Just last week we heard the announcement from Microsoft that Azure Synapse Analytics is now generally available (GA)… A full year on, plus a few weeks, since first seeing Synapse at the big USA conferences in November 2019.
Today I’ve been attempting to use the resource with a view to implementing it for several customer projects. Although GA, it would seem that many part of the technology are far from ready.
In this brief blog I’m exposing some of the pain I’ve faced so far in simply trying to deploy a second instance of Azure Synapse Analytics using ARM templates.
Part 1 – Deploying the Workspace
Using the Azure portal to manually deploy a vanilla Synapse Workspace yields the following auto generated ARM template, snippet image:
Get the full JSON file from GitHub here.
Straight away just skimming the 260 lines of JSON reveals:
- A bunch of ‘preview’ API calls. Not GA then?
- Firewall rules created with open IP address ranges 0.0.0.0 – 255.255.255.255.
- SQL authentication admin credentials that don’t require passwords and can be left blank.
- A mass of parameters to configure things not exposed via the portal UI and without a great deal of context.
- Hard coded resource security role GUID’s.
- An apparent disconnect between the workspace resource and the required Data Lake storage. Maybe this is a good thing.
I expect most will have to do a lot of cleaning up here before this template can be using for a production deployment. Anyway, once you have this, stash this somewhere.
Part 2 – Deploying Workspace Artifacts
Assuming you can live with the Workspace defaults etc, you’ll next need to connect it to a source code repository using either GitHub or Azure DevOps. This follows a very similar approach to Azure Data Factory, if your familiar, by setting up its own xxxxx_publish branch in the repo.
In the publish branch, you’ll then find a file named TemplateForWorkspace.json. This includes some, not all, of your Synapse Workspace developments:
- Linked Services
- SQL Scripts
These can also be viewed separately in your feature branches, all as separate JSON files. Even the SQL Scripts are created as JSON.
If you don’t yet have the TemplateForWorkspace.json file you simply need to Commit and Publish your workspace content via the UI (the same as Data Factory).
However, a current bug in the GA service means that if you do this more than once you’ll probably get a write error…
Apparently the Synapse can’t overwrite the TemplateForWorkspace.json file in the publish branch if it already exists using the *cough* Team Foundation Web Server! The only workaround I’ve found is to navigate to the publish branch and delete the auto generated template and parameter files before hitting the publish toolbar button a again!… Come on Microsoft!
Maybe stash this somewhere as well, or use it directly from the publish branch.
Moving on again.
Part 3 – A Release Pipeline
Assuming once again you can live with these issues, it is possible to create a release pipeline for all Synapse things.
You’ll need a think about which code branch to use if you want to include the auto generated workspace artifacts template and you’ll need to define your own template for the workspace itself.
All of the above is far from ideal. For production we will need something much better including Terraform and PowerShell support.
This is just where we are at with the deployment of Azure Synapse Analytics today using ARM templates.
I hope things will improve very soon in Synapse and we don’t need to wait another year for some basic resource features and resilience.
Many thanks for reading.