Just last week we heard the announcement from Microsoft that Azure Synapse Analytics is now generally available (GA)… A full year on, plus a few weeks, since first seeing Synapse at the big USA conferences in November 2019.
Today I’ve been attempting to use the resource with a view to implementing it for several customer projects. Although GA, it would seem that many part of the technology are far from ready.
In this brief blog I’m exposing some of the pain I’ve faced so far in simply trying to deploy a second instance of Azure Synapse Analytics using ARM templates.
Part 1 – Deploying the Workspace
Using the Azure portal to manually deploy a vanilla Synapse Workspace yields the following auto generated ARM template, snippet image:
Get the full JSON file from GitHub here.
Straight away just skimming the 260 lines of JSON reveals:
- A bunch of ‘preview’ API calls. Not GA then?
- Firewall rules created with open IP address ranges 0.0.0.0 – 255.255.255.255.
- SQL authentication admin credentials that don’t require passwords and can be left blank.
- A mass of parameters to configure things not exposed via the portal UI and without a great deal of context.
- Hard coded resource security role GUID’s.
- An apparent disconnect between the workspace resource and the required Data Lake storage. Maybe this is a good thing.
I expect most will have to do a lot of cleaning up here before this template can be using for a production deployment. Anyway, once you have this, stash this somewhere.
Moving on.
Part 2 – Deploying Workspace Artifacts
Assuming you can live with the Workspace defaults etc, you’ll next need to connect it to a source code repository using either GitHub or Azure DevOps. This follows a very similar approach to Azure Data Factory, if your familiar, by setting up its own xxxxx_publish branch in the repo.
In the publish branch, you’ll then find a file named TemplateForWorkspace.json. This includes some, not all, of your Synapse Workspace developments:
- Credentials
- Pipelines
- Datasets
- Linked Services
- Notebooks
- SQL Scripts
These can also be viewed separately in your feature branches, all as separate JSON files. Even the SQL Scripts are created as JSON.
If you don’t yet have the TemplateForWorkspace.json file you simply need to Commit and Publish your workspace content via the UI (the same as Data Factory).
However, a current bug in the GA service means that if you do this more than once you’ll probably get a write error…
Apparently the Synapse can’t overwrite the TemplateForWorkspace.json file in the publish branch if it already exists using the *cough* Team Foundation Web Server! The only workaround I’ve found is to navigate to the publish branch and delete the auto generated template and parameter files before hitting the publish toolbar button a again!… Come on Microsoft!
Maybe stash this somewhere as well, or use it directly from the publish branch.
Moving on again.
Part 3 – A Release Pipeline
Assuming once again you can live with these issues, it is possible to create a release pipeline for all Synapse things.
You’ll need a think about which code branch to use if you want to include the auto generated workspace artifacts template and you’ll need to define your own template for the workspace itself.
All of the above is far from ideal. For production we will need something much better including Terraform and PowerShell support.
This is just where we are at with the deployment of Azure Synapse Analytics today using ARM templates.
I hope things will improve very soon in Synapse and we don’t need to wait another year for some basic resource features and resilience.
Many thanks for reading.
Welcome to the Blog of Paul Andrew 
Hi Mr Paul Andrew,
I have tried to deploy my ARM template in multiple ways (via azure portal and release pipeline) but always get to the same issue.
Deploying an empty Synapse workspace works.
The problems start when I add a dataset or a synapse pipeline for example and then try to run the release pipeline.
For some reason i always get the error MissingApiVersionParameter: The api-version query parameter is required for al requests.
This is strange to me as it was synapse it self who generated these ARM templates and for each resource there is an apiversion defined.
Did you have any experience with that?
Btw, thanks for the article
With kind regards
Ward
LikeLike
Hi, no, I didn’t have that issue. Synapse is a moving target at the moment so I expect the blog post content is already out of date while Microsoft attempts to get the GA service working as expected.
Cheers
Paul
LikeLike
Hey Paul,
I just ran into this problem and have spent the last 2 days solving it. I think the issue is that the documentation for CI/CD and azure synapse is quite opaque. It seems that there are two ARM templates required. The first template is used only to instantiate the resources for the azure synapse instance and any other required resources (dedicated pools, etc). The published templates from synapse (found in workspace_publish) are only used in the “Synapse deployment task for workspace”.
I think the easiest way is to create the difference synapse environments through the azure portals and skip the ARM template deployment task. Your deployment will only have one task, which is the “Synapse deployment task for workspace”, which will copy over workspace details.
Cheers,
Zach
LikeLike
Hi Zach,
Did you actually manage to get the “Synapse deployment task for workspace” working? I have been trying this for days now with absolutely no success?
Regarding the API-version error problem, I have the same issue so have just skipped the first part of the MS documentation that details the ARM Template Deployment, it just doesn’t work. Now trying just the second part, having manually setup my resources in another environment.
Thanks.
LikeLike
Hello Ward, i’m having the same problem. When i try to release Synapse trough DevOps I also receive the error “The api-version query parameter (?api-version=) is required for all requests.”
Did you already solved this issue?
With kind regards,
Mark
LikeLike
Hi Paul Andrew,
Thanks for sharing nice information about the new Azure Deployment process. This blog post more useful to the Azure and DevOps experts.
LikeLike
Hi Paul,
Here’s another little gem for your growing list… try creating a Managed VNet via the ARM template!
If you create a Synapse instance via the portal there’s a tick-box on the Networking tab “Enable managed virtual network”. Ticking that box and then reviewing the ARM template prior to deployment reveals the property “managedVirtualNetwork”: “default” and one below it “managedVirtualNetworkSettings” with whatever you select for the linked tenants. Continuing with the deployment via the portal is all tickety-boo. And navigating to Synapse Studio -> Manage -> Integration Runtimes -> New -> Azure will reveal the ‘Virtual network configuration’ boxes enabled and available for use.
However (and here’s the fun bit….) if you create an ARM template with exactly the same VNet settings as the auto-generated ARM templates and then deploy that, the ‘Virtual network configuration’ boxes are disabled
Furthermore, since a Managed VNet *can only* be defined at the point of creation and not afterwards, you’re screwed. Especially as MS Docs say managed VNets for connecting to data sources is the recommended practice. The only recourse is to tear the DevOps deployed Synapse instance down and re-do it via the portal.
We’re not at the point of deploying this to UAT or Prod yet, but where I work now developers don’t have access to those environments, so manual creation is a no-go. We’re fingers crossed this gets fixed before we get to that stage.
I’ve tried various values in the “managedVirtualNetwork” property: “default”, “Enabled”, true, “”, “On”, “Yes”, “JFDI you annoying contraption” and none of them work. Not even that last one (although it was satisfying to try that at the time).
There’s also *no* property for it in the Azure CLI command “az synapse workspace create”
So MS position is: managed VNets are recommended best practice, but neither ARM nor CLI contain a working command to enable it. Fun, eh?
John
LikeLike
Ha, nice, thanks John
LikeLike
Hi John, Paul, nice read! Did you manage to solve the issue with the managedVirtualNetwork property?
LikeLike