With Azure Synapse Analytics now in public preview is was time to find out how compatible my Azure Data Factory metadata driven processing framework (ADF.procfwk) is with the Synapse Orchestrate features.
Firstly, as Synapse doesn’t yet have any source control or DevOps support I had to manually rebuild the framework pipelines in the browser, copying content via the ADF developer UI to the Synapse Workspace. Not a fun experience, but hopefully this will be a one off!
Next, I can happily report that the framework pipeline hierarchy (or core pipeline) are compatible when running from Synapse Orchestrate. This includes the framework pipelines:
So, on the surface this is good start. However, we aren’t done yet.
We do sadly have an important limitation if we choose to use Synapse and want to inject the procfwk into the Orchestrate area. This limitation relates to our Worker pipelines and how they are called. The framework (as you may know) uses a set of Azure Function to execute, check and capture Activity errors.
The framework Functions use the Microsoft NuGet package and libraries:
The Synapse versions are unsurprisingly still in pre-release:
Also, critically the Synapse.Models namespace doesn’t yet have an equivalent to the ADF PipelineRun class.
For the framework this means that core pipelines can run in Synapse, but they can only interact with Worker pipelines in Data Factory.
To visualise that in the context of all orchestration scenarios if Worker pipelines are separate or combined with the core (Grandparent, Parent, Child and Infant) pipelines. What is currently supported?
|Setup||Core Pipelines||Worker Pipeline(s)||Supported|
To summarise, if Worker pipelines have been created in Azure Synapse Orchestrate they cannot be called by the current release of ADF.procfwk due to a limitation in the pre-release version of the Microsoft SDK.
I’ll keep a close eye on the affected name spaces and post updates once available.
Thanks for reading.