ADF.procfwk and Azure Synapse Orchestrate – Preview Limitations

With Azure Synapse Analytics now in public preview is was time to find out how compatible my Azure Data Factory metadata driven processing framework (ADF.procfwk) is with the Synapse Orchestrate features.

Firstly, as Synapse doesn’t yet have any source control or DevOps support I had to manually rebuild the framework pipelines in the browser, copying content via the ADF developer UI to the Synapse Workspace. Not a fun experience, but hopefully this will be a one off!

Next, I can happily report that the framework pipeline hierarchy (or core pipeline) are compatible when running from Synapse Orchestrate. This includes the framework pipelines:

• Grandparent
• Parent
• Child
• Infant

So, on the surface this is good start. However, we aren’t done yet.

We do sadly have an important limitation if we choose to use Synapse and want to inject the procfwk into the Orchestrate area. This limitation relates to our Worker pipelines and how they are called. The framework (as you may know) uses a set of Azure Function to execute, check and capture Activity errors.

The framework Functions use the Microsoft NuGet package and libraries:

• Microsoft.Azure.Management.DataFactory
• Microsoft.Azure.Management.DataFactory.Models
  • https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.management.datafactory.models

The Synapse versions are unsurprisingly still in pre-release:

• Microsoft.Azure.Management.Synapse
• Microsoft.Azure.Management.Synapse.Models
  • https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.synapse.models

Also, critically the Synapse.Models namespace doesn’t yet have an equivalent to the ADF PipelineRun class.

For the framework this means that core pipelines can run in Synapse, but they can only interact with Worker pipelines in Data Factory.

To visualise that in the context of all orchestration scenarios if Worker pipelines are separate or combined with the core (Grandparent, Parent, Child and Infant) pipelines. What is currently supported?

Setup	Core Pipelines		Worker Pipeline(s)	Supported
1
2
3
4
5
6

Summary

To summarise, if Worker pipelines have been created in Azure Synapse Orchestrate they cannot be called by the current release of ADF.procfwk due to a limitation in the pre-release version of the Microsoft SDK.

I’ll keep a close eye on the affected name spaces and post updates once available.

Thanks for reading.