From my YouTube channel: https://www.youtube.com/c/mrpaulandrew Special guest Richard Swinbank talks about how you can use an NUnit project in Visual Studio to automate the testing of Data Factory pipelines. Richard kindly helped me apply this and test the pipelines in the ADF.procfwk to ensure post release everything still works in the open source code project. … Continue reading Using NUnit to Automate the Testing of Azure Data Factory Pipelines
Tag: Azure Data Factory
How To Use ‘Specify dynamic contents in JSON format’ in Azure Data Factory Linked Services
Make any Azure Data Factory Linked Service dynamic! In a few different community circles I've been asked 'how to handle dynamic Linked Service connections in Azure Data Factory if the UI doesn't naturally support the addition of parameters'. Depending on the Linked Service the support for this varies. Well, the answer, or should I say, … Continue reading How To Use ‘Specify dynamic contents in JSON format’ in Azure Data Factory Linked Services
ADF.procfwk v1.8 – Complete Pipeline Dependency Chains For Failure Handling
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.8 – Complete Pipeline Dependency Chains For Failure Handling
ADF.procfwk v1.7.2 – NULL Pipeline Parameters Handled
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.7.2 – NULL Pipeline Parameters Handled
ADF.procfwk – The 300! Scaled Out Worker Pipeline Test
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk – The 300! Scaled Out Worker Pipeline Test
ADF.procfwk v1.7.1 – Alerting Bug Fix And Pipeline Parameter Size Limit Removed
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.7.1 – Alerting Bug Fix And Pipeline Parameter Size Limit Removed
ADF.procfwk – Video Setup How To Guide
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk – Video Setup How To Guide
ADF.procfwk v1.7 – Pipeline Email Alerting
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.7 – Pipeline Email Alerting
ADF.procfwk and Azure Synapse Orchestrate – Preview Limitations
With Azure Synapse Analytics now in public preview is was time to find out how compatible my Azure Data Factory metadata driven processing framework (ADF.procfwk) is with the Synapse Orchestrate features. Firstly, as Synapse doesn't yet have any source control or DevOps support I had to manually rebuild the framework pipelines in the browser, copying … Continue reading ADF.procfwk and Azure Synapse Orchestrate – Preview Limitations
When Should I Use Multiple Azure Data Factory’s?
Hi all, I get asked this question a lot. So I decided to do a full post on the subject based on my current thoughts and experience. When should I use multiple Azure Data Factory's? The consultants answer, it depends 😉 Ok, let's go deeper and explore why it depends! Also, to be clear, we … Continue reading When Should I Use Multiple Azure Data Factory’s?
