A quick blog friends… I’ve done a few different thing now with Azure Functions and Azure Data Factory (ADF). They are definitely two of my favourite Azure Resources. In previous post I’ve:
- Executed Any Azure Data Factory Pipeline with an Azure Function
- Get Any Azure Data Factory Pipeline Run Status with Azure Functions
Building on this theme I’ve put together a similar Function to now return the error details for our failed Pipeline Activities.
Via the ADF monitoring portal something like the below might be a common sight (no judgement!). However, in my case, as the Pipeline name suggests this is an ‘Intentional Error’ 🙂
My pipeline contains three Activities:
- A simple Wait, left with all default values.
- An Azure SQLDB Stored Procedure call.
- An Azure Databricks Notebook call.
In my Stored Procedure and Notebook I’ve done nothing more than raise/throw exceptions using a Pipeline parameter to establish if I actually want an error status to be created. T-SQL and Scala code snippets below.
Anyway, these bits aren’t really the point of the blog.
The main thing to consider is how these error details are reported programmatically, via C# in my case, from the ADF Pipeline run. The main class used is called ‘Query By Pipeline Run‘ which in the .NET SDK is available via the DataFactoryManagementClient. This query response contains details of everything about the pipeline run and all executed Activities; success or fail. Therefore exacting the error message information can be a little tricky when presented with a huge response body for every Activity. This is my motivation for wanting to simplify things into a targeted Azure Function call. Furthermore, given my Pipeline structure above an array is required if we need to deal with multiple Activities responses.
Get Activity Error Details
In my Function, after creating the ADF client, I firstly query my Pipeline using the Run ID as the primary filter and use these to get the Activity Run details.
Next, I parse the response to extract the error information required for each failed Activity and construct my own cleaner Function response content.
The output for the ADF Pipeline shown above looks like this, via Postman:
I hope you agree this refined response to the Activity run(s) is a lot nicer when error details is really all you want to know.
As with my previous blogs the Function body should contain the following details:
{
"tenantId": "1234-1234-1234-1234-1234",
"applicationId": "1234-1234-1234-1234-1234",
"authenticationKey": "Passw0rd123!",
"subscriptionId": "1234-1234-1234-1234-1234",
"resourceGroup": "CommunityDemos",
"factoryName": "PaulsFunFactoryV2",
"pipelineName": "Intentional Error",
"runId": "1234-1234-1234-1234-1234"
}
Feel free to grab the code from my usual GitHub repo…
Blog Supporting Content in my GitHub repository:
https://github.com/mrpaulandrew/BlobSupportingContent/{Blog_Title}
Also, for those following my ADF.procfwk blogs you can probably guess where I’m heading with this and the reason for wanting to develop a discrete Function to get Activity error details 😉
Many thanks for reading.
Welcome to the Blog of Paul Andrew 
Hi Paul, very useful content, thanks for sharing.. Is there way achieve the same without using scala or any other databricks code ? sorry if it is a daff question !
LikeLike
Hi, interesting one, you can certainly hit an Azure Function from a Databricks Notebook if you want. My first thoughts are why though? Normally I’d expect ADF to be calling Databricks. So why do you want error details about an ADF pipeline in a Notebook? To do what? Cheers
LikeLike
This function reminds me of the old script component we would all have to write to get the “ErrorColumnName” within an SSIS Pipeline.
This really is great. You’re the Jamie Thompson and Koen Verbeeck of ADF!
LikeLike
Erm, thanks! This post was just part of a wider framework project. Check out procfwk.com
LikeLike
Hi Paul,
Thank you for your post.
I’m actually developing a pipeline including a Foreach box.
When my pipeline is KO, i only retrieve the first error level : Operation on target ForEach1 failed: Activity failed because an inner activity failed
Is there a way to retrieve the error occured in my Foreach box.
Many thanks !
LikeLike
Hey, I suggest you use a child pipeline for the inner ForEach activities. Then inspect the run id of the child to get the second level error information. Not perfect. But works. Cheers
LikeLike
Hi Paul,
Thank you for your post.
I’m actually developing a pipeline including a Foreach box.
When my pipeline is KO, i only retrieve the first error level : Operation on target ForEach1 failed: Activity failed because an inner activity failed
Is there a way to retrieve the error occured in my Foreach box.
Many thanks !
LikeLike
Thank you Andrew, Is there any way I can get latest pipeline run id from ADF monitor system for a specific pipeline?
Also, if we use parent pipeline name , does it provides error for child pipeline as well?
LikeLike
Yes, check out my other post on getting the pipeline status. It has a set of filter params to find the latest run ID.
LikeLike
I have a data factory with multiple child pipelines and there are activities within which might run into exception . Is there a way to bubble up the exception to the parent pipeline error message?
LikeLike
In short, no. Check out how I structure pipelines as part of my processing framework… procfwk.com
LikeLike
Hi ,I saw the code in github .Great work
I am not able to find where is my application id for synapse is?
Can you help me with that?
LikeLike
Check out the latest implementation of the functions in my procfwk repo. They include beta support for Synapse integration pipelines.
LikeLike
Hi Paul, thank you for your write ups, they have been very help. I recently was able to log my pipeline runs in an SQL database with logic app but what I couldn’t get was being able to link child/nested pipeline runs with their parent or grandparent pipepline run. Is there a parameter that links them together so I’m able to do a view of a parent and child pipeline runs.
LikeLike
Paul, thanks for the code. Looks great! Just wondering, what triggers the function? Is it running in a scheduled manner and extracts all pipelines with errors? So say run every 5 min and get any pipeline error?
LikeLike
Thanks a lot Paul for this beautiful work, i would like to know is there a way to set up a retry function for every ADF pipeline instance that fails? in other words, if a pipeline fails, i want to ensure that there is a retry mechanism in place that re-executes the failed pipeline instance multiple times in a parametrizable way. like a loop with a set number of retries inside a function called after every failed pipeline due to an unexpected event. Thanks a lot !
LikeLike