A quick blog friends… I’ve done a few different thing now with Azure Functions and Azure Data Factory (ADF). They are definitely two of my favourite Azure Resources. In previous post I’ve:
- Executed Any Azure Data Factory Pipeline with an Azure Function
- Get Any Azure Data Factory Pipeline Run Status with Azure Functions
Building on this theme I’ve put together a similar Function to now return the error details for our failed Pipeline Activities.
Via the ADF monitoring portal something like the below might be a common sight (no judgement!). However, in my case, as the Pipeline name suggests this is an ‘Intentional Error’ 🙂
My pipeline contains three Activities:
- A simple Wait, left with all default values.
- An Azure SQLDB Stored Procedure call.
- An Azure Databricks Notebook call.
In my Stored Procedure and Notebook I’ve done nothing more than raise/throw exceptions using a Pipeline parameter to establish if I actually want an error status to be created. T-SQL and Scala code snippets below.
Anyway, these bits aren’t really the point of the blog.
The main thing to consider is how these error details are reported programmatically, via C# in my case, from the ADF Pipeline run. The main class used is called ‘Query By Pipeline Run‘ which in the .NET SDK is available via the DataFactoryManagementClient. This query response contains details of everything about the pipeline run and all executed Activities; success or fail. Therefore exacting the error message information can be a little tricky when presented with a huge response body for every Activity. This is my motivation for wanting to simplify things into a targeted Azure Function call. Furthermore, given my Pipeline structure above an array is required if we need to deal with multiple Activities responses.
Get Activity Error Details
In my Function, after creating the ADF client, I firstly query my Pipeline using the Run ID as the primary filter and use these to get the Activity Run details.
Next, I parse the response to extract the error information required for each failed Activity and construct my own cleaner Function response content.
The output for the ADF Pipeline shown above looks like this, via Postman:
I hope you agree this refined response to the Activity run(s) is a lot nicer when error details is really all you want to know.
As with my previous blogs the Function body should contain the following details:
{
"tenantId": "1234-1234-1234-1234-1234",
"applicationId": "1234-1234-1234-1234-1234",
"authenticationKey": "Passw0rd123!",
"subscriptionId": "1234-1234-1234-1234-1234",
"resourceGroup": "CommunityDemos",
"factoryName": "PaulsFunFactoryV2",
"pipelineName": "Intentional Error",
"runId": "1234-1234-1234-1234-1234"
}
Feel free to grab the code from my usual GitHub repo…
Blog Supporting Content in my GitHub repository:
https://github.com/mrpaulandrew/BlobSupportingContent/{Blog_Title}
Also, for those following my ADF.procfwk blogs you can probably guess where I’m heading with this and the reason for wanting to develop a discrete Function to get Activity error details 😉
Many thanks for reading.
Hi Paul, very useful content, thanks for sharing.. Is there way achieve the same without using scala or any other databricks code ? sorry if it is a daff question !
LikeLike
Hi, interesting one, you can certainly hit an Azure Function from a Databricks Notebook if you want. My first thoughts are why though? Normally I’d expect ADF to be calling Databricks. So why do you want error details about an ADF pipeline in a Notebook? To do what? Cheers
LikeLike
This function reminds me of the old script component we would all have to write to get the “ErrorColumnName” within an SSIS Pipeline.
This really is great. You’re the Jamie Thompson and Koen Verbeeck of ADF!
LikeLike
Erm, thanks! This post was just part of a wider framework project. Check out procfwk.com
LikeLike
Hi Paul,
Thank you for your post.
I’m actually developing a pipeline including a Foreach box.
When my pipeline is KO, i only retrieve the first error level : Operation on target ForEach1 failed: Activity failed because an inner activity failed
Is there a way to retrieve the error occured in my Foreach box.
Many thanks !
LikeLike
Hey, I suggest you use a child pipeline for the inner ForEach activities. Then inspect the run id of the child to get the second level error information. Not perfect. But works. Cheers
LikeLike
Hi Paul,
Thank you for your post.
I’m actually developing a pipeline including a Foreach box.
When my pipeline is KO, i only retrieve the first error level : Operation on target ForEach1 failed: Activity failed because an inner activity failed
Is there a way to retrieve the error occured in my Foreach box.
Many thanks !
LikeLike