Get Any Azure Data Factory Pipeline Activity Error Details with Azure Functions

A quick blog friends… I’ve done a few different thing now with Azure Functions and Azure Data Factory (ADF). They are definitely two of my favourite Azure Resources. In previous post I’ve:

• Executed Any Azure Data Factory Pipeline with an Azure Function
• Got Any Azure Data Factory Pipeline Run Status with Azure Functions

Building on this theme I’ve put together a similar Function to now return the error details for our failed Pipeline Activities.

Via the ADF monitoring portal something like the below might be a common sight (no judgement!). However, in my case, as the Pipeline name suggests this is an ‘Intentional Error’ 🙂

My pipeline contains three Activities:

1. A simple Wait, left with all default values.
2. An Azure SQLDB Stored Procedure call.
3. An Azure Databricks Notebook call.

In my Stored Procedure and Notebook I’ve done nothing more than raise/throw exceptions using a Pipeline parameter to establish if I actually want an error status to be created. T-SQL and Scala code snippets below.

Anyway, these bits aren’t really the point of the blog.

The main thing to consider is how these error details are reported programmatically, via C# in my case, from the ADF Pipeline run. The main class used is called ‘Query By Pipeline Run‘ which in the .NET SDK is available via the DataFactoryManagementClient. This query response contains details of everything about the pipeline run and all executed Activities; success or fail. Therefore exacting the error message information can be a little tricky when presented with a huge response body for every Activity. This is my motivation for wanting to simplify things into a targeted Azure Function call. Furthermore, given my Pipeline structure above an array is required if we need to deal with multiple Activities responses.

Get Activity Error Details

In my Function, after creating the ADF client, I firstly query my Pipeline using the Run ID as the primary filter and use these to get the Activity Run details.

Next, I parse the response to extract the error information required for each failed Activity and construct my own cleaner Function response content.

The output for the ADF Pipeline shown above looks like this, via Postman:

I hope you agree this refined response to the Activity run(s) is a lot nicer when error details is really all you want to know.

As with my previous blogs the Function body should contain the following details:
{
"tenantId": "1234-1234-1234-1234-1234",
"applicationId": "1234-1234-1234-1234-1234",
"authenticationKey": "Passw0rd123!",
"subscriptionId": "1234-1234-1234-1234-1234",
"resourceGroup": "CommunityDemos",
"factoryName": "PaulsFunFactoryV2",
"pipelineName": "Intentional Error",
"runId": "1234-1234-1234-1234-1234"
}

Feel free to grab the code from my usual GitHub repo…

---

Blog Supporting Content in my GitHub repository:
https://github.com/mrpaulandrew/BlobSupportingContent/{Blog_Title}

---

Also, for those following my ADF.procfwk blogs you can probably guess where I’m heading with this and the reason for wanting to develop a discrete Function to get Activity error details 😉

Many thanks for reading.