Get Any Azure Data Factory Pipeline Activity Error Details with Azure Functions

A quick blog friends… I’ve done a few different thing now with Azure Functions and Azure Data Factory (ADF). They are definitely two of my favourite Azure Resources. In previous post I’ve:

Building on this theme I’ve put together a similar Function to now return the error details for our failed Pipeline Activities.

Via the ADF monitoring portal something like the below might be a common sight (no judgement!). However, in my case, as the Pipeline name suggests this is an ‘Intentional Error’ πŸ™‚

My pipeline contains three Activities:

  1. A simple Wait, left with all default values.
  2. An Azure SQLDB Stored Procedure call.
  3. An Azure Databricks Notebook call.

In my Stored Procedure and Notebook I’ve done nothing more than raise/throw exceptions using a Pipeline parameter to establish if I actually want an error status to be created. T-SQL and Scala code snippets below.

Anyway, these bits aren’t really the point of the blog.

The main thing to consider is how these error details are reported programmatically, via C# in my case, from the ADF Pipeline run. The main class used is called ‘Query By Pipeline Run‘ which in the .NET SDK is available via the DataFactoryManagementClient. This query response contains details of everything about the pipeline run and all executed Activities; success or fail. Therefore exacting the error message information can be a little tricky when presented with a huge response body for every Activity. This is my motivation for wanting to simplify things into a targeted Azure Function call. Furthermore, given my Pipeline structure above an array is required if we need to deal with multiple Activities responses.

Get Activity Error Details

In my Function, after creating the ADF client, I firstly query my Pipeline using the Run ID as the primary filter and use these to get the Activity Run details.

Next, I parse the response to extract the error information required for each failed Activity and construct my own cleaner Function response content.

The output for the ADF Pipeline shown above looks like this, via Postman:

I hope you agree this refined response to the Activity run(s) is a lot nicer when error details is really all you want to know.

As with my previous blogs the Function body should contain the following details:
{
"tenantId": "1234-1234-1234-1234-1234",
"applicationId": "1234-1234-1234-1234-1234",
"authenticationKey": "Passw0rd123!",
"subscriptionId": "1234-1234-1234-1234-1234",
"resourceGroup": "CommunityDemos",
"factoryName": "PaulsFunFactoryV2",
"pipelineName": "Intentional Error",
"runId": "1234-1234-1234-1234-1234"
}

Feel free to grab the code from my usual GitHub repo…


Blog Supporting Content in my GitHub repository:
https://github.com/mrpaulandrew/BlobSupportingContent/{Blog_Title}


Also, for those following my ADF.procfwk blogs you can probably guess where I’m heading with this and the reason for wanting to develop a discrete Function to get Activity error details πŸ˜‰

Many thanks for reading.

4 thoughts on “Get Any Azure Data Factory Pipeline Activity Error Details with Azure Functions

  1. Hi Paul, very useful content, thanks for sharing.. Is there way achieve the same without using scala or any other databricks code ? sorry if it is a daff question !

    Like

    1. Hi, interesting one, you can certainly hit an Azure Function from a Databricks Notebook if you want. My first thoughts are why though? Normally I’d expect ADF to be calling Databricks. So why do you want error details about an ADF pipeline in a Notebook? To do what? Cheers

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.