Walking the Walk With Real-Time Analytics
I hope you enjoyed 2023 as much we have. Founding my own data consultancy business with Dan was certainly a highlight. But then, to put that machine, that experience and that passion for technology into practice, delivering not one, but two customer projects is simply the best feeling. And using Microsoft Fabric.
In this blog I want to share with you some of the technical thinking that went into those platform architectures, continuing the thought process from my other blogs around Microsoft Fabric as the product matures.
Marketing Analytics Platform with Outreach insights and Campaign Tracking
The solution involves a near real-time stream of marketing outreach data into Power BI dashboards to provide up to the minute analytics on the performance of campaigns, this informs future content and audience enrolment for a professional services customer.
The raw data source APIs needed some wrapping with Azure Functions before they could be accessed directly by Microsoft Fabric. Early development work used Web Activities within the Fabric Integration pipeline experience, but sadly, the handling of the payload wasn’t complex enough for the requirements and Copy activity support was limited at the time of delivery.
Beyond the data ingestion, the approach followed a classic lambda architecture with cold/batch data supplementing the near real-time feed when polled by a simple Python Notebooks.
- Download the complete case study on the Cloud Formations website here. >>>
Satellite Internet (Starlink) Telemetry Analytics Platform
The requirement was for a marine electronic service provider was for a near real-time feed of Starlink usage telemetry, but from an API data source, therefore we had to get creative with the ingestion handling. Using a highly scaled set of Azure Functions, bootstrapped with Fabric Integration pipelines to deliver a constant feed of data from the source. The pipelines and nested levels of functions were called using metadata making the ingestion highly dynamic and easy to update for support teams.
Downstream the functions wrote messages to an Event Stream endpoint before passing content to a KQL database, which supported a push dataset to the resulting Power BI dashboards.
Supplementary reference datasets were handling using Notebooks, again due to limitations in the pipeline Copy activity when writing API payloads to the Lakehouse.
- Download the complete case study on the Cloud Formations website here. >>>
Many thanks for reading.
Looking forward to 2024!
Welcome to the Blog of Paul Andrew 
Very interesting, can you please provide details on the SQL endpoint? Is it SQL MI?
LikeLike
No, the default SQL endpoint made available with the Lakehouse experience in Fabric.
LikeLike
what do you think about the missing git integration of fabric. Didn’t it complicate the development for you?
LikeLike
Yes, it made the dev/ops story difficult and requiring manual updates for some items. Less than ideal. Especially for the Event Stream artifact which requires data inputs before the output connectors can be provisioned.
LikeLike