Considering a Medallion Architecture vs Microsoft Fabric

Data Ingestion vs Data Serving

Context

It seems like the concept of a medallion architecture has stuck in the industry. ETL isn’t cool anymore and decribing the layers of our data solution directly with names like raw, cleansed and modelled was (apparently) too obvious, or maybe too interchangable, meaning people couldn’t agree. That’s fine. I don’t mind. If we are all now happy using metal to describe how we process data thats good enough for me. Bronze, Silver, Gold it is.

To continue this then, how do we now wrap that metal with some fabric. Or some cloth, maybe leather! Ok sorry. I’ll stop 🙂

In this post, I want to explore the conceptual layers of our data solution vs the technology we use to implement them.

---

Current Stance

Microsoft Fabric gives us a unified set of tools for data processing and at the same time has managed to deliver both centralised and de-centralised concepts for organising resources. Great. Given that, and based on these capabilities (and a maturity timeline) I’m suggesting we do the following with our layers:

1. Use hardened Azure Data Platform resources to ingest our data.
2. Use versitile Microsoft Fabric experiences to model and serve our data.

Or, to be explicit.

1. Bronze – Azure Data Platform.
2. Gold – Microsoft Fabric.

Today, given the technical capabilitites offered, Silver could be Azure and/or Fabric, depending on the complexity of the source data. Visualised below, with some layer characteristics included.

To continue this thinking, given what we know about Microsoft Fabric’s OneLake, storing everything as Delta Lake tables, I don’t think we can consider using it for our Bronze layer. Furthermore, Microsoft Fabric is currently lacking a lot of connectors when dealing with disparate and third party hosted data sources.

Where Microsoft Fabric does help us, is if we mount/shortcut our bronze, silver layers in our LakeHouse for further downstream processing. Including data modelling and the ability to create logical domains for reporting outputs.

With all of the above, I’m happy to be wrong. As mentioned we need to consider a timeline in what I’m suggesting. As the Microsoft Fabric ingestion capabilities improve the green will probably start to dominate over the blue.

What do you think?

---

Many thanks for reading.