Automated DQM Framework
A reusable PySpark data-quality accelerator on Microsoft Fabric that automates schema extraction, rule generation, validation, threshold monitoring, and alerting across production-scale lakehouse pipelines.
What was broken
Data-quality checks were manual, repetitive, and hard to scale across large tables and pipelines. Teams needed a reusable framework that could validate schemas, enforce data rules, detect anomalies, and raise alerts without bespoke checks for every new table.
How I built it
I built an automated DQM framework on Microsoft Fabric using Python and PySpark. It dynamically extracts schemas from lakehouse tables, generates a config-driven rule set (nulls, duplicates, schema drift, data types, key uniqueness, row count reconciliation, threshold anomalies, referential and range checks, freshness, and pipeline SLA monitoring), runs validations as Fabric pipeline steps across Bronze / Silver / Gold layers, persists results, and fires threshold-based alerts with reporting on top.
The reasoning
DQM has to scale with the lakehouse, not behind it. Auto-generating rules from schema means coverage grows as new tables land; running checks in PySpark on the Medallion layers means validation happens where the data already lives; config-driven rules keep changes reviewable instead of silent.
How the pieces fit
Key components
- 01Dynamic schema extraction across any Delta / Lakehouse table
- 02Config-driven rule generation for reusable, reviewable validations
- 03Null, duplicate, schema drift, data type, and primary / business key checks
- 04Row count reconciliation, threshold anomaly, referential, range, and freshness checks
- 05Pipeline failure / SLA monitoring as first-class DQM signals
- 06Result persistence in dedicated DQM tables with full traceability
- 07Threshold-based alerting wired into Azure Monitor
- 08Reusable framework design for new Fabric projects on day one
Tech stack
In-browser demo
A scripted, in-browser walkthrough of a real run, traced step by step. Press play to watch the agents fire.