DataShield
Real-time data observability: quality, lineage, and anomaly detection.
Problem
Data platforms fail silently. A broken upstream job or a schema drift can corrupt dozens of downstream tables before anyone notices, and engineers lack a clear view of blast radius.
Solution
An observability layer continuously checks data quality, maps lineage to compute blast radius, and uses ML to detect anomalies before they propagate, all surfaced through an interactive demo.
Architecture
A data-quality engine evaluates rules and statistics over datasets, a lineage graph traces dependencies to compute blast radius, and an ML anomaly detector flags unusual patterns. Everything is exposed through FastAPI with an interactive Streamlit demo.
Tech stack
Data QualityLineageBlast RadiusML Anomaly DetectionFastAPIStreamlit