Case Study
MDQM / MDM Data Governance Platform
Enterprise master data management for dataset governance, quality validation, and metadata enrichment.
Overview
MDQM is a full-stack Master Data Management platform built to help data teams govern, validate, and profile datasets before they enter analytics pipelines. The platform combines a React dashboard with a FastAPI backend, PostgreSQL for operational storage, and Snowflake for analytics workloads — delivering a unified experience for data stewards and engineers.
Problem Statement
Enterprise teams often struggle with inconsistent master data across systems. Manual validation is error-prone, metadata is scattered, and there's no single place to profile incoming datasets or enforce quality rules before data reaches Snowflake warehouses. MDQM was built to solve this by providing centralized dataset management, automated validation, and AI-assisted metadata enrichment.
Solution Architecture
The platform follows a three-tier architecture: a React + Tailwind CSS frontend for dashboards and dataset workflows, a FastAPI service layer exposing RESTful APIs with JWT authentication, and a dual-database backend — PostgreSQL for transactional operations (users, rules, metadata) and Snowflake for analytics queries and warehouse-scale profiling. Docker containers orchestrate local and deployment environments, with CI/CD via GitHub Actions.
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ React Dashboard │────▶│ FastAPI REST API │────▶│ PostgreSQL │
│ (Tailwind CSS) │ │ (JWT Auth) │ │ (Operational) │
└─────────────────┘ └────────┬─────────┘ └─────────────────┘
│
▼
┌─────────────────┐
│ Snowflake │
│ (Analytics) │
└─────────────────┘Technology Stack
- React
- TypeScript
- Tailwind CSS
- Python
- FastAPI
- PostgreSQL
- Snowflake
- JWT
- Docker
- GitHub Actions
Key Features
Dataset Management
Upload, version, and manage master datasets with quarantine workflows for records that fail validation — preventing bad data from propagating downstream.
Data Profiling
Automatic column-level statistics including null counts, distinct values, data types, and distribution summaries to give data stewards immediate visibility.
Metadata Management
Centralized metadata store for datasets, columns, and business glossary terms — enabling consistent definitions across teams.
AI Column Descriptions
AI-generated column descriptions that accelerate data onboarding by suggesting human-readable explanations based on column names and sample values.
FastAPI REST APIs
Production-grade APIs with Pydantic validation, async endpoints, JWT auth, and OpenAPI documentation for seamless frontend integration.
Database Design
Normalized PostgreSQL schema for governance entities with Snowflake integration for warehouse-scale analytics and cross-database reporting.
Dataset Management
Users can upload CSV and structured datasets, track versions, and route failed records to a quarantine queue. The upload pipeline validates file structure, maps columns, and applies configured rules before committing data to the master store.
Data Profiling
Upon upload, the profiling engine computes column-level metrics — completeness, uniqueness, min/max values, and pattern detection. Results render in interactive dashboard cards so stewards can assess data quality at a glance.
Metadata Management
A dedicated metadata layer stores dataset descriptions, column definitions, tags, and ownership information. This creates a searchable catalog that bridges technical schemas and business context.
AI Column Descriptions
An AI service analyzes column names and sample data to generate suggested descriptions. Stewards can review, edit, and approve descriptions — reducing the manual effort of documenting large datasets.
FastAPI APIs
- Authentication endpoints with JWT token issuance and refresh
- CRUD APIs for datasets, rules, metadata, and profiling results
- Async query endpoints for Snowflake analytics workloads
- Webhook-style notifications for quarantine events
Database Design
PostgreSQL handles users, roles, dataset metadata, validation rules, and audit logs with foreign-key integrity. Snowflake stores profiling aggregates and analytics-ready views, connected via a sync layer that maintains referential consistency without duplicating operational writes.
Challenges Faced
- 01Building a dynamic rules engine that supports configurable validation logic without redeploying backend code
- 02Managing dual-database consistency between PostgreSQL operational data and Snowflake analytics views
- 03Rendering large dataset previews in the browser without blocking the main thread or degrading dashboard responsiveness
- 04Designing JWT auth flows that work across SPA navigation and API middleware without token leakage
- 05Optimizing FastAPI endpoints that aggregate profiling statistics from multiple database sources
Lessons Learned
- Separating operational storage (PostgreSQL) from analytics (Snowflake) early prevents costly refactoring when data volume grows
- A flexible rules engine pays dividends — hard-coded validations become unmaintainable as business rules evolve
- AI-assisted metadata is most valuable as a suggestion layer; human review remains essential for governance trust
- OpenAPI-first API design with FastAPI accelerates frontend development and reduces integration bugs
- Docker-based local environments that mirror production eliminate 'works on my machine' deployment issues