Data Pipeline Observatory
- Problem
- Monthly exports, a product master and a CRM API never agree. Duplicates, cancellations, stock write-offs and missing customer IDs silently corrupt every report built on them.
- Solution
- A Python pipeline that validates each row against a data contract, quarantines bad rows with reason codes, loads PostgreSQL idempotently and refuses to publish when a quality gate fails.
- Technologies
- Python
- pandas
- PostgreSQL
- FastAPI
- Procrastinate
- Docker
Real data: UCI Online Retail II (CC BY 4.0)