DGHS CleanPulse
Validates Bangladesh dengue and district climate data through a Medallion-architecture pipeline with no silent data drops.
TL;DR
A Medallion-architecture data engineering pipeline with a Pandera validation firewall, DuckDB analytics API, and Next.js console for Bangladesh dengue surveillance and district-level climate data. Source: README.md:5 @ be055e4.
Context & motivation
The official DGHS dengue dashboard returned HTTP 404 when last checked (2026-11-15), so CleanPulse ingests a secondary historical dengue series (a third-party transcription of DGHS daily press releases, Mendeley Data, CC BY 4.0) and Open-Meteo ERA5 reanalysis climate data (CC BY 4.0). Raw responses land unchanged in Bronze with SHA-256 provenance sidecars; a Pandera firewall builds validated Silver Parquet tables and records anomalies; a FastAPI + DuckDB API and a Next.js console sit on top. The validation demo uses synthetic fault-injection input, and live DGHS scraping fails loudly until the official feed returns. Source: README.md:5–7, 37, 47–48, 106–119 @ be055e4.
My contribution
- Built a Medallion pipeline whose Bronze layer stores immutable raw responses, each paired with a SHA-256 provenance sidecar.
- Built a Pandera validation firewall that records anomalies instead of dropping rows silently, tested against a synthetic fault-injection fixture.
- Built a FastAPI + DuckDB analytics API and a Next.js console.
How I used AI tools
Implementation by an AI coding agent (Google Antigravity). I set the scope, approved every plan, ran the checks, and accepted only work that passed automated gates.
Key decisions
Drop Dataset A’s bundled weather columns and use ERA5 instead
- Why
- Their upstream source is unstated and internally inconsistent.
Shift report dates by -1 day
- Why
- DGHS daily press releases cover the 24 hours ending 08:00 Asia/Dhaka on the report date.
Results
Limitations & next steps
- The official DGHS dengue feed returned HTTP 404 Not Found when last checked on 2026-11-15; the live scraper remains fail-loud and unvalidated against the live DGHS HTML structure until the official feed returns. source: README.md:7, 119 @ be055e4
Data sources
- Data: Real secondary sourcesREADME.md:7, 106–114 @ be055e4