Tahmid Jawad Shafi data engineering, home
← All projects
pipeline · case study

DGHS CleanPulse

Validates Bangladesh dengue and district climate data through a Medallion-architecture pipeline with no silent data drops.

RoleDesigned and built
TeamSolo
Period2026-10 – present
Statusin-progress
DataReal secondary sources
StackPython · Pandas · Pandera · DuckDB · Parquet · FastAPI · Next.js · Pytest

TL;DR

A Medallion-architecture data engineering pipeline with a Pandera validation firewall, DuckDB analytics API, and Next.js console for Bangladesh dengue surveillance and district-level climate data. Source: README.md:5 @ be055e4.

Context & motivation

The official DGHS dengue dashboard returned HTTP 404 when last checked (2026-11-15), so CleanPulse ingests a secondary historical dengue series (a third-party transcription of DGHS daily press releases, Mendeley Data, CC BY 4.0) and Open-Meteo ERA5 reanalysis climate data (CC BY 4.0). Raw responses land unchanged in Bronze with SHA-256 provenance sidecars; a Pandera firewall builds validated Silver Parquet tables and records anomalies; a FastAPI + DuckDB API and a Next.js console sit on top. The validation demo uses synthetic fault-injection input, and live DGHS scraping fails loudly until the official feed returns. Source: README.md:5–7, 37, 47–48, 106–119 @ be055e4.

My contribution

  • Built a Medallion pipeline whose Bronze layer stores immutable raw responses, each paired with a SHA-256 provenance sidecar.
  • Built a Pandera validation firewall that records anomalies instead of dropping rows silently, tested against a synthetic fault-injection fixture.
  • Built a FastAPI + DuckDB analytics API and a Next.js console.

How I used AI tools

Implementation by an AI coding agent (Google Antigravity). I set the scope, approved every plan, ran the checks, and accepted only work that passed automated gates.

Key decisions

  1. Drop Dataset A’s bundled weather columns and use ERA5 instead

    Why
    Their upstream source is unstated and internally inconsistent.
    source: docs/analysis_plan.md:20 @ be055e4
  2. Shift report dates by -1 day

    Why
    DGHS daily press releases cover the 24 hours ending 08:00 Asia/Dhaka on the report date.
    source: README.md:109 @ be055e4

Results

Status: in-progress. Verified result metrics are withheld until instrumentation and evaluation are complete.

Limitations & next steps

  • The official DGHS dengue feed returned HTTP 404 Not Found when last checked on 2026-11-15; the live scraper remains fail-loud and unvalidated against the live DGHS HTML structure until the official feed returns. source: README.md:7, 119 @ be055e4

Data sources