Tahmid Jawad Shafi data engineering, home
← All projects
pipeline · case study

Bari-Link

Deduplicates NGO beneficiary registries that share no identifier, using kinship constraints to cut costly false merges.

RoleDesigned and shipped
TeamSolo
Period2026-11 – 2026-11
Statuscompleted
DataSynthetic
StackPython · Splink · DuckDB-Wasm · React · TypeScript · Vite · Pytest · Playwright

TL;DR

Kinship-aware probabilistic deduplication and household inference for multi-module NGO beneficiary registries that share no identifier, evaluated on a synthetic population modelled on rural Bangladesh. Source: README.md:6 @ c518135.

Context & motivation

NGO programmes often enrol the same people through separate modules with no identifier shared across them. Bangla names are romanised inconsistently, a woman may be recorded with her father's name in one register and her husband's in another, and phones are often shared within a household. Unlinked duplicates inflate reported reach; wrong merges can deny a person a transfer. Source: README.md:18 @ c518135.

My contribution

  • Built a synthetic population engine and noise model, romanised-name normalisation, and a Splink Fellegi-Sunter model with kinship cannot-link constraints (Pipeline K).
  • Added multi-channel household inference on top of the resolved person clusters.
  • Committed hypotheses before their results and confirmed the champion pipeline on five fresh holdout seeds.
  • Shipped a static DuckDB-Wasm demo of the results.

How I used AI tools

Implementation by an AI coding agent (Google Antigravity). I set the scope, approved every plan, ran the checks, and accepted only work that passed automated gates.

Key decisions

  1. Use kinship only as a hard cannot-link rule (Pipeline K)

    Alternatives
    Kinship as probabilistic Fellegi-Sunter evidence (the reference model)
    Why
    Roster overlap signals “same household”, not “same person”, which breaks the Fellegi-Sunter independence assumption. Pipeline K lowered cost against the reference on 5/5 fresh seeds (mean -76.0 points; one-sided sign test p = 0.031).
    Trade-off
    Better only when false merges cost at least 2x missed duplicates (tie at 1:1). In a namesake stress test at matched recall 0.50, K let through more cross-household namesakes (55.6% vs 49.0%).
    source: README.md:94 @ c518135
  2. Commit hypotheses before their results, and confirm H3 only on five new seeds

    Why
    H3 was suggested by earlier holdout results.
    Trade-off
    Pipeline K’s link threshold lies on the lower edge of the tested grid and was not re-tuned, to protect the confirmation.
    source: README.md:95, 108 @ c518135

Challenges

Problem: The generator filled fields the module forms never collect (guardian type, sex), so the model saw ground truth.
Approach: A schema audit found the leak, and the results were rebuilt clean.
Outcome: On clean rebuilds of holdout_0..4, splink_v1 cluster precision moved 0.9404 → 0.9243 and recall 0.4741 → 0.4408. An earlier 0.61 figure that a clean rebuild could not reproduce was corrected after a fresh-clone reproducibility check.
source: README.md:93 @ c518135
Problem: Household cuts were non-deterministic: tied min-cuts at 2.0x weights depended on PYTHONHASHSEED-dependent edge order.
Approach: Sorted subgraph edges before nx.stoer_wagner / nx.minimum_cut (commit 897b32e).
Outcome: A 30-node tied-min-cut regression test was added.
source: README.md:98, 163 @ c518135

Results

812.2Operational cost at a 5:1 false-merge cost ratio, Pipeline K vs the Splink reference (holdout seeds 5–9, lower is better) (reference 888.2; -8.56%; better on 5/5 seeds, one-sided sign test p = 0.031)verified 2026-11 · source: README.md:48–50 @ c518135
0.9656Cluster-level precision, Pipeline K (holdout seeds 5–9) (± 0.0106; Splink reference 0.9457 ± 0.0127)verified 2026-11 · source: README.md:47–48 @ c518135

Limitations & next steps

Full list in README

Data sources