Tahmid Jawad Shafi data engineering, home
← All projects
quality · case study

Curbstone

Ranks survey enumerators by risk of fabricating interviews that range and skip checks let through.

RoleDesigned and shipped
TeamSolo
Period2026-11 – 2026-11
Statuscompleted
DataSynthetic
StackPython · SQL · DuckDB-Wasm · Parquet · Pytest

TL;DR

Detecting likely fabricated survey interviews ("curbstoning") at the enumerator level, using synthetic data with known ground truth. Source: README.md:5 @ 363c7b4.

Context & motivation

Range and skip checks pass fabricated interviews, because a careful fabricator enters plausible values. Curbstone compares each enumerator’s behaviour with peers working in the same villages, ranks enumerators by risk, and allocates a fixed back-check budget with a random floor so that unflagged staff are still audited. Source: README.md:11 @ 363c7b4.

My contribution

  • Designed peer-comparison features (interview duration, daily throughput, near-duplicate answers, answer dispersion, GPS spread, age heaping, non-response) and a frozen risk-scoring engine.
  • Built a fabrication simulator with five fraud modes plus one mode held out from development, so detection performance is measured instead of asserted.
  • Ran the same 12 SQL rule checks in Python and DuckDB-Wasm with a cross-engine parity check, which caught a timezone bug.

How I used AI tools

Implementation by an AI coding agent (Google Antigravity). I set the scope, approved every plan, ran the checks, and accepted only work that passed automated gates.

Key decisions

  1. Deliver risk ranking with a fixed back-check budget

    Alternatives
    Calibrated false-discovery-rate (FDR) flagging
    Why
    Scoring v1 failed its pre-registered honest-data acceptance test; two documented amendments and a post-hoc v2 also failed; v3, evaluated only on fresh seeds, still flagged at least one honest enumerator in 11 of 20 honest-only rounds.
    Trade-off
    The delivered method is a risk ranking, not a calibrated flagging decision.
    source: README.md:47–51 @ 363c7b4
  2. Treat speed as weak evidence

    Alternatives
    An absolute duration threshold (the rule checks)
    Why
    Honest enumerators differ in speed.
    Trade-off
    Rules catch rushing better than the statistical ranking (0.63 vs 0.35).
    source: README.md:56 @ 363c7b4
  3. Keep F10 (household size vs interview duration) after it showed no signal

    Alternatives
    Remove it
    Why
    Avoid removing features after seeing results.
    Trade-off
    F10 has no signal for rushing by construction, and it produced false flags in the low-workload demo round.
    source: README.md:52 @ 363c7b4

Challenges

Problem: Timestamps stored as +06:00 ISO strings were parsed differently by the Python engine and DuckDB-Wasm (736 vs 0 hits on R07, the out-of-hours rule).
Approach: Offset-free parsing in both engines.
Outcome: The parity check confirms 12/12 rules match in both demo scenarios.
source: README.md:60 @ 363c7b4
Problem: Cached copies of two Parquet files had wrong checksums and were partly zero-filled after DuckDB-Wasm read them with HTTP range requests.
Approach: Load whole files and pass in-memory Arrow tables to DuckDB.
Outcome: No range requests are made.
source: README.md:61 @ 363c7b4
Problem: F5 (asset dispersion) is not centred on honest data. This was found after the study, because the original test failed from NaN propagation before reaching the median assertion.
Approach: Added non-NaN coverage checks and a strict xfail test for the F5 limitation.
Outcome: Headline numbers are unaffected: the engine code is unchanged since commit 51a97e1, where the study ran.
source: README.md:83, 103 @ 363c7b4

Results

+0.131Precision@4 gain over the rule-count baseline, 95% CI (72 configurations × 20 rounds) (interval +0.119 to +0.142)verified 2026-11 · source: README.md:19–21 @ 363c7b4
+0.195Average-precision gain over the rule-count baseline, 95% CI (interval +0.178 to +0.211)verified 2026-11 · source: README.md:19, 21 @ 363c7b4
11/15Held-out fraud mode (partial_section) detected, risk ranking vs rule checks (rules: 1/15)verified 2026-11 · source: README.md:37 @ 363c7b4

Limitations & next steps

  • Simulated fabricators follow fixed rules. Real ones adapt to known checks, so detection rates here are likely an upper bound. source: README.md:79 @ 363c7b4
  • The same author designed both the fraud modes and the detectors. The held-out mode reduces this risk but does not remove it. source: README.md:80 @ 363c7b4
  • F5 (asset dispersion) is not centred on honest data (median -0.152 against a ±0.1 target over seeds 0..19). Frozen Scoring v3 uses it without re-centring, which plausibly contributes to honest false flags (untested). source: README.md:83 @ 363c7b4
  • There is no real-world validation. Applying this to real survey exports (ODK/Kobo-style schema) is the obvious next test. source: README.md:84 @ 363c7b4

Full list in README

Data sources