Curbstone
Ranks survey enumerators by risk of fabricating interviews that range and skip checks let through.
TL;DR
Detecting likely fabricated survey interviews ("curbstoning") at the enumerator level, using synthetic data with known ground truth. Source: README.md:5 @ 363c7b4.
Context & motivation
Range and skip checks pass fabricated interviews, because a careful fabricator enters plausible values. Curbstone compares each enumerator’s behaviour with peers working in the same villages, ranks enumerators by risk, and allocates a fixed back-check budget with a random floor so that unflagged staff are still audited. Source: README.md:11 @ 363c7b4.
My contribution
- Designed peer-comparison features (interview duration, daily throughput, near-duplicate answers, answer dispersion, GPS spread, age heaping, non-response) and a frozen risk-scoring engine.
- Built a fabrication simulator with five fraud modes plus one mode held out from development, so detection performance is measured instead of asserted.
- Ran the same 12 SQL rule checks in Python and DuckDB-Wasm with a cross-engine parity check, which caught a timezone bug.
How I used AI tools
Implementation by an AI coding agent (Google Antigravity). I set the scope, approved every plan, ran the checks, and accepted only work that passed automated gates.
Key decisions
Deliver risk ranking with a fixed back-check budget
- Alternatives
- Calibrated false-discovery-rate (FDR) flagging
- Why
- Scoring v1 failed its pre-registered honest-data acceptance test; two documented amendments and a post-hoc v2 also failed; v3, evaluated only on fresh seeds, still flagged at least one honest enumerator in 11 of 20 honest-only rounds.
- Trade-off
- The delivered method is a risk ranking, not a calibrated flagging decision.
Treat speed as weak evidence
- Alternatives
- An absolute duration threshold (the rule checks)
- Why
- Honest enumerators differ in speed.
- Trade-off
- Rules catch rushing better than the statistical ranking (0.63 vs 0.35).
Keep F10 (household size vs interview duration) after it showed no signal
- Alternatives
- Remove it
- Why
- Avoid removing features after seeing results.
- Trade-off
- F10 has no signal for rushing by construction, and it produced false flags in the low-workload demo round.
Challenges
Results
Limitations & next steps
- Simulated fabricators follow fixed rules. Real ones adapt to known checks, so detection rates here are likely an upper bound. source: README.md:79 @ 363c7b4
- The same author designed both the fraud modes and the detectors. The held-out mode reduces this risk but does not remove it. source: README.md:80 @ 363c7b4
- F5 (asset dispersion) is not centred on honest data (median -0.152 against a ±0.1 target over seeds 0..19). Frozen Scoring v3 uses it without re-centring, which plausibly contributes to honest false flags (untested). source: README.md:83 @ 363c7b4
- There is no real-world validation. Applying this to real survey exports (ODK/Kobo-style schema) is the obvious next test. source: README.md:84 @ 363c7b4
Data sources
- Data: SyntheticREADME.md:13 @ 363c7b4
- Metric: Precision@4 gain over the rule-count baseline, 95% CI (72 configurations × 20 rounds) (+0.131)README.md:19–21 @ 363c7b4 · 2026-11
- Metric: Average-precision gain over the rule-count baseline, 95% CI (+0.195)README.md:19, 21 @ 363c7b4 · 2026-11
- Metric: Held-out fraud mode (partial_section) detected, risk ranking vs rule checks (11/15)README.md:37 @ 363c7b4 · 2026-11