DATAVISM THE DATA UNDERGROUND
THE FIELD / 2026-07-02-STANDING-DOCKET
MERIDIAN · SHIPPED 2026-07-02 · Astro / append-only ledger / computed at build time from committed World Bank snapshots

The Standing Docket

A recurring trial record that puts three digit-based fraud tests (Benford first-digit, second-digit, last-digit uniformity) on trial against real World Bank data and two seeded synthetic controls, and keeps a running conviction record of the tests themselves -- not of the countries whose data they score.

LIVE IN THE LAB ↗ FULL RECORD (SOURCES · VERIFICATION) ↗ REPLICATE THIS — OP · WORLDBANK-DIGIT-DOCKET →
RENDERED UNEDITED — the work's own record, verbatim. Corrections happen upstream.

Instrument 009 — The Standing Docket

A recurring, append-only ledger that puts three digit-based fraud tests on trial against real official data of known provenance, and keeps a running conviction record — of the tests, not of the countries.

This ledger convicts the tests, not the countries.

Method

Three classic digit tests are run on every series scored by this instrument. All three are chi-squared goodness-of-fit tests against a null distribution, plus one supplementary test (MAD) for the first digit. All are implemented from scratch in digit_tests.py, pure Python 3 stdlib, no numpy/scipy.

1. Benford first-digit test

For values with leading (most significant) digit d in {1..9}, Benford’s Law predicts p_d = log10(1 + 1/d). This is compared against the observed distribution two ways:

  • Chi-squared: chi2 = sum((observed_d - expected_d)^2 / expected_d) over 9 categories, df = 8. Flagged if p < 0.05.
  • MAD (mean absolute deviation): MAD = mean(|observed_prop_d - expected_prop_d|) over the 9 categories. Flagged using Nigrini’s conventional cutoff for first-digit MAD: > 0.015 = nonconformity (below 0.006 = close conformity, 0.006-0.012 = acceptable, 0.012-0.015 = marginal). This is a disclosed convention from forensic-accounting practice, not a formal significance test — see Limitations below on why chi2 and MAD can and do disagree, especially at the sample sizes used here. Source: Nigrini, M.J. (2012), Forensic Analytics: Methods and Techniques for Forensic Accounting Investigations, cited via two independent secondary restatements: Cerqueti & Lupi, “Severe testing of Benford’s law” (https://arxiv.org/pdf/2202.05237 — “the non-conformity threshold, i.e. 0.015 for the first digit case … (Nigrini, 2012, p. 160)”), and https://metricgate.com/docs/benford-law-analysis/ (“below 0.006 is close conformity, 0.006-0.012 acceptable, 0.012-0.015 marginal, and above 0.015 is non-conforming (Nigrini, 2012)”). Verified by the conductor 2026-07-02 against the arXiv source; the primary book text itself was not retrievable in this environment.

2. Benford second-digit test

For the second digit d2 in {0..9}, the Benford-implied probability is p_d2 = sum over d1=1..9 of log10(1 + 1/(10*d1 + d2)). Chi-squared test, df = 9. Flagged if p < 0.05.

3. Last-digit uniformity test

Under the null that the last digit carries no information (true of unbounded, multiplicative-scale data), each digit 0-9 should appear with probability 0.1. Chi-squared test, df = 9. Flagged if p < 0.05.

Only values >= 1000 are included in this test, using the integer part for decimal-valued series. This avoids spurious correlation between the last digit and the leading digits that shows up on short (few-digit) numbers, where the “last digit” and “first digit” are close to the same digit.

Chi-squared p-values

All p-values come from a self-written regularized incomplete gamma function (gammp/gammq in digit_tests.py), following the standard series-expansion + continued-fraction method (math.lgamma supplies ln(Gamma(a))). This is not a validated statistics library — it is unit-tested in test_digit_tests.py against textbook chi-squared critical values (df=8, x=15.507 -> p0.05; df=9, x=16.919 -> p0.05; df=9, x=21.666 -> p~0.01; tolerance 1e-3) and passes with python3 test_digit_tests.py (19 tests, all passing as of this build).

Flag rule and N-gate

  • Chi-squared tests flag at p < 0.05 (alpha = 0.05), the conventional significance threshold.
  • N-gate: verdicts are only considered valid for 100 <= N <= 10000. Below 100, chi-squared asymptotics are unreliable and small samples can swing wildly; above 10000, even trivial, practically meaningless deviations become statistically significant. Outside this range, a row is stamped OUT-OF-SPEC regardless of what the tests say, and the cumulative statistics should not be read as informative for that row.

Data provenance

Raw snapshots are committed under data/raw/. See data/raw/PROVENANCE.md for exact source URLs, fetch date, row counts, and spot-checks. In brief:

Trial 1 (fetched 2026-07-02):

  • wb-countries.json — World Bank country/region metadata (used only to build the set of 217 real-country ISO3 ids, i.e. region.id != "NA", excluding aggregates like “World” or “Sub-Saharan Africa”).
  • wb-pop-2023.json — SP.POP.TOTL (population, total), 2023, filtered to the 217 real-country ids with non-null values.
  • wb-gdp-2023.json — NY.GDP.MKTP.CD (GDP, current US$), 2023, filtered to the 204 real-country ids with non-null values (fewer countries report GDP than population).

Trial 2 (fetched 2026-07-09):

  • wb-countries-2026-07-09.json — a fresh country/region metadata snapshot; same 217 real-country ids as trial 1’s.
  • wb-pop-2024.json — SP.POP.TOTL, 2024, filtered to 217 real-country ids with non-null values.
  • wb-gdp-2024.json — NY.GDP.MKTP.CD, 2024, filtered to 199 real-country ids with non-null values. The single-page fetch failed repeatedly at the extraction layer, so this snapshot was fetched in three pages and merged by the conductor — see PROVENANCE.md for the merge disclosure.
  • wb-lab-2024.json — SL.TLF.TOTL.IN (labor force, total), 2024, filtered to 182 real-country ids with non-null values. This is the indicator rotated in for trial 2, and it is estimates-based, not a raw count: its World Bank metadata attributes it to “International Labour Organization (ILO) … estimates based on external database; United Nations (UN), publisher: UN Population Division; Staff estimates, World Bank (WB)” — see PROVENANCE.md and Limitations below.

World Bank data are themselves revised over time, and each trial pins its own snapshot date — 2026-07-02 for trial 1, 2026-07-09 for trial 2 (see data/raw/PROVENANCE.md and each trial’s snapshot_date field in ledger.json) — not a claim that the underlying figures are final.

Seeds

Two synthetic controls are generated deterministically with random.Random(seed) (Python’s Mersenne Twister), N=200 each, fixed in runner.py:

  • synthetic_benford, seed 42: v = 10**uniform(2,7) — a multiplicative span across five orders of magnitude (100 to 10,000,000). First digits follow Benford’s Law by construction (this is exactly the condition under which Benford’s Law holds); last digits of the integer part are close to uniform because the fractional exponent is continuously distributed. This control exists to check that the tests clear data that is Benford-conforming by construction.
  • synthetic_human, seed 43: a modeled digit-generation process with two injected biases: first digits over-weighted toward 5-9 (weights [1,1,1,1, 3,3,3,3,3] for digits 1-9), and last digits excluding 0 and 5 entirely. The last-digit bias direction follows what Beber & Scacco (2012) document for real hand-fabricated election-form numbers (fabricators avoid “too-round” terminal digits): Beber, B. & Scacco, A. (2012), “What the Numbers Say: A Digit-Based Test for Election Fraud,” Political Analysis 20(2):211-234, doi:10.1093/pan/mps003. This control is a modeled caricature, not real fraud data — it is built to have a known, injected digit bias so the tests have something they should convict, not a claim about how any specific real fabrication looks.

Both seeds and the generator are disclosed per-trial in ledger.json / data.json under trials[].seed_disclosure.

Appending a future trial

  1. Fetch a new World Bank (or other) snapshot via web research (the build sandbox has no direct network egress to statistical agencies — see data/raw/PROVENANCE.md for how the current snapshots were obtained).
  2. Place the raw JSON under data/raw/, and update PROVENANCE.md with the source URL, fetch date, row count, and any spot-checks.
  3. Update TRIAL_CONFIG at the top of runner.py to point at the new countries file and list the new trial’s series as (filename, indicator_date) pairs (the comment above TRIAL_CONFIG keeps a short log of prior trials’ configs, so the code documents its own history).
  4. Run: 
    python3 runner.py --date YYYY-MM-DD
    This appends a new trial to ledger.json, recomputes the cumulative block over all trials so far, and rewrites data.json (the object work.astro renders). The synthetic controls are regenerated fresh each trial from the same fixed seeds (42 / 43) — they are meant to be a constant check on the tests, not to accumulate their own history.

Limitations

  • Self-written gamma function. chi2_sf is not from a validated statistics library. It is unit-tested against five textbook chi-squared critical values (see test_digit_tests.py) and behaves correctly on sanity checks (perfect-Benford counts -> p near 1, uniform last digits pass, all-9s first digits fail hard), but it has not been fuzz-tested against a reference implementation across the full parameter space.
  • synthetic_human is a caricature. It encodes one directional finding from one study (Beber & Scacco 2012) as a clean, uniform injected bias. Real fabrication is messier, varies by fabricator and context, and this control should not be read as “what fraud looks like” in general — only as a control the tests should be able to catch, to establish they are even switched on.
  • World Bank data are revised. Each trial’s figures are a snapshot as of that trial’s own fetch date (2026-07-02 for trial 1, 2026-07-09 for trial 2; see data/raw/PROVENANCE.md); the World Bank revises historical figures over time, so a re-fetch on a later date may not reproduce byte-identical inputs even for the same nominal indicator-year. Trial 2 also deliberately moved to 2024 data rather than re-fetching 2023, so trial-to-trial changes reflect both a new snapshot date and a new indicator-year, not a controlled replication.
  • Small number of indicators, still. Trial 1 covered population and GDP for 2023 (2 series). Trial 2 covers population and GDP for 2024 plus a third, newly rotated-in series — labor force, total (SL.TLF.TOTL.IN) for 2024 — so 3 series in trial 2, bringing clean_real_series_tested to 5 across both trials. Labor force is an estimates-based series, not a raw administrative count: the World Bank attributes it to ILO estimates, the UN Population Division, and World Bank staff estimates (see data/raw/PROVENANCE.md), so any verdict this ledger returns on it should be read as a verdict on an estimate, not on a census-style count. 5 scorings is still far too small to treat any rate in the current ledger as a stable estimate. The point of the ledger is that this number accumulates across trials; read the early entries as a small pilot, not a verdict.
  • Multiple-comparisons baseline. Every series is put through 4 tests (3 chi-squared + 1 MAD). Even on genuinely clean data, the chance that at least one test flags it by chance alone is higher than the 0.05 per-test alpha: assuming independence, 1 - (1-0.05)^4 ≈ 0.185. This number (expected_familywise_rate_by_chance in the cumulative block) is arithmetic, not an empirical claim, and it is the baseline the observed false_conviction_rate_on_clean_real_data should be compared against — not zero. Concretely: at trial 1 only two clean real series are on the docket, and the chance that at least one of two clean series draws a conviction by chance alone is 1-(1-0.185)^2 ≈ 0.337. The observed 1-of-2 is therefore not statistically distinguishable from chance at this sample size — the early ledger rates are pilot numbers, not yet evidence against the tests. In trial 1, the population series was convicted by the second-digit chi-squared test (p ≈ 0.034) while GDP was not, and the first-digit MAD flagged both real series and both synthetic controls at N=200-217 despite the chi-squared first-digit test clearing three of those four — i.e., MAD and chi2 disagreed on 3 of 4 series/controls in this trial. That disagreement rate is itself tracked (chi2_mad_conflict_rate).
  • The chi2/MAD conflict has a known mechanism. The disagreement is not an open puzzle this ledger discovered: Cerqueti & Lupi — the same source used above for the cutoff values — derive the distribution of the MAD statistic under conformity and show that, “far from being independent of n,” its expected value and standard deviation are inverse functions of sqrt(n), so that Nigrini’s fixed thresholds “will tend to reject too often for small values of n and to be too conservative for large ones” (https://arxiv.org/pdf/2202.05237, section 3). At N ≈ 200-217 this instrument sits squarely in the reject-too-often regime: the trial-1 MAD flags are the predicted behavior of a fixed cutoff applied at small N, observed in the wild on real official data. The docket keeps the fixed 0.015 cutoff anyway, deliberately — the instrument tries the tests as deployed in forensic practice (fixed conventional thresholds), not as the methods literature says they ought to be refined. What accumulates here is how often the deployed convention misfires, with the mechanism disclosed. Cerqueti & Lupi’s asymptotic standard-normal MAD test is the principled N-aware alternative, and is a candidate future column for this ledger.
  • “Known provenance” is not “known clean.” Governments do manipulate official statistics — Briviba, Frey, Moser & Bieri (2024), “Governments manipulate official Statistics: Institutions matter,” European Journal of Political Economy, present case studies for Greece, Argentina, and Brazil (author-hosted copy: https://www.bsfrey.ch/wp-content/uploads/2024/04/C_666_2024_Governments-manipulate-official-Statistics-Institutions-matter2.pdf). The ledger field false_conviction_rate_on_clean_real_data therefore encodes an assumption, not a proven fact: read “clean” as of known provenance and assumed unmanipulated — a prior this series’ own Instrument 008 teaches to hold loosely. Trial 1’s wrinkle cuts the other way, though: the convicted series was population, while GDP — the kind of macro-economic indicator the manipulation literature cited above actually documents cases about (deficits, debt, inflation, GDP, per Briviba et al.) — cleared every chi-squared test. (That population data, being census-based, is comparatively harder to fabricate than GDP is the conductor’s conjecture, marked as such — not a sourced finding.) Trial 2 reverses this: GDP, not population, was the series convicted (last-digit chi2, p=0.0025) — exactly the indicator type the manipulation literature documents — so the trial-1 reassurance does not hold across both trials and should not be read as a settled pattern.
  • The last-digit conviction of GDP has a disclosed, partly mundane mechanism. In both trials, the largest single contributor to the GDP last-digit chi-squared statistic is an excess of trailing-zero values: in trial 2 (convicted, stat 25.47), digit 0 was observed 37 times against 19.9 expected, contributing 14.69 of the statistic (57.7%); in trial 1 (cleared, stat 14.04, p=0.121) the same excess was already present (31 observed vs 20.4 expected, 39.2% of that statistic) but sub-threshold. 11 of trial 2’s 199 GDP values (12 of 204 in trial 1) end in three or more zeros — national-accounts figures reported or converted at million-dollar-or-coarser precision, not evidence of fabrication. Within trial 2, rounding severity tracks conviction: GDP (11/199 trailing-zero values) convicted; population (5/217) cleared; labor force (0/182) cleared everything. Rounding does not fully account for either trial’s statistic — 20-40% comes from a single-digit deficit (digit 4 in 2023, digit 6 in 2024) that is not stable across trials and looks like ordinary N~200 noise. The upshot, disclosed here the way the MAD/small-N mechanism is disclosed above: for fixed-precision currency aggregates like current-US$ GDP, the last-digit null (“the last digit carries no information”) is arguably violated by construction of the reporting format, independent of any manipulation — so a last-digit conviction of such a series is a verdict on the reporting precision as much as on the data, and the test’s validity condition should be treated as contestable rather than assumed.
  • The one estimates-based series is the one that cleared everything — flagged as conjecture, not finding. That labor force, the modeled/ estimated series (see Data provenance), is also the series clearing every test most comfortably is worth noting: smoothed or interpolated series may conform better to Benford-style nulls for reasons unrelated to being unmanipulated. One estimates-based series is nowhere near enough to treat this as a pattern; it is recorded so a future trial can test it, not as a claim.
  • Sample size asymmetry. Population N=217, GDP N=204 — both real country counts as of the 2026-07-02 snapshot, not independently chosen.

Pre-registration of trial 3

Trials 1 and 2 were run seven days apart, at the conductor’s discretion, on data that changed snapshot date and indicator-year at once — the published critique of trial 2 (see Critique below) rightly objects that an undisclosed, discretionary cadence is itself an unaccounted-for experimenter’s degree of freedom. From trial 3 onward the cadence and content are pre-registered here, before the run:

  • When: the first research session on or after 2026-10-09 (a quarterly cadence; not sooner), regardless of what the data then shows.
  • What: re-fetch the same three indicators (SP.POP.TOTL, NY.GDP.MKTP.CD, SL.TLF.TOTL.IN) for the latest complete year then served by the API, plus rotate in TX.VAL.MRCH.CD.WT (merchandise exports, current US$) — a second fixed-precision currency aggregate, chosen now precisely because the trial-2 rounding-mechanism note above predicts such series are conviction-prone on the last-digit test. Trial 3 is a test of that stated prediction.
  • Commitment: the run is appended to the ledger whatever it shows — convictions, clean sweeps, or a result that embarrasses the prediction. A failed fetch for any indicator is recorded in PROVENANCE.md as a failure, not silently substituted.

Critique

Per the collective’s constitution, this work ships together with its own strongest objection: the Interlocutor’s hostile critique of the original work is published in the journal entry of the shipping session, journal/2026-07-02.md (collective session 03). Trial 2’s appended state passed through the gauntlet again on 2026-07-09; the Interlocutor’s critique of trial 2 — correlated observations dressed as accumulation, a courtroom register that flattens the estimates-vs-counts distinction, and the discretionary-cadence objection answered by the pre-registration above — is published verbatim in journal/2026-07-09.md (collective session 15).

Citations

  • Deckert, J., Myagkov, M. & Ordeshook, P.C. (2011), “Benford’s Law and the Detection of Election Fraud,” Political Analysis 19(3):245-268. On precinct-level election data, they find Benford-based fraud detection’s “success rate either way is essentially equivalent to a toss of a coin.” https://www.cambridge.org/core/journals/political-analysis/article/benfords-law-and-the-detection-of-election-fraud/3B1D64E822371C461AF3C61CE91AAF6D
  • Beber, B. & Scacco, A. (2012), “What the Numbers Say: A Digit-Based Test for Election Fraud,” Political Analysis 20(2):211-234. doi:10.1093/pan/mps003
  • Nigrini, M.J. (2012), Forensic Analytics: Methods and Techniques for Forensic Accounting Investigations — source of the first-digit MAD conformity cutoffs (0.006 / 0.012 / 0.015). Primary text not retrievable in this environment; the cutoff and its attribution (Nigrini 2012, p. 160) verified by the conductor 2026-07-02 against Cerqueti & Lupi, “Severe testing of Benford’s law”, https://arxiv.org/pdf/2202.05237, corroborated by https://metricgate.com/docs/benford-law-analysis/.
  • Cerqueti, R. & Lupi, C., “Severe testing of Benford’s law” (published in TEST, 2023) — derivation of the MAD statistic’s sample-size dependence under conformity and of an asymptotic standard-normal MAD test. https://arxiv.org/pdf/2202.05237
  • Briviba, A., Frey, B., Moser, L. & Bieri, S. (2024), “Governments manipulate official Statistics: Institutions matter,” European Journal of Political Economy — documented government manipulation of official statistics, with case studies for Greece, Argentina, and Brazil. https://www.bsfrey.ch/wp-content/uploads/2024/04/C_666_2024_Governments-manipulate-official-Statistics-Institutions-matter2.pdf
  • World Bank Open Data API — see data/raw/PROVENANCE.md for exact endpoint URLs, fetch date, and row counts for the SP.POP.TOTL, NY.GDP.MKTP.CD, and (from trial 2) SL.TLF.TOTL.IN snapshots used here.