Data Diary and Reproducibility
Media & Document Verification

Data Diary and Reproducibility

Skip to main content
Table of Contents
< All Topics
Print

Data Diary and Reproducibility

Last updated: November 14, 2026

Instructions

We use this skill whenever a piece rests on data: a downloaded table, an API query, a spreadsheet someone built, a script. The standard is the one the Cronkite School’s data course sets: “Your data, its analysis and the way it’s characterized in publication must be demonstrably accurate. That means understanding exactly what you did, why, where it all is and how it should be communicated” (Data reporting @ Cronkite, “Replication and the data diary”). Three rules follow, and each is checked before publication:

  1. Every published number traces to a dated, versioned source: which release or file, when

it was retrieved, and the exact query or script that produced the figure (§2, §3).

  1. Every transform is written down in the data diary, in the order it was done (§1, §4).
  2. A second person can re-run it from the diary alone and get the same numbers (§5).

It composes four existing skills and restates none of them:

  • document-research-specialist owns the tools: ingestion, OCR, cleaning and record

linkage. Its “Evidence and Provenance” rule (every finding traces to a document ID, page or URL; machine-derived text is labelled) is rule 1 applied to documents, and its §5 rule (“flag ‘likely duplicate’ instead of silent merge”) is a diary entry, not a judgment call.

  • ai-document-analysis already does half of a data diary: its provenance tags

(AI-extracted, AI-inferred, Human-verified) go on every value a model pulled from a document, and those tags are copied into the diary (§4.4).

  • agency-damage-assessment (ADA) consumes USAspending and OPM figures as evidence for D6 and

D1 under its Rule 6-c, and its “Primary-record data” section is the house rule for those two sources: cite the source URL and the file’s retrieved date; a missing unit is “unverified”, never “no harm” (Rule 6-d). The caveat cards (§6) are written to match it.

  • legislative-impact-scoring (LIA) has the same discipline for estimates: its Provenance

Gate (Section 1) refuses an assessment with no citable primary text, and its knobs (Section 5) make every value ship with its baseline, unit and data vintage (knob 16), with not_disclosed rendered as a flag. A data diary is how a figure earns those fields.

Use a different skill when:

  • the question is whether a number is used correctly (per capita, percentage points, inflation,

margins of error): numbers-and-statistics-integrity checks the arithmetic and framing; this skill proves where the number came from and that it can be produced again;

  • the question is whether a source or outlet is reliable: source-credibility-scoring;
  • the task is scoring an agency or an action: ADA and LIA own their scales; this skill only

documents the data they cite.


1. The data diary

Open the diary before the first download, and write down what you do as you do it. “Think of the data work the same way you think about interview notes or transcripts”, and keep it “in at least the same detail” (Cronkite, above). It is a working file for the editor, never reader-facing.

Where it lives. Beside the draft’s other source material: agent-base/knowledgebase/_sources//data-diary.md. _sources/ is staging and is never published. Raw downloads do not go in the repository (size, and some carry individual rows): keep them in the research run directory or the dataset directory a loader names, and record their SHA-256 in the diary (§3).

Sections, following Cronkite’s model:

Section What goes in it
Sourcing Who collects the data and why it exists; where you got it; how you know it is authentic; other stories and academic work that used it, with their warnings; alternative sources
Documentation and flaws The record layout or codebook (link or copy); each integrity check you ran and what it found; whether your totals match the agency’s own published totals, and why not; scope decisions (“serious crimes” means these codes, because…); questions put to the agency and its answers
Processing notes Every hand-made correction (or how you made them, e.g. OpenRefine clustering, linked); geocoding hit and miss counts; how messy data became a table
Analysis Each question asked of the data, the query or code that answered it, and the answer, in plain language beside the code
Vetting Who re-ran or reviewed it (§5), what they found, and why the finding might be wrong

Cronkite’s examples are worth copying: a Los Angeles Times notebook for an FEC analysis, Talia Buford’s annotated diary from the Center for Public Integrity, and a BuzzFeed News repository documenting an opioid-deaths story end to end. The Associated Press data team’s version is a “one-person bus rule”: “If someone is hit by a bus, someone else should be able to pick up the project without wasting any time” (Meghan Hoyer, quoted there).

Stages. Paul Bradshaw’s Inverted Pyramid of Data Journalism names the stages a diary follows: Conceive, Compile, Clean, Context, Combine, Communicate, with questioning “at every stage”, including “how authoritative the source is, and how reliable their methodology is” (Online Journalism Blog, 2024 update). Context is where the caveat cards (§6) come in; Combine is where denominators and joins (§4.3) come in.


2. The number trace

One row for every number the piece will publish. It extends the numbers register in numbers-and-statistics-integrity (which grades whether the number is used correctly) with the columns that prove where it came from.

# Number as published Dataset and release Retrieved (date, how) Query or script, and the commit or command Diary steps Caveat card Re-run by, result

A number with an empty “Dataset and release” or “Query or script” cell does not publish. A number the re-run did not reproduce does not publish until the difference is explained in the diary.


3. Retrieval: dated and versioned

  • Through the research CLI where a client exists. `scripts/ca_research.py api

“” –agent –run-dir

` writes every call to the run directory’s ledger.jsonl (time, arguments, status, the file it saved). Copy the ledger line, or its path, into the diary. The executors for the three carded datasets are in §6.

  • Offline loaders pin the version for you. Loaders that read a file a human downloaded

refuse a file with no release date in its name (the Deportation Data Project loader refuses ddp_287g.xlsx and reads only ddp_287g_.xlsx), so a figure always names its release. Never rename a file to get past that refusal.

  • Committed aggregates carry their own date. The ADA data files under

docs/ada-pilot/data/ record a retrieved date; cite it, and refresh before relying on a figure older than the document’s as-of date (ADA, “Primary-record data”).

  • A manual download gets four lines in the diary: the URL of the download page, the date

and time, the file name as saved, and shasum -a 256 . The hash is what lets a second person prove they re-ran against the same file.

  • Never overwrite a raw file. Raw downloads are read-only; every change happens in a copy

or in code, and the diary says which.

  • Data that changes after you read it. Agencies revise. Record the publication or

certification date the source shows (USAspending publishes both per agency submission, §6.2), and re-pull before publication if the source has updated since.


4. Transforms: write each one down

4.1 Code over hand edits

A script or notebook is its own diary of what it did; a hand edit is not. Prefer code for any transform that touches more than a handful of rows, and keep the script with the diary. Where a hand edit is unavoidable, list each one, or describe the method and link the file (Cronkite).

4.2 Cleaning

document-research-specialist §5 names the tools (OpenRefine for clustering and reconciliation, pandas in scripts). Jonathan Soma’s IRE 2024 session, “30 data cleaning tips in 60 minutes”, is a good working reference for the common moves: substitute, type-cast, split, regular expressions, long-to-wide reshaping, fuzzy matching (“Jonathan Soma,” “Jonathan P Soma,” and “Soma, Jonathan” into one listing) and geocoding (jsoma.github.io/ire24-cleaning). Each move goes in the diary with its before-and-after row count.

Two cleaning facts to write down every time:

  • Missing is not zero. Record how blanks were treated. Soma notes that in pandas a column

with missing values must be cast to float, not int, “even if there are no decimals in the numbers”: the blanks survive as missing values, and the diary says what was done with them.

  • A merge of two records is a decision. Fuzzy matches are flagged and adjudicated by a

person for anything that will publish (document-research-specialist §5), and each adjudication is a diary line.

4.3 Joins and denominators

Every join records its key, how many rows matched, how many did not, and what happened to the unmatched. Every rate records its denominator’s source and year in the diary, and numbers-and-statistics-integrity checks that the two years agree.

4.4 Values a model extracted

A value pulled from a PDF or scan by a model keeps its ai-document-analysis provenance tag (AI-extracted, AI-inferred, Human-verified) in the dataset itself, in a column, not only in the diary. Soma’s tip of passing bad OCR through a language model (“Even if it hallucinates, it’s probably better than what you had initially (and you’re sure to double-check, right??)”) is allowed only on that footing: the output is AI-extracted until a person checks it against the page, and no AI-extracted value publishes.

4.5 Building your own dataset

When the data does not exist and the newsroom builds it from documents, observation or a survey, the diary also records the collection method. Jennifer LaFleur and Helena Bengtsson’s advice at GIJC23: structure the data before collecting, set cut-off dates, “leave time for follow-up after collecting it”, know the universe and its subgroups, test survey questions on a small group first, and “be cautious of documents with lots of tiny numbers” (GIJN, “Expert Tips for Journalists on Building Your Own Datasets”, 2023). A methodology built from observation is vetted by a statistician before collection begins (same source).

4.6 Definitions decide the count

Two databases of “the same thing” can disagree because they define it differently. GIJN’s “Best Practices for Working With Mass Shootings Data” starts from this problem: “There can be confusion among journalists about ‘mass shootings’ data” (gijn.org/stories/best-practices-mass-shootings-data). Write the definition the dataset uses into the diary, and say in the story which one the number follows.


5. The re-run

Before publication a second person, who did not do the analysis, re-runs it from the diary and the raw files:

  1. Start from the hashes in §3; confirm the files match.
  2. Follow the diary in order without asking the author. Every point where they had to ask is a

gap in the diary; fix the diary, not just the number.

  1. Reproduce each number in the trace (§2). Record “reproduced”, or the difference and its

cause.

  1. Read the caveat card for each dataset (§6) against the draft’s wording.

How much review a story needs depends on how much of it hangs on the analysis: “If the entire story hinges on conclusions that you found through an analysis, then it’s worth comparing notes with someone else”; a stronger form is two people choosing a method independently and then comparing (GIJN, “Peer Reviewing Our Data Stories”).

Outside reviewers see the method, not the findings. The same GIJN piece suggests sharing “a description of your methodology, without revealing specific findings” for sensitive stories. That is the only form in which Citizen Analyst sends unpublished work outside: never the findings, never source names, never unpublished leads.

After publication, show the work. “Even a one-sentence summary where you got the data from and what you did with it is better than nothing at all” (same source). Every data-led piece carries at least that sentence; a piece whose finding rests on the analysis carries a methodology note drawn from the diary.


6. Caveat cards

A caveat card says what a dataset counts, what it misses, where its series breaks, and how to cite it. Read the card before writing a sentence that uses the data, and name the card in the number trace. The cards were written 2026-11-13 from the sources linked in each; the datasets’ own notes change with each release, so re-read them when a new release arrives.

6.1 Deportation Data Project: ICE arrests, detainers, detentions, removals (and 287(g), facilities)

Get it. `scripts/ca_research.py api api.deportation_data ““`. The loader reads files a person downloaded in a browser (deportationdata.org answers scripts with a bot challenge, so nothing is fetched), saved as ddp__.xlsx in the dataset directory. It returns findings from every DDP file present, each naming its dataset and release, with the project’s caveats attached. It aggregates the individual-level tables to counts per state (per area of responsibility for removals) and never lists a row.

What it counts.

  • Arrests: one row per ICE administrative arrest. Rows are arrests, not people.
  • Detainers: one row per detainer request to a jail or prison.
  • Detentions: “Each row in the detentions table represents time in a specific detention

facility from book-in to book-out.” One person can have several stints.

  • Removals: one row per deportation.
  • 287(g): one row per agreement, compiled by the project. It records the legal scope of an

agreement, not enforcement.

  • Facilities: one row per facility, compiled by the project.

What it misses.

  • Only ICE Enforcement and Removal Operations: asked whether the data include other components’

actions, the project answers “No, they only include actions by ICE Enforcement and Removal Operations (ERO).” Actions by other ICE components, and by other agencies such as Border Patrol, are not in it.

  • Detainers: “There does not appear to be any way to determine from these data whether a jail

or prison is holding individuals for up to 48 hours in response to a detainer request.” A request is not proof a jail complied.

  • 287(g): “does not capture informal interactions between state or local agencies and ICE, nor

does it measure the number of people arrested, detained, or transferred as a result of each agreement.”

  • Facilities: “No single public source provides a complete or up-to-date list of ICE detention

facilities”.

  • The state field “is accurate, but sometimes missing.” Report the missing count.

Known breaks in the series.

  • Removals: the project reports reliability problems “resolved in the late July 2025 data

release but reappeared in the October 2025, March 2026, and August 2026 releases”, and says “we continue to have questions about the reliability of the removals data”. Do not build a removals trend across those releases without saying so; prefer arrests or detentions for a trend.

  • “ICE appears to update records retroactively in a relatively small number of cases”: two

releases can disagree about the same month. Pin the release, and never splice two releases into one series without a diary note.

  • “There are several types of potential duplicates in the data and multiple reasonable

approaches to resolving them”: the diary says which approach was used.

  • Some spreadsheets “are split over multiple sheets. The sheets should be stacked before being

analyzed.”

  • 287(g) end dates: “there is no document that specifies the end date of the agreements”; an

agreement is known to have ended only when ICE drops it from its list.

How to cite. The data are CC0, and the project asks for this wording: “government data provided by ICE in response to a FOIA request to the Deportation Data Project and analyzed by Citizen Analyst” (arrests, detainers, detentions, removals), or “government data published by ICE, collated by the Deportation Data Project, and analyzed by Citizen Analyst” (287(g), facilities). Add the release date. ICE is the record holder and the project is a secondary compilation. For which agencies hold a 287(g) agreement today, ICE’s own participating-agencies spreadsheet (scripts/ca_research.py api api.ice_287g "") is the primary record; the project’s 287(g) file adds history and geography.

Sources: the project’s FAQ (deportationdata.org/docs/ice.html#faq), its processed-data page, its 287(g) and facilities codebooks and its 2026-10-16 287(g) release note, quoted in the loader’s docstring as read on 2026-11-13 through the Internet Archive’s copies (the live site challenges scripts).

6.2 USAspending

Get it. scripts/ca_research.py api api.usaspending "" returns budget authority and obligations per fiscal year for one toptier agency (a department or independent agency), with FY2024 as the baseline. The committed table docs/ada-pilot/data/usaspending-budgetary-resources.json holds all toptier agencies and its retrieved date. Award-level searches go through api.usaspending_awards.

What it counts. Federal award data (contracts, grants, loans and other assistance) and agency account data (budget authority, obligations, outlays). USAspending’s “About the Data” says award data “are available on USAspending.gov starting from Fiscal Year 2001”, and account data, “covering both award and non-award spending”, from “the second quarter of Fiscal Year 2017 (i.e., January 1, 2017), as required by the” DATA Act (usaspending.gov, “About the Data”).

What it misses.

  • Bureaus. A bureau (FEMA, NIH, CDC) cannot be separated from its department in the

budgetary-resources data; the client declines a bureau by name.

  • Cause. Rescissions and level changes Congress made are inside the numbers.
  • Completeness at the margins. In the DATA Act’s first reporting quarter GAO found awards for

160 financial assistance programs, an estimated $80.8 billion a year, omitted (GAO, “DATA Act: OMB, Treasury, and Agencies Need to Improve …”). Check an agency’s submission record before relying on a thin number.

Known breaks in the series.

  • “Data reporting requirements change over time, and … higher quality data are generally found

in more recent fiscal years” (About the Data). Account data do not exist before FY2017 Q2; do not run an account-data trend across that line.

  • From the June 2020 reporting period, agencies with COVID-19 relief funding reported monthly

(About the Data); reporting frequency differs across that date.

  • Budget authority includes multi-year and carry-over money: DHS FY2025 budget authority more

than doubled on one-time reconciliation funds, so a lower obligation rate can be slower spending against a larger denominator, not withholding (ADA, “Primary-record data”).

  • The current fiscal year is partial and is never compared with a full year.
  • Agencies revise submissions. The submission statistics table shows each submission’s

publication date and certification date, “the date when an agency’s Senior Accountable Official (SAO) confirms that the financial data submission is valid and reliable” (usaspending.gov, submission statistics). Record both in the diary.

How to cite. “USAspending.gov, (toptier code ), fiscal year , retrieved ", with the endpoint or the download page. In an ADA profile, end with the house line: evidence, not a score or proof of withholding.

6.3 OPM federal workforce data (formerly FedScope)

Get it. scripts/ca_research.py api api.opm_fwd "" --slug reads the committed aggregate docs/ada-pilot/data/opm-workforce.json, built from OPM's record-level files by scripts/ada_opm_workforce.py. The official source is data.opm.gov.

What it counts. Headcount: people on the payroll at the file's date, by agency and sub-element, monthly, plus separations and accessions by personnel action. OPM launched the Federal Workforce Data site at data.opm.gov as "a modern platform that replaces the outdated FedScope system", with "predictable monthly data updates" and new data on "retirement eligibility, telework and remote work, administrative leave, performance ratings, and federal hiring activity" (OPM news release).

What it misses.

  • People, not FTE. Say which when citing.
  • Bodies OPM does not cover: among them the Postal Service, the CIA, NSA, DIA, NGA, ODNI, the

White House Office, State Department Foreign Service personnel and the legislative branch (the full list is in the data file's caveats). A body OPM does not cover is "not covered", not zero.

  • Demographics. FedScoop reported that "Data on disability status, gender, race and national

origin, and ethnicity aren't included" in the new site, citing the site itself and administration directives on diversity programs (FedScoop, Madison Alder, 2026-01-09).

  • Why. A separation category is OPM's coding of the action, not a finding of why someone left;

a deferred resignation is voluntary and is reported beside, never as, a firing.

  • A unit inside a larger OPM sub-element is context only, never that unit's headcount.

Known breaks in the series.

  • FedScope to Federal Workforce Data. FedScope offered "data updates on a quarterly basis

that lagged by months"; the new site is monthly, and FedScope would no longer be available as of January 28, 2026 (FedScoop, above). A series that joins FedScope-era figures to new-site figures crosses a change of system, cadence and fields: check the definitions match and say in the diary that the series is spliced.

  • Files are re-issued in versions; record the version read. Late-reported actions are counted

separately.

  • Seasonal employers (parks, the Census Bureau) compare same season to same season.

How to cite. "OPM Federal Workforce Data (data.opm.gov), , , version , retrieved "; for the committed aggregate, its retrieved date and refresh command. Detail and the sources that could not be used: docs/ada-osint-sources.md in the citizen-analyst repository.


7. Before publication: the checklist

  • [ ] Diary exists in _sources// and covers every section in §1.
  • [ ] Every published number has a full row in the number trace (§2).
  • [ ] Raw files are hashed; no raw file was edited in place (§3).
  • [ ] Every transform, join and hand edit is in the diary with row counts (§4).
  • [ ] No AI-extracted value publishes without a person's check (§4.4).
  • [ ] A second person re-ran it, and every number reproduced or the difference is explained (§5).
  • [ ] Each dataset's caveat card was read against the draft's wording (§6).
  • [ ] The piece says where the data came from and what was done to it, in at least one sentence (§5).
  • [ ] Figures that feed ADA or LIA carry the source, retrieval date and data vintage those

skills require.

Was this article helpful?
0 out of 5 stars
5 Stars 0%
4 Stars 0%
3 Stars 0%
2 Stars 0%
1 Stars 0%
5
Please Share Your Feedback
How Can We Improve This Article?