Document Analysis — Overview
Last updated: October 21, 2026
Purpose: Turn heterogeneous document drops (FOIA, leaks, archives) into searchable, citable evidence.
A FOIA release or a court production rarely arrives as a tidy set of searchable PDFs. It comes as scans, email exports, spreadsheets and attachments nested inside attachments. The pipeline below makes all of it searchable, so a reporter can find every mention of a name and cite the exact page it came from.
Core pipeline
- Inventory — List formats (PDF, email, images, spreadsheets).
- Extract text — Born-digital PDFs vs. scans (OCR).
- Normalize — Consistent filenames, date fields, deduplication keys.
- Index — Full-text index; optional entity extraction downstream.
- Search — Iterative queries; save search strings in a research log.
Tools for each stage
- Extraction. Apache Tika detects and extracts text and metadata from more than a thousand file types, including PowerPoint, Excel and PDF, through a single interface. Its maintainers describe it as useful for search-engine indexing and content analysis. Scanned pages need OCR first; see OCR Tools.
- Normalization. OpenRefine is a free, open-source tool for cleaning messy data. Its clustering feature merges near-identical values, such as one company spelled three ways, and it runs on your own machine.
- Index and search. ICIJ’s Datashare is open-source and self-hosted. It indexes PDFs, emails, office documents, images and archives, runs OCR, detects people, organizations and locations, and supports boolean, wildcard and fuzzy queries. A server mode lets a team share tags. Because it runs on your own machines, sensitive material does not go to an outside cloud.
When to escalate
| Situation | Action |
|---|---|
| >100k pages, team collaboration | Consider Datashare, Aleph, or dedicated investigation platform |
| Audio/video central to case | Plan transcription pass; see Transcription Services |
| Messy relational data | OpenRefine before joining to corporate or FEC tables |
Note on Aleph (checked 2026-10-21): OCCRP says it has rewritten Aleph from scratch as Aleph Pro and is focusing its resources there. It committed to maintaining the open-source repository only until December 31, 2025. Check the current terms before you build a new project on self-hosted Aleph. Platform comparisons are in Search and Document Platforms.
Related skill
document-research-specialist
Sources
- Apache Tika (read 2026-10-21)
- OpenRefine (read 2026-10-21)
- ICIJ, Datashare README (read 2026-10-21)
- OCCRP, Aleph repository README, including the Aleph Pro transition notice (read 2026-10-21)
Expanded with AI research and drafting assistance from the primary sources cited; editorial review pending.
