Document Analysis — Overview
Investigative Tools

Document Analysis — Overview

Skip to main content
Table of Contents
< All Topics
Print

Document Analysis — Overview

Last updated: October 21, 2026

Purpose: Turn heterogeneous document drops (FOIA, leaks, archives) into searchable, citable evidence.

A FOIA release or a court production rarely arrives as a tidy set of searchable PDFs. It comes as scans, email exports, spreadsheets and attachments nested inside attachments. The pipeline below makes all of it searchable, so a reporter can find every mention of a name and cite the exact page it came from.

Core pipeline

  1. Inventory — List formats (PDF, email, images, spreadsheets).
  2. Extract text — Born-digital PDFs vs. scans (OCR).
  3. Normalize — Consistent filenames, date fields, deduplication keys.
  4. Index — Full-text index; optional entity extraction downstream.
  5. Search — Iterative queries; save search strings in a research log.

Tools for each stage

  • Extraction. Apache Tika detects and extracts text and metadata from more than a thousand file types, including PowerPoint, Excel and PDF, through a single interface. Its maintainers describe it as useful for search-engine indexing and content analysis. Scanned pages need OCR first; see OCR Tools.
  • Normalization. OpenRefine is a free, open-source tool for cleaning messy data. Its clustering feature merges near-identical values, such as one company spelled three ways, and it runs on your own machine.
  • Index and search. ICIJ’s Datashare is open-source and self-hosted. It indexes PDFs, emails, office documents, images and archives, runs OCR, detects people, organizations and locations, and supports boolean, wildcard and fuzzy queries. A server mode lets a team share tags. Because it runs on your own machines, sensitive material does not go to an outside cloud.

When to escalate

Situation Action
>100k pages, team collaboration Consider Datashare, Aleph, or dedicated investigation platform
Audio/video central to case Plan transcription pass; see Transcription Services
Messy relational data OpenRefine before joining to corporate or FEC tables

Note on Aleph (checked 2026-10-21): OCCRP says it has rewritten Aleph from scratch as Aleph Pro and is focusing its resources there. It committed to maintaining the open-source repository only until December 31, 2025. Check the current terms before you build a new project on self-hosted Aleph. Platform comparisons are in Search and Document Platforms.

Related skill

  • document-research-specialist

Sources

Expanded with AI research and drafting assistance from the primary sources cited; editorial review pending.

Was this article helpful?
0 out of 5 stars
5 Stars 0%
4 Stars 0%
3 Stars 0%
2 Stars 0%
1 Stars 0%
5
Please Share Your Feedback
How Can We Improve This Article?