Scraping Ethics and Methods
Last updated: November 14, 2026
Instructions
We use this skill before any script collects data from a website: a one-off scraper, a scheduled crawler, a no-code browser extension or an AI agent told to “get everything on this site”. It answers three questions in order: **should we scrape at all, may we, and how do we do it so a reader, the site owner and a court would all recognise it as fair?**
It is the conduct and law layer. It does not choose tools or write crawlers:
osint-automation-frameworksis the sibling. It covers frameworks (SpiderFoot, Recon-ng,
BBOT, sn0int, Maltego), passive-first design and API-key hygiene, and it stops at “without writing custom scrapers first”. This skill starts where a custom scraper or crawler begins, and governs any tool that skill picks.
public-records-research-specialistfinds the official record first. Most of what a
scraper would collect from a government site is published somewhere as a file or an API.
legal-research-specialistis the method for every legal statement here. §7 lists what
was read and when; anything beyond it is researched with that skill’s protocol (citator check included) through scripts/ca_research.py, never stated from memory.
patriot-private-citizen-inclusion-gateruns before any scraped record about a private
person is kept or named (§5).
foia-and-sunshine-lawsis the route when the data owner is a public body that will not
publish the file (§1).
Method, in order:
- Ask for the data instead if any route in §1 exists.
- Read the licence and the terms (§2). A non-commercial or research-only licence ends it.
- Read robots.txt (§3).
- Identify ourselves and throttle (§4). Stop at a login, a bot challenge or a
cease-and-desist.
- Keep only what the story needs (§5).
- Archive and log what was fetched (§6).
- Fill the scraping plan card (§8) before the first request, and keep it with the data.
Use a different skill when:
- the question is which framework or module to run:
osint-automation-frameworks; - the data sits behind a records request:
foia-and-sunshine-laws; - the question is whether a scraped page is authentic or manipulated:
media-verification-specialist;
- the question is a legal one this skill does not answer from a source:
legal-research-specialist,
then counsel.
1. Ask for the data instead
A scraper is the last route, not the first. Work down this ladder and stop at the first rung that gives the data:
| Rung | Route | How we do it |
|---|---|---|
| 1 | An official API | scripts/ca_research.py api , the house clients in scripts/lib/osint_api/. A missing client is a code-lane request, not a scraper |
| 2 | A bulk download or data portal | Read the site’s “data”, “download” or “developers” page. A CSV the publisher maintains beats one we rebuilt |
| 3 | The data behind the page | Open the browser’s web inspector. It can reveal the raw data behind graphics and a site’s source code (GIJN, “Digging Up Hidden Data with the Web Inspector”, Samantha Sunne at IRE21). A JSON file the page already loads is still the site’s data: §2–§4 apply to it unchanged |
| 4 | Ask the owner | Email the agency, court clerk or organisation for the file. Record the request and the answer in the plan card |
| 5 | A records request | Public bodies: foia-and-sunshine-laws. A refusal or silence is itself reportable |
| 6 | Scrape | Only now, under §2–§6 |
2. Licence and terms of service
Read the site’s terms of use and any data licence before the first request. Quote the clause that governs automated access or reuse in the plan card, with the URL and the date read.
2.1 House licence rule
**Data whose terms are non-commercial, academic or research-only is not used by Citizen Analyst**, whether it arrives by API, download or scraper. Peter’s rulings:
| Source | Terms | Ruling |
|---|---|---|
| Police Scorecard API | “only open for select non-commercial use”; no public keys | Dropped 2026-11-13. Skills may point reporters to the website |
| Vera Institute, Incarceration Trends | Licence grants academic, nonprofit and research use only; forbids commercial or profit-making use and redistribution | Dropped 2026-11-14 |
| Public Overpass API instance | Commercial use should use a self-hosted or paid server | 2026-11-14: the client requires a self-hosted endpoint (OVERPASS_URL), never the public one |
When the terms are ambiguous, the answer is “not yet”: record the question in the plan card and ask an editor. When the terms cannot be found at all, stop that source. A dataset whose licence is unread gets no loader (journalist-resources PLAN §3 rule 7).
2.2 What the terms can and cannot do
- Accepted terms bind us. In hiQ v. LinkedIn, after the CFAA questions, the district
court granted LinkedIn summary judgment on breach of contract in November 2022: hiQ had agreed to the user agreement when it created a corporate account, and violated it by automated scraping and by hiring workers to create fake profiles. The case ended in December 2022 with a consent judgment: a permanent injunction against scraping, deletion of the scraped data and code, and $500,000 in damages (ZwillGen, “hiQ v. LinkedIn Wrapped Up”, 21 Dec 2022).
- Never create an account to scrape, and never scrape while logged in to an account whose
terms forbid it. Never use fake or borrowed accounts. That is the conduct that lost hiQ.
- Terms written for users may not reach a logged-out visitor. In Meta v. Bright Data
(N.D. Cal., 23 Jan 2024, Judge Chen) the court held that Meta’s terms do not prohibit logged-off scraping of data that is public on Facebook and Instagram, because the terms govern “your use” of Meta’s products (Farella Braun + Martel; Quinn Emanuel summary). That is one district court reading one company’s terms. It is not a licence to ignore terms; it is why we read them closely.
- House practice is stricter than the law. Whatever a court might allow, we honour a
site’s terms unless an editor records a public-interest exception in the plan card (§8). The exception is an editor’s decision, written down before the scrape, never the scraper’s.
3. robots.txt
The Robots Exclusion Protocol is RFC 9309 (RFC Editor, RFC 9309). What it says, and what we add:
- It is a request, not a lock. “These rules are not a form of access authorization” (§1),
and the protocol “is not a substitute for valid content security measures” (§3). A path missing from robots.txt is not permission; a disallowed path is still a clear statement of the owner’s wishes.
- Longest match wins. The crawler must use the most specific matching rule, the one with the
most octets; if an allow and a disallow rule are equivalent, allow should be used (§2.2.2).
- No file is not a no. If robots.txt returns a 4xx status the crawler may access any
resource. If it is unreachable because of a 5xx or a network error, the crawler must assume complete disallow (§2.3.1.3–2.3.1.4).
- Re-read it daily. A cached copy should not be used for more than 24 hours unless the file
is unreachable (§2.4).
Crawl-delayis not in the standard. Crawlers may interpret other records (§2.2.4). **We
honour a Crawl-delay when one is present**, and throttle per §4 when none is.
- House rule: honour robots.txt for every automated collection, unless an editor records a
public-interest exception in the plan card. Reading one page by hand, or one page through scripts/ca_research.py extract for research, is reading, not crawling; a script that walks many URLs on a site is a crawler and checks robots.txt first.
4. Identify ourselves, throttle, and know when to stop
4.1 Identification
- Send the house User-Agent. Our HTTP layer (
scripts/lib/http.pythrough
lib.osint_api._base.user_agent()) sends CitizenAnalyst/1.0 (civic accountability research; factcheck@citizenanalyst.ai) by default. A scraper we write uses that string or a more specific one that still names us and gives a contact.
- Never disguise the scraper as a browser. Teaching material often shows sending a real
browser’s headers to get past a block, and random pauses “to look less bot-like” (Mindy McAdams, Python Beginners, “Web Scraping, Part 3”). We keep the pauses and drop the disguise: no spoofed browser User-Agents, no rotating User-Agents or proxies, no residential-proxy services, no fingerprint evasion.
4.2 Throttling
- Slow by default. One request at a time per host, with a pause between requests; the same
guide is plain that “it’s very bad to overload or overwork a website” (McAdams, above). Our own GDELT client keeps at least 6 seconds between calls because GDELT asks for one request every 5 seconds (scripts/lib/osint_api/gdelt_doc.py); start there for a site that publishes no limit, and never go faster than its Crawl-delay.
- Set the limit explicitly. Crawling libraries do not do it for you: Crawlee’s
maxRequestsPerMinute defaults to Infinity, and it has a respectRobotsTxtFile option (Crawlee API, BasicCrawlerOptions). Set both.
- Back off on 429 and 5xx.
lib.http.fetchalready retries those with per-host exponential
backoff and honours Retry-After; a hand-written scraper must do the same, not retry in a tight loop.
- Run small first. Test on a handful of pages, check the output, then run the rest. Prefer
off-peak hours for anything large.
- Unattended runs on the compose box go through its egress allowlist. A host that is not in
deploy/linux/harden/egress-allowlist.txt is added by PR; a 403 Filtered from the box is never routed around.
4.3 Stop rules
Stop the run, keep what was collected, and tell an editor when any of these happens:
| Signal | What we do |
|---|---|
| A login, paywall or “members only” wall | Stop. Never scrape behind a login |
| A bot challenge (Cloudflare “just a moment”, a CAPTCHA, a 403 challenge page) | Stop. Never solve, bypass or outsource a challenge. Build a loader that reads a file a human downloaded, and document that manual-download path (PLAN §3 rule 6). lib.http.fetch tries the Wayback Machine’s copy once after a bot wall; that reads a public archive, it does not get past the challenge |
| A cease-and-desist or any written request to stop | Stop at once and escalate. The DOJ’s charging policy treats an unambiguous written cease-and-desist that the recipient receives and understands as revoking authorization from that point (§7.3) |
| A 429 that does not clear after backoff, or the site slows down | Stop for the day |
| Personal data the plan card did not expect | Stop and re-run §5 before continuing |
5. Personal data minimisation
- Store only the fields the story needs. Decide them in the plan card before the first
request. A scraper that saves whole pages “in case” is collecting personal data we have no reason to hold.
- Private people go through the gate. Before any scraped record about a person who is not a
public official or public figure is kept, linked or named, run patriot-private-citizen-inclusion-gate. Bulk harvesting of private people’s details (names with addresses, phone numbers, family members) is refused outright, as osint-automation-frameworks already refuses “non-public personal data harvesting”.
- Aggregate where you can. If the story needs a count, keep the count and drop the rows.
- Never log raw targets in product logs (
osint-automation-frameworks§3, the zero-PII
rule).
- Retention. Raw scraped files with personal data live in the run directory or the
evidence store, not in the knowledgebase. Delete what the story did not use when it is published or abandoned, and say in the plan card that you did.
- Outside the US the law is different. The US and EU approach scraping very differently
(Ugo Pagallo and Jacopo Ciani, “Anatomy of Web Data Scraping: Ethics, Standards, and the Troubles of the Law”, 2023). A scrape of a non-US site, or of records about people in the EU, goes to an editor and counsel first.
6. Archive what you scraped, so it can be re-run
A scraped number is only as good as our ability to show where it came from on the day.
- Keep the raw response. Save each fetched page or JSON file with its URL, fetch time,
HTTP status and a content hash. scripts/ca_research.py does this for every fetch in its run directory (extracted/, the http.jsonl and ledger.jsonl logs), and make fetch URL= drives the same layer from a terminal.
- Archive the pages you cite. For a public page a story cites, ask the Wayback Machine to
capture it (scripts/lib/wayback_spn2.py, opt-in with CA_OSINT_WAYBACK=1 and its keys), so a reader can follow the citation after the page changes.
- Records a profile cites go in the committed evidence store (
scripts/lib/evidence_store.py,
docs/evidence/), which refuses any record containing a secret.
- Version the scraper. The script, its settings (rate, User-Agent, selectors) and the plan
card are committed or kept beside the data. A second person must be able to re-run it and see what changed.
- Write down every transform from raw page to published figure: what was parsed, dropped,
deduplicated or corrected by hand.
7. The law in the United States, as read on 2026-11-13
Every statement in this section was read through scripts/ca_research.py (search, extract and CourtListener) on 2026-11-13; the sources are linked. It is a map for reporters and editors, not legal advice. Before a publication-facing decision rests on any of it, re-verify with legal-research-specialist‘s protocol (a citator check included) and, where the stakes are real, ask counsel. Never add a holding to this section from memory.
7.1 The Computer Fraud and Abuse Act and Van Buren v. United States
- The case. Van Buren v. United States, No. 19-783 (U.S. June 3, 2021). Justice Barrett
wrote for six Justices; Justice Thomas dissented, joined by the Chief Justice and Justice Alito (opinion, LII).
- The holding. An individual “exceeds authorized access” under 18 U.S.C. § 1030(a)(2) “when
he accesses a computer with authorization but then obtains information located in particular areas of the computer—such as files, folders, or databases—that are off limits to him”. The provision “does not cover those who, like Van Buren, have improper motives for obtaining information that is otherwise available to them” (Part IV and the opening of the opinion).
- Gates up or down. The Court read the statute as a “gates-up-or-down inquiry—one either can
or cannot access a computer system, and one either can or cannot access certain areas within the system”.
- Terms of service. The Court noted that many websites authorize access only on agreement
to terms of service, and that the Government’s broader reading would have reached violations of those terms too.
- Left open. Footnote 8: the Court did “not address whether this inquiry turns only on
technological (or ‘code-based’) limitations on access, or instead also looks to limits contained in contracts or policies”.
7.2 hiQ Labs v. LinkedIn (9th Cir. 2022)
- The case. hiQ Labs, Inc. v. LinkedIn Corp., No. 17-16783, 31 F.4th 1180 (9th Cir.
Apr. 18, 2022), on remand from the Supreme Court after Van Buren (opinion, Ninth Circuit; CourtListener).
- What it decided. It again affirmed a preliminary injunction stopping LinkedIn from
blocking hiQ’s access to public member profiles. The panel concluded that hiQ “raised a serious question” whether the CFAA’s “without authorization” concept is inapplicable where prior authorization is not generally required but a particular person, or bot, is refused access, and that Van Buren‘s reasoning reinforced that reading (Proskauer New Media and Technology Law Blog, 21 Apr 2022). A public website “has erected no gates to lift or lower in the first place” (JLCW case note).
- What it did not decide. It was a preliminary-injunction ruling on “serious questions”, not
a final judgment, and the scraping was later held to breach LinkedIn’s user agreement (§2.2). The CFAA question is narrower than “scraping public data is legal”.
7.3 The Justice Department’s charging policy
The Justice Manual, 9-48.000 (justice.gov), read 2026-11-13. It is prosecutorial policy, not law, and can change:
- The Department will not charge “exceeds authorized access” unless the division of a computer
into areas “is established in a computational sense, that is, through computer code or configuration, rather than through contracts, terms of service agreements, or employee policies”.
- “A CFAA prosecution may not be brought on the theory that a defendant exceeds authorized
access solely by violating an access restriction contained in a contractual agreement or term of service with an Internet service provider or web service available to the general public.”
- But: “when authorizers later expressly revoke authorization—for example, through
unambiguous written cease and desist communications that defendants receive and understand—the Department will consider defendants from that point onward not to be authorized.” This is why a cease-and-desist stops us (§4.3).
- Prosecutors should decline cases where the conduct consisted of, and was intended as,
good-faith security research, defined as testing, investigating or correcting a security flaw or vulnerability. It is not a journalism exception.
7.4 AI agents acting for a user
Amazon.com Services, LLC v. Perplexity AI, Inc., No. 26-1444 (9th Cir. Aug. 4, 2026), vacated a preliminary injunction against Perplexity’s Comet browser agent: on that record it was the user, not the agent’s developer, who “accesses” Amazon’s computers under the CFAA and California’s CDAFA (Cooley; Jones Day). For us the lesson is the reverse: an AI agent we send to collect data is our scraper, and this whole skill applies to it.
7.5 Copyright in what we collect
Facts are not copyrightable. A compilation of facts is protected only for the original way its facts are “selected, coordinated, or arranged”, not for the facts themselves (Feist Publications, Inc. v. Rural Telephone Service Co., 499 U.S. 340 (1991), LII). Scraping facts and reporting them is a different act from republishing a site’s text, photos or its arrangement. We report facts, quote briefly and link; we do not republish scraped text or images wholesale.
7.6 Other claims a site can bring
State computer-access laws (California’s CDAFA appears beside the CFAA in *Amazon v. Perplexity) and contract claims (hiQ*, §2.2) are separate from the CFAA. A favourable CFAA reading does not answer them.
8. The scraping plan card
Fill this before the first request and keep it with the data. An editor reads it before a scheduled or large scrape starts, and before anything from it is published.
| Field | Answer |
|---|---|
| Target site and URLs | |
| Story need: the question this data answers | |
| Why not an API, a download, the owner or a records request (§1) | |
| Terms of use / licence: URL, date read, the clause quoted (§2) | |
| Licence verdict: usable, not usable (non-commercial/research-only), or editor question | |
robots.txt: URL, date read, rules for our paths, Crawl-delay (§3) |
|
| User-Agent sent (§4.1) | |
| Rate: requests per minute, concurrency, schedule (§4.2) | |
| Fields kept; personal data present; inclusion-gate result (§5) | |
| Where raw files, logs and archives live (§6) | |
| Editor’s public-interest exception, if any: who, when, why | |
| Stop events and what was done (§4.3) |
Sources
Read on 2026-11-13 through scripts/ca_research.py (search, extract and CourtListener search), run directory $TMPDIR/jr/p10-1/research.
- Van Buren v. United States, No. 19-783 (U.S. 2021): opinion and syllabus, LII.
- hiQ Labs v. LinkedIn, 31 F.4th 1180 (9th Cir. 2022): Ninth Circuit opinion PDF and
CourtListener (search results), Proskauer, JLCW case note; the 2022 district-court contract ruling and consent judgment, ZwillGen and Meitar.
- U.S. Department of Justice, Justice Manual 9-48.000, Computer Fraud and Abuse Act.
- Meta Platforms v. Bright Data (N.D. Cal. 2024): Farella Braun + Martel; Quinn Emanuel.
- Amazon.com Services v. Perplexity AI, No. 26-1444 (9th Cir. 2026): CourtListener search,
Cooley, Jones Day.
- Feist Publications v. Rural Telephone Service, 499 U.S. 340 (1991): LII.
- RFC 9309, Robots Exclusion Protocol (RFC Editor).
- GIJN, “Digging Up Hidden Data with the Web Inspector” (Samantha Sunne, IRE21).
- Mindy McAdams, Python Beginners (University of Florida), “Web Scraping, Part 3”.
- Crawlee API reference,
BasicCrawlerOptions. - Pagallo and Ciani, “Anatomy of Web Data Scraping: Ethics, Standards, and the Troubles of the
Law” (SSRN, 2023).
- Indicator, David Weisz, “No code, no budget: 7 web scrapers put to the test” (7 Jul 2026),
for the state of no-code tools; its point that such tools now get past protections like Cloudflare is exactly what §4.3 forbids us to use them for.
Cross-references
osint-automation-frameworks— sibling: framework and module choice, passive-first design.public-records-research-specialist— find the official record before scraping.legal-research-specialist— method for every legal statement; CourtListener quota rules.patriot-private-citizen-inclusion-gate— before keeping or naming a private person.foia-and-sunshine-laws— when a public body holds the data.media-verification-specialist— archiving and verifying captured pages.document-research-specialist— ingesting what was collected.
Safety
- Never bypass a technical barrier: no login we are not entitled to, no CAPTCHA solving,
no challenge bypass, no spoofed or rotating identity, no proxies to evade a block.
- Never state the law from memory. §7 is what was read, with dates. New questions go
through legal-research-specialist and scripts/ca_research.py.
- Non-commercial or research-only data is not ours to use, however we could get it.
- Stop on a cease-and-desist and escalate; never “finish the run first”.
- Minimise personal data and gate private people before anything is kept or named.
- Leave the site as we found it. If a scraper could slow a small site, it runs slower or
not at all.
