Fighting AI by blocking the Wayback Machine is a terrible trade
%20(1).avif)
Written by Derrick KityoPublishers are restricting the archive reporters use to check the past. I understand the fear of AI scraping. I think the public record deserves a better answer.
I don’t want Google, OpenAI or anyone else treating the web as an unlimited supply of free work. I also don’t want the fight over AI to leave us unable to check what used to be on it.
That’s where the dispute over the Wayback Machine has got to. Publishers want more control over their journalism. Some are restricting the organisation that preserves old versions of their pages.
On 19 October 2026, Internet Archive founder Brewster Kahle made a fresh public case against those restrictions in The Atlantic. I agree with him about the risk to the public record. I think publishers are making a terrible trade when protecting their work also makes its history harder to inspect.
There’s a reporter on the other side of the block
BJ Mendelson explains the problem rather better than an abstract argument about data. In a public statement supporting the Internet Archive, he describes relying on old reporting from publications that have closed or stopped functioning properly.
Tech policy journalist Ashley Belanger describes using archived websites, terms of service and government documents to compare what changed. That’s a fairly basic part of holding an organisation to its word. You need the earlier version as well as the one it wants you to see today.
A useful old page might contain a name, a promise or a detail that suddenly matters years later. Nobody gets to know in advance which sentence a future reporter will need.
When that record disappears, the cost lands on someone trying to establish what happened. They may have the current statement and a broken link where the previous one used to be.
The publishers have a real problem
I can understand why a publisher would worry about another route into its content. Paying people to report a story and watching somebody else package it into an answer is a miserable prospect.
In January, the Guardian’s Robert Hahn told Nieman Lab that structured archive access could offer a route for AI extraction. The Guardian had restricted article access through parts of the Archive’s systems while keeping homepages and other landing pages available.
The same report described the New York Times actively blocking Archive crawlers. Its stated concern was unauthorised access to its journalism, including by AI companies.
Those are different restrictions, with consequences worth examining separately. Describing every publisher as having deleted its entire archive would make this argument easier and less accurate.
The Guardian told Nieman Lab its measures were preventive; it hadn’t documented a specific instance of AI companies taking its pages through the Wayback Machine. That leaves a genuine question about risk. It doesn’t make the possibility of abuse imaginary.
The question I’d put to publishers is whether the restriction leaves a workable way for people to consult the record. The answer matters even when the original concern is justified.
Want to apply this to your own website or app?
Tell me what you are working on and where it is getting stuck. I will help you turn the useful ideas into shipped work.
The damage can reach far beyond one newspaper
In a May 2026 analysis, Nieman Lab counted 342 local news sites among 382 news sites restricting at least one tracked archive-related crawler. Several large newspaper groups appeared in its sample.
That’s a count of restrictions in a particular sample, rather than proof that 342 complete archives have vanished. Some tracked rules even named a bot the Archive says it doesn’t use. The distinction matters when describing the scale of the problem.
The underlying concern is still serious. If restrictions spread across a newspaper group, decisions made centrally can affect the record of many different places. A reporter in one town has very little influence over that choice.
I don’t think the people making those decisions need to want history erased for the result to be damaging. A routine infrastructure change can do it perfectly well.
Human access deserves its own argument
Mark Graham, the Wayback Machine’s director, says the Archive uses rate limits, filtering and monitoring to respond to abusive access. The Archive explains its position and those safeguards here. That’s its account of its controls, rather than an independent guarantee that nobody can misuse the service.
I’d want publishers and archivists to work through the practical distinction between someone consulting an old article and a business extracting material at scale. They have different purposes, volumes and consequences.
Some cases will be difficult. A newsroom researching years of coverage may need more than a handful of pages. An archive still has to run a service people can use.
That’s a reason to agree access arrangements, improve abuse reporting and review restrictions when circumstances change. A blanket block is easy to leave in place long after anyone has checked whether it still makes sense.
Preserving public journalism also needs to sit alongside proper handling of personal information and legitimate removal requests. Taking the public record seriously means dealing with those questions carefully.
A website’s history belongs in the conversation
People who build and manage websites have a part in this too. When I’m thinking about a site’s future, I want to ask what happens to the material already published.
Who keeps the old URLs working? What survives a CMS migration? Where can somebody find a previous public statement? Those decisions deserve attention while there’s still something to preserve.
I’d like the companies making money from AI to put serious support behind that preservation work, with arrangements that respect publishers’ rights. Leaving publishers and a nonprofit archive to absorb the consequences is a poor outcome for everyone who needs reliable sources.
People in power can rewrite their own websites. The rest of us need a record that lets us notice.
Tags
SEO & AI searchFrequently asked questions
Why are some publishers restricting the Wayback Machine?
Publishers have raised concerns that AI companies could obtain their journalism through archived copies. Restrictions differ: Nieman Lab's January reporting described limits on particular Guardian interfaces and active crawler blocking by the New York Times. A restriction isn't necessarily a complete removal of a publication's history.
Does the figure of 342 local news sites mean their complete archives have disappeared?
No. Nieman Lab counted 342 local sites among 382 news sites restricting at least one tracked archive-related crawler in its May 2026 sample. It wasn't a count of complete archives lost. The methodology also included some rules for a bot the Archive says it doesn't use.
What does the Internet Archive say it does about abusive scraping?
The Archive says it uses rate limiting, filtering and monitoring to identify and respond to abusive access. Those are its stated safeguards, not a guarantee that misuse is impossible. Its FAQ also explains the role of archived pages in reporting and research.
.jpg)
