University library websites are facing growing disruption from automated web crawlers, with a newly published report from the University of Edinburgh confirming that UK institutions are introducing protections to keep digital collections available to genuine users.

The problem has been documented across research institutions internationally. Bots can send enormous numbers of requests to online catalogues, archives and repositories, slowing services or making them unavailable to students and researchers. Some operators seek information for artificial intelligence systems, although not every automated request can be definitively linked to AI training.
UK universities introduce safeguards
The University of Edinburgh’s 2026 Library Annual Report, published on 16 November, says its digital systems team has faced increasing pressure from large volumes of crawlers seeking to harvest content for training large language models. The team, which supports more than 100 websites and services, says it has introduced measures to protect performance and preserve access for legitimate users.
Edinburgh’s report describes the pressure and its response. It does not establish that the university suffered a specific outage caused by AI bots.
At Loughborough University, a separate anti-scraping measure called Botstopper was introduced on 19 February 2026 for Library Catalogue Plus and Articles Plus. Some users accessing those services off-campus without a university VPN connection now see an automated check that they are not a bot.
Millions of requests blocked after US library outages
The consequences have been particularly clear at Duke University Libraries in the United States. Its technical team reported extended outages affecting the Duke Digital Repository, archives collection guides and books and media catalogue during April to June 2025.
In a June 2025 technical report, Duke said an open-source protection system called Anubis blocked around 90% of traffic across three affected applications, equivalent to more than four million unwanted HTTP requests a day. The report said service was restored and performance improved for ordinary users.
Those figures count requests, not four million individual bots. They also do not demonstrate that every blocked request came from a named AI company. Automated traffic can be distributed across many internet addresses and can have different purposes.
National libraries face the same problem
In August 2026, the International Federation of Library Associations and Institutions reported that web requests to US Library of Congress properties increased from around 50 million in June 2025 to almost 200 million in June 2026. It said traffic analysis attributed almost all of that growth to bots, which could slow or temporarily prevent access.
The American Library Association issued a warning in September about automated mass searches in library catalogues. It advised institutions to watch for unusually rapid, repeated searches and warned that bot activity can affect performance for genuine users.
The problem extends beyond traditional library websites. In a 2025 survey of 66 open-access repositories, the Confederation of Open Access Repositories found that more than 90% of respondents had encountered aggressive crawlers, commonly more than once a week. The organisation cautioned that the exact purpose of those crawlers cannot always be established.
Could blocking bots also block the public?
Libraries have traditionally welcomed responsible search-engine crawlers because they help people discover collections. More aggressive systems can overwhelm the same infrastructure, however, forcing institutions to choose between keeping information openly accessible and protecting their services.
Defences come with trade-offs. Duke warns that users with older browsers or disabled JavaScript and cookies may be mistakenly blocked. The repositories organisation has likewise warned that protective measures can obstruct legitimate people and well-behaved automated tools.
In a CTV News interview published on 17 November, Ontario Tech University researcher Peter Lewis discussed the strain that automated collection places on institutions and raised possible approaches to funding access, including negotiated agreements or charges for heavy use. Those are proposals, not an established charging system.
What remains unclear
The evidence establishes that aggressive automated crawling has caused service disruption and that British universities are actively defending their systems. It does not identify a single company responsible for all incidents, prove that every bot is collecting AI training data or provide a verified industry-wide financial cost.
The central question is how libraries can continue making knowledge available to the public without the traffic required to harvest that knowledge overwhelming their systems.
Sources
- University of Edinburgh Library, Re:Connect Annual Report 2026, p. 25
- Loughborough University, Botstopper announcement, 17 February 2026
- Duke University Libraries, Anubis Pilot Project Report, 26 June 2025
- IFLA, national libraries and AI crawling, 2026
- COAR, repository survey results, 3 June 2025
- CTV News interview with Peter Lewis, 17 November 2026

