• Skip to primary navigation
  • Skip to main content

F4N

  • Home
  • UK
  • World
  • Politics
  • Business
  • Travel
  • Sport
  • Opinion
  • Analysis
  • Profiles
NewsBusiness News · 19 Nov

AI Crawlers Disrupt Library Websites as UK Universities Strengthen Defences

UK universities are introducing defences against aggressive AI-related web crawling as libraries worldwide report disrupted online services.

Staff Reporter19 November 2026

University library websites are facing growing disruption from automated web crawlers, with a newly published report from the University of Edinburgh confirming that UK institutions are introducing protections to keep digital collections available to genuine users.

Fish 4 News illustration of a university library website overwhelmed by automated AI crawler traffic, with a digital overload warning

The problem has been documented across research institutions internationally. Bots can send enormous numbers of requests to online catalogues, archives and repositories, slowing services or making them unavailable to students and researchers. Some operators seek information for artificial intelligence systems, although not every automated request can be definitively linked to AI training.

UK universities introduce safeguards

The University of Edinburgh’s 2026 Library Annual Report, published on 16 November, says its digital systems team has faced increasing pressure from large volumes of crawlers seeking to harvest content for training large language models. The team, which supports more than 100 websites and services, says it has introduced measures to protect performance and preserve access for legitimate users.

Edinburgh’s report describes the pressure and its response. It does not establish that the university suffered a specific outage caused by AI bots.

At Loughborough University, a separate anti-scraping measure called Botstopper was introduced on 19 February 2026 for Library Catalogue Plus and Articles Plus. Some users accessing those services off-campus without a university VPN connection now see an automated check that they are not a bot.

Millions of requests blocked after US library outages

The consequences have been particularly clear at Duke University Libraries in the United States. Its technical team reported extended outages affecting the Duke Digital Repository, archives collection guides and books and media catalogue during April to June 2025.

In a June 2025 technical report, Duke said an open-source protection system called Anubis blocked around 90% of traffic across three affected applications, equivalent to more than four million unwanted HTTP requests a day. The report said service was restored and performance improved for ordinary users.

Those figures count requests, not four million individual bots. They also do not demonstrate that every blocked request came from a named AI company. Automated traffic can be distributed across many internet addresses and can have different purposes.

National libraries face the same problem

In August 2026, the International Federation of Library Associations and Institutions reported that web requests to US Library of Congress properties increased from around 50 million in June 2025 to almost 200 million in June 2026. It said traffic analysis attributed almost all of that growth to bots, which could slow or temporarily prevent access.

The American Library Association issued a warning in September about automated mass searches in library catalogues. It advised institutions to watch for unusually rapid, repeated searches and warned that bot activity can affect performance for genuine users.

The problem extends beyond traditional library websites. In a 2025 survey of 66 open-access repositories, the Confederation of Open Access Repositories found that more than 90% of respondents had encountered aggressive crawlers, commonly more than once a week. The organisation cautioned that the exact purpose of those crawlers cannot always be established.

Could blocking bots also block the public?

Libraries have traditionally welcomed responsible search-engine crawlers because they help people discover collections. More aggressive systems can overwhelm the same infrastructure, however, forcing institutions to choose between keeping information openly accessible and protecting their services.

Defences come with trade-offs. Duke warns that users with older browsers or disabled JavaScript and cookies may be mistakenly blocked. The repositories organisation has likewise warned that protective measures can obstruct legitimate people and well-behaved automated tools.

In a CTV News interview published on 17 November, Ontario Tech University researcher Peter Lewis discussed the strain that automated collection places on institutions and raised possible approaches to funding access, including negotiated agreements or charges for heavy use. Those are proposals, not an established charging system.

What remains unclear

The evidence establishes that aggressive automated crawling has caused service disruption and that British universities are actively defending their systems. It does not identify a single company responsible for all incidents, prove that every bot is collecting AI training data or provide a verified industry-wide financial cost.

The central question is how libraries can continue making knowledge available to the public without the traffic required to harvest that knowledge overwhelming their systems.

Sources

  • University of Edinburgh Library, Re:Connect Annual Report 2026, p. 25
  • Loughborough University, Botstopper announcement, 17 February 2026
  • Duke University Libraries, Anubis Pilot Project Report, 26 June 2025
  • IFLA, national libraries and AI crawling, 2026
  • COAR, repository survey results, 3 June 2025
  • CTV News interview with Peter Lewis, 17 November 2026

More from Fish 4 News

Editorial composite of a British court building with the headline 'Serco takes MoJ to court', illustrating the High Court procurement claim against the Ministry of Justice.
NewsBusiness News · 19 Nov

Serco takes Ministry of Justice to High Court in procurement dispute

Serco has filed a High Court procurement claim against the Ministry of Justice. The reason remains undisclosed amid scrutiny of justice contracts.

Illustrative composite of a TGJones-style high street shopfront with the text 'TGJONES FOR SALE?' and 'OWNER EXPLORES BUYER INTEREST'.
NewsBusiness News · 19 Nov

TGJones put up for sale less than 18 months after WHSmith takeover

TGJones is reviewing buyer offers less than 18 months after acquiring WHSmith’s high street shops. What the proposed sale means for stores, staff and Post Offices.

Fish 4 News

Independent news, context and analysis designed to help readers make up their own minds. News informs and verifies; opinion is clearly labelled.

Explore

  • Opinion
  • Analysis
  • Profiles
  • Editorial Standards
  • Corrections

Information

  • About Fish 4 News
  • Contact
  • Privacy Policy
  • Cookie Policy
  • Terms / Disclaimer
F4NFish 4 News

We don’t tell you what to think. We tell you what we know.

© 2026 Fish 4 News