AI Scraping Bots Overwhelm University Libraries and Digital Archives
Academic libraries across North America, Europe, and Oceania are experiencing severe disruptions from AI data-scraping bots that crash servers and slow access for human users. While major AI companies have backed a standardized opt-out signal, compliance remains voluntary and does not cover all third-party scrapers.
Academic libraries and cultural heritage institutions are facing swarms of data-scraping bots that are slowing or crashing websites intended for students, researchers, and the public. Reports of these bot attacks on libraries began appearing in late 2024 across North America, Europe, and Oceania.
The scale of the automated traffic is significant. The University of North Carolina at Chapel Hill reported receiving more than 11,000 searches for Finnish music originating from thousands of different internet addresses in a single day. These AI scraping bots sometimes ignore standard web protocols such as robots.txt instructions and can request thousands of documents per second. As a result, library sites are being knocked offline or slowed down for students, faculty, and staff, leaving regular users competing with infinite traffic from bot farms.
Public access to digitized historical and government documents is now at risk. Libraries face an increased workload and mounting technical challenges to maintain open access, creating a disincentive for institutions to make their collections available online due to bot abuse.
Some organizations have attempted to accommodate automated requests responsibly. Project Gutenberg offers a one-click download of its entire library, but notes that many AI bots are too poorly programmed to use this respectful method.
In response to the growing disruption, Anthropic, OpenAI, Google, and Microsoft have expressed support for honoring a single, standardized opt-out signal in websites' robots.txt files to block their web crawlers. However, compliance with the new opt-out signal is voluntary and excludes problematic bots operated by third-party companies and smaller AI startups.
The situation has led some institutions to consider restrictive measures, such as implementing CAPTCHAs or login requirements, to protect their servers. Experts argue that such steps could further hinder open access to digital archives.