Simon Willison highlights a growing problem of abusive web crawlers consuming disproportionate server resources, citing Konstantin Ryabitsev's observation that git.kernel.org spends more CPU cycles rendering commits for scrapers than on all legitimate access combined. Willison expresses concern about the implications for Datasette, which serves a large number of crawlable web pages.
Background
Web scraping and crawling have become increasingly resource-intensive for open-source projects and public repositories, with many sites reporting that automated scrapers consume more bandwidth and compute than genuine users.
- Source
- Simon Willison
- Published
- Sep 8, 2026 at 07:08 AM
- Score
- 6.0 / 10