The author of a large 1.5 million-page website reveals that 99% of their traffic consists of bots and scrapers, highlighting the severe impact of web scraping on legitimate site operations. The article details the technical and financial challenges of fighting these automated requests while trying to maintain service for real users.
Background
Web scraping has become a major issue for large content websites, often consuming bandwidth and resources meant for legitimate users. Many site owners struggle to distinguish between beneficial bots (like search engine crawlers) and malicious scrapers.
- Source
- Hacker News (RSS)
- Published
- Aug 7, 2026 at 10:51 PM
- Score
- 6.0 / 10