An update on residential proxies and the scraper situation
lwn.net
July 10, 2026
16 min read
🔥🔥🔥🔥🔥
64/100
Summary
Scraper attacks originate from numerous sources, often involving millions of unique IP addresses making multiple requests in a short period. The user-agent field in these requests is typically manipulated to appear as legitimate browser traffic.
Key Takeaways
- Scraper attacks originate from millions of unique IP addresses, often using residential and mobile networks controlled by central command-and-control nodes.
- Media-streaming devices have been identified as significant carriers of malicious scraping software, either through initial compromise or poor security.
- Companies like Bright Data offer "ethically sourced" IP addresses and VPN services that allow users' devices to be used as endpoints in residential-proxy networks for web scraping.
- Residential-proxy networks have the potential to access resources on connected networks, raising concerns beyond just web scraping activities.
Community Sentiment
Positives
- A better common crawl could reset the baseline for information accessibility, leveling the playing field against AI labs hoarding data.
- The push for a common crawl database would eliminate the need for scraping, benefiting both website operators and users.
- Residential proxies are seen as the last anti-hero of the open internet, providing access where traditional methods fail.
Concerns
- The sheer volume of scraping is alarming, with millions of requests raising ethical concerns and the potential for abuse.
- Using residential proxies is highly unethical, with many comparing it to botnets, which undermines trust in internet practices.
- The intensity of bot traffic is damaging to low-volume users, as aggressive bot defense measures end up blocking legitimate crawlers.
