[LWN.net
News from the source](/)
ContentWeekly EditionArchivesSearchKernelSecurityEvents calendarUnread commentsLWN FAQWrite for us
Weekly Edition
Archives
Search
Kernel
Security
Events calendar
Unread comments
LWN FAQ
Write for us
An update on the scraper situation
[LWN subscriber-only content]
LWN counts on its subscribing readers to support its mission of creating relevant human-written news coverage of the Linux and free-software communities. Please subscribe to LWN and help to keep us on the net.
As a special offer, subscribe to LWN now for at least six months, and receive a 25% discount on your subscription.
[Fighting the AI scraper bot
scourge](/Articles/1008897/)
Residential proxies
As was described last year, scraper attacks come from a huge number of
sources across the net. It is not unusual to see coordinated requests from
millions of unique IP addresses over the course of a few hours, each of
which hits the site at most two or three times. Attacker-controlled data,
such as the user-agent field, is entirely fictional; each hit is meant to
look like just another human with a web browser. There are ways to tell
the difference — the bots usually do not fetch images or CSS, for example —
but, by the time that determination is made, the address in question will
not be used again. Blocking the address at that point is just a waste of
time.
This traffic comes predominantly from residential and mobile networks,
directed by central command-and-control nodes. Software is installed on
ordinary systems that takes orders from a control node, fetches web pages
on demand, and forwards the resulting data back to the controller. Much of
the time, this activity occurs without the knowledge or consent of the
owner of the device in question. The term "residential proxies" is used to
describe systems that are used in this way.
There are a few different (on the surface, at least) types of operator
running residential-proxy networks to attack web sites. One type is purely
criminal, running scrapers on systems that have been compromised with some
sort of malware. At the beginning of the year, Google acted
to take down a bot network called IPIDEA and provided a lot of
information about how these operations work. The shutdown of IPIDEA
correlated with a significant reduction in scraper traffic here at LWN;
things were relatively peaceful for a few months. That period of peace has
since come to an end, though.
[acted
to take down a bot network called IPIDEA](https://cloud.google.com/blog/topics/threat-intelligence/disrupting-largest-residential-proxy-network)
More recently, media-streaming devices have been identified
as a major carrier of malicious scraping software. Sometimes the devices
are compromised at the source; other times, they are just poorly secured
and easily compromised after the fact.
The second sort of operator works more overtly, pretending to a degree of
legitimacy and offering "ethically sourced" IP addresses. A company called
Bright Data is one of the most prominent of these; it happily advertises
its prowess at getting around web-site access controls and traffic limits.
Bright Data offers a "free" VPN service; all that is needed is for the user
to give Bright Data the ability to route traffic through the user's device
— to become a part of the company's residential-proxy network, in other
words. Every phone or other device that makes use of this VPN becomes yet
another endpoint that will be used to attack web sites.
There are many other examples of this type of operator out there; often
they offer a library that app developers can link into their offerings and
be paid for hijacking their users' network connections. One of them even
sent us a query about running an ad for its SDK on LWN; that was, it
suffices to say, a short conversation. In general, these companies range
from those that aspire toward some appearance of legitimacy, advertising
"GDPR compliance" for example, to others that are just overtly sleazy.
While these residential-proxy networks are used for web-site scraping, it
is worth emphasizing that these operators have the ability to run code that
accesses resources on whatever networks millions of devices happen to be
connected to. To assume that this type of access would only be used for
scraping would be naive at best.
Then, of course, there are the high-profile companies developing models as
their core business. These companies do their own scraping; the traffic
that can be easily attributed to them is clearly identified in the
user-agent field and, as a general rule, observes measures like
robots.txt. They, too, will scrape an entire site, repeatedly,
seemingly on the theory that articles written in 2003 might somehow have
changed in the last day, but they do not generate overwhelming amounts of
traffic from millions of systems and are not the biggest problem.
What isn't clear is who is using the residential proxies; somebody
is paying them to run these attacks on web sites. There is no
evidence (that I am aware of) that the frontier-model companies are using
those networks. If were to turn out that they are doing so, though,
the increase in global astonishment would barely register. Those companies
are feeding their models somehow, they are not forthcoming about how they
get their training data, and they have not distinguished themselves with
their level of respect toward content creators — or toward anybody who
might have concerns about their operations.
For every public model, though, there must be a vast number of undercover
models. Many companies are surely trying to build their own; after all, we
are reliably informed that AI is going to take over the world and the
companies that come out on top of that race will be worth untold amounts of
money. There must be shadowy government agencies in many countries working
on their own models and groping for training data wherever they can find
it. Large-scale criminal organizations (to the extent that they are
distinct from governments) probably also want to have their own models.
These tools are seen as weapons, and there is an arms race underway. The
Internet as a whole is caught in the crossfire.
Defending the open Internet
In response to all of this, web-site operators have been scrambling to
defend their sites while minimizing the effect on their actual users. Anubis, which attempts to fend off scrapers by
requiring a proof of work, is now widespread. Other sites use commercial
services, which sometimes make themselves known with a "prove you are
human" button. Or sites force users to pick out squares containing
streetlights (but only those with LED bulbs), place puzzle pieces, or hum a
song while holding down the space bar. Many site features have been placed
behind login gates or paywalls. Some sites attempt to actively poison the
data sent to scrapers with tools like iocaine.
Both the need to set up and maintain these mechanisms, and the requirement
that users cope with them to access a web site, constitute a heavy tax
placed on the world as a whole by scrapers and those who pay them.
Recently, LWN was subjected what was, by far, the heaviest scraper attack
yet. Thanks to the defenses that have been implemented, the site bore the
traffic well enough that most actual readers probably did not even notice.
There have been requests to describe the measures we have taken to defend
the site; for obvious reasons we do not wish to discuss them in any detail.
It is an arms race at this level too.
What we can say is that we have tried to minimize the impact on real
readers as much as possible. We have not gone with tools like Anubis,
partly because it causes annoying delays for those trying to get to the
site, but also partly because it seems inevitable that the scrapers will
eventually find their way around it. Indeed, there are some indications
that is already happening. A proof-of-work requirement is not a huge
obstacle when you have millions of other people's machines to do the work
on.
There is also a desire to not impede the operation of legitimate search
engines, the Internet Archive, and other such groups. Some sites may add
explicit allowlists to, for example, give the dominant search engine access
to the site. Such measures have the effect of further entrenching a
monopoly that already serves us poorly and should be avoided. We have,
thus far, succeeded in that.
We have aggressively optimized parts of the site, and found ways to
minimize expensive operations during times when the site is under attack.
Anonymous readers may occasionally encounter one of those measures;
logged-in users will not. Amusingly, the response time when the site is
under attack is often better than during the calm times, when the defensive
measures are dormant. We have learned better than to think that the
problem is solved, though; consideration must be given to our next steps
once the current measures are no longer effective.
On July 2, Google announced
that it had, in coordination with the US Federal Bureau of Investigation
and others, taken down a residential-proxy network called "NetNut". For
the time being, that action would, indeed, seem to have succeeded in
reducing the level of scraper attacks somewhat. Experience shows, though,
that this welcome peace will only last so long. Google takes pains to
point out that its Play Store will now check for NetNut-infected apps, but
all of the major vendors are silent on the topic of why it is so easy to
put apps with residential-proxy functionality into their app stores.
It would be good to find a more lasting solution before the entire Internet
is driven behind defensive walls, and the open network that inspired so
much creativity is lost. The industry that is driving these attacks seems
entirely at ease with turning independent web sites into smoking craters
after having pillaged their contents — an attitude that extends to the
planet and its economies as well. Some of us, though, object to that idea
and will fight against it. Someday, with luck, the world as a whole will
decide to hold the companies behind large language models and related
technologies to a minimal ethical standard. Until then, though, this
behavior will continue, and we will have no choice but to defend ourselves
against it.
How to avoid running a residential proxy (especially on Android phones)?
Posted Jul 10, 2026 15:59 UTC (Fri)
by KJ7RRV (subscriber, #153595)
[Link] (3 responses)
How to avoid running a residential proxy (especially on Android phones)?
Posted Jul 10, 2026 16:25 UTC (Fri)
by farnz (subscriber, #17727)
[Link]
Other than that, you can ask Android to tell you how much mobile data an app used; go to "Settings", then "Apps", and it's in the "App Info" screen for individual apps. You can also disable "Background data" if you're suspicious of an app - that stops it using data when you don't have it open and are on mobile; IIRC, this also covers WiFi networks set as "metered", but ICBW on that one.
How to avoid running a residential proxy (especially on Android phones)?
Posted Jul 10, 2026 20:26 UTC (Fri)
by mirabilos (subscriber, #84359)
[Link]
Note this is hearsay. I have not installed that äpp nor sniffed its traffic myself. But it was the first one where I heard about what is now called residential proxies, via “trojaned” äpps, and from multiple sources.
I’m sure the “link fetchers” from GAFAM äpps can also occasionally do that…
How to avoid running a residential proxy (especially on Android phones)?
Posted Jul 10, 2026 21:42 UTC (Fri)
by rgmoore (✭ supporter ✭, #75)
[Link]
Some of the other posters have commented on how to avoid these kinds of proxies on your own system, but I want to point out that individual action won't be enough to solve the broader problem. I don't want to discourage anyone from trying to keep their own system clean, but we will never get rid of these kinds of proxies by encouraging each person with a mobile device to avoid them. You need to block access to a large majority of the available devices to make any real progress, and voluntary programs that involve real effort are never going to get that level of participation. This stuff needs to be taken care of at the system level- getting the apps out of app stores, making this kind of thing clearly illegal and prosecuting the offenders, etc.- not the individual user level.
Thanks for your efforts and keep up the good work!
Posted Jul 10, 2026 16:08 UTC (Fri)
by wtarreau (subscriber, #51152)
[Link] (2 responses)
I think that many of us here are totally aware of the problem these bots and proxies are causing to web sites like this one, and the difficulties in fighting them. At least I can say that I haven't noticed anything abnormal on the site here, so your actions were fine from the user experience perspective. Thanks for this!
You're right, it's important never to publicly explain the counter-measures that you apply. Very often some are extremely simple and effective (some easy tricks I've deployed 9 months ago that I imagined would only last one week are still working fine). Also they're often very specific to the site and would hardly adapt to other ones (except for the main principle), so there's little to share by explaining everyone how your specific site is fighting these.
I noticed a 30% drop of traffic on July 2nd, after a 50% one on June 25th that I couldn't explain (mostly attributed to user-agent "sleepbot"). So yes, it seems that such networks are progressively getting dismantled, probably to re-appear somewhere else soon, given that infected browsers (and their unsuspecting users) are just waiting for another C&C to take care of them :-/
I must confess I'm a bit worried about the risk of losing a lot of legit content indexing on the net in the coming years due to installed counter-measures against non-humans. If sites cannot be found via search engines it will become a problem. All this due to AI startups racing in training their own models (or variants).
Maybe it would work better to set up a static central registry of the whole internet's contents that could be scraped by such companies as much as they want without killing small web sites. It could still take a lot of time before we start to see something like this happen though.
Thanks for your efforts and keep up the good work!
Posted Jul 10, 2026 17:15 UTC (Fri)
by daroc (editor, #160859)
[Link] (1 responses)
The idea is that you contribute resources to their project, they have _one_ scraper that downloads web content in a polite way (respecting rate limits and robots.txt), and then anyone who wants to can make use of the scraped content without having to re-scrape it.
There are some problems with the approach, but I'm generally pretty happy when I see Common Crawl go by in the server logs because I know that's a bunch of unrelated projects that don't need to send us multiple requests.
Thanks for your efforts and keep up the good work!
Posted Jul 10, 2026 17:46 UTC (Fri)
by wtarreau (subscriber, #51152)
[Link]
Oh, thanks for the link, I wasn't aware. I'll try to make sure not to block that one!
Preventing this at the app store level is hard
Posted Jul 10, 2026 20:42 UTC (Fri)
by roc (subscriber, #30627)
[Link]
Google's rules already forbid non-user-authorized "residential proxies": https://support.google.com/googleplay/android-developer/a...
But if the user gives genuine consent to it, should they still be banned? That's a tough call.
https://support.google.com/googleplay/android-developer/a...
Of course developers can and do violate those rules. Then the problem is that it's not really possible in general to detect that code is going to break those rules just by inspecting it. So how do you detect violations other than by people reporting them and then shutting them down after the fact? That's roughly what happens now and it's better than nothing, but still whack-a-mole.
IP blocks should be neither authentication nor authorization
Posted Jul 10, 2026 20:59 UTC (Fri)
by quotemstr (subscriber, #45331)
[Link]
It already is in large part. Have you tried browsing the internet from outside a well-known residential or end-user-VPN IP block? Tons of sites block access out of the gate or put up so many "click the traffic lights" gates that they might as well have blocked you.
Residential VPNs are, yes, often scummy, but also an understandable reaction to much of the internet using routing tables as a proxy for proof of humanity. You can't stamp them out, either: there will always be people willing to trade their home bandwidth for trinkets. You can't stop them without stamping out general-purpose computing altogether, and I don't think anyone wants that.
What we need instead is an open protocol through which network clients can provide proof of interactive humanity under zero knowledge in such a way as to resist cloning and Sybil attacks. I believe recent advances in zkVMs and remote attestation make such a protocol possible. If we had it,
- residential VPNs would cease to be special and incentives for scummy tricks would disappear,
- assurance of humanity could shift from annoying interstitials to automatic protocol exchange (because we could prove recent human interaction end-to-end authenticated from hardware without sacrificing privacy), and
- site operators could, in principle, create a market for non-interactive access permits (e.g. under an anonymous cap-and-trade scheme), naturally rate-limiting accesses while avoiding the monopoly-reinforcing effects of just whitelisting IP blocks owned by this or that archive or search engine.
Interactive users couldn't sell their interactivity for trinkets without setting up robots to manipulate their input devices or doing tedious link-clicking themselves. Channel- and hardware-binding would mitigate proxy attacks, again anonymously. Remote attestation doesn't have to be a privacy nightmare. It can enable privacy!
Such a scheme would be compatible with free software operating systems too, since input attestation would be pass-through. You'd just prove, remotely, that the same TPM generated both the input attestation and your TLS session key. Linux can drive this hardware just fine.
Privacy-preserving protocols like this have become practical just recently, over the past few years. It's a shame we haven't yet begun to explore their potential. The alternative is something like the Cloudflare Monetization Gateway [1], which accomplishes similar goals, but without the privacy or the democracy. A world where Cloudflare becomes de-facto internet gatekeeper is a worse world than one with an open attestation ecostystem.
[1] https://blog.cloudflare.com/monetization-gateway/
https://blog.cloudflare.com/monetization-gateway/
Copyright © 2026, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds