You should probably check on your smart appliances
Published on 2026-07-14, 1008 words, 4 minutes to read
TL;DR: is your refrigerator running malware? If so, you better catch it!
The scraping problem is worse than anyone can imagine and thanks to my friends at Sourceware we have some real data to prove it.
I've been working more on Anubis' reputation database and I've run into a really weird discovery: 80-90% of the hits created by the honeypot feature are from IP addresses that do not belong to any existing threat monitoring lists.
Here's a breakdown of the honeypot hits Sourceware has gotten in the last few months:
Assessment of ./data/manually-submitted/sourceware/202607141625.txt against ./var/reputationdb.mmdb
In case this interests you, I have put the full tables in Appendix A: Full tables for the reputation database input.
Appendix A: Full tables for the reputation database input
Flags (of flagged addresses)
Categories (6 distinct, of flagged addresses)
Providers (126 distinct, of flagged addresses)
Methodology note: "provider" here means one of two things:
The company or organization associated with the IP address.
The place the list was gotten from.
For example, scaleway is based off of Scaleway's publicly posted IP address ranges, firehol-level1 is based on a daily snapshot of FireHOL's Level 1 IP list, and fdo is based on data contributed by the administrators of freedesktop.org.
Scaleway's publicly posted IP address ranges
(remainder snipped for brevity)
Countries (229 distinct, of all addresses)
This doesn't list data from 204 additional countries. Given that the ISO 3166-1 standard comprises 249 countries (193 of which are UN members), it's safe to say this is a global problem.
ASNs (21116 distinct, of all addresses)
There are 18069 more ASNs not listed.
How Anubis' honeypot works
In order to collect data on how widespread the scraper problem is, I added a honeypot feature to Anubis. On every challenge page it adds semantically invalid HTML akin to the following:
Visiting that page gets you cheap to generate vacuous anti-content that has two links to other pages. This is intended to get badly written scrapers caught in the honeypot so they scrape that instead of the protected website. I made it on a whim but thought it would be great for collecting data on how widespread this problem actually is.
This is a global problem
Based on the data I've seen, this is a global problem. If I had to guess where most of this traffic is coming from, it's from compromised smart appliances contributing traffic to proxy networks. I don't think there's any way to make a real impact on this problem without concerted simultaneous global action.
TL;DR: the scraping problem is actually widespread enough that web application firewalls like Anubis make sense.
Facts and circumstances may have changed since publication. Please contact me before jumping to conclusions if something seems wrong or unclear.
Tags: