A Weblog About Topics and Issues Discussed in the Book Spam Kings by Brian McWilliams

« "Reformed" Richter still blacklisted | Main | Smathers sentenced »

August 16, 2005

Blue Frog and the blurry hash

Blue Security logo

I'm all in favor of creative solutions to the spam problem, but I have some serious concerns about the privacy protection features in an anti-spam system from Blue Security.

Blue Security has created what it calls a Do Not Intrude Registry -- an encrypted database of email addresses of Internet users who've signed up to use the free service.

Blue Security's system was launched despite (or perhaps because of) a June 2004 decision by the Federal Trade Commission not to implement a national do-not-email registry. (A team of eminent computer scientists advised the FTC that such databases present serious practical and technical problems.)

Under the Blue Security system, spammers are supposed to scrub their mailing lists using a free program provided by Blue Security. Doing so produces a list of "protected" email addresses -- presumably those of Blug Frog users who are on both the spammer's list and the Blue Security list of users who've asked to have their emails protected by the service.

If a spammer sends junk email to any of those addresses and fails to heed warnings, the system is designed to retaliate by flooding the spammer's online order forms with complaints (a concept Blue Security calls "revenue loss induction.")

The ethics of such fight-abuse-with-abuse retaliation are a worthy topic of debate in their own right. But my concerns resulted when I decided to test out Blue Security's list-cleaning system.

I tried running the cleaner against a list composed of spammer email addresses. To my surprise, a large number of the addresses appeared in the "protected" list created by Blue Security's list-cleaning program.

Next, I tried cleaning a mailing list I obtained from spammers. Before running the Blue Security program, I "munged" some of the addresses, changing the letter "i" to the letter "y". Hence, addresses ending in earthlink.net became earthlynk.net; adelphia.net became adelphya.net; ucdavis.edu became ucdavys.edu, etc.

Again, several of these non-existent addresses showed up in the list of those "protected" by Blue Security.

I asked Blue Security for an explanation. The company's marketing director said that the system is designed to generate some "fake entries" in an effort to protect the privacy of users. The company calls this approach "blurry hashing," a concept it explains in more detail in this whitepaper .

I leave it up to better minds to decide whether Blue Security's technology is an elegant solution to the problems that befuddled the FTC's expert panel (which included Professors Ed Felten, Avi Rubin, and Matthew Bishop).

But I can say that this built in "noise" in Blue Security's list-cleaning registry is going to give it an air of unreliability with spammers. As the company's white paper explains:

When a spammer notices that an e-mail address has been deleted from his list, he has no way of knowing if it was filtered because it was a legitimate user's e-mail address or if it matched one of the random entries in the blurry hashed Registry.

Blue Security says there is a 1/5000 probability that an address will be considered protected even though it is not in the registry. My tests suggest the probability is much greater -- more like 1/1000. So, if a spammer "cleans" a list of 10M addresses using the system, 10,000 will be erroneously flagged as protected.

Is it reasonable to expect spammers to use the Blue Security registry when the removal system isn't completely accurate?

Posted by brian at August 16, 2005 12:23 PM

Comments

I see no problem with a system that allows those affected by spam to lodge a single complaint thru the only mechanism to actually reach the spammer - their order form. The Blue Frog does this with a 1 spam = 1 complaint ratio, which is more than fair. I don't see how irate users lodging a complaint can be considered 'abuse' or a DDOS.

As for the Blue Security registry producing a false positive every few thousand entries (you claimed 1 in 1000 but didn't publish your numbers or details of the test, Blue claims 1 in 5000 max), I really don't care. I doubt spammers really care about their data quality to 99.99% either. In fact, criticizing the registry design is the weakest anti-Blue Security argument I've heard so far. I would have expected better from you given your experience with these lowlifes, that the only thing which will stop them spamming is a ton of complaints that end up costing them money.

Posted by: at September 2, 2005 6:41 PM

Few people actually get it. Fewer still, get it and then attempt a concept to do something about it. Fewer still, (I think two or three) have actually hit upon the actual solution. But nothing gets through the noise or private interest groups... or... the other 99.99% of those who still don't get it.

If indeed Brian actually reads the comments to this blog, I should like to discuss this -- privately. :-)

Posted by: Fred Showker at October 7, 2005 9:53 AM

Ignorance is bliss, and unfortunately often cloaked in the mantel of expertise.

Posted by: Yellow Hornet at October 25, 2005 4:16 AM

My understanding is that if a BF user gets a spam message, they "complain" via a form. If they get another spam message from the same source (determined by BS experts) all BF users automagically "complain" via the form. 1 spam = 1 complaint, 2 spams = 15001 complaints, hence the DDOS. But I could be wrong about this, it's not 100% clear.

Posted by: Adam at December 6, 2005 1:51 AM

Adam,

No, that's not how it works. 1 spam = 1 complaint, 2 spams = 2 complaints, etc. Like others have said, that's perfectly legitimate. If someone sends me email, I can send an email back asking them to stop. And, yes, Brian's beef about the blurry hashing makes no sense.

Hooray for Blue Security. Finally, someone has built an effective system to fight spam.

Posted by: Mark Scheuern at April 11, 2006 8:29 PM

Yes that is correct. 1 complaint per spam received,it is totally proportional.

In addition complaints are not sent out straight away. The spammer is contacted and given 10 days to clean his lists. If he refuses the complaints procedure starts.

There is no DDOS, since careful measures have been taken not to overload the offending website, but at the same time making sure our complaints are heard.

I can see this article is quite old now, and since the time of writing the bluefrog model has been refined further.

Perhaps readers may like to do their own fact finding on this subject, since there does seem to be a lot of misinformation floating around.

Posted by: vres at May 4, 2006 7:30 AM

I am tired of this bend over and accept spam
philosophy. Something has to be done and
until someone has better solution then
BlueFrog will be running on every workstation
on my network. Bend over is no way to live
life.

Posted by: MadAsHell at May 6, 2006 10:00 AM

I'm not keen on the blurry hash approach -- after all, the privacy protection this grants is minimal (i.e., the recent spam attacks on members likely included some non-members... but why would the spammers care?), and I agree that it isn't quite "playing fair".

What's your opinion on the service *without* that element, though? I'm sure Blue Security can be convinced to remove the blurry hash -- then their tool is just a tool to automate the complaint process -- which is beyond recrimination, isn't it?

Think about it. Limiting us to manually visiting each website, finding the form, and manually typing out a personalized removal request would be like limiting the spammers to personally surfing the web looking for email addresses, and manually typing out each spam email. Not likely, right? But mass emailing isn't illegal or even necessarily unethical, when you're sending email to people who want it and have a way to unsubscribe. Same thing with submitting complaints -- it's legal and ethical for me to complain to a company who either spammed me directly or paid someone else to.

Okay, so many of us want to complain for each spam. Why should that process be manual and extremely expensive? That's the kind of tedious problem computers were invented to solve! There's nothing wrong with posting a single removal request for a single spam received. If the spammers automate the sending process to pump out massive quantities of spam to mostly unwelcoming recipients, they have to be ready to handle massive quantities of complaints when that process is also automated.

This isn't retribution; it's playing by the rules even though many spammers (violating CAN-SPAM, using open relays and bot nets, sending from servers in China, etc.) are NOT thus restricted.

So yes, Blue Security: please drop the blurry hash (and don't protect emails who might not want it!). Then you've got a truly beautiful thing going.

Brian Williams -- I hope you'll read this and consider it. Hopefully it's clear that I'm not the average uninformed spam-hater (I'm a software engineer, actually); feel free to contact me if you have questions.

Cheers!

Posted by: Rob at May 8, 2006 12:28 AM

No, they don't need to drop the blurry hash. There is no harm if it "protects" a few thousand people who are not Blue Frog members. There is no chance that it will accidentally remove anyone from a mailing list which he or she asked to receive, because the "do not intrude" registry is only used to remove addresses from UNSOLICITED mailing lists.

Posted by: Anonymous at May 15, 2006 5:49 AM