« Russians, not Ralsky, now rule the spam world | Main | Blacklist stats suggest anti-spam progress »

November 28, 2005

FTC fudges spam filter study

The U.S. Federal Trade Commission just announced the results of a study of spam filters and address harvesting. Unfortunately, the agency's research sheds little new insight into the spam problem.

To test the effectiveness of spam filtering, the FTC created dozens of spam trap email accounts at each of three different (and unnamed) Internet service providers. Two of the ISPs offered spam filtering; the third didn't.

After two weeks, the email accounts at the ISP without spam filters received 2,129 junk emails. The accounts at the two filtered ISPs netted a total of just 469 and 95 spams. According to the study, the two ISPs with spam filters respectively blocked 78% and 96% of spam messages.

Interesting numbers, but when you look a little closer, things don't add up.

For some reason, the FTC report doesn't mention whether the filtered accounts had spam folders. (To my knowledge, the spam filters employed by most big webmail providers don't just delete all spam; they shunt suspicious stuff in to a special junk folder.)

Instead, the FTC researchers make a rather tenuous extrapolation, apparently in the dark about how much spam was rejected by the ISPs mail server or landed in each account's spam folder. According to the report,

FTC staff was able to calculate the percentage of spam messages blocked by the two ISPs’ spam filters by comparing the number of messages received in each of the Unfiltered Addresses to the number of messages received in Filtered ISP 1 and in Filtered ISP 2.

In other words, the FTC assumed each ISP received exactly the same amount of spam, and used the unfiltered ISP as a control. That's a pretty big assumption -- a fact the FTC seemed to acknowledge in a footnote to the report:

We assumed that spammers who harvested the addresses were not biased in favor or against a particular ISP when sending spam. It is possible, however, that the number of spam messages sent to the Unfiltered Addresses differed from the number of messages sent to Filtered ISP 1 or Filtered ISP 2.

It's common knowledge that many spammers, frustrated by spam filters, are currently targeting smaller ISPs that don't offer spam filtering to their users. Had its researchers paid more attention to methodology, the FTC might have shed some interesting new quantitative light on this phenomenon. Unfortunately, this is a missed opportunity.

The FTC report also examined how spammers harvest email addresses from web sites and Usenet. The agency found that unprotected, published FTC spam-trap addresses quickly began receiving spam. But this information has been already been gleaned by much more methodical researchers.

Posted by brian at November 28, 2005 11:53 PM