CGNET GateKeeper: Maximum Spam and Virus Protection with Minimum Effort

Why Users Are Still Complaining about Spam

By now, spam is nothing new. Virtually all IT departments are doing something about it, either in-house or through a service provider. It's fairly common to be able to remove more than 90% of the spam coming at us from the Internet. But users are still complaining. What do we have to do to satisfy our users?

The conventional wisdom is that there's little more we can do. We're told that the creators of both spam and viruses are smart, dedicated people who are highly motivated to create better "products." This is particularly true of spammers, because the sad truth is that sending spam makes money. When you send out a million advertisements, a very small rate of return can pay for the effort, if all you are doing is sending bulk email.

There's also a tendency to blame the users who are still complaining. Perhaps their email address is on too many Websites or in too many newsgroups. It's something they should fix.

This kind of analysis, however, is not really fair to the user, because it overstates how good a job spam filtering products are doing. From the users' point of view, we have to get very close to removing 100% of the spam from their mailboxes before they may perceive an adequate improvement.

Consider the following graph:

Today, approximately 70% of all email on the Internet is spam. If a user has no spam protection, then 70% percent of the messages in the inbox are spam. Out of 100 messages, 70 are spam and 30 are real. If you filter out 30% of the spam (21 messages), then 49 messages in the mailbox will be spam and 30 will be real. 49 spams out of 79 total messages is about 62%. So filtering out 30% of the spam reduces the percentage of spam in the mailbox, which is what the user perceives, by only 8%.

As you increase the percentage of spam filtered, the effect on the user is not directly proportional. The effect is non-linear, producing rapid reductions in the percentages of spam in the user's mailbox only as you approach 100% removal.

Today, the best spam-filtering products on the market remove about 93% of the spam from the Internet. Taking out more spam results in an unacceptably high number of "false positives," i.e., real messages removed incorrectly as spam. Removing 93% of the spam from the Internet still leaves about 13% of the messages in the average user's mailbox being spam, however. Many users still find that percentage of spam unacceptable.

Add to this the variation among users from the other causes mentioned above, such as differential exposure of certain addresses, and the effect gets worse. For example, if approximately 90% of the messages bound for a particularly exposed user are spam, filtering out 95% of the spam still leaves more than 30% of the messages in the box being spam.

When the amount of spam you can remove is more than 90%, adding just a few points to the percentage of spam removed produces a big payoff, because the curve is steep as it approaches 100%. For example, when CGNET discovered that it was able to increase the percentage of spam that it removed by six points, from 93% to 99%, it found that the amount of spam in the average user's mailbox declined from more than 13% to 2%, that is, to one-sixth the amount.

Going the Last Mile

Getting the percentage of spam removed as close as possible to 100% makes a big difference to users, then. But how is it possible?

Basically, we've combined two different approaches to spam control, filtering and router-based "squelching."

CGNET has used Brightmail Anti-Spam as its main spam protection for several years. We've been proud of this, because Brightmail consistently tops comparative reviews of spam fighters and almost always has the industry's lowest ratio of false positives. We've also supplied another industry-leading product, Trend Micro's Internet Virus Wall, to filter out email-borne viruses.

This solution worked as well as anything else out there, and better than most, but because some users were still unsatisfied, we kept trying.

Last fall, we began testing Spam Squelcher from the Electronic Privacy Group. The product, which was purchased this summer by Symantec, is now called the TurnTide Anti-Spam Router. TurnTide's technical approach differs significantly from Brightmail's.

Brightmail, like most of the products on the market, filters spam. It looks at each inbound email message and runs a lot of tests to determine if it is spam. If so, it removes the spam from the mail flow and quarantines it on one of our servers. Periodically, we notify each user of all quarantined email, in case a false positive does occur. In practice, our users only request quarantined messages to be resent about once for every 41,000 messages. Brightmail is really good that way.

TurnTide's anti-spam router uses a different approach. It never looks inside an email message. Instead, it looks at the routing information that is part of a transmission, including the sender, the addressee, how many addressees, and other information about the router-to-router conversation that facilitates any message. From this, it is able to identify about half as much spam as Brightmail does.

What the router does then is to slow down its reception of messages from any domains identified as probably sending spam. It also refuses to take on multiple "conversations" with devices from that domain. One result is that the amount of spam coming into CGNET's servers declines significantly. Another result is that the server that is sending the spam experiences difficulties in getting its messages sent to CGNET quickly.

The second-order consequences of this are even more interesting. On CGNET's end, the reduced amount of spam coming in seems not to include some of the spam that Brightmail was missing. It also seems to allow Brightmail to filter the remaining spam more efficiently. Also, it appears that at least some of the spammers have built routines into their programs to avoid addresses where mail is not delivered quickly. Thus, over time, the overall amount of spam being sent to CGNET has declined.

Another nice feature of TurnTide's system is that it generates no false positives at all! This is because it merely slows down the flow of mail, rather than removing any messages from it. The worst consequence of TurnTide mis-identifying a sender is that mail from it might be delayed, but never lost.

CGNET has been testing this system for several months, before and after the cut-over to TurnTide, both in terms of test mailboxes that attract spam and in terms of our overall measurement of the mail flow to our users. By combining this data for analysis, we've seen the overall amount of spam removed jump from 93% to 99%, as we mentioned above.

The Numbers

CGNET measures its spam protection in two principal ways. First, we maintain a number of test mailboxes where we can count how much spam gets through our service. Combining this with information about how much mail is filtered yields percentages of effectiveness.

Second, we have records of how many messages are filtered and how many are sent on to our actual users. By combining the data from these two sources, we can get a complete picture of how our system is functioning.

Before we installed the TurnTide Anti-Spam Router, Brightmail was our main defense against spam. Over a three-month measurement period, we got the results shown in the chart below:

Over this period, 68.2% of all messages coming to our users' domains over the Internet were spam. We filtered out about 93% of the spam, but 4.8% of all the messages sent, about 13% of the contents of the user's inbox, still were spam. The way we got these numbers was to establish the percentage of spam caught by looking at our test mailboxes and to then extend that percentage to our overall mail service, where the proportion of mail filtered to mail delivered was known.

After installing the TurnTide Anti-Spam Router, the situation in the following three months looked like this:

Spam from the Internet headed for our users' mailboxes first hit the TurnTide router, where 44.7% of the spam (30.5% of the total messages, good and bad) was deterred. The remaining messages were filtered by Brightmail. Brightmail's efficiency in filtering the remaining spam actually increased to 98.1% (37% of all messages). This left 2% of all the messages in the mailboxes (0.7% of all messages sent) being spam.

Why Brightmail did so well in combination with TurnTide is a matter of speculation. One explanation is that the kind of mail deterred by TurnTide is somewhat different from that filtered by Brightmail, so the mail that TurnTide does not deter is easier for Brightmail to filter. Another explanation is that the reduced volume of mail Brightmail has to handle allows it to work more efficiently. Regardless, the result is that the two products work much better together than separately. Perhaps this is why Symantec has recently bought both companies.

The Good News

To sum up, CGNET is finding it possible to filter out 99% of all spam by combining Brightmail's and TurnTide's solutions. Frankly, as far as we know, this is unprecedented. And because of the way users perceive the ratio of spam to mail in their inboxes, this increase is a difference they really notice. Going the last mile matters.

How GateKeeper Works

CGNET GateKeeper is a hosted solution. A customer changes its domain's MX record so that mail bound for the domain is sent to CGNET. CGNET then deters and filters the spam and removes the viruses, finally forwarding the mail on to the customer's servers.

In the process, if the customer wishes, CGNET can also store all good (non-spam, virus-free) messages for a designated time (the default is two weeks). This is done at no additional cost. It gives the customer the ability to retrieve any amount of good mail at any time, ranging from a single message that a user has mistakenly deleted to all the mail, in the event of some kind of local server and/or backup failure.

If the customer does not desire this service, however, the mail can be protected against spam and viruses without being stored at CGNET.

Maximum Effectiveness

It is not necessary at this point to repeat the story of how GateKeeper is now removing 99% of all Internet-based spam. What does bear mentioning, however, is CGNET's use of the Trend Micro Internet Virus Wall. This product is the industry leader, both in terms of its measured effectiveness and in terms of its market share. CGNET has been very satisfied with the Internet Virus Wall's performance, and Trend Micro's speed in responding to virus problems and furnishing updates.

Minimum Effort

Because GateKeeper is a hosted solution, all a customer's IT department has to do is to supply CGNET with information, including its domain's MX record and a list of its mailboxes. CGNET does the rest, including all maintenance and updating of the systems.

CGNET also provides 24/7/365 live telephone and email technical support, if backups needs to be restored, mail classified as spam needs to be resent, or for any other reason.

GateKeeper's cost is also competitively low, beginning at $2 per user per month and declining from there as the number of users in an organization increases.