Defend against a billion spammers (and win)

This is the story of how a filtering appliance successfully stood up to more than a billion pieces of spam- and virus-laden e-mail.While other units buckled under this deluge, Sendio Inc.'s I.C.E. Box, which looks at spam in a completely different way, shot down 100 percent of the bad e-mail, generated no false positives and successfully delivered the good e-mail. It took on an army of bad guys and won.We learned about I.C.E.'s superior spam-fighting capabilities the hard way'trying to sort through our own mountain of junk e-mail. We set up the GCN Lab test network to take in a lot of spam and viruses for our testing zoo. It's an important part of how we test filtering and e-mail scanning devices.And while this approach worked for many years, we had become victims of our own success. The lab network was getting over 10,000 spam e-mail messages per hour, along with perhaps two or three items of legitimate e-mail. But the good ones were not being delivered in a timely manner, and sometimes not at all.The lab had purchased a Barracuda 200 Spam Firewall to shoot down all the bad e-mail. We figured that, because we only have a handful of valid users, the lower-end Barracuda should be fine.But we didn't take into account the huge volume of spam. The Barracuda could not handle the load. It was running at between 95 and 98 percent capacity, creating a three-hour queue for e-mail. And even though it was only a small percentage of the total, so much was getting through that the e-mail server was getting overloaded even after the filtering step.The problem was that even though we only have about five valid user accounts, the Barracuda was processing everything that came into the lab, even if it was going to former employees or to nonvalid, made-up addresses such as bob@gcnlab.com or hrmanager@gcnlab.com.What we needed was for the Barracuda to drop everything that was not going to a valid account.Why waste processing power when we know there is no valid recipient on the other end? But Barracuda tech support's response was less than stellar.Apparently, such validation is a feature that exists on Barracuda models, but only on the 400s and above.This is a ridiculous limitation, considering the 200s are marketed to smaller networks, which can be crippled without this needed feature.Considering that admins of smaller networks, who could easily identify the valid users, would get the most out of this feature, this blatant up-selling is distasteful from a company that formerly held our respect. If anyone wants to buy a slightly used Barracuda 200, let us know.Given that we were facing 10,000 spam e-mails per hour, with spikes going much higher than that, adding up to millions per week and about a billion every two months, we were open to new options.That is when we heard about Sendio's I.C.E. (Intercept, Confirm or Eliminate) Box appliance. The I.C.E. Box takes a different approach to spam filtering, and it is one we feel will make all other filtering appliances obsolete.The I.C.E. Box performs Sender Address Verification (SAV). SAV is fundamentally different than filtering because it is not content-based: Messages are not read or scanned, and no guesses are made as to proper content. Every message is checked to evaluate whether or not the purported sender of the message has been added to the recipient's Accept List.We had a bit of a rocky relationship with Sendio to start out, because the company insists on taking over the brunt of the setup work, something they do for every customer. You give them your IP addresses and let them know what holes you are opening in the firewall for the appliance.While most agencies will appreciate this, we in the lab like to get our hands dirty. Still, when the glowing blue I.C.E. Box arrived, it was literally ready to plug in and go.That hardware on the box is impressive, though it is not as necessary as with filtering-type appliances since it is not scanning every e-mail. It has a 3-GHz Pentium 4 processor, two 160GB hard drives, two NIC cards and 1GB of RAM. It fits into a 1U space in a rack and should be powerful enough to support about 1,000 users without extra networking.When an e-mail comes into the I.C.E. Box, it is stored in a temporary folder. A challenge e-mail is sent back to the sender explaining that this is the first time they have communicated with the recipient since the I.C.E. Box was installed.It asks the sender to simply reply to the challenge. When they do, their e-mail is added to the approved-sender list, and the original mail is sent forward.The sender receives a note thanking them for their participation, and telling that their original mail is being delivered and that in the future they won't have to go through the challenge-and-response program. If no response is given within two weeks (a default value which can be changed in the administration interface), the original e-mail is deleted.You can log into the I.C.E. Box by going to its IP address and typing in your e-mail user name and password.From there, you can see all your held mail as well as the approved user list.If you happen to see a valid user in the pending folder, you can manually authorize them, assuming the administrator has given you permission to do so.Or if someone should get onto the authorized list who should not be there, they can be removed.Also, users can be pre-approved so they never have to go through the challenge-and-response program. The user interface is intuitive and extremely speedy. It's one of the best Web interfaces we have seen.So, what if spammers reply to the e-mail challenge and become authorized users? It may sound hard to believe at first, but that won't happen.Spammers need to retain their anonymous status, and most of the time the server and routing info they put on their mail is faked so nobody can track them or reply to them. That means the challenge won't make it back to the real source.Additionally, computers that are not set up to respond to anyone generate most of the spam traffic. Even if spammers could receive a challenge, putting forth the effort to respond ruins their business model. After a month of testing and watching nearly a billion spam e-mails pass through the I.C.E. Box (most of which were natural, though we generated some), not one spammer ever replied to a challenge.And even if a spammer did somehow reply, the anomaly of seeing a spam e-mail in your box'we never saw one once the I.C.E. Box was installed'would trigger a user to log in manually and ban them. We doubt you will ever have to do this.And unless you have a valid user who can't be bothered to simply press reply to the challenge the first time they contact you, there will be no false positives.Remember also that e-mail coming from within your agency won't touch the I.C.E. Box, since it sits at your gateway and is unconcerned about interoffice traffic.In fact, you should set the I.C.E. Box to automatically kill any mail that appears to come from your own domain, because if the I.C.E. Box sees it, it means the mail actually came from the outside. Doing this eliminates a common spam technique where the mail seems to come from your boss or co-workers, but is in fact completely fake.The one area where you might get something blocked that you want is with bulk mail. If you have signed up for, say, one of GCN's newsfeeds or the weekly special list at Best Buy, those newsletters could be tagged as bulk mail and sent to quarantine. Bulk mailers normally don't reply to challenges either, so the mail will sit there for two weeks and get deleted. However, the I.C.E. Box checks a bulk mailer list that legitimate senders register with, and tags the mail as bulk in the pending folder. So a user can easily look in their quarantine folder and authorize the bulk mail that they want to receive.There is also a universal setting that lets an administrator allow or block bulk mail by default. We set ours to block bulk mail, but authorizing our vitally important Sci Fi Network newsletter took only one step.And what happened to our original problem with the Barracuda tying itself up trying to process millions of e-mail to bogus recipients? That does not happen with the I.C.E. Box. It smartly reads our mail server's user table. Any mail not addressed to someone on the table is dropped without even sending a challenge.You can set the I.C.E. Box to scan your user table at regular intervals, perhaps every night at midnight, to see if any users have been added or dropped. The I.C.E. Box will then configure itself appropriately. The admin never has to touch the I.C.E. Box itself. If they just do what they always do with the mail server for a new user, the I.C.E. Box will follow their lead. Of course you can trigger a forced look at the table, or manually add a new user to the I.C.E. Box if the new person needs instant authorization, or you have someone leaving your agency under bad terms and no longer want to accept e-mail for them.And although the I.C.E. Box uses SAV to avoid content scanning, it does still scan for viruses, as we found out by slamming it with several hundred. All antivirus scanning takes place within the initial SMTP portion of the transaction.Only e-mails from existing domains, as determined by DNS checking, which are sent to existing addresses, as determined by the I.C.E. Box, are subject to virus scanning. The I.C.E. Box does not store or pass through an e-mail with a known virus. If a message is found to contain a virus, a 550 (FATAL) response code is returned during the SMTP portion of the transaction. This 550 response code explains that the message was rejected due to the presence of a virus and includes the name of the virus.The 550 response code from the I.C.E. Box is not a bounce, but a reject. Therefore, the I.C.E. Box is not sending the virus back to the purported sender.In the end, the greatest thing we can say about the I.C.E. Box is that it works. It took a nearly crippled e-mail system that we thought was beyond repair and fixed it as soon as it was installed. The I.C.E. Box was our magic bullet, and the lab intends to purchase our test unit to protect the network. It easily earns our Reviewer's Choice designation and is on track to be one of the best products we have reviewed all year.Traditional filtering-appliance companies should be put on notice. Once people learn about the I.C.E. Box and how it works, such old-school appliances won't be needed anymore.