DOE lab calls on big Bro for protection

Networking at Lawrence Berkeley National Laboratory is all about bandwidth, because visualization, simulation and other high-end applications eat up bandwidth voraciously.

The Energy Department's Berkeley, Calif., lab network supports about 15,000 devices and 4,000 users. To keep the bits flowing, the network services group kicks up bandwidth a notch whenever average usage on a link reaches 10 percent of capacity.

'We are not quality-of-service advocates,' senior network engineer Michael Bennett said. 'There is a lot of religious debate' about quality-of-service guarantees versus just boosting bandwidth. 'We've found throwing bandwidth at it is cheaper and simpler all around,' Bennett said.

But there are plenty of challenges in this approach, which often brings the lab staff to the bleeding edge of network technology.

'We've been running Gigabit Ethernet since before the standard was cooked,' Bennett said. 'We're starting to think about the transition to 10 Gbps now.'

The Institute of Electrical and Electronics Engineers adopted the 802.3ae standard for so-called 10-G Ethernet in June. But early hardware and applications for new standards seldom deliver full rates.

The lab's network services group, a perennial early adopter, usually tinkers with such products to develop their full potential.

Fast connections

The lab, a two-time Bandwidth Challenge winner at the annual High Performance Networking and Computing conference, plans to defend its title this month in Baltimore with a 10.2-Gbps connection between computer clusters running a high-speed visualization program. That is a threefold increase over last year's winning 3.3 Gbps.

Such speeds put a strain on security measures. Lawrence Berkeley users have free access to resources in their open environment, which emphasizes security applications rather than policy. The lab relies on Bro, an open-source intrusion detection system written by the lab's Vern Paxson in 1998.

Bro is not an acronym. 'It's named after Orwell's Big Brother,' Bennett said.

Like Big Brother, Bro watches everything. It consists of a packet-filtering event engine and a policy script interpreter that applies policy to observed events. Making it work in real time on a high-speed network is critical, Bennett said.

'You have to consider the worst-case scenario,' he said. 'If you don't capture a packet, it might be the one that brings a worm in.'

Bro first ran in the lab's Fast Ethernet demilitarized zone. About three years ago, as average traffic approached the 10-Mbps level, 'we moved our DMZ to Gig Ethernet,' Bennett said. 'At that time, there was no intrusion detection system that could run even close to Gig-E speeds.'
Bringing Bro up to speed was a matter of finding the right network interface card and the right drivers. Given the state of gigabit technology at the time, that meant tweaking and reworking the drivers. 'That's what it was all about, trying to get every bit per second out of the cards,' Bennett said.

The server card finally chosen was the SK-9844, a 1000Base-SX dual-port connector from Germany's SysKonnect GmbH.

'They needed a dual-port Gigabit Ethernet adapter' to deliver packets to multiple machines in real time, said Kenny Tam, SysKonnect's vice president of technical services. Using one dual-port NIC instead of two cards reduced the overhead for running Bro, he said.

When introduced, the SK-9844 was a single-driver NIC intended as a failover device for redundancy on Gigabit Ethernet networks.

Three drivers

'Three or four years ago, it was a niche product for applications like that,' Tam said. Partly thanks to the Lawrence Berkeley team's modification of the driver so it could be used with an optical splitter for the intrusion detection system, the NIC now comes with three drivers.

One driver is for the original failover application, one drives dual active ports and one is for link aggregation. The dual-active-port configuration serves security needs such as the lab's.

'The market for that kind of thing boomed overnight,' said Lester Fong, a SysKonnect technical support engineer. The company worked with the lab on the modification.

Although the NIC did not originally support the FreeBSD operating system, that is what Lawrence Berkeley developer Craig Leres used to modify the driver for Bro. By customizing the software, Leres got the cards working at their full gigabit rating.

Now there are off-the-shelf intrusion detection products for Gigabit Ethernet, but the network staff at Lawrence Berkeley will stick with Bro. They prefer openness.

'We like our system because we can tune it,' Bennett said. 'It's very flexible.'

Bro runs on ordinary 933-MHz Pentium III servers. In Leres' last trial, blasting packets at the intrusion detection system, Bro captured them at full wire speed without dropping a packet. For the time being, that's good enough.

'We're not quite to the point where we have to upgrade the DMZ to 10 Gbps,' Bennett said. And that's fortunate, because there is a scarcity of 10-Gbps hardware.

'We will eventually produce a 10-Gbps adapter,' Tam said. 'But the switch market leads the adapter market by eight to 12 months. We're waiting for the switch market to mature.'