Internet2 opens HOV lanes

As Big Science gets bigger, demands on research networks to enable collaboration are growing exponentially. The Energy Department has seen traffic on its Energy Sciences Network (ESnet), which links researchers at its major research labs and universities, increase tenfold every 47 months since 1990.'To a certain extent, this is inevitable,' said William Johnston, ESnet department head at DOE's Lawrence Berkeley National Laboratory. 'Scientific instruments follow Moore's law,' and more sensitive and powerful instruments are producing more data.As a smaller number of larger, more sophisticated instruments are built, collaboration communities are growing and sharing more data. ESnet had an average steady-state load of 1.5 gigabits/sec on its New York-Chicago- San Francisco link in July 2006. This summer, the granddaddy of all scientific instruments, the Large Hadron Collider particle accelerator operated by the European Organization for Nuclear Research, is expected to go online, promising a quantum leap in the amount of experimental data being shuttled among scientists worldwide.'A bandwidth of 10 gigabits/sec site-to-site connectivity is needed now,' according to a 2006 DOE assessment of the needs of ESnet, 'and 100 gigabits/sec will be needed by 2010.'In fact, that capacity is already here. The bandwidth is being provided for ESnet through a 2006 partnership with Internet2, an advanced research and education network operated by a consortium of companies, universities and other organizations. The backbone is from Level 3 Communications, which finished the 13,500-mile optical fiber network for Internet2 in June 2007.'To go from zero to 100 gigabits/sec in nine months is impressive,' said Randy Brogle, director of the research and education segment at Level 3. That expanded bandwidth is provided by bundling 10 gigabits/sec optical channels through 300 photonic network elements from Infinera. Using dense wavelength division multiplexing, additional 10 gigabits/sec channels can be added by installing additional cards at each end of the fiber-optic links.Further scaling will be possible as optical interfaces increase to 40 gigabits/sec and then 100 gigabits/sec. 'It can easily scale to 400 gigabits/sec,' Brogle said of the Internet2 backbone.The 100 gigabits/sec speed is expected to accommodate ESnet's bandwidth needs until around 2012. After that, requirements are not so clear.'We have a couple of segments that are using the 100 gigabits/ sec now, so we will be going to 200 this year,' said Rob Vietzke, Internet2's executive director of network services.But the ability to create dedicated channels on short notice to move large volumes of data from point to point is as important as total bandwidth. For this, Internet2 is using Dynamic Circuit Networking, a technique that combines the best qualities of IP and circuit-switched networks to segment traffic and provide dedicated channels on demand.Dedicated commercial circuits between two points are not economically feasible when a laboratory only needs such a channel for a few minutes or hours. And a packet-switched network cannot afford to build its entire infrastructure to accommodate the peak needs of those few minutes or hours. Virtual circuits, provided by dedicating optical wavelengths or paths between two points, allow efficient use of network resources.'The network is being seen like a supercomputer, as a schedulable resource,' said Internet2 spokeswoman Lauren Rotman.To reserve and claim circuits, ESnet developed the On-demand Secure Circuits and Advance Reservation System, a Web-based application that lets users allocate bandwidth, a job usually done by the network carrier. A virtual network operations center giving users a view into the Level 3 backbone is needed to enable OSCAR, Brogle said.'It needs to happen in real time,' he said. 'We had to work with them on having a robust virtual [network operations center] between the two entities.'Internet2 was established in 1996 to pick up where the NSFNet Internet backbone left off.'After the Internet became commercialized, the education and research community felt it needed something to serve their needs,' Rotman said. 'We provide a breakable test bed' on which users can try new approaches without worrying about the operational requirements of a commercial network. It also provides bigger pipes to accommodate the needs of power users in the research community.The consortium operating Internet2 includes 212 universities, 70 corporations and 45 affiliate members. Not all users are doing Big Science. The Philadelphia Orchestra uses Internet2 to multicast performances. The New World Symphony, a teaching orchestra building a new facility in Miami, uses the network for distance-learning programs.But the research opportunities provided by the Large Hadron Collider, which will be used by scientists worldwide, was a major driver for the recent upgrade of Internet2 and ESnet. In 2006, Internet2 was a 10 gigabits/sec network with international links to research networks in 30 countries. By 2007, Level 3 had built out the 13,500-mile purpose- built Internet2 optical backbone with bundled 10 gigabits/sec channels to provide 100 gigabits/sec capacity. The backbone infrastructure now includes:Dynamic Circuit Networking uses Multiprotocol Label Switching to establish dedicated circuits at layers 2 and 3 of the Open Systems Interconnection stack for moving large amounts of data for a brief time, and OSCAR sets up the routes.Dynamic Circuits are a great match for Internet2 and ESnet, Vietzke said. 'You get quality of service, [and] you're not on a shared network,' he said. 'We have done some tests where we have configured a 100 gigabits/sec span between New York and Chicago in seven minutes.'Eventually, circuits could be requested and provisioned quickly without a user going to a Web site to reserve the channel.The bandwidth and the ability to provide dedicated channels on short notice was just what DOE needed for ESnet. The network serves all DOE research laboratories and provides connections with major international research and education networks. Twenty years ago, the first generation of ESnet consisted of 45 megabits/sec point-to-point links. This was upgraded to 155 megabits/sec, and by 2006, the third-generation network core was a 10 gigabits/sec packet-over-Sonet ring connected with labs by single commercial telecom circuits.But the trend of scientists worldwide collaborating on fewer, larger, more data-intense experiments required a major expansion to a bigger, faster and more reliable fourth-generation ESnet. The need for 100 gigabits/sec and faster links in the near future became obvious.Because the fastest growth in bandwidth consumption is with high-end users, ESnet 4 is being built with dual nationwide cores. The first core is a 10 gigabits/sec IP ring network for routine administrative and small-science traffic. The second core, the Science Data Network, will be a 100 gigabits/sec ring created by aggregating 10 gigabits/sec optical channels. The SDN core will interconnect with the administrative core and have nine hubs with peering routers for other networks.'We're right on schedule,' said Johnston, although there has been some slippage. 'We expected to have the core of the [storage-area network] deployed some time ago,' but contracts for the big switches were not awarded on time.The 10 gigabits/sec IP national core consisting of five rings was completed several months ago. 'We have some segments of the SDN in place,' Johnston said. Completion of the first full 100 gigabits/sec SDN ring is awaiting delivery of the core routing and switching equipment. Upgrades of metropolitan area networks in San Francisco, Chicago and Long Island, N.Y., also have been completed.'Delivery of the waves is not an issue,' Johnston said.