Now Online: First Internet2 Next Generation Infrastructure 400G Ethernet Links for New Routed Network

Subscribe for more like this



Estimated reading time: 4 minutes

By Chris Wilkinson, Internet2 Network Services Director of Planning and Architecture

Internet2 marks the beginning of this week having deployed its first operational 400 Gbps Ethernet connections for its new Next Generation Infrastructure (NGI) platform. Cisco 8201 routers in adjacent cities are now connected using high capacity coherent optical links provided by Ciena Waveserver 5 transponders. This work builds on the successful deployment of the first 800 Gbps single-carrier optical circuit between Phoenix and Tucson that was announced at the end of February.

The seven completed packet network links as of March 22, 2021, include: Albany, NY to Hartford, CT; Ashburn, VA to Pittsburgh, PA; Dallas, TX to Kansas City, MO; El Paso, TX to Tucson, AZ; Las Vegas, NV to Salt Lake City, UT; Phoenix, AZ to Tucson, AZ; Reno, NV to Salt Lake City, UT.

Current Deployment Status of 400G packet backbone as of March 22, 2021

Each intra-city link represented in the map above connects 400Gbps native interfaces on adjacent Cisco 8201 routers via Ciena Waveserver 5 modules; 400 Gbps active optical cables (AOCs) are used to connect the key devices. The Waveserver 5 encapsulates the 400G Ethernet frames within a 600-800 Gbps single-carrier optical circuit which traverses the Ciena 6500 flex-grid optical line system.

An example block diagram of an inter-metro 400G link

On the optical platform, single carrier optical performance between cities has consistently performed above the rates in the original pre-deployment models. For example, 800 Gbps of Ethernet capacity was achieved on the 518 kilometer amp chain between El Paso and Tucson; the projected performance on this segment as originally modeled was 650 Gbps. 

A sample of the original model compared to rates delivered by the Waveserver 5 platform

LinkPre-deployment Modeled Line Rate Validated Ethernet Performance Day 1 Operational Configuration
Albany – Hartford
549 km
N/A1700 Gbps600 Gbps, OTN Enabled
400GE + 100GE + OTL4.4
Ashburn – Pittsburgh
453 km
600 Gbps800 Gbps600 Gbps, OTN Enabled
400GE + 100GE + OTL4.4
Dallas – Kansas City
928 km
550 Gbps700 Gbps600 Gbps, OTN Enabled
400GE + 100GE + OTL4.4
El Paso – Tucson
518 km
650 Gbps800 Gbps600 Gbps, OTN Enabled
400GE + 100GE + OTL4.4
Phoenix – Tucson
202 km
700 Gbps800 Gbps600 Gbps, OTN Enabled
400GE + 100GE + OTL4.4
Las Vegas – Salt Lake
731 km
550 Gbps700 Gbps600 Gbps, OTN Enabled
400GE + 100GE + OTL4.4
Reno – Salt Lake
876 km
550 Gbps700 Gbps700 Gbps, Ethernet-only
400GE + 100GE + 100GE + 100GE
1 No model was run for the specified route

As the Internet2 optical system provides 400GE, 100GE, and 10GE services using a uniform transport layer, Mixed OTN-Ethernet mapping is required for the majority of the Waveserver 5 links being deployed. While this does reduce the delivered capacity on a given link, the deployed solution meets NGI 400 Gbps packet design requirements while also providing 10 Gbps and 100 Gbps wave services; additionally, this combined strategy reduces overall power consumption.

Next Steps: Router Configuration, Testing and Connector Integration

The NGI packet platform deployment will proceed with a focus on east to west connectivity at 400 Gbps. The deployment will also include the movement of router configurations from the lab to the future production network platform, as well as the initial use of Cisco Network Services Orchestrator (NSO) for system deployment.

In the coming weeks, the team plans to build a linear prototype path to stress test the baseline backbone hardware and software, culminating in the launch of 400 Gbps of test traffic into the platform. 

Example of planned linear prototype path and 400G+ perfSONAR test configuration

Deployment, testing, and system validation will expand to the entire footprint in April, followed shortly by connector integration in May and June.

The above activities are the culmination of thousands of hours of effort from both the Internet2 community and teams in many organizations, including Internet2, GlobalNOC at Indiana University, Ciena, Cisco, Lumen, and GDT.