Internet2 Next Generation Infrastructure Update: First Member Traffic Moved onto the New Backbone, Remaining Service Migrations Planned Over Next Six Weeks

By Chris Wilkinson, Internet2 Network Services Director of Planning and Architecture

Estimated reading time: 5 minutes

Internet2 last week migrated the first of its member connections onto the Next Generation Infrastructure (NGI). This major milestone marks the start of member traffic flow onto the new Internet2 backbone, following the successful completion of the NGI shim phase.

Working with teams at the Connecticut Education Network (CEN) and Utah Education and Telehealth Network (UETN), we transitioned an initial set of their BGP peering sessions from our legacy network onto the Cisco 8200 platform. These software-driven migrations leveraged the Cisco Network Services Orchestrator (NSO), with the results demonstrating that the translation scripts are working effectively and traffic is flowing as expected on the new infrastructure.

In Progress: Member-Facing Peering Migrations

During the next three weeks, more traffic will transition to the new backbone as the NGI team rapidly moves approximately 800 of the 2,000 total remaining BGP sessions from the legacy network. In this first phase of the service migration process, the team is working closely with operational collaborators at each regional network to move member-facing peering exchanges and research and education services router-by-router.

View the Internet2 NGI BGP Migration Schedule and progress dashboard.

Prior to each scheduled migration, a series of scripts scrapes configurations from the legacy devices and translates them into language that can be parsed by Cisco’s NSO automation platform. NSO then provisions the final on-device configurations for the new Cisco 8200 platform. 

Most of the configuration can be deployed in preparation for the maintenance window, with a few final pieces activated during the window using a few simple command lines for each BGP session. These commands first gracefully deactivate the participant’s legacy BGP session to allow traffic to shed to alternate paths. Then the replacement session on the new Cisco router is activated. In addition, Internet2 has developed a toolset that will take a snapshot before and after of the BGP routes received and intelligently compare those states to confirm the translation scripts are performing accurately.

Our high-level approach is to start with routers that have the smallest number of sessions, allowing engineers to make any necessary adjustments as we scale up to migrate devices with more sessions. In the interim, interconnects between the legacy and NGI networks are handling the exchange between those that have and have not yet migrated.

What’s Next: OESS, Wave Services, and Commercial Peering Migrations

During the next several weeks, the NGI team will concurrently prepare for or perform OESS service migrations, Wave service migrations, and Commercial Peering migrations. All migrations are on track to ramp down by the end of September.

OESS services, which encompass layer 2 and layer 3 virtual circuit networks on MPLS-enabled switches, will follow a similar migration path to what is outlined above for member-facing I2PX and R&E BGP services. One dependency for the OESS/Cloud migration strategy is that all access control lists (ACLs) managed by OESS must be moved to the Cisco interfaces just prior to the migration. As a result, during the 7-10 day window for OESS migrations, any required ACL changes will need to be conducted manually in coordination with the Internet2 NOC.

With Ciena Waveserver 5 devices already in service on the NGI 400G backbone, we are also moving 39 10G and 13 100G AL1S Wave Services from traditional transponder cards to the new Waveserver 5 flex-grid technology. The transition approach will focus on hub and adjacent sites grouped together by region, with an intention to compartmentalize the Wave service migrations into small, verifiable events. Beginning in August, we will migrate regions from west to east, easing into the more complex layer 1 configurations on the East Coast over the course of a six-week period.

Plans are also underway to migrate Internet2’s seven commercial peering locations. Each location has its own unique configuration and collectively they support several hundred peering connections. The scope includes moving Rapid Private Interconnect (RPI) and Internet2 Peer Exchange (I2PX) physical connections and BGP peerings from the legacy devices to the Cisco 8200 platform. Internet2 Cloud Connect (I2CC) services will move to the Cisco Network Convergence System (NCS) which is required to support layer 2 shaping. 

Under the current provisional schedule, Chicago will be the first of the peering sites to migrate this week, followed generally by one site per week (i.e., Dallas, San Jose, New York, Seattle) through mid-September. The timeline for Ashburn and Los Angeles will be extended across six weeks to accommodate larger, more complex moves—we are coordinating a suite relocation and subsequent cross-connect moves with Equinix in Ashburn and bringing up a new peering site in Los Angeles. 

For the majority of the peering locations, the current I2PX connections are mere feet away from where they need to go to connect to the NGI equipment. This should make for a smooth and speedy transition that ensures system continuity during the session migration and verification process. 

I2CC and RPI will be moved in lockstep with the OESS transition. These migrations are slated for mid- to late-September in light of dependencies on other services. This plan puts us on track to complete the full migration of services by the end of September, followed by decommissioning and removal of legacy equipment through the end of the year.

A Thank You to the Community

The above activities are the culmination of years of effort from the Internet2 community and staff teams in many organizations, including Internet2, GlobalNOC at Indiana University, Ciena, Cisco, Lumen, GDT, and global partners.

If you have questions about NGI or want to learn more about the next phases, please email the team at networkdevelopment@internet2.edu.

Resources

Watch the video about Next Generation Infrastructure. Internet2 Director of Operations and Engineering, Chris Robb, reviews what has been accomplished with the NGI project.