Internet2 Next Generation Infrastructure Project Substantially Complete: Productive, Unexpected 2021 Ready for the History Books

By Rob Vietzke, Vice President, Network Services

Estimated reading time: 7 minutes

Although the planning for Internet2’s Next Generation Infrastructure (NGI) project has been underway since 2016, a huge surge of technical work occurred during the 2021 calendar year – including nationwide equipment replacement, development of a whole new automation system, and over 2,400 service transitions. Gratitude and appreciation are due to dozens of individuals throughout the community who helped achieve this milestone during the most unusual 20 months when technical meetings and installations had to be conducted virtually.

With over 2,400 individual services transitioned to the new infrastructure in October and November, and over 50% of the old network decommissioned in December, a process that began with community requirements gathering in 2016 is now coming to a close. 

So, what did we accomplish? What are the lessons learned? What’s next now that the transition is complete? Read on to find out more.

NGI Project Accomplishments

• Implemented a new software orchestration layer that will allow more rapid feature rollout through standardization and automation of configuration management. This will be the basis for many of our future service offerings, including enhancements to Cloud Connect and some of the promised edge-to-edge offerings envisioned in the original NGI requirements gathering.
• Designed a new platform agreement to bundle more bandwidth access together with new security and cloud access capabilities for regional network connectors. Almost 50% of the connectors have already moved to this new agreement and have started to adopt the expanded NGI capabilities.
• Deployed a new 100G attached performance node at every Internet2 node. This will enable community performance testing and be a model for how high-performance network federations like the U.S. research and education community can assure network quality.
• Deployed a new secure management network, further hardening the security posture of the network against increasingly sophisticated infrastructure attacks.
• Tested our team’s ability to design, deploy, and support complicated disaggregated network topologies (and to do it remotely). They both grew as a team and knocked it out of the park in terms of completing the deployment during a pandemic on a tight schedule.
• Replaced every backbone router with a smaller and more scalable device that saves power and offers immediately consumable spare ports.
• Replaced approximately 80% of Internet2’s 1998-era national fiber footprint with 2019-era fiber that offers greater capacity with less equipment. 
• Replaced the nX100 Gbps backbone with the new nX400 Gbps backbone, bringing at least 800 Gbps to every Internet2 node and often 1-3 Tbps of capacity.
• Savings from a 70% reduction in space and power combined with a nearly 40% reduction in equipment maintenance costs are enabling reinvestment in software, security, and other areas, while also allowing us to keep our costs controlled for regional network connectors.

Lessons Learned

An undeniable highlight of the overall NGI effort was the community’s involvement in the requirements gathering and procurement phases of the project. More than 100 community members wrote papers, participated in workshops, joined committee meetings, or served on a procurement team. These contributions, the design principles they brought about, and the overall achievements listed above can all be linked to that community direction-setting and collaboration. After two network rollouts driven by externalities, we are reminded of how rewarding the community process can be when we can engage fully.

While a full debrief hasn’t yet occurred, the team is already talking about choices that were made and options to better prepare for the next upgrade. The pandemic plays a large role in many of the initial thoughts, with the inability to meet in person for design meetings, inability to travel for installations, issues with shipping, and other impacts causing distractions throughout the project. Thoughts on how to better streamline outsourced resources, mock-up deployments more fully in advance, and consolidate various parts of the program seem to be emerging as well. We expect we’ll have a long list of observations and hope to memorialize them for the next team to pursue such a project in the future.

What’s Next

In early 2022, we’ll continue to wrap up final NGI tasks, including finishing the decommissioning and resale of the old equipment and deploying patches for system bugs identified since deployment. From that point forward, our attention will turn to new features enabled by the new platform. 

The first deliverable for new features will be a refresh of the router proxy used by the technical community, followed very quickly by a new routing security console and enhanced Cloud Connect portal. The routing security console will facilitate much more dynamic management of the address space and autonomous systems that underpin our network. This work will increase our compliance with Mutually Agreed Norms for Routing Security (MANRS) and other security principles. The Cloud Connect portal will update the user interface for the popular Cloud Connect capabilities and reposition us to more quickly deliver new features from the cloud providers. 

Each of these projects will leverage a tighter adoption of InCommon for role-based security that will also enhance the usability of the network services. These new and enhanced offerings begin the continuous evolution of software-enabled features that we expect in the coming years and should all be available by the end of 2022.

We also expect Internet2 connectors to leverage the expanded bandwidth that the platform agreements offer. Five connectors have already agreed to be the “First to 400G” and will be upgrading their connections to 400 Gbps in early 2022. Other connectors are beginning to think about resiliency and capacity strategies to take advantage of their new connections. We expect these enhancements will enable the explosive traffic growth we saw in 2021 to continue in 2022, and we anticipate the 300 petabytes per month of traffic we are currently seeing during the academic year will continue to grow by 20-40% per year.

There is also a great deal of work in the community that we’ll be able to return our attention to now that the NGI build-out is done. We’re tracking several important telemetry, security, and global network projects that we expect to more fully engage in as the team returns from NGI deployments. Our software, security, and network engineering teams are eager to return to more regular engagements in these projects!

Final Thoughts for 2021

This has been an incredibly productive year for Internet2’s network services team despite the uncertainty of the pandemic and all the pressures it brought to each of our households. I have never been more aware of or pleased with the amazing team of collaborators with whom I have the pleasure to work each day. 

This was an incredibly complex, sustained, and demanding set of activities, and the team absolutely knocked it out of the park. Across Internet2, GlobalNOC, vendor, and community teams, there has been good cheer and resilience throughout the NGI project. As a result, we have never been in a better place as a community to build new collaborations and find new ways to support this amazing community. I am grateful and amazed by the teamwork. Thank you all, and happy new year.

ICYMI

• Automation and Orchestration No Longer a ‘Nice to Have’ for Internet2’s Next Generation Infrastructure
• Internet2: A New Network for a 25th Anniversary