By Igor Sfiligoi, Lead Scientific Software Developer and Researcher at UCSD-SDSC
Estimated reading time: 3 minutes
Register for Characterizing Networking In and Out of the Clouds event to be held Thursday, September 2, at 1 p.m. ET.
Igor Sifilgoi presents on the IceCube project at the Internet2 booth at SC19.
Commercial cloud providers continually make news showcasing their compute capabilities. Very few compute problems, however, happen in isolation.
Getting input data to the compute nodes and delivering the computed results to permanent storage is often equally important. Without enough network throughput, the provisioned compute capacity can get wasted while waiting for data to be moved.
When the IceCube Neutrino Observatory decided to consume a large amount of cloud compute capacity, we could not find any reliable information on the available network capabilities of the various cloud providers—so my colleagues and I did the measurements ourselves.
The measured data showed that there is plenty of network bandwidth inside the provider-operated networks and that connectivity to on-prem research networks is plentiful, too. IceCube was thus able to make good use of cloud compute resources, with minimal networking-related constraints.
One of the cloud runs was however especially data intensive, and we were worried about the cost of the network transfers. Unlike on-prem research networks, some cloud networking is a billable commodity.
We engaged with the Internet2 network team to provision on-demand dedicated links using the Internet2 Cloud Connect service. They are billed by the cloud providers at a substantial discount on data egress charges, although they do come with additional configuration requirements.
We provisioned significantly more bandwidth at the same time with multiple providers. This was the first time a research team had attempted to accomplish this, so the setup and operation of those links turned out to be trickier than expected. Yet the resulting cloud run was a success and delivered on the price reduction promises.
Join Me for the Talk
I hope you will join me on Thursday, September 2, at 1 p.m. ET as I share lessons learned, and provide both an overview of the experimental setups and the measured results. The slides and recording of the talk will be posted soon as part of Internet2’s TechEXtra21 series.
TechExtra21 will offer a wide selection of opportunities for the Internet2 and InCommon technical community to convene virtually through small bites of the Technology Exchange experience that is dubbed “TechEXtras.”
Over the last 18 months, the research and education community has addressed constant and changing conditions – the shift to remote teaching and learning, working from home, planning for an in-person return to the classrooms, securing and improving infrastructure, workforce development and upskilling, and managing ever-changing safety guidelines – just to name a few.
There are myriad stories to tell and lessons to share. Although our ability to meet in person is limited this fall, we will continue to meet virtually to share, teach, and learn from each other through this series.
During this year’s TechEXtra21 series, Internet2 invites community partners to tell their stories. Internet2 is seeking community proposals to round out the fall programming!