R&E networks at the state, regional, national, and global levels are architected to ensure the high-performance paths needed to support researchers in moving and accessing data securely and as quickly as possible. But when a campus employs a scrubbing service to protect against DDoS attacks, some or all of the traffic to the campus is diverted to the DDoS vendor’s scrubbing facilities via the commercial internet rather than utilizing our R&E networks. While effective at removing attack traffic, these scrubbing services aren’t architected to provide the high-performance network services that data-intensive researchers need.
The challenge then becomes how to best balance DDoS protection while retaining support for researchers.
Always-On vs. Dynamic Scrubbing Services
Always-on scrubbing services present a more complex challenge, particularly when they encompass the entire campus’s network traffic. By design, these services offer the advantages of providing continuous protection and allowing the scrubbing service to learn and adapt to normal traffic. However, the major disadvantage is that network performance is always adversely affected – even when no DDoS attack is taking place.
Dynamic scrubbing services, on the other hand, only engage during a DDoS attack. This tends to minimize the impact on researchers. When an attack isn’t happening, traffic can take the best-performing path, rather than having to be directed to the cloud-based scrubbing provider. These services can also be configured to only divert subnets targeted by an attack, not the entire campus network. It’s uncommon for DDoS attacks to specifically target research infrastructure. If the research subnets are separate from the day-to-day services (web servers, DNS, etc.), there’s a good chance that diverted traffic may have no effect on researchers.
Several practices can mitigate the effects DDoS scrubbing services may have on researchers:
- At least so far, most scrubbing services focus on IPv4 traffic. A campus can opt to have its research infrastructure accessible via IPv6 and ensure services have AAAA DNS records so the IPv6 address is preferred, which allows the research traffic to bypass the scrubbing facility.
- The worldwide R&E network infrastructure represents roughly 2% of global routes. These networks also engage in a high level of collaboration, including promoting practices that lessen the risk that these networks contribute to DDoS attacks. Given the lower risk for DDoS coming from this sector, a campus can ensure its IP routes announced towards R&E (e.g., the Internet2 network) are as specific as those announced by the scrubbing provider.
- Where appropriate, high-energy physics projects participating in the LHCONE overlay network should ensure those resources receive their traffic directly and bypass the scrubbing facilities.
- Additionally, Internet2 is actively pursuing relationships with scrubbing provider networks through Internet2 Peer Exchange (I2PX) so that, where possible, clean traffic returning from the scrubbing facility can take advantage of a campus’s I2PX connectivity.
Defending against DDoS attacks while supporting researchers is a specific example of our general challenge to ensure the best network performance for mission support across the R&E community. Many of our networks are multihomed. Without applying the appropriate routing policy, it’s possible that research traffic may use commercial internet service providers, rather than the Internet2 network.
Multihomed networks use BGP routing policy to control and influence how traffic enters and leaves the network. The default behavior of routers without an explicit configuration, including the standard BGP route selection process, is a routing policy. This section will introduce a common routing policy that networks connected to Internet2 use, an explanation of why this is a good starting point, and some things to consider.
First, some routing basics.
Inbound route policy refers to (manipulating) characteristics of routing updates you learn from neighbors. Often this involves modifying the default BGP Local Preference (LOCAL_PREF). This inbound route policy affects outbound traffic. Conversely, outbound routing policy – how your network is advertised to the outside world – affects inbound traffic. Generally, we have a lot of control over outbound traffic because we essentially have total control over the inbound route policy. Controlling inbound traffic is more of a suggestion using a limited set of tools (such as BGP communities and AS Path prepending). You can make adjustments to these parameters, but it is up to the other networks to honor them.
Routers make the longest match decisions to determine the next-hop/outgoing interface used to get a packet closer to its destination. The longest match wins, no matter what other route characteristics are present in less specific matches because a more specific route is a different route than its parent. For example, let’s say a router has the following in its route table:
| Prefix |
Next-hop |
| 172.16.0.0/16 |
et-2/1/0 |
| 172.16.0.0/17 |
gr-2/2/0 |
A packet destined for 172.16.12.22 matches both of these prefixes, and the router will pick the longest match, also called the more-specific. In this case, that’s 172.16.0.0/17. This happens before any protocol-specific characteristics are taken into account, such as BGP Local Preference.
Many campus networks implement a route policy that prioritizes using the Internet2 network if possible, followed by settlement-free peering if present, and finally commercial transit as a last resort.
Why is this a common policy? The Internet2 network and interconnected state and regional R&E networks are high-capacity, low-latency, and no-congestion. This R&E network ecosystem also comprises friendly, like-minded operators when it comes to transparency and collaboration. Jumbo frames are supported. In almost all cases, you’ll find better performance via Internet2 than any other path. If you encounter any problems, you’ll likely find helpful people at all the networks along the path. This is a very unique environment compared with commercial providers.
How might this policy be implemented? For an inbound route policy, LOCAL_PREF can be configured as follows (remember, higher LOCAL_PREF is preferred):
| Route Source |
LOCAL_PREF |
| Internal and Customers |
400 |
| R&E Peers and Internet2 |
300 |
| Settlement-Free Peering |
200 |
| Commercial Transit |
100 (default) |
The result of this policy is that traffic destined to sites that are reachable via R&E networks such as Internet2 will follow that path, even if the AS Path is shorter via other providers. Generally, this is desirable for the reasons stated above.
As mentioned above, inbound traffic is harder to control. One of the few tools available for operators to influence inbound traffic is AS Prepending, which artificially makes an AS Path appear longer than it really is by adding your own AS more than the default one time. For example, to implement the ‘Prefer Internet2’ policy, your outbound route advertisements might look like the following:
| BGP Neighbor Type |
Number of Prepends |
| R&E Peers and Internet2 |
None |
| Settlement-Free Peering |
None |
| Commercial Transit |
Two |
Because AS Path length is near the top of the list in the BGP selection process, networks connected to Internet2 may use that path over others. Some networks may do this for all the address space under its control, others, only for selected subnets.
When a network advertises more specific routes via a DDoS scrubbing service than it does to Internet2, almost all traffic will follow the more specific route to the DDoS scrubbing service. If the scrubbing service is “always-on” an organization may not be taking full advantage of the benefits of the Internet2 network and global R&E infrastructure. This may be intentional, or it may be the result of adding DDoS scrubbing without accounting for the impact on a long-standing routing policy.
Questions and Support
We would be happy to review your campus routing policy and discuss your specific needs to help support data-intensive science. To start that conversation, please email networkdevelopment@internet2.edu.