Extensibility

Segment Routing Over IP/UDP

MPLS separates the underlay (transport) from the overlay (service) layers. Labels are commonly used to represent forwarding instructions for both layers. That said, the transport layer has always been capable of using non-MPLS tunneling technologies such as GRE, IPsec, or MPLS-over-IP/UDP tunnels, even as the service layer retains labels. SR over IP/UDP (shorted to SRoUDP) takes advantage of this.

A growing number of brownfield deployments are investigating tunneling a stack of SIDs over IP-only islands. SIDs remain relevant to SR-MPLS capable routers. Non-MPLS routers are oblivious to the SID stack and only need to perform IP forwarding. The IP islands – IPv4 or IPv6 – can remain as they are in perpetuity, allowing coexistence of SR-MPLS and IP-only networks.

SRoUDP works by segregating the network into SR-MPLS capable and incapable nodes. This is determined by the advertisement of, or lack thereof, a node SID. When an SR-MPLS capable router’s next hop is IP-only, it encapsulates the SID stack into an IP/UDP packet. The IP packet is tunneled either to the next SR-MPLS capable router or the final destination based on the SID stack. Currently, Junos implements the latter.

Let’s make the P routers in Newark and Philadelphia into IPv4-only routers. They should no longer advertise node SIDs, nor have LDP, or RSVP-TE enabled. This leaves the SR-MPLS capable P routers in New York and Washington connected over an IP expanse.

The New York PE routers will still impose one or more MPLS labels to reach the PEs in Washington. The local New York P routers that receive long-distance labeled traffic recognize that the destination is now reachable over IP next hops. They build a dynamic IP/UDP tunnel to the remote PE and encapsulate MPLS traffic within that. The Washington PE decapsulates the tunnel and forwards the payload.

This requires decapsulation configuration at the PE routers, and encapsulation configuration at the SR-MPLS P routers. The IP-only P routers require no reconfiguration.

Encapsulation configuration: New York and Washington P routers

Decapsulation configuration: New York and Washington PE routers

The P routers are configured to create dynamic IP/UDP tunnels to any destination in the range that encompasses the addresses assigned to the routers’ lo0. The udp-tunneling encapsulation knob allows IS-IS to include IP next hops during its route calculation. The PE routers are configured with a firewall filter that matches and decapsulates traffic on the well-known MPLS-over-UDP destination port. This filter is applied to all core-facing interfaces as the IP/UDP tunneled traffic could arrive over any of them.

After committing this change, something curious happens to the P routers’ inet.3 table. Where before it only contained L-ISIS routes to area-local SR-MPLS routers, it additionally contains routes to remote P/PE routers. Let’s take a look.

Control plane: p1.nyc adds routes to P/PE routers in D.C.

Congruently, the mpls.0 table has an interesting label forwarding action associated with the remote routers’ node SIDs. Instead of the typical push, next, and continue actions, you see the label being swapped for explicit-null and pushed into a dynamic IP/UDP tunnel. The explicit-null ensures that when pe1.iad receives the packet it performs an MPLS lookup on the inner packet.

Control plane: p1.nyc swaps inbound label 1010 (pe1.iad) with label 0 before tunneling

Control plane: p1.nyc creates a dynamic tunnel to each SR-MPLS router in Washington

The traceroute between the CEs is most transformed:

Gone are the previously visible hops. Remember that the current behavior has p1.nyc establish a tunnel to the ultimate next hop; an alternative would be to establish a tunnel to the next closest SR-MPLS-capable router. With either approach, not all the transit routers are required to have an MPLS data plane. For many environments, that is a matter of fact. SRoUDP allows a SID stack to be transported over these IP islands.

FlexAlgo

Those who have dealt with multitopology routing (MTR) may shudder from its complexity. While not free of blemishes, MTR is a generally useful concept. The path less traveled may be favored over the shortest. Many operators maximize the IGP metric and optimize routing using the IGP traffic engineering metric. The TE metric may represent latency, which is arguably the metric that most affects typical end user experience.

MTR did not enjoy widespread acceptance. Practical implementations limited the number of topologies to an IPv4 and IPv6 topology, with unicast and multicast variants. More than anything, this failure in imagination is what led to its marginalization.

As the name implies, FlexAlgo aims to hand operators control over topology definition. All nodes and links participate in the default topology. Additional subset topologies are calculated by defining constraints and optimization objectives. To take a prototypical example, a red topology could include all links that are colored red, a familiar concept carried over from RSVP-TE; a supplementary blue topology could only include blue links. Finally, a purple topology could include those links that are both red and blue.

The first step is to define what constitutes a topology. With MTR, the operator would have to visit each router and configure each link’s topological participation. This was an error-prone process, scaling poorly with growing networks. Discovering, validating, and enforcing membership in a topology required significant homegrown tooling.

FlexAlgo formalizes topology definition by advertising a Flexible Algorithm Definition (FAD) in the IGP itself. This attribute encodes which metric to calculate, which algorithm to use for the calculation, and most interestingly, link inclusion and exclusion based on extended administrative groups (EAG). Each FAD is numbered from 128-255; for operational convenience, it may be referred to nominally (‘red’ FAD, ‘blue’ FAD) but is ultimately encoded as a FAD number.

A FAD only need be configured on a subset of routers in the network. The remaining routers learn the FADs via the IGP. Each FAD also has an associated priority, with the highest numerical value being selected. The FADs must be identically configured across the subset of routers. In case of a conflict – one router advertising a ‘red’ FAD that includes links with red EAGs, and another advertising a ‘red’ FAD that accidentally includes links with blue EAGs – the highest priority FAD wins. As a tie breaker, the IGP system ID or router ID is used.

Once topology definitions have been learned, either via local configuration or IGP advertisement, routers can be configured to participate in those topologies. Again, this is not accomplished by configuring individual links, a la MTR. The FAD acts as a sieve for the default topology. The subset topologies are a result of each router pruning ineligible links, then optimizing for the metric type.

FlexAlgo can serve as an lightweight form of label compression. Rather than lengthy explicit paths, per-FAD node SIDs can provide loose-source routing with shallow label stacks.

An example will clarify. To keep things simple, let’s configure an additional topology on pe1.nyc and p1.nyc. This new topology will optimize for the TE, instead of the IGP, metric. For now, it will not prune any links from the default topology. The resulting configuration numbers the FAD, specifies its optimization objective, and assigns it a priority:

This simply results in a new topology being defined without any routers participating in it yet. As a result, the FAD isn’t even flooded by the IGP. The next step should be to have all New York routers participate in the topology – note that pe2.nyc and p2.nyc have learned about the FAD from IS-IS, not configuration:

With two routers participating in the topology they are describing, multiple sub-TLVs now augment pe1.nyc and p1.nyc’s IS-IS LSPs.

Control plane: Viewing pe1.nyc’s IS-IS LSP on pe2.nyc

With two routers participating in the topology they are describing, multiple sub-TLVs now augment pe1.nyc and p1.nyc’s IS-IS LSPs. In addition to a new router capability that advertises the FAD itself, there is a new per-topology prefix SID being advertised.

This brings up another advantage of FlexAlgo when used to create dual or even multi-planar topologies. Often, for the same service route, BGP’s protocol next hop (PNH) is modified via policy so it is different for each plane. RSVP-TE LSPs are spun to each PNH. The LSPs are routed differently, accomplishing dual-plane behavior. Other than the soft-state that RSVP-TE imposes, this additionally requires multiple IPv4 or IPv6 addresses to be allocated to each PE’s loopback. Each address represents reachability to that PE, via a specific plane.

In contrast, FlexAlgo can assign a node SID per-topology. Each node SID remains associated with the same loopback IPv4 or IPv6 address. Service route PNH resolution can then be contained within different routing tables, each corresponding to a different topology. In this way, a single IP address can remain the PNH. Services routes can use different node SIDs to resolve the next hop, based on a combination of extended BGP color communities and their association with a FlexAlgo topology.

To demonstrate a simple dual-plane topology, let’s have pe1.nyc prefer p1.nyc to reach pe2.nyc; in turn, pe2.nyc will prefer p2.nyc to reach pe1.nyc. Currently the PEs are reachable over equal-cost routes via the P routers. The earlier config snippet already set this up by using unequal TE metrics, which the new topology optimizes for. Go ahead and add per-topology node SID configuration to pe2.nyc and p2.nyc.

Control plane: pe1.nyc’s new topology is a subset of the default topology

The new topology has a corresponding new routing information base (RIB). In that new topology, there are fewer available paths between pe1.nyc and pe2.nyc, and the metric between them is also different. Note the additional prefix SID (1000 + 16) advertised by pe2.nyc, associated with the same loopback address.

We began, and conclude, our journey with a simple example. Segment routing has reignited conversations about what it means to build flexible, efficient, and fungible networks.

Leap in.