# Hierarchical Rate Limits Overview

In another type of rate limiting, rate-limit hierarchies enable lower priority traffic to access unused bandwidth allocated for real-time traffic, such as voice or video, during times when no real-time traffic is flowing. IP subscribers receive multiple services, such as Web, video, and file transfer, that have a maximum bandwidth. A rate-limit hierarchy can apply a common rate limit to several classified flows, enabling them to share bandwidth according to the preferences set in the hierarchical rate limits.

You can also use rate-limit hierarchies in a layer 2 (ATM) access network for DSL where many routing gateways lead into one Broadband Access Server. The Broadband Access Server uses rate-limit hierarchies to allocate shareable bandwidth to each routing gateway, which enables unused bandwidth from one routing gateway to be used by others. The hierarchy in the rate limit represents the hierarchy in the access network.

Rate-limit hierarchies enable you to share unused bandwidth dynamically, taking unused preferred bandwidth. They also enable real-time traffic to use all guaranteed bandwidth at any time without violating the configured limit on the total interface bandwidth. While preferred traffic fluctuates, the interface rate limit adjusts, dropping non-preferred packets to keep the total flow through the interface under a configured maximum rate, because preferred packets cannot be dropped by the shared rate limits, only by their individual rate limits.

Shared rate limits in the hierarchy keep the combined traffic below a configured maximum without dropping preferred packets. Preferred packets always reduce tokens on these rate limits, making their token counts negative, if necessary. Later non-preferred packets are then dropped in greater volume, bringing the total traffic through the shared rate limit below its configured maximum.

Every packet passing through a rate limit hierarchy has an owner, which is the last rate limit that can modify the packet; for example, by changing its color or dropping it. Preferred packets are owned by their individual preferred rate limits, which do not transfer ownership of the packet while the packet traverses the hierarchy. Ownership of non-preferred packets is transferred while they move from one rate-limit to the next in the hierarchy, so shared rate limits can change the packet color or drop them.

## Hierarchical Classifier Groups

Rate-limit hierarchies can be intra-interface, where different flows from classifier groups are in one policy attachment on an interface. Each time the policy is attached to another interface the rate-limit hierarchy is replicated, with no rate limits shared between attachments. Hierarchical rate-limits are only applied at forwarding interfaces because they provide the most accurate classification of packets.

You can configure rate-limit hierarchies by defining a hierarchy of policy classifier and parent groups, each with a rate limit. This hierarchy applies to the packet flow on one interface attachment for the policy. Each policy attachment creates its own copy of the rate-limit hierarchy. There are no shared rate limits across interface attachments.

A policy-based rate-limit hierarchy consists of classifier groups with an aggregate node policy object. Aggregate nodes create the interior nodes of a policy-based hierarchy; they are not classifier groups and the only policy rule applicable to them is the rate limit rule. Every classifier group or aggregate node can select another aggregate node as its parent. The policy manager ensures that these choices always result in a hierarchy. Not every classifier group with a parent aggregate node must have a rate limit rule; multiple classifier groups can share a common parent group, which may have a rate limit rule.

A policy imposes a limit of three parent groups that can be traversed from any classifier group. However, the total number of parent groups in one policy can be up to 512, but every packet must pass through no more than three parent groups at any point.

In a hierarchy of rate limits, a rate limit can be color-blind or color-aware; color-blind rate limits run the same algorithm for all packets, regardless of their color. Color-aware rate limits can change the algorithm used, depending on the color of the incoming packet (possibly set in the previous rate limit or an earlier policy, such as a VLAN policy on ingress or an IP policy). The color mark profile action changes the ToS field for the packet, depending on packet type (EXP for MPLS, DSCP or ToS for IPv4), and transmits the packet. If the mark action uses a color-mark profile, the ToS values marked can depend on the color of the packet.

## Hierarchical Rate-Limit Profiles

Hierarchical rate-limit profiles are independent from interface types. You can apply the green, yellow, or red mark values to the rate-limit profile for every type of forwarding interface that accepts ToS marking for packets. The same rate limit can be reused for a different interface type. Hierarchical rate limits have two-rate or TCP-friendly rate types.

The value applied to the ToS field is configured in the CLACL group for green, yellow, or red packets but the coloring of the packet as green, yellow, or red depends on the entire rate-limit hierarchy.

- Preferred packets are transmitted unconditionally. Rate limits that process packets transmitted unconditionally always decrement their token count, if necessary, making it negative.
- Red packets cannot be transmitted unconditionally, to avoid cases where an aggregate rate limit is oversubscribed with transmit-unconditional rates.
- Color-aware uses the incoming packet color in its algorithm
- Not promoting packets means that if the packet enters the rate limit as yellow and the rate-limit then determines that it is green, the packet remains yellow. If the rate limit determines it is red, then the packet is colored red.

A rate-limit rule is an instance of a rate-limit profile. The same profile can be used to create many rate-limit rules in the same hierarchy or in different rate-limit hierarchies. The classifier group that defines the flow can use a mark rule with color-mark profile to set the packet ToS field based on the packet color. A rate-limit hierarchy invoked from the classifier group is one way of changing the packet color; the rate-limit hierarchy is invoked before the classifier group runs the mark rule to set the packet ToS.

## Hierarchical Rate-Limit Actions

Every packet traversing a rate-limit hierarchy has an owner that is defined by the last rate limit that can apply its actions to the packet; this is a configuration option.

A rate limit in the hierarchy that does not own the packet only decrements its tokens, but cannot perform any of the following actions:

- Transfer ownership of the packet to the next rate limit.
- Retain ownership of the packet but consume tokens from the remaining rate limits in the hierarchy.
- Exit the rate-limit hierarchy, making that rate limit the final one for the packet.

These actions become the same action if the hierarchy has only one rate limit. Combining these actions with the additional choices to transmit or drop packets results in the following possible actions:

- Drop—Drops the packet at that rate limit in the hierarchy. The packet does not change the state of any rate limit further down the hierarchy.
- Transmit final—Sets the packet color and ends the packet's traversal of the rate-limit hierarchy at the current rate limit. The packet is forwarded and the rate limits further down the hierarchy are not affected. Because transmit final is based on the result of the rate limit, transmit is not an attribute of the node in the rate-limit hierarchy. Committed packets can exit the hierarchy while conformed and exceeded packets continue to the next rate limit.
- Transmit conditional—Sets the packet color to the result calculated by the rate limit and forwards the packet to the next rate limit for processing, also transferring ownership of the packet to the next rate limit. The next rate limit can then set the packet color according to the state of its token buckets and apply its actions to the packet. The transmit conditional option is the same as connecting the two rate limits in series.
- Transmit unconditional—Sets the packet color to the result calculated by the rate limit, retains ownership of the packet, and forwards the packet to the next rate limit. Later rate limits only decrement their current token counts by the packet length but do not otherwise affect the packet, either by changing its color or applying their actions to it. Although the packet is not affected, the remaining rate limits change because the token counts are reduced, making them more likely to make other packets conformed or exceeded. Transmit unconditional is not allowed as an exceeded action.

  After the transmit-unconditional completes, the packet traverses to the end of the hierarchy. Because ownership of the packet has been retained, no rate limit further down can apply its actions to it. Some of the later rate limits might already have very low token counts, which must still be decremented when processing a transmit-unconditional packet (if necessary, by making the token count negative). Negative token counts enable the remaining rate limits to restrict the total traffic through them to their peak rate (over a large enough averaging interval, which is a function of rates and burst sizes only). Transmit unconditional packets traversing the rate-limit hierarchy reduce the number of tokens available for other packets.

A rate limit has one of the four preceding actions configured for each possible result: committed, conformed, and exceeded. (Transmit unconditional is not allowed as an exceeded action.) The action taken depends only on the result of that rate limit, its rates, burst sizes, and current token state. In addition, the rate limit assigns a color to the packet, depending on both the result of the rate limit and the packet's incoming color. The final color after a packet has finished traversing a rate-limit hierarchy is a function of all the rate limits that owned the packet.

Policy actions are processed in the following order:

1. log
2. filter
3. traffic class
4. user packet class
5. next hop
6. rate limit
7. color status
8. color action
9. parent group
10. mark

The mark action is the last action that occurs, after parent-group, so that the color-mark profile can mark the packet with the final color from the hierarchy.

| Note: To avoid saturation when using dual token buckets, the total amount of yellow transmit unconditional traffic should be less than the peak rate minus the committed rate; the green transmit unconditional traffic should be less than the committed rate.

## Example: Multiple Flows Sharing Preferred Bandwidth Rate-Limiting Hierarchical Policy

Figure 2 shows an interface with an attached policy that has a Video classifier that singles out a substream of the packets flowing on that interface. The Video classifier can be allocated 6 Mbps out of the 10 Mbps interface rate. All other packets on the interface are Internet. The common rate limit cannot drop Video packets, but must limit the total flow (Video and Internet) to under 10 Mbps. Internet traffic can use the Video bandwidth when there are no active Video calls, while avoiding hard partitioning of interface bandwidth.

| Note: To avoid rate-limit saturation, we recommend that you set the rate limit profile to color-aware when the rate limit is set to receive transmit conditional.

Figure 2: Multiple Flows Sharing Preferred Bandwidth

host1(config)#rate-limit-profile video two-rate hierarchical host1(config-rate-limit-profile)#committed-action transmit unconditional host1(config-rate-limit-profile)#conformed-action transmit unconditional host1(config-rate-limit-profile)#exceeded-action drop host1(config-rate-limit-profile)#peak-rate 60000000 host1(config-rate-limit-profile)#exit host1(config)#rate-limit-profile common two-rate hierarchical host1(config-rate-limit-profile)# color-aware host1(config-rate-limit-profile)#committed-action transmit conditional host1(config-rate-limit-profile)#conformed-action transmit conditional host1(config-rate-limit-profile)#exceeded-action drop host1(config-rate-limit-profile)#peak-rate 100000000 host1(config-rate-limit-profile)#exit host1(config)#policy-list mycompany host1(config-policy-list)#classifier-group video parent-group all host1(config-policy-list-classifier-group)#rate-limit-profile video host1(config-policy-list-classifier-group)#exit host1(config-policy-list)#classifier-group * parent-group all host1(config-policy-list-classifier-group)#forward host1(config-policy-list-classifier-group)#exit host1(config-policy-list)#parent-group all host1(config-policy-list-parent-group)#rate-limit-profile common host1(config-policy-list-parent-group)#exit

In this example, the rate limit Common is color-aware, using the color of the incoming packets instead of setting them to Green. This causes the rate limit Preferred to send 6 Mbps of yellow, transmit unconditional packets. The rate limit Common counts the packets against the yellow token bucket, which has a rate of 10 Mbps. However, if the rate limit Common is color-blind, it treats all packets as Green so the green token bucket gets 6 Mbps of transmit unconditional traffic, which eventually causes all packets to be saturated and dropped.

## Example: Multiple Flows Sharing a Rate Limit Hierarchical Policy

Figure 3 shows an interface that has one rate limit and three classified flows, A, B, and C. The combined traffic for A, B, and C must be below a peak rate of 4 Mbps, but each individual flow can burst up to that amount. Statistics can be collected separately on A, B, and C, while limiting only the aggregate of all three. None of the flows has any preference in accessing the rate limit, and the rate limit is shared on a first-come first-serve basis.

Figure 3: Multiple Packet Flows Sharing a Rate Limit

This example uses committed and conformed actions for a preferred rate limit profile so that the common rate limit drops only exceeded packets (those packets that raise the traffic load above 4 Mbps); packets below 4 Mbps are transmitted. By specifying classifier-group * parent-group all, all packets are sent to the parent group. There is no individual rate limit so that those packet use any available, unused bandwidth in the parent group rate limit.

host1(config)#rate-limit-profile All two-rate hierarchical host1(config-rate-limit-profile)#committed-action transmit conditional host1(config-rate-limit-profile)#conformed-action transmit conditional host1(config-rate-limit-profile)#exceeded-action drop host1(config-rate-limit-profile)#peak-rate 40000000 host1(config-rate-limit-profile)#exit host1(config)#policy-list rlpshare host1(config-policy-list)#classifier-group A parent-group All host1(config-policy-list-classifier-group)#forward host1(config-policy-list-classifier-group)#exit host1(config-policy-list)#classifier-group B parent-group All host1(config-policy-list-classifier-group)#forward host1(config-policy-list-classifier-group)#exit host1(config-policy-list)#classifier-group C parent-group All host1(config-policy-list-classifier-group)#forward host1(config-policy-list-classifier-group)#exit host1(config-policy-list)#parent-group All host1(config-policy-list-parent-group)#rate-limit-profile All host1(config-policy-list-parent-group)#exit

## Example: Shared Pool of Additional Bandwidth with Select Flows Rate-Limiting Hierarchical Policy

Figure 4 shows three classified flows, A, B, and C, each of which has an individual rate limit with a peak rate of 1 Mbps. If flow A is exceeding its peak rate, rather than drop the packet, the flow tries to use any bandwidth left in a shared rate limit (extrabw) of peak rate of 2 Mbps. The packet is dropped only if both the individual and the shared rate limit have no bandwidth left.

The total flow is limited to 5 Mbps, which is the sum of all the individual peak rates plus the peak rate of the shared rate limit. Individual flows A, B, and C are limited to a maximum of 3 Mbps (1 Mbps from its individual rate limit and up to 2 Mbps if it can consume the entire shared pool); however, it cannot go below a 1 Mbps rate because of the other flows. A shared rate limit enables many flows to share the extra bandwidth dynamically.

Figure 4: Shared Pool of Additional Bandwidth with Select Flows

This example uses transmit final so that those packets do not pass through the common rate limit. Transmit final also indicates that there is no shared maximum. If the packets are committed or conformed, they do not need to borrow extra bandwidth or subtract tokens from it. The example uses exceeded action transmit conditional so that packets above the individual rate-limit maximum are not dropped but sent to the next rate limit in the hierarchy. Because this is transmit conditional, ownership of the packet also transfers so the common rate limit can drop these packets if it has no bandwidth left.

host1(config)#rate-limit-profile indiv two-rate hierarchical host1(config-rate-limit-profile)#committed-action transmit final host1(config-rate-limit-profile)#conformed-action transmit final host1(config-rate-limit-profile)#exceeded-action transmit conditional host1(config-rate-limit-profile)#peak-rate 10000000 host1(config-rate-limit-profile)#exit host1(config)#rate-limit-profile extrabw two-rate hierarchical host1(config-rate-limit-profile)#committed-action transmit conditional host1(config-rate-limit-profile)#conformed-action transmit conditional host1(config-rate-limit-profile)#exceeded-action drop host1(config-rate-limit-profile)#peak-rate 20000000 host1(config-rate-limit-profile)#exit host1(config)#policy-list mypolicy host1(config-policy-list)#classifier-group A parent-group extrabw host1(config-policy-list-classifier-group)#rate-limit-profile indiv host1(config-policy-list-classifier-group)#exit host1(config-policy-list)#classifier-group B parent-group extrabw host1(config-policy-list-classifier-group)#rate-limit-profile indiv host1(config-policy-list-classifier-group)#exit host1(config-policy-list)#classifier-group C parent-group extrabw host1(config-policy-list-classifier-group)#rate-limit-profile indiv host1(config-policy-list-classifier-group)#exit host1(config-policy-list)#parent-group extrabw host1(config-policy-list-parent-group)#rate-limit-profile extrabw host1(config-policy-list-parent-group)#exit

## Example: Aggregate Marking with Oversubscription Rate-Limiting Hierarchical Policy

Figure 5 shows an aggregate rate limit that enables up to 2 Mbps of traffic to be sent with ToS marking TOS1. Traffic above that rate is sent with marking TOS2 or TOS3 (depending on packet type) and traffic above 6 Mbps is dropped. The 2 Mbps of TOS1 is oversubscribed among individual flows A, B, and C, each of which can have up to 1 Mbps of TOS1 traffic. An individual flow can mark a packet TOS1, but if there is insufficient bandwidth at the shared rate limit because of oversubscription, the packet is demoted and remarked.

The demoted packets from flow A are marked as TOS2 but the demoted packets from flows B and C are marked as TOS3. The shared rate limit determines whether to demote the packet, in which case each individual rate limit selects the new ToS marking. Individual flows are not required to mark demoted packets with the same value.

The committed and conformed actions are transmit conditional so that all packets also go through rate limit S, because rate limit S imposes the limit of 2 Mbps of TOS1 traffic (total across A, B, and C).

Committed packets are transmitted conditionally to rate limit S, which has a peak rate of 6 Mbps and a committed rate of 2 Mbps; these packets can be demoted by S to Y (yellow), in which case they are remarked TOS2 or TOS3. If S leaves them as G (green), they are marked as TOS1. All conformed packets from A, B, and C are also transmitted conditionally to S but arrive as Y because rate limits do not promote packets in color. S is color-aware so these Y packets do not take away G tokens, leaving them reserved only for the G packets coming from A, B, and C.

Figure 5: Aggregate Marking with Oversubscription

host1(config)#rate-limit-profile indiv two-rate hierarchical host1(config-rate-limit-profile)#committed-action transmit conditional host1(config-rate-limit-profile)#conformed-action transmit conditional host1(config-rate-limit-profile)#exceeded-action drop host1(config-rate-limit-profile)#committed-rate 10000000 host1(config-rate-limit-profile)#peak-rate 20000000 host1(config-rate-limit-profile)#exit host1(config)#rate-limit-profile S two-rate hierarchical host1(config-rate-limit-profile)#committed-action transmit conditional host1(config-rate-limit-profile)#conformed-action transmit conditional host1(config-rate-limit-profile)#exceeded-action drop host1(config-rate-limit-profile)#committed-rate 20000000 host1(config-rate-limit-profile)#peak-rate 60000000 host1(config-rate-limit-profile)#color-aware host1(config-rate-limit-profile)#exit host1(config)#ip color-mark-profile A host1(config-color-mark-profile)#green-mark TOS1 host1(config-color-mark-profile)#yellow-mark TOS2 host1(config-color-mark-profile)#red-mark TOS2 host1(config-color-mark-profile)#exit host1(config)#ip color-mark-profile BC host1(config-color-mark-profile)# green-mark TOS1 host1(config-color-mark-profile)# yellow-mark TOS3 host1(config-color-mark-profile)# red-mark TOS3 host1(config-color-mark-profile)# exit host1(config)#policy-list TOS1_oversubsribed host1(config-policy-list)#classifier-group A parent-group S host1(config-policy-list-classifier-group)#rate-limit-profile indiv host1(config-policy-list-classifier-group)#mark profile A host1(config-classifier-group)#exit host1(config-policy-list)#classifier-group B parent-group S host1(config-policy-list-classifier-group)#rate-limit-profile indiv host1(config-policy-list-classifier-group)#mark profile BC host1(config-classifier-group)#exit host1(config-policy-list)#classifier-group C parent-group S host1(config-policy-list-classifier-group)#rate-limit-profile indiv host1(config-policy-list-classifier-group)#mark profile BC host1(config-policy-list-classifier-group)#exit host1(config-policy-list)#parent-group S host1(config-policy-list-parent-group)#rate-limit-profile S host1(config-policy-list-parent-group)#exit

## Color-Aware Configuration for Rate-Limiting Hierarchical Policy

Common to many rate-limit hierarchies is a large aggregate rate limit that receives packets from many smaller individual rate limits. An individual rate limit can mark a packet yellow but, if few individual flows are active, the aggregate rate limit is likely to try to promote it to green, overriding the individual rate limit. For this reason, rate limits never promote packets in color; color-aware rate limits use the incoming color in their algorithm, but the final result is always equal to or less than the initial packet color.

Rate-limit profiles for rate-limit hierarchies include a non-default configuration option for color-aware. For two-rate rate limits this option enables the color-aware algorithm. If hierarchical, TCP-friendly one-rate rate limits have a color-aware algorithm defined.

In the following color-aware example, the non-preferred packets do not take any green tokens from rate-limit A, leaving them all for preferred packets. Preferred packets may take green and also take yellow tokens (which reduces the flow of non-preferred). In this way the non-preferred packets do not reduce the number of green preferred packets, only the number of yellow preferred packets; preferred packets are then marked from a color-mark profile.

class non-preferred parent A color yellow class preferred parent A mark profile cm parent A rate-limit A !! a color-aware rate limit

The color-mark profile translates the packet color, which is independent of its type, to a type-dependent mark for ToS or EXP and applies it to a packet after it has exited the rate-limit hierarchy. If no translation is configured for a color, then packets of that color are not changed.

Transmit-unconditional packets entering a color-aware rate limit uses the color on the packet for the rate-limit algorithm. Doing this ensures that the color-aware rate limit depletes tokens from the token buckets to account for these packets.

Every packet sent through a rate-limit hierarchy is either dropped inside the hierarchy or emerges with a green, yellow, or red color assigned to it by the rate-limit hierarchy. The color depends on the last rate limit in the hierarchy that owned the packet and all prior rate limits. The green, yellow, or red classification applies to packets of any type and is not interface-type dependent.

A packet that has traversed the hierarchy either has been dropped or emerges with a color (green, yellow or red). This final color can be used by a mark rule with a color-mark profile to select the ToS marking for the packet. Because this operation is interface-type dependent, the actual value is configured where the packet entered the hierarchy; however, the color is set by the entire rate-limit hierarchy.

We recommend that all rate-limit profiles that receive transmit unconditional packets should be color-aware. If not color-aware, yellow transmit unconditional packets are processed through both the green and yellow token buckets; if the green rate is low, this causes an oversubscription of transmit unconditional packets and leads to saturation. By making the rate limit color-aware, the yellow transmit unconditional packets ar counted only against the yellow token bucket.

#### Related Topics

- color
- color-aware
- color-mark-profile
- green-mark
- red-mark
- yellow-mark