Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Troubleshooting Layer 3 VPNs

 

Diagnosing Common Layer 3 VPN Problems

Problem

Description: To troubleshoot problems in the Layer 3 VPN configuration, start at one end of the VPN (the local customer edge [CE] router) and follow the routes to the other end of the VPN (the remote CE router).

Solution

The following troubleshooting steps should help you diagnose common problems:

  1. If you configured a routing protocol between the local provider edge (PE) and CE routers, check that the peering and adjacency are fully operational. When you do this, be sure to specify the name of the routing instance. For example, to check OSPF adjacencies, enter the show ospf neighbor instance routing-instance-name command on the PE router.

    If the peering and adjacency are not fully operational, check the routing protocol configuration on the CE router and check the routing protocol configuration for the associated VPN routing instance on the PE router.

  2. Check that the local CE and PE routers can ping each other.

    To check that the local CE router can ping the VPN interface on the local PE router, use a ping command in the following format, specifying the IP address or name of the PE router:

    To check that the local PE router can ping the CE router, use a ping command in the following format, specifying the IP address or name of the CE router, the name of the interface used for the VPN, and the source IP address (the local address) in outgoing echo request packets:

    Often, the peering or adjacency between the local CE and local PE routers must come up before a ping command is successful. To check that a link is operational in a lab setting, remove the interface from the VPN routing and forwarding (VRF) by deleting the interface statement from the [edit routing-instance routing-instance-name] hierarchy level and recommitting the configuration. Doing this removes the interface from the VPN. Then try the ping command again. If the command is successful, configure the interface back into the VPN and check the routing protocol configuration on the local CE and PE routers again.

  3. On the local PE router, check that the routes from the local CE router are in the VRF table (routing-instance-name.inet.0):

    The following example shows the routing table entries. Here, the loopback address of the CE router is 10.255.14.155/32 and the routing protocol between the PE and CE routers is BGP. The entry looks like any ordinary BGP announcement.

    If the routes from the local CE router are not present in the VRF routing table, check that the CE router is advertising routes to the PE router. If static routing is used between the CE and PE routers, make sure the proper static routes are configured.

  4. On a remote PE router, check that the routes from the local CE router are present in the bgp.l3vpn.0 routing table:
    user@host> show route table bgp.l3vpn.0 extensive

    The output of the show route table bgp.l3vpn.0 extensive command contains the following information specific to the VPN:

    • In the prefix name (the first line of the output), the route distinguisher is added to the route prefix of the local CE router. Because the route distinguisher is unique within the Internet, the concatenation of the route distinguisher and IP prefix provides unique VPN-IP version 4 (IPv4) routing entries.

    • The Route Distinguisher field lists the route distinguisher separately from the VPN-IPv4 address.

    • The label-switched-path field shows the name of the label-switched path (LSP) used to carry the VPN traffic.

    • The Push field shows both labels being carried in the VPN-IPv4 packet. The first label is the inner label, which is the VPN label that was assigned by the PE router. The second label is the outer label, which is an RSVP label.

    • The Communities field lists the target community.

    • The Secondary tables field lists other routing tables on this router into which this route has been installed.

    If routes from the local CE router are not present in the bgp.l3vpn.0 routing table on the remote PE router, do the following:

    • Check the VRF import filter on the remote PE router, which is configured in the vrf-import statement. (On the local PE router, you check the VRF export filter, which is configured with the vrf-export statement.)

    • Check that there is an operational LSP or an LDP path between the PE routers. To do this, check that the IBGP next-hop addresses are in the inet.3 table.

    • Check that the IBGP session between the PE routers is established and configured properly.

    • Check for “hidden” routes, which usually means that routes were not labeled properly. To do this, use the show route table bgp.l3vpn.0 hidden command.

    • Check that the inner label matches the inner VPN label that is assigned by the local PE router. To do this, use the show route table mpls command.

    The following example shows the output of this command on the remote PE router. Here, the inner label is 100004.

    The following example shows the output of this command on the local PE router, which shows that the inner label of 100004 matches the inner label on the remote PE router:

  5. On the remote PE router, check that the routes from the local CE router are present in the VRF table (routing-instance-name.inet.0):
    user@host> show route table routing-instance-name.inet.0 detail

    In this routing table, the route distinguisher is no longer prepended to the prefix. The last line, Primary Routing Table, lists the table from which this route was learned.

    If the routes are not present in this routing table, but were present in the bgp.l3vpn.0 routing table on the local CE router, the routes might have not passed the VRF import policy on the remote PE router.

    If a VPN-IPv4 route matches no vrf-import policy, the route does not show up in the bgp.l3vpn table at all and hence is not present in the VRF table. If this occurs, it might indicate that on the PE router, you have configured another vrf-import statement on another VPN (with a common target), and the routes show up in the bgp.l3vpn.0 table, but are imported into the wrong VPN.

  6. On the remote CE router, check that the routes from the local CE router are present in the routing table (inet.0):

    If the routes are not present, check the routing protocol configuration between the remote PE and CE routers, and make sure that peers and adjacencies (or static routes) between the PE and CE routers are correct.

  7. If you determine that routes originated from the local CE router are correct, check the routes originated from the remote CE router by repeating this procedure.

Example: Troubleshooting Layer 3 VPNs

This example shows how to use the ping command to check the accessibility of various routers in a VPN topology, and how to use the traceroute command to check the path that packets travel between the VPN routers.

Requirements

This example uses the following hardware and software components:

  • M Series routers

  • Junos OS Release 10.0R1 and later

Overview

Topology

The topology shown in Figure 1 illustrates the network used in this example to demonstrate how to employ the ping and traceroute commands to test connectivity between the routers participating in a Layer 3 VPN.

Figure 1: Layer 3 VPN Topology for ping and traceroute Examples
Layer 3 VPN Topology for ping
and traceroute Examples

Pinging the CE Router from Another CE Router

Step-by-Step Procedure

The following describes how to use the ping and traceroute commands to troubleshoot Layer 3 VPN topologies. You can ping one CE router from the other by specifying the other CE router’s loopback address as the IP address in the ping command. This ping command succeeds if the loopback addresses have been announced by the CE routers to their directly connected PE routers. The success of these ping commands also means that Router CE1 can ping any network devices beyond Router CE2, and vice versa. Figure 1 shows the topology referenced in the following steps:

  1. Ping Router CE2 (VPN5) from Router CE1 (VPN4):
    user@vpn4> ping 10.255.10.5 local 10.255.10.4 count 3
  2. To determine the path from Router CE1’s loopback interface to Router CE2’s loopback interface, use the traceroute command:
    user@vpn4> traceroute 10.255.10.5 source 10.255.10.4
  3. When you use the traceroute command to examine the path used by a Layer 3 VPN, the provider (P) routers in the service provider’s network are not displayed. As shown above, the jump from Router VPN1 to Router VPN2 is displayed as a single hop. The P router (VPN3) shown in Figure 1 is not displayed.
  4. Ping Router CE1 (VPN4) from Router CE2 (VPN5):
    user@vpn5> ping 10.255.10.4 local 10.255.10.5 count 3
  5. To determine the path from Router CE2 to Router CE1, use the traceroute command:
    user@vpn5> traceroute 10.255.10.4 source 10.255.10.5

Pinging the Remote PE and CE Routers from the Local CE Router

Step-by-Step Procedure

From the local CE router, you can ping the VPN interfaces on the remote PE and CE routers, which are point-to-point interfaces. Figure 1 shows the topology referenced in the following examples:

  1. Ping router CE2 from router CE1.
    user@vpn4> ping 192.168.193.5 local 10.255.10.4 count 3
  2. To determine the path from Router CE1’s loopback interface to Router CE2’s directly connected interface, use the traceroute command:
    user@vpn4> traceroute 192.168.193.5 source 10.255.10.4
  3. Ping Router PE2 (VPN2) from Router CE1 (VPN4). In this case, packets that originate at Router CE1 go to Router PE2, then to Router CE2, and back to Router PE2 before Router PE2 can respond to Internet Control Message Protocol (ICMP) requests. You can verify this by using the traceroute command.
    user@vpn4> ping 192.168.193.2 local 10.255.10.4 count 3
  4. To determine the path from Router CE1 to Router PE2, use the traceroute command:
    user@vpn4> traceroute 192.168.193.2 source 10.255.10.4

Pinging a CE Router from a Multiaccess Interface

Step-by-Step Procedure

You cannot ping one CE router from the other if the VPN interface is a multiaccess interface, such as the fe-1/1/2.0 interface on Router CE1. To ping Router CE1 from Router CE2, you must either include the vrf-table-label statement at the [edit routing-instances routing-instance-name] hierarchy level on Router PE1 or configure a static route on Router PE1 to the VPN interface of Router CE1. If you include the vrf-table-label statement to ping a router, you cannot configure a static route.

  1. If you configure a static route on Router PE1 to the VPN interface of Router CE1, its next hop must point to Router CE1 (at the [edit routing-instance routing-instance-name] hierarchy level), and this route must be announced from Router PE1 to Router PE2 as shown in the following configuration:
  2. Now you can ping Router CE1 from Router CE2:
    user@vpn5> ping 192.168.192.4 local 10.255.10.5 count 3
  3. To determine the path between these two interfaces, use the traceroute command:
    user@vpn5> traceroute 192.168.192.4 source 10.255.10.5

Pinging the Directly Connected PE Routers from the CE Routers

Step-by-Step Procedure

From the loopback interfaces on the CE routers, you can ping the VPN interface on the directly connected PE router. Figure 1 shows the topology referenced in this procedure:

  1. From the loopback interface on Router CE1 (VPN4), ping the VPN interface, fe-1/1/0.0, on Router PE1:
    user@vpn4> ping 192.168.192.1 local 10.255.10.4 count 3
  2. From the loopback interface on Router CE2 (VPN5), ping the VPN interface, t3-0/0/3.0, on Router PE2:
    user@vpn5> ping 192.168.193.2 local 10.255.10.5 count 3
  3. From the loopback interface on Router CE2 (VPN5), ping the VPN interface, t3-0/0/3.0, on Router PE2:
    user@vpn5> ping 192.168.193.2 local 10.255.10.5 count 3
  4. To determine the path from the loopback interface on Router CE2 to the VPN interfaces on Router PE2, use the traceroute command:
    user@vpn5> traceroute 192.168.193.2 source 10.255.10.5

Pinging the Directly Connected CE Routers from the PE Routers

Step-by-Step Procedure

From the VPN and loopback interfaces on the PE routers, you can ping the VPN interface on the directly connected CE router. Figure 1 shows the topology referenced in this procedure:

  1. From the VPN interface on the PE router (router PE1), you can ping the VPN or loopback interface on the directly connected CE router (router CE1).

    From the VPN interface on Router PE1 (VPN1), ping the VPN interface, fe-1/1/0.0, on Router CE1:

    user@vpn1> ping 192.168.192.4 interface fe-1/1/0.0 local 192.168.192.1 count 3
  2. From the VPN interface on Router PE1 (VPN1), ping the loopback interface, 10.255.10.4, on Router CE1:
    user@vpn1> ping 10.255.10.4 interface fe-1/1/0.0 local 192.168.192.1 count 3
  3. To determine the path from the VPN interface on Router PE1 to the VPN and loopback interfaces on Router CE1, respectively, use the following traceroute commands:
    user@vpn1> traceroute 10.255.10.4 interface fe-1/1/0.0 source 192.168.192.1
  4. From the VPN interface on Router PE2 (VPN2), ping the VPN interface, t3-0/0/3.0, on Router CE2:
    user@vpn2> ping 192.168.193.5 interface t3-0/0/3.0 local 192.168.193.2 count 3
  5. From the VPN interface on Router PE2 (VPN2), ping the loopback interface, 10.255.10.5, on Router CE2:
    user@vpn2> ping 10.255.10.5 interface t3-0/0/3.0 local 192.168.193.2 count 3
  6. To determine the path from the VPN interface on Router PE2 to the VPN and loopback interfaces on Router CE2, respectively, use the following traceroute commands:
    user@vpn2> traceroute 10.255.10.5 interface t3-0/0/3.0 source 192.168.193.2

Pinging the Remote CE Router from the Local PE Router

Step-by-Step Procedure

The following procedure is effective for Layer 3 VPNs only. To ping a remote CE router from a local PE router in a Layer 3 VPN, you need to configure the following interfaces:

  1. Configure a logical unit for the loopback interface.

    To configure an additional logical unit on the loopback interface of the PE router, configure the unit statement at the [edit interfaces lo0] hierarchy level:

  2. Configure the loopback interface for the Layer 3 VPN routing instance on the local PE router. You can associate one logical loopback interface with each Layer 3 VPN routing instance, enabling you to ping a specific routing instance on a router.

    Specify the loopback interface you configured in Step 1 using the interface statement at the [edit routing-instances routing-instance-name] hierarchy level:

    The interface-name is the logical unit on the loopback interface (for example, lo0.1).

  3. From the VPN interface on PE router, you can now ping the logical unit on the loopback interface on the remote CE router:

    Use interface to specify the new logical unit on the loopback interface (for example, lo0.1). For more information about how to use the ping interface command, see the Junos Interfaces Command Reference.

Troubleshooting Inconsistently Advertised Routes from Gigabit Ethernet Interfaces

Step-by-Step Procedure

For direct routes on a LAN in a Layer 3 VPN, the Junos OS attempts to locate a CE router that can be designated as the next hop. If this cannot be done, advertised routes from Gigabit Ethernet interfaces are dropped.

In such instances:

  1. Use the static statement at the [edit routing-options] or [edit logical-systems logical-system-name routing-options] hierarchy levels in the VRF routing instance to a CE router on the LAN subnet, configuring the CE router as the next hop. All traffic to directly destinations on this LAN will go to the CE router. You can add two static routes to two CE routers on the LAN for redundancy.
  2. Configure the vrf-table-label statement at the [edit routing-instances routing-instance-name] hierarchy levels to map the inner label of a packet to a specific VRF routing table. This allows the examination of the encapsulated IP header to force IP lookups on the VRF routing instance for all traffic.Note

    The vrf-table-label statement is not available for every core-facing interface; for example, channelized interfaces are not supported. See Filtering Packets in Layer 3 VPNs Based on IP Headers for information about support for the vrf-table-label statement over Ethernet and SONET/SDH interfaces.

This example shows how to disable TTL decrementing in a single VRF routing instance in a Layer 3 VPN scenario.

Requirements

Before you begin:

  • Configure the router interfaces. See the Network Interfaces Configuration Guide.

Overview

To diagnose networking problems related to VPNs, it can be useful to disable normal time-to-live (TTL) decrementing. The IP header includes a TTL field that serves as a hop counter. At every routed hop, the TTL is decremented by one; if the TTL reaches zero before the packet reaches its destination, the packet is discarded and (optionally) an ICMP TTL exceeded message is sent to the source. MPLS labels also have a TTL field. MPLS routers copy the TTL of an IP packet when it enters a label-switched path (LSP). An IP packet with a TTL of 27 receives an MPLS label with a TTL of 27. Junos OS decrements the MPLS TTL of an MPLS-encapsulated packet in place of the IP TTL, at every label-switched hop. Because the MPLS TTL is copied (or propagated) from the IP TTL, a traceroute lists every hop in the path, be it routed or label-switched. When the packet exits the LSP, the decremented MPLS TTL is propagated back into the IP TTL field.

By default, TTL propagation is enabled. The global no-propagate-ttl statement disables TTL propagation at the router level and affects all RSVP-signalled or LDP-signalled LSPs. When a router acts as an ingress router for an LSP and the router configuration includes the no-propagate-ttl statement, the router pushes an MPLS header with a TTL value of 255, regardless of the IP packet TTL. When a router acts as the penultimate router, it pops the MPLS header without propagating the MPLS TTL into the IP packet. Thus the IP packet TTL value is preserved, regardless of the hop count of the LSP.

Instead of configuring TTL propagation behavior at the router level, you can configure the behavior for the routes in a VRF routing instance. This example shows how to disable TTL propagation for the routes in a single VRF routing instance instead of at the global router level.

The per-VRF configuration takes precedence over the global router configuration. If you disable TTL propagation on the router and explicitly enable TTL propagation for a single VRF routing instance, TTL propagation is in effect for that routing instance. To explicitly enable TTL propagation on a VRF routing instance, include the vrf-propagate-ttl statement in the routing instance.

When you change the TTL propagation behavior, old next hops for VRF routes are deleted from the inet.3 routing table and new next hops are added.

You need only configure the vrf-propagate-ttl or no-vrf-propagate-ttl statement on the ingress routers.

Topology Diagram

Figure 2 shows the topology used in this example. Router PE1 and Router PE2 have two VPNs---VPN-A and VPN-B. Devices CE1 and CE4 belong to VPN-A. Devices CE2 and CE5 belong to VPN-B. In this example, Router PE1 has TTL propagation disabled on VPN-A but not on VPN-B. Packets received by PE1 on the interface connected to CE1 have TTL propagation disabled. This example shows the configuration on Router PE1. You do not need to include the no-vrf-propagate-ttl statement on the egress router (PE2).

Figure 2: Disabling TTL Propagation for a Single VPN
Disabling TTL Propagation
for a Single VPN

Configuration

CLI Quick Configuration

To quickly disable TTL propagation in a VRF routing instance, copy the following commands and paste the commands into the CLI.

Step-by-Step Procedure

The following example requires you to navigate various levels in the configuration hierarchy. For information about navigating the CLI, see Using the CLI Editor in Configuration Mode.

To configure a flow map:

  1. Configure the loopback interface.

  2. Configure the routing protocols.

    The internal BGP neighbor address is the loopback interface address of Router PE2 in Figure 2.

  3. Configure routing policies for VPN-A and VPN-B.

  4. Configure the VPN-A and VPN-B routing instances, including the no-vrf-propagate-ttl statement in VPN-A.

  5. Define the local autonomous system.

  6. If you are done configuring the device, commit the configuration.

Results

Confirm your configuration by entering the show interfaces, show policy-options, show protocols, show routing-instances, and show routing-options commands.

Verification

To verify the operation, run the following commands:

  • See the TTL Action field in the output of the show route extensive table VPN-A command.

  • See the TTL Action field in the output of the show route extensive table VPN-B command.

  • On Device CE1, run the traceroute command to Device CE4's loopback address.

  • On Device CE4, run the traceroute command to Device CE1's loopback address.