Examining a CSPF Failure - MPLS Network Operations Guide Log Reference

When a local CSPF failure indicates that no path meets the constraints configured for the LSP, you must perform CSPF-based tracing and be familiar with the contents of the traffic engineering database to resolve the problem. See Examine the Traffic Engineering Database for an analysis of the traffic engineering database.

Verify the CSPF Failure

Purpose

To simulate a configuration error on the network, router R5 has the administrative group coloring removed from interface so-0/0/0. The result is a CSPF failure at R5 because there is no longer a path between R1 and R6 that includes the red color.

Action

To confirm that the LSP is down and verify the configuration on routers R1 and R5, enter the following JUNOS CLI operational mode commands:

Sample Output 1

user@R1> clear mpls lsp 
[edit protocols mpls]
user@R1#  run show mpls lsp extensive
Ingress LSP: 1 sessions

10.0.0.6
  From: 0.0.0.0,  State: Dn
 , ActiveRoute: 0, LSPname: R1-to-R6
   ActivePath: (none)
  LoadBalance: Random
  Metric: 100
  Encoding type: Packet, Switching type: Packet, GPID: IPv4
  Primary                    State: Dn
          Include: red    Exclude: blue
    Will be enqueued for recomputation in 24 second(s).
     9 May 11 20:12:28 CSPF failed: no route toward 10.0.0.6
    8 May 11 20:12:28 Clear Call
    7 May 11 20:12:28 Deselected as active
    6 May 11 19:31:42 Selected as active path
    5 May 11 19:31:42 Record Route:  10.1.15.2 10.1.56.2
    4 May 11 19:31:42 Up
    3 May 11 19:31:42 Originate Call
    2 May 11 19:31:42 CSPF: computation result accepted
    1 May 11 19:31:12 CSPF failed: no route toward 10.0.0.6[5 times]
  Created: Wed May 11 19:29:17 2005
Total 1 displayed, Up 0, Down 1
[...Output truncated...]

Sample Output 2

Meaning

Sample Output 1 from ingress router R1 shows that the clear mpls lsp command was issued to confirm that R1 cannot reestablish LSP R1-to-R6. The sample output from the show mpls lsp extensive command shows that LSP R1-to-R6 is down, State: Dn and ActivePath: (None); and that the CSPF has failed, CSPF failed: no route toward 10.0.0.6.

Sample Output 2 from transit router R5 shows that LSP R1-to-R6 is not included in the output, indicating that the LSP is not transiting R5.

Most network problems appear as a local CSPF failure, as shown in the sample output. The CSPF failure indicates that no path meeting the constraints for the LSP can be found in the router’s traffic engineering database. To resolve these problems effectively, use CSPF tracing on the ingress router, and analyze the traffic engineering database to locate the node that should meet the constraints.

Examine the CSPF Log File

Purpose

After you have confirmed that the LSP is down, obtain more information about the possible cause of the failure.

Action

To examine the CSPF log file, enter the following JUNOS CLI operational mode commands:

Sample Output

user@R1> monitor start cspf 

[edit protocols mpls]
user@R1#  run show log cspf-failed3    
May 27 10:22:23 trace_on:  Tracing to "/var/log/cspf"
  started
May 27 10:22:29 CSPF adding path R1-to-R6(primary ) to CSPF queue 1
May 27 10:22:29 CSPF creating CSPF job
May 27 10:22:29
May 27 10:22:29 CSPF for path R1-to-R6(primary ), begin at R1.00 , starting
May 27 10:22:29          path include: 0x00000100
 << administration group red
May 27 10:22:29          path exclude: 0x00000010
 << administration group blue
May 27 10:22:29         bandwidth: CT0=0bps ; setup priority: 0; random
May 27 10:22:29 CSPF final destination 10.0.0.6
May 27 10:22:29 CSPF starting from R1.00 (10.0.0.1) to 10.0.0.6, hoplimit 254
May 27 10:22:29         constraint include 0x00000100
May 27 10:22:29         constraint exclude 0x00000010
May 27 10:22:29    Node R1.00 (10.0.0.1) metric 0, hops 0, avail 32000 32000 32000 32000
May 27 10:22:29      Link 10.1.12.1->10.1.12.2(R2.00/10.0.0.2) metric 10 color 0x00000000 bw 155.52Mbps
May 27 10:22:29         Reverse Link for 10.1.12.1->10.1.12.2 is 10.1.12.2->10.1.12.1
May 27 10:22:29         link fails include 0x00000100
May 27 10:22:29      Link 10.1.15.1->10.1.15.2(R5.00/10.0.0.5) metric 10 color 0x00000100 bw 155.52Mbps
May 27 10:22:29         Reverse Link for 10.1.15.1->10.1.15.2 is 10.1.15.2->10.1.15.1
May 27 10:22:29         link's interface switch capability descriptor #1
May 27 10:22:29           encoding: Packet, switching: Packet
May 27 10:22:29          link passes constraints
May 27 10:22:29      Link 10.1.13.1->10.1.13.2(R3.00/10.0.0.3) metric 10 color 0x00000010 bw 155.52Mbps
May 27 10:22:29         Reverse Link for 10.1.13.1->10.1.13.2 is 10.1.13.2->10.1.13.1
May 27 10:22:29         link fails include 0x00000100
May 27 10:22:29    Node R5.00 (10.0.0.5) metric 10, hops 1, avail 32000 32000 32000 32000
May 27 10:22:29      Link 10.1.15.2->10.1.15.1(R1.00/10.0.0.1) metric 10 color 0x00000100 bw 155.52Mbps
May 27 10:22:29         skipped: end point already visited
May 27 10:22:29      Link 10.1.45.2->10.1.45.1(R4.00/10.0.0.4) metric 10 color 0x00000000 bw 155.52Mbps
May 27 10:22:29         Reverse Link for 10.1.45.2->10.1.45.1 is 10.1.45.1->10.1.45.2
May 27 10:22:29         link fails include 0x00000100
May 27 10:22:29      Link 10.1.56.1->10.1.56.2(R6.00/10.0.0.6) metric 10 color 0x00000000 bw 155.52Mbps
May 27 10:22:29          Reverse Link for 10.1.56.1->10.1.56.2 is 10.1.56.2->10.1.56.1
May 27 10:22:29          link fails include 0x00000100
May 27 10:22:29 CSPF completed in 0s
May 27 10:22:29 CSPF couldn't find a route to 10.0.0.6
May 27 10:22:29 CSPF for R1-to-R6 done!
monitor stop

Meaning

The sample output shows that the monitor start cspf command was issued to start displaying entries in the cspf log file in real time. The cspf log file is generated by the routing protocol process after the file is configured with the traceoptions statement at the [edit protocols mpls] hierarchy level. In this example, the cspf log file is configured with the cspf, cspf-node, and cspf-link flags to provide the most granular information about the steps taken by the CSPF algorithm. For information on configuring CSPF tracing, see Configuring CSPF Tracing.

The only link that passes the color constraint is between R1 and R5, 10.1.15.0/32. The CSPF algorithm is a locally run algorithm, which makes its calculations on a given router. When the CSPF algorithm runs on R5, it prunes 10.1.15.2 and selects 10.1.56.1 to send the message to R6.The link between R5 and R6 10.1.56.0/32 does not pass the color constraints, indicating a problem with R5. At this stage, it is useful to examine the traffic engineering database to determine which link on R5 should be associated with the red color.

Examine the Traffic Engineering Database

Purpose

Examining the traffic engineering database is another way to locate the node that should meet the constraints but does not. Once identified, you can concentrate your troubleshooting efforts on why that node is not being represented accurately in the database.

The contents of the traffic engineering database are consistent among all routers within a given traffic engineering domain. Therefore, you can issue the show ted database command from any router in the same traffic engineering domain to obtain more granular information about the CSPF failure.

CSPF integrates topology link-state information that is learned from the IGP traffic engineering extensions and maintained in the traffic engineering database. The information stored in the traffic engineering database includes attributes associated with the state of the network resources (such as total link bandwidth, reserved link bandwidth, available link bandwidth, and link color). When calculating a path, the CSPF algorithm factors in user–provided information such as bandwidth requirements, maximum allowed hop count, and administrative groups, all of which are obtained from user configuration. (See Figure 5).

Action

To examine the traffic engineering database, enter the following JUNOS CLI operational mode commands:

Sample Output 1

[edit protocols mpls]
user@R1#  run show ted database extensive
TED database: 6 ISIS nodes 6 INET nodes
[...Output truncated...]
NodeID: R5.00(10.0.0.5)
   Type: Rtr , Age: 103 secs, LinkIn: 3, LinkOut: 3
   Protocol: IS-IS(2)
    To: R1.00(10.0.0.1), Local: 10.1.15.2, Remote: 10.1.15.1
      Color: 0x100 red
      Metric: 10
      Static BW: 155.52Mbps
      Reservable BW: 155.52Mbps
      Available BW [priority] bps:
          [0] 155.52Mbps   [1] 155.52Mbps  [2] 155.52Mbps  [3] 155.52Mbps  
          [4] 155.52Mbps   [5] 155.52Mbps  [6] 155.52Mbps  [7] 155.52Mbps  
      Interface Switching Capability Descriptor(1):
        Switching type: Packet
        Encoding type: Packet
        Maximum LSP BW [priority] bps:
          [0] 155.52Mbps   [1] 155.52Mbps  [2] 155.52Mbps  [3] 155.52Mbps  
          [4] 155.52Mbps   [5] 155.52Mbps  [6] 155.52Mbps  [7] 155.52Mbps  
     To: R4.00(10.0.0.4) , Local: 10.1.45.2, Remote: 10.1.45.1
      Color: 0 <none>
      Metric: 10
      Static BW: 155.52Mbps
      Reservable BW: 155.52Mbps
      Available BW [priority] bps:
          [0] 155.52Mbps   [1] 155.52Mbps  [2] 155.52Mbps  [3] 155.52Mbps  
          [4] 155.52Mbps   [5] 155.52Mbps  [6] 155.52Mbps  [7] 155.52Mbps  
      Interface Switching Capability Descriptor(1):
        Switching type: Packet
        Encoding type: Packet
        Maximum LSP BW [priority] bps:
          [0] 155.52Mbps   [1] 155.52Mbps  [2] 155.52Mbps  [3] 155.52Mbps  
          [4] 155.52Mbps   [5] 155.52Mbps  [6] 155.52Mbps  [7] 155.52Mbps  
    To: R6.00(10.0.0.6), Local: 10.1.56.1, Remote: 10.1.56.2
      Color: 0 <none>
      Metric: 10
      Static BW: 155.52Mbps
      Reservable BW: 155.52Mbps
      Available BW [priority] bps:
          [0] 155.52Mbps   [1] 155.52Mbps  [2] 155.52Mbps  [3] 155.52Mbps  
          [4] 155.52Mbps   [5] 155.52Mbps  [6] 155.52Mbps  [7] 155.52Mbps  
      Interface Switching Capability Descriptor(1):
        Switching type: Packet
        Encoding type: Packet
        Maximum LSP BW [priority] bps:
          [0] 155.52Mbps   [1] 155.52Mbps  [2] 155.52Mbps  [3] 155.52Mbps  
          [4] 155.52Mbps   [5] 155.52Mbps  [6] 155.52Mbps  [7] 155.52Mbps 
[...Output truncated...]

Sample Output 2

[edit protocols]
user@R1#  run show ted database extensive R5.00 | match "(NodeID|To:|Color)" 
NodeID: R5.00(10.0.0.5)
    To: R1.00(10.0.0.1), Local: 10.1.15.2, Remote: 10.1.15.1
      Color: 0x100 red
    To: R4.00(10.0.0.4), Local: 10.1.45.2, Remote: 10.1.45.1
      Color: 0 <none>
    To: R6.00(10.0.0.6), Local: 10.1.56.1, Remote: 10.1.56.2
      Color: 0 <none>
    To: R1.00(10.0.0.1), Local: 10.1.15.2, Remote: 10.1.15.1
      Color: 0x100 red
    To: R4.00(10.0.0.4), Local: 10.1.45.2, Remote: 10.1.45.1
      Color: 0 <none>
     To: R6.00(10.0.0.6), Local: 10.1.56.1, Remote: 10.1.56.2
      Color: 0 <none>

Meaning

Sample Output 1 from ingress router R1 shows a wealth of information on each node in the network, although only a portion is included in this example. The output shows the total number of IS-IS and INET nodes in the traffic engineering domain. The portion of the traffic engineering database shown represents a node (R5), and the Type field indicates Rtr (router). The Type field could also indicate Net (network) if the node were a pseudo node. The node (R5) has three input and output links that are running IS-IS Level 2, Protocol: IS-IS(2). The links lead to nodes R1, R4, and R6. The local address and remote address for each link is specified. The information on each node includes administrative groups (Color:), metrics, static bandwidth, reservable bandwidth, and available bandwidth priority level. The information contained in the traffic engineering database should be the same across all routers in the same traffic engineering domain. For a detailed description of the fields in the output of the show ted database extensive command, see the JUNOS Routing Protocols and Policies Command Reference.

Sample Output 2 shows filtered output that allows you to focus on exactly what is missing or incorrect.

Both outputs confirm that the link between R1 and R5, 10.1.15.0/32, is associated with the red color, while the link between R5 and R6, 10.1.56.0/32, is not associated with a color. In the network shown in Figure 4, for the LSP to establish correctly, link 10.1.56.1 must be associated with the red color.

Check the Administrative Group Configuration on R5

Purpose

Focus on R5 to determine which interfaces are associated with the red color, and make any necessary corrections.

Action

To check the administrative group configuration on R5 and make any necessary corrections, enter the following JUNOS CLI commands:

Sample Output 1

Sample Output 2

[edit protocols mpls]
user@R5# delete interface so-0/0/1 admin-group 

[edit protocols mpls]
user@R5# set interface so-0/0/0 admin-group red  

[edit protocols mpls]
user@R5# show  
admin-groups {
    red 8;
    blue 4;
}
interface fxp0.0 {
    disable;
}
interface so-0/0/0.0 { <<<correct interface configured with admin-group
    admin-group red;
}
interface so-0/0/1.0;
interface so-0/0/2.0;

[edit protocols mpls]
user@R5# commit  
commit complete

Sample Output 3

Meaning

Sample Output 1 from transit router R5 shows that at the [edit protocols mpls] hierarchy level, interface so-0/0/1 is incorrectly configured with the admin-group red statement. The so-0/0/0 interface should be configured with the admin-group red statement.

Sample Output 2 shows the steps taken to correct the configuration. The administration group has been deleted from so-0/0/1 and so-0/0/0 is now associated with the red color.