Determining Why Mastership Switched

Mastership can switch between the master Routing Engine and the backup Routing Engine for the following reasons:

• Hardware problems.
• The master Routing Engine is pulled.
• Software issues, such as a Routing Engine kernel crash.

Action

View the log file /var/log/mastership for redundancy logging. This file contains hardware and software transitions to help debug auto-redundancy issues.

user@host> show log mastership

Table 107 lists the event codes that can be displayed in the mastership log.

Table 107: Logging Events

Event Code	Description
E_NULL = 0	The event is a null event.
E_CFG_M	The Routing Engine is configured as master.
E_CFG_B	The Routing Engine is configured as backup.
E_CFG_D	The Routing Engine is configured as disabled.
E_MAXTRY	The maximum number of tries to acquire or release mastership was exceeded.
E_REQ_C	A claim mastership request was sent.
E_ACK_C	A claim mastership acknowledgement was received.
E_NAK_C	A claim mastership request was not acknowledged.
E_REQ_Y	Confirmation of mastership is requested.
E_ACK_Y	Mastership is acknowledged.
E_NAK_Y	Mastership is not acknowledged.
E_REQ_G	A giveup mastership request was sent by a Routing Engine.
E_ACK_G	The Routing Engine acknowledges giveup of mastership.
E_CMD_A	The command request chassis routing-engine master acquire was issued from the backup Routing Engine.
E_CMD_F	Force switchover command was issued.
E_CMD_R	The command request chassis routing-engine master release was issued from the master Routing Engine.
E_CMD_S	The command request chassis routing-engine master switch was issued from a Routing Engine.
E_NO_ORE	No other Routing Engine is detected.
E_TMOUT	A request timed out.
E_NO_IPC	Routing Engine connection was lost.
E_ORE_M	Other Routing Engine state was changed to master.
E_ORE_B	Other Routing Engine state was changed to backup.
E_ORE_D	Other Routing Engine state was changed to disabled.

Sample Output

user@host> show log mastership

Jan 12 21:50:05  clear-log[865]: logfile cleared

Jan 12 21:50:18 failed to receive keepalives from other RE for the last 60 sec 
Jan 12 21:50:23 failed to send RE info/keepalive: errno=22, total=6 in the last 
20 sec

Jan 12 21:50:23 failed to send RE info/keepalive: errno=22, total=6 in the last 
20 sec

Jan 12 21:50:34 event = E_CMD_R, state = master, param = 0x0 Jan 12 21:50:34 
send "you are the master" request Jan 12 21:50:34 Failed to send RE mastership 
cmd. err = 65 Jan 12 21:50:34 Currentstate: master  NextState:giveup

                   reason_code: 1

Jan 12 21:50:34 timestamp: Wed Jan 12 21:50:34 2000

Jan 12 21:50:34 new state = giveup

Jan 12 21:50:36 event = E_TMOUT, state = giveup, param = 0x0 Jan 12 21:50:36 
send "you are the master" request

Jan 12 21:50:36 Failed to send RE mastership cmd. err = 65 Jan 12 21:50:36 
Currentstate: giveup  NextState:giveup

                   reason_code: 1

Jan 12 21:50:36 new state = giveup

Jan 12 21:50:38 event = E_TMOUT, state = giveup, param = 0x0

Jan 12 21:50:38 send "you are the master" request

Jan 12 21:50:38 Failed to send RE mastership cmd. err = 65

Jan 12 21:50:38 Currentstate: giveup  NextState:giveup

                   reason_code: 1

Jan 12 21:50:38 new state = giveup

Jan 12 21:50:40 failed to receive keepalives from other RE for the last 80 sec 
Jan 12 21:50:41 event = E_TMOUT, state = giveup, param = 0x0

Jan 12 21:50:41 send "you are the master" request

Jan 12 21:50:41 Failed to send RE mastership cmd. err = 65

Jan 12 21:50:41 Currentstate: giveup  NextState:giveup

                   reason_code: 1

Jan 12 21:50:41 new state = giveup

Jan 12 21:50:43 event = E_TMOUT, state = giveup, param = 0x0

Jan 12 21:50:43 send "you are the master" request

Jan 12 21:50:43 Failed to send RE mastership cmd. err = 65

Jan 12 21:50:43 Currentstate: giveup  NextState:giveup

                   reason_code: 1

Jan 12 21:50:43 new state = giveup

Jan 12 21:50:46 failed to send RE info/keepalive: errno=35, total=7 in the last 
20 sec

Jan 12 21:50:46 failed to send RE info/keepalive: errno=35, total=7 in the last 
20 sec

Jan 12 21:50:46 event = E_TMOUT, state = giveup, param = 0x0

Jan 12 21:50:46 send "you are the master" request

Jan 12 21:50:46 Failed to send RE mastership cmd. err = 65

Jan 12 21:50:46 Currentstate: giveup  NextState:giveup

                   reason_code: 1

Jan 12 21:50:46 new state = giveup

Jan 12 21:50:48 event = E_TMOUT, state = giveup, param = 0x0

Jan 12 21:50:48 send "you are the master" request

Jan 12 21:50:48 Failed to send RE mastership cmd. err = 65

Jan 12 21:50:48 Currentstate: giveup  NextState:giveup

                   reason_code: 1

Jan 12 21:50:48 new state = giveup

Jan 12 21:50:50 event = E_TMOUT, state = giveup, param = 0x0

Jan 12 21:50:50 send "you are the master" request

Jan 12 21:50:50 Failed to send RE mastership cmd. err = 65

Jan 12 21:50:50 Currentstate: giveup  NextState:giveup

                   reason_code: 1

Jan 12 21:50:50 new state = giveup

Jan 12 21:50:53 event = E_MAXTRY, state = giveup, param = 0x0

Jan 12 21:50:53 Currentstate: giveup  NextState:master

                   reason_code: 1

Jan 12 21:50:53 timestamp: Wed

Jan 12 21:50:53 2000

Jan 12 21:50:53 new state = master

Jan 12 21:51:01 failed to receive keepalives from other RE for the last 100 sec 
Jan 12 21:51:06 failed to send RE info/keepalive: errno=65, total=7 in the last 
20 sec

Jan 12 21:51:06 failed to send RE info/keepalive: errno=65, total=7 in the last 
20 sec

Jan 12 21:51:21 failed to receive keepalives from other RE for the last 120 sec 
Jan 12 21:51:26 failed to send RE info/keepalive: errno=22, total=6 in the last 
20 sec

Jan 12 21:51:26 failed to send RE info/keepalive: errno=22, total=6 in the last 
20 sec

What It Means

The beginning of the log shows that keepalives are not being responded to and the state of the Routing Engine changed from master to giveup after the request chassis routing-engine master release command was issued. However, the other Routing Engine is not taking over mastership because it is unreachable. Eventually a timeout (E_TMOUT) occurs until the Routing Engine reaches the maximum number of attempts permitted (E_MAXTRY). The output then shows the Routing Engine state changing from giveup back to master.

The output doesn't indicate why the mastership switchover did not work. However, it is clear that the backup Routing Engine is unreachable.