Tracing SNMP Activity
Monitoring SNMP Activity and Tracking Problems That Affect SNMP Performance on a Device Running Junos OS
The following sections contain information about monitoring the SNMP activity on devices running the Junos OS and identifying problems that might impact the SNMP performance on devices running Junos OS:
- Checking for MIB Objects Registered with the snmpd
- Tracking SNMP Activity
- Monitoring SNMP Statistics
- Checking CPU Utilization
- Checking Kernel and Packet Forwarding Engine Response
Checking for MIB Objects Registered with the snmpd
For the SNMP process to be able to access data related to a MIB object, the MIB object must be registered with the snmpd. When an SNMP subagent comes online, it tries to register the associated MIB objects with the snmpd. The snmpd maintains a mapping of the objects and the subagents with which the objects are associated. However, the registration attempt fails occasionally, and the objects remain unregistered with the snmpd until the next time the subagent restarts and successfully registers the objects.
When a network management system polls for data related to objects
that are not registered with the snmpd, the snmpd returns either a noSuchName
error (for SNMPv1 objects) or a noSuchObject
error (for SNMPv2 objects).
You can use the following commands to check for MIB objects that are registered with the snmpd:
show snmp registered-objects
—Creates a/var/log/snmp_reg_objs
file that contains the list of registered objects and their mapping to various subagents.file show /var/log/snmp_reg_objs
—Displays the contents of the/var/log/snmp_reg_objs
file.
The following example shows the steps for creating and displaying
the /var/log/snmp_reg_objs
file:
user@host> show snmp registered-objects user@host> file show /var/log/snmp_reg_objs -------------------------------------------------------------- Registered MIB Objects root_name = -------------------------------------------------------------- .1.2.840.10006.300.43.1.1.1.1.2 (dot3adAggMACAddress) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.1.1.1.3 (dot3adAggActorSystemPriority) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.1.1.1.4 (dot3adAggActorSystemID) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.1.1.1.5 (dot3adAggAggregateOrIndividual) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.1.1.1.6 (dot3adAggActorAdminKey) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.1.1.1.7 (dot3adAggActorOperKey) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.1.1.1.8 (dot3adAggPartnerSystemID) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.1.1.1.9 (dot3adAggPartnerSystemPriority) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.1.1.1.10 (dot3adAggPartnerOperKey) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.1.1.1.11 (dot3adAggCollectorMaxDelay) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.1.2.1.1 (dot3adAggPortListPorts) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.2.1.1.2 (dot3adAggPortActorSystemPriority) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.2.1.1.3 (dot3adAggPortActorSystemID) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.2.1.1.4 (dot3adAggPortActorAdminKey) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.2.1.1.5 (dot3adAggPortActorOperKey) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.2.1.1.6 (dot3adAggPortPartnerAdminSystemPriority) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.2.1.1.7 (dot3adAggPortPartnerOperSystemPriority) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.2.1.1.8 (dot3adAggPortPartnerAdminSystemID) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.2.1.1.9 (dot3adAggPortPartnerOperSystemID) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.2.1.1.10 (dot3adAggPortPartnerAdminKey) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.2.1.1.11 (dot3adAggPortPartnerOperKey) (/var/run/mib2d-11) .1.2.840.10006.300.43.1.2.1.1.12 (dot3adAggPortSelectedAggID) (/var/run/mib2d-11) ---(more)---
The /var/log/snmp_reg_objs file contains only those objects that are associated with the Junos
OS processes that are up and running and registered with the snmpd,
at the time of executing the show snmp registered-objects
command. If a MIB object related to a Junos OS process that is up
and running is not shown in the list of registered objects, you might
want to restart the software process to retry object registration
with the snmpd.
Tracking SNMP Activity
SNMP tracing operations track activity of SNMP agents and record
the information in log files. The logged event descriptions provide
detailed information to help you solve problems faster. By default,
Junos OS does not trace any SNMP activity. To enable tracking of SNMP
activities on a device running Junos OS, include the traceoptions
statement at the [edit snmp]
hierarchy level.
A sample traceoptions
configuration might
look like:
[edit snmp] set traceoptions flag all;
When the traceoptions flag all
statement is included
at the [edit snmp]
hierarchy level, the following log files
are created:
snmpd
mib2d
rmopd
You can use the show log log-filename
operational mode command to view the contents of the log file.
In the snmpd log file (see the following example), a sequence of >>>
represents an incoming packet, whereas a sequence
of <<<
represents an outgoing
packet. Note that the request response pair might not follow any sequence
if there are multiple network management systems polling the device
at the same time. You can use the source and request ID combinations
to match requests and responses. However, note that no response log
is created in the log file if the SNMP master agent or the SNMP subagent
has not responded to a request.
A careful analysis of the request-response time can help you identify and understand delayed responses.
Reviewing a Log File
The following example shows the output for the show log
snmpd
command:
user@host> show log snmpd Apr 12 06:40:03 snmpd[7ee783df] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 12 06:40:03 snmpd[7ee783df] >>> Get-Bulk-Request Apr 12 06:40:03 snmpd[7ee783df] >>> Source: 10.209.63.42 Apr 12 06:40:03 snmpd[7ee783df] >>> Destination: 10.209.2.242 Apr 12 06:40:03 snmpd[7ee783df] >>> Version: SNMPv2 Apr 12 06:40:03 snmpd[7ee783df] >>> Request_id: 0x7ee783df Apr 12 06:40:03 snmpd[7ee783df] >>> Community: public Apr 12 06:40:03 snmpd[7ee783df] >>> Non-repeaters: 0 Apr 12 06:40:03 snmpd[7ee783df] >>> Max-repetitions: 10 Apr 12 06:40:03 snmpd[7ee783df] >>> OID : jnxContentsType.6.1.2.0 Apr 12 06:40:03 snmpd[7ee783df] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 12 06:40:03 snmpd[7ee783df] <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Apr 12 06:40:03 snmpd[7ee783df] <<< Get-Response Apr 12 06:40:03 snmpd[7ee783df] <<< Source: 10.209.63.42 Apr 12 06:40:03 snmpd[7ee783df] <<< Destination: 10.209.2.242 Apr 12 06:40:03 snmpd[7ee783df] <<< Version: SNMPv2 Apr 12 06:40:03 snmpd[7ee783df] <<< Request_id: 0x7ee783df Apr 12 06:40:03 snmpd[7ee783df] <<< Community: public Apr 12 06:40:03 snmpd[7ee783df] <<< Error: status=0 / vb_index=0 Apr 12 06:40:03 snmpd[7ee783df] <<< Apr 12 06:40:03 snmpd[7ee783df] <<< OID : jnxContentsType.7.1.0.0 Apr 12 06:40:03 snmpd[7ee783df] <<< type : Object Apr 12 06:40:03 snmpd[7ee783df] <<< value: jnxM10iFPC.0 Apr 12 06:40:03 snmpd[7ee783df] <<< Apr 12 06:40:03 snmpd[7ee783df] <<< OID : jnxContentsType.7.1.1.0 Apr 12 06:40:03 snmpd[7ee783df] <<< type : Object Apr 12 06:40:03 snmpd[7ee783df] <<< value: jnxChassisTempSensor.0 Apr 12 06:40:03 snmpd[7ee783df] <<< Apr 12 06:40:03 snmpd[7ee783df] <<< OID : jnxContentsType.7.2.0.0 Apr 12 06:40:03 snmpd[7ee783df] <<< type : Object Apr 12 06:40:03 snmpd[7ee783df] <<< value: jnxM10iFPC.0 Apr 12 06:40:03 snmpd[7ee783df] <<< Apr 12 06:40:03 snmpd[7ee783df] <<< OID : jnxContentsType.7.2.1.0 Apr 12 06:40:03 snmpd[7ee783df] <<< type : Object Apr 12 06:40:03 snmpd[7ee783df] <<< value: jnxChassisTempSensor.0 Apr 12 06:40:03 snmpd[7ee783df] <<< Apr 12 06:40:03 snmpd[7ee783df] <<< OID : jnxContentsType.9.1.0.0 Apr 12 06:40:03 snmpd[7ee783df] <<< type : Object Apr 12 06:40:03 snmpd[7ee783df] <<< value: jnxM10iRE.0 Apr 12 06:40:03 snmpd[7ee783df] <<< Apr 12 06:40:03 snmpd[7ee783df] <<< OID : jnxContentsType.9.1.1.0 Apr 12 06:40:03 snmpd[7ee783df] <<< type : Object Apr 12 06:40:03 snmpd[7ee783df] <<< value: jnxPCMCIACard.0 Apr 12 06:40:03 snmpd[7ee783df] <<< Apr 12 06:40:03 snmpd[7ee783df] <<< OID : jnxContentsType.9.2.0.0 Apr 12 06:40:03 snmpd[7ee783df] <<< type : Object Apr 12 06:40:03 snmpd[7ee783df] <<< value: jnxM10iRE.0 Apr 12 06:40:03 snmpd[7ee783df] <<< Apr 12 06:40:03 snmpd[7ee783df] <<< OID : jnxContentsType.9.2.1.0 Apr 12 06:40:03 snmpd[7ee783df] <<< type : Object Apr 12 06:40:03 snmpd[7ee783df] <<< value: jnxPCMCIACard.0 Apr 12 06:40:03 snmpd[7ee783df] <<< Apr 12 06:40:03 snmpd[7ee783df] <<< OID : jnxContentsType.12.1.0.0 Apr 12 06:40:03 snmpd[7ee783df] <<< type : Object Apr 12 06:40:03 snmpd[7ee783df] <<< value: jnxM10iHCM.0 Apr 12 06:40:03 snmpd[7ee783df] <<< Apr 12 06:40:03 snmpd[7ee783df] <<< OID : jnxContentsType.12.2.0.0 Apr 12 06:40:03 snmpd[7ee783df] <<< type : Object Apr 12 06:40:03 snmpd[7ee783df] <<< value: jnxM10iHCM.0 Apr 12 06:40:03 snmpd[7ee783df] <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Apr 12 06:40:03 snmpd[7ee783e0] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 12 06:40:03 snmpd[7ee783e0] >>> Get-Bulk-Request Apr 12 06:40:03 snmpd[7ee783e0] >>> Source: 10.209.63.42 Apr 12 06:40:03 snmpd[7ee783e0] >>> Destination: 10.209.2.242 Apr 12 06:40:03 snmpd[7ee783e0] >>> Version: SNMPv2 Apr 12 06:40:03 snmpd[7ee783e0] >>> Request_id: 0x7ee783e0 Apr 12 06:40:03 snmpd[7ee783e0] >>> Community: public Apr 12 06:40:03 snmpd[7ee783e0] >>> Non-repeaters: 0 Apr 12 06:40:03 snmpd[7ee783e0] >>> Max-repetitions: 10 Apr 12 06:40:03 snmpd[7ee783e0] >>> OID : jnxContentsType.12.2.0.0 Apr 12 06:40:03 snmpd[7ee783e0] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> …… ……
Monitoring SNMP Statistics
The show snmp statistics extensive
operational mode
command provides you with an option to review SNMP traffic, including
traps, on a device. Output for the show snmp statistics extensive
command shows real-time values and can be used to monitor values
such as throttle drops, currently active, max active, not found, time
out, max latency, current queued, total queued, and overflows. You
can identify slowness in SNMP responses by monitoring the currently
active count, because a constant increase in the currently active
count is directly linked to slow or no response to SNMP requests.
Sample Output for the show snmp statistics extensive Command
user@host> show snmp statistics extensive SNMP statistics: Input: Packets: 226656, Bad versions: 0, Bad community names: 0, Bad community uses: 0, ASN parse errors: 0, Too bigs: 0, No such names: 0, Bad values: 0, Read onlys: 0, General errors: 0, Total request varbinds: 1967606, Total set varbinds: 0, Get requests: 18478, Get nexts: 75794, Set requests: 0, Get responses: 0, Traps: 0, Silent drops: 0, Proxy drops: 0, Commit pending drops: 0, Throttle drops: 27084, Duplicate request drops: 0 V3 Input: Unknown security models: 0, Invalid messages: 0 Unknown pdu handlers: 0, Unavailable contexts: 0 Unknown contexts: 0, Unsupported security levels: 0 Not in time windows: 0, Unknown user names: 0 Unknown engine ids: 0, Wrong digests: 0, Decryption errors: 0 Output: Packets: 226537, Too bigs: 0, No such names: 0, Bad values: 0, General errors: 0, Get requests: 0, Get nexts: 0, Set requests: 0, Get responses: 226155, Traps: 382 SA Control Blocks: Total: 222984, Currently Active: 501, Max Active: 501, Not found: 0, Timed Out: 0, Max Latency: 25 SA Registration: Registers: 0, Deregisters: 0, Removes: 0 Trap Queue Stats: Current queued: 0, Total queued: 0, Discards: 0, Overflows: 0 Trap Throttle Stats: Current throttled: 0, Throttles needed: 0 Snmp Set Stats: Commit pending failures: 0, Config lock failures: 0 Rpc failures: 0, Journal write failures: 0 Mgd connect failures: 0, General commit failures: 0
Checking CPU Utilization
High CPU usage of the software processes that are being queried,
such as snmpd or mib2d, is another factor that can lead to slow response
or no response. You can use the show system processes extensive
operational mode command to check the CPU usage levels of the Junos
OS processes.
Sample Output of show system processes extensive Command
user@host> show system processes extensive last pid: 1415; load averages: 0.00, 0.00, 0.00 up 0+02:20:54 10:26:25 117 processes: 2 running, 98 sleeping, 17 waiting Mem: 180M Active, 54M Inact, 39M Wired, 195M Cache, 69M Buf, 272M Free Swap: 1536M Total, 1536M Free PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 11 root 1 171 52 0K 12K RUN 132:09 95.21% idle 1184 root 1 97 0 35580K 9324K select 4:16 1.61% chassisd 177 root 1 -8 0 0K 12K mdwait 0:51 0.00% md7 119 root 1 -8 0 0K 12K mdwait 0:20 0.00% md4 13 root 1 -20 -139 0K 12K WAIT 0:16 0.00% swi7: clock sio 1373 root 1 96 0 15008K 12712K select 0:09 0.00% snmpd 1371 root 1 96 0 9520K 5032K select 0:08 0.00% jdiameterd 12 root 1 -40 -159 0K 12K WAIT 0:07 0.00% swi2: net 1375 root 2 96 0 15016K 5812K select 0:06 0.00% pfed 49 root 1 -8 0 0K 12K mdwait 0:05 0.00% md0 1345 root 1 96 0 10088K 4480K select 0:05 0.00% l2ald 1181 root 1 96 0 1608K 908K select 0:05 0.00% bslockd 23 root 1 -68 -187 0K 12K WAIT 0:04 0.00% irq10: fxp1 30 root 1 171 52 0K 12K pgzero 0:04 0.00% pagezero 1344 root 1 4 0 39704K 11444K kqread 0:03 0.00% rpd 1205 root 1 96 0 3152K 912K select 0:03 0.00% license-check 1372 root 1 96 0 28364K 6696K select 0:03 0.00% dcd 1374 root 1 96 0 11764K 7632K select 0:02 0.00% mib2d 1405 user 1 96 0 15892K 11132K select 0:02 0.00% cli 139 root 1 -8 0 0K 12K mdwait 0:02 0.00% md5 22 root 1 -80 -199 0K 12K WAIT 0:02 0.00% irq9: cbb1 fxp0 1185 root 1 96 0 4472K 2036K select 0:02 0.00% alarmd 4 root 1 -8 0 0K 12K - 0:02 0.00% g_down 3 root 1 -8 0 0K 12K - 0:02 0.00% g_up 43 root 1 -16 0 0K 12K psleep 0:02 0.00% vmkmemdaemon 1377 root 1 96 0 3776K 2256K select 0:01 0.00% irsd 48 root 1 -16 0 0K 12K - 0:01 0.00% schedcpu 99 root 1 -8 0 0K 12K mdwait 0:01 0.00% md3 953 root 1 96 0 4168K 2428K select 0:01 0.00% eventd 1364 root 1 96 0 4872K 2808K select 0:01 0.00% cfmd 15 root 1 -16 0 0K 12K - 0:01 0.00% yarrow 1350 root 1 96 0 31580K 7248K select 0:01 0.00% cosd 1378 root 1 96 0 19776K 6292K select 0:01 0.00% lpdfd ...
Checking Kernel and Packet Forwarding Engine Response
As mentioned in Understanding SNMP Implementation in Junos OS, some SNMP MIB data are maintained by the kernel or Packet Forwarding
Engine. For such data to be available for the network management system,
the kernel has to provide the required information to the SNMP subagent
in mib2d. A slow response from the kernel can cause a delay in mib2d
returning the data to the network management system. Junos OS adds
an entry in the mib2d log file every time that an interface takes
more than 10,000 microseconds to respond to a request for interface
statistics. You can use the show log log-filename | grep “kernel response time”
command to find
out the response time taken by the kernel.
Checking the Kernel Response Time
user@host> show log mib2d | grep “kernel response time” Aug 17 22:39:37 == kernel response time for COS_IPVPN_DEFAULT_OUTPUT-t1-7/3/0:10:27.0-o: 9.126471 sec, range (0.000007, 11.000806) Aug 17 22:39:53 == kernel response time for COS_IPVPN_DEFAULT_INPUT-t1-7/2/0:5:15.0-i: 5.387321 sec, range (0.000007, 11.000806) Aug 17 22:39:53 == kernel response time for ct1-6/1/0:9:15: 0.695406 sec, range (0.000007, 11.000806) Aug 17 22:40:04 == kernel response time for t1-6/3/0:6:19: 1.878542 sec, range (0.000007, 11.000806) Aug 17 22:40:22 == kernel response time for lsq-7/0/0: 2.556592 sec, range (0.000007, 11.000806)
Tracing SNMP Activity on a Device Running Junos OS
SNMP tracing operations track activity for SNMP agents and record the information in log files. The logged error descriptions provide detailed information to help you solve problems faster.
By default, Junos OS does not trace any SNMP activity. If you
include the traceoptions
statement at the [edit snmp]
hierarchy level, the default tracing behavior is:
Important activities are logged in files located in the /var/log directory. Each log is named after the SNMP agent that generates it. Currently, the following log files are created in the /var/log directory when the
traceoptions
statement is used:chassisd
craftd
ilmid
mib2d
rmopd
serviced
snmpd
When a trace file named filename reaches its maximum size, it is renamed filename.0, then filename.1, and so on, until the maximum number of trace files is reached. Then the oldest trace file is overwritten. (For more information about how log files are created, see the System Log Explorer.)
Log files can be accessed only by the user who configured the tracing operation.
You cannot change the directory (/var/log) in which trace files are located. However, you can customize the
other trace file settings by including the following statements at
the [edit snmp]
hierarchy level:
[edit snmp] traceoptions { file <files number> <match regular-expression> <size size> <world-readable | no-world-readable>; flag flag; memory-trace; no-remote-trace; no-default-memory-trace; }
These statements are described in the following sections:
- Configuring the Number and Size of SNMP Log Files
- Configuring Access to the Log File
- Configuring a Regular Expression for Lines to Be Logged
- Configuring the Trace Operations
Configuring the Number and Size of SNMP Log Files
By default, when the trace file reaches 128 kilobytes (KB) in size, it is renamed filename.0, then filename.1, and so on, until there are three trace files. Then the oldest trace file (filename.2) is overwritten.
You can configure the limits on the number and size of
trace files by including the following statements at the [edit
snmp traceoptions]
hierarchy level:
[edit snmp traceoptions] file files number size size;
For example, set the maximum file size to 2 MB, and the maximum number of files to 20. When the file that receives the output of the tracing operation (filename) reaches 2 MB, filename is renamed filename.0, and a new file called filename is created. When the new filename reaches 2 MB, filename.0 is renamed filename.1 and filename is renamed filename.0. This process repeats until there are 20 trace files. Then the oldest file (filename.19) is overwritten by the newest file (filename.0).
The number of files can be from 2 through 1000 files. The file size of each file can be from 10 KB through 1 gigabyte (GB).
Configuring Access to the Log File
By default, log files can be accessed only by the user who configured the tracing operation.
To specify that any user can read all log files, include
the file world-readable
statement at the [edit snmp
traceoptions]
hierarchy level:
[edit snmp traceoptions] file world-readable;
To explicitly set the default behavior, include the file no-world-readable
statement at the [edit snmp traceoptions]
hierarchy level:
[edit snmp traceoptions] file no-world-readable;
Configuring a Regular Expression for Lines to Be Logged
By default, the trace operation output includes all lines relevant to the logged activities.
You can refine the output by including the match
statement at the [edit snmp traceoptions file filename]
hierarchy level and specifying a regular expression (regex)
to be matched:
[edit snmp traceoptions] file filename match regular-expression;
Configuring the Trace Operations
By default, only important activities are logged. You
can specify which trace operations are to be logged by including the
following flag
statement (with one or more tracing flags)
at the [edit snmp traceoptions]
hierarchy level:
[edit snmp traceoptions] flag { all; configuration; database; events; general; interface-stats; nonvolatile-sets; pdu; policy; protocol-timeouts; routing-socket; server; subagent; timer; varbind-error; }
Table 1 describes the meaning of the SNMP tracing flags.
Flag |
Description |
Default Setting |
---|---|---|
|
Log all operations. |
Off |
|
Log reading of the configuration at the |
Off |
|
Log events involving storage and retrieval in the events database. |
Off |
|
Log important events. |
Off |
|
Log general events. |
Off |
|
Log physical and logical interface statistics. |
Off |
|
Log nonvolatile SNMP set request handling. |
Off |
|
Log SNMP request and response packets. |
Off |
|
Log policy processing. |
Off |
|
Log SNMP response timeouts. |
Off |
|
Log routing socket calls. |
Off |
|
Log communication with processes that are generating events. |
Off |
|
Log subagent restarts. |
Off |
|
Log internal timer events. |
Off |
|
Log variable binding errors. |
Off |
To display the end of the log for an agent, issue the show log agentd | last
operational mode
command:
[edit] user@host# run show log agentd | last
where agent
is the name of an
SNMP agent.
Example: Tracing SNMP Activity
Trace information about SNMP packets:
[edit] snmp { traceoptions { file size 10k files 5; flag pdu; flag protocol-timeouts; flag varbind-error; } }
Configure the Certificate Expiration Trap
Before you begin:
Understand how certificates works on VPN. Read Understanding Certificate Chains.
This topic shows how to configure certificate expiration trap and configure the number of days before to generate the trap.
See Also
Enable Peer Down and IPsec Tunnel Down Traps
This topic shows how to enable peer-down
and ipsec-tunnel-down
traps.