Define Junos PyEZ Operational Tables for Parsing Unstructured Output
Create custom Tables that select data from CLI or vty command output to extract operational information from a Junos device.
Junos PyEZ operational (op) Tables for unstructured output extract data from the text output of a CLI command executed on a Junos device or a vty command executed on a given Flexible PIC Concentrator (FPC). The extracted data is then converted to JSON. This enables you to quickly retrieve and analyze operational state information for the device. Junos PyEZ op Tables for unstructured output are particularly useful when you need to parse command output that cannot be returned in a structured format such as XML.
This topic discusses the different components of the Table.
Summary of Parameters in Op Tables for Parsing Unstructured Output
Junos PyEZ Tables are formatted using YAML. Op Table definitions can include a number of required and optional parameters, which are summarized in Table 1.
Table Parameter Name |
Table Parameter |
Description |
|---|---|---|
Table name |
– |
User-defined identifier for the Table. |
Command |
|
CLI or vty command to execute. |
Command arguments |
|
(Optional) When you define the command as a Jinja template, |
Target FPC |
|
(Optional) Flexible PIC Concentrator (FPC) on which to execute a vty command. |
Table item |
|
(Optional) String or regular expression that defines how to split the output into sections. These sections become the iterable reference for the associated View. Specify '*' to extract and match against the whole string rather than each line. |
Table item key |
|
(Optional) String or list of strings that define one or more keys that uniquely identify each Table item. |
Selected keys |
|
(Optional) List of one or more Table item keys for which to return data. |
Section title |
|
(Optional) String that selects the section of output containing the data to parse. |
Field delimiter |
|
(Optional) Delimiter that defines how to split the data in command output comprised of key-value pairs. The extracted data is stored as key-value pairs in a dictionary. |
Eval expression |
|
(Optional) Associative array, or dictionary, of one or
more key-value pairs that map a user-defined key to a string containing
a mathematical expression. When you retrieve the data, the expression
is evaluated using the Python |
Table View |
|
(Optional) View that is used to extract field data from the Table items. |
TextFSM template |
|
(Optional) Boolean that specifies whether a TextFSM template is used to parse the data. |
TextFSM template platform identifier |
|
(Optional) When a TextFSM template is used, specify the platform for the template. |
Table Name
The Table name is a user-defined identifier for the Table.
The YAML file or string can contain one or more Tables. The start
of the YAML document must be left justified. The following example
defines a Table named ChassisFanTable:
--- ChassisFanTable: command: show chassis fan key: fan-name view: ChassisFanView
Command
A Junos PyEZ op Table for unstructured output extracts data
from the text output of a CLI or vty command. You must include the command property in the Table definition to specify
the CLI command to execute on a device or the vty command to execute
on a given FPC. You can define the command as a simple string or a
Jinja template.
For example, the following Table executes the show
chassis fan command on the device.
--- ChassisFanTable: command: show chassis fan key: fan-name view: ChassisFanView
The following Table executes the show cmerror module
brief vty command on the target FPC.
--- CMErrorTable: command: show cmerror module brief target: fpc1 key: module view: CMErrorView
When you define the command as a Jinja template, you must also
supply the args parameter with a dictionary
of key-value pairs that map the variables in the template to the values
used when the template is rendered. For information about defining
the command as a Jinja template, see Command Arguments (args).
Command Arguments (args)
You can define the CLI or vty command for the command parameter using a Jinja template and substitute variables for the
command arguments. When you use a Jinja template, you must also define
the args parameter, which is a dictionary
of key-value pairs that map the variable names in the Jinja template
to the values used when the template is rendered. You can provide
default values for the template variables in the Table definition,
and you can also define the values in the Junos PyEZ application.
To define default values for the template variables,
include the args parameter in the Table
definition and map each template variable to its default value. The
following Table defines a command using a Jinja template that has
one variable, protocol. The args parameter defines the default value for protocol, which is used when you call the get() method in your script and do not provide an argument
that overrides that default.
---
DdosPolicerStatsTable:
command: show ddos policer stats {{ protocol }}
args:
protocol: ospf
target: Null
title: "DDOS Policer Statistics:"
key: location
view: DdosPolicerStatsViewAdditionally, you can define the args argument as a dictionary in the Table’s get() method to:
define a value for any template variable that does not have a default value defined in the Table
override the default value defined in the Table for one or more template variables
The following example executes the command in the previous Table using protocol 'bgp' instead of the default value of 'ospf'.
from jnpr.junos import Device from jnpr.junos.command.pfe_ddos_policer import DdosPolicerStatsTable from pprint import pprint import json with Device(host='router1.example.com') as dev: stats = DdosPolicerStatsTable(dev) stats.get(target='fpc0', args={'protocol':'bgp'}) pprint(json.loads(stats.to_json()))
Target FPC (fpc)
Junos PyEZ op Tables can execute vty commands on a specific
Flexible PIC Concentrator (FPC). When you use a vty command, the Table
must include the target parameter to define
the target FPC. You can set target to Null and force the user to specify the target FPC in
the application, or you can set target to
a default FPC, and the user can optionally override this value in
the application.
The following Table executes the show memory vty command, but sets target: Null, which
requires that the user supply the target FPC in the Junos PyEZ application:
---
FpcMemory:
command: show memory
target: Null
key: ID
key_items:
- 0
- 1
view: FPCMemoryViewThe following Table executes the show memory vty command on FPC 1, unless the user overrides this value in the
Junos PyEZ application.
---
FpcMemory:
command: show memory
target: fpc1
key: ID
key_items:
- 0
- 1
view: FPCMemoryViewIn the Junos PyEZ application, to define the target FPC
or override the default FPC defined in the Table, set the target argument in the Table’s get() method to the desired FPC, for example:
from jnpr.junos import Device from jnpr.junos.command.fpc_memory import FpcMemory from pprint import pprint import json with Device(host='router.example.com') as dev: stats = FpcMemory(dev) stats.get(target='fpc0') pprint(json.loads(stats.to_json()))
Table Item (item)
The optional Table item property
is a string or regular expression that defines how to split the command
output for parsing. If the output has similar, repeating sets of data,
you can define item to match on and extract
each iteration of the data. For example, show interfaces returns a similar set of data for many interfaces. Alternatively,
you can define item: '*' when you need
to extract the data as a single block of text.
Consider the following output for the show devices
local vty command:
TSEC Ethernet Device Driver: .le1, Control 0x4296c218, (1000Mbit)
HW reg base 0xff724000
[0]: TxBD base 0x7853ce20, RxBD Base 0x7853d640
[1]: TxBD base 0x7853d860, RxBD Base 0x7853e080
[2]: TxBD base 0x7853e2a0, RxBD Base 0x785422c0
[3]: TxBD base 0x785426e0, RxBD Base 0x78544700
Receive:
185584608 bytes, 2250212 packets, 0 FCS errors, 0 multicast packets
107271 broadcast packets, 0 control frame packets
0 PAUSE frame packets, 0 unknown OP codes
0 alignment errors, 0 frame length errors
0 code errors, 0 carrier sense errors
0 undersize packets, 0 oversize packets
0 fragments, 0 jabbers, 0 drops
Receive per queue:
[0]: 0 bytes, 0 packets, 0 dropped
0 jumbo, 0 truncated jumbo
[1]: 0 bytes, 0 packets, 0 dropped
0 jumbo, 0 truncated jumbo
[2]: 0 bytes, 0 packets, 0 dropped
0 jumbo, 0 truncated jumbo
[3]: 203586808 bytes, 2250219 packets, 0 dropped
0 jumbo, 0 truncated jumbo
Transmit:
288184646 bytes, 2038370 packets, 0 multicast packets
106531 broadcast packets, 0 PAUSE control frames
0 deferral packets, 0 excessive deferral packets
0 single collision packets, 0 multiple collision packets
0 late collision packets, 0 excessive collision packets
0 total collisions, 0 drop frames, 0 jabber frames
0 FCS errors, 0 control frames, 0 oversize frames
0 undersize frames, 0 fragments frames
Transmit per queue:
[0]: 10300254 bytes, 72537 packets
0 dropped, 0 fifo errors
[1]: 4474302 bytes, 106531 packets
0 dropped, 0 fifo errors
[2]: 260203538 bytes, 1857137 packets
0 dropped, 0 fifo errors
[3]: 199334 bytes, 2179 packets
0 dropped, 0 fifo errors
TSEC status counters:
kernel_dropped:0, rx_large:0 rx_short: 0
rx_nonoctet: 0, rx_crcerr: 0, rx_overrun: 0
rx_bsy: 0,rx_babr:0, rx_trunc: 0
rx_length_errors: 0, rx_frame_errors: 0 rx_crc_errors: 0
rx_errors: 0, rx_ints: 2250110, collisions: 0
eberr:0, tx_babt: 0, tx_underrun: 0
tx_timeout: 0, tx_timeout: 0,tx_window_errors: 0
tx_aborted_errors: 0, tx_ints: 2038385, resets: 1
TSEC Ethernet Device Driver: .le3, Control 0x42979220, (1000Mbit)
HW reg base 0xff726000
[0]: TxBD base 0x78545720, RxBD Base 0x78545f40
[1]: TxBD base 0x78546160, RxBD Base 0x78546980
[2]: TxBD base 0x78546ba0, RxBD Base 0x7854abc0
[3]: TxBD base 0x7854afe0, RxBD Base 0x7854d000
Receive:
0 bytes, 0 packets, 0 FCS errors, 0 multicast packets
0 broadcast packets, 0 control frame packets
0 PAUSE frame packets, 0 unknown OP codes
0 alignment errors, 0 frame length errors
0 code errors, 0 carrier sense errors
0 undersize packets, 0 oversize packets
0 fragments, 0 jabbers, 0 drops
Receive per queue:
[0]: 0 bytes, 0 packets, 0 dropped
0 jumbo, 0 truncated jumbo
[1]: 0 bytes, 0 packets, 0 dropped
0 jumbo, 0 truncated jumbo
[2]: 0 bytes, 0 packets, 0 dropped
0 jumbo, 0 truncated jumbo
[3]: 0 bytes, 0 packets, 0 dropped
0 jumbo, 0 truncated jumbo
Transmit:
6817984 bytes, 106531 packets, 0 multicast packets
106531 broadcast packets, 0 PAUSE control frames
0 deferral packets, 0 excessive deferral packets
0 single collision packets, 0 multiple collision packets
0 late collision packets, 0 excessive collision packets
0 total collisions, 0 drop frames, 0 jabber frames
0 FCS errors, 0 control frames, 0 oversize frames
0 undersize frames, 0 fragments frames
Transmit per queue:
[0]: 0 bytes, 0 packets
0 dropped, 0 fifo errors
[1]: 4474302 bytes, 106531 packets
0 dropped, 0 fifo errors
[2]: 0 bytes, 0 packets
0 dropped, 0 fifo errors
[3]: 0 bytes, 0 packets
0 dropped, 0 fifo errors
TSEC status counters:
kernel_dropped:0, rx_large:0 rx_short: 0
rx_nonoctet: 0, rx_crcerr: 0, rx_overrun: 0
rx_bsy: 0,rx_babr:0, rx_trunc: 0
rx_length_errors: 0, rx_frame_errors: 0 rx_crc_errors: 0
rx_errors: 0, rx_ints: 0, collisions: 0
eberr:0, tx_babt: 0, tx_underrun: 0
tx_timeout: 0, tx_timeout: 0,tx_window_errors: 0
tx_aborted_errors: 0, tx_ints: 106531, resets: 1
The following Table extracts each section of the output
that starts with TSEC Ethernet Device Driver:. In this case, the value for key: name is derived from the capturing group in the regular expression defined
in item.
---
DevicesLocalTable:
command: show devices local
target: fpc1
item: 'TSEC Ethernet Device Driver: (\.?\w+),'
key: name
view: DevicesLocalView
DevicesLocalView:
fields:
TSEC_status_counters: _TSECStatusCountersTable
receive_counters: _ReceiveTable
transmit_per_queue: _TransmitQueueTable
You can also define item as an asterisk
('*') if you want to match against the entire section of output instead
of matching each line. When you include item: '*', in most cases, you must also include the title parameter to specify the heading for the section of output to extract.
Extracting the output using item: '*' is
useful when you include the regex parameter
in a View and want each expression to match against the entire text
string. Otherwise, the regex expressions
are combined and matched against each line.
The following Table extracts the text block under the
heading Receive: and matches each regular
expression against the entire text string:
_ReceiveTable:
item: '*'
title: 'Receive:'
view: _ReceiveView
_ReceiveView:
regex:
bytes: '(\d+) bytes'
packets: '(\d+) packets'
FCS_errors: '(\d+) FCS errors'
broadcast_packets: '(\d+) broadcast packets'Table Item Key (key)
The optional key property defines
the output fields that are used to uniquely identify a Table item.
You can identify a Table item using a single key or a list of keys.
If the Table and View return iterative data, the key value must reference a variable or field name defined in the View.
Consider the following show chassis fan output:
Item Status RPM Measurement
Fan 1 OK 5280 Spinning at normal speed
Fan 2 OK 5280 Spinning at normal speed
Fan 3 OK 5280 Spinning at normal speed
Fan 4 OK 5280 Spinning at normal speed
Fan 5 OK 5280 Spinning at normal speed
The following Table defines the Table item key as fan-name, which maps to the value under the Item column in the output.
ChassisFanTable:
command: show chassis fan
key: fan-name
view: ChassisFanView
ChassisFanView:
columns:
fan-name: Item
fan-status: Status
fan-rpm: RPM
fan-measurement: MeasurementWhen you retrieve and print the data in the Junos PyEZ application, each item in the resulting dictionary uses this field’s value as its key.
dict_keys(['Fan 1', 'Fan 2', 'Fan 3', 'Fan 4', 'Fan 5'])
You can also define key as
a list to identify a Table item using a composite key. For example:
---
FPCIPV4AddressTable:
command: show ipv4 address
target: fpc1
key:
- name
- addr
view: FPCIPV4AddressView
FPCIPV4AddressView:
columns:
index: Index
addr: Address
name: Name
Selected Keys (key_items)
The key parameter defines the output
fields that uniquely identify a Table item. When you include the key parameter in a Table, you can use the optional key_items parameter to only return data for certain
key values. key_items defines the key or
list of keys of the Table items for which to retrieve data. You can
define the key_items parameter in the Table
definition or in the Junos PyEZ application.
Consider the following show chassis fan output:
Item Status RPM Measurement
Fan 1 OK 5280 Spinning at normal speed
Fan 2 OK 5280 Spinning at normal speed
Fan 3 OK 5280 Spinning at normal speed
Fan 4 OK 5280 Spinning at normal speed
Fan 5 OK 5280 Spinning at normal speed
The following Table defines the Table item key as fan-name and only retrieves the data for the Table
item with a key value equal to Fan 1.
ChassisFanTable:
command: show chassis fan
key: fan-name
key_items:
- Fan 1
view: ChassisFanView
ChassisFanView:
columns:
fan-name: Item
fan-status: Status
fan-rpm: RPM
fan-measurement: MeasurementIn the Junos PyEZ application, to define the key_items to return or to override the key_items defined in the Table, set the key_items argument in the Table’s get() method to a list or tuple of the desired items,
for example:
from jnpr.junos import Device from jnpr.junos.command.chassis_fan import ChassisFanTable from pprint import pprint import json with Device(host='router.example.com') as dev: fans = ChassisFanTable(dev) fans.get(key_items=['Fan 1', 'Fan 2']) pprint(json.loads(fans.to_json()))
user@host:~$ python3 junos-pyez-get-fan-data.py
{'Fan 1': {'fan-measurement': 'Spinning at normal speed',
'fan-name': 'Fan 1',
'fan-rpm': 5280,
'fan-status': 'OK'},
'Fan 2': {'fan-measurement': 'Spinning at normal speed',
'fan-name': 'Fan 2',
'fan-rpm': 5280,
'fan-status': 'OK'}}
Section Title (title)
Tables can include the optional title parameter to define the starting point for a section in the command
output from which to extract and parse the data. When the Table defines item: '*', you must include title to specify the heading for the section of output to extract
For example, consider the following command output, which is enclosed within a larger set of output:
... TSEC status counters: kernel_dropped:0, rx_large:0 rx_short: 0 rx_nonoctet: 0, rx_crcerr: 0, rx_overrun: 0 rx_bsy: 0,rx_babr:0, rx_trunc: 0 rx_length_errors: 0, rx_frame_errors: 0 rx_crc_errors: 0 rx_errors: 0, rx_ints: 2250110, collisions: 0 eberr:0, tx_babt: 0, tx_underrun: 0 tx_timeout: 0, tx_timeout: 0,tx_window_errors: 0 tx_aborted_errors: 0, tx_ints: 2038385, resets: 1 ...
The following Table uses the title parameter to extract and parse data from the TSEC
status counters section of the output. In this case,
the Table defines item as '*', which considers
the data as a single text string.
_TSECStatusCountersTable:
item: '*'
title: 'TSEC status counters:'
view: _TSECStatusCountersView
_TSECStatusCountersView:
regex:
kernel_dropped: 'kernel_dropped:(\d+)'
rx_large: 'rx_large:(\d+)'
Field Delimiter (delimiter)
Some operational commands return output comprised of only key-value
pairs. If you want to simply retrieve the entire set of data and extract
each key-value pair, you can use the delimiter parameter to define how to split each data pair instead of defining
a separate View. Junos PyEZ uses the delimiter to split the data at
the specified boundary and stores each key-value pair in a dictionary.
Consider the following output for the show
link stats vty command.
PPP LCP/NCP: 0 HDLC keepalives: 0 RSVP: 0 ISIS: 0 OSPF Hello: 539156 OAM: 0 BFD: 15 UBFD: 0 LMI: 0 LACP: 0 ETHOAM: 0 SYNCE: 0 PTP: 0 L2TP: 0 LNS-PPP: 0 ARP: 4292 ELMI: 0 VXLAN MRESOLVE: 0 Unknown protocol: 42
The following Table defines the delimiter as the colon (:) character:
--- FPCLinkStatTable: command: show link stats target: fpc1 delimiter: ":"
When you retrieve the data in the Junos PyEZ application, the Table splits each line of output at the delimiter and stores the key-value pairs in a dictionary.
{'ARP': 4292, 'ELMI': 0, 'SYNCE': 0, 'ISIS': 0, 'BFD': 15, 'PPP LCP/NCP': 0,
'OAM': 0, 'ETHOAM': 0, 'LACP': 0, 'LMI': 0, 'Unknown protocol': 42,
'UBFD': 0, 'L2TP': 0, 'HDLC keepalives': 0, 'LNS-PPP': 0,
'OSPF Hello': 539156, 'RSVP': 0, 'VXLAN MRESOLVE': 0, 'PTP': 0}
Eval Expression (eval)
You can use the optional eval parameter
to add or modify key-value pairs in the final data returned by the
Table and View. The eval parameter maps
a key name to a string containing a mathematical expression that gets
evaluated by the Python eval function.
You can include the eval parameter in both
Tables and Views, and eval can define and
calculate multiple values.
When you use eval in a Table, it
references the full data dictionary for the calculation, and the key
and calculated value are added as a single additional item to the
dictionary. When you use eval in a View,
the expression is calculated on each iteration of the data, and the
calculated value is added to the data for that iteration. If the eval key name matches a key defined in the View, eval replaces the value of that key with the calculated
value. If the eval key name does not match
a key defined in the View, eval adds the
new key and calculated value to the data.
The eval expression can reference
the data dictionary returned by the View.
The expression must reference data using
a Jinja template variable, so that Junos PyEZ can replace the variable
with the dictionary when the expression is evaluated.
The following example uses eval in the Table definition to include a single additional entry in
the data dictionary. The added item’s key is cchip_errors_from_lkup_chip, and its value is the sum of the number of interrupts.
---
CChipLiInterruptStatsTable:
command: show xmchip {{ chip_instance }} li interrupt-stats
target: NULL
args:
chip_instance: 0
key:
- li_block
- name
eval:
cchip_errors_from_lkup_chip: "reduce(lambda x,y: x+y, [v['interrupts'] for k,v in {{ data }}.items()])"
view: CChipLiInterruptStatsView
CChipLiInterruptStatsView:
columns:
li_block: LI Block
name: Interrupt Name
interrupts: Number of Interrupts
last_occurance: Last Occurrence
You can also define eval in the Table
to calculate and add multiple key-values pairs, for example:
---
CChipDRDErrTable:
command: show xmchip {{ instance }} drd error-stats
args:
instance: 0
target: NULL
key: Interrupt Name
item: '*'
eval:
cchip_drd_wan_errors: sum([v['interrupt_count'] for k, v in {{ data }}.items() if k.endswith('_0')])
cchip_drd_fab_errors: sum([v['interrupt_count'] for k, v in {{ data }}.items() if k.endswith('_1')])
view: CChipDRDErrView
CChipDRDErrView:
regex:
cchip_drd_wan_timeouts: 'Total WAN reorder ID timeout errors:\s+(\d+)'
cchip_drd_fab_timeouts: 'Total fabric reorder ID timeout errors:\s+(\d+)'
columns:
interrupt_name: Interrupt Name
interrupt_count: Number of Interrupts
filters:
- interrupt_count
For information about using eval in
a View, see Eval Expression (eval).
Table View (view)
The view property associates the
Table definition with a particular View. A View defines how the Table
output should be parsed and maps your user-defined Python variable
names to output fields in the selected Table items. You can customize
the View to only select certain fields from the Table items.
If the output consists of only key-value pairs, you can use
the Table’s delimiter parameter to
extract the data and store the key-value pairs in a dictionary. In
this case, you do not need to define a separate View.
TextFSM Templates (platform and use_textfsm)
Junos PyEZ Tables can reference a TextFSM template to parse
command output from Juniper Networks devices or other vendors’
devices. You must install the ntc-templates library on your Junos PyEZ server or virtual environment
in order to use TextFSM templates in your Tables.
To use a TextFSM template to parse the command output, include use_textfsm: True in the Table. You can use a TextFSM
template by itself or in conjunction with a Junos PyEZ View. Junos
PyEZ uses the platform and command values to determine the template’s filename.
For example, the following Table uses the juniper_junos_show_arp_no-resolve.textfsm template to parse command output from a Juniper Networks Junos device:
--- ArpTableTextFSM: command: show arp no-resolve platform: juniper_junos key: MAC use_textfsm: True
For detailed information about using Junos PyEZ Tables with TextFSM templates, see Use Junos PyEZ Tables with TextFSM Templates.