ON THIS PAGE
Cisco Device Agent
Cisco NX-OS Device Agent Overview
Although the preferred method of installing device system agents is from the Apstra GUI, you can manually install Apstra agents from the CLI. Only in rare exceptions would you need to manually install agents, which requires more effort and is error-prone. Before manually installing agents, you should have an in-depth understanding of the various device states, configuration stages, and agent operations . For assistance, contact Juniper Support.
You can also use Apstra ZTP to automatically boot and install agents and prerequisite configuration on switches. Using Apstra ZTP is simpler and easier to support at scale than manually installing agents.
Manually installing an agent for Cisco devices involves the following steps:
- Update the guestshell disk size, memory and cpu, then enable/reboot the guestshell.
- Install the device agent.
- Update the aos.config file.
- Start service.
The Cisco GuestShell is not partitioned to be unique with Apstra. If there are other applications hosting on the guestshell, any changes in the guestshell could impact them.
Commands in the "Bootstrap" or "Pristine" configuration may interfere with Apstra configuration added during fabric deployment.
If you configure NX-OS "system jumbomtu" with a value lower than the MTUs that Apstra uses, then Apstra MTU commands will fail.
Device Configuration Requirements
Configure the device in the following order: VRF, NXAPI, GuestShell, Create
Management VRF. To allow for agent-server communication Apstra's device agent uses
the VRF name management
. Ensure these lines appear in the running
configuration.
! no password strength-check username admin password admin-password role network-admin copp profile strict ! vrf context management ip route 0.0.0.0/0 <Management Default Gateway> ! interface mgmt0 vrf member management ip address <Management CIDR Address> !
Resize and Enable Guestshell
- Run the following commands to resize the disk space, memory and
CPU:
guestshell resize rootfs 1024 guestshell resize memory 2048 guestshell resize cpu 6
- If the guestshell is not enabled, run the command
guestshell enable
to activate the changes. - If the guestshell was already enabled, run the command
guestshell reboot
to restart the shell and activate the changes. - Run the command
switch# show guestshell detail
and verify that the guestshell has been activated.
Download Agent Installer
You can easily copy the installation agents over HTTPS from the Apstra server. After downloading, confirm the MD5sum of your downloaded copy matches what Apstra stores.
To retrieve the agent file, the Cisco device connects to the Apstra server using HTTPS. Before proceeding, make sure this connectivity is functioning.
Apstra ships with the agent from the Apstra Server. We can copy it to the
/volatile
, or volatile:
filesystem location.
Apstra also ships with an md5sum file in the /home/admin
folder on
the Apstra Server.
Replace the aos_server_ip
variable and aos_version
from the run file below. For the Apstra server version (4.1.0-115 for example),
navigate to Platform > About).
switch# guestshell run sudo chvrf management wget --no-check-certificate -o /volatile/aos_download.log -O /volatile/aos.run https://<aos_server_ip>/device_agent_images/aos_device_agent_<aos_version>.run guestshell run sudo chvrf management wget --no-check-certificate -o /volatile/aos_download.log -O /volatile/aos.run.md5 https://<aos_server_ip>/device_agent_images/aos_device_agent_<aos_version>.run.md5
Validate that the file was downloaded correctly.
switch# show file volatile:aos.run md5 a28780880a8d674f6eb6a397509db101 switch# show file volatile:aos.run.md5 a28780880a8d674f6eb6a397509db101 aos_device_agent_<aos_version>.run
Install Cisco Device Agent
We recommend that you run the command copy running-config
startup-config
to save your latest changes, in case any issues
arise.
From the Cisco NX-OS switch guestshell, run the command to install the agent as shown below:
switch# guestshell run sudo chmod +x /volatile/aos.run switch# guestshell run sudo /volatile/aos.run -- --no-start <omitted output> created 7855 files created 1386 directories created 602 symlinks created 0 devices created 0 fifos + [[ True == \T\r\u\e ]] + true + systemctl enable aos
Update Agent Config File and Start Service
After installing the agent and before starting service, update the
aos.conf
file so it will connect to the server.
Configure the Cisco NX-OS device agent configuration file located at
/etc/aos/aos.conf
. See Apstra device agent
configuration file for parameters.
After updating the file, run the command service aos start
to start
the Apstra device agent.
Activate Apstra Devices on Apstra Server
When the Apstra device agent communicates with Apstra, it uses a ‘device key’ to identify itself. For Cisco NXOS switches, the device key is the MAC address of the management interface ‘eth0’.
root@Cisco:/etc/aos# ip link show dev eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether 08:00:27:8a:39:05 brd ff:ff:ff:ff:ff:ff
Deploy Device
From the left navigation menu of the Apstra GUI, navigate to Devices > Managed Devices. When the agent is up and running it appears in this list, and can be acknowledged and assigned to a blueprint using the GUI per standard procedure.
Reset Apstra Device Agent
If you need to reset the Apstra agent for some reason (changing blueprints, redeploying, restoring device from backup, etc.) it's best to clear the Apstra agent metadata, re-register the device, and redeploy to the blueprint.
C9K-172-20-65-5# guestshell [guestshell@guestshell ~]$ sudo su - [root@guestshell ~]# systemctl stop aos [root@guestshell ~]# rm -rf /var/log/aos/* [root@guestshell ~]# systemctl start aos Starting AOS Agents...root@guestshell ~]#
Uninstall Apstra Device Agent
To uninstall the agent, first undeploy and unassign it from the blueprint per standard procedures using the GUI. You can also delete it entirely from the Managed Devices page.
To remove the Apstra package from NX-OS, destroy the guestshell. Do this only if no other applications are using the guestshell:
C9K-172-20-65-5# guestshell destroy Remove remaining AOS data from system Removing the guest-shell deletes most of the data left by AOS. Some files are still on the bootflash:/.aos folder. C9K-172-20-65-5# delete bootflash:.aos no-prompt
Remove Apstra EEM Scripts
The Apstra device agent installs some event manager applets to assist with telemetry. These can be safely removed
C9K-172-20-65-5(config)# no event manager applet AOS_PROTO_VSH_LAUNCH C9K-172-20-65-5(config)# no event manager applet AOS_STATS_VSH_LAUNCH C9K-172-20-65-5(config)# no event manager applet aos_bgp_applet C9K-172-20-65-5(config)# no event manager applet aos_ifdown_applet C9K-172-20-65-5(config)# no event manager applet aos_ifup_applet
Cisco Agent Troubleshooting
The Apstra agent runs under the NXOS guestshell to interact with the underlying bash and Linux environments. This is an internal Linux Container (LXC) in which Apstra operates. Under LXC, Apstra makes use of the NXAPI and other methods to directly communicate with NXOS. For security reasons, Cisco partitions much of the LXC interface away from the rest of the NXOS device, so we must drop to the guest shell bash prompt to perform more troubleshooting commands.
Confirm the Guest Shell is running on NX-OS The Apstra agent runs under the NXOS Guest Shell to interact with the underlying bash and linux environments. This is an internal Linux Container (LXC) in which Apstra operates. We are checking to make sure the guest shell is activated and running.
C9K-172-20-65-5# show guestshell detail Virtual service guestshell+ detail State : Activated Package information Name : guestshell.ova Path : /isanboot/bin/guestshell.ova Application Name : GuestShell Installed version : 2.1(0.0) Description : Cisco Systems Guest Shell Signing Key type : Cisco release key Method : SHA-1 Licensing Name : None Version : None Resource reservation Disk : 1024 MB Memory : 3072 MB CPU : 6% system CPU Attached devices Type Name Alias --------------------------------------------- Disk _rootfs Disk /cisco/core Serial/shell Serial/aux Serial/Syslog serial2 Serial/Trace serial3
Showing registered services
C9K-172-20-65-5# show virtual-service list Virtual Service List: Name Status Package Name ----------------------------------------------------------------------- guestshell+ Activated guestshell.ova
- Confirm Network Reachability to Apstra
- Confirm Agent Installation
- Check that Apstra Agent is Running
- Check for Presence of Files in /etc/aos
- Check for Apstra Data in /var/log/aos
- Determine Apstra Agent Version
- DNS Resolution Failure
- Apstra Service Takes Long Time to Start on Cisco NX-OS
- Apstra Stops and ails Without Errors (MGMT VRF)
- Verify MGMT VRF in NX-OS Guest Shell
Confirm Network Reachability to Apstra
Within the guest shell, ping to the Apstra server to check ICMP Ping. When
running commands within the context of a VRF, use the command chvrf
<vrf>
In this case, it's management
VRF.
[guestshell@guestshell ~]$ chvrf management ping 172.20.65.3 PING 172.20.65.3 (172.20.65.3) 56(84) bytes of data. 64 bytes from 172.20.65.3: icmp_seq=1 ttl=64 time=0.239 ms 64 bytes from 172.20.65.3: icmp_seq=2 ttl=64 time=0.215 ms
Confirm Agent Installation
Check if the Apstra device agent package is installed. In NXOS, the Apstra agent
installs to /etc/rc.d/init.d/aos
to start when the guestshell
instance starts.
[guestshell@guestshell ~]$ systemctl status aos aos.service - LSB: Start AOS device agents Loaded: loaded (/etc/rc.d/init.d/aos) Active: active (running) since Tue 2016-11-15 00:10:49 UTC; 3h 54min ago Process: 30 ExecStart=/etc/rc.d/init.d/aos start (code=exited, status=0/SUCCESS) CGroup: /system.slice/aos.service ├─113 tacspawner --daemonize=/var/log/aos/aos.log --pidfile=/var/run/aos.pid --name=SAL2028T5NE --hostname=localhost --domainSocket=aos_spawner_sock --hostSysdbAddress=tb... ├─115 tacleafsysdb --agentName=SAL2028T5NE-LocalTasks-SAL2028T5NE-0 --partition= --storage-mode=persistent --eventLogDir=. --eventLogSev=TaccSpawner/error,Mounter/error,M... ├─116 /usr/bin/python /bin/aos_agent --class=aos.device.common.ProxyDeploymentAgent.ProxyDeploymentAgent --name=DeploymentProxyAgent device_type=Cisco serial_number=@(SWI... ├─117 /usr/bin/python /bin/aos_agent --class=aos.device.common.ProxyCountersAgent.ProxyCountersAgent --name=CounterProxyAgent device_type=Cisco serial_number=@(SWITCH_UNI... └─118 /usr/bin/python /bin/aos_agent --class=aos.device.cisco.CiscoTelemetryAgent.CiscoTelemetryAgent --name=DeviceTelemetryAgent serial_number=@(SWITCH_UNIQUE_ID)
Check that Apstra Agent is Running
Check the running system state with the ‘service’ command, and check running processes with the ‘ps’ command. We are looking to confirm aos_agent is running properly.
[root@guestshell ~]# service aos status aos is running [root@guestshell ~]# ps wax PID TTY STAT TIME COMMAND 1 ? Ss 0:00 /sbin/init 9 ? Ss 0:00 /usr/lib/systemd/systemd-journald 19 ? Ss 0:00 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 22 ? Ss 0:00 /usr/lib/systemd/systemd-logind 29 ? Ss 0:00 /usr/sbin/sshd -D -f /etc/ssh/sshd_config-cisco -p 17682 -o ListenAddress=localhost 38 ? Ss 0:00 /usr/sbin/crond -n 55 pts/1Ss+0:00 /sbin/agetty --noclear ttyS1 56 pts/0Ss+0:00 /sbin/agetty --noclear ttyS0 113 ? Sl 0:01 tacspawner --daemonize=/var/log/aos/aos.log --pidfile=/var/run/aos.pid --name=C9K --hostname=localhost --domainSocket=aos_spawner_sock --hostSysdbAdd 115 ? S 0:03 tacleafsysdb --agentName=C9K-LocalTasks-C9K-0 --partition= --storage-mode=persistent --eventLogDir=. --eventLogSev=TaccSpawner/error,Mounter/ 116 ? Sl 0:01 /usr/bin/python /bin/aos_agent --class=aos.device.common.ProxyDeploymentAgent.ProxyDeploymentAgent --name=DeploymentProxyAgent device_type=Cisco serial_numbe 117 ? Sl 0:19 /usr/bin/python /bin/aos_agent --class=aos.device.common.ProxyCountersAgent.ProxyCountersAgent --name=CounterProxyAgent device_type=Cisco serial_number=@(SWI 118 ? Sl 0:02 /usr/bin/python /bin/aos_agent --class=aos.device.cisco.CiscoTelemetryAgent.CiscoTelemetryAgent --name=DeviceTelemetryAgent serial_number=@(SWITCH_UNIQUE_ID) 700 ? Ss 0:00 sshd: guestshell [priv] 702 ? S 0:00 sshd: guestshell@pts/4 703 pts/4Ss 0:00 bash -li 732 pts/4S 0:00 sudo su - 733 pts/4S 0:00 su - 734 pts/4S 0:00 -bash 823 pts/4R+ 0:00 ps wax
Check for Presence of Files in /etc/aos
Under the guest shell, Apstra stores a number of configuration files under /etc/aos.
[root@guestshell aos]# ls -lah /etc/aos total 44K drwxr-xr-x 2 root root 4.0K Nov 15 00:05 . drwxr-xr-x 63 root root 4.0K Nov 15 00:09 .. -rwxr-xr-x 1 root root 1.1K Nov 14 22:26 agent.json -rw-r--r-- 1 root root 1.1K Nov 15 00:05 aos.conf -rwxr-xr-x 1 root root 992 Nov 14 22:26 common_functions -rwxr-xr-x 1 root root 1.4K Nov 14 22:26 health_check_functions -rwxr-xr-x 1 root root 450 Nov 14 22:26 iproute2_functions -rwxr-xr-x 1 root root 916 Nov 14 22:26 lsb_functions -rwxr-xr-x 1 root root 4.5K Nov 14 22:26 platform_functions -rwxr-xr-x 1 root root 156 Nov 14 22:26 version
Check for Apstra Data in /var/log/aos
Apstra writes the internal database to /var/log/aos
[root@guestshell aos]# ls -lah /var/log/aos total 500K drwxr-xr-x 2 root root 480 Nov 15 00:10 . drwxr-xr-x 3 root root 120 Nov 15 00:10 .. -rw-r--r-- 1 root root 3.2K Nov 15 00:11 CounterProxyAgent.117.1479168658.log -rw-r--r-- 1 root root 289K Nov 15 02:27 CounterProxyAgent.err -rw-r--r-- 1 root root0 Nov 15 00:10 CounterProxyAgent.out -rw------- 1 root root 31K Nov 15 00:11 CounterProxyAgentC9K_2016-11-15--00-10-59_117-2016-11-15--00-10-59.tel -rw-r--r-- 1 root root 104 Nov 15 00:45 DeploymentProxyAgent.116.1479168650.log -rw-r--r-- 1 root root 12K Nov 15 00:45 DeploymentProxyAgent.err -rw-r--r-- 1 root root0 Nov 15 00:10 DeploymentProxyAgent.out -rw------- 1 root root 31K Nov 15 00:10 DeploymentProxyAgentC9K_2016-11-15--00-10-51_116-2016-11-15--00-10-51.tel -rw-r--r-- 1 root root 4.1K Nov 15 00:11 DeviceTelemetryAgent.118.1479168657.log -rw-r--r-- 1 root root 1.4K Nov 15 00:11 DeviceTelemetryAgent.err -rw-r--r-- 1 root root0 Nov 15 00:10 DeviceTelemetryAgent.out -rw------- 1 root root 31K Nov 15 00:11 DeviceTelemetryAgentC9K_2016-11-15--00-10-58_118-2016-11-15--00-10-58.tel -rw-r--r-- 1 root root0 Nov 15 00:10 C9K-0.115.1479168649.log -rw-r--r-- 1 root root0 Nov 15 00:10 C9K-0.err -rw-r--r-- 1 root root0 Nov 15 00:10 C9K-0.out -rw------- 1 root root 39K Nov 15 00:10 C9K-LocalTasks-C9K-0_2016-11-15--00-10-50_115-2016-11-15--00-10-50.tel -rw------- 1 root root 36K Nov 15 00:10 Spawner-C9K_2016-11-15--00-10-49_111-2016-11-15--00-10-49.tel -rw------- 1 root root 634 Nov 15 00:10 _C9K-00000000582a528a-0001744b-checkpoint -rw-r--r-- 1 root root0 Nov 15 00:10 _C9K-00000000582a528a-0001744b-checkpoint-valid -rw------- 1 root root0 Nov 15 00:10 _C9K-00000000582a528a-0001744b-log -rw-r--r-- 1 root root0 Nov 15 00:10 _C9K-00000000582a528a-0001744b-log-valid -rw-r--r-- 1 root root0 Nov 15 00:10 aos.log [root@guestshell aos]#
Determine Apstra Agent Version
The Apstra agent version is available in /etc/aos/version. Before executing this command we need to attach to aos service.
[root@guestshell admin]# service aos attach aos@guestshell:/# cat /etc/aos/version VERSION=99.0.0-3874 BUILD_ID=AOS_latest_OB.3874 BRANCH_NAME=master COMMIT_ID=d3eb2585608f0509a11b95fb9d07aed6e26d6c32 BUILD_DATETIME=2018-05-20_10:22:32_PDT AOS_DI_RELEASE=2.2.0-169 aos@guestshell:/#
DNS Resolution Failure
Apstra agent is sensitive to the DNS resolution of the metadb connection. Ensure that the IP and/or DNS from /etc/aos/aos.conf is reachable from the device eth0 management port.
[root@guestshell ~]# aos_show_tech | grep -i dns [2016/10/20 23:04:20.534538UTC@event-'warning']:(textMsg=Failing outgoing mount to <'tbt://aos-server:29731/Data/ReplicaStatus?flags=i','/Metadb/ReplicaStatus'>' due to code 'resynchronizing' and reason 'Dns lookup issue "Temporary failure in name resolution" Unknown error 18446744073709551613) [2016/10/20 23:04:21.540444UTC@OutgoingMountConnectionError-'warning']:(connectionName=--NONE--,localPath=/Metadb/ReplicaStatus,remotePath=tbt://aos-server:29731/Data/ReplicaStatus?flags=i,msg=Tac::ErrnoException: Dns lookup issue "Temporary failure in name resolution" Unknown error 18446744073709551613) [2016/10/20 23:04:21.541174UTC@event-'warning']:(textMsg=Failing outgoing mount to <'tbt://aos-server:29731/Data/ReplicaStatus?flags=i','/Metadb/ReplicaStatus'>' due to code 'resynchronizing' and reason 'Dns lookup issue "Temporary failure in name resolution" Unknown error 18446744073709551613) Insufficient Guestshell filesystem size An error message ‘AOS Agent needs XXMB on the / filesystem’ will occur if the rootfs partition is not at least 1GB large. Please make sure to resize the guestshell filesystem to 2gb ram, 1gb disk, and 6% CPU. <snip> + popd /tmp/selfgz18527139 + rpm -Uvh --nodeps --force /tmp/selfgz18527139/aos-device-agent-1.1.0-0.1.1108.x86_64.rpm Preparing... ################################# [100%] installing package aos-device-agent-1.1.0-0.1.1108.x86_64 needs 55MB on the / filesystem
Apstra Service Takes Long Time to Start on Cisco NX-OS
It takes a few minutes for the GuestShell on Cisco NX-OS to initialize the NXAPI within the LXC container. This is normal. To account for this delay, a wait-delay has been added to the Apstra script initialization.
Apstra Stops and ails Without Errors (MGMT VRF)
Ensure that the guestshell is properly behind management VRF.
We should not be able to ping the Apstra server when running ‘ping’ command by default:
Below - we expect a ping from global default routing table to Apstra server at 172.20.156.3 to fail, but succeed under the guest shell.
SAL2028T5PP-172-20-156-5# ping 172.20.156.3 PING 172.20.156.3 (172.20.156.3): 56 data bytes ping: sendto 172.20.156.3 64 chars, No route to host ^C --- 172.20.156.3 ping statistics --- 1 packets transmitted, 0 packets received, 100.00% packet loss SAL2028T5PP-172-20-156-5# ping 172.20.156.3 vrf management PING 172.20.156.3 (172.20.156.3): 56 data bytes 64 bytes from 172.20.156.3: icmp_seq=0 ttl=63 time=0.649 ms 64 bytes from 172.20.156.3: icmp_seq=1 ttl=63 time=0.449 ms 64 bytes from 172.20.156.3: icmp_seq=2 ttl=63 time=0.428 ms 64 bytes from 172.20.156.3: icmp_seq=3 ttl=63 time=0.423 ms 64 bytes from 172.20.156.3: icmp_seq=4 ttl=63 time=0.404 ms ^C
Verify MGMT VRF in NX-OS Guest Shell
[root@guestshell ~]# ping 172.20.157.3 connect: Network is unreachable [root@guestshell ~]# sudo ip netns exec management ping 172.20.156.3 PING 172.20.156.3 (172.20.156.3) 56(84) bytes of data. 64 bytes from 172.20.156.3: icmp_seq=1 ttl=64 time=0.226 ms 64 bytes from 172.20.156.3: icmp_seq=2 ttl=64 time=0.232 ms ^C