Apstra ZTP

Overview

Apstra ZTP is a Zero-Touch-Provisioning server for data center infrastructure systems. (Apstra ZTP replaces the community-supported Aeon-ZTPS software that Apstra previously used for ZTP implementation.) Apstra ZTP enables you to bootstrap Apstra data center devices without considering the differences in underlying NOS mechanisms. ZTP, from an Apstra perspective, is a process that takes a device from initial boot to a point where it is managed by Apstra via device agents.

Depending on how ZTP is configured, the process may include (but not always) the following capabilities:

  • A DHCP service
  • Setting the device admin/root password
  • Creating a device user for device system agent
  • Upgrading / downgrading device OS
  • Installing license (Cumulus only)
  • On-box or Off-box Device System Agent installation

Note

To prevent being locked out of a device when there is a problem during the ZTP process, ZTP uses default, hard-coded credentials. These credentials are:

  • root / admin
  • aosadmin / aosadmin

Apstra ZTP is a reference implementation. The two supported Apstra ZTP versions are:

  • Apstra ZTP 1.0.0 - supports Apstra versions 3.1.1 through 3.3.0 and devices running Juniper Junos, Cisco NX-OS, Arista EOS, and Cumulus Linux.
  • Apstra ZTP 2.0.0 - released with version 3.3.0a supports devices running Juniper Junos, Enterprise SONiC, Cisco NX-OS, Arista EOS, and Cumulus Linux.

You can use an Apstra-provided VM image (.ova, .tar.gz, .qcow2.gz, .vhdx.gz) or build your own ZTP server and use the Apstra-provided device provisioning scripts as part of the existing ZTP/DHCP process to automatically install agents on devices as part of the boot process. The reference implementation consists of the following three phases:

  1. Generic DHCP Phase
    • The device requests an IP address via DHCP.
    • The device receives the assigned IP address and a pointer to a script to execute (or an OS image to install if using the Apstra-provided VM image).
  2. Initialization Phase
    • The device downloads the ZTP script using TFTP.
    • The device executes the downloaded script preparing it for Apstra management. This includes verifying that the device is running an Apstra-supported OS.
  3. Agent Installation Phase
    • The ZTP script makes an API call to Apstra to install a device system agent on the device.

Apstra ZTP 2.0.0

Apstra ZTP 2.0.0 was introduced with version 3.3.0a and supports ZTP for Juniper Junos, Cisco NX-OS, Arista EOS, Cumulus Linux, and Enterprise SONiC devices.

Apstra ZTP 2.0.0 VM Server Resource Requirements

Apstra ZTP 2.0.0 runs as an Ubuntu 18.04LTS server running a DHCP, HTTP, and TFTP server and includes Apstra-provided ZTP scripts that must be customized for your environment. The table below shows the minimum server specifications for a production environment:

Resource Setting
Guest OS Type Ubuntu 18.04 LTS 64-bit
Memory 2 GB
CPU 1 vCPU
Disk Storage 64 GB
Network At least 1 network adapter. Configured for DHCP initially

Apstra ZTP 2.0.0 Network requirements

Source Destination Ports Role
Device agents DHCP Server (renewals) & Broadcast (requests) udp/67 -> udp/68 DHCP Client
Device agents Apstra ZTP any -> tcp/80 Bootstrap and API scripts
Arista and Cisco Device agents Apstra ZTP any -> udp/69 TFTP for POAP and ZTP
Apstra ZTP Apstra Server any -> tcp/443 Device System Agent Installer API

In addition to the ZTP-specific network requirements, the Apstra ZTP server and device agents require connectivity to Apstra. Refer to Apstra Network Security Requirements for more information.

Download and Deploy Apstra ZTP VM on Standalone Apstra ZTP VM

  1. Download the appropriate Apstra 2.0.0 ZTP image from https://support.juniper.net/support/downloads/?p=apstra>.

    • Linux KVM QCOW2 image - apstra-ztp-2.0.0-<build-version>.qcow2.gz (example: apstra-ztp-2.0.0-53.qcow2.gz)
    • VMware OVA image - apstra-ztp-2.0.0-<build-version>.ova (example: apstra-ztp-2.0.0-53.ova)
    • Microsoft Hyper-V - apstra-ztp-2.0.0-<build-version>.vhdx.gz (example: apstra-ztp-2.0.0-53.vhdx.gz)
  2. Validate the downloaded file against the SHA512/MD5 checksums provided.

  3. Deploy the VM with the appropriate resources.

  4. By default, TFTP, NGINX (HTTP) and DHCP Server Docker containers are enabled to run by default.

    admin@apstra-ztp:~$ docker ps
    CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                                                        NAMES
    3f6c53304501        apstra/tftp         "sh /init.sh"            4 hours ago         Up 12 minutes       0.0.0.0:69->69/udp                                                                           tftp
    ba53ebd62b04        apstra/nginx        "sh /init.sh"            4 hours ago         Up 12 minutes       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:31415->31415/tcp   nginx
    0fbda7d068e1        apstra/status       "sh /init.sh"            4 hours ago         Up 12 minutes       8080/tcp                                                                                     status
    59a61c037096        apstra_ztp_dhcpd    "sh /init.sh"            4 hours ago         Up 12 minutes                                                                                                    dhcpd
    caf901aac317        mysql:8             "docker-entrypoint.s..."   4 hours ago         Up 12 minutes       3306/tcp, 33060/tcp                                                                          db
    admin@apstra-ztp:~$
    
  5. If you don’t want to use the Apstra ZTP DHCP Server, stop and disable the dhcpd container.

    admin@apstra-ztp:~$ docker stop dhcpd
    dhcpd
    admin@apstra-ztp:~$ docker update --restart=no dhcpd
    dhcpd
    admin@apstra-ztp:~$
    

Configuring Static Management IP Address

By default, the Apstra ZTP Server attempts to assign an IP address for its eth0 interface via DHCP. If you’re using the Apstra ZTP Server as a DHCP server, you must set a static management IP address.

  1. Log into the Apstra server as user admin. To configure a static management IP address, edit the /etc/netplan/01-netcfg.yaml file.

    admin@apstra-ztp:~$ sudo vi /etc/netplan/01-netcfg.yaml
    [sudo] password for admin:
    
    # This file describes the network interfaces available on your system
    # For more information, see netplan(5).
    network:
      version: 2
      renderer: networkd
      ethernets:
        eth0:
          dhcp4: no
          addresses: [192.168.59.4/24]
          gateway4: 192.168.59.1
          nameservers:
              search: [example.com, example.net]
              addresses: [69.16.169.11, 69.16.170.11]
    
  2. Apply the IP address change with the sudo netplan apply command, or reboot the Apstra server with the sudo reboot command.

For more information on using netplan, see https://netplan.io/examples.

Configure DHCP Server

An ISC DHCP server is provided on the Apstra ZTP VM. You may decide to use another DHCP for the device management network. This guide describes the process to edit the supplied DHCP server. You are responsible for configuring the same options if you’re using a different DHCP server. For example, if you’re using Cumulus Linux, you must ensure that your server contains the following, so the devices download the Apstra ZTP ztp.py file.

option cumulus-provision-url "tftp://192.168.59.4/ztp.py";

On the Apstra ZTP VM, the DHCP configuration files are in the /containers_data/dhcp directory:

admin@apstra-ztp:~$ sudo ls -l /containers_data/dhcp
total 16
-rw------- 1 root root 2533 Oct 21 00:35 dhcpd.conf
-rw------- 1 root root  146 Oct 21 00:35 Dockerfile
-rw------- 1 root root  932 Oct 21 00:35 init.sh
-rw------- 1 root root 1896 Oct 21 00:35 rsyslog.conf
admin@apstra-ztp:~$

Note

All configuration files are owned by root. You must use sudo to run commands as root using the sudo command or after becoming root with the sudo -s command.

  1. Edit the dhcpd.conf file with vi or nano.

    admin@apstra-ztp:~$ sudo nano /containers_data/dhcp/dhcpd.conf
    
  2. Add a “group” corresponding to the management network:

    group {
        option tftp-server-name "192.168.59.4";
        subnet 192.168.59.0 netmask 255.255.255.0 {
          range 192.168.59.21 192.168.59.99;
          option routers 192.168.59.1;
        }
        host my-switch {
            hardware ethernet 34:17:eb:1e:41:80;
            fixed-address 192.168.59.100;
        }
    }
    
    • tftp-server-name: IP address of ZTP server (not a URL)
    • subnet: IP management network and netmask
    • range: Range of dynamic DHCP IP addresses. Ensure the full range is available and no statically configured IP addresses from that range are used.
    • option routers: Default gateway router for management network
    • host: Static DHCP IP address (fixed-address) for device with hardware ethernet MAC. Use the Switch MAC address (hardware ethernet) of the management interface used for DHCP negotiations.
  3. The following DHCP parameters are optional:

    ddns-update-style none;
    option domain-search "example.internal";
    option domain-name "example.internal";
    option domain-name-servers 8.8.8.8, 8.8.4.4;
    
  4. If you’re using ZTP with Cumulus Linux, edit the following:

    class "cumulus" {
        match if (substring(option host-name, 0, 7) = "cumulus");
        option cumulus-provision-url "tftp://192.168.59.4/ztp.py";
    }
    
    • cumulus-provision-url: TFTP URL with IP address of ZTP server
  5. If using ZTP with SONiC, edit the following:

    class "sonic" {
        match if (substring(option host-name, 0, 5) = "sonic");
        option sonic-provision-url "tftp://192.168.59.4/ztp.py";
    }
    
    • sonic-provision-url: TFTP URL with IP address of ZTP server
  6. After modifying any DHCP configuration change, restart the Apstra ZTP DHCP process with the sudo docker restart dhcpd command.

    admin@apstra-ztp:~$ docker restart dhcpd
    dhcpd
    admin@apstra-ztp:~$
    

Configure ZTP User on Apstra Server

You may use any configured GUI user that has API write access (such as admin), but we recommended that you create a designated user (for example “ztp”) that is assigned the predefined role device_ztp. The device_ztp role allows you to make API calls to the Apstra server to request device system agent installation.

_images/ztp_user_330a.png

Configure Apstra Server IP for ZTP

You must configure the Apstra server IP and Apstra ZTP username in the /containers_data/status/app/aos.conf file on the Apstra ZTP 2.0.0 Server.

admin@apstra-ztp:~$ sudo nano /containers_data/status/app/aos.conf
{
   "ip": "192.168.0.3",
   "user": "ztp",
   "password": "ztp-user-password"
}
  • ip: The IP Address of the Apstra server
  • user: The username of the ZTP or admin user
  • password: The user’s password

Edit ZTP Configuration File

Apstra ZTP VM includes a TFTP and nginx HTTP server. These servers do not require configuration. Both servers serve files out of the /containers_data/tftp directory.

admin@apstra-ztp:~$ sudo ls -l /containers_data/tftp/
total 220
-rw------- 1 root root  2448 Oct 21 00:35 config_verifier.py
-rw------- 1 root root   393 Oct 21 00:35 container_init.sh
-rw------- 1 root root   170 Oct 21 00:35 cumulus_custom.sh
-rw------- 1 root root    55 Oct 21 00:35 cumulus_license_file
-rw------- 1 root root   192 Oct 21 00:35 Dockerfile
-rw------- 1 root root   107 Oct 21 00:35 eos_custom.sh
-rw------- 1 root root  4721 Oct 21 00:35 junos_apstra_ztp_bootstrap.sh
-rw------- 1 root root  1799 Oct 21 00:35 junos_custom.sh
-rw------- 1 root root    86 Oct 21 00:35 nxos_custom.sh
-rwx------ 1 root root   205 Oct 21 00:35 poap-md5sum
-rw------- 1 root root  1843 Oct 21 00:35 rsyslog.conf
-rw------- 1 root root  1910 Oct 21 00:35 ztp.json
-rw------- 1 root root 84855 Oct 21 00:36 ztp.py
-rw------- 1 root root 84812 Oct 21 00:36 ztp.py.md5
admin@apstra-ztp:~$

The ztp.json file contains all the configuration for the Apstra ZTP script ztp.py.

  1. Edit the ztp.json file with vi or nano.

    admin@apstra-ztp:~$ sudo nano /containers_data/tftp/ztp.json
    
  2. The ztp.json file is organized by the following:

    • defaults: Values are used for all devices unless more specific keys are defined.

      "defaults": {
        "device-root-password": "root-password-123",
        "device-user": "admin",
        "device-user-password": "admin-password-123",
        "system-agent-params": {
          "agent_type": "onbox",
          "install_requirements": false
        }
      }
      
    • platform: Values are used for all devices for a network platform (“cumulus”, “nxos”, “eos”, “junos”, “sonic”) unless more specific keys are defined.

      "cumulus": {
        "cumulus-versions": ["3.7.12", "3.7.11"],
        "cumulus-image": "http://192.168.59.4/cumulus-linux-3.7.12-bcm-amd64.bin",
        "license": "cumulus_license_file",
        "custom-config": "cumulus_custom.sh",
      }
      
    • model: Values are used for all devices for a specific device model (for example “N9K-C93180YC-FX”).

      "N9K-C93180YC-FXC3396": {
        "custom-config": "93180_cumulus_custom.sh",
      }
      
    • serial number: Values are used for a device matching a specific device serial number (for example “525400B3C311”).

      "525400B3C311": {
        "cumulus-versions": [ "3.7.13" ],
        "cumulus-image": "http://192.168.59.4/cumulus-linux-3.7.13-bcm-amd64.bin"
      }
      

    More specific data takes precedence over other data. For example, data for a specific serial number takes precedence over any other data, then model, then platform, then finally default data.

  3. The ztp.json file uses the following keys:

    • nxos-versions: Valid versions for NX-OS devices. If a device is not running a version in this list, ZTP upgrades the device with the nxos-image image.

      "nxos-versions": [ "7.0(3)I7(4)", "9.2(2)"]
      
    • nxos-image: This is filename of the NX-OS image to be loaded if the running version does not match a version in the nxos-versions list.

      By default, the image name will be loaded from the ZTP server via TFTP from the ZTP server’s /container_data/tftp/ directory. To use any HTTP server for image transfer, enter a valid HTTP URL with IP address. For example:

      "nxos-image": "http://192.168.59.4/nxos.9.2.2.bin"
      

      The above example uses HTTP from the ZTP server to transfer the Cisco NX-OS image.

    • cumulus-versions: Valid versions for Cumulus Linux devices. If a device is not running a version in this list, ZTP upgrades the device with the cumulus-image image.

      "cumulus-versions": ["3.7.12", "3.7.11"]
      
    • cumulus-image: This is the filename of the Cumulus Linux ONIE BIN image to be loaded if the running version does not match a version in the cumulus-versions list.

      By default, the image name is loaded from the ZTP server via TFTP from the ZTP server’s /container_data/tftp/ directory. To use any HTTP server for image transfer, enter a valid HTTP URL with IP address. For example:

      "cumulus-image": "http://192.168.59.4/cumulus-linux-3.7.12-bcm-amd64.bin"
      

      The above example will use HTTP from the ZTP server to transfer the Cumulus Linux image.

    • eos-versions: Valid versions for Arista EOS devices. If a device is not running a version in this list, ZTP upgrades the device with the eos-image image.

      "eos-versions": ["4.20.11M", "4.21.5.1F"]
      
    • eos-image: This is the filename of the Arista EOS SWI image to be loaded if the running version does not match a version in the eos-versions list.

      By default, the image name is loaded from the ZTP server via TFTP from the ZTP server’s /container_data/tftp/ directory. To use any HTTP server for image transfer, enter a valid HTTP URL with IP address. For example:

      "eos-image": "http://192.168.59.3/dos_images/EOS-4.21.5.1F.swi"
      

      The above example uses HTTP from the Apstra server to transfer the Arista EOS image.

    • junos-versions: Valid versions for Juniper Junos devices. If a device is not running a version in this list, ZTP upgrades the device with the junos-image image.

      "junos-versions": [ "18.4R3-S4.2" ]
      
    • junos-image: This is the filename of the Juniper Junos TGZ image to be loaded if the running version does not match a version in the junos-versions list.

      By default, the image name is loaded from the ZTP server via TFTP from the ZTP server’s /container_data/tftp/ directory. To use any HTTP server for image transfer, enter a valid HTTP URL with IP address. For example:

      "junos-image": "http://192.168.59.4/jinstall-host-qfx-5-18.4R3-S4.2-signed.tgz"
      

      The above example uses HTTP from the Apstra server to transfer the Juniper Junos image.

    • sonic-versions: Valid versions for SONiC devices. If a device is not running a version in this list, ZTP upgrades the device with the sonic-image image.

      "sonic-versions": [ "SONiC-OS-3.1.0a-Enterprise_Base" ]
      
    • sonic-image: This is filename of the SONiC ONIE BIN image to be loaded if the running version does not match a version in the sonic-versions list.

      By default, the image name is loaded from the ZTP server via TFTP from the ZTP server’s /container_data/tftp/ directory. To use any HTTP server for image transfer, enter a valid HTTP URL with IP address. For example:

      "sonic-image": "http://192.168.59.3/sonic-3.1.0a-bcm.bin"
      

      The above example uses HTTP from the Apstra server to transfer the SONiC image.

    • device-root-password: The ZTP process sets the device root password to this value. For Arista EOS and Cisco NX-OS devices, the device-root-password is used to set the password for the system admin password.

      "device-root-password": "root-admin-password"
      
    • device-user / device-user-password: This is the username and password Apstra uses for the device system agent. Also, if necessary, the ZTP process creates a user on the device with this username and password.

      "device-user": "aosadmin",
      "device-user-password": "aosadmin-password"
      
    • license: The filename of the Cumulus Linux license file in the TFTP directory or a URL pointing to the file on a HTTP server.

      "license": "cumulus_license_file"
      
    • custom-config: The filename of the custom configuration shell script in the TFTP directory or a URL pointing to the file on a HTTP server. This shell script is run during ZTP allowing you to add custom configuration to the device. See Platform Specific Information section below for more information.

      "custom-config": "cumulus_custom.sh"
      
    • system-agent-params: The information that Apstra uses to create a new user and device system agent on the device.

      • agent_type: The agent type, onbox or offbox.

        "agent_type": "onbox"
        
      • install_requirements: Always set to false. Not needed for any supported Network Operating System.

        "install_requirements": false
        
      • job_on_create: (On-box agents only) Set to install to install the on-box agent on the device.

        "job_on_create": "install"
        
      • platform: (Required for off-box agents only) Set to the device platform (“eos”, “nxos”, “junos”). Must be in all lowercase.

        "platform": "junos"
        
      • open_options: (Off-box agents only) Set to enable HTTPS between off-box agent to device API interface. Connection defaults to HTTP if open_options is not defined.

        "open_options": {
          "proto": "https",
          "port": "443"
        }
        
      • packages: Set to configure which additional SDK or extended telemetry packages to upload to the system agent.

        "packages": [
          "aos-deployment-helper-nxos",
          "aosstdcollectors-builtin-nxos",
          "aosstdcollectors-custom-nxos"
        ]
        

Refer to the Swagger Platform REST API Documentation for /api/system-agents for all available system-agent-params options.

Apstra ZTP 1.0.0

Apstra ZTP 1.0.0 VM Server Resource Requirements

Apstra ZTP 1.0.0 runs as an Ubuntu 18.04LTS Server running a DHCP, HTTP, and TFTP server and includes Apstra provided ZTP scripts that must be customized for your environment. Below table shows the minimum server specifications for a production environment:

Resource Setting
Guest OS Type Ubuntu 18.04 LTS 64-bit
Memory 2 GB
CPU 1 vCPU
Disk Storage 64 GB
Network At least 1 network adapter. Configured for DHCP initially

Apstra ZTP 1.0.0 Network requirements

Source Destination Ports Role
Device agents DHCP Server (renewals) & Broadcast (requests) udp/67 -> udp/68 DHCP Client
Device agents Apstra ZTP any -> tcp/80 Bootstrap and API scripts
Arista and Cisco Device agents Apstra ZTP any -> udp/69 TFTP for POAP and ZTP
Apstra ZTP Apstra Server any -> tcp/443 Device System Agent Installer API

In addition to the ZTP-specific network requirements, the Apstra ZTP server and device agents require connectivity to Apstra. Refer to Apstra Network Security Requirements for more information.

Download and Deploy the Apstra ZTP VM

Download the Apstra ZTP VM image Linux KVM/QEMU QCOW2 and VMware OVA images are available:

  • Linux KVM QCOW2 image apstra-ztp-<version>.qcow2.gz (for example apstra-ztp-1.0.0-48.qcow2.gz)
  • VMware OVA image apstra-ztp-<version>.ova (for example apstra-ztp-1.0.0-48.ova)

Validate the downloaded file against the SHA512/MD5 checksums provided.

Deploy the VM with the appropriate resources outlined in Apstra ZTP 1.0.0 VM Server Resource Requirements.

Deploy Apstra ZTP on the Apstra Server VM

You may install Apstra ZTP directly on the Apstra server, or on another Ubuntu Linux 18.04.3 LTS server with Docker installed. This involves the following steps:

  1. Download the apstra-ztp-<version>.tar.gz (for example apstra-ztp-1.0.0-48.tar.gz) from https://support.juniper.net/support/downloads/?p=apstra and copy it to the Apstra server.

    Note

    If you’re using the Apstra ZTP DHCP server on the Apstra server, we recommend changing the Apstra server network interface to a static IP address. For more information, see Configuring Static Management IP Address.

  2. On the AOS server VM Linux CLI, as root, use the tar command to extract the files to the root (/) directory:

    admin@aos-server:~$ sudo tar zxvf apstra-ztp-1.0.0-48.tar.gz -C /
    etc/apstra_ztp/
    etc/apstra_ztp/version
    etc/apstra_ztp/docker-compose-nohttp.yml
    etc/apstra_ztp/docker-compose.yml
    containers_data/
    containers_data/tftp/
    containers_data/tftp/eos_custom.sh
    containers_data/tftp/ztp.py
    containers_data/tftp/ztp.json
    containers_data/tftp/junos_custom.sh
    containers_data/tftp/cumulus_custom.sh
    containers_data/tftp/poap-md5sum
    containers_data/tftp/cumulus_license_file
    containers_data/tftp/nxos_custom.sh
    containers_data/tftp/junos_apstra_ztp_bootstrap.sh
    containers_data/Dockerfile.tftp
    containers_data/dhcp/
    containers_data/dhcp/dhcpd.conf
    containers_data/dhcp/dhcpd.leases
    containers_data/init
    
  3. On the AOS server VM Linux CLI, as root, run the docker-compose command to install the Apstra ZTP dhcpd and tftp Docker containers.

    Note

    To download the required packages from the Docker repository, Internet access is required. If the Apstra server VM does not have access, you must manually upload and install the dhcpd and tftp Docker containers images before running the docker-compose command. Contact Juniper Support for assistance.

    admin@aos-server:~$ sudo docker-compose -f /etc/apstra_ztp/docker-compose-nohttp.yml up --detach
    WARNING: The CONTAINER_DATA_BASE_DIR variable is not set. Defaulting to a blank string.
    Creating network "apstra_ztp_default" with the default driver
    Building tftp
    Step 1/7 : FROM alpine:3.4
    3.4: Pulling from library/alpine
    c1e54eec4b57: Pull complete
    Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
    Status: Downloaded newer image for alpine:3.4
     ---> b7c5ffe56db7
    Step 2/7 : RUN apk update && apk add --no-cache tftp-hpa rsyslog
     ---> Running in 5261c8b831f6
    fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
    fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
    v3.4.6-316-g63ea6d0 [http://dl-cdn.alpinelinux.org/alpine/v3.4/main]
    v3.4.6-160-g14ad2a3 [http://dl-cdn.alpinelinux.org/alpine/v3.4/community]
    OK: 5973 distinct packages available
    fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
    fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
    (1/9) Installing libestr (0.1.10-r0)
    (2/9) Installing libfastjson (0.99.2-r0)
    (3/9) Installing libgpg-error (1.23-r0)
    (4/9) Installing libgcrypt (1.7.9-r0)
    (5/9) Installing liblogging (1.0.5-r1)
    (6/9) Installing libnet (1.1.6-r2)
    (7/9) Installing libuuid (2.28-r3)
    (8/9) Installing rsyslog (8.18.0-r0)
    (9/9) Installing tftp-hpa (5.2-r2)
    Executing busybox-1.24.2-r14.trigger
    OK: 8 MiB in 20 packages
    Removing intermediate container 5261c8b831f6
     ---> 07754ef3a795
    Step 3/7 : COPY init /
     ---> c964d72e19aa
    Step 4/7 : VOLUME /var/tftpboot
     ---> Running in 231e4640f053
    Removing intermediate container 231e4640f053
     ---> 3951abf2f5f6
    Step 5/7 : EXPOSE 69/udp
     ---> Running in 2eaaa53dcc59
    Removing intermediate container 2eaaa53dcc59
     ---> edf66a3456ae
    Step 6/7 : ENTRYPOINT ["sh"]
     ---> Running in c0c705a3f73d
    Removing intermediate container c0c705a3f73d
     ---> f929bb9c9b93
    Step 7/7 : CMD ["/init"]
     ---> Running in c1a3bed09729
    Removing intermediate container c1a3bed09729
     ---> 301c97f4656a
    
    Successfully built 301c97f4656a
    Successfully tagged apstra/tftp:latest
    WARNING: Image for service tftp was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
    Pulling dhcpd (networkboot/dhcpd:)...
    latest: Pulling from networkboot/dhcpd
    898c46f3b1a1: Pull complete
    63366dfa0a50: Pull complete
    041d4cd74a92: Pull complete
    6e1bee0f8701: Pull complete
    114483241095: Pull complete
    ef446bdcb1f0: Pull complete
    Digest: sha256:fdc7ff6f265249a104f32f1d7aed0aedaf2f2fc62ea10eebf596e2af3b670477
    Status: Downloaded newer image for networkboot/dhcpd:latest
    Creating apstra_ztp_dhcpd_1 ... done
    Creating apstra_ztp_tftp_1  ... done
    admin@aos-server:~$
    
  4. You can verify the running Docker Containers with the docker ps command.

    admin@aos-server:~$ docker ps
    CONTAINER ID        IMAGE                 COMMAND                  CREATED              STATUS              PORTS                NAMES
    1319db66b7b6        networkboot/dhcpd     "/entrypoint.sh eth0"    About a minute ago   Up About a minute                        apstra_ztp_dhcpd_1
    2f6e5f5ce4f3        apstra/tftp           "sh /init"               About a minute ago   Up About a minute   0.0.0.0:69->69/udp   apstra_ztp_tftp_1
    452f4c93c5f5        aos:3.1.1-179         "/usr/bin/aos_launch…"   About an hour ago    Up About an hour                         aos_metadb_1
    40f226709b4a        aos:3.1.1-179         "/usr/bin/aos_launch…"   About an hour ago    Up About an hour                         aos_auth_1
    021a50908640        aos:3.1.1-179         "/usr/bin/aos_launch…"   About an hour ago    Up About an hour                         aos_sysdb_1
    b41de404ec14        aos:3.1.1-179         "/usr/bin/aos_launch…"   About an hour ago    Up About an hour                         aos_controller_1
    ed2e77565552        nginx:1.14.2-upload   "nginx -g 'daemon of…"   About an hour ago    Up About an hour                         aos_nginx_1
    admin@aos-server:~$
    

Using the Apstra Server VM as an HTTP image server for ZTP

By default, the ZTP process uses TFTP to transfer ZTP upgrade images to devices. The TFTP process can be slow for large images and networks with higher latency.

Beginning with Apstra ZTP version 1.0.0-33, you can use HTTP to transfer ZTP upgrade images to devices.

For the Apstra ZTP server VM, a Nginx HTTP server has been added beginning with Apstra ZTP version 1.0.0-33. The HTTP server serves the same directory as the TFTP server (/containers_data/tftp).

Whether you’re using Apstra ZTP on an Apstra ZTP VM or an Apstra server VM, you may use the Nginx HTTP server on the Apstra ZTP server VM for ZTP images.

You must upload “OS images” to the Apstra Server via the GUI. For more information, see Upgrading Device OS.

The HTTP URL for files is available under Devices > OS Images.

_images/apstra_ztp_6_311.png

By default, the Apstra server VM redirects HTTP connections to an HTTPS connection. If the Apstra server is not configured with a signed SSL certificate, you must reconfigure the Apstra server Nginx process to not redirect for the dos_images directory.

/etc/aos/nginx.conf.d/nginx.conf
server {
    listen       80;
    server_name _;
    location /api { return 307 https://$host$request_uri; }
    location / { return 301 https://$host$request_uri; }
    # for this URI we allow plain HTTP access
    location /dos_images { root   /usr/share/nginx/html; }
}

After reconfiguring this file, signal Nginx to reload the configuration file.

root@aos-server:/etc/aos/nginx.conf.d# docker kill -s HUP aos_nginx_1
aos_nginx_1
root@aos-server:/etc/aos/nginx.conf.d#

Configure the Apstra ZTP Server VM

After the Apstra ZTP server completes its first boot-up, you can log in and set management IP address information and the Apstra server IP address.

Note

Apstra ZTP expects a DHCP IP address on first boot. You must connect to the Apstra ZTP server VM via a console to configure a static IP address.

Log in through the hypervisor console to configure the management IP address the first time.

Note

Initial username and password are admin and admin. Change the admin password with the passwd command.

Logging in to Apstra ZTP
Ubuntu 18.04.3 LTS apstra-ztp tty1

apstra-ztp login: admin
Password:

Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

admin@apstra-ztp:~$

Management IP address

Note

If you’re using the Apstra ZTP server as a DHCP server, you must reconfigure the eth0 interface with a static IP address.

Configure the IP address by modifying the /etc/netplan/01-netcfg.yaml file with the sudo vi /etc/netplan/01-netcfg.yaml command (or another text editor).

/etc/netplan/01-netcfg.yaml
# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: no
      addresses: [192.168.59.4/24]
      gateway4: 192.168.59.1

Apply the IP address change with the sudo netplan apply command or reboot the Apstra ZTP server with the sudo reboot command.

Configure Apstra ZTP 1.0.0 User on Apstra Server

We recommend that you create a “ztp” user with a Role-Based Access Control (RBAC) role of “device_ztp”. This role only allows this user to make an API call to the Apstra server to request device system agent installation.

  1. From the left navigation menu, navigate to Platform > User Management >Users.

    _images/apstra_ztp_1_310.png
  2. Click Create User and enter the Username, Password and select the Role of device_ztp.

    _images/apstra_ztp_2_310.png
  3. Click Create to create the profile.

For more information about RBAC see the Providers section.

Use of this role is optional. You may use the admin Apstra user or any other configured user that has API write access.

Configuring the DHCP Server

An ISC DHCP server is provided on the Apstra ZTP VM. You may decide to use another DHCP for the device management network. This guide describes the process to edit the supplied DHCP server. You are responsible for configuring the same options if you’re using a different DHCP server. For example, if you’re using Cumulus Linux, you must ensure that your server contains the following information so devices download the Apstra ZTP ztp.py file.

option cumulus-provision-url "tftp://192.168.59.4/ztp.py";

On the Apstra ZTP VM, the DHCP configuration files are in the /containers_data/dhcp directory:

admin@apstra-ztp:/containers_data/dhcp$ ls -l
total 12
-rw-r--r-- 1 root            root            1526 Aug 27 19:36 dhcpd.conf
-rw-r--r-- 1 root            root             274 Aug 28 00:37 dhcpd.leases
-rw-r--r-- 1 systemd-resolve systemd-journal  274 Aug 27 19:36 dhcpd.leases~
admin@apstra-ztp:/containers_data/dhcp$

Note

All configuration files are owned by root. You must use sudo to run commands as root.

  1. Edit the dhcpd.conf file with vi or nano.

    admin@apstra-ztp:/containers_data/dhcp$ sudo nano dhcpd.conf
    
  2. Add a “group” corresponding to the management network:

    group {
        option tftp-server-name "192.168.59.4";
        subnet 192.168.59.0 netmask 255.255.255.0 {
          range 192.168.59.21 192.168.59.99;
          option routers 192.168.59.1;
        }
        host my-switch {
            hardware ethernet 34:17:eb:1e:41:80;
            fixed-address 192.168.59.100;
        }
    }
    
    • tftp-server-name: IP address of ZTP server (not a URL)
    • subnet: IP management network and netmask
    • range: Range of dynamic DHCP IP addresses. Ensure the full range is available and no statically configured IP addresses from that range are used.
    • option routers: Default gateway router for management network
    • host: Static DHCP IP address (fixed-address) for device with hardware ethernet MAC. Use the Switch MAC address of the management interface used for DHCP negotiations.
  3. The following DHCP parameters are optional:

    ddns-update-style none;
    option domain-search "example.internal";
    option domain-name "example.internal";
    option domain-name-servers 8.8.8.8, 8.8.4.4;
    

Warning

Incorrect DNS settings may result in timeouts as the server attempts to resolve IP addresses. When in doubt, remove the DNS section.

  1. If you’re using ZTP with Cumulus Linux, edit the following:

    class "cumulus" {
        match if (substring(option host-name, 0, 7) = "cumulus");
        option cumulus-provision-url "tftp://192.168.59.4/ztp.py";
    }
    
    • cumulus-provision-url: TFTP URL with IP address of ZTP server

After modifying any DHCP configuration change, restart the Apstra ZTP DHCP process with the sudo docker restart apstra_ztp_dhcpd_1 command.

admin@apstra-ztp:~$ sudo docker restart apstra_ztp_dhcpd_1
apstra_ztp_dhcpd_1
admin@apstra-ztp:~$

Editing the ZTP Configuration File

A TFTP server is provided on the Apstra ZTP VM. The TFTP server requires no configuration. The TFTP server serves files out of the /containers_data/tftp directory.

admin@apstra-ztp:/containers_data/tftp$ ls -l
total 56
-rw-r--r-- 1 root root   170 Aug 27 19:36 cumulus_custom.sh
-rw-r--r-- 1 root root    55 Aug 27 19:36 cumulus_license_file
-rw-r--r-- 1 root root    82 Aug 27 19:36 eos_custom.sh
-rw-r--r-- 1 root root    86 Aug 27 19:36 nxos_custom.sh
-rw-r--r-- 1 root root   117 Aug 27 19:36 poap-md5sum
-rw-r--r-- 1 root root  2103 Aug 27 19:36 ztp.json
-rw-r--r-- 1 root root 35100 Aug 27 19:36 ztp.py
admin@apstra-ztp:/containers_data/tftp$

Note

All configuration files are owned by root. You must use sudo to run commands as root.

The ztp.json file contains all the configuration for the Apstra ZTP script ztp.py.

  1. Edit the ztp.json file with vi or nano.

    admin@apstra-ztp:/containers_data/tftp$ sudo nano ztp.json
    
  2. The ztp.json file is organized by the following:

    • Defaults: Values are used for all devices unless more specific keys are defined.
    • Platform: Values are used for all devices for a network platform (“cumulus”, “nxos”, “eos”) unless more specific keys are defined.
    • Model: Values are used for all devices for a specific device model (for example “C3396”).
    • Serial Number: Values are used for a device matching a specific device serial number.
    {
      "defaults": {
        "_comment# if nxos-versions is empty, we will just assume all versions are accepted": 0,
        "nxos-versions": ["9.2(2)", "7.0(3)I7(4)"],
        "nxos-image": "aos_nxos_image.bin",
    ......
       },
    
       "cumulus": {
          "device-root-password": "admin123",
          "custom-config": "cumulus_custom.sh"
       },
    
       "nxos": {
          "custom-config": "nxos_custom.sh"
       },
    
       "C3396": {
          "Device-root-password": "c3396-custom-pass"
       }
    
       "some-serial-number": {
          "nxos-image": "device-specific-nxos-image"
       }
    }
    
  3. The ztp.json file uses the following keys:

    • nxos-versions: Valid versions for NX-OS devices. If a device is not running a version in this list, ZTP upgrades the device with the nxos-image image.

    • nxos-image: This is the filename of the NX-OS image to be loaded if the running version does not match a version in the nxos-versions list.

      By default, the image name is loaded from the ZTP server via TFTP from the ZTP server’s /container_data/tftp/ directory. Beginning in Apstra ZTP version 1.0.0-33, you can use HTTP to transfer ZTP upgrade images to devices. To use any HTTP server for image transfer, enter a valid HTTP URL with IP address for nxos-image. For example:

      "nxos-versions": [ "7.0(3)I7(4)", "9.2(2)"],
      "nxos-image": "nxos.9.2.2.bin",
      

      The above example uses TFTP from the ZTP server to transfer the Cisco NX-OS image.

    • cumulus-versions: Valid versions for Cumulus Linux devices. If a device is not running a version in this list, ZTP upgrades the device with the cumulus-image image.

    • cumulus-image: This is filename of the Cumulus Linux ONIE image to be loaded if the running version does not match a version in the cumulus-versions list.

      By default, the image name is loaded from the ZTP server via TFTP from the ZTP server’s /container_data/tftp/ directory. Beginning in Apstra ZTP version 1.0.0-33, you can use HTTP to transfer ZTP upgrade images to devices. To use any HTTP server for image transfer, enter a valid HTTP URL with IP address for cumulus-image. For example:

      "cumulus-versions": ["3.7.11", "3.7.12"],
      "cumulus-image": "http://192.168.59.4/cumulus-linux-3.7.12-bcm-amd64.bin",
      

      The above example uses HTTP from the ZTP server to transfer the Cumulus Linux image.

    • eos-versions: Valid versions for Arista EOS devices. If a device is not running a version in this list, ZTP upgrades the device with the eos-image image.

    • eos-image: This is the filename of the Arista EOS SWI image to be loaded if the running version does not match a version in the eos-versions list.

      By default, the image name is loaded from the ZTP server via TFTP from the ZTP server’s /container_data/tftp/ directory. Beginning in Apstra ZTP version 1.0.0-33, you can use HTTP to transfer ZTP upgrade images to devices. To use any HTTP server for image transfer, enter a valid HTTP URL with IP address for eos-image. For example:

      "eos-versions": ["4.21.5.1F", "4.22.3M"],
      "eos-image": "http://192.168.59.4/dos_images/EOS-4.22.3M.swi",
      

      The above example uses HTTP from the Apstra server to transfer the Arista EOS image.

    • aos-server: The IP Address of the Apstra server.

    • aos-user: The ZTP user that has been configured on the Apstra server (such as “ztp”), see Configure Apstra ZTP 1.0.0 User on Apstra Server.

    • aos-password: The password configured for the ZTP user on the Apstra

      Server.

    • device-root-password: The ZTP process sets the device root password to this value.

    • license: The filename of the Cumulus Linux license file in the TFTP directory.

    • custom-config: The filename of the custom configuration file in the TFTP directory. See NOS-specific section below for more information.

    • system-agent-params: The information that Apstra uses to create a new user and device system agent on the device.

      • username: The username of an account on the device that is used to create the device system agent. This user is newly created on the device.
      • password: The password for the new account on the device that is used to create the device system agent.
      • agent_type: The agent type, onbox or offbox.
      • installer_options: (only for version 3.1) Options for the device system agent.
        • enable: Set false to let you manually enable the agent. Set true to automatically enable the agent.
        • intent: Set to install to install the device system agent.
        • install_policy: Set to once.
        • install_requirements: Set to false. Not needed for Cumulus Linux, Cisco NX-OS or Arista EOS.
      • job_on_create: (only for version 3.2) Set to install to install the on-box agent. Do not use this option for off-box agents.

Refer to the Swagger Platform REST API documentation for /api/system-agents for all available system-agent-params options.

Note

To skip agent install, only specify username and password under system-agent-params.

Platform Specific Information

Cumulus Linux

Note

Apstra ZTP has limited support for virtual Cumulus VX (CVX) devices.

  • ZTP Cumulus Linux upgrades are not supported on CVX devices. Cumulus Linux versions for CVX device must match cumulus-versions set in ztp.json file.
  • ZTP Logging to an Apstra controller does not work for CVX devices due to an unsupported device serial number (MAC address). This will be addressed in a future Apstra version.

Ensure that the device has sufficient disk space for the new OS image that is copied to the switch as part of the ZTP process. Insufficient disk space causes failure.

cumulus@cumulus:~$ df -h /dev/sda4
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4       5.8

If ZTP is installing Cumulus Linux image, copy the image (for example cumulus-linux-3.7.12-bcm-amd64.bin) to the /containers_data/tftp directory. Modify file permissions to allow for download via tftp or http, for example:

admin@aos-server:/containers_data/tftp$ sudo chmod a+r cl-3.7.12-mlx-amd64.bin
[sudo] password for admin:
admin@aos-server:/containers_data/tftp$

Important

If you are using ONIE to install Cumulus Linux on a device, copy the image to the /containers_data/tftp directory and rename onie-installer or another ONIE download name (for example onie-installer-x86_64-dell_s3000_c2338-r0). When rebooting in ONIE, the device looks for this file on the TFTP server. If the file is not found, ZTP fails.

Example Cumulus Linux Apstra ZTP ztp.json

3.2.0 On-box Agent / Apstra ZTP 1.0.0
{
  "cumulus": {
    "cumulus-versions": ["3.7.11"],
    "cumulus-image": "http://192.168.59.4/cumulus-linux-3.7.11-bcm-amd64.bin",
    "aos-server": "192.168.59.3",
    "aos-user": "ztp",
    "aos-password": "ztp-password",
    "device-root-password": "root-password",
    "license": "cumulus_license_file",
    "custom-config": "cumulus_custom.sh",
    "system-agent-params": {
      "username": "admin",
      "password": "admin-password",
      "agent_type": "onbox",
      "job_on_create": "install"
    }
  }
}
3.3.0a/3.3.0c On-box Agent / Apstra ZTP 2.0.0
{
  "cumulus": {
    "cumulus-versions": ["3.7.12"],
    "cumulus-image": "http://192.168.59.4/cumulus-linux-3.7.12-bcm-amd64.bin",
    "device-root-password": "root-password",
    "license": "cumulus_license_file",
    "custom-config": "cumulus_custom.sh",
    "device-user": "admin",
    "device-user-password": "admin-password",
    "system-agent-params": {
      "agent_type": "onbox",
      "job_on_create": "install"
    }
  }
}

Cumulus Linux Custom Config File

When configuring custom-config for Cumulus Linux devices, refer to the example cumulus_custom.sh which is a bash executable file executed during the ZTP process. It can set the SSH login banner or other system configuration to be set prior to device system agent installation.

#!/bin/bash

sed -i s/"#Banner.*"/"Banner \/etc\/issue.net"/ /etc/ssh/sshd_config

cat >& /etc/issue.net << EOF
Provisioned by AOS
Date: $(date)
EOF

service ssh restart

Restarting Cumulus Linux ZTP

Warning

If an agent is already installed on the device, you must remove the agent either via the GUI device agent installer or manually via the device CLI before restarting the device ZTP process.

admin@cumulus:mgmt-vrf:~$ sudo dpkg -r aos-device-agent
(Reading database ... 25366 files and directories currently installed.)
Removing aos-device-agent (2.0.0-210) ...
Processing triggers for systemd (215-17+deb8u4) ...

For more information, see the documentation on how to Remove the AOS package from Cumulus.

To restart the ONIE install and Cumulus Linux ZTP process:

cumulus@cumulus:mgmt-vrf:~$ sudo curl -o image.bin tftp://<tftpserver-ip>/cumulus-linux-3.7.5-bcm-amd64.bin
cumulus@cumulus:mgmt-vrf:~$ sudo onie-install -a -f -i image.bin
cumulus@cumulus:mgmt-vrf:~$ sudo reboot

To just do the Cumulus Linux ZTP process:

cumulus@cumulus:mgmt-vrf:~$ sudo echo "cumulus" >& /etc/hostname
cumulus@cumulus:mgmt-vrf:~$ sudo net del vrf mgmt && net commit
cumulus@cumulus:mgmt-vrf:~$ sudo ztp -R && sudo ztp -s
cumulus@cumulus:mgmt-vrf:~$ sudo reboot

Troubleshooting Cumulus Linux ZTP

When in ZTP mode, the switch downloads the ztp.py, ztp.json, OS image and license files to the /mnt/persist directory. For diagonstics, take note of the /mnt/persist/aosztp.log file.

You can find additional useful messages in /var/log/syslog (search for ‘ztp’)

Cisco NX-OS

Ensure that the device has sufficient disk space for the new OS image that is copied to the switch as part of the ZTP process. Insufficient disk space causes failure.

switch1# dir bootflash: | include free|total
1296171008 bytes free
3537219584 bytes total

If ZTP is installing Cisco NX-OS image, copy the image (for example nxos.7.0.3.I7.7.bin) to the /containers_data/tftp directory to ensure correct file permissions.

Example Cisco NX-OS Apstra ZTP ztp.json

3.2.0 On-box Agent / Apstra ZTP 1.0.0
{
  "nxos": {
    "nxos-versions": ["9.2(2)"],
    "nxos-image": "http://192.168.0.6/nxos.9.2.2.bin",
    "aos-server": "192.168.0.3",
    "aos-user": "ztp",
    "aos-password": "ztp-password",
    "device-root-password": "admin-password",
    "custom-config": "nxos_custom.sh",
    "system-agent-params": {
      "username": "admin",
      "password": "admin-password",
      "agent_type": "onbox",
      "job_on_create": "install"
    }
  }
}
3.2.0 Off-box Agent / Apstra ZTP 1.0.0
{
  "nxos": {
    "nxos-versions": ["9.2(2)"],
    "nxos-image": "http://192.168.0.6/nxos.9.2.2.bin",
    "aos-server": "192.168.0.3",
    "aos-user": "ztp",
    "aos-password": "ztp-password",
    "device-root-password": "admin-password",
    "custom-config": "nxos_custom.sh",
    "system-agent-params": {
      "username": "admin",
      "password": "admin-password",
      "agent_type": "offbox",
      "operation_mode": "full_control",
      "platform": "nxos",
      "open_options": {
        "proto": "https",
        "port": "443"
      },
      "packages": [
        "aos-deployment-helper-nxos",
        "aosstdcollectors-builtin-nxos",
        "aosstdcollectors-custom-nxos"
      ]
    }
  }
}

This configuration enables secure off-box agent HTTPS (port 443) between the off-box agent on the server and the device API.

Cisco NX-OS Custom Config File

When configuring custom-config for Cisco NX-OS devices, refer to the example nxos_custom.sh, which is a bash executable file executed during the ZTP process. It can execute NX-OS configuration commands to set the SSH login banner or other system configuration to be set prior to device system agent installation.

Note

You must add copp profile strict via the NX-OS custom-config file.

#!/bin/sh

/isan/bin/vsh -c "conf ; copp profile strict ; banner motd ~
########################################################
BANNER BANNER BANNER BANNER BANNER BANNER BANNER BANNER
########################################################
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Donec gravida, arcu vitae tincidunt sagittis, ligula
massa dignissim blah, eu sollicitudin nisl dui at massa.
Aliquam erat volutpat. Vitae pellentesque elit at
pulvinar volutpat. Etiam lacinia derp lacus, non
pellentesque nunc venenatis rhoncus.
########################################################
~"

Cisco NX-OS Offbox Agent Custom Config File

If you’re using Apstra ZTP to prepare a Cisco NX-OS device for use with off-box agents, enable the following NX-OS in the custom-config file configuration commands.

feature nxapi
feature bash-shell
feature scp-server
feature evmed
copp profile strict
nxapi http port 80

The following nxos_custom.sh can be used to add these along with a banner.

#!/bin/sh

/isan/bin/vsh -c "conf ; feature nxapi ; nxapi http port 80 ; feature bash-shell ; feature scp-server ; feature evmed ; copp profile strict ; banner motd ~
########################################################
BANNER BANNER BANNER BANNER BANNER BANNER BANNER BANNER
########################################################
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Donec gravida, arcu vitae tincidunt sagittis, ligula
massa dignissim blah, eu sollicitudin nisl dui at massa.
Aliquam erat volutpat. Vitae pellentesque elit at
pulvinar volutpat. Etiam lacinia derp lacus, non
pellentesque nunc venenatis rhoncus.
########################################################
~"

Restarting Cisco NX-OS ZTP

Warning

If an agent is already installed on the device, remove the agent either via GUI device agent installer or manually via the device CLI before restarting the device ZTP process.

C9K-172-20-65-5# guestshell destroy

Remove remaining AOS data from system
Removing the guest-shell deletes most of the data left by AOS.  Some files are
still on the bootflash:/.aos folder.

C9K-172-20-65-5# delete bootflash:.aos no-prompt

See the documentation about Uninstalling the AOS device agent for more information.

To restart Cisco NX-OS ZTP process:

switch# write erase
switch# reload

Arista EOS

Note

Apstra ZTP has limited support and known issues for virtual Arista EOS (vEOS) devices.

  • ZTP EOS upgrades are not supported on vEOS devices. EOS versions for vEOS device must match eos-versions set in ztp.json file.
  • ZTP Logging to an Apstra controller does not work for vEOS devices due to lack of a device serial number. Future Apstra versions may support this.

Ensure that the device has sufficient disk space for the new OS image that is copied to the switch as part of the ZTP process. Insufficient disk space causes failure.

switch1#dir flash:
Directory of flash:/

<...>

3957878784 bytes total (3074723840 bytes free)

If ZTP is installing Arista EOS image, copy the image (for example EOS-4.22.3M.swi) to the /containers_data/tftp directory.

Example Arista EOS Apstra ZTP ztp.json

3.2.0 On-box Agent / Apstra ZTP 1.0.0
{
  "eos": {
    "eos-versions": ["4.22.3M"],
    "eos-image": "http://192.168.59.4/EOS-4.22.3M.swi",
    "aos-server": "192.168.59.3",
    "aos-user": "ztp",
    "aos-password": "ztp-password",
    "device-root-password": "admin-password",
    "custom-config": "eos_custom.sh",
    "system-agent-params": {
      "username": "admin",
      "password": "admin-password",
      "agent_type": "onbox",
      "job_on_create": "install"
    }
  }
}
3.2.0 Off-box Agent / Apstra ZTP 1.0.0
{
  "eos": {
    "eos-versions": ["4.21.9M"],
    "eos-image": "http://192.168.59.4/EOS-4.21.9M.swi",
    "aos-server": "192.168.59.3",
    "aos-user": "ztp",
    "aos-password": "ztp-password",
    "device-root-password": "admin-password",
    "custom-config": "eos_custom.sh",
    "system-agent-params": {
      "username": "admin",
      "password": "admin-password",
      "agent_type": "offbox",
      "operation_mode": "full_control",
      "platform": "eos",
      "open_options": {
        "proto": "https",
        "port": "443"
      }
    }
  }
}
3.3.0a/3.3.0c On-box Agent / Apstra ZTP 2.0.0
{
  "eos": {
    "eos-versions": [ "4.22.3M" ],
    "eos-image": "http://192.168.59.3/EOS-4.22.3M.swi",
    "custom-config": "eos_custom.sh",
    "device-user": "admin",
    "device-user-password": "admin-password",
    "system-agent-params": {
      "agent_type": "onbox",
      "job_on_create": "install"
    }
  }
}

Arista EOS Custom Config File

When configuring custom-config for Arista EOS devices, refer to the example eos_custom.sh which is a bash executable file executed during the ZTP process. It can execute EOS configuration commands to set the SSH login banner or other system configuration to be set prior to device system agent installation.

#!/bin/sh

FastCli -p 15 -c $'conf t\n service routing protocols model multi-agent\n hardware tcam\n system profile vxlan-routing\n banner login\n
########################################################
UNAUTHORIZED ACCESS TO THIS DEVICE IS PROHIBITED
########################################################\n EOF\n'

Note

During the ZTP process, the EOS banner login is set to text saying “The device is in Zero Touch Provisioning mode …”. By default, this is copied to the permanent configuration by the ZTP script.

To prevent this, you must configure the custom-config pointing to a script (for example eos_custom.sh), which will configure a different banner login or configure no banner login.

There must be a space after any \n.

Note

If you’re using EOS 4.22, we recommend adding the service routing protocols model multi-agent to the device configuration along with any other configuration during ZTP which requires a device reboot to activate (for example system profile vxlan-routing). This ensures that this configuration is applied on reboot and added to the device pristine configuration.

Restarting Arista EOS ZTP

Warning

If an agent is already installed on the device, the remove the Agent extension either via the GUI Device Agent Installer or manually via the device CLI before restarting the device ZTP process.

l2-virtual-001-leaf1#sho extensions
Name                                      Version/Release    Status    Extension
----------------------------------------- ------------------ --------- ---------
aos-device-agent-3.1.0-0.1.205.i386.rpm   3.1.0/0.1.205      A, I      1


A: available | NA: not available | I: installed | NI: not installed | F: forced
l2-virtual-001-leaf1#delete extension:aos-device-agent-3.1.0-0.1.205.i386.rpm
l2-virtual-001-leaf1#no extension aos-device-agent-3.1.0-0.1.205.i386.rpm
l2-virtual-001-leaf1#copy installed-extensions boot-extensions
Copy completed successfully.
l2-virtual-001-leaf1#

See Arista Device Agent for more information.

To restart Arista EOS ZTP process:

localhost# delete flash:zerotouch-config
localhost# write erase
localhost# reload

Juniper Junos

Note

Apstra ZTP 1.0.0-48 used with version 3.3.0 has support for Juniper Junos devices. ZTP is supported for devices which start the ZTP process with Junos version 14.1, 15.1, and 17.3 only.

Apstra manages Juniper Junos devices using off-box agents. Apstra ZTP manages the bootstrap and the lifecycle of Juniper Junos devices. Apstra ZTP handles off-box agent creation, local user creation and can set other system configuration using a custom script.

Ensure that the device has sufficient disk space for the new OS image that is copied to the switch as part of the ZTP process. Insufficient disk space causes failure.

root@leaf001-001-2> show system storage
Filesystem         Size  Used  Avail  Capacity   Mounted on
/dev/gpt/junos     6.0G  1.0G   4.5G       18%  /.mount
<...>

Example Juniper Junos Apstra ZTP ztp.json

3.3.0 Off-box Agent / Apstra ZTP 1.0.0
{
  "junos": {
    "junos-version": [ "18.4R2-S4.10" ],
    "junos-image": "http://192.168.59.4/jinstall-host-qfx-5-18.4R2-S4.10-signed.tgz",
    "aos-password": "admin",
    "aos-user": "aos-admin-password",
    "aos-server": "192.168.59.3",
    "device-root-password": "root-password"
    "system-agent-params": {
      "username": "admin",
      "password": "admin-password",
      "platform": "junos",
      "agent_type": "offbox",
      "operation_mode": "full_control"
    }
  }
}
3.3.0a/3.3.0c Off-box Agent / Apstra ZTP 2.0.0
{
  "junos": {
    "junos-versions": [ "18.4R2-S4.10" ],
    "junos-image": "http://192.168.59.4/jinstall-host-qfx-5-18.4R2-S4.10-signed.tgz",
    "device-root-password": "root-password",
    "device-user": "admin",
    "device-user-password": "admin-password",
    "custom-config": "junos_custom.sh",
    "system-agent-params": {
      "platform": "junos",
      "agent_type": "offbox"
    }
  }
}

Juniper Junos Bootstrap File

Apstra ZTP uses a python script ztp.py to provision the device during ZTP. On Junos, additional configurations are required to allow a python script to run on the device. The script junos_apstra_ztp_bootstrap.sh is used to bootstrap Apstra ZTP on Junos. It downloads the ZTP script and runs it.

Juniper Junos Custom Config File

When configuring custom-config for Juniper Junos devices, refer to the example junos_custom.sh which is a bash executable file executed during the ZTP process. It can set system configuration for example Syslog, NTP, SNMP authentication prior to device system agent installation.

#!/bin/sh

SOURCE_IP=$(cli -c "show conf interfaces em0.0" | grep address | sed 's/.*address \([0-9.]*\).*/\1/')

# Syslog
SYSLOG_SERVER="192.168.59.4"
SYSLOG_PORT="514"
# NTP
NTP_SERVER="192.168.59.4"
# SNMP
SNMP_NAME="SAMPLE"
SNMP_SERVER="192.168.59.3"

# Syslog
cli -c "configure; \
set system syslog host $SYSLOG_SERVER any notice ; \
set system syslog host $SYSLOG_SERVER authorization any ; \
set system syslog host $SYSLOG_SERVER port $SYSLOG_PORT ; \
set system syslog host $SYSLOG_SERVER routing-instance mgmt_junos ; \
commit and-quit"
cli -c "configure; \
set system syslog file messages any notice ; \
set system syslog file messages authorization any ; \
commit and-quit"

# NTP
cli -c "configure; \
set system ntp server $NTP_SERVER routing-instance mgmt_junos ; \
set system ntp source-address $SOURCE_IP routing-instance mgmt_junos ; \
commit and-quit;"

# SNMP
cli -c "configure; \
set snmp name $SNMP_NAME; \
set snmp community public clients $SNMP_SERVER/32 ; \
set snmp community public routing-instance mgmt_junos ; \
set snmp routing-instance-access access-list mgmt_junos ; \
commit and-quit"

Warning

If you set external AAA authentication (for example authentication-order), replicate the device system agent device-user and device-user-password in the AAA system. Otherwise, the device system agent generates an authentication error.

Restarting Juniper Junos ZTP

To erase (zeroize) the device and restart Juniper Junos ZTP process:

root@leaf3> request system zeroize

Enterprise SONiC

Note

Apstra ZTP 2.0.0 used with 3.3.0a/3.3.0c has support for SONiC Enterprise Distribution devices. There is no support for any SONiC devices with earlier versions of Apstra ZTP and Apstra.

Apstra manages Enterprise SONiC devices only using on-box agents. Apstra ZTP manages the bootstrap and the lifecycle of Enterprise SONiC devices. Apstra ZTP handles on-box agent creation, local user creation and can set other system configuration using a custom script.

Ensure that sufficient disk space is available on the switch.

Important

If you’re using ONIE to install Enterprise SONiC on a device, copy the image to the /containers_data/tftp directory and rename onie-installer or another ONIE download name (for example onie-installer-x86_64-dell_z9100_c2538-r0). When rebooting in ONIE, the device looks for this file on the HTTP then TFTP server. Iff the file is not found, ZTP fails. Once ONIE SONiC installation successfully completes, the SONiC device automatically starts ZTP.

Example Enterprise SONiC Apstra ZTP ztp.json

3.3.0a On-box Agent / Apstra ZTP 2.0.0
{
  "sonic": {
    "sonic-versions": [ "SONiC-OS-3.1.0a-Enterprise_Base" ],
    "sonic-image": "http://192.168.59.4/sonic-3.1.0a-bcm.bin",
    "device-root-password": "root-password",
    "device-user": "admin",
    "device-user-password": "admin-password",
    "custom-config": "sonic_custom.sh",
    "system-agent-params": {
      "agent_type": "onbox",
      "job_on_create": "install"
    }
  }
}
3.3.0a On-box Agent / Apstra ZTP 2.0.0
{
  "sonic": {
    "sonic-versions": [ "SONiC-OS-3.1.0a-Enterprise_Base" ],
    "sonic-image": "http://192.168.59.4/sonic-3.1.0a-bcm.bin",
    "device-root-password": "root-password",
    "device-user": "aosadmin",
    "device-user-password": "aosadmin-password",
    "custom-config": "sonic_custom.sh",
    "system-agent-params": {
      "agent_type": "onbox",
      "job_on_create": "install"
    }
  }
}

Note

If you use another device-user besides admin (for example aosadmin) Apstra ZTP creates this new user, but it does not change the password for the default SONiC admin user (password set to YourPaSsWoRd by default).

Enterprise SONiC Custom Config File

When configuring custom-config for Enterprise SONiC devices, refer to the example sonic_custom.sh which is a bash executable file executed during the ZTP process. It can set system configuration for example Radius authentication prior to device system agent installation.

#!/bin/bash

sed -i s/"#Banner.*"/"Banner \/etc\/issue.net"/ /etc/ssh/sshd_config

cat >& /etc/issue.net << EOF
Provisioned by AOS
Date: $(date)
EOF

service ssh restart

Restarting Enterprise SONiC ZTP

To restart the SONiC ZTP process, use the sudo ztp enable and sudo ztp run commands.

admin@sonic:~$ sudo ztp enable
admin@sonic:~$ sudo ztp run
ZTP will be restarted. You may lose switch data and connectivity, continue?[yes/NO] yes
admin@sonic:~$

Monitoring DHCP, TFTP and HTTP Logs

From the Apstra ZTP VM, you may monitor logs from the DHCP Server (if used), the TFTP server and the HTTP server with the docker logs commands for the apstra_ztp_dhcpd_1, apstra_ztp_tftp_1, and apstra_ztp_http_1 Docker containers.

admin@apstra-ztp:~$ docker logs --tail 10 apstra_ztp_dhcpd_1
DHCPACK on 192.168.59.30 to 50:00:00:0a:00:00 (cumulus) via eth0
DHCPREQUEST for 192.168.59.24 from 50:00:00:03:00:00 (spine1) via eth0
DHCPACK on 192.168.59.24 to 50:00:00:03:00:00 (spine1) via eth0
DHCPREQUEST for 192.168.59.26 from 00:50:00:00:06:00 (ubuntu-xenial) via eth0
DHCPACK on 192.168.59.26 to 00:50:00:00:06:00 (ubuntu-xenial) via eth0
DHCPDISCOVER from 50:00:00:0b:00:00 via eth0
DHCPOFFER on 192.168.59.31 to 50:00:00:0b:00:00 via eth0
DHCPREQUEST for 192.168.59.31 (192.168.59.4) from 50:00:00:0b:00:00 via eth0
DHCPACK on 192.168.59.31 to 50:00:00:0b:00:00 via eth0
DHCPRELEASE of 192.168.59.31 from 50:00:00:0b:00:00 via eth0 (found)
admin@apstra-ztp:~$

The following log messages indicate successful download of the various files. If none or some of these are seen this is an indication the ZTP process did not run correctly. Review your ZTP configuration against what has been detailed in this document:

root@apstra-ztp:/containers_data/tftp# docker logs --tail 10 apstra_ztp_tftp_1
2019-10-07T15:14:14.695422+00:00 e4f048f72774 in.tftpd[4193]: RRQ from 192.168.59.10 filename ztp.py
2019-10-07T15:14:14.955079+00:00 e4f048f72774 in.tftpd[4194]: RRQ from 192.168.59.10 filename ztp.json
2019-10-07T15:14:15.037284+00:00 e4f048f72774 in.tftpd[4195]: RRQ from 192.168.59.10 filename ztp.py
2019-10-07T15:14:15.703449+00:00 e4f048f72774 in.tftpd[4197]: RRQ from 192.168.59.10 filename cumulus_license_file
2019-10-07T15:14:36.020385+00:00 e4f048f72774 in.tftpd[4218]: RRQ from 192.168.59.10 filename cumulus_custom.sh
admin@apstra-ztp:~# docker logs --tail 10 apstra_ztp_http_1
2019/12/16 18:36:43 [error] 6#6: *1 directory index of "/usr/share/nginx/html/" is forbidden, client: 10.1.252.70, server: localhost, request: "GET / HTTP/1.1", host: "192.168.59.6"
10.1.252.70 - - [16/Dec/2019:18:36:43 +0000] "GET / HTTP/1.1" 403 555 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" "-"
2019/12/16 18:36:43 [error] 6#6: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 10.1.252.70, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "192.168.59.6", referrer: "http://192.168.59.6/"
10.1.252.70 - - [16/Dec/2019:18:36:43 +0000] "GET /favicon.ico HTTP/1.1" 404 555 "http://192.168.59.6/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" "-"
10.1.252.70 - - [16/Dec/2019:18:51:46 +0000] "GET /EOS-4.21.5.1F.swi HTTP/1.1" 200 749757299 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" "-"
192.168.59.101 - - [16/Dec/2019:19:16:08 +0000] "GET /EOS-4.21.5.1F.swi HTTP/1.1" 200 749757299 "-" "lftp/4.6.4" "-"
admin@apstra-ztp:~#

See the docker logs --help command for more information.

ZTP Logs

Note

There is no support for ZTP logs for virtual Arista EOS (vEOS) and virtual Cumulus VX (CVX) devices.

When you execute the ZTP script, logs are sent to the Apstra server via API.

Monitoring Apstra ZTP 1.0.0

From the left navigation menu, navigate to Devices > ZTP Logs to monitor the Apstra ZTP 1.0.0 process.

_images/apstra_ztp_3_310.png

To view logs, click the serial number of the device.

_images/apstra_ztp_4_310.png

You can monitor device system agents installation. From the left navigation menu, navigate to Devices > System Agent > Agents. As the agent installation completes the status changes from “In Process” to “Success”.

_images/apstra_ztp_5_310.png

Monitoring Apstra ZTP 2.0.0

Starting with Apstra ZTP 2.0.0, log files for all processes are located in the /containers_data/logs directory. Apstra ZTP 2.0.0 logs are not available from the GUI.

root@apstra-ztp:/containers_data/logs# ls -l
total 7132
-rw-r--r-- 1 root root 6351759 Oct 28 17:47 debug.log
drwxr-xr-x 2 root root    4096 Oct 27 19:20 devices
-rw------- 1 root root       0 Oct 23 20:02 dhcpd.leases
-rw-r--r-- 1 root root  926980 Oct 28 17:39 info.log
-rw------- 1 root root      58 Oct 23 20:02 README
-rw------- 1 root root     469 Oct 27 02:13 rsyslog.log
root@apstra-ztp:/containers_data/logs# tail info.log
2020-10-28 17:16:38,786     root.status    INFO Incoming: dhcpd dhcpd[18]: DHCPACK on 192.168.59.9 to 04:f8:f8:6b:36:91 via eth0
2020-10-28 17:18:04,299     root.status    INFO Incoming: dhcpd dhcpd[18]: DHCPREQUEST for 192.168.59.9 from 04:f8:f8:6b:36:91 via eth0
2020-10-28 17:18:04,300     root.status    INFO Incoming: dhcpd dhcpd[18]: DHCPACK on 192.168.59.9 to 04:f8:f8:6b:36:91 via eth0
2020-10-28 17:19:29,250     root.status    INFO Incoming: dhcpd : -- MARK --
2020-10-28 17:19:29,442     root.status   ERROR Failed to update status of all containers: /api/ztp/service 404 b'{"errors":"Resource not found"}'
2020-10-28 17:33:29,353     root.status    INFO Incoming: tftp : -- MARK --
2020-10-28 17:33:29,538     root.status   ERROR Failed to update status of all containers: /api/ztp/service 404 b'{"errors":"Resource not found"}'
2020-10-28 17:33:34,768     root.status    INFO Incoming: status : -- MARK --
2020-10-28 17:39:29,349     root.status    INFO Incoming: dhcpd : -- MARK --
2020-10-28 17:39:29,539     root.status   ERROR Failed to update status of all containers: /api/ztp/service 404 b'{"errors":"Resource not found"}'
root@apstra-ztp:/containers_data/logs#