Upgrade In-Place
To upgrade the Apstra server you need Apstra OS admin user privileges and Apstra admin user group permissions.
Pre-Upgrade Validation (In-Place)
- Confirm that you are upgrading to a supported version.
-
Log into the Apstra server as admin (For example, if your Apstra
server IP address were 10.28.105.3, the command would be
ssh admin@10.28.105.3
). -
Check that memory utilization is less than 50% to confirm that the VM has
enough memory to hold two versions of the Apstra software at the same time.
Check resources by running the command
free -h
.admin@aos-server:~$ free -h total used free shared buff/cache available Mem: 15G 5.1G 8.8G 7.8M 1.8G 10G Swap: 3.8G 0B 3.8G
- If utilization is > 50%, gracefully shutdown the Apstra server, add resources, then restart the Apstra server.
-
Check that the server is active and has no issues by running the command
service aos status
.admin@aos-server:~$ service aos status * aos.service - LSB: Start AOS management system Loaded: loaded (/etc/init.d/aos; generated) Active: active (exited) since Thu 2021-10-28 17:11:27 UTC; 24h ago Docs: man:systemd-sysv-generator(8) Process: 1157 ExecStart=/etc/init.d/aos start (code=exited, status=0/SUCCESS)
-
Review each blueprint to confirm that all Service Config has
succeeded. If necessary, undeploy and remove devices from the blueprint to
resolve any pending or failed service config.
- Review each blueprint for probe anomalies, and resolve them as much as possible. Take notes of any remaining anomalies.
- Refer to Qualified Device and NOS to verify that the devices' NOS versions are qualified on the new Apstra version. Upgrade or downgrade as needed, to one of the supported versions.
- Remove any device AAA configuration. During device upgrade, configured device agent credentials are required for SSH access.
- Remove any configlets used to configure firewalls. If you use FW's Routing Engine filters on devices, you'll need to update them to include the IP address of the new controller and worker VMs.
-
Run an Apstra System "Check" job for all devices (Devices > Agents) and
verify that all job states are SUCCESS.
Note:
To upgrade device system agents, Apstra software must SSH to all devices using the configured credentials that were used when creating the device system agent. To verify SSH using these credentials, we recommend running an Apstra System "Check" job for all devices. If any check job fails, resolve the issue before proceeding with the Apstra upgrade.
-
As root user, back up the Apstra server by running the command
sudo aos_backup
.Note:admin@aos-server:~$ sudo aos_backup [sudo] password for admin: ==================================================================== Backup operation completed successfully. ==================================================================== New AOS snapshot: 2021-10-29_18-58-56
-
Copy the backup files from
/var/lib/aos/snapshot/<shapshot_name>
to an external location.
Deploy New Apstra Server (In-Place)
-
Download the Apstra installer package for your platform from Juniper Support Downloads
(
aos_4.0.1-1045.run.gz
for example) and transfer it to the Apstra server.admin@aos-server:~$ ls -l total 823228 -rw------- 1 admin admin 842984302 Oct 26 00:44 aos_4.0.1-1045.run.gz
-
Unzip the Apstra installer package.
admin@aos-server:~$ gunzip aos_4.0.1-1045.run.gz admin@aos-server:~$ ls -l total 823108 -rw------- 1 admin admin 842860338 Oct 26 00:44 aos_4.0.1-1045.run
- If you're using an Apstra cluster (off-box agents, IBA probes), download the installer package to the worker nodes as well. You'll upgrade the worker nodes in a later step.
- Log into the Apstra server as admin.
-
Run the
sudo bash aos_<aos_version>.run
command, where<aos_version>
is the version of the run file. For example, if the version is4.0.1-1045
the command would besudo bash aos_4.0.1-1045.run
as shown below.admin@aos-server:~$ sudo bash aos_4.0.1-1045.run [sudo] password for admin: Verifying archive integrity... All good. Uncompressing AOS installer 100% ===================================================================== Backup operation completed successfully. ===================================================================== AOS[2021-10-28_01:28:52]: Loading AOS 4.0.1-1045 image AOS[2021-10-28_01:29:44]: Initiating upgrade pre-checker AOS[2021-10-28_01:29:45]: Initiating docker library import DONE AOS[2021-10-28_01:30:59]: Preparing to retrieve data from running AOS Server. DONE AOS[2021-10-28_01:31:11]: Retrieving data from running AOS Server. This step can take up to 10 minutes DONE AOS[2021-10-28_01:34:30]: Importing retrieved state to AOS pre-checker. This step can take up to 20 minutes DONE AOS[2021-10-28_01:34:35]: Waiting for blueprint <3db44826-807f-4ab9-8ca0-e25040af7ef6> processing to finish. DONE AOS[2021-10-28_01:34:42]: Waiting for blueprint <964211f7-7f3c-4b0a-b6b7-137790c461f5> processing to finish. DONE Summary saved to /tmp/aos-upgrade-config-summary-2021.10.28-013449
When you run this command, if any previous Apstra versions are detected, the script enters upgrade mode instead of new installation mode. The new Docker container installs next to the Docker containers from the previous version. The script imports the data from the previous version and migrates it to the Apstra SysDB on the new version.
You’ll be shown a summary of configuration changes that will be pushed to devices as part of the upgrade
Apstra Upgrade Summary ======================================================================= This is a summary of configuration pushed to devices logically grouped into sections. Use 'q' to exit this view. For more device specific configurations, use the menu after quitting this view BLUEPRINT: 3db44826-807f-4ab9-8ca0-e25040af7ef6 (BP2) BLUEPRINT: 964211f7-7f3c-4b0a-b6b7-137790c461f5 (BP1) Section: FULL_CONFIG ~~~~~~~~~~~~~~~~~~~~ Full configuration apply. Configuration Role Systems ================================================================================== Spine spine2 [525400E3EF4A, 10.28.105.10] spine1 [52540006D434, 10.28.105.9] ---------------------------------------------------------------------------------- Leaf l2-virtual-ext-001-leaf1 [5254006260B2, 10.28.105.11] l2-virtual-ext-002-leaf1 [5254009D09D6, 10.28.105.12]
As of Apstra version 4.0.1, the Apstra Upgrade Summary shows information separated by device roles (superspine, spine, leaf, leaf pair, and access switch for example). If an incremental config was applied instead of a full config, more details are displayed about the changes.
-
After you've reviewed the summary, enter
q
to exit the summary. The AOS Upgrade: Interactive Menu appears where you can access additional information, and continue or quit the upgrade.AOS Upgrade: Interactive Menu ================================================== <Device SN> - display config changes using a specific device serial number (s)ummary - display config change summary (l)ist - list all devices with config changes (d)ump - dump all config changes to a file (c)ontinue - continue with AOS upgrade (q)uit - quit AOS upgrade aos-upgrade (h for help)#
CAUTION:The Apstra Reference Design in the new Apstra release may have changed in a way that invalidates configlets. To avoid unexpected outcomes, verify that your configlets don’t conflict with the newly rendered config. If you need to update your configlets, quit the upgrade, update your configlets, then run the upgrade again.
-
If you want to continue with the upgrade after reviewing pending changes,
enter
c
. The older Apstra version is deleted and the new Apstra version is activated on the server. When the upgrade is complete, you can check the version by navigating to Platform > About in the Apstra GUI.CAUTION:Upgrading the Apstra server is a disruptive process. When you upgrade in-place (same VM) and continue with the upgrade from this point, you cannot roll back the upgrade. The only way to return to the previous version is to reinstall a new VM with the previous version and restore the database from the backup that you previously made.
-
If you want to stop the upgrade, enter
q
to abort the process. If you quit at this point and later decide to upgrade, you must start the process from the beginning. - If you're using an Apstra cluster, the worker nodes disconnect from the Apstra controller and change to the FAILED state. This state means that off-box agents and the IBA probe containers that are on the worker nodes are not available; devices that are managed by the off-box agents do remain in service though. After you upgrade the agents in a later step, you'll upgrade the worker nodes in your Apstra cluster and the agents and/or probes will become available.
Change Operation Mode to Normal (In-place)
When you initiate an Apstra server upgrade, the operation mode changes from Normal to Maintenance automatically. After you've completed the upgrade you must manually change the mode back to Normal.
-
From the left navigation menu in the Apstra GUI, navigate to
Platform > Apstra Cluster > Cluster
Management.
-
Click the Change Operation Mode button, select
Normal, then click Update.
When you change the mode to Normal, any configured
off-box agents are activated, but you must initiate the upgrade of any
on-box agents (in the next section).
You can also access the Cluster Management page from the lower left section of any page. You have continuous platform health visibility from here as well, based on the colors.
Note:This feature has been classified as a Juniper Apstra Technology Preview feature. These features are "as is" and voluntary use. Juniper Support will attempt to resolve any issues that customers experience when using these features and create bug reports on behalf of support cases. However, Juniper may not provide comprehensive support services to Tech Preview features.
For additional information, refer to the Juniper Apstra Technology Previews page or contact Juniper Support.
From the bottom of the left navigation menu, click one of the dots, then click Operation Mode to go to Cluster Management. Click the Change Operation Mode button, select Normal, then click Update.
Because they're still in the process of upgrading, the agents won't be connected. When the upgrade has completed, the agents reconnect to the server and come back online. On the blueprint dashboard the Liveness anomalies for spine and leaf will also resolve.
Upgrade On-box Agents (In-Place)
Devices are still connected to the old Apstra server. (see Devices > Managed Devices). By upgrading the agents, the devices disconnect from the old Apstra server and connect to the new Apstra server.
When you initiate agent upgrade you cannot roll back to the previous version. The only way to return to the previous version is to reinstall a new VM with the previous version and restore the database from the backup that you previously made.
- Log into the Apstra GUI as user admin.
-
From the left navigation menu, navigate to Devices > System
Agents > Agents and select the device(s) to upgrade (up
to 100 devices at a time as of Apstra version 4.0.1). You can upgrade
multiple on-box agents at the same time, but the order of device upgrade is
important.
- Upgrade agents for superspines first.
- Upgrade agents for spines second.
- Upgrade agents for leafs third.
- Click the Install button to initiate the install process. The job state changes to IN PROGRESS. If agents are using a previous version of the Apstra software, they are automatically upgraded to the new version. Then they connect to the server and push any pending configuration changes to the devices. Telemetry also resumes, and the job states change to SUCCESS.
-
In the Liveness section on the blueprint dashboard confirm that you don't
have any device anomalies .
Note:
If you need to roll back to the previous Apstra version after initiating agent upgrade, you must build a new VM with the previous Apstra version and restore the configuration to that VM. For assistance, contact Juniper Support.
Upgrade Worker Nodes (Apstra Cluster Only)
If you're using an Apstra cluster (for off-box agents and/or IBA probes), you need to upgrade the worker nodes as well as the controller node that you have already upgraded.
- If you didn't download the Apstra installer package to the worker nodes when you downloaded it to the Apstra server, do that now.
-
From each Apstra worker node, run the
sudo bash aos_<aos_version>.run
command, where<aos_version>
is the version of the run file. For example, if the version is4.0.1-1045
the command would besudo bash aos_4.0.1-1045.run
(no options). This is the same file you used to upgrade the controller. There are no prompts during the worker node upgrade.admin@aos-server:~$ sudo bash aos_4.0.1-1045.run Verifying archive integrity... All good. Uncompressing AOS installer 100% ===================================================================== Backup operation completed successfully. ===================================================================== AOS[2021-10-28_23:15:29]: Removing installed (3.3.0.2-46) AOS package 6babb56be259: Loading layer [==================================================>] 65.51MB/65.51MB 4b61bdd19e28: Loading layer [==================================================>] 5.632kB/5.632kB bd9a55afbdce: Loading layer [==================================================>] 44.03kB/44.03kB f495d3ee1163: Loading layer [==================================================>] 13.31kB/13.31kB 0222f30a89f7: Loading layer [==================================================>] 1.114GB/1.114GB 15f1b266e91a: Loading layer [==================================================>] 3.072kB/3.072kB 3cebea5ed20e: Loading layer [==================================================>] 5.632kB/5.632kB 07d63988038c: Loading layer [==================================================>] 25.6kB/25.6kB 82bbad94c148: Loading layer [==================================================>] 88.41MB/88.41MB 30c5cc7507d8: Loading layer [==================================================>] 58.8MB/58.8MB c3a6272b640d: Loading layer [==================================================>] 242.4MB/242.4MB 236ebbddf13a: Loading layer [==================================================>] 118.3MB/118.3MB fcd29376258b: Loading layer [==================================================>] 25.77MB/25.77MB 214893e2d628: Loading layer [==================================================>] 4.608kB/4.608kB Loaded image: aos:4.0.1-1045 AOS[2021-10-28_23:16:15]: Installing AOS 4.0.1-1045 package