ON THIS PAGE
Upgrade Procedure
Below are instructions for upgrading Control Center. This is the procedure that should normally be followed. An alternative procedure is described in the next chapter.
Be sure to refer to the current Paragon Active Assurance Installation Guide.
-
Make backups according to the Operations Guide, chapter Backing Up Product Data, starting with the section "Backing Up the PostgreSQL Database".
In addition, back up the file
/etc/netrounds/secret_keyand keep it with the backup archives:cp /etc/netrounds/secret_key /etc/netrounds/secret_key-backup
Note:Taking backup of certificates and secret key files are required only if you are upgrading from Release 4.1 or earlier releases.
-
Make backups of the disks or disk partitions used by Control Center (or take a VM snapshot).
-
Disable the
apache2andnetrounds-callexecuterservices completely:sudo systemctl disable apache2 sudo systemctl disable netrounds-callexecuter
-
Stop all Paragon Active Assurance services:
sudo systemctl stop "netrounds-*" apache2 openvpn@netrounds
-
Verify that all services are stopped:
sudo systemctl status "netrounds-*" apache2 openvpn@netrounds
If some service is not in status stopped, run the following command once more:
sudo systemctl stop "netrounds-*" apache2 openvpn@netrounds
If the problem persists, please contact Juniper technical support.
-
Upgrade Ubuntu from version 18.04 to version 22.04. This can be done in two steps as described on the pages linked below:
- Upgrade from 18.04 to 20.04 (Click the "Clear message" link to bring up the page with instructions.)
- Upgrade from 20.04 to 22.04 (Here, version 21.10 is taken as starting point, but it is equally possible to start from 20.04.)
-
Verify the integrity of the tarball containing the new Control Center version:
# Compute the checksum for the tar file and verify that it is equal to the SHA256 # checksum provided on the download page export CC_VERSION=4.4.0.12 sha256sum paa-control-center_${CC_VERSION}.tar.gz -
Unpack the Control Center tarball:
tar -xzf paa-control-center_${CC_VERSION}.tar.gz -
Install new Control Center packages.
Note the following:
- In the file
/etc/netrounds/netrounds.confyou need to replace the settingPASSWORD_RESET_TIMEOUT_DAYSwithPASSWORD_RESET_TIMEOUT. The former gives the timeout in days, whereas the latter has seconds as unit. - In the file
/etc/netrounds/netrounds.confyou can also optionally configure theSPEEDTEST_ADDRESSsetting (if you are going to use Speedtest). This can either point to the same IP address thatSITE_URLresolves to, or it can have a hostname of its own.
Warning:You will be prompted about overwriting existing configuration files. Before proceeding, please read all the information about settings below.
Note:- We highly recommend that you first inspect the difference between your old configuration and the new one using the "D" choice. In most cases you will then want to keep your old settings by pressing "N" (do not overwrite).
- New optional and updated settings may be available in the example configuration files provided in the packages. We recommend that you review these and add new options as appropriate for your installation.
Warning:For the Apache configuration files found in
/etc/apache2/sites-available/
you need to press "Y", which is the "package maintainer's version".
If you have installed proper SSL certificates (as recommended) instead of the default snakeoil ones, you will have to modify the file again to point to the correct path in the
SSLCertificateFileandSSLCertificateKeyFilesettings after the Debian package installation has completed. See the Installation Guide, chapter Service Configuration, section "SSL Certificate Configuration".sudo apt-get update sudo apt-get install ./paa-control-center_${CC_VERSION}/*.deb - In the file
-
Run the database migration:
Warning:If you have changed the database password from the default, make sure you also change this in the
db-passwordsetting in the/etc/netrounds/plugin.yamlfile before runningncc migrate. Otherwise, the command will fail.Note:- This is a sensitive command, and care should be taken when executing it on a remote
machine. In such a scenario it is strongly recommended that you use a program like
screen(generally installed by default on popular Linux distributions) ortmux(runsudo apt-get install tmuxto install) so that the migrate command will continue running even if the ssh session breaks. - This command takes considerable time to execute.
sudo ncc migrate
- This is a sensitive command, and care should be taken when executing it on a remote
machine. In such a scenario it is strongly recommended that you use a program like
-
Change the
/var/lib/netrounds/openvpnowner tonetrounds:sudo chown -R netrounds: /var/lib/netrounds/openvpn
-
Restart all Paragon Active Assurance services:
sudo ncc services restart
-
Install the new Test Agent repository and plugins.
The plugins are used by Test Agent Applications.
TA_APPLIANCE_BUILD=4.4.0.12 TA_APPLICATION_BUILD=4.4.0.12 PLUGIN_BUILD=4.4.0.24 # Compute checksums for the repositories and verify that they match the # SHA256 checksums provided on the download page sha256sum paa-test-agent_${TA_APPLIANCE_BUILD}_all.deb sha256sum paa-test-agent-application_${TA_APPLICATION_BUILD}_all.deb sha256sum paa-test-agent-plugins_${PLUGIN_BUILD}_all.deb # Start the installation sudo apt-get install ./paa-test-agent_${TA_APPLIANCE_BUILD}_all.deb sudo apt-get install ./paa-test-agent-application_${TA_APPLICATION_BUILD}_all.deb sudo apt-get install ./paa-test-agent-plugins_${PLUGIN_BUILD}_all.deb
-
Enable services as follows:
sudo ncc services enable apache2 sudo ncc services enable kafka sudo ncc services enable callexecuter
Note:If the TimescaleDB and Metrics services are expected to be running after the upgrade, you need to enable these as well since they are not enabled by default:
sudo ncc services enable timescaledb sudo ncc services enable metrics
-
Restart all Paragon Active Assurance services:
Note:You must do this to get the services up and running again after the upgrade.
sudo ncc services restart
-
To activate the new configuration, you also need to run:
sudo systemctl reload apache2
-
Check that the system is up and running with the commands
ncc status sudo systemctl status "netrounds-*"
-
Run the following script to enable the latest version of all plugins in all accounts:
export PORT=49900 # Replace with your desired port number while true; do if netstat -lnt | grep ":$PORT " > /dev/null; then echo "$(date): Plugin service is listening on port $PORT" echo "Enabling latest plugins for all accounts" sudo ncc plugins edit enabled-version --all-plugins --latest-version --all-accounts --exit-on-failure=false --verbose break fi echo "Waiting for plugin service listening on $PORT" sleep 3 doneIf you encounter the following error after you run the script, create a support case by attaching the script output in order to troubleshoot the error.
2023-11-17T09:22:46Z ERR ../../app/api/handlers/update_plugin.go:246 > Failed to change enabled plugin version error="unknown account shortname account_name" host=ip-10-0-0-11 service=plugin-service short_name=account_name src=core
For more information on how to manage plugins using the Control Center CLI, see the in-app help under "Plugins".
- Log in to the Control Center GUI and go to the Test Agents view. Next to each Test Agent for which an upgrade is available, an up-arrow icon appears. Click that icon to go ahead with the upgrade.
Upgrading Postgres
In this section we will upgrade the netrounds and paa-plugins
databases to use Postgres 14 instead of Postgres 10.
Work through the following steps:
-
Verify that Postgres 10 is currently used:
sudo -u postgres psql netrounds -c "SELECT version();"
The expected output is:
version -------------------------------------------------------------------------------------------------------------------------------------- PostgreSQL 10.23 (Ubuntu 10.23-0ubuntu0.18.04.2) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit (1 row)
If the above command reports Postgres 14, don't take any further actions.
-
Stop Control Center servies:
sudo ncc services stop
-
Stop both Postgres clusters:
sudo -u postgres /usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/main stop sudo -u postgres /usr/lib/postgresql/14/bin/pg_ctl -D /var/lib/postgresql/14/main stop sudo systemctl stop postgresql -
Before doing the actual upgrade, run the following check:
cd /tmp sudo -u postgres \ /usr/lib/postgresql/14/bin/pg_upgrade \ -b /usr/lib/postgresql/10/bin \ -B /usr/lib/postgresql/14/bin \ -d /etc/postgresql/10/main \ -D /etc/postgresql/14/main \ -p 5432 \ -P 5434 \ --checkIf the
pg_upgradecheck reports errors that are not easily fixable, contact Juniper technical support.If there are no errors and
pg_upgradereported that the clusters are compatible, you can proceed with the upgrade. -
Upgrade Postgres:
sudo -u postgres \ /usr/lib/postgresql/14/bin/pg_upgrade \ -b /usr/lib/postgresql/10/bin \ -B /usr/lib/postgresql/14/bin \ -d /etc/postgresql/10/main \ -D /etc/postgresql/14/main \ -p 5432 \ -P 5434
-
Change the port assignment so that Postgres 10 uses port 5434 and Postgres 14 uses port 5432.
Update the value of port in
/etc/postgresql/14/main/postgresql.confto 5432:sudo vim /etc/postgresql/14/main/postgresql.conf
Update the value of port in
/etc/postgresql/10/main/postgresql.confto 5434:sudo vim /etc/postgresql/10/main/postgresql.conf
-
Start the
postgresqlservice:sudo systemctl start postgresql
-
Alter the password for the
netroundsuser:sudo -u postgres psql -c "alter user netrounds with encrypted password 'netrounds';"
-
Run the statistics optimizer:
sudo -u postgres /usr/lib/postgresql/14/bin/vacuumdb --all --analyze-in-stages
-
Start Control Center services:
sudo ncc services start
-
Verify that Postgres 14 is now being used for
netroundsdatabase:sudo -u postgres psql netrounds -c "SELECT version();"
Expected output:
version ---------------------------------------------------------------------------------------------------------------------------------------- PostgreSQL 14.8 (Ubuntu 14.8-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, 64-bit (1 row)
sudo DJANGO_SETTINGS_MODULE=netrounds.settings.local django-admin shell -c "from django.db import connection; print(connection.cursor().connection.server_version)"
Expected output:
140008
-
Verify that Control Center is functional and remove old cluster data:
sudo -u postgres sh delete_old_cluster.sh
-
Optionally remove postgresql-10 and postgresql-12 client/server packages:
sudo apt purge postgresql-client-10 postgresql-10 postgresql-client-12 postgresql-12
Upgrading ConfD
If the ConfD service is installed in Control Center, do the following:
-
Remove the existing version of
paa-netconf-yangby running:sudo apt-get remove paa-netconf-yang python3-pycryptodomex python3-pysnmp
- Follow the instructions in the Install ConfD section of the Installation Guide for version 4.2 (chapter Installing Control Center and Related Tasks
Troubleshooting
- Password Authentication Failed For User
- Target WSGI Script Not Found
- Same Origin Policy Disallows Reading the Remote Resource
- openvpn@netrounds Service Fails to Find dh.pem File
- openvpn@netrounds Service Reports Weak CA Key
- Control Center Installation Fails Due to the Conflict with the containers-common Package
Password Authentication Failed For User
If the ncc migrate command fails with an error message
Failed to connect to database error="pq: password authentication failed for user \"netrounds\"" db-host=localhost db-name=paa-plugins db-port=5432 ...
you must update the variable db-password in the
/etc/netrounds/plugin.yaml file as explained in the warning above. Edit this
file and then rerun ncc migrate.
Target WSGI Script Not Found
If you accidentally selected "N" for the Apache configuration files (see this step above) and got an error message like the one below
[wsgi:error] [pid 29401:tid 140567451211520] [client 127.0.0.1:37172] Target WSGI script not found or unable to stat: /usr/lib/python2.7/dist-packages/netrounds/wsgi.py
run the following commands to get back on track:
export CC_VERSION=4.4.0.12
dpkg-deb --fsys-tarfile paa-webapp_${CC_VERSION}_all.deb | tar -x --wildcards ./etc/apache2/sites-available/*.conf --strip-components 4
sudo mv netrounds*.conf /etc/apache2/sites-available/
sudo chown -R root:root /etc/apache2/sites-available/
sudo systemctl reload apache2This overwrites the old configuration with the new one in the updated package.
Again, if you have installed proper SSL certificates (as recommended) instead of the
default snakeoil ones, you will have to modify the file again to point to the correct path
in the SSLCertificateFile and SSLCertificateKeyFile
settings after the Debian package installation has completed. See the Installation Guide,
chapter Service Configuration, section "SSL
Certificate Configuration".
Same Origin Policy Disallows Reading the Remote Resource
This or some similar error may occur if you have set SITE_URL and
SPEEDTEST_ADDRESS to different values in
/etc/netrounds/netrounds.conf. You then need to change
ALLOWED_ORIGINS in /etc/netrounds/restol.conf to allow
both of these values in the restol.conf file. The simplest way to achieve
this is to delete any value previously assigned to ALLOWED_ORIGINS. That
setting will then get a default value which allows SITE_URL and
SPEEDTEST_ADDRESS as found in
/etc/netrounds/netrounds.conf.
openvpn@netrounds Service Fails to Find dh.pem File
If this happens, output like that below will be given:
Sep 04 09:56:47 ip-10-0-0-11 systemd[1]: Stopped OpenVPN connection to netrounds. Sep 04 09:56:47 ip-10-0-0-11 systemd[1]: Starting OpenVPN connection to netrounds... Sep 04 09:56:47 ip-10-0-0-11 ovpn-netrounds[1741772]: --cipher is not set. Previous OpenVPN version defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers. Sep 04 09:56:47 ip-10-0-0-11 ovpn-netrounds[1741772]: Options error: --dh fails with '/var/lib/netrounds/openvpn/dh.pem': No such file or directory (errno=2) Sep 04 09:56:47 ip-10-0-0-11 ovpn-netrounds[1741772]: Options error: Please correct these errors. Sep 04 09:56:47 ip-10-0-0-11 ovpn-netrounds[1741772]: Use --help for more information. Sep 04 09:56:47 ip-10-0-0-11 systemd[1]: openvpn@netrounds.service: Main process exited, code=exited, status=1/FAILURE Sep 04 09:56:47 ip-10-0-0-11 systemd[1]: openvpn@netrounds.service: Failed with result 'exit-code'. Sep 04 09:56:47 ip-10-0-0-11 systemd[1]: Failed to start OpenVPN connection to netrounds. Sep 04 09:56:52 ip-10-0-0-11 systemd[1]: openvpn@netrounds.service: Scheduled restart job, restart counter is at 9. Sep 04 09:56:52 ip-10-0-0-11 systemd[1]: Stopped OpenVPN connection to netrounds.
To resolve, run the following commands:
sudo mv /var/lib/netrounds/openvpn/certs_by_serial/dh1024.pem /var/lib/netrounds/openvpn/ sudo chown -R netrounds: /var/lib/netrounds/openvpn sudo ncc services restart openvpn
openvpn@netrounds Service Reports Weak CA Key
In this case, the output will look like this:
Aug 30 12:49:34 ip-10-0-0-11 systemd[1]: Starting OpenVPN connection to netrounds... Aug 30 12:49:34 ip-10-0-0-11 ovpn-netrounds[1848602]: --cipher is not set. Previous OpenVPN version defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers. Aug 30 12:49:34 ip-10-0-0-11 ovpn-netrounds[1848602]: WARNING: file '/var/lib/netrounds/openvpn/private/server.key' is group or others accessible Aug 30 12:49:34 ip-10-0-0-11 ovpn-netrounds[1848602]: OpenVPN 2.5.5 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jul 14 2022 Aug 30 12:49:34 ip-10-0-0-11 ovpn-netrounds[1848602]: library versions: OpenSSL 3.0.2 15 Mar 2022, LZO 2.10 Aug 30 12:49:34 ip-10-0-0-11 ovpn-netrounds[1848602]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts Aug 30 12:49:34 ip-10-0-0-11 systemd[1]: Started OpenVPN connection to netrounds. Aug 30 12:49:34 ip-10-0-0-11 ovpn-netrounds[1848602]: OpenSSL: error:0A00018E:SSL routines::ca md too weak Aug 30 12:49:34 ip-10-0-0-11 ovpn-netrounds[1848602]: Cannot load certificate file /var/lib/netrounds/openvpn/issued/server.crt Aug 30 12:49:34 ip-10-0-0-11 ovpn-netrounds[1848602]: Exiting due to fatal error Aug 30 12:49:34 ip-10-0-0-11 systemd[1]: openvpn@netrounds.service: Main process exited, code=exited, status=1/FAILURE Aug 30 12:49:34 ip-10-0-0-11 systemd[1]: openvpn@netrounds.service: Failed with result 'exit-code'. Aug 30 12:49:39 ip-10-0-0-11 systemd[1]: openvpn@netrounds.service: Scheduled restart job, restart counter is at 14918. Aug 30 12:49:39 ip-10-0-0-11 systemd[1]: Stopped OpenVPN connection to netrounds.
What you need to do is to modify the /etc/openvpn/netrounds.conf file by
appending the line below at the end of the file:
tls-cipher "DEFAULT:@SECLEVEL=0"
Then restart the openpvn service:
sudo ncc services restart openvpn
Control Center Installation Fails Due to the Conflict with the containers-common Package
Sometimes, during the Ubuntu system upgrade, not all deprecated packages are removed and therefore the Control Center installation fails with the following error message:
... Selecting previously unselected package conmon. Preparing to unpack .../04-conmon_2.0.25+ds1-1.1_amd64.deb ... Unpacking conmon (2.0.25+ds1-1.1) ... Selecting previously unselected package golang-github-containers-image. Preparing to unpack .../05-golang-github-containers-image_5.16.0-3_all.deb ... Unpacking golang-github-containers-image (5.16.0-3) ... dpkg: error processing archive /tmp/apt-dpkg-install-KnNDVu/05-golang-github-containers-image_5.16.0-3_all.deb (--unpack): trying to overwrite '/etc/containers/registries.conf', which is also in package containers-common 100:1-14 Selecting previously unselected package golang-github-containers-common. Preparing to unpack .../06-golang-github-containers-common_0.44.4+ds1-1_all.deb ... Unpacking golang-github-containers-common (0.44.4+ds1-1) ... dpkg: error processing archive /tmp/apt-dpkg-install-KnNDVu/06-golang-github-containers-common_0.44.4+ds1-1_all.deb (--unpack): trying to overwrite '/etc/containers/policy.json', which is also in package containers-common 100:1-14 ... Errors were encountered while processing: /tmp/apt-dpkg-install-KnNDVu/05-golang-github-containers-image_5.16.0-3_all.deb /tmp/apt-dpkg-install-KnNDVu/06-golang-github-containers-common_0.44.4+ds1-1_all.deb needrestart is being skipped since dpkg has failed E: Sub-process /usr/bin/dpkg returned an error code (1)
In this case, run the apt-cache rdepends --installed containers-common
command to check which package holds containers-common package. If this command returns
the following output, then you can delete the containers-common package,
and reinstall Control Center.
apt-cache rdepends —-installed containers-common containers-common Reverse Depends:
To delete the containers-common package, run the following command:
sudo apt-get purge containers-common