Migrate Data from NorthStar to Paragon Automation

Copy the files locally to the current node:

Copy the files to the target NorthStar installation.

Source the NorthStar environment file.

Run the export script.

Table 1: Historical Event Data Tables

keyspace	table
taskscheduler	taskstatus
pcs	topology, lsp_topo, lsp_link, ntad, messages, pcs_lsp_event, link_event, node_event
pcs_provision	provision

Running the script exports the contents of the Cassandra database (according to db_schema.json) to the export_csv folder in the current working directory. The script pipes the progress output from the dsbulk invocations to stdout. Each table has its own sub-directory with one or more CSV files. The procedure may take a long time for larger databases.

Note:

The exported CSV files also serve as a backup of the Cassandra DB data. We recommend archiving the files in case data needs to be restored in the future.

Copy the exported files to the correct directory on worker3 node.

For example:

You must specify the following parameters while running the ns_data_migration.py script.

• -a APP, --app APP—IP address or hostname of the application server
• -su SSHUSER, --sshuser SSHUSER—SSH username (default is root)
• -sp SSHPASS, --sshpass SSHPASS—SSH password
• -so SSHPORT, --sshport SSHPORT—SSH port (default is 22)
• -du DBUSER, --dbuser DBUSER—Cassandra DB username (default is cassandra)
• -dp DBPASS, --dbpass DBPASS—Cassandra DB password
• -do DBPORT, --dbport DBPORT—Cassandra DB port (default is 9042)
• -dh DBHOST, --dbhost DBHOST—Comma-separated host IP addresses of Cassandra DB
• -pu PGUSER, --pguser PGUSER—Postgres DB username (default is northstar)
• -dcsv DBCSV, --dbCsvPath DBCSV—The path with CSV data exported from Cassandra
• -pp PGPASS, --pgpass PGPASS—Postgres DB password
• -ph PGHOST, --pghost PGHOST—Postgres DB host (default is atom-db.common)
• -po PGPORT, --pgport PGPORT—Postgres DB port (default is 5432)
• -pah PARAGONHOST, --paragonHost PARAGONHOST—Virtual IP (VIP) address of Paragon Automation Web UI
• -pau PARAGONUSER, --paragonUser PARAGONUSER—Paragon Automation Web UI username
• -pap PARAGONPASSWORD, --paragonPassword PARAGONPASSWORD—Paragon Automation Web UI user password
• -dr DISCOVERYRETRIES, --discoveryRetries DISCOVERYRETRIES—Device discovery retries (default is 2).

  You use the dr DISCOVERYRETRIES option for DeviceProfile migration when Paragon Automation fails to discover devices at the first attempt. There are multiple reasons for discovery failure, such as devices not being reachable or device credentials being incorrect. Despite discovery failure for devices with incorrect information, Paragon Automation discovers devices with correct information. Partial failure for a subset of devices while discovering multiple devices at a time is possible. To determine the exact reason of failure, see the Monitoring > Jobs page in the Paragon Automation Web UI.

  If the dr option is set to more than 1, on getting a discovery failure, the ns_data_migration.py script retries the discovery for all the devices. This attempt does not impact the devices that are already discovered. However, the chances of successfully discovering devices in subsequent attempts for any failed device discovery is minimal. We recommend that you set the maximum value for the dr option to 2, which is the default value. If there are too many devices in the network, then use a value of 1 to avoid unnecessary retries.

Note:

When migrating Cassandra DB data from NorthStar to Paragon Automation, large tables with millions of rows might cause the migration to proceed very slowly and take a long time. Often these large tables contain historical event data that you can discard during migration. To skip migrating this data, you can manually set the '--dbSkipHistoricalData' flag while calling the 'ns_data_migration.py' script. This means that the data in the historical event tables listed in Table 1 is not available in Paragon Automation. This data is permanently lost if not backed up once the NorthStar instance is removed.

Log in to Paragon Automation Web UI and navigate to Configuration > Device. Verify that all the devices are discovered and present. Also, verify that the configuration information is the same as that in the NorthStar device profile.

To view the device discovery result, go to the Monitoring > Jobs page in the Paragon Automation Web UI.

The log output of the ns_data_migration.py script indicates whether there were any problems migrating data from Cassandra. You can also run a script to verify the data in Paragon Automation against the exported CSV files. Note, this may take a long time for larger databases. From the nsmigration container, run:

The script outputs (rows verified)/(rows checked) in each table (see lines beginning with "Verify") to stdout and debug_migration.log. Note that some rows may have been updated after the data was imported but before it was verified, so 'rows verified' may not always equal 'rows checked'. The exported CSV files can be removed once the migration is complete by simply removing the /var/local/ns_db_migration/export_csv directory on the relevant node.

You must specify the following import_es_data.py script options:.

• Statistics type—By default, supports interface, label-switched path (LSP), and link-latency statistics data. You can select a specific type by using the --type option.

• Rollups type—By default, supports daily and hourly time periods. You can select a specific type by using the --rollup_type option.

• Migration schema—The mapping of ES DB to Influx DB schema is defined in the /opt/northstar/util/db/nsmigration/es_influx_mapping.json file.

• Rollup ages—By default, fetches hourly and daily data for the last 180 days and 1000 days, respectively. You can change the ages by using the --hourly_age and --daily_age options.

• ETL parallel worker process—By default, uses four ETL parallel worker processes. You can change the worker process by using the --wp option.

• Execution time—The script execution time varies based on data volume, the number of days, and the number of ETL parallel worker processes. For example, if four ETL workers use a migration rate of 1500, then:

  • 25,000 LSP statistics of 180 days hourly can take 5 hours

  • 50,000 LSP statistics of 180 days hourly can take 10 hours

For more information about script arguments, see help 'import_es_data.py -h'.

• To query all tunnel traffic data for the last 30 days in Influx DB, run the /opt/pcs/bin/getTrafficFiles.py script inside the dcscheduler pod:

• To query all egress interface traffic data for the last 30 days in Influx DB, run the /opt/pcs/bin/getTrafficFiles.py script inside the dcscheduler pod: