How to Backup and Restore Contrail Databases Using the OpenStackClient 16.1 Director Deployment
The document explains how to back up and restore the Contrail databases—Cassandra and Zookeeper—in JSON format. It focuses on scenarios where Contrail Networking operates in OpenStack-orchestrated environments deployed using the Red Hat OpenStack 16.1 director.
If you are deploying Contrail Networking in an OpenStack-orchestrated environment and are using an OpenStack 13-based or Ansible deployer for the deployment, see How to Backup and Restore Contrail Databases in JSON Format in Openstack Environments Using the Openstack 13 or Ansible Deployer.
Contrail Networking initially supports Openstack environments through the Openstack 16.1 director deployment in Contrail Networking Release 2008. See Contrail Networking Supported Platforms for a matrix that outlines the compatibility of Contrail Networking releases with orchestration platforms and deployers.
Before You Begin
You must complete the backup and restore procedure for nodes running the same Contrail Networking release. This procedure backs up only the Contrail Networking databases and excludes instructions for backing up orchestration system databases.
Database backups must be consistent across all systems because the state of the Contrail database is associated with other system databases, such as OpenStack databases. Database changes associated with northbound APIs must be stopped on all the systems before performing any backup operation. For example, you might block the external VIP for northbound APIs at the load balancer level, such as HAproxy.
Administrators must ensure database backups remain consistent across all systems because the state of the Contrail database depends on other system databases, such as OpenStack databases. Before performing any backup operation, administrators must stop database changes associated with northbound APIs on all systems. For example, administrators might block the external VIP for northbound APIs at the load balancer level, such as HAproxy.
Simple Database Backup in JSON Format
This procedure provides a simple database backup in JSON format.
This procedure is performed using the db_json_exim.py script located inside the config-api container in /usr/lib/python2.7/site-packages/cfgm_common on the controller node.
To perform this database backup:
From a controller node, ensure the
db_json_exim.pyscript is available:(overcloud) [user@overcloud-contrailcontroller-0 heat-admin]# podman exec -it contrail_config_api bash (config-api)[user@overcloud-contrailcontroller-0 /]$ ls /usr/lib/python2.7/site-packages/cfgm_common/db_json_exim.py /usr/lib/python2.7/site-packages/cfgm_common/db_json_exim.py
Log into one of the contrail controller nodes. Create the /tmp/db-dump directory on any of the contrail controller node hosts.
mkdir /tmp/db-dump
Note:/tmp/db-dump is a user-defined directory name. You can assign any name to a directory.
On the same contrail controller node, copy the
contrail-api.conffile from the container to the host.podman cp contrail_config_api:/etc/contrail/contrail-api-0.conf /tmp/db-dump/contrail-api.conf
The Cassandra database instance on any contrail controller node includes the complete Cassandra database for all contrail controller nodes in the cluster. Steps 2 and 3, therefore, only need to be performed on one contrail controller node.
On all Contrail controller nodes, stop the following Contrail configuration services:
systemctl stop tripleo_contrail_config_svc_monitor.service systemctl stop tripleo_contrail_config_device_manager.service systemctl stop tripleo_contrail_config_schema.service systemctl stop tripleo_contrail_config_api.service systemctl stop tripleo_contrail_config_nodemgr.service systemctl stop tripleo_contrail_config_database_nodemgr.service
This step must be performed on each individual controller node in the cluster.
On all nodes hosting Contrail Analytics containers, stop the following analytics services:
systemctl stop tripleo_contrail_analytics_kafka.service systemctl stop tripleo_contrail_analytics_snmp_nodemgr.service systemctl stop tripleo_contrail_analytics_alarmgen.service systemctl stop tripleo_contrail_analytics_alarm_nodemgr.service systemctl stop tripleo_contrail_analytics_topology.service systemctl stop tripleo_contrail_analytics_collector.service systemctl stop tripleo_contrail_analytics_nodemgr.service systemctl stop tripleo_contrail_analytics_snmp_collector.service systemctl stop tripleo_contrail_analytics_api.service
This step must be performed on each individual analytics node in the cluster.
Return to the contrail controller node where you performed steps 2 and 3.
Use the podman images command to list the name or ID of the config api image.
podman images | grep config-api
Example:
(overcloud) [user@overcloud-contrailcontroller-0 db-dump]# podman images | grep config-api 192.168.24.1:8787/contrail/contrail-controller-config-api 2011.L1.297 2dcd2feaeed5 2 months ago 876 MB
-
From the same contrail controller node, start the config api container by pointing the
entrypoint.shscript to the /bin/bash directory then mapping /tmp/db-dump directory from the host to the /tmp directory inside the container. You perform this step to ensure that the API services are not started on the contrail controller node.Enter the -v /etc/contrail/ssl:/etc/contrail/ssl:ro command option when
cassandra_use_sslis used as the api-server configuration parameter to ensure TLS certificates are mounted to the Contrail SSL directory. This mounting ensures that the backup procedure succeeds in environments with endpoints that require TLS authentication.podman run --rm -it -v /tmp/db-dump/:/tmp:Z -v /etc/contrail/ssl:/etc/contrail/ssl:ro -v /etc/ipa:/etc/ipa:ro --network host --entrypoint=/bin/bash registry_name/contrail-controller-config-api:container_tag
Example:
podman run --rm -it -v /tmp/db-dump/:/tmp:Z -v /etc/contrail/ssl:/etc/contrail/ssl:ro -v /etc/ipa:/etc/ipa:ro --network host --entrypoint=/bin/bash 192.168.24.1:8787/contrail-nightly/contrail-controller-config-api:21.4.L3.441
Note:The cluster is TLS-enabled if TLS-related files are used during Contrail Networking deployment. The TLS-related files are part of the deployment script. These are custom files and can have any name. For example:
~/tripleo-heat-templates/environments/contrail/contrail-tls.yaml \ ~/tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \ ~/tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \ ~/tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml \
-
From the container created on the contrail controller node, use the
db_json_exim.pyscript to backup data in JSON format. The db dump file will be saved in the /tmp/db-dump/ directory on this contrail controller node. Example:(config-api)[user@overcloud-contrailcontroller-0 /]$ cd /usr/lib/python2.7/site-packages/cfgm_common (config-api)[user@overcloud-contrailcontroller-0 /usr/lib/python2.7/site-packages/cfgm_common]$ python db_json_exim.py --export-to /tmp/db-dump.json --api-conf /tmp/contrail-api.conf 2021-06-30 19:47:27,120 INFO: Cassandra DB dumped 2021-06-30 19:47:28,878 INFO: Zookeeper DB dumped 2021-06-30 19:47:28,895 INFO: DB dump wrote to file /tmp/db-dump.json
The Cassandra database instance on any contrail controller node includes the complete Cassandra database for all contrail controller nodes in the cluster. You, therefore, only need to perform step 4 through 6 from one of the contrail controller nodes.
(Optional. Recommended) From the same contrail controller node, enter the
cat /tmp/db-dump.json | python -m json.tool | lesscommand to view a more readable version of the file transfer.cat /tmp/db-dump.json | python -m json.tool | less
From the same contrail controller node, exit out of the config api container. This will stop the container.
exit
On each contrail controller node, start the following configuration services:
systemctl start tripleo_contrail_config_svc_monitor.service systemctl start tripleo_contrail_config_device_manager.service systemctl start tripleo_contrail_config_schema.service systemctl start tripleo_contrail_config_api.service systemctl start tripleo_contrail_config_nodemgr.service systemctl start tripleo_contrail_config_database_nodemgr.service
This step must be performed on each individual contrail controller node.
On each contrail analytics node, start the following analytics services:
systemctl start tripleo_contrail_analytics_kafka.service systemctl start tripleo_contrail_analytics_snmp_nodemgr.service systemctl start tripleo_contrail_analytics_alarmgen.service systemctl start tripleo_contrail_analytics_alarm_nodemgr.service systemctl start tripleo_contrail_analytics_topology.service systemctl start tripleo_contrail_analytics_collector.service systemctl start tripleo_contrail_analytics_nodemgr.service systemctl start tripleo_contrail_analytics_snmp_collector.service systemctl start tripleo_contrail_analytics_api.service
This step must be performed on each individual contrail analytics node.
On each contrail controller node, enter the
contrail-statuscommand to confirm that services are in the active or running states.Note:Some command output and output fields are removed for readability. Output shown is from a single node hosting configuration and analytics services.
contrail-status Pod Service Original Name State analytics api contrail-analytics-api running analytics collector contrail-analytics-collector running analytics nodemgr contrail-nodemgr running analytics provisioner contrail-provisioner running analytics redis contrail-external-redis running analytics-alarm alarm-gen contrail-analytics-alarm-gen running analytics-alarm kafka contrail-external-kafka running <some output removed for readability> == Contrail control == control: active nodemgr: active named: active dns: active == Contrail analytics-alarm == nodemgr: active kafka: active alarm-gen: active == Contrail database == nodemgr: active query-engine: active cassandra: active == Contrail analytics == nodemgr: active api: active collector: active == Contrail config-database == nodemgr: active zookeeper: active rabbitmq: active cassandra: active == Contrail webui == web: active job: active == Contrail analytics-snmp == snmp-collector: active nodemgr: active topology: active == Contrail config == svc-monitor: active nodemgr: active device-manager: active api: active schema: active
Restore Database from the Backup in JSON Format
This procedure outlines steps to restore Contrail configuration databases— Cassandra and
ZooKeeper databases— on a three-node config-database cluster from a simple database
backup JSON file. For example, a file produced by db_json_exim.py
export.
It covers containerized deployments where Contrail runs as Docker containers managed with Docker or Docker Compose. It also covers TripleO deployments, where OpenStack 16.1 director manages containers with Podman and services with Systemd.
Commands for both deployment types are provided in every step. A DOCKER block is for containerized deployments managed with docker and a TRIPLEO block is for Openstack 16.1 director deployments, which use podman and systemd. Run only the variant that matches your network deployment.
Throughout this procedure, Node1 is the "seed" node where you run the import. Node2 and Node3 are the other two config-database nodes. Each step has a RUN ON badge. ALL THREE NODES means run the code block on node1, node2, and node3. A node1 / node2 / node3 badge means run it only on that specific node.
Prerequisites
Before you begin to restore the databases, you need to ensure that the following prerequisites are fulfilled:
-
All three configuration database nodes are reachable, and you can run Docker commands (or Podman for TripleO) on each one.
-
The backup file, db-dump.json, is present on node1 under /tmp/db-dump/ . Copy the contrail-api.conf file into the same directory.
-
All nodes are running the same Contrail Networking release as the release the backup was taken from.
Steps to Restore Database
To restore a system from a backup JSON file:
Copy the contrail-api.conf file [Run on: Node1]
Export the contrail-api.conf file to the host. Create the dump directory and copy the file out of the running config-api container. Because the Cassandra database on any one node holds the complete data for the cluster. This process is required on a single node, such as Node1, or any controller node in a TripleO deployment.
Docker
mkdir -p /tmp/db-dump docker cp config-api-1:/etc/contrail/contrail-api-0.conf \ /tmp/db-dump/contrail-api.conf
TRIPLEO (podman/systemctl)
mkdir -p /tmp/db-dump podman cp contrail_config_api:/etc/contrail/contrail-api-0.conf \ /tmp/db-dump/contrail-api.conf
Note: Place the backup file,db-dump.json, in the /tmp/db-dump/ directory on Node1 so both files are available to the import container in Step 7.Stop All Contrail Services [Run on: ALL THREE NODES]
Stop all Contrail services before updating the databases. Ensure no data is being written to Cassandra or ZooKeeper during the wipe and restore process. Stop the application services, analytics services, and control services followed by the databases.
Docker
# application, analytics and control services docker stop config-svcmonitor-1 docker stop config-devicemgr-1 docker stop config-schema-1 docker stop config-api-1 docker stop config-nodemgr-1 docker stop config_database-nodemgr-1 docker stop analytics_snmp-snmp-collector-1 docker stop analytics_snmp-topology-1 docker stop analytics_alarm-alarm-gen-1 docker stop analytics-api-1 docker stop analytics-collector-1 docker stop analytics_alarm-kafka-1 docker stop control-control-1 # databases docker stop config_database-cassandra-1 docker stop config_database-zookeeper-1
TRIPLEO (podman/systemctl)
# configuration services (all controller nodes) systemctl stop tripleo_contrail_config_svc_monitor.service systemctl stop tripleo_contrail_config_device_manager.service systemctl stop tripleo_contrail_config_schema.service systemctl stop tripleo_contrail_config_api.service systemctl stop tripleo_contrail_config_nodemgr.service systemctl stop tripleo_contrail_config_database_nodemgr.service # analytics services (all analytics nodes) systemctl stop tripleo_contrail_analytics_kafka.service systemctl stop tripleo_contrail_analytics_snmp_nodemgr.service systemctl stop tripleo_contrail_analytics_alarmgen.service systemctl stop tripleo_contrail_analytics_alarm_nodemgr.service systemctl stop tripleo_contrail_analytics_topology.service systemctl stop tripleo_contrail_analytics_collector.service systemctl stop tripleo_contrail_analytics_nodemgr.service systemctl stop tripleo_contrail_analytics_snmp_collector.service systemctl stop tripleo_contrail_analytics_api.service # databases systemctl stop tripleo_contrail_config_database.service systemctl stop tripleo_contrail_config_zookeeper.service systemctl stop tripleo_contrail_analytics_zookeeper.service
Wipe ZooKeeper Data [Run On: ALL THREE NODES]
Empty ZooKeeper’s data directory on all three nodes. On a three-node ensemble, wiping a single node is not enough because the wiped node re-syncs old data from the leader. This must be done on all three nodes while every ZooKeeper remains stopped.
Docker
cd /var/lib/docker/volumes/config_database_config_zookeeper/ cp -R _data/version-2/ version-2-save rm -rf _data/version-2/*
TRIPLEO (podman/systemctl)
cd /var/lib/contrail/config_zookeeper cp -aR version-2/ zookper-bkp.save rm -rf version-2
Note:The cp line is an optional single-use backup of the current ZooKeeper data. Skip this step if a reliable backup already exists.
Wipe Cassandra Data [Run On: ALL THREE NODES]
A Cassandra node stores its data in two volumes, and both must be cleared:
Named volume: /var/lib/cassandra/data — the SSTables, including system.peers_v2 and system.local (the ring and host_id).
Anonymous volume: /var/lib/cassandra — the commitlog/ , hints/, and saved_caches/directories.
Clearing only the data directory is insufficient. On startup, the commitlog replays and rewrites the old ring, including the host_ids of other nodes, causing the node to boot with the other two marked as DN (down) and “schema unreachable.” You must clear the commitlog, hints, and saved_caches as well.
Stop the Cassandra container, then use a throwaway container with --volumes-from to mount both volumes at their fixed in-container paths. This approach ensures the same command works on every node without requiring host path lookups.
Docker
docker stop -t 30 config_database-cassandra-1 IMG=$(docker inspect config_database-cassandra-1 --format '{{.Config.Image}}') docker run --rm --volumes-from config_database-cassandra-1 \ --entrypoint bash "$IMG" -c 'rm -rf /var/lib/cassandra/data/* \ /var/lib/cassandra/commitlog/* /var/lib/cassandra/hints/* \ /var/lib/cassandra/saved_caches/*'Verify both locations are empty (both must report 0):
docker run --rm --volumes-from config_database-cassandra-1 \ --entrypoint bash "$IMG" \ -c 'echo "data=$(ls -A /var/lib/cassandra/data | wc -l) \ commitlog=$(ls -A /var/lib/cassandra/commitlog | wc -l)"'
Example output:
data=0 commitlog=0
Note:Host-path fallback (Docker). If you prefer to delete from the host instead of using a temporary container, find the host paths with docker inspect and delete within those paths the following code:
DATA=$(docker inspect config_database-cassandra-1 --format \ '{{range .Mounts}}{{if eq .Destination "/var/lib/cassandra/data"}}{{.Source}}{{end}} {{end}}') ROOT=$(docker inspect config_database-cassandra-1 --format \ '{{range .Mounts}}{{if eq .Destination "/var/lib/cassandra"}}{{.Source}}{{end}}{{end }}') rm -rf "$DATA"/* rm -rf "$ROOT"/commitlog/* "$ROOT"/hints/* "$ROOT"/saved_caches/*TripleO
Clear the same set of Cassandra directories on each controller. The host bind mount /var/lib/contrail/config_cassandra maps to /var/lib/cassandra inside the container, so remove data/, commitlog/, hints/, and saved_caches/ beneath it:
cd /var/lib/contrail/config_cassandra cp -aR data/ Cassandra_data-save rm -rf data/* commitlog/* hints/* saved_caches/*
Start Cassandra One Node at a Time
Bring the cluster up in order and verify each node before adding the next. Node1 is the seed. If you start Node2 or Node3 while they still contain outdated data, the seed recalculates the old ring through gossip. Start the seed alone first and confirm it shows only itself—with zero ghost peers—before starting the other nodes.
Start the seed alone [Node1]
On Node1, start Cassandra and wait until the native transport is up:
Docker
docker start config_database-cassandra-1 until docker exec config_database-cassandra-1 \ nodetool -p 7201 info 2>/dev/null \ | grep -q 'Native Transport active: true'; do sleep 6; done
Confirm the seed shows only Node1—a single UN line with zero DN or UJ ghost rows. If any ghost rows appear here, stop and re-wipe that peer (repeat Step 4); the peer was not clean.
docker exec config_database-cassandra-1 \ nodetool -p 7201 status | grep -E '^(UN|DN|UJ|UL)'
Example output:
-- Address Load Tokens Owns Host ID Rack UN 10.0.0.1 1.04 MiB 256 100.0% 9b1f...c7a2 rack1
Start Node2
Start Cassandra on Node2, and then from Node1, confirm Node1 and Node2 are both UN before continuing:
Docker
docker start config_database-cassandra-1 # then, on node1: docker exec config_database-cassandra-1 \ nodetool -p 7201 status | grep -E '^(UN|DN|UJ)'
Start Node3
The process transitions through UJ joining first. Then, from Node1, verify that all three nodes are in the UN state and that the schema is consistent with a single version.
Docker
docker start config_database-cassandra-1 # then, on node1: docker exec config_database-cassandra-1 \ nodetool -p 7201 status | grep -E '^(UN|DN|UJ)' docker exec config_database-cassandra-1 \ nodetool -p 7201 describecluster
Example Output
UN 10.0.0.1 1.04 MiB 256 100.0% 9b1f...c7a2 rack1 UN 10.0.0.2 1.02 MiB 256 100.0% 3d77...01be rack1 UN 10.0.0.3 1.03 MiB 256 100.0% a6c0...5f49 rack1 Schema versions: 2757ebb3-...: [10.0.0.1, 10.0.0.2, 10.0.0.3]
TripleO
The same one-at-a-time rule applies. Start the Cassandra service on Node1 first and wait until the node tool status reports Node1 as UN with no inactive peers. Repeat the process on Node2 and Node3, verifying each node's status before proceeding to the next:
TRIPLEO (podman/systemctl)
# node1 (seed): start, then wait until Native Transport is up systemctl start tripleo_contrail_config_database.service until podman exec contrail_config_database \ nodetool -p 7201 info 2>/dev/null \ | grep -q 'Native Transport active: true'; do sleep 6; done # confirm node1 shows a single UN line (zero DN/UJ ghost rows); # if a ghost peer appears, stop and re-wipe that node (Step 4): podman exec contrail_config_database \ nodetool -p 7201 status | grep -E '^(UN|DN|UJ|UL)' # node2: start, then from node1 confirm node1 + node2 are both UN systemctl start tripleo_contrail_config_database.service podman exec contrail_config_database \ nodetool -p 7201 status | grep -E '^(UN|DN|UJ)' # node3: start, then from node1 confirm all three are UN systemctl start tripleo_contrail_config_database.service podman exec contrail_config_database \ nodetool -p 7201 status | grep -E '^(UN|DN|UJ)' podman exec contrail_config_database \ nodetool -p 7201 describecluster
Start ZooKeeper [Run On: ALL THREE NODES]
Start ZooKeeper on all three nodes after Cassandra confirms all nodes are UN.
Docker
docker start config_database-zookeeper-1
Wait for the ZooKeeper ensemble to elect a leader and sync the followers before importing. The import writes thousands of znodes, so the ensemble must be settled first. A 180 second margin is safe for a three-node ensemble.
sleep 180
TripleO
Start the ZooKeeper services on the configuration and analytics controllers.
TRIPLEO (podman/systemctl)
systemctl start tripleo_contrail_config_zookeeper.service systemctl start tripleo_contrail_analytics_zookeeper.service
Note:Optional. Check— one leader and two follower nodes across the three hosts:
docker exec config_database-zookeeper-1 \ bash -c 'echo srvr | nc $(hostname -i) 2181' | grep -i mode
Import the Database Dump [Run On: Node1]
Run the import on Node1, the seed node where the dump file resides. First, locate the config-api image for this deployment (the tag and registry vary). It is the image used by the running config-api-1 container. Use docker inspect to identify the image, even if the container is currently stopped.
Docker
CONFIG_API_IMG=$(docker inspect config-api-1 --format '{{.Config.Image}}') echo "$CONFIG_API_IMG"Launch that image as a shell, mounting the dump directory and the SSL directory, then run an import inside the container. The -v /etc/contrail/ssl:/etc/contrail/ssl:ro mount supplies the TLS certificates needed when cassandra_use_ssl is enabled:
docker run --rm -it -v /tmp/db-dump/:/tmp/ \ -v /etc/contrail/ssl:/etc/contrail/ssl:ro \ --network host --entrypoint=/bin/bash "$CONFIG_API_IMG" # inside the container: cd /.venv/lib/python3.12/site-packages/cfgm_common/ python3 db_json_exim.py --import-from /tmp/db-dump.json \ --api-conf /tmp/contrail-api.conf exit
Example Output
2026-06-23 17:22:17,157 INFO: DB dump file loaded 2026-06-23 17:23:12,227 INFO: Cassandra DB restored 2026-06-23 17:23:14,236 INFO: Zookeeper DB restored
TripleO
The equivalent uses podman. The import prints the same INFO lines shown above:
TRIPLEO (podman / systemctl)
podman image ls | grep config-api # use the registry_name and container_tag from the image list: podman run --rm -it -v /tmp/db-dump/:/tmp:Z \ -v /etc/contrail/ssl:/etc/contrail/ssl:ro \ --network host --entrypoint=/bin/bash \ <registry_name>/contrail-controller-config-api:<container_tag> # inside the container: cd /.venv/lib/python3.12/site-packages/cfgm_common/ python3 db_json_exim.py --import-from /tmp/db-dump.json \ --api-conf /tmp/contrail-api.conf exit
Start All Contrail Services [Run On: ALL THREE NODES]
After the import completes, restart all services on each of the three nodes:
Docker
docker start config-svcmonitor-1 docker start config-devicemgr-1 docker start config-schema-1 docker start config-api-1 docker start config-nodemgr-1 docker start config_database-nodemgr-1 docker start analytics_snmp-snmp-collector-1 docker start analytics_snmp-topology-1 docker start analytics_alarm-alarm-gen-1 docker start analytics-api-1 docker start analytics-collector-1 docker start analytics_alarm-kafka-1 docker start control-control-1
TRIPLEO (podman/systemctl)
# configuration services (all controller nodes) systemctl start tripleo_contrail_config_svc_monitor.service systemctl start tripleo_contrail_config_device_manager.service systemctl start tripleo_contrail_config_schema.service systemctl start tripleo_contrail_config_api.service systemctl start tripleo_contrail_config_nodemgr.service systemctl start tripleo_contrail_config_database_nodemgr.service # analytics services (all analytics nodes) systemctl start tripleo_contrail_analytics_kafka.service systemctl start tripleo_contrail_analytics_snmp_nodemgr.service systemctl start tripleo_contrail_analytics_alarmgen.service systemctl start tripleo_contrail_analytics_alarm_nodemgr.service systemctl start tripleo_contrail_analytics_topology.service systemctl start tripleo_contrail_analytics_collector.service systemctl start tripleo_contrail_analytics_nodemgr.service systemctl start tripleo_contrail_analytics_snmp_collector.service systemctl start tripleo_contrail_analytics_api.service
Post-Restore Kafka Fix [Run On: ALL THREE NODES]
After a restore, the collector commonly reports a Kafka connection error such as:
Symptom
collector: initializing (KafkaPub:<host>:9092 not connected)
Because ZooKeeper was wiped and restored, the ZooKeeper topic IDs no longer match the
partition.metadatastamps in Kafka’s on-disk log directory (/tmp/kafka-logs, the container’s temporary overlay file system). A plain Docker restart does not clear that directory, which is why the problem persists. Wiping it and restarting is equivalent to up --force-recreate but keeps the container. This step is harmless to run regardless, so always run it:Docker
docker exec analytics_alarm-kafka-1 rm -rf /tmp/kafka-logs \ && docker restart -t 0 analytics_alarm-kafka-1 docker restart analytics-collector-1
Note:More surgical variant (Docker). To keep existing UVE data and drop only the topic-id stamp, delete just the
partition.metadatafiles instead:docker exec analytics_alarm-kafka-1 \ bash -c 'find /tmp/kafka-logs -name partition.metadata -delete' \ && docker restart -t 0 analytics_alarm-kafka-1
TRIPLEO (podman/systemctl)
podman exec contrail_analytics_kafka rm -rf /tmp/kafka-logs systemctl restart tripleo_contrail_analytics_kafka.service systemctl restart tripleo_contrail_analytics_collector.service
Verify the Restore
Confirm the cluster is healthy. On each node, verify that the configuration, control, analytics, and configuration database containers are running:
Docker
docker ps --format 'table {{.Names}}\t{{.Status}}'From Node1, verify Cassandra displays all three nodes UN and a single schema version:
docker exec config_database-cassandra-1 \ nodetool -p 7201 status | grep -E '^(UN|DN|UJ)' docker exec config_database-cassandra-1 \ nodetool -p 7201 describecluster
TripleO confirm services are in the active states or running states with contrail-status on each controller and analytics node.
TRIPLEO (podman/systemctl)
contrail-status
Finally, confirm the restored objects are visible in the Contrail Networking web UI or through the API, and that the collector does not report the Kafka connection error in Step 9.