TechLibrary

Table of Contents

Guide That Contains This Content

[+] Expand All

[-] Collapse All

Installing and Administering IP/MPLSView Replication and Resync in a Two-Server Environment

This section describes how to set up, install, start, and administer IP/MPLSView in a two-server distributed environment. It includes step-by-step procedures for failing over to the backup server when the primary server goes down, synchronizing the data, and switching back to the primary server when it resumes operation.

Figure 1 shows a typical setup for using IP/MPLSView in a distributed environment that consists of two servers—a primary application server and a backup application server—each of which runs both the IP/MPLSView application and the IP/MPLSView database.

Figure 1: Example IP/MPLSView Setup for a Two-Server Distributed Environment

You can install the replication and rsync package as part of the IP/MPLSView installation script. If you choose not to install the replication and rsync package during the standard installation procedure, you can install it later by running the replication/instrepl.sh script as the wandl user.

The sample setup consists of the following hardware and software components:

One primary application server (labeled Primary AP)
One backup application server (labeled Backup AP)
Two Traffic Data Collectors (each labeled DC)
Replication and rsync software running between the primary and backup application servers
MongoDB fault management database
MariaDB performance management database on the primary and backup application servers
Java Message Service (JMS), which sends messages between the application servers and the Traffic Data Collectors

Installing IP/MPLSView with Replication and Resync in a Two-Server Environment

To install IP/MPLSView with replication and rsync software in a two-server distributed environment:

Use the server/install.sh script in the installation directory and follow the standard installation procedure.
%install_directory% /server/install.sh

Install the rsync and replication package when prompted.

Install Rsync & Database Replication Package (default=no)? [y/n] y

Configure the primary application server with the appropriate replication and rsync settings.

Rsync & Replication Settings
1.)Setup Rsync for Application Server.....YES
2.)Install Database Replication Package...YES
3.)Setup as Primary or Backup Server......PRIMARY
4.)Preserve files on target server........YES (Recommended)

Configure the backup application server with the appropriate replication and rsync settings.

Rsync & Replication Settings
1.)Setup Rsync for Application Server.....YES
2.)Install Database Replication Package...YES
3.)Setup as Primary or Backup Server......BACKUP
4.)Preserve files on target server........YES (Recommended)

Enter the IP address of the backup server on the primary servers and, conversely, enter the IP address of the primary server on the backup server.
```
Please enter the IP address of the alternate MPLSView server: ip-address
```
Enter the ssh directory of the alternate IP/MPLSView server if the actual paths do not conform to the default paths.
Enter the SSH host key of the wandl user, for example, /home/wandl/.ssh/id_rsa.
Check the size of the IP/MPLSView home (/home/wandl) directory to make sure the backup server has sufficient disk space in the home directory.
du -ks /home/wandl
By default, the administrative home directory is not synchronized. To synchronize this directory, check the size of the home directory to make sure it is not too large to copy its entire contents to the backup server.
After the installation, you can modify which directories and files are synchronized by editing the /u/wandl/bin/rsync.sh script. You must exclude the database files from the rsync.

Synchronize directories and files when prompted.

Synchronize the following files/directories?:
You can later manually modify the entries in
/u/wandl/bin/rsync.sh [YES]y
1.) /u/wandl/data/
2.) [YES] /u/wandl/db/config/snmptrap.store
3.) [YES] /u/wandl/db/config/subscriptions.store
4.) [YES] /u/wandl/db/config/eventtypes.store
5.) [YES] /u/wandl/db/config/productionscopes.store
6.) [YES] /u/wandl/db/config/collectioncmds.xml
7.) [YES] /u/wandl/db/config/diagnosticcmds
8.) [YES] /u/wandl/db/config/shownodecmds
9.) [YES] /u/wandl/db/config/showvpncmds
10.) [YES] /u/wandl/db/command/
11.) [YES] /u/wandl/data/.network/
12.) [YES] /u/wandl/data/.TaskManager/profile/
13.) [YES] /u/wandl/data/.TaskManager/tmp/.diag
14.) [YES] /u/wandl/data/device/
15.) [YES] /u/wandl/data/ping/
16.) [YES] /u/wandl/data/sla/
17.) [YES] /u/wandl/data/summary/
18.) [YES] /u/wandl/data/latency/
19.) [YES] /u/wandl/data/event/
20.) [YES] - /u/wandl/data/LDPTraffic/
Please select a number to modify. [<CR>=accept]:
Accept these values (default=no)? [y/n] y

Schedule the time interval to use for synchronizing the backup application server with the primary application server.
```
Please select the crontab interval in minutes (60): [0-60] <#>
```
(Optional) To modify the time interval after the installation, run the following script:
/install_dir/replication/instrepl.sh
Alternatively, you can directly modify the crontab settings as the wandl user and export the EDITOR variable by using the export EDITOR command. For example, you can enter EDITOR=vi to set the editor to vi, and then run the crontab -e command.

Starting IP/MPLSView in a Two-Server Environment

To start IP/MPLSView on the primary and backup application servers:

Start IP/MPLSView on the primary application server (master).
If it does not start during installation, use the following command:
/u/wandl/bin/startup_mplsview
Make sure the following entry is in the /u/wandl/db/config/mongodb.conf file:
replSet = rs0
Start MongoDB on the backup application server.
/u/wandl/bin/mongodb.sh start
Start MongoDB in slave mode on the primary application server.
/u/wandl/bin/mongodb_repl.sh addslave

Check the status of MongoDB on both the primary and the backup application servers.

/u/wandl/bin/mongodb_repl.sh status

Primary MongoDB is running
Primary replication position: 1
Secondary: <ip-address>:27017 is up and running
Secondary: <ip-address>:27017 replication position 1

(Optional) To start IP/MPLSView if the addslave command does not cause the MongoDB to run in slave mode:
1. Configure the primary and backup application servers with the replication settings in the /u/wandl/db/config/mongodb.conf file.
2. Make sure the following entry appears in the /u/wandl/db/config/mongodb.conf file:
  replSet = rs0
3. Access the MongoDB CLI (/u/wandl/thirdparty/mongodb/bin/mongo) on the primary application server, and issue the following commands to initiate MongoDB and run it in slave mode:
  run rs.initiate();
  rs.add({_id: 1, host: “<slave-ip>:27017”, priority: 0.5});
Start the backup MariaDB server in slave mode.
/u/wandl/bin/.mysql start slave
On the backup application server, register the MariaDB server with the master MariaDB server.
/u/wandl/bin/mysql_repl.sh registermaster
Check the status of the MariaDB slave server.
/u/wandl/bin/mysql_repl.sh status
```
Slave Mysql server connected to Master: primary-server ip-address/hostname
```

Check the status of the MariaDB master server.

/u/wandl/bin/mysql_repl.sh status

Master Mysql server is running
Slave server: backup-server ip-address/hostname

Verify that rsync is enabled.
crontab -l
On the primary application server, the /u/wandl/bin/rsync.sh script runs according to the interval you specify during the installation. Depending on your synchronization requirements, specifying an interval of 30 minutes or 60 minutes should be sufficiently frequent.
Each time the rsync.sh script runs, it synchronizes the directories in the /u/wandl/data directory, excluding the MariaDB traffic database and MongoDB directories, between the primary application server and the backup application server. This is referred to as push synchronization. This process also replicates the MariaDB and MongoDB databases.
After the interval you specified for rsync, check the /u/wandl/data directory on the backup application server to make sure the file system data has been copied.

Failing Over to the Backup Server

In the event that the IP/MPLSView primary application server fails, you can fail over to the backup application server and initiate operation on the backup server until the primary server recovers.

To fail over to the backup application server if the primary application server fails:

Stop the remaining services on the primary application server.
/u/wandl/bin/stop_mplsview
On the primary application server, stop the cronjob for the /u/wandl/bin/rsync.sh script as both the root user and the wandl user.
crontab -e
The cronjob is owned by the administrative user ID who installed IP/MPLSView. By default, this is the wandl user.
Comment out (disable) the following line in crontab:
#0,30 * * * * /u/wandl/bin/rsync.sh exec > /u/wandl/log/rsync.log
Stop the backup application server.
/u/wandl/bin/stop_mplsview
Make sure the cronjob is disabled (commented out).
crontab -e
Start IP/MPLSView on the backup application server.
This action causes the backup application server to become the primary (master) application server.
/u/wandl/bin/startup_mplsview
Comment out the following entry in the /u/wandl/db/config/mongodb.conf file:
replSet = rs0
Restart the MongoDB process.
/u/wandl/bin/mongodb.sh stop
/u/wandl/bin/mongodb.sh start
Access the MongoDB CLI and issue the following commands:
/u/wandl/thirdparty/mongodb/bin/mongo
use local;
db.dropDatabase();

Synchronizing the Data from the Backup Server to the Original Primary Server

When the original primary application server resumes operation, you must synchronize the data from the backup application server to the primary application server.

To synchronize data from the backup server to the original primary server:

Start the original primary MariaDB server in slave mode.
/u/wandl/bin/.mysql start slave
```
Slave Mysql server connected to master: backup-server ip-address
```
Register the master MariaDB server on the backup application server.
/u/wandl/bin/.mysql registermaster
Check the last data received on both the primary application server and the backup application server to make sure the data is synchronized.
/u/wandl/bin/mysql_repl.sh lastdata
When you issue this command, the software displays an 11-digit timestamp that corresponds to the time of the last data collection. When both the primary MariaDB server and the master MariaDB server display the same 11-digit timestamp, the data is synchronized.
For more information about the format of the timestamp, see Troubleshooting Database Synchronization in a Distributed Environment.
Start the MongoDB.
/u/wandl/bin/mongodb.sh start
Leave (stop) the MongoDB cluster on the original primary application server.
/u/wandl/bin/mongodb_repl.sh leavecluster
On the backup application server, restart the MongoDB in cluster mode, and uncomment the following entry in the /u/wandl/db/config/mongodb.conf file:
replSet = rs0
Stop and restart the MongoDB process.
/u/wandl/bin/mongodb.sh stop
/u/wandl/bin/mongodb.sh start
Start the MongoDB in slave mode on the backup application server.
/u/wandl/bin/mongodb_repl.sh addslave
Stop IP/MPLSView on the backup application server.
/u/wandl/bin/stop_mplsview
If changes occurred in the network while the primary application server was out of service, issue the following command only once on the backup application server to synchronize the original primary application server with the backup application server:
/u/wandl/bin/rsync.sh exec
Verify that the data has been synchronized.
/u/wandl/bin/mongodb_repl.sh status
/u/wandl/bin/mongodb_repl.sh lastdata

Switching Back to the Original Primary Server

To switch back to the original primary application server after it resumes operation:

Stop IP/MPLSView on the backup application server and then on the primary application server, in that order.
Check the status of IP/MPLSView on both application servers to confirm that it has been stopped.
/u/wandl/bin/status_mplsview
On the backup application server, start MongoDB and leave the cluster.
/u/wandl/bin/mongodb.sh start
/u/wandl/bin/mongodb_repl.sh leavecluster
On the primary application server, start MongoDB and leave the cluster.
/u/wandl/bin/mongodb.sh start
/u/wandl/bin/mongodb_repl.sh leavecluster
/u/wandl/bin/mongodb.sh stop
Start IP/MPLSView on the primary application server.
/u/wandl/bin/startup_mplsview
Start MongoDB in slave mode on the primary application server.
/u/wandl/bin/mongodb_repl.sh addslave
Check the status of MongoDB on the primary application server and then on the backup application server, in that order.
/u/wandl/bin/mongodb_repl.sh status
Start the MariaDB server in slave mode on the backup application server.
/u/wandl/bin/.mysql start slave
On the backup application server, register the MariaDB server with the master MariaDB server.
/u/wandl/bin/mysql_repl.sh registermaster
Check the status on the primary and backup MariaDB servers.
/u/wandl/bin/mysql_repl.sh status
Check the data consistency on the primary MariaDB server.
/u/wandl/bin/mysql_repl.sh lastdata
On the primary application server, check crontab for the root and wandl user accounts to verify if rsync.sh is commented out.
crontab -l
If cronjob is commented out in the /u/wandl/bin/rsync.sh script on the primary application server, reenable it.
```
crontab -e
```
The cronjob is owned by the administrative user ID who installed IP/MPLSView. By default, this is the wandl user.

Uncomment (enable) the following line in crontab:

#0,30 * * * * /u/wandl/bin/rsync.sh exec > /u/wandl/log/rsync.log

Stop IP/MPLSView on the backup application server.
```
/u/wandl/bin/stop_mplsview
```
Make sure the cronjob is commented out (disabled).
```
crontab -e
```
If the cronjob is not disabled on the backup application server, comment out (disable) the following line in crontab:
#0,30 * * * * /u/wandl/bin/rsync.sh exec > /u/wandl/log/rsync.log