Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Navigation
Guide That Contains This Content
[+] Expand All
[-] Collapse All

    Installing and Administering IP/MPLSView Replication and Resync in a Two-Server Environment

    This section describes how to set up, install, start, and administer IP/MPLSView in a two-server distributed environment. It includes step-by-step procedures for failing over to the backup server when the primary server goes down, synchronizing the data, and switching back to the primary server when it resumes operation.

    Figure 1 shows a typical setup for using IP/MPLSView in a distributed environment that consists of two servers—a primary application server and a backup application server—each of which runs both the IP/MPLSView application and the IP/MPLSView database.

    Figure 1: Example IP/MPLSView Setup for a Two-Server Distributed Environment

    Example
IP/MPLSView Setup for a Two-Server Distributed Environment

    You can install the replication and rsync package as part of the IP/MPLSView installation script. If you choose not to install the replication and rsync package during the standard installation procedure, you can install it later by running the replication/instrepl.sh script as the wandl user.

    The sample setup consists of the following hardware and software components:

    • One primary application server (labeled Primary AP)
    • One backup application server (labeled Backup AP)
    • Two Traffic Data Collectors (each labeled DC)
    • Replication and rsync software running between the primary and backup application servers
    • MongoDB fault management database
    • MariaDB performance management database on the primary and backup application servers
    • Java Message Service (JMS), which sends messages between the application servers and the Traffic Data Collectors

    Installing IP/MPLSView with Replication and Resync in a Two-Server Environment

    To install IP/MPLSView with replication and rsync software in a two-server distributed environment:

    1. Use the server/install.sh script in the installation directory and follow the standard installation procedure.
      %install_directory% /server/install.sh
    2. Install the rsync and replication package when prompted.
      Install Rsync & Database Replication Package (default=no)? [y/n] y
    3. Configure the primary application server with the appropriate replication and rsync settings.
      Rsync & Replication Settings
      1.)Setup Rsync for Application Server.....YES
      2.)Install Database Replication Package...YES
      3.)Setup as Primary or Backup Server......PRIMARY
      4.)Preserve files on target server........YES (Recommended)
    4. Configure the backup application server with the appropriate replication and rsync settings.
      Rsync & Replication Settings
      1.)Setup Rsync for Application Server.....YES
      2.)Install Database Replication Package...YES
      3.)Setup as Primary or Backup Server......BACKUP
      4.)Preserve files on target server........YES (Recommended)
    5. Enter the IP address of the backup server on the primary servers and, conversely, enter the IP address of the primary server on the backup server.
      Please enter the IP address of the alternate MPLSView server: ip-address
    6. Enter the ssh directory of the alternate IP/MPLSView server if the actual paths do not conform to the default paths.
    7. Enter the SSH host key of the wandl user, for example, /home/wandl/.ssh/id_rsa.
    8. Check the size of the IP/MPLSView home (/home/wandl) directory to make sure the backup server has sufficient disk space in the home directory.
      du -ks /home/wandl

      By default, the administrative home directory is not synchronized. To synchronize this directory, check the size of the home directory to make sure it is not too large to copy its entire contents to the backup server.

      After the installation, you can modify which directories and files are synchronized by editing the /u/wandl/bin/rsync.sh script. You must exclude the database files from the rsync.

    9. Synchronize directories and files when prompted.
      Synchronize the following files/directories?:
      You can later manually modify the entries in
      /u/wandl/bin/rsync.sh [YES]y
      1.) /u/wandl/data/
      2.) [YES] /u/wandl/db/config/snmptrap.store
      3.) [YES] /u/wandl/db/config/subscriptions.store
      4.) [YES] /u/wandl/db/config/eventtypes.store
      5.) [YES] /u/wandl/db/config/productionscopes.store
      6.) [YES] /u/wandl/db/config/collectioncmds.xml
      7.) [YES] /u/wandl/db/config/diagnosticcmds
      8.) [YES] /u/wandl/db/config/shownodecmds
      9.) [YES] /u/wandl/db/config/showvpncmds
      10.) [YES] /u/wandl/db/command/
      11.) [YES] /u/wandl/data/.network/
      12.) [YES] /u/wandl/data/.TaskManager/profile/
      13.) [YES] /u/wandl/data/.TaskManager/tmp/.diag
      14.) [YES] /u/wandl/data/device/
      15.) [YES] /u/wandl/data/ping/
      16.) [YES] /u/wandl/data/sla/
      17.) [YES] /u/wandl/data/summary/
      18.) [YES] /u/wandl/data/latency/
      19.) [YES] /u/wandl/data/event/
      20.) [YES] - /u/wandl/data/LDPTraffic/
      Please select a number to modify. [<CR>=accept]:
      Accept these values (default=no)? [y/n] y
    10. Schedule the time interval to use for synchronizing the backup application server with the primary application server.
      Please select the crontab interval in minutes (60): [0-60] <#>
    11. (Optional) To modify the time interval after the installation, run the following script:
      /install_dir/replication/instrepl.sh

      Alternatively, you can directly modify the crontab settings as the wandl user and export the EDITOR variable by using the export EDITOR command. For example, you can enter EDITOR=vi to set the editor to vi, and then run the crontab -e command.

    Starting IP/MPLSView in a Two-Server Environment

    To start IP/MPLSView on the primary and backup application servers:

    1. Start IP/MPLSView on the primary application server (master).

      If it does not start during installation, use the following command:

      /u/wandl/bin/startup_mplsview
    2. Make sure the following entry is in the /u/wandl/db/config/mongodb.conf file:
      replSet = rs0
    3. Start MongoDB on the backup application server.
      /u/wandl/bin/mongodb.sh start
    4. Start MongoDB in slave mode on the primary application server.
      /u/wandl/bin/mongodb_repl.sh addslave
    5. Check the status of MongoDB on both the primary and the backup application servers.
      /u/wandl/bin/mongodb_repl.sh status
      Primary MongoDB is running
      Primary replication position: 1
      Secondary: <ip-address>:27017 is up and running
      Secondary: <ip-address>:27017 replication position 1
    6. (Optional) To start IP/MPLSView if the addslave command does not cause the MongoDB to run in slave mode:
      1. Configure the primary and backup application servers with the replication settings in the /u/wandl/db/config/mongodb.conf file.
      2. Make sure the following entry appears in the /u/wandl/db/config/mongodb.conf file:
        replSet = rs0
      3. Access the MongoDB CLI (/u/wandl/thirdparty/mongodb/bin/mongo) on the primary application server, and issue the following commands to initiate MongoDB and run it in slave mode:
        run rs.initiate();
        rs.add({_id: 1, host: “<slave-ip>:27017”, priority: 0.5});
    7. Start the backup MariaDB server in slave mode.
      /u/wandl/bin/.mysql start slave
    8. On the backup application server, register the MariaDB server with the master MariaDB server.
      /u/wandl/bin/mysql_repl.sh registermaster
    9. Check the status of the MariaDB slave server.
      /u/wandl/bin/mysql_repl.sh status
      Slave Mysql server connected to Master: primary-server ip-address/hostname
    10. Check the status of the MariaDB master server.
      /u/wandl/bin/mysql_repl.sh status
      Master Mysql server is running
      Slave server: backup-server ip-address/hostname
    11. Verify that rsync is enabled.
      crontab -l

      On the primary application server, the /u/wandl/bin/rsync.sh script runs according to the interval you specify during the installation. Depending on your synchronization requirements, specifying an interval of 30 minutes or 60 minutes should be sufficiently frequent.

      Each time the rsync.sh script runs, it synchronizes the directories in the /u/wandl/data directory, excluding the MariaDB traffic database and MongoDB directories, between the primary application server and the backup application server. This is referred to as push synchronization. This process also replicates the MariaDB and MongoDB databases.

    12. After the interval you specified for rsync, check the /u/wandl/data directory on the backup application server to make sure the file system data has been copied.

    Failing Over to the Backup Server

    In the event that the IP/MPLSView primary application server fails, you can fail over to the backup application server and initiate operation on the backup server until the primary server recovers.

    To fail over to the backup application server if the primary application server fails:

    1. Stop the remaining services on the primary application server.
      /u/wandl/bin/stop_mplsview
    2. On the primary application server, stop the cronjob for the /u/wandl/bin/rsync.sh script as both the root user and the wandl user.
      crontab -e

      The cronjob is owned by the administrative user ID who installed IP/MPLSView. By default, this is the wandl user.

    3. Comment out (disable) the following line in crontab:
      #0,30 * * * * /u/wandl/bin/rsync.sh exec > /u/wandl/log/rsync.log
    4. Stop the backup application server.
      /u/wandl/bin/stop_mplsview
    5. Make sure the cronjob is disabled (commented out).
      crontab -e
    6. Start IP/MPLSView on the backup application server.

      This action causes the backup application server to become the primary (master) application server.

      /u/wandl/bin/startup_mplsview
    7. Comment out the following entry in the /u/wandl/db/config/mongodb.conf file:
      replSet = rs0
    8. Restart the MongoDB process.
      /u/wandl/bin/mongodb.sh stop
      /u/wandl/bin/mongodb.sh start
    9. Access the MongoDB CLI and issue the following commands:
      /u/wandl/thirdparty/mongodb/bin/mongo
      use local;
      db.dropDatabase();

    Synchronizing the Data from the Backup Server to the Original Primary Server

    When the original primary application server resumes operation, you must synchronize the data from the backup application server to the primary application server.

    To synchronize data from the backup server to the original primary server:

    1. Start the original primary MariaDB server in slave mode.
      /u/wandl/bin/.mysql start slave
      Slave Mysql server connected to master: backup-server ip-address
    2. Register the master MariaDB server on the backup application server.
      /u/wandl/bin/.mysql registermaster
    3. Check the last data received on both the primary application server and the backup application server to make sure the data is synchronized.
      /u/wandl/bin/mysql_repl.sh lastdata

      When you issue this command, the software displays an 11-digit timestamp that corresponds to the time of the last data collection. When both the primary MariaDB server and the master MariaDB server display the same 11-digit timestamp, the data is synchronized.

      For more information about the format of the timestamp, see Troubleshooting Database Synchronization in a Distributed Environment.

    4. Start the MongoDB.
      /u/wandl/bin/mongodb.sh start
    5. Leave (stop) the MongoDB cluster on the original primary application server.
      /u/wandl/bin/mongodb_repl.sh leavecluster
    6. On the backup application server, restart the MongoDB in cluster mode, and uncomment the following entry in the /u/wandl/db/config/mongodb.conf file:
      replSet = rs0
    7. Stop and restart the MongoDB process.
      /u/wandl/bin/mongodb.sh stop
      /u/wandl/bin/mongodb.sh start
    8. Start the MongoDB in slave mode on the backup application server.
      /u/wandl/bin/mongodb_repl.sh addslave
    9. Stop IP/MPLSView on the backup application server.
      /u/wandl/bin/stop_mplsview
    10. If changes occurred in the network while the primary application server was out of service, issue the following command only once on the backup application server to synchronize the original primary application server with the backup application server:
      /u/wandl/bin/rsync.sh exec
    11. Verify that the data has been synchronized.
      /u/wandl/bin/mongodb_repl.sh status
      /u/wandl/bin/mongodb_repl.sh lastdata

    Switching Back to the Original Primary Server

    To switch back to the original primary application server after it resumes operation:

    1. Stop IP/MPLSView on the backup application server and then on the primary application server, in that order.
    2. Check the status of IP/MPLSView on both application servers to confirm that it has been stopped.
      /u/wandl/bin/status_mplsview
    3. On the backup application server, start MongoDB and leave the cluster.
      /u/wandl/bin/mongodb.sh start
      /u/wandl/bin/mongodb_repl.sh leavecluster
    4. On the primary application server, start MongoDB and leave the cluster.
      /u/wandl/bin/mongodb.sh start
      /u/wandl/bin/mongodb_repl.sh leavecluster
      /u/wandl/bin/mongodb.sh stop
    5. Start IP/MPLSView on the primary application server.
      /u/wandl/bin/startup_mplsview
    6. Start MongoDB in slave mode on the primary application server.
      /u/wandl/bin/mongodb_repl.sh addslave
    7. Check the status of MongoDB on the primary application server and then on the backup application server, in that order.
      /u/wandl/bin/mongodb_repl.sh status
    8. Start the MariaDB server in slave mode on the backup application server.
      /u/wandl/bin/.mysql start slave
    9. On the backup application server, register the MariaDB server with the master MariaDB server.
      /u/wandl/bin/mysql_repl.sh registermaster
    10. Check the status on the primary and backup MariaDB servers.
      /u/wandl/bin/mysql_repl.sh status
    11. Check the data consistency on the primary MariaDB server.
      /u/wandl/bin/mysql_repl.sh lastdata
    12. On the primary application server, check crontab for the root and wandl user accounts to verify if rsync.sh is commented out.
      crontab -l
    13. If cronjob is commented out in the /u/wandl/bin/rsync.sh script on the primary application server, reenable it.
      crontab -e

      The cronjob is owned by the administrative user ID who installed IP/MPLSView. By default, this is the wandl user.

    14. Uncomment (enable) the following line in crontab:
      #0,30 * * * * /u/wandl/bin/rsync.sh exec > /u/wandl/log/rsync.log
    15. Stop IP/MPLSView on the backup application server.
      /u/wandl/bin/stop_mplsview
    16. Make sure the cronjob is commented out (disabled).
      crontab -e
    17. If the cronjob is not disabled on the backup application server, comment out (disable) the following line in crontab:
      #0,30 * * * * /u/wandl/bin/rsync.sh exec > /u/wandl/log/rsync.log

    Modified: 2016-11-08