Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Navigation
Guide That Contains This Content
[+] Expand All
[-] Collapse All

    Installing and Administering IP/MPLSView Replication and Rsync in a Four-Server Environment

    This section describes how to set up, install, start, and administer IP/MPLSView in a four-server distributed environment. It includes step-by-step procedures for failing over to the backup server when the primary server goes down, synchronizing the data, and switching back to the primary server when it resumes operation.

    Figure 1 shows a typical setup for using IP/MPLSView in a distributed environment that consists of two application servers and two database servers. In this configuration, the primary and backup application servers run the IP/MPLSView application, and the primary and backup database servers run the IP/MPLSView database.

    For optimal performance and to simplify setup and administration, we recommend that you pair the primary application server with the primary database server, and the backup application server with the backup database server. After you complete the installation, verify that the remote database is working properly and that the servers are configured with the correct IP address pairings.

    Figure 1: Example IP/MPLSView Setup for a Four-Server Distributed Environment

    Example
IP/MPLSView Setup for a Four-Server Distributed Environment

    The example setup consists of the following hardware and software components:

    • One primary application server (labeled Primary AP) paired with one primary database server (labeled Primary DB)
    • One backup application server (labeled Backup AP) paired with one backup database server (labeled Backup DB)
    • Two traffic data collectors (each labeled DC)
    • Replication and rsync software running between the primary and backup application servers
    • Replication software running between the primary and backup database servers
    • MongoDB fault management database
    • MariaDB performance management database on the primary and backup database servers
    • Java Message Service (JMS), which sends messages between the primary and backup application servers and the traffic data collectors

    Installing and Administering IP/MPLS View in a Four-Server Distributed Environment

    To install IP/MPLSView with replication and rsync software in a four-server distributed environment:

    1. Follow the standard installation procedure in the installation directory.
      install_dir/server/install.sh

      You can install the replication and rsync software package as part of the IP/MPLSView installation. If you choose not to install the replication and rsync package during the standard installation procedure, you can install it later by running the replication/instrepl.sh script as the wandl user.

    2. Install the rsync and replication package when prompted.
      Install Rsync & Database Replication Package (default=no)? [y/n] y
    3. Configure the primary application server with the appropriate replication and rsync settings.
      Rsync & Replication Settings
      1.)Setup Rsync for Application Server.....YES
      2.)Install Database Replication Package...YES
      3.)Setup as Primary or Backup Server......PRIMARY
      4.)Preserve files on target server........YES
    4. Configure the primary database server with the appropriate replication and rsync settings.
      Rsync & Replication Settings
      1.)Setup Rsync for Application Server.....NO
      2.)Install Database Replication Package...YES
      3.)Setup as Primary or Backup Server......PRIMARY
      4.)Preserve files on target server........YES
    5. Configure the backup application server with the appropriate replication and rsync settings.
      Rsync & Replication Settings
      1.)Setup Rsync for Application Server.....YES
      2.)Install Database Replication Package...YES
      3.)Setup as Primary or Backup Server......BACKUP
      4.)Preserve files on target server........YES
    6. Configure the backup database server with the appropriate replication and rsync settings.
      Rsync & Replication Settings
      1.)Setup Rsync for Application Server.....NO
      2.)Install Database Replication Package...YES
      3.)Setup as Primary or Backup Server......BACKUP
      4.)Preserve files on target server........YES
    7. Enter the IP address of the corresponding backup server for the primary servers and, conversely, enter the IP address of the primary server on the corresponding backup servers.
      Please enter the IP address of the alternate MPLSView server:ip-address
    8. Enter the ssh directory of the alternate IP/MPLSView server if the actual paths do not conform to the default paths.
    9. Enter the SSH host key of the wandl user, for example, /home/wandl/.ssh/id_rsa.
    10. Check the size of the IP/MPLSView home directory (for example, /home/wandl) to make sure the backup server has sufficient disk space in the home directory.
      du -ks /home/wandl

      By default, the administrative home directory is not synchronized. To synchronize this directory, check the size of the home directory to make sure it is not too large to copy its entire contents to the backup server.

      After the installation, you can modify which directories and files are synchronized by editing the /u/wandl/bin/rsync.sh script. You must exclude the database files from the rsync.

    11. Synchronize directories and files when prompted.
      Synchronize the following files/directories?:
      You can later manually modify the entries in
      /u/wandl/bin/rsync.sh [YES]y
      1.) /u/wandl/data/
      2.) [YES] /u/wandl/db/config/snmptrap.store
      3.) [YES] /u/wandl/db/config/subscriptions.store
      4.) [YES] /u/wandl/db/config/eventtypes.store
      5.) [YES] /u/wandl/db/config/productionscopes.store
      6.) [YES] /u/wandl/db/config/collectioncmds.xml
      7.) [YES] /u/wandl/db/config/diagnosticcmds
      8.) [YES] /u/wandl/db/config/shownodecmds
      9.) [YES] /u/wandl/db/config/showvpncmds
      10.) [YES] /u/wandl/db/command/
      11.) [YES] /u/wandl/data/.network/
      12.) [YES] /u/wandl/data/.TaskManager/profile/
      13.) [YES] /u/wandl/data/.TaskManager/tmp/.diag
      14.) [YES] /u/wandl/data/device/
      15.) [YES] /u/wandl/data/ping/
      16.) [YES] /u/wandl/data/sla/
      17.) [YES] /u/wandl/data/summary/
      18.) [YES] /u/wandl/data/latency/
      19.) [YES] /u/wandl/data/event/
      20.) [YES] - /u/wandl/data/LDPTraffic/
      Please select a number to modify. [<CR>=accept]:
      Accept these values (default=no)? [y/n] y
      
    12. Schedule the time interval to use for synchronizing the backup application server with the primary application server.
      Please select the crontab interval in minutes (60): [0-60] <#>
    13. (Optional) To modify the time interval after the installation, run the following script:
      /install_dir/replication/instrepl.sh

      Alternatively, you can directly modify the crontab settings as the wandl user. For example, you can enter EDITOR=vi to set the editor to vi, export the EDITOR variable by using the export EDITOR command, and then run the crontab -e command.

    Starting IP/MPLSView in a Four-Server Environment

    To ensure that the data is synchronized when you use IP/MPLSView in a four-server distributed environment, you must start the paired primary application server and primary database server in active mode, and start the paired backup application server and backup database server in slave mode.

    To start IP/MPLSView on the primary and backup application servers:

    1. Make sure the MariaDB is started on the primary database server.
    2. Make sure the following entry appears in the /u/wandl/db/config/mongodb.conf file on both the primary application server and the backup application server:
      replSet = rs0
    3. Start IP/MPLSView.
      /u/wandl/bin/startup_mplsview
    4. Start MongoDB on the backup application server.
      mongodb.sh start
    5. Start MongoDB in slave mode on the primary application server.
      /u/wandl/bin/mongodb_repl.sh addslave
    6. Check the status of MongoDB on both application servers.
      /u/wandl/bin/mongodb_repl.sh status
      Primary MongoDB is running
      Primary replication position: 1
      Secondary: <ip-address>:27017 is up and running
      Secondary: <ip-address>:27017 replication position 1
    7. (Optional) To start IP/MPLSView if the addslave command does not cause the MongoDB to run in slave mode:
      1. Configure the primary and backup application servers with the replication settings in the /u/wandl/db/config/mongodb.conf file.
      2. Make sure the following entry appears in the /u/wandl/db/config/mongodb.conf file:
        replSet = rs0
      3. Access the MongoDB CLI (/u/wandl/thirdparty/mongodb/bin/mongo) on the primary application server, and issue the following commands to initiate MongoDB and run it in slave mode:
        run rs.initiate();
        rs.add({_id: 1, host: “<slave-ip>:27017”, priority: 0.5});
    8. Verify that rsync is enabled.
      crontab -l

      On the primary application server, the /u/wandl/bin/rsync.sh script runs according to the interval you specify during the installation. Depending on your synchronization requirements, specifying an interval of 30 minutes or 60 minutes should be sufficiently frequent.

      Each time the rsync.sh script runs, it synchronizes the directories in /u/wandl/data, excluding the MongoDB data directory, between the primary application server and the backup application servers. This is referred to as push synchronization.

    9. After the interval you specified for rsync, check the /u/wandl/data directory on the backup application server to make sure the file system data has been copied.

      You can exclude both the MongoDB databases and the MariaDB traffic collection databases. MongoDB is replicated by means of the replication process, and MariaDB is handled on the database server.

    Starting IP/MPLSView on the Database Servers in a Four-Server Environment

    To start IP/MPLSView on the primary and backup database servers:

    1. Start the MariaDB database on the primary (master) database server.
      /u/wandl/bin/startup_mplsview

      You must start MariaDB on the primary database server before you start IP/MPLSView on the primary application server.

    2. Start MariaDB on the backup database server in slave mode.
      /u/wandl/bin/.mysql start slave
    3. Register the master MariaDB server on the backup application server.
      /u/wandl/bin/.mysql registermaster
    4. Check the status of the MariaDB slave server.
      /u/wandl/bin/mysql_repl.sh status
      Slave Mysql server connected to Master: primary-server ip-address/hostname
    5. Check the status of the MariaDB master server.
      /u/wandl/bin/mysql_repl.sh status
      Master Mysql server is running
      Slave server:backup-server ip-address/hostname

    Configuring an Rsync Cronjob to Replicate Traffic Collection Data

    When you use IP/MPLSView, the database servers aggregate the day’s traffic collection at the end of each day and save the data in object format files. To replicate the traffic collection files from the database servers to the application servers as a cronjob, you must run the inst_agg_rsync.sh script once on each database server.

    You do not need to run the inst_agg_rsync.sh script every time you upgrade IP/MPLSView unless you are explicitly told to do so by the Juniper Networks Technical Assistance Center (JTAC).

    To configure the rsync cronjob on the database servers, run theinst_agg_rsync.sh script once on the primary database server and once on the backup database server:

    wandl@server /u/wandl/bin> ./inst_agg_rsync.sh
    Please enter the complete path of rsync [/usr/local/bin/rsync]:
    
    This script will generate replication scripts and schedule a task in the
    crontab, do you wish to continue? [n]:y
    The installation directory was detected as '/home/wandl/ipmplsview’, is this
    correct? (default=yes)? [y/n]
    
    [1] Add a target server
    [2] Remove a target server
    [3] Show current targets
    [4] Regenerate script files
    [5] Quit
    
    Please enter your selection: 1
    
    [Add server selected]
    
    Please enter
    IP address of the target server: 172.17.7.7
    Remote IP/MPLSView product home directory [/u/wandl]:
    Remote login ID associated with the SSH authorized key [wandl]:
    Remote rsync installation path [/usr/local/bin/rsync]:
    Aggregation rsync script generated...
    
    Settings for 172.17.7.7 saved successfully, press any key to continue...
    
    [1] Add a target server
    [2] Remove a target server
    [3] Show current targets
    [4] Regenerate script files
    [5] Quit
    
    Please enter your selection: 5
    

    Failing Over to the Backup Servers in a Four-Server Environment

    In the event that the IP/MPLSView primary application server or primary database server fails, you can fail over to the corresponding backup application server or backup database server and initiate operation on the backup servers until the primary servers recover.

    When you use IP/MPLSView in a four-server distributed environment, we recommend that you pair the primary application server with the primary database server, and the backup application server with the backup database server. As a result, when either the primary application server or primary database server fails, the server pair must fail over from primary to backup together.

    If either the primary database server or the primary application server fails, stop the remaining services on the primary server. In this situation, you must fail over the primary database server first to prevent the data gateway server that is running on the primary application server from losing connectivity with the database.

    To fail over to the backup database server if the primary database server fails:

    1. Stop the backup database server.
      /u/wandl/bin/stop_mplsview
    2. Start IP/MPLSView on the backup database server.
      /u/wandl/bin/startup_mplsview

      This action causes the backup database server to become the master (primary) database server.

    3. Make sure the aggregated traffic cronjob is disabled on the original primary database server and enabled on the backup database server.
    4. To fail over to the backup application server if the primary application server fails, stop the remaining services on the primary application server.
      /u/wandl/bin/stop_mplsview
    5. On the primary application server, stop the rsync cronjob for the /u/wandl/bin/rsync.sh script as both the root user and the wandl user.
      crontab -e

      The cronjob is owned by the administrative user ID who installed IP/MPLSView. By default, this is the wandl user.

    6. Comment out (disable) the following line in crontab:
      #0,30 * * * * /u/wandl/bin/rsync.sh exec > /u/wandl/log/rsync.log
    7. Stop the backup application server.
      /u/wandl/bin/stop_mplsview
    8. Make sure the rsync cronjob is disabled (commented out).
      crontab -e
    9. Start IP/MPLSView on the backup application server.
      /u/wandl/bin/startup_mplsview

      This action causes the backup application server to become the primary (master) application server.

    10. Comment out the following entry in the /u/wandl/db/config/mongodb.conf file:
      replSet = rs0
    11. Restart the MongoDB process.
      /u/wandl/bin/mongodb.sh stop
      /u/wandl/bin/mongodb.sh start
    12. Access the MongoDB CLI and issue the following commands:
      /u/wandl/thirdparty/mongodb/bin/mongo
      use local;
      db.dropDatabase();

    Synchronizing the Database from the Backup Server to the Original Primary Server

    When the original primary application server or the original primary database server resumes operating, you must synchronize the data from the backup server to the original primary server.

    To synchronize data from the backup database server to the original primary database server:

    1. Start the original primary MariaDB server in slave mode.
      /u/wandl/bin/.mysql start slave
      Slave Mysql server connected to master: backup-server ip-address
    2. Register the master MariaDB server on the backup application server.
      /u/wandl/bin/.mysql registermaster
    3. Check the last data received on both the primary application server and the backup application server to make sure the data is synchronized.
      /u/wandl/bin/mysql_repl.sh lastdata

      When you issue this command, the software displays an 11-digit timestamp that corresponds to the time of the last data collection. When both the primary MariaDB server and the master MariaDB server display the same 11-digit timestamp, the data is synchronized.

      For more information about the format of the timestamp, see Troubleshooting Database Synchronization in a Distributed Environment.

    4. To synchronize data from the backup application server to the original primary application server, start MongoDB.
      /u/wandl/bin/mongodb.sh start
    5. Leave (stop) the MongoDB cluster on the original primary application server.
      /u/wandl/bin/mongodb_repl.sh leavecluster
    6. On the backup application server, restart the MongoDB in cluster mode, and uncomment the following entry in the /u/wandl/db/config/mongodb.conf file:
      replSet = rs0
    7. Stop and restart the MongoDB process.
      /u/wandl/bin/mongodb.sh stop
      /u/wandl/bin/mongodb.sh start
    8. Start MongoDB in slave mode on the backup application server.
      /u/wandl/bin/mongodb_repl.sh addslave
    9. Stop IP/MPLSView on the backup application server.
      /u/wandl/bin/stop_mplsview
    10. If changes occurred in the network while the primary application server is out of service, use the following command only once on the backup application server to synchronize the original primary application server with the backup application server:
      /u/wandl/bin/rsync.sh exec
    11. Verify that the data has been synchronized.
      /u/wandl/bin/mongodb_repl.sh status
      /u/wandl/bin/mongodb_repl.sh lastdata

    Switching Back to the Original Primary Server

    After you synchronize the data from the backup application or database server to the original primary application or database server, you can switch back to the original primary server and resume operation.

    To switch back to the original primary database server:

    1. Stop IP/MPLSView on the backup database server and then on the primary database server, in that order.
      /u/wandl/bin/stop_mplsview
    2. Start the MariaDB server in slave mode on the backup database server.
      /u/wandl/bin/.mysql start slave
    3. Start MariaDB on the primary database server.
      /u/wandl/bin/startup_mplsview
    4. On the backup application server, register the MariaDB server with the master MariaDB server.
      /u/wandl/bin/mysql_repl.sh registermaster
    5. Check the status on the primary and backup MariaDB servers.
      /u/wandl/bin/mysql_repl.sh status
    6. Check the data consistency on the primary MariaDB server.
      /u/wandl/bin/mysql_repl.sh lastdata
    7. To switch back to the original primary application server, on the backup application server, start the MongoDB and then leave the cluster.
      /u/wandl/bin/mongodb.sh start
      /u/wandl/bin/mongodb_repl.sh leavecluster
    8. On the primary application server, start MongoDB and leave the cluster.
      /u/wandl/bin/mongodb.sh start
      /u/wandl/bin/mongodb_repl.sh leavecluster
      /u/wandl/bin/mongodb.sh stop
    9. Restart the MongoDB in cluster node, and uncomment the following entry in the /u/wandl/db/config/mongodb.conf file:
      replSet = rs0
    10. Start IP/MPLSView on the primary application server.
      /u/wandl/bin/startup_mplsview
    11. Start MongoDB in slave mode on the primary application server.
      /u/wandl/bin/mongodb_repl.sh addslave
    12. Check the status of MongoDB on the primary application server and then on the backup application server, in that order.
      /u/wandl/bin/mongodb_repl.sh status
    13. If cronjob is commented out in the /u/wandl/bin/rsync.sh script on the primary application server, reenable it.
      crontab -e

      The cronjob is owned by the administrative user ID who installed IP/MPLSView. By default, this is the wandl user.

    14. Uncomment (enable) the following line in crontab:
      #0,30 * * * * /u/wandl/bin/rsync.sh exec > /u/wandl/log/rsync.log

    Modified: 2016-11-08