Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Navigation
Guide That Contains This Content
[+] Expand All
[-] Collapse All

    Troubleshooting Database Synchronization in a Distributed Environment

    During the MariaDB replication process, the IP/MPLSView database tables can become unsynchronized between the primary application server and the backup database server. An out-of-sync condition can occur for one or more of the following reasons:

    • Either server is improperly shut down.
    • Either server is interrupted by an event such as a power outage.

    To troubleshoot synchronization problems that can occur during the MariaDB replication process, you must determine whether the databases are unsynchronized, and if so, use the following the procedure to resynchronize the databases.

    To determine whether the database tables between the primary application server and the backup database server are synchronized:

    1. Check the last data received on both the primary application server and the backup database server.
      /u/wandl/bin/mysql_repl.sh lastdata

      When you issue the mysql_repl.sh lastdata command, the software displays the timestamp of the last data written to the database tables. If the command output for the primary application server and the backup database server displays the same timestamp, the database tables are synchronized. Conversely, if the output for the primary application server and the backup database server displays different timestamps, the database tables on the servers are potentially unsynchronized.

      The timestamp is an 11-digit number in the format YYYYMMDDIII, where YYYY is the 4-digit year, MM is the 2-digit month, DD is the 2-digit day, and III is the 3-digit interval. The month is represented as a 2-digit integer starting with 00 for January through 11 for December. The interval is represented as a multiple of 5 minutes after 12:00 A.M., with a total of 288 five-minute intervals in a 24-hour period (one day).

      For example, the timestamp 20101023133 (year 2010, month 10, day 23, interval 133) is November 23, 2010 at 11:05 A.M.

    2. Run the mysql_repl.sh lastdata command again after at least 5 minutes to confirm that the database tables are unsynchronized.

      Wait at least 5 minutes before running the command again to allow sufficient time for one MariaDB replication cycle to complete.

      If the output for the primary application server and the backup database server still displays different timestamps, the database tables are unsynchronized.

    The following procedure uses the MariaDB CLI to resynchronize the IP/MPLSView databases between the primary application server and the backup database server. In this procedure, the terms backup and slave are equivalent, as are the terms primary and master.

    To resynchronize the IP/MPLSView databases when they become unsynchronized:

    1. On the backup (slave) database, access the MariaDB CLI.
      . /u/wandl/bin/mplsenvsetup.sh
      /u/wandl/thirdparty/mysql/bin/mysql -uroot -pwandlroot -A wandltraffic
    2. Stop the backup database replication thread.
      stop slave;
    3. Reset (clean) the backup replication data on the backup database, and exit the MariaDB CLI.
      reset slave;
      exit
    4. Stop the backup (slave) database.
      /u/wandl/bin/.mysql stop slave
    5. On the primary (master) database, access the MariaDB CLI.
      . /u/wandl/bin/mplsenvsetup.sh
      /u/wandl/thirdparty/mysql/bin/mysql -uroot -pwandlroot -A wandltraffic
    6. Reset the master database replication record, and exit the MariaDB CLI.
      reset master;
      exit
    7. Stop IP/MPLSView on the primary application server.
      /u/wandl/bin/stop_mplsview
    8. (Optional) Back up the contents of the /u/wandl/data/mysql/data/wandltraffic directory on the backup database server.
    9. Copy the directory and all files in the /u/wandl/data/mysql/data/wandltraffic directory from the primary application server to the backup database server.
    10. Repair the MariaDB database tables on the unsynchronized server.
      /u/wandl/bin/fixmysql.sh
    11. When the repair completes, start IP/MPLSView on the primary application server.
      /u/wandl/bin/startup_mplsview
    12. Start the backup (slave) database.
      /u/wandl/bin/.mysql start slave
    13. Register the backup (slave) to master, and start replication thread.
      /u/wandl/bin/mysql_repl.sh registermaster
    14. On the backup database, access the MariaDB CLI.
      /u/wandl/bin/mplsenvsetup.sh
      /u/wandl/thirdparty/mysql/bin/mysql -uroot -pwandlroot -A wandltraffic
    15. Check the backup database for error messages, where database-drive is the letter of the drive on which the backup database resides (for example, \G).
      show slave status \database-drive;
    16. Exit the MariaDB CLI.
      exit
    17. After waiting at least 5 minutes, check the last data received on both the primary application server and the backup database server to make sure the database tables are resynchronized.
      /u/wandl/bin/mysql_repl.sh lastdata

      Waiting at least 5 minutes before running the mysql_repl.sh lastdata command allows sufficient time for one MariaDB replication cycle to complete.

      If the command output for the primary application server and the backup database server displays the same timestamp, the database tables are resynchronized.

    Modified: 2017-01-09