Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Navigation
Guide That Contains This Content
[+] Expand All
[-] Collapse All

    Executing Job Management Scripts and Java Applications

    You execute job management scripts and Java applications to view information about jobs triggered from Junos Space nodes. The JobInfoCollector.jar Java application and the job management scripts are stored at the following location: /var/log/space-debug/debug-utilities/jobManagement. When you execute the JobInfoCollector.jar Java application and job management scripts, the output is stored as .txt files at the same location. You can also specify a custom path to store the output. The following sections list the steps to execute the scripts to collect information about jobs.

    Executing the Java Application to Collect Job Information

    You execute the JobInfoCollector.jar application to collect job information. Ensure that you have updated the MySQL query in the JobInfoCollector.properties file before executing the Java application. The file also contains an example configuration.

    To execute the Java application to collect job information:

    1. Log in to the CLI of the Junos Space node.
    2. Enter the administrator username and password at the Junos Space login prompt and press Enter.

      The default username is admin and the default password is abc123.

      The Junos Space Settings Menu is displayed.

    3. Enter 6 (if you are using a hardware appliance) or 7 (if you are using a virtual appliance) at the Junos Space Settings Menu prompt to run shell commands.

      You are prompted to enter your password.

      The following is a sample output from a virtual appliance:

      admin@10.206.41.183's password:
      Last login: Mon Aug 17 06:17:58 2015 from 10.206.41.42
      
      Welcome to the Junos Space network settings utility.
      
      Initializing, please wait
      
      
      Junos Space Settings Menu
      
      1> Change Password
      2> Change Network Settings
      3> Change Time Options
      4> Retrieve Logs
      5> Security
      6> Expand VM Drive Size
      7> (Debug) run shell
      
      A> Apply changes
      Q> Quit
      R> Redraw Menu
      
      Choice [1-7,AQR]: 7
    4. Type the administrator password and press Enter.

      The shell prompt appears.

    5. Type cd /var/log/space-debug/debug-utilities/jobManagement at the shell prompt and press Enter.
    6. (Optional) To view the list of job management scripts and Java applications, enter ls and press Enter.
    7. Type /usr/bin/java -jar JobInfoCollector.jar and press Enter.

      The following options will be displayed:

      Please select an option from below:
      1. Display job parent-children hierarchy
      2. Filter from job data
      3. Display jobs not yet picked up by Job Dispatcher
      
    8. Type 1, 2, or 3 to select an option.
    9. Enter the database username.
    10. Enter the database password.
    11. The inputs displayed depend on the option you selected.

      If you selected option 1, you need to enter the parent job ID.

      • Enter a job ID.

        The output from this command is saved to the JobHierarchy-<job-ID>.txt file in the same directory. The following is a sample output if you selected 1:

        =====Job details for 491529 =====
        
        NODE: 10.206.41.57
        STARTTIMESTAMP: 17 Jul 2015 17:51:00 GMT
        JOBSTATE: Done
        JOBSTATUS: Success
        ENDTIMESTAMP: 17 Jul 2015 17:52:15 GMT
        CHILDCOUNT: 2
        
         Child job details:
        
        ID || NODE || STARTTIMESTAMP || JOBSTATE || JOBSTATUS || ENDTIMESTAMP || CHILDCOUNT ||
         491530 | 10.206.41.57 | 17 Jul 2015 17:51:01 GMT | Done | Success | 17 Jul 2015 17:51:13 GMT | 0 |
         491531 | 10.206.41.57 | 17 Jul 2015 17:51:01 GMT | Done | Success | 17 Jul 2015 17:51:13 GMT | 0 |

      If you selected option 2, you need to enter a MySQL query.

      • Enter a MySQL query—for example, mostate=’Failure’.

        This query filters the jobs that failed. The following is a sample output if you selected 2:

        ID || NAME || NODE || STARTTIMESTAMP || JOBSTATE || JOBSTATUS || ENDTIMESTAMP ||
         196608 | Discover Network Elements-196608 | 10.206.41.184 | 10 Aug 2015 10:44:48 GMT | Done | Failure | 10 Aug 2015 10:48:47 GMT |
         196624 | Discover Network Elements-196624 | 10.206.41.184 | 10 Aug 2015 10:49:52 GMT | Done | Failure | 10 Aug 2015 10:49:58 GMT |
         196643 | Mail User Password-196643 | 10.206.41.184 | 10 Aug 2015 14:11:21 GMT | Done | Failure | 10 Aug 2015 14:11:21 GMT |
         196649 | Mail User Password-196649 | 10.206.41.184 | 11 Aug 2015 02:39:18 GMT | Done | Failure | 11 Aug 2015 02:39:18 GMT |
         196650 | Mail User Password-196650 | 10.206.41.184 | 11 Aug 2015 02:39:18 GMT | Done | Failure | 11 Aug 2015 02:39:18 GMT |
         196660 | Sync Files-196660 | 10.206.41.187 | 11 Aug 2015 11:54:13 GMT | Done | Failure | 11 Aug 2015 11:54:46 GMT |
         360448 | Resync Network Elements-360448 | 10.206.41.184 | 11 Aug 2015 18:06:07 GMT | Done | Failure | 11 Aug 2015 18:06:08 GMT |
         360449 | Discover Network Elements-360449 | 10.206.41.187 | 11 Aug 2015 18:12:43 GMT | Done | Failure | 11 Aug 2015 18:13:16 GMT |

      If you selected option 3, the list of unscheduled jobs is saved to the UnscheduledJobs.txt file in the same directory.

      • The following is a sample output if you selected 3:
        ID || NAME || JOBSTATE || PARENTJOBID ||
         393501 | ND Discovery-393501 | Done | 0 |
         393858 | Backup Configuration Files-393858 | Done | 0 |
         720906 | ND Discovery-720906 | Done | 0 |
         721067 | Backup Configuration Files-721067 | Done | 0 |
         721884 | Backup Configuration Files-721884 | Done | 0 |
         1048659 | Backup Configuration Files-1048659 | Done | 0 |
         1182260 | Enable Script-1182260 | InProgress | 0 |
         1182413 | Execute Script-1182413 | InProgress | 0 |
         1183914 | Backup Configuration Files-1183914 | Done | 0 |
         1184310 | Delete Device-1184310 | Done | 0 |
         1189992 | Stage Script-1189992 | InProgress | 0 |
         1190105 | Update Network Element-1190105 | InProgress | 0 |
         1442231 | Backup Configuration Files-1442231 | Done | 0 |
         1442232 | -1442232 | InProgress | 1184310 |
         1442233 | -1442233 | InProgress | 1184310 |
         1442234 | -1442234 | InProgress | 1184310 |
         1442235 | -1442235 | InProgress | 1184310 |
         1442236 | -1442236 | InProgress | 1184310 |
         1442237 | -1442237 | InProgress | 1184310 |
         1442238 | -1442238 | InProgress | 1184310 |
         1442239 | -1442239 | InProgress | 1184310 |
         1442240 | -1442240 | InProgress | 1184310 |
         1445400 | Generate SD LR Report-1445400 | Scheduled | 0 |
         1445528 | Cloud Infrastructure Event Purge-1445528 | Scheduled | 0 |
         1445602 | Policy Hits Collection-1445602 | Scheduled | 0 |
         1903749 | useraccounts-1903749 | Done | 0 |

    You can view the hierarchy of a subjob (input as the parent job ID), list of jobs that are currently unscheduled, or the output based on the query you entered in the JobInfoCollector.properties file.

    Executing the Script to View the Stack Trace of a Job

    You execute the getJobThreadDump.sh script to view the stack trace of a job.

    To execute the script to view the stack trace of a job:

    1. Log in to the CLI of the Junos Space node.
    2. Enter the administrator username and password at the Junos Space login prompt and press Enter.

      The default username is admin and the default password is abc123.

      The Junos Space Settings Menu is displayed.

    3. Enter 6 (if you are using a hardware appliance) or 7 (if you are using a virtual appliance) at the Junos Space Settings Menu prompt to run shell commands.

      You are prompted to enter your password.

    4. Type the administrator password and press Enter.

      The shell prompt appears.

    5. Type cd /var/log/space-debug/debug-utilities/jobManagement at the shell prompt and press Enter.
    6. (Optional) To view the list of job management scripts and Java applications, type ls and press Enter.
    7. Type ./getJobThreadDump.sh <job-ID> and press Enter.

      The output from this command is saved to the JobThreadDump-<job-ID>.txt file in the same directory.

    8. (Optional) To save the stack trace of the job at a custom output location, type ./getJobThreadDump.sh <job-ID> <output-file-path> and press Enter.

    Executing the Script to View Job Information on Nodes

    You execute the SystemLoadViewer.sh script to view job information on a node.

    To execute the script to view job information on a node:

    1. Log in to the CLI of the Junos Space node.
    2. Enter the administrator username and password at the Junos Space login prompt and press Enter.

      The default username is admin and the default password is abc123.

      The Junos Space Settings Menu is displayed.

    3. Enter 6 (if you are using a hardware appliance) or 7 (if you are using a virtual appliance) at the Junos Space Settings Menu prompt to run shell commands.

      You are prompted to enter your password.

    4. Type the administrator password and press Enter.

      The shell prompt appears.

    5. Type cd /var/log/space-debug/debug-utilities/jobManagement at the shell prompt and press Enter.
    6. (Optional) To view the list of job management scripts and Java applications, type ls and press Enter.
    7. Type ./SystemLoadViewer.sh and press Enter.

      The output from this command is saved to the ./SystemLoadInfo.txt file in the same directory. The following is a sample output from the command:

      ------------------------------------------------------------
      Time of execution: Mon Jul 20 06:54:14 UTC 2015
      ------------------------------------------------------------
      
      =====================NODE SUMMARY======================
      Node IP: 10.206.41.35
      Maximum JVM memory: 2040528896
      Estimated memory usage per node: 40465
      Reserved memory size: 20000000
      (Available Memory = Maximum JVM memor - Estimated memory usage per node - Reserved memory)
      Available Memory:2020488431
      Number of first root jobs on this node:1
      Number of next root jobs on this node:0
      Number of sub jobs on this node:0
      -------------------------------------------
      ===========================================
      Node load statistics:
      ===========================================
      Resource Id:autoresync-group:132832:user1@host:0360471000000000008G
      Type:autoresync-group
      SubType:132832
      Context type:FIRST_ROOT_JOB
      State:RUNNING
      Queue:
      Node IP:10.206.41.35
      Creation Time:2015-07-20 06:54:13.166
      Last modification time:2015-07-20 06:54:13.166
      Estimated Memory:40465
      ===========================================
      ============================SYSTEM LOAD SUMMARY FOR LOCAL NODE=============================
      Filesystem           1K-blocks      Used Available Use% Mounted on
      /dev/mapper/jmpvgnocf-lvvar
                            52772604   2584204  47373632   6% /var
      ======Command executed: LINES=20 COLUMNS=120 top -n 2 -b -c | tail -n +22==================
      top - 06:54:18 up 3 days, 19:01,  2 users,  load average: 1.24, 0.88, 0.82
      Tasks: 271 total,   2 running, 268 sleeping,   0 stopped,   1 zombie
      Cpu(s): 42.2%us, 23.5%sy,  0.5%ni, 32.7%id,  0.3%wa,  0.1%hi,  0.7%si,  0.0%st
      Mem:   7937672k total,  7827836k used,   109836k free,   539116k buffers
      Swap:  8193140k total,   132684k used,  8060456k free,  1246572k cached
      
        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
      30383 user1      20   0 83616  24m 2508 S 36.8  0.3   0:02.16 /usr/bin/perl /var/www/cgi-bin/secure/resourceMonitoring
       4204 jboss     20   0 4081m 2.7g  14m S 27.6 36.0 395:59.73 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/jre/bin/jav
      31240 user1      20   0  124m  15m 2448 S  7.6  0.2   0:00.23 perl -e use lib("/usr/nma/lib"); use NmaUtil; print NmaUtil
      31251 user1      20   0 72016  14m 2384 R  6.9  0.2   0:00.21 perl -e use lib("/usr/nma/lib"); use NmaDb; print NmaDb::ge
      30719 mysql     16  -4 2936m 744m 6628 S  3.6  9.6  62:17.79 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --
      32329 opennms   25   5 3446m 877m  21m S  3.0 11.3 164:35.59 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Djava.endor
       3120 user1      20   0 84612  21m 2460 S  1.0  0.3  13:51.23 /usr/bin/perl /usr/bin/jmp-sync
      30651 user1      20   0 12896 1436  952 R  1.0  0.0   0:00.07 top -n 2 -b -c
      13332 user1      20   0  154m 9824 4480 S  0.7  0.1  26:56.92 /usr/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a
      18186 postgres  20   0  268m  12m 9252 S  0.7  0.2   0:00.38 postgres: opennms opennms 127.0.0.1(55330) idle
       2589 jboss     20   0  924m 197m  10m S  0.3  2.5   6:11.24 java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log
      15335 apache    23   3  141m 4936 3012 S  0.3  0.1   0:00.62 /usr/sbin/httpd
      ======Command executed: COLUMNS=120 top -n 1 -b -c | egrep '(mysqld|java|postgres)'========
       4204 jboss     20   0 4081m 2.7g  14m S  1.9 36.0 395:59.74 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/jre/bin/jav
      32329 opennms   25   5 3446m 877m  21m S  1.9 11.3 164:35.60 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Djava.endor
       2556 jboss     20   0  527m  47m 9800 S  0.0  0.6   4:44.43 java -D[Process Controller] -server -Xms32m -Xmx128m -XX:Ma
       2589 jboss     20   0  924m 197m  10m S  0.0  2.5   6:11.24 java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log
       2886 postgres  20   0  268m  10m 7524 S  0.0  0.1   0:00.10 postgres: opennms opennms 127.0.0.1(55289) idle
       2888 postgres  20   0  268m  10m 7580 S  0.0  0.1   0:00.08 postgres: opennms opennms 127.0.0.1(55290) idle
       2889 postgres  20   0  268m  10m 8040 S  0.0  0.1   0:00.15 postgres: opennms opennms 127.0.0.1(55291) idle
       2891 postgres  20   0  268m  12m 9352 S  0.0  0.2   0:00.31 postgres: opennms opennms 127.0.0.1(55292) idle
       2893 postgres  20   0  267m  10m 7612 S  0.0  0.1   0:00.09 postgres: opennms opennms 127.0.0.1(55293) idle
       2895 postgres  20   0  267m 8968 6548 S  0.0  0.1   0:00.03 postgres: opennms opennms 127.0.0.1(55294) idle
       3507 jboss     20   0  528m  47m 9856 S  0.0  0.6   4:51.80 java -D[Process Controller] -server -Xms32m -Xmx128m -XX:Ma
       3561 jboss     20   0  566m 143m  10m S  0.0  1.8   5:43.77 java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log
       5570 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36280) idle
       5572 postgres  20   0  266m 4688 2792 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36281) idle
       5573 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36282) idle
       5575 postgres  20   0  266m 4684 2788 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36283) idle
       5578 postgres  20   0  266m 4696 2800 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36284) idle
       5579 postgres  20   0  266m 4688 2792 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36285) idle
       5581 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36286) idle
       5583 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36287) idle
       5586 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36288) idle
       5587 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36289) idle
       5590 postgres  20   0  266m 4692 2796 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36290) idle
       5591 postgres  20   0  266m 4692 2796 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36291) idle
       5593 postgres  20   0  266m 4696 2800 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36292) idle
       5595 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36293) idle
       5597 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36294) idle
       5599 postgres  20   0  266m 4684 2788 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36295) idle
       5601 postgres  20   0  268m  10m 8172 S  0.0  0.1   0:00.14 postgres: opennms opennms 127.0.0.1(36296) idle
       5603 postgres  20   0  268m  10m 7376 S  0.0  0.1   0:00.07 postgres: opennms opennms 127.0.0.1(36297) idle
       5605 postgres  20   0  267m 9.8m 7228 S  0.0  0.1   0:00.05 postgres: opennms opennms 127.0.0.1(36298) idle
       5607 postgres  20   0  267m 9972 7132 S  0.0  0.1   0:00.06 postgres: opennms opennms 127.0.0.1(36299) idle
       5609 postgres  20   0  268m  10m 8252 S  0.0  0.1   0:00.20 postgres: opennms opennms 127.0.0.1(36300) idle
       5611 postgres  20   0  268m  11m 8848 S  0.0  0.2   0:00.21 postgres: opennms opennms 127.0.0.1(36301) idle
       8123 postgres  20   0  268m  10m 8160 S  0.0  0.1   0:00.14 postgres: opennms opennms 127.0.0.1(56849) idle
      17000 user1      20   0 2299m  68m 8072 S  0.0  0.9   6:32.60 ./jre/bin/java -Djava.compiler=NONE -cp /usr/StorMan/RaidMa
      18186 postgres  20   0  268m  12m 9256 S  0.0  0.2   0:00.38 postgres: opennms opennms 127.0.0.1(55330) idle
      18187 postgres  20   0  268m  11m 8800 S  0.0  0.2   0:00.18 postgres: opennms opennms 127.0.0.1(55331) idle
      18188 postgres  20   0  268m  10m 7484 S  0.0  0.1   0:00.09 postgres: opennms opennms 127.0.0.1(55332) idle
      18190 postgres  20   0  268m  10m 7800 S  0.0  0.1   0:00.10 postgres: opennms opennms 127.0.0.1(55333) idle
      20287 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36339) idle
      20289 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36340) idle
      20291 postgres  20   0  266m 4684 2788 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36341) idle
      20297 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36342) idle
      20298 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36343) idle
      20301 postgres  20   0  267m 7448 5228 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36344) idle
      20306 postgres  20   0  267m 7804 5548 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36345) idle
      20308 postgres  20   0  267m 9660 6872 S  0.0  0.1   0:00.06 postgres: opennms opennms 127.0.0.1(36346) idle
      20311 postgres  20   0  268m  10m 7692 S  0.0  0.1   0:00.14 postgres: opennms opennms 127.0.0.1(36347) idle
      22848 postgres  20   0  266m 4684 2788 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(56892) idle
      22850 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(56893) idle
      22852 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(56894) idle
      22858 postgres  20   0  266m 6388 4312 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(56895) idle
      22860 postgres  20   0  268m  10m 7452 S  0.0  0.1   0:00.11 postgres: opennms opennms 127.0.0.1(56896) idle
      22863 postgres  20   0  268m  10m 7968 S  0.0  0.1   0:00.22 postgres: opennms opennms 127.0.0.1(56897) idle
      22864 postgres  20   0  267m 7608 5368 S  0.0  0.1   0:00.02 postgres: opennms opennms 127.0.0.1(56898) idle
      22866 postgres  20   0  268m  10m 7528 S  0.0  0.1   0:00.13 postgres: opennms opennms 127.0.0.1(56899) idle
      29715 user1      16  -4 13052 1068 1064 S  0.0  0.0   0:00.11 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid
      30719 mysql     16  -4 2936m 744m 6628 S  0.0  9.6  62:17.79 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --
      31064 postgres  20   0  265m  17m  16m S  0.0  0.2   0:32.35 /usr/pgsql-9.4/bin/postmaster -p 5432 -D /var/lib/pgsql/9.4
      31072 postgres  20   0  119m 1728  868 S  0.0  0.0   0:03.28 postgres: logger process
      31074 postgres  20   0  265m  42m  41m S  0.0  0.5   0:39.87 postgres: checkpointer process
      31075 postgres  20   0  265m 7304 6368 S  0.0  0.1   0:05.45 postgres: writer process
      31076 postgres  20   0  265m 6076 5156 S  0.0  0.1   0:53.84 postgres: wal writer process
      31077 postgres  20   0  265m 2688 1496 S  0.0  0.0   0:12.46 postgres: autovacuum launcher process
      31078 postgres  20   0  119m 2056  924 S  0.0  0.0   0:58.40 postgres: stats collector process
      31304 user1      20   0 61220  768  664 S  0.0  0.0   0:00.00 egrep (mysqld|java|postgres)
    8. (Optional) To save job information about a node at a custom output location, enter ./SystemLoadViewer.sh <output-file-path>.

    You can view information such as the memory on the nodes, number of root jobs and subjobs on each of the nodes, and so on.

    Modified: 2017-09-13