Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Navigation  Back up to About Overview 
  
[+] Expand All
[-] Collapse All

Utilities to Measure the Performance of an Existing System

This section describes the following utilities available in the Solaris 10 operating system used to measure and view the performance of an existing system:

prstat

The prstat or top utility shows the current active processes information on the system and reports statistics based on the selected output mode and sort order. The following is a sample output with no arguments for an SBRC system under load for reference:

PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
 28193 root      248M  142M cpu0    42    0   0:23:26  94% radius_generic/285
 27695 hadm      169M  129M sleep   59    0   0:00:17 0.1% mysqld/18
 27609 hadm      131M   25M sleep   59    0   0:00:15 0.1% ndb_mgmd/10
 28244 root     3888K 3248K cpu2    59    0   0:00:00 0.0% prstat/1
  1916 noaccess  166M   79M sleep   59    0   1:58:27 0.0% java/18
  1737 root     4968K 2160K sleep   59    0   0:28:51 0.0% nmbd/1
 27947 root     6688K 4424K sleep   59    0   0:00:00 0.0% sshd/1
…
Total: 82 processes, 559 lwps, load averages: 12.43, 5.73, 2.29

The command on Linux is similar:

top - 11:40:40 up 5 days, 19:52,  1 user,  load average: 0.09, 0.04, 0.01
Tasks: 101 total,   1 running, 100 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.8%sy,  0.0%ni, 97.8%id,  1.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3923200k total,  1872928k used,  2050272k free,   147100k buffers
Swap:  6160376k total,        0k used,  6160376k free,   481212k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 2631 root      -5   0 1871m 899m 8088 S  1.7 23.5 135:24.36 ndbmtd             
 1858 root      20   0  727m  90m  12m S  0.7  2.4  23:51.54 radius             
 1641 hadm      20   0  842m  60m 5652 S  0.3  1.6  24:30.98 mysqld             
    1 root      20   0 19272 1456 1176 S  0.0  0.0   0:00.78 init             
  

Table 8 describes the key fields of the prstat utility output.

Table 8: prstat Output Description

Field

Description

CPU

Shows the percentage of in-use CPU across the entire machine. The CPU time is regressed over the number of seconds, so this value does not show spike utilization, but is averaged over time.

SIZE

Shows the total working-set virtual memory size of the process (on some versions of Solaris, this reads as SWAP). In general, this should be less than the system memory size, and for the SBRC front-end application, as a 32-bit application, must be less than 4G.

RSS

The resident set size (RSS) shows the amount of memory that is paged in. Some paging out of SBRC is tolerable, but if this value is less than the working-set size, check for other parts of the system processes using too much memory.

NLWP

Number of light-weight processes (or threads).

TIME

Shows the amount of CPU time that the process has been executing since the start.

STATE

The information is not very useful in a multithreaded application in the thread roll-up display, but in prstat –L, it will show the state of the thread.

PRI

The information is not very useful in a multithreaded application in the thread roll-up display, but in prstat –L, it will show the thread priority. See Thread Priority for more information.

The following is an output of prstat –L showing per-thread utilization:

   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/LWPID      
 28193 root      252M  146M sleep   51    0   0:01:20 2.2% radius_generic/65
 28193 root      252M  146M sleep   53    0   0:01:21 2.0% radius_generic/28
 28193 root      252M  146M sleep   53    0   0:01:00 1.3% radius_generic/66
 28193 root      252M  146M sleep   54    0   0:00:14 0.8% radius_generic/156
 28193 root      252M  146M sleep   58    0   0:00:19 0.8% radius_generic/122
 28193 root      252M  146M cpu4    53    0   0:00:17 0.7% radius_generic/167
 28193 root      252M  146M sleep   55    0   0:00:21 0.7% radius_generic/93
…

Size and RSS refer to the process and not to the specific thread. STATE, CPU and Time refer to the specific thread.

Note: The maximum CPU for a given thread is equal to 100 divided by the number of virtual CPUs active on the system. In this case, since SBRC runs on an M3000 with 8 virtual CPUs enabled, the maximum would be 12.5 percent. Since all CPU utilization is less than 12.5 percent and evenly distributed across multiple threads, you will get more than 90 percent of the total CPU on the system. In this case, SBRC works well under the load.

For more general usage information on the prstat utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kqdk/index.html.

On Linux, the top command accepts the uppercase “H” subcommand to switch between per-process (the default) and per-thread reporting:

top - 11:34:34 up 5 days, 19:46,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 239 total,   1 running, 238 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.5%us,  1.2%sy,  0.0%ni, 97.2%id,  1.0%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   3923200k total,  1873052k used,  2050148k free,   147100k buffers
Swap:  6160376k total,        0k used,  6160376k free,   481212k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 2631 root      -5   0 1871m 899m 8088 S  1.3 23.5 104:18.06 ndbmtd             
 2636 root      -5   0 1871m 899m 8088 S  1.0 23.5  54:27.83 ndbmtd             
 1681 hadm      20   0  842m  60m 5652 S  0.7  1.6  18:51.67 mysqld             
 2637 root      -5   0 1871m 899m 8088 S  0.7 23.5  67:02.36 ndbmtd             
 1680 hadm      20   0  842m  60m 5652 S  0.3  1.6  18:12.45 mysqld             
 2624 root      20   0  727m  90m  12m S  0.3  2.4  15:55.08 radius             
 2625 root      20   0  727m  90m  12m S  0.3  2.4  16:52.83 radius             
 2634 root      -5   0 1871m 899m 8088 S  0.3 23.5   1:48.90 ndbmtd             
 2638 root      -5   0 1871m 899m 8088 S  0.3 23.5  41:18.35 ndbmtd             
 2639 root      -5   0 1871m 899m 8088 S  0.3 23.5  35:21.34 ndbmtd             
 2666 root      -5   0 1871m 899m 8088 S  0.3 23.5   2:37.54 ndbmtd       

Unlike Solaris, the default %CPU on Linux totals to 100% times the number of processors. Thus, 100% utilization for a thread corresponds to one virtual CPU. It does not appear to be the case that 100% CPU utilization is realistic on a hyper threaded system. Further, in this calculation, %System time is not accounted for.

It is safe to assume, given that the highest CPU utilization threads for SBR are generally those involved with network I/O (which takes the bulk of the %SY time), that if the highest CPU utilization thread plus the overall %Sys time approximates to 100%, then that thread is at its maximum.

sar

This is a system activity reporter utility. This Solaris command shows the overall activity of the system. The following is a sample output:

# sar 1 10 [every 1 second, display the results 10 times]
SunOS sbr-perf2 5.10 Generic_141444-09 sun4u    01/10/2011

15:14:10    %usr    %sys    %wio   %idle
15:14:11      70      25       0       5
15:14:12      71      26       0       3
…

For more general usage information on the sar utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kqgf/index.html.

The following is a sample output:

# sar –A 1 10 [every 1 second display all the results 10 times]
15:13:06    %usr    %sys    %wio   %idle
           device        %busy   avque   r+w/s  blks/s  avwait  avserv
         runq-sz %runocc swpq-sz %swpocc
         bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
         swpin/s bswin/s swpot/s bswot/s pswch/s
         scall/s sread/s swrit/s  fork/s  exec/s rchar/s wchar/s
          iget/s namei/s dirbk/s
         rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s
          proc-sz    ov  inod-sz    ov  file-sz    ov   lock-sz
           msg/s  sema/s
          atch/s  pgin/s ppgin/s  pflt/s  vflt/s slock/s
          pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
         freemem freeswap
         sml_mem   alloc  fail  lg_mem   alloc  fail  ovsz_alloc  fail
…
15:13:08      72      26       0       2
           nfs1              0     0.0       0       0     0.0     0.0
           nfs2              0     0.0       0       0     0.0     0.0
           nfs3              0     0.0       0       0     0.0     0.0
           nfs8              0     0.0       0       0     0.0     0.0
           nfs9              0     0.0       0       0     0.0     0.0
           sd0               0     0.0       0       0     0.0     0.0
           sd0,a             0     0.0       0       0     0.0     0.0
           sd0,b             0     0.0       0       0     0.0     0.0
           sd0,c             0     0.0       0       0     0.0     0.0
           sd1               2     0.0       3    1958     0.0     8.0
           sd1,a             0     0.0       0       0     0.0     0.0
           sd1,b             0     0.0       0       0     0.0     0.0
           sd1,c             0     0.0       0       0     0.0     0.0
           sd1,d             0     0.0       0       0     0.0     0.0
           sd1,e             0     0.0       0       0     0.0     5.8
           sd1,f             2     0.0       2    1956     0.0     9.0
           sd1,g             0     0.0       0       0     0.0     0.0
           sd2               0     0.0       0       0     0.0     0.0
             7.0     100     0.0       0
               0    7935     100       8     240      97       0       0
            0.00     0.0    0.00     0.0  143189
          346430      26   22887    0.00    0.00    6541 3517768
               1       2       0
               0       0    1903       0       0       0
           86/30000    0 32815/129797    0  615/615     0    0/0   
            0.00    0.00
            0.00    0.00    0.00    0.00    0.00    0.00
             0.00     0.00     0.00     0.00     0.00
         1548184 39927664
         423872064 45155219     0 1032183808 911917808     0   165756928     0
…

The sar utility runs in the background to display metrics on the system, with an interval of every few minutes on a nominal system, or every 5 or 10 seconds on a system that is under examination or undergoing troubleshooting to help diagnose system anomalies that impact SBRC (for example, to identify periods of spikiness).

iostat

The iostat utility reports I/O statistics such as terminal, disk, and tape I/O activity, as well as CPU utilization. The following is a sample output:

# iostat 1 10
   tty        sd0           sd1           sd2           nfs1           cpu
 tin tout kps tps serv  kps tps serv  kps tps serv  kps tps serv   us sy wt id
   0   55  11   1    7   93   2   14    0   0    0    0   0    0    6  2  0 91
   0  235   0   0    0  1021   1   13    0   0    0    0   0    0   72 25  0  3
   0   81   0   0    0  1026   1   10    0   0    0    0   0    0   72 25  0  3
   0   81   0   0    0  1024   1   13    0   0    0    0   0    0   72 25  0  3
…

If the CPU utilization from prstat is relatively low but the I/O utilization (kps on any device) is nearing the I/O bandwidth utilization, you may have a problem with I/O.

For more general usage information on the iostat utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kq4q/index.html.

vmstat

The vmstat utility shows virtual memory usage statistics regarding kernel thread, virtual memory, disk, trap, and CPU activity. The following is a sample output:

# vmstat 1 10
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s0 s1 s2 --   in   sy   cs us sy id
 0 0 0 19956200 13155320 43 379 11 26 26 0 4 1 2 -0  0 8658 31671 14768 6 2 91
 11 0 0 19964120 12374728 9 40 0 0 0  0  0  0  1  0  0 60815 358620 149035 72 25 3
 3 0 0 19963920 12374576 0 5 0  0  0  0  0  0  0  0  0 61047 352131 146928 72 25 3

The vmstat utility is useful to determine if the memory utilization is too high. For example, any high pi/po values indicate memory thrashing (that is, more memory is in use by the working set than is available on the system), and you should diagnose for stale sessions, leaks, or add more memory.

For more general usage information on the vmstat utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kqms/index.html.

prtdiag and sosreport

The prtdiag and sosreport utilities displays system configuration and diagnostic information. The following is a sample output:

# prtdiag
System Configuration:  Sun Microsystems  sun4u Sun SPARC Enterprise M3000 Server
System clock frequency: 1224 MHz
Memory size: 16384 Megabytes
==================================== CPUs =================================== 
      CPU                 CPU                         Run    L2$    CPU   CPU 
LSB   Chip                 ID                         MHz     MB    Impl. Mask
---   ----  ----------------------------------------  ----   ---    ----- ----
 00     0      0,   1,   2,   3,   4,   5,   6,   7   2750   5.0        7  161
============================ Memory Configuration ===========================  
       Memory  Available           Memory     DIMM    # of  Mirror  Interleave
LSB    Group   Size                Status     Size    DIMMs Mode    Factor    
---    ------  ------------------  -------    ------  ----- ------- ----------
 00    A         8192MB            okay       2048MB      4 no       2-way
 00    B         8192MB            okay       2048MB      4 no       2-way
========================= I/O Cards ==========================================
…

For more general usage information on the prtdiag utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kqdn/index.html.

An equivalent of prtdiag on Linux is a sosreport (which may require an additional package, which can be installed with “yum install sosreport”). The sosreport generates a package of files in /tmp, which can be inspected or forwarded to JTAC.

sosreport
sosreport (version 2.2)

This utility will collect some detailed  information about the
hardware and setup of your Red Hat Enterprise Linux system.
The information is collected and an archive is  packaged under
/tmp, which you can send to a support representative.
Red Hat Enterprise Linux will use this information for diagnostic purposes ONLY
and it will be considered confidential information.

This process may take a while to complete.
No changes will be made to your system.

Press ENTER to continue, or CTRL-C to quit.

Please enter your first initial and last name [defaultusername]: juser
Please enter the case number that you are generating this report for: 12345

  Running plugins. Please wait ...

  Completed [51/51] ...      
Creating compressed archive...

Your sosreport has been generated and saved in:
  /tmp/sosreport-juser.12345-20120312121724-8658.tar.xz

The md5sum is: 68c7948130edf27d71ccc683ebe28658

Please send this file to your support representative.

The report can be viewed as follows:

# xz -d sosreport-juser.12345-20120312121724-8658.tar.xz
# tar xf sosreport-juser.12345-20120312121724-8658.tar
# cd %ltmachinename%gt-%ltdatetime%gt; ls
boot       etc             java         lspci    pstree    sos_commands  uptime
chkconfig  free            lib          mount    root      sos_logs      var
date       hostname        lsb-release  netstat  route     sos_reports
df         ifconfig        lsmod        proc     sar12     sys
dmidecode  installed-rpms  lsof         ps       sestatus  uname

The preceding generated trees may be inspected freely and remotely, and are very valuable for debugging issues.

psrinfo

The psrinfo utility displays information about virtual processors. The following is an example output:

# psrinfo
0       on-line   since 07/14/2010 18:36:36
1       on-line   since 10/04/2010 11:41:31
2       on-line   since 07/14/2010 18:36:36
3       on-line   since 10/04/2010 11:41:31
4       on-line   since 07/14/2010 18:36:36
5       on-line   since 10/04/2010 11:41:31
6       on-line   since 07/14/2010 18:36:36
7       on-line   since 10/04/2010 11:41:31

For more general usage information on the psrinfo utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kqdt/index.html.

psradm

The psradm utility changes the status of the virtual processors. The following is a sample output:

# psradm -f 1
# psrinfo
0       on-line   since 07/14/2010 18:36:36
1       off-line  since 01/10/2011 15:19:20
2       on-line   since 07/14/2010 18:36:36
3       on-line   since 10/04/2010 11:41:31
4       on-line   since 07/14/2010 18:36:36
5       on-line   since 10/04/2010 11:41:31
6       on-line   since 07/14/2010 18:36:36
7       on-line   since 10/04/2010 11:41:31
# psradm -n 1
# psrinfo
0       on-line   since 07/14/2010 18:36:36
1       on-line   since 01/10/2011 15:19:47
2       on-line   since 07/14/2010 18:36:36
3       on-line   since 10/04/2010 11:41:31
4       on-line   since 07/14/2010 18:36:36
5       on-line   since 10/04/2010 11:41:31
6       on-line   since 07/14/2010 18:36:36
7       on-line   since 10/04/2010 11:41:31

For more general usage information on the psradm utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kqds/index.html.

cpuinfo on Linux

On Linux, the equivalent of psrinfo is /proc/cpuinfo, which returns information like:

# more /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           X5687  @ 3.60GHz
stepping        : 2
cpu MHz         : 1600.000
cache size      : 12288 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat 
pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp 
lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf 
pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse
4_1 sse4_2 popcnt aes lahf_lm ida arat epb tpr_shadow vnmi flexpriority ept vpid
bogomips        : 7197.00
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:
processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           X5687  @ 3.60GHz
...etc...

Please see the man page for “proc” for more information.

Summary information can be seen via the lscpu command.

# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    4
CPU socket(s):         2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 44
Stepping:              2
CPU MHz:               1600.000
BogoMIPS:              7197.24
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              12288K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15

Please see the man page for lscpu for more information.

Taskset on Linux

Taskset is used to set or retrieve the CPU affinity of a running process, given a PID or a command. This could be used to increase the performance in certain instances.

taskset –p –c <cpuset> <PID>

Example:

taskset –p –c 0-7 20320

Where –p option indicates the last argument is a PID, the –c option indicates the second-to-last argument is a list of CPU IDs (rather than, the default, a bitmap representing which CPUs). The effect of this command would be to make process 20320 bias to the first 8 cores.

Disabling a CPU on Linux

To temporarily disable a CPU on Linux, use the following command:

# echo 0 >> /sys/devices/system/cpu/cpu[#]/online

Where [#] is the processor ID of the core to disable. It appears that disabling CPU 0 is not permitted. To enable, echo the value 1. It does not appear to persist between reboots.

System Statistics

You use the Statistics page in Web GUI to display summary statistics for authentication, accounting, and proxy forwarding transactions. You can measure the current, average, and peak TPS values of authentication and accounting transactions.

Authentication statistics summarize the number of authentication acceptances and rejections, with summary totals for each type of rejection or retry (see Figure 2).

Figure 1: Viewing Authentication Statistics

Viewing
Authentication Statistics

Figure 2: Viewing Authentication Statistics

Viewing
Authentication Statistics

Accounting statistics provide information such as the number of transaction starts and stops and the reasons for rejecting attempted transactions. The start and stop numbers rarely match, because many transactions can be in progress at any given time (see Figure 4).

Figure 3: Viewing Accounting Statistics

Viewing Accounting Statistics

Figure 4: Viewing Accounting Statistics

Viewing
Accounting Statistics

Modified: 2016-11-15