Utilities to Measure the Performance of an Existing System
This section describes the following utilities available in the Solaris 10 operating system used to measure and view the performance of an existing system:
prstat
prstat
The prstat or top utility shows the current active processes information on the system and reports statistics based on the selected output mode and sort order. The following is a sample output with no arguments for an SBRC system under load for reference:
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 28193 root 248M 142M cpu0 42 0 0:23:26 94% radius_generic/285 27695 hadm 169M 129M sleep 59 0 0:00:17 0.1% mysqld/18 27609 hadm 131M 25M sleep 59 0 0:00:15 0.1% ndb_mgmd/10 28244 root 3888K 3248K cpu2 59 0 0:00:00 0.0% prstat/1 1916 noaccess 166M 79M sleep 59 0 1:58:27 0.0% java/18 1737 root 4968K 2160K sleep 59 0 0:28:51 0.0% nmbd/1 27947 root 6688K 4424K sleep 59 0 0:00:00 0.0% sshd/1 … Total: 82 processes, 559 lwps, load averages: 12.43, 5.73, 2.29
The command on Linux is similar:
top - 11:40:40 up 5 days, 19:52, 1 user, load average: 0.09, 0.04, 0.01
Tasks: 101 total, 1 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.8%sy, 0.0%ni, 97.8%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3923200k total, 1872928k used, 2050272k free, 147100k buffers
Swap: 6160376k total, 0k used, 6160376k free, 481212k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2631 root -5 0 1871m 899m 8088 S 1.7 23.5 135:24.36 ndbmtd
1858 root 20 0 727m 90m 12m S 0.7 2.4 23:51.54 radius
1641 hadm 20 0 842m 60m 5652 S 0.3 1.6 24:30.98 mysqld
1 root 20 0 19272 1456 1176 S 0.0 0.0 0:00.78 init
Table 9 describes the key fields of the prstat utility output.
Table 9: prstat Output Description
Field | Description |
|---|---|
CPU | Shows the percentage of in-use CPU across the entire machine. The CPU time is regressed over the number of seconds, so this value does not show spike utilization, but is averaged over time. |
SIZE | Shows the total working-set virtual memory size of the process (on some versions of Solaris, this reads as SWAP). In general, this should be less than the system memory size, and for the SBRC front-end application, as a 32-bit application, must be less than 4G. |
RSS | The resident set size (RSS) shows the amount of memory that is paged in. Some paging out of SBRC is tolerable, but if this value is less than the working-set size, check for other parts of the system processes using too much memory. |
NLWP | Number of light-weight processes (or threads). |
TIME | Shows the amount of CPU time that the process has been executing since the start. |
STATE | The information is not very useful in a multithreaded application in the thread roll-up display, but in prstat –L, it will show the state of the thread. |
PRI | The information is not very useful in a multithreaded application in the thread roll-up display, but in prstat –L, it will show the thread priority. See Thread Priority for more information. |
The following is an output of prstat –L showing per-thread utilization:
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID 28193 root 252M 146M sleep 51 0 0:01:20 2.2% radius_generic/65 28193 root 252M 146M sleep 53 0 0:01:21 2.0% radius_generic/28 28193 root 252M 146M sleep 53 0 0:01:00 1.3% radius_generic/66 28193 root 252M 146M sleep 54 0 0:00:14 0.8% radius_generic/156 28193 root 252M 146M sleep 58 0 0:00:19 0.8% radius_generic/122 28193 root 252M 146M cpu4 53 0 0:00:17 0.7% radius_generic/167 28193 root 252M 146M sleep 55 0 0:00:21 0.7% radius_generic/93 …
Size and RSS refer to the process and not to the specific thread. STATE, CPU and Time refer to the specific thread.
The maximum CPU for a given thread is equal to 100 divided by the number of virtual CPUs active on the system. In this case, since SBRC runs on an M3000 with 8 virtual CPUs enabled, the maximum would be 12.5 percent. Since all CPU utilization is less than 12.5 percent and evenly distributed across multiple threads, you will get more than 90 percent of the total CPU on the system. In this case, SBRC works well under the load.
For more general usage information on the prstat utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kqdk/index.html.
On Linux, the top command accepts the uppercase “H” subcommand to switch between per-process (the default) and per-thread reporting:
top - 11:34:34 up 5 days, 19:46, 1 user, load average: 0.00, 0.00, 0.00 Tasks: 239 total, 1 running, 238 sleeping, 0 stopped, 0 zombie Cpu(s): 0.5%us, 1.2%sy, 0.0%ni, 97.2%id, 1.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 3923200k total, 1873052k used, 2050148k free, 147100k buffers Swap: 6160376k total, 0k used, 6160376k free, 481212k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2631 root -5 0 1871m 899m 8088 S 1.3 23.5 104:18.06 ndbmtd 2636 root -5 0 1871m 899m 8088 S 1.0 23.5 54:27.83 ndbmtd 1681 hadm 20 0 842m 60m 5652 S 0.7 1.6 18:51.67 mysqld 2637 root -5 0 1871m 899m 8088 S 0.7 23.5 67:02.36 ndbmtd 1680 hadm 20 0 842m 60m 5652 S 0.3 1.6 18:12.45 mysqld 2624 root 20 0 727m 90m 12m S 0.3 2.4 15:55.08 radius 2625 root 20 0 727m 90m 12m S 0.3 2.4 16:52.83 radius 2634 root -5 0 1871m 899m 8088 S 0.3 23.5 1:48.90 ndbmtd 2638 root -5 0 1871m 899m 8088 S 0.3 23.5 41:18.35 ndbmtd 2639 root -5 0 1871m 899m 8088 S 0.3 23.5 35:21.34 ndbmtd 2666 root -5 0 1871m 899m 8088 S 0.3 23.5 2:37.54 ndbmtd
Unlike Solaris, the default %CPU on Linux totals to 100% times the number of processors. Thus, 100% utilization for a thread corresponds to one virtual CPU. It does not appear to be the case that 100% CPU utilization is realistic on a hyper threaded system. Further, in this calculation, %System time is not accounted for.
It is safe to assume, given that the highest CPU utilization threads for SBR are generally those involved with network I/O (which takes the bulk of the %SY time), that if the highest CPU utilization thread plus the overall %Sys time approximates to 100%, then that thread is at its maximum.
sar
sar
This is a system activity reporter utility. This Solaris command shows the overall activity of the system. The following is a sample output:
# sar 1 10 [every 1 second, display the results 10 times]SunOS sbr-perf2 5.10 Generic_141444-09 sun4u 01/10/2011 15:14:10 %usr %sys %wio %idle 15:14:11 70 25 0 5 15:14:12 71 26 0 3 …
For more general usage information on the sar utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kqgf/index.html.
The following is a sample output:
# sar –A 1 10 [every 1 second display all the results
10 times]15:13:06 %usr %sys %wio %idle
device %busy avque r+w/s blks/s avwait avserv
runq-sz %runocc swpq-sz %swpocc
bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
swpin/s bswin/s swpot/s bswot/s pswch/s
scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s
iget/s namei/s dirbk/s
rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s
proc-sz ov inod-sz ov file-sz ov lock-sz
msg/s sema/s
atch/s pgin/s ppgin/s pflt/s vflt/s slock/s
pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
freemem freeswap
sml_mem alloc fail lg_mem alloc fail ovsz_alloc fail
…
15:13:08 72 26 0 2
nfs1 0 0.0 0 0 0.0 0.0
nfs2 0 0.0 0 0 0.0 0.0
nfs3 0 0.0 0 0 0.0 0.0
nfs8 0 0.0 0 0 0.0 0.0
nfs9 0 0.0 0 0 0.0 0.0
sd0 0 0.0 0 0 0.0 0.0
sd0,a 0 0.0 0 0 0.0 0.0
sd0,b 0 0.0 0 0 0.0 0.0
sd0,c 0 0.0 0 0 0.0 0.0
sd1 2 0.0 3 1958 0.0 8.0
sd1,a 0 0.0 0 0 0.0 0.0
sd1,b 0 0.0 0 0 0.0 0.0
sd1,c 0 0.0 0 0 0.0 0.0
sd1,d 0 0.0 0 0 0.0 0.0
sd1,e 0 0.0 0 0 0.0 5.8
sd1,f 2 0.0 2 1956 0.0 9.0
sd1,g 0 0.0 0 0 0.0 0.0
sd2 0 0.0 0 0 0.0 0.0
7.0 100 0.0 0
0 7935 100 8 240 97 0 0
0.00 0.0 0.00 0.0 143189
346430 26 22887 0.00 0.00 6541 3517768
1 2 0
0 0 1903 0 0 0
86/30000 0 32815/129797 0 615/615 0 0/0
0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
1548184 39927664
423872064 45155219 0 1032183808 911917808 0 165756928 0
…
The sar utility runs in the background to display metrics on the system, with an interval of every few minutes on a nominal system, or every 5 or 10 seconds on a system that is under examination or undergoing troubleshooting to help diagnose system anomalies that impact SBRC (for example, to identify periods of spikiness).
iostat
iostat
The iostat utility reports I/O statistics such as terminal, disk, and tape I/O activity, as well as CPU utilization. The following is a sample output:
# iostat 1 10tty sd0 sd1 sd2 nfs1 cpu tin tout kps tps serv kps tps serv kps tps serv kps tps serv us sy wt id 0 55 11 1 7 93 2 14 0 0 0 0 0 0 6 2 0 91 0 235 0 0 0 1021 1 13 0 0 0 0 0 0 72 25 0 3 0 81 0 0 0 1026 1 10 0 0 0 0 0 0 72 25 0 3 0 81 0 0 0 1024 1 13 0 0 0 0 0 0 72 25 0 3 …
If the CPU utilization from prstat is relatively low but the I/O utilization (kps on any device) is nearing the I/O bandwidth utilization, you may have a problem with I/O.
For more general usage information on the iostat utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kq4q/index.html.
vmstat
vmstat
The vmstat utility shows virtual memory usage statistics regarding kernel thread, virtual memory, disk, trap, and CPU activity. The following is a sample output:
# vmstat 1 10kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s1 s2 -- in sy cs us sy id 0 0 0 19956200 13155320 43 379 11 26 26 0 4 1 2 -0 0 8658 31671 14768 6 2 91 11 0 0 19964120 12374728 9 40 0 0 0 0 0 0 1 0 0 60815 358620 149035 72 25 3 3 0 0 19963920 12374576 0 5 0 0 0 0 0 0 0 0 0 61047 352131 146928 72 25 3
The vmstat utility is useful to determine if the memory utilization is too high. For example, any high pi/po values indicate memory thrashing (that is, more memory is in use by the working set than is available on the system), and you should diagnose for stale sessions, leaks, or add more memory.
For more general usage information on the vmstat utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kqms/index.html.
prtdiag and sosreport
prtdiag and sosreport
The prtdiag and sosreport utilities displays system configuration and diagnostic information. The following is a sample output:
# prtdiagSystem Configuration: Sun Microsystems sun4u Sun SPARC Enterprise M3000 Server
System clock frequency: 1224 MHz
Memory size: 16384 Megabytes
==================================== CPUs ===================================
CPU CPU Run L2$ CPU CPU
LSB Chip ID MHz MB Impl. Mask
--- ---- ---------------------------------------- ---- --- ----- ----
00 0 0, 1, 2, 3, 4, 5, 6, 7 2750 5.0 7 161
============================ Memory Configuration ===========================
Memory Available Memory DIMM # of Mirror Interleave
LSB Group Size Status Size DIMMs Mode Factor
--- ------ ------------------ ------- ------ ----- ------- ----------
00 A 8192MB okay 2048MB 4 no 2-way
00 B 8192MB okay 2048MB 4 no 2-way
========================= I/O Cards ==========================================
…
For more general usage information on the prtdiag utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kqdn/index.html.
An equivalent of prtdiag on Linux is a sosreport (which may require an additional package, which can be installed with “yum install sosreport”). The sosreport generates a package of files in /tmp, which can be inspected or forwarded to JTAC.
sosreportsosreport (version 2.2) This utility will collect some detailed information about the hardware and setup of your Red Hat Enterprise Linux system. The information is collected and an archive is packaged under /tmp, which you can send to a support representative. Red Hat Enterprise Linux will use this information for diagnostic purposes ONLY and it will be considered confidential information. This process may take a while to complete. No changes will be made to your system. Press ENTER to continue, or CTRL-C to quit. Please enter your first initial and last name [defaultusername]: juser Please enter the case number that you are generating this report for: 12345 Running plugins. Please wait ... Completed [51/51] ... Creating compressed archive... Your sosreport has been generated and saved in: /tmp/sosreport-juser.12345-20120312121724-8658.tar.xz The md5sum is: 68c7948130edf27d71ccc683ebe28658 Please send this file to your support representative.
The report can be viewed as follows:
# xz -d sosreport-juser.12345-20120312121724-8658.tar.xz# tar xf sosreport-juser.12345-20120312121724-8658.tar# cd %ltmachinename%gt-%ltdatetime%gt; ls boot etc java lspci pstree sos_commands uptime chkconfig free lib mount root sos_logs var date hostname lsb-release netstat route sos_reports df ifconfig lsmod proc sar12 sys dmidecode installed-rpms lsof ps sestatus uname
The preceding generated trees may be inspected freely and remotely, and are very valuable for debugging issues.
psrinfo
psrinfo
The psrinfo utility displays information about virtual processors. The following is an example output:
# psrinfo0 on-line since 07/14/2010 18:36:36 1 on-line since 10/04/2010 11:41:31 2 on-line since 07/14/2010 18:36:36 3 on-line since 10/04/2010 11:41:31 4 on-line since 07/14/2010 18:36:36 5 on-line since 10/04/2010 11:41:31 6 on-line since 07/14/2010 18:36:36 7 on-line since 10/04/2010 11:41:31
For more general usage information on the psrinfo utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kqdt/index.html.
psradm
psradm
The psradm utility changes the status of the virtual processors. The following is a sample output:
# psradm -f 1# psrinfo0 on-line since 07/14/2010 18:36:36 1 off-line since 01/10/2011 15:19:20 2 on-line since 07/14/2010 18:36:36 3 on-line since 10/04/2010 11:41:31 4 on-line since 07/14/2010 18:36:36 5 on-line since 10/04/2010 11:41:31 6 on-line since 07/14/2010 18:36:36 7 on-line since 10/04/2010 11:41:31
# psradm -n 1# psrinfo0 on-line since 07/14/2010 18:36:36 1 on-line since 01/10/2011 15:19:47 2 on-line since 07/14/2010 18:36:36 3 on-line since 10/04/2010 11:41:31 4 on-line since 07/14/2010 18:36:36 5 on-line since 10/04/2010 11:41:31 6 on-line since 07/14/2010 18:36:36 7 on-line since 10/04/2010 11:41:31
For more general usage information on the psradm utility, see the man page at http://download.oracle.com/docs/cd/E19253-01/816-5166/6mbb1kqds/index.html.
cpuinfo on Linux
cpuinfo on Linux
On Linux, the equivalent of psrinfo is /proc/cpuinfo, which returns information like:
# more /proc/cpuinfoprocessor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Intel(R) Xeon(R) CPU X5687 @ 3.60GHz stepping : 2 cpu MHz : 1600.000 cache size : 12288 KB physical id : 0 siblings : 8 core id : 0 cpu cores : 4 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse 4_1 sse4_2 popcnt aes lahf_lm ida arat epb tpr_shadow vnmi flexpriority ept vpid bogomips : 7197.00 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Intel(R) Xeon(R) CPU X5687 @ 3.60GHz ...etc...
Please see the man page for “proc” for more information.
Summary information can be seen via the lscpu command.
# lscpuArchitecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 2 Core(s) per socket: 4 CPU socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 44 Stepping: 2 CPU MHz: 1600.000 BogoMIPS: 7197.24 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 12288K NUMA node0 CPU(s): 0,2,4,6,8,10,12,14 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15
Please see the man page for lscpu for more information.
Taskset on Linux
Taskset on Linux
Taskset is used to set or retrieve the CPU affinity of a running process, given a PID or a command. This could be used to increase the performance in certain instances.
taskset –p –c <cpuset> <PID>Example:
taskset –p –c 0-7 20320
Where –p option indicates the last argument is a PID, the –c option indicates the second-to-last argument is a list of CPU IDs (rather than, the default, a bitmap representing which CPUs). The effect of this command would be to make process 20320 bias to the first 8 cores.
Disabling a CPU on Linux
Disabling a CPU on Linux
To temporarily disable a CPU on Linux, use the following command:
# echo 0 >> /sys/devices/system/cpu/cpu[#]/onlineWhere [#] is the processor ID of the core to disable. It appears that disabling CPU 0 is not permitted. To enable, echo the value 1. It does not appear to persist between reboots.
System Statistics
System Statistics
You use the Statistics page in Web GUI to display summary statistics for authentication, accounting, and proxy forwarding transactions. You can measure the current, average, and peak TPS values of authentication and accounting transactions.
Authentication statistics summarize the number of authentication acceptances and rejections, with summary totals for each type of rejection or retry (see Figure 2).
Accounting statistics provide information such as the number of transaction starts and stops and the reasons for rejecting attempted transactions. The start and stop numbers rarely match, because many transactions can be in progress at any given time (see Figure 4).
