AIX > Administrator > Performance

CPU Threading Efficiency: Tactical AIX Monitoring of the Runqueue Value

AIX Runqueue

This is the third installment in an ongoing series on improving AIX performance by emphasizing CPU threading efficiency. Read part 1 and part 2.

In my previous article, "Recognizing the Efficiency Benefits of CPU Threading," I offered an outline of instructions for tactical AIX monitoring that begins with:

  1. Understand the runqueue thread count relative to the total count of logical CPUs.
  2. Monitor the value of AIX:vmstat -IWwt 1:kthr:r until familiar.

With this article, I'll elaborate on point 1b, specifically AIX:vmstat –IWwt 1, because it's my favorite AIX monitoring syntax. This is an extremely lightweight command for casual 24-7 monitoring of your important LPARs―the keyword being "lightweight." Of course, the better known AIX:nmon does the same type of monitoring, but it's far more CPU-intensive.

The command syntax of AIX:vmstat -IWwt 1 offers six groups of statistics per second, namely:

  • The count of a variety of kernel threads (kthr:r/:b/:p/:w)
  • Active virtual memory and free memory in 4k pages (memory:avm/:fre)
  • Default JFS/JFS2 reads/writes and paging space ins/outs in 4k pages (page:fi/:fo/:pi/:po)
  • The count of device interrupts, system calls and context switches (faults:in/:sy:/:cs)
  • CPU usage percentages, processor consumed and entitlement used (cpu:us/:sy/:id/:wa/:pc/:ec)
  • The time of each interval (time:hr/:mi/:se).

One great advantage with this syntax is that it provides an up-scrolling history of past seconds as the next second appears with each new line. While most other AIX interval commands (including mpstat, iostat and lvmstat) display in this fashion, this capability contrasts sharply with nmon output, which doesn't show any interval history. The up-scrolling history is perfect for following the dynamic rise and fall of the numbers of an on-going workload.

For the remainder of this article, I'll focus on a single statistical value: AIX:vmstat –IWwt 1:kthr:r.

Tactical Detail

Follow along with your own LPAR to observe AIX:vmstat -IWwt 1 output, or, if it's more convenient, see figure 1.

The AIX:vmstat -IWwt 1:kthr:r value is the average count of running plus runnable threads over each one-second interval, (aka the runqueue). Threads are groups of instructions that execute on the logical CPUs of a virtual CPU while this virtual CPU is running on a CPUcore.

Now, you might be thinking: “But I thought processes run on CPUs? Then what are logical CPUs and virtual CPUs running on a CPUcore?” At this point, it's critical to consider the structure specific to POWER CPU virtualization: process:threads are mapped to CPUcore:vCPU:logicalCPUs when executing. I've taught customer classes weekly for nearly five years, and most attendees don't understand the significance of this. Nor do they see how these threads comprise the runqueue that is the AIX:vmstat -IWwt 1:kthr:r value itself.

Historically, a process is a stream of instructions manipulating data in process memory (aka private RAM). This process’s single stream of instructions is a discrete task working with data―i.e., one process:one task:one dataset or [1:1:1].

Next, envision a process with two discrete streams of instructions sharing access to the same data in process memory. In other words, imagine two tasks of the same process manipulating the same dataset in RAM―i.e., one process:two tasks:one dataset or [1:2:1]. This is more efficient than two processes with two copies of the same data in two different regions of process memory―i.e., 2*[1:1:1].

Now envision a process with eight discrete streams of instructions sharing access to the same data in process memory. In other words, imagine eight tasks of the same process manipulating the same dataset in RAM―i.e., one process:eight tasks:one dataset or [1:8:1]. Again, this is more efficient than eight processes with eight copies of the same data in eight different regions of process memory―i.e., 8*[1:1:1].

Given all of this, each discrete stream of instructions (or tasks) of the same process is a thread. Processes are comprised of one or more threads sharing the same data in process memory―i.e., multithreaded processes or [1:X:1].

AIX displays this structure of processes and its comprising threads with AIX:ps -elmo THREAD, as seen in figure 2.

Notice the PID (Process ID) column. PIDs are always even-numbered in AIX. Also take note of the TID (Thread ID) column. TIDs are always odd-numbered in AIX. A TID number uniquely identifies each thread of every process listed by PID. (Exception: /etc/init always has a PID=1 and a PPID=0.)

Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.



2017 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.

Achieving a Resilient Data Center

Implement these techniques to improve data-center resiliency.


AIO: The Fast Path to Great Performance

AIX Enhancements -- Workload Partitioning

The most exciting POWER6 enhancement, live partition mobility, allows one to migrate a running LPAR to another physical box and is designed to move running partitions from one POWER6 processor-based server to another without any application downtime whatsoever.

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store
AIX News Sign Up Today! Past News Letters