AIX > Administrator > Performance

Slicing Time: Curing the Context Switch in AIX

AIX Context Switch

We take the number of LPs in our system and look at the total number of context switches as reported by vmstat. Let’s say we have a POWER7 system with 12 LPs. Let’s further assume we’re using a 2-second sampling rate for our vmstats to get a quick idea of the rate of context switching. In this scenario, we wouldn’t be surprised to see a total context switching rate somewhere in the low hundreds when the system is quiet. Even on a system at apparent rest, many internal processes are active, making our cs column non-zero.

Over time, you should watch vmstats and take careful note of application, database or other activity that would begin to drive the cs counters upward. Then implement vmstats over an extended period—weeks or even months—using a consistent sampling rate to baseline context switching activity. Say we take vmstats for a month and find that we average 1,000 total context switches per sampling rate while the system is under a heavy workload. Now we start our calculations:

Take the total number of context switches in a sample (1,000) and divide that number by the total number of LPs in our system (12). That gives us an average of about 83 context switches, per LP, per sample.

(One more important backtrack: It's not enough to have a general idea as to our workload and the number of logical CPUs in our system. We also need to know how that workload uses our CPUs. Remember that most single-threaded workloads—like databases—will use only the first, or primary, LP in any CPU, and when that LP is saturated, the workload will fall over to the next primary LP on the next CPU. The generic term for this type of CPU usage is “raw throughput.” Multi-threaded applications tend to use more LPs on each CPU before they cascade to the next CPU; this type of CPU usage is called “scaled throughput.” You can see that these workload characteristics are very important in determining your context switch/LP average. More on this in future articles.)

If our system’s performance is good with this number of context switches, we simply continue our statistics. But let’s say we see a sustained increase in context switching: our cs counters have increased from 1,000 to 10,000 every 2 seconds, and users are telling us that things are slowing down. We don’t see anything else in our performance data other than high cs rates, so what's happening to our system’s performance and what can we do about it?

The Timeslice Parameter

When a system’s cs rate is too high, working threads are bumped off CPUs before they’ve had time to complete their tasks, and the system’s logical CPUs rapidly turn their attention from one thread to another. Simply put, they change their context... much too quickly. We need to get back to a state where our working threads stay on an LP long enough to do everything we expect of them. Fortunately, a CPU tunable in AIX lets us adjust this time; it's called “timeslice.”

The timeslice parameter tells us how many clock ticks of a CPU can happen in any 10-millisecond period. If we raise the timeslice tunable’s value, we increase the number of CPU clock ticks that occur in 10 milliseconds. So you can see that if our working threads initially have x-number of clock ticks to run on a CPU within 10 milliseconds, and we increase the number of clock ticks to 10x, 100x, 1,000x, etc., our threads will have more time to run before they are taken off a logical CPU. We use the “schedo” command to determine how the timeslice value is set in our system. Schedo is short for “scheduling options” and is the suite of tunables that governs CPU performance in an AIX system. To view the timeslice value, use the “schedo –FL timeslice” command. You’ll see output like this:

lpar # schedo -FL timeslice
NAME                      CUR    DEF    BOOT   MIN    MAX        UNIT           TYPE
timeslice                     1         1          1         0         2G-1   clock ticks       D

We start raising the timeslice parameter conservatively, say, in powers of 10. So we first increase our timeslice from the default of 1 to a value of 10 using the following command (adjustments to the timeslice parameter are dynamic; they take effect immediately, without a system reboot):

   schedo –o timeslice=10

Now we return to our vmstats. Has our rate of total context switching gone down? If we're not satisfied with our cs rate, we continue adjusting our timeslice tunable upward to 100, 1,000, 10,000, etc., until we see our cs rate fall off. As the timeslice gets bigger, our cs rate should get smaller. When we're satisfied that our cs rate no longer hampers our system’s performance, we stop tuning the timeslice.

Use Timeslice Wisely

As I said, there are many reasons a system’s cs rate may be too high. With many reasons come multiple cures, but we have to start someplace. Short of a code change, adjusting the timeslice is the quickest way to rid yourself of too-frequent context switches. Of course you need a complete picture of your system’s performance if you intend to adjust any CPU tunable, timeslice included; utilities you can run safely over extended periods (like the “stats” programs: vmstat, iostat and netstat) as well as snapshot diagnostic tools like PerfPMR will give you this picture.

A final caveat: Use caution if you're adjusting timeslice in a production environment, and by no means should you raise timeslice if your system is already performing well. Timeslice isn't a performance-enhancer. Its only proper use is for fixing very specific problems, like excessive context switching. It shouldn't be adjusted for any other purpose.

Mark J. Ray has been working with AIX for 23 years, 18 of which have been spent in performance. His mission is to make the diagnosis and remediation of the most difficult and complex performance issues easy to understand and implement. Mark can be reached at mjray@optonline.net



Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.



Advertisement

Advertisement

2017 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.

Achieving a Resilient Data Center

Implement these techniques to improve data-center resiliency.

AIX > ADMINISTRATOR > PERFORMANCE

AIO: The Fast Path to Great Performance

AIX Enhancements -- Workload Partitioning

The most exciting POWER6 enhancement, live partition mobility, allows one to migrate a running LPAR to another physical box and is designed to move running partitions from one POWER6 processor-based server to another without any application downtime whatsoever.

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store
AIX News Sign Up Today! Past News Letters