MAINFRAME > Hot Topics

Metering and Capping on z/OS for Apache Spark


Hot Topics content is written by IBM technical experts - the people that design and write the code. The articles provide insight into products and functionality from a developer's perspective.

One of the strengths of z/OS is the ability to run multiple workloads simultaneously, within one z/OS image or across multiple images, while maintaining high system utilization. z/OS workloads have different, often competing, performance completion and resource requirements. The Workload Manager (WLM) component of z/OS is designed to balance these requirements by making the best use of system resources while maintaining the optimal throughput and system responsiveness.

Unlike many resource managers on distributed platforms, WLM allows the user to define performance goals and importance to each goal in business terms, rather than in hardware terms. WLM then decides how much resource, such as CPU and memory, to give to a workload to meet a goal. WLM constantly monitors the system and adapts processing to meet defined performance goals. However, WLM’s primary focus is to ensure that performance goals are met, and it doesn’t always prevent a workload from overachieving and consuming a large amount of shared system resources.

For many new workloads on z/OS, having a means of controlling or capping resource consumption might be useful. IBM z/OS Platform for Apache Spark (z/OS Spark), for example, is known for its aggressive in-memory data caching to achieve performance gains. IBM Cloud Provisioning and Management for z/OS is another example that requires hard limits to support its multi-tenancy. The new metering and capping support for z/OS allows the system capacity planner more granular control over CPU and memory consumption for various workloads and enables the system to host new workloads more easily. For instance, by limiting the z/OS Spark service class, you can safely allow interactive data analysis to be performed on the same system as production transactions.

The new metering and capping support consists of two major features:

  • Honor Priority by Service Class
  • Memory Capping

Honor Priority by Service Class

Honor Priority by Service Class allows granular control over which work is eligible for z Systems Integrated Information Processor (zIIP) can be processed by the general-purpose central processors (GCPs). Before this enhancement, the IIPHONORPRIORITY option in the IEAOPTxx parmlib member controlled whether the GCPs could run zIIP-eligible work for all workloads on the system. This situation was not always ideal if you wanted to restrict certain workloads (such as z/OS Spark) to only zIIPs while allowing others (such as Db2) to obtain help from the GCPs.

As its name implies, the new feature adds an Honor Priority setting to the WLM service class definition. If you accept the default value for or specify the Honor Priority value to DEFAULT for a service class, then the IIPHONORPRIORITY setting is used to determine whether work within the service class can overflow to GCPs. If you specify the Honor Priority value to NO, then zIIP-eligible work within the service class is not allowed to overflow to GCPs, regardless of the IIPHONORPRIORITY setting. Since DEFAULT and NO are the only available options, you cannot enable work within a service class to overflow to GCPs if IIPHONORPRIORITY is set to NO. Note that this new feature applies to work eligible for zEnterprise Application Assist Processor as well.

Memory Capping

The other main feature, Memory Capping, puts an upper limit (in GBs) on real memory that can be consumed by address spaces that are associated with a WLM resource group. A new field, Memory Limit, is added to the WLM resource group definition. When a resource group is defined with a nonzero Memory Limit, the system creates a memory pool, which represents the accounting of physical frames used by address spaces that are associated with the resource group (these address spaces are also called pool members). The maximum size of a memory pool equals the Memory Limit value of the corresponding resource group. If a memory pool is approaching its limit, the z/OS system will take actions against its members, thus preventing negative impact to other workloads on the system.

Elpida Tzortzatos is a senior technical staff member and team leader for the z/OS Core Development Team.

Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.

comments powered by Disqus



2019 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store
Mainframe News Sign Up Today! Past News Letters