JES2 JEC: Use UNIX Pipes to Pass Data Between Concurrent Jobs
This is the fifth and final article on Job Execution Control (JEC). Read parts one, two, three, and four.
In z/OS 2.2, JES2 introduced new functions to facilitate the scheduling of dependent batch jobs. These functions comprise JEC and can be utilized by making use of the new JOBGROUP and related Job Control Language (JCL) statements.
The primary goal of JEC is to provide simple controls to break down multistep batch jobs into constituent pieces. Provided that these constituent pieces can be run in parallel, improved throughput can be achieved by exploiting the concurrency functions provided by JEC.
This article expands on the discussion of concurrent dependencies provided in article three by showing how UNIX named pipes can be used to pass data between simultaneously executing jobs. This article is divided into the following sections:
- Introduction to UNIX Named Pipes
- Pipelining in a Multistep Batch Job
- Improved Pipelining With Concurrent Jobs
- Putting It All Together
Introduction to UNIX Named Pipes
Pipelines are built by chaining together processes that interact through special communication channels called pipes. These pipelines are used to accomplish complex tasks by breaking them down into distinct steps, each of which can be performed by a single utility.
A named pipe, also known as a FIFO special pipe, is a type of pipe that exists independently of any particular process. One or more processes can connect to a named pipe, write to it, and read from it at will. z/OS UNIX provides native support for named pipes in the z/OS File System (zFS).
The following JCL shows how a named pipe can be defined in a batch job:
//PIPE1 DD PATH='/u/&SYSUID/tmp/pipe1',DSNTYPE=PIPE,
Notice that this data definition (DD) is very similar to a DD used to define a “regular” UNIX file in a batch job. Here is a description of each of the parameters used in the above JCL:
◦ Defines the name of this pipe (pipe1) and the directory that it
resides in (/u/&SYSUID/tmp/).
◦ Indicates that this z/OS UNIX file is to be treated as a pipe.
◦ Defines fixed length records of 80 bytes that are to be read
in 3200 byte blocks (40 records at a time)
◦ OCREAT specifies that if the pipe does not exist,
it should be created. If the /u/&SYSUID/tmp/ directory does not exist,
the pipe will not be created and the job will be failed.
◦ OWRONLY requests that the system uses the pipe for writing only.
Because pipes ingest and dispense data in a first-in, first-out manner, they are well-suited to the task of passing data between two concurrently running batch jobs.
Pipelining in a Multistep Batch Job
In order for a pipe to be useful, there must be at least two processes simultaneously writing to and reading from the pipe. Because steps in a multistep batch job execute sequentially rather than concurrently, pipes cannot be used to pass data between steps in a multistep batch job.
The following example presents a crude form of pipelining in a multistep job by using temporary UNIX files to pass data from step to step. Experienced JCL users may notice that the following multistep job could actually be performed in fewer than three steps. For the sake of the discussion in this article, assume that it is necessary to run the job in the three steps that are described.
STEP1 uses the IEBGENER utility to copy data from JES2.DATA(ARTICLE2) to a temporary file on the UNIX file system:
//NOTPIPED JOB MSGLEVEL=(1,1)
//STEP1 EXEC PGM=IEBGENER
//SYSUT1 DD DSN=JES2.DATA(ARTICLE2),
//SYSUT2 DD PATH='/u/&SYSUID/tmp/temp1',
//SYSIN DD DUMMY
//SYSPRINT DD DUMMY
The next step reads records from the temporary dataset created in STEP1, runs the SORT facility to convert all records to uppercase, and then writes the output to a second temporary file:
//STEP2 EXEC PGM=SORT
//SORTIN DD PATH='/u/&SYSUID/tmp/temp1',
//SORTOUT DD PATH='/u/&SYSUID/tmp/temp2',
//SYSIN DD *
//SYSOUT DD DUMMY
Finally, STEP3 uses the SORT facility again to filter out all records that don’t contain the string PRESIDENT from the temporary file created in STEP2. All records containing PRESIDENT are written as SYSOUT:
//STEP3 EXEC PGM=SORT
//SORTIN DD PATH='/u/&SYSUID/tmp/temp2',
//SORTOUT DD SYSOUT=Z
//SYSIN DD *
//SYSOUT DD DUMMY
Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.
comments powered by