MAINFRAME > Tips & Techniques > Systems Management

JES2 JEC: Use UNIX Pipes to Pass Data Between Concurrent Jobs

UNIX pipes
 

This is the fifth and final article on Job Execution Control (JEC). Read parts one, two, three, and four.

In z/OS 2.2, JES2 introduced new functions to facilitate the scheduling of dependent batch jobs. These functions comprise JEC and can be utilized by making use of the new JOBGROUP and related Job Control Language (JCL) statements.

The primary goal of JEC is to provide simple controls to break down multistep batch jobs into constituent pieces. Provided that these constituent pieces can be run in parallel, improved throughput can be achieved by exploiting the concurrency functions provided by JEC.

This article expands on the discussion of concurrent dependencies provided in article three by showing how UNIX named pipes can be used to pass data between simultaneously executing jobs. This article is divided into the following sections:

  • Introduction to UNIX Named Pipes
  • Pipelining in a Multistep Batch Job
  • Improved Pipelining With Concurrent Jobs
  • Putting It All Together

Introduction to UNIX Named Pipes

Pipelines are built by chaining together processes that interact through special communication channels called pipes. These pipelines are used to accomplish complex tasks by breaking them down into distinct steps, each of which can be performed by a single utility.

A named pipe, also known as a FIFO special pipe, is a type of pipe that exists independently of any particular process. One or more processes can connect to a named pipe, write to it, and read from it at will. z/OS UNIX provides native support for named pipes in the z/OS File System (zFS).

The following JCL shows how a named pipe can be defined in a batch job:

//PIPE1  DD PATH='/u/&SYSUID/tmp/pipe1',DSNTYPE=PIPE,
//      LRECL=80,BLKSIZE=3200,RECFM=FB,
//      PATHOPTS=(OCREAT,OWRONLY),
//      PATHDISP=KEEP

Notice that this data definition (DD) is very similar to a DD used to define a “regular” UNIX file in a batch job. Here is a description of each of the parameters used in the above JCL:

•	PATH='/u/&SYSUID/tmp/pipe1'
◦	Defines the name of this pipe (pipe1) and the directory that it 
resides in (/u/&SYSUID/tmp/).
•	DSNTYPE=PIPE
◦	Indicates that this z/OS UNIX file is to be treated as a pipe.
•	LRECL=80,BLKSIZE=3200,RECFM=FB
◦	Defines fixed length records of 80 bytes that are to be read 
in 3200 byte blocks (40 records at a time)
•	PATHOPTS=(OCREAT,OWRONLY)
◦	OCREAT specifies that if the pipe does not exist, 
it should be created. If the /u/&SYSUID/tmp/ directory does not exist, 
the pipe will not be created and the job will be failed.
◦	OWRONLY requests that the system uses the pipe for writing only.

Because pipes ingest and dispense data in a first-in, first-out manner, they are well-suited to the task of passing data between two concurrently running batch jobs.

Pipelining in a Multistep Batch Job

In order for a pipe to be useful, there must be at least two processes simultaneously writing to and reading from the pipe. Because steps in a multistep batch job execute sequentially rather than concurrently, pipes cannot be used to pass data between steps in a multistep batch job.

The following example presents a crude form of pipelining in a multistep job by using temporary UNIX files to pass data from step to step. Experienced JCL users may notice that the following multistep job could actually be performed in fewer than three steps. For the sake of the discussion in this article, assume that it is necessary to run the job in the three steps that are described.

STEP1 uses the IEBGENER utility to copy data from JES2.DATA(ARTICLE2) to a temporary file on the UNIX file system:

//NOTPIPED JOB MSGLEVEL=(1,1)
//STEP1  EXEC PGM=IEBGENER 
//SYSUT1  DD DSN=JES2.DATA(ARTICLE2),
//      DISP=SHR, 
//      VOL=SER=J2SHR2,
//      UNIT=3390 
//SYSUT2  DD PATH='/u/&SYSUID/tmp/temp1', 
//      LRECL=80,BLKSIZE=3200,RECFM=FB,
//      PATHOPTS=(OCREAT,OWRONLY), 
//      PATHDISP=(KEEP,DELETE) 
//SYSIN  DD DUMMY 
//SYSPRINT DD DUMMY

The next step reads records from the temporary dataset created in STEP1, runs the SORT facility to convert all records to uppercase, and then writes the output to a second temporary file:

//STEP2  EXEC PGM=SORT 
//SORTIN  DD PATH='/u/&SYSUID/tmp/temp1', 
//      LRECL=80,BLKSIZE=3200,RECFM=FB,
//      PATHOPTS=ORDONLY, 
//      PATHDISP=DELETE 
//SORTOUT DD PATH='/u/&SYSUID/tmp/temp2', 
//      LRECL=80,BLKSIZE=3200,RECFM=FB,
//      PATHOPTS=(OCREAT,OWRONLY), 
//      PATHDISP=KEEP
//SYSIN  DD * 
 SORT FIELDS=COPY 
 OUTREC FIELDS=(1,80,TRAN=LTOU) 
/* 
//SYSOUT  DD DUMMY 

Finally, STEP3 uses the SORT facility again to filter out all records that don’t contain the string PRESIDENT from the temporary file created in STEP2. All records containing PRESIDENT are written as SYSOUT:

//STEP3  EXEC PGM=SORT 
//SORTIN  DD PATH='/u/&SYSUID/tmp/temp2', 
//      LRECL=80,BLKSIZE=3200,RECFM=FB,
//      PATHOPTS=ORDONLY, 
//      PATHDISP=DELETE
//SORTOUT DD SYSOUT=Z 
//SYSIN  DD * 
 SORT FIELDS=COPY 
 INCLUDE COND=(1,80,SS,EQ,C'PRESIDENT') 
/* 
//SYSOUT  DD DUMMY



Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.


comments powered by Disqus

Advertisement

Advertisement

2017 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.

Optimal Service Delivery

Overcome eight key challenges and reduce costs

MAINFRAME > TIPS & TECHNIQUES > SYSTEMS MANAGEMENT

An Accurate Benchmark Is Important

Advice for the Lazy Administrator

Steps you can take to avoid late nights and system frights.

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store
Mainframe News Sign Up Today! Past News Letters