Skip to main content

DATA-INTO Plus DATA-GEN for Web Services

Combining these two RPG operations with the YAJL library simplifies using web services from RPG.

illustration of gears on a light blue background.

It seems that barely a month can go by without us receiving a request from a client concerning web services. They’ve been tasked with using or providing web services, and are asking for advice on how to generate and consume JSON. Some years ago, such requests would have involved XML, but these days JSON with its leaner profile has almost completely taken over. In fact, our most recent request involved a SOAP web service which used JSON for both requests and responses. This was a little unusual in that SOAP is an XML based protocol!

Generating and Consuming JSON with RPG

Of course, one could always do it the hard way, i.e. stringing together requests and parsing responses "by hand". However, RPG gives us three main alternatives to this: 

  •  Embed SQL and use its JSON capabilities
  •  Use the JSON creation and parsing APIs from Scott Klement's YAJL library
  •  Use RPG's native facilities DATA-GEN and DATA-INTO in combination with facilities from the YAJL library

The third option will be the focus of this article since we find it generally to be the most flexible solution. As some readers will recall, we’ve written about the use of YAJL before. However, in light of the questions we continue to get we decided it would be appropriate to consolidate some of that information into one more cohesive "How To" article.

The Task at Hand

For the purposes of this article we are going to suppose that we are building a quotation for a customer and that the prices to be used in the quote are retrieved by using a web service. Each web service request will contain the data for a variable number of items from 1 to 50.

The basic sequence of operations therefore is this:

  1. Gather the data for the customer quote
  2.  Build the JSON request
  3. Send the JSON request to the web service and receive the response
  4.  Parse the response JSON to extract the pricing information
  5. Complete the processing of the quotation.

Regardless of how the request was received and how the quote is to be delivered, the basic pricing operation laid out here will always be the same.

The JSON Formats

If you are not familiar with JSON we suggest that you might want to read our article "An RPGer's First Steps With JSON".

The required JSON for the web service request looks like this:

{

  "QuotationId": "A12345", 

  "CustomerNumber":123456, 

  "CustomerClass":"D2", 

  "LineItems": [ {

                  "ItemNumber":1, 

                  "Quantity":2

                  },

          ... repeats of the item as needed ...

                ] 

}                    


The JSON returned by the web service is basically the same except that each LineItems entry will have had three additional fields added to it containing the pricing information. So, that portion of the response looks like this:

     {

      "ItemNumber":1,

     "Quantity":2, 

     "UnitPrice": 123.45, 

     "UnitDiscount": 12.345, 

     "ExtendedPrice":12345.67

     }  

Formatting the Request

We’re going to use RPG's DATA-GEN for this, but before we concern ourselves with the mechanics of the opcode, we need to design a Data Structure (DS) that will be used as the data source for building the JSON.

This DS must match the JSON both in terms of the field names used and in the relationships between those fields. Why? Because that is how DATA-GEN (and for that matter XML-INTO and DATA-INTO) work! Here's the DS that we designed:

(A)  dcl-ds Request qualified  Inz;

(B)    QuotationId     char(6);

       CustomerNumber  char(6);

       CustomerClass   char(2);

(C)    num_LineItems   int(5);  // Count controls number of elements    

(D)     dcl-ds  LineItems Dim(50);  

          ItemNumber        char(10);

          Quantity          int(5);

        end-ds  LineItems;

     end-ds  Request;                                                  

(A) The name of the DS (Request) is arbitrary as JSON doesn’t name its root element (unlike XML). We do however need to specify it as QUALIFIED because we will need to use the same field names later in the DS that we design for handling the response data.

(B) This is the exact same name used to identify the element in the JSON. It mustmatch exactly. We'll talk a little later about how to handle situations where the JSON name contains characters that are illegal in an RPG field name. The sequence of these fields is not important, but their hierarchical position (In this case at the first level within the DS) is.

(C) This field is optional, but if it were not present then DATA-GEN would generate a JSON array element for each of the 50 elements present in the LineItems DS array—including all the empty ones! Most web services wouldn’tt be happy about that. By setting a value in this field we tell DATA-GEN to only generate the required number of elements. The association between the DS array and this count field is via the name. i.e. the name is the same except that the count field's name has a prefix of "num_". We’ll show you how the association is made when we look at the DATA-GEN operation.

(D) The JSON LineItems element consists of an array (indicated by the [ ] characters) so it is coded as a nested DS array consisting of the ItemNumber and Quantity fields.

Cases like this are why we really appreciate the ability to directly nest data structures—it makes it so much easier to see that the "shape" of our DS does indeed match the shape of the JSON.

We don't have time to go into all of the details here, we will cover that in a later article, but if you have difficulties in working out what your DS should look like there is help available. Scott Klement has recently added to the YAJL library the program YAJLGENand its associated command. This program will read a sample JSON document and generate the associated DS. That DS can then be used by either DATA-GEN or DATA-INTO. It won't be perfect because it has to guess at many things, but it can be a big help in getting started.  

Once the DS is in place then we can code the DATA-GEN operation needed to produce the JSON text.


(E)    dcl-s JSONRequestText varchar(4000)  inz;

(F)    dcl-s  itemCount int(5);  // Number of items in the quotation


(G)    // Assorted logic to load the Request DS with the required items


      Request.num_LineItems = itemCount; // Set count into request DS


(H)    Data-Gen Request

                 %Data( JSONRequestText : 'countprefix=num_' )

                 %Gen('YAJL/YAJLDTAGEN');


(E) This is the variable that will hold the generated JSON text. Its contents will subsequently be passed to the web service. In our simple example 4,000 bytes is long enough—just adjust the size to fit your needs.

(F) The preliminary processing of the quotation will increment this counter. We could of course have built the count in Request.num_LineItemsdirectly but since not showing that logic this seems a good way to make it clear that the count must be set before calling DATA-GEN.

(G) This is the point at which our RPG logic would set the contents of the quotation request DS. 

(H) Data-Gen starts by identifying the field to receive the generated data (Request). The %Data BIF identifies the DS that will be the source for the generated data and also specifies that the characters "num_" are to be used as the prefix to identify count fields. Last but not least we specify the program or procedure to perform the generation process. In our case it is the YAJLDTAGEN program supplied as part of Scott's YAJL library. 

Note: This particular program is a relatively recent addition to the YAJL library, so if it does not appear to be on your system it just means that you need to install an updated version.

At this point in time we have the request JSON built and simply need to send it off to the web service by whatever mechanism we have chosen to use. The web service in turn will return the JSON response and before we can process the results we will need to convert it to a more RPG-friendly format. That will be a job for DATA-INTO.

Processing the Result

Just as we did with DATA-GEN, the first thing we need to do is to design a DS to hold the decoded JSON. In this case it was basically a case of copying the Request DS, renaming it to Response and adding the additional three pricing fields. The result looks like this:

     dcl-ds Response qualified;

       QuotationId     char(6);

       CustomerNumber  char(6);

(I)    num_LineItems        int(5);  // Count of elements found

        dcl-ds LineItems  Dim(50);

         ItemNumber     char(15);

         Quantity       int(5);

         UnitPrice      packed(9:3);

         UnitDiscount   packed(9:3);

         ExtendedPrice  packed(15:2);

        end-ds  LineItems;

     end-ds  Response; 


Notice that we have included a count field in the DS as we did before. There is however no corresponding value in the JSON.  DATA-INTO fills in this value based on the number of LineItems elements that it finds in the JSON.

Having created the DS the only thing left to do is to code the DATA-INTO operation to fill the DS from the web service response. Here's that code:

   Data-Into Response

             %Data( jsonResponse :

                   'countprefix=num_ case=any' )

            %Parser('YAJL/YAJLINTO');

The first operand is the target DS. 

Next comes the %Data BIF which provides the name of the field containing the web services response data (jsonResponse) and the processing options to be applied. You've already met the count prefix option, but case=any may require a little explanation for those of you unfamiliar with XML-INTO. Basically its purpose is to deal with the fact that RPG field names are not case-sensitive. Under the covers the compiler treats the field names abc, Abc, and ABC as all being the name ABC. This causes problems when DATA-INTO attempts to match the name of an item identified by the parser with the corresponding field names. In our example, the parser will return names such as QuotationIdand CustomerNumber but RPG knows the equivalent fields in the DS as QUOTATIONID and CUSTOMERNUMBER and as a result the names would not match. case=any deals with this in that it tells DATA-INTO to convert any names it receives from the parser to upper case before attempting to make the comparison. In other words, you are pretty much always going to need to specify this option. 

The final operand is the %Parser BIF, which identifies the program or subprocedure that will parse the document. While IBM supplies an example JSON parser, it is intended for educational purposes and is not "production ready.” We’ve used the YAJL library's YAJLINTO program which works very well and is very fast.

And that's all we need to do. All the data from the JSON response has been unpacked into the Response DS and is ready for us to process. Here's a short extract from the example program that we used that shows how the resulting data can be accessed. If you would like to play with this code and study how it works you can download it here.

   Dsply ( 'Processed ' + %char(Response.num_LineItems) +

           ' items - details:' );

   // Process all Line Items returned ...

   For i = 1 to Response.num_LineItems;

      Dsply ( '  Item: ' + Response.LineItems(i).ItemNumber );

      Dsply ( '  Price: ' + %Char(Response.LineItems(i).UnitPrice) +

              '  Discount: ' +

             %Char(Response.LineItems(i).UnitDiscount) );

   EndFor;

Going Beyond the Basics

Once we began writing this article it quickly became apparent that there was a lot more that could be said. Hopefully this piece will serve as a good introduction to the basics of using these two opcodes in combination to handle many of your JSON processing needs. But there are always exceptional situations, more than we can cover in this article. However, we’ll quickly touch on one question that has come up a number of times in our discussions with customers. "What if I only need to process part of a JSON document?". 

There are of course many ways to deal with this and the best one to choose depends on the format of the JSON and your specific requirements. To give you one example—suppose that to speed processing we wanted to ignore the first part of our web service response. A reasonable thing to do considering that the data should be exactly the same as the data that we sent in the first place.


Here's the data definition and the revised DATA-INTO that we could use to achieve this.

(J)   dcl-ds  Psds PSDS;

         elementCount  int(20) Pos(372);

      end-Ds;


      dcl-ds LineItems  Qualified  Dim(50);

        ItemNumber     char(15);

        Quantity       int(5);

        UnitPrice      packed(9:3);

         UnitDiscount   packed(9:3);

        ExtendedPrice  packed(15:2);

      end-ds  LineItems;


      Data-Into  LineItems

(K)             %Data( jsonResponse :

                       'path=jsonData/LineItems case=any' )

(L)              %Parser('YAJL/YAJLINTO':

                          '{ "document_name" : "jsonData" }' );


(J) Because the target of DATA-INTO is an array, RPG automatically keeps a count of the number of elements found in position 372 of the PSDS.

(K) Here we have used the path= option to tell DATA-INTO to drill down to the start of the LineItems entry before beginning processing. But there's a problem with that - JSON documents do not have a name associated with the outer element like XML does. We deal with that by making up our own root name - jsonData in our case but then ...

(L) We need to tell the YAJLINTO parser what that name should be. This is specified via the second parameter to the %Parser BIF. The options themselves are in the form of JSON text. In our case we are only using the document_name option and so the JSON is very simple.

Summary

That's all there is to it. Hopefully this gives you a good idea of how this process hangs together. But as we said, there are a lot of other situations that you may encounter and we will address those own our next column. In the meantime if there is anything in this area that you need more immediate help with then please contact us at contact@Partner400.com.

IBM Systems Webinar Icon

View upcoming and on-demand (IBM Z, IBM i, AIX, Power Systems) webinars.
Register now →