Skip to main content

Diving Deeper Into Web Services with DATA-INTO and DATA-GEN

Technical editors Jon Paris and Susan Gantner continue their series on DATA-INTO.

image of block cube on a blue background.

In part 1 of this series we discussed the basic usage of RPG's DATA-INTO and DATA-GEN opcodes as a means of generating a JSON web service request and processing the response. In cases where your code is actually supplying the web service you would of course simply reverse these functions and use DATA-INTO to process the incoming request and DATA-GEN to prepare the response.

This time we want to probe a little deeper into some of the additional processing options available for these opcodes and discuss when you might need to use them. Note that some of these options apply only to the YAJLINTO parser and YAJLDTAGEN generator - and so will be specified as part of the %PARSER and %GEN BIFs; others apply to either the DATA-INTO or DATA-GEN opcodes themselves and will be specified as part of the %DATA BIF. 

YAJLINTO Options for Special Data Types

JSON supports a boolean data type (i.e. an indicator in RPG terms) - the values for such elements being 'true' and 'false'. By default, YAJLINTO maps these to to '1' (true) and '0' (false). These defaults work well if you want to map the JSON element to an indicator. However, in some cases, you may be mapping to an existing character flag in your application. Perhaps a 'Y' is used to indicate the true condition and 'N' for the false. In such cases you will need to override the defaults. 

This is achieved by specifying an option string to the %PARSER BIF as shown here: 


Data-Into  customerData2

           %Data( jsonData :

                  'case=any countprefix=num_ ' )

           %Parser('YAJL/YAJLINTO' "

              '{ "value_true":"Y", "value_false":"N" }');

You are not limited to the length of the substitution data and could, for example, have used something like this if the application called for it:


"value_true":"Yes", "value_false":"No"

Another aspect of JSON that requires special handling are null values. A null value can be used for any type of JSON data and this can require some thought when determining how to process such documents in RPG. 

In traditional RPG applications, blanks are commonly used to indicate the equivalent of a null value in a character field. But in numeric fields all bets are off and everything from zero to high-values to negative numbers may be used. By default, YAJLINTO will return the character string '*NULL' when it encounters a null value in the JSON. The problem of course is that this causes an error if you attempt to assign it to a numeric field. So how to deal with this?

If you know that the only elements in the JSON that could have a null value are numeric, then you could use the value_null %PARSER option as shown below to set the target field to a value of -1 - or whatever other value you might want. Remember however that your value must be expressed as a string. So to use a value of -1 you would code the option like this:


     %Parser('YAJL/YAJLINTO' :

             '{ "value_null" : "-1" }' );

However, if you need different values for different fields, or if the JSON can contain null values for a variety of data types, then you are probably better off using the default setting and mapping all such values into character fields with a minimum length of 5 characters. You would then test each field for '*NULL' and set an appropriate value based on the field's data type and usage in the subsequent RPG logic

Handling "Invalid" RPG Field Names

Now that we know how to handle data that is not directly supported in RPG, how do we handle names that are valid element names in JSON but not valid field names in RPG? We'll look at two ways to address this, depending on what kind of issue we have with the names. The first uses an option in DATA-INTO; the second is an option implemented by YAJLINTO. 

When processing JSON it is important to remember that it is not uncommon for JSON element names to include a space—something that you can never do in RPG. There is DATA-INTO support to solve this in the same way that XML-INTO does. Those of you familiar with XML-INTO will recognize it.

It uses the case=convert option for the %DATA BIF to convert most non-RPG names to valid RPG names. See our earlier article "New and Improved XML-INTO" for more details, but a simple example of this technique follows.

When you use this option, any element names that are not valid as RPG field names will be modified to make them compatible. In many cases the invalid character will be replaced by an underscore.

The brief example below shows how to code the DS and the DATA-INTO option to handle this. Note the space in the JSON element name "customer id" and how this is mapped to the field customer_id in the DS via the use of the case=convert option.


dcl-s  jsonData  varChar(4000)

         inz('{ "customers": [ -

               { "customer id": 12345, -

                  "name": "Paris" }, -

         ... ] }');


dcl-ds  customerData  Inz  Qualified;

   num_customers  int(5);

   dcl-ds  customers  Dim(99);

      customer_id       Zoned(5);      

      name              Char(40);

   end-ds;

end-Ds;


Data-Into  customerData

           %Data( jsonData :

                  'case=convert countprefix=num_ ' )

           %Parser('YAJL/YAJLINTO');


Another potential naming issue involves JSON element names that begin with numbers. While case=convert on its own will convert them to valid names, it does so by simply stripping the leading digits off the name. This may not always be a good solution

For such cases, the YAJLINTO %PARSER function has a number_prefix option, which allows us to prefix any JSON element name that starts with a number with characters so that it becomes a valid RPG name. 

So if we specify to %PARSER the option { "number_prefix" : "Num_" } in combination with case=convert, then the JSON element name "30 day balance" would be associated with the RPG variable Num_30_day_balance.

There are other options that you may need from time to time but this covers the majority of the ones we have used so far.

DATA-GEN %DATA Options

Time now to take a look at some of the processing options used for DATA-GEN. Later we will explore options specific to YAJLDTAGEN.

Let's start with the countprefix option. When specified as an option to DATA-INTO, it identifies a field into which RPG will place a count of the number of elements of that name found in the document.

So as you might reasonably suppose, when we use this option with DATA-GEN it tells RPG how many of the named elements to generate. Using the DS customerData from the previous example as the DATA-GEN source, if there were a value of 2 in the field num_customers then we would generate 2 occurrences of the customers element. 

Whereas you can often get away without using countprefix with DATA-INTO, it is normally essential when using DATA-GEN. The reason is that if we were to omit the count, then DATA-GEN would generate 99 occurrences (i.e. the number of elements in the array.) This would not endear us to the people who are going to consume our JSON as they do not want to have to deal with ignoring 97 empty elements!

Another use for this feature (which may not be immediately obvious) is to use it to control the output of optional portions of the document. For example we might want to send JSON like this if a request succeeded:

{

"success": true,

"detail": {

    "customer_id": 123,

    "name": "Jones and Company" } 

And like this if an error were detected:

{

"success": false,

"errors": {

    "errorCode": "A123",

    "errorText": "My error message" }

}


By using counts at both the customers and errors levels we could achieve this. The following DS should give you an idea of how it might look and the field comments will hopefully make the usage obvious:


dcl-ds  customerData  Inz  Qualified;

   success  ind;  // Set on for success - off for error

   num_errors  int(5); // Set non-zero for errors

   dcl-ds errors;

      errorCode  char(5);

      errorText  varchar(60);

   end-ds;

   num_detail  int(5); // Set to zero on error customers

   dcl-ds  detail;

      customer_id       Zoned(5);


Another approach is to build the JSON in pieces. DATA-GEN gives you this ability by allowing you to combine the output of a series of DATA-GEN runs. If there is enough interest we will look at this feature in a future article.

Renaming Elements

Because of the naming differences between RPG and JSON that we mentioned earlier, you will find occasions when you need to use the renameprefix option. This allows us to provide a substitute name for an element. For example, if we specify renameprefix=json_ to the %DATA BIF then we could use a DS like this to rename the related fields:


dcl-ds  customerData  Inz  Qualified;

   json_shipTo  Varchar(15) Inz('Ship to address'); 

   dcl-ds  shipTo;

      json_street  Varchar(14) Inz('street address');

      street       Char(40);

      city         Char(30);

      state        Char(2);

      zip          Char(5);

   end-ds;

end-ds;


The resulting JSON will use "Ship to address" as the object name for the shipTo values and within that the content of the street field will have the name "street address". 

YAJLDTAGEN Parser Options

Since we were discussing the topic of renaming, our first option is number_prefix. This is also a valid option for YAJLDTAGEN'S %Parser BIF. In this case however, it performs the reverse task of removing the designated prefix from the element name being generated. So the option { "number_prefix" : "Num_" } in this case means to remove the character string Num_ from the start of any RPG variable name and use the resulting truncated name in the generated JSON.

Another %PARSER option is beautify. The default is false but if set to true the generated json will be formatted with indentation and line feeds to make it more readable. This is a useful option to use when debugging, but in the interests of keeping the payload as small as possible should not be used in production. 

Examples:  { "beautify" : true } or { "beautify" : false }

There are other options, and Scott Klement continues to add more as he enhances the tool. 

But since the topic of this series of articles has been using RPG and JSON in web services, it is appropriate to mention one last related option. This is "write to stdout.” If this is set to true then the generated JSON will be written to the standard out device. Why is this useful? Well if the RPG program running this code was called as a web service from the Apache server, then standard out is where you must write the response. So DATA-GEN coupled with YAJLDTAGEN can be used to provide a simple web service.

YAJLINTO also provides a similar option to process a JSON request from stdin (Standard input) which as you may have guessed is where the Apache server will place any input requests received. So the combination of DATA-INTO and DATA-GEN together with Scott's YAJL tools can be used to build a simple web service. We haven't got the space to provide an example here but Scott's presentation "Providing Web Services on IBM I," which you can find here.


Wrapping Up

We hope that you have found this short series useful. We have tried to cover all of the major features that our own explorations have revealed to us, but if there is anything you think we have left out or you would like to have covered please let us know via the comments section below.

IBM Systems Webinar Icon

View upcoming and on-demand (IBM Z, IBM i, AIX, Power Systems) webinars.
Register now →