AIX > Tips & Techniques > Application Development

End Users: A Programmer’s Best Friend and Worst Enemy

I’ve worked in the mainframe IT industry for more than 35 years, in a spectrum of roles from senior IBM Systems Engineer to founder, CEO and lead consultant of a consulting firm to senior programmer/ analyst. I’ve had the privilege of working with more than 300 customers ranging from the largest banks, insurance companies, manufacturers, educational institutions, etc., through small VSE (originally DOS) companies, plus two IBM development labs. Events recently have caused me to reflect on my career–which I’ve enjoyed greatly, good and bad–and in doing so, realize that writing a “how-not-to” series is a good way of sharing some of that experience to help others avoid those mistakes.

While the IT department usually gets credited with development of applications or enhancement of existing applications, it’s my experience that successful projects come from a dynamic chemistry and synergy among those who produce the code/data, those who use it and those who assure the application’s quality via testing. But even when IT does its part correctly, that doesn’t guarantee success; the problem doesn’t always lie with programming.

Things Aren’t Always Simple

An example of this was an accounts receivable (A/R) application where a late fee assessment was added to delinquency-processing code in a program that handled credit orders paid via installment plans. Initially and superficially considered a minor project, the complexity of existing code and late fee processing should have tripped red flags as coding got underway. Several business rules–mostly undocumented–really complicated programming. For example:

  1. If full payment comes in after the due date, leaving only late fee as due, the late fee must be “rolled into” the following payment, because a late fee by itself cannot make an account past due.
  2. If the balance due is less than a dollar, it is not considered past due and is rolled into the next payment.
  3. If the late fee is less than $1 and the only amount due, it is rolled into the next payment.
  4. In the final payment, all due amounts remain, nothing is rolled; late fees may be assessed.
  5. Merchandise returns prior to first due date are subtracted from the first installment. If the refund is greater than first payment, the remainder is applied to second payment, etc.
  6. Late fees assessed vary by state based on statute.

The above rules are only a sample. And there’s interdependency between rules; for example, a merchandise return reduces balance due, which can cause it to be rolled into the next installment, which changes the date a late fee is assessed. Further, the rules change from first payment to following payments, and change again for the last payment.

A Challenging Project

After initial coding and testing, more in-depth testing by quality assurance (QA) to validate all anticipated scenarios for all payment periods was initiated. The plethora of business rules and exceptions created surprisingly numerous scenarios that existed in three different situations: first payment, following payments and last payment. Issues arose as unanticipated exceptions or unexpected results arose due to combinations of account profiles and/or rules producing unintended outcomes. Far more scenarios arose than projected, which only became apparent as the new code was run against production data clones, then compared against real production data. It took a huge amount of effort and time to isolate and resolve all of the discrepancies. Because of rule interdependencies, this was very convoluted. Business rules were modified due to new exceptions or account characteristics; what had been considered a minor project became major. A/R is very complex, and initial investigation had missed many factors, exceptions and issues.

Another complication not apparent until in-depth testing was that a new subroutine used for online A/R inquiry had unexpected limitations. It used account-based rather than order-based A/R logic; while this added desirable function, the subroutine lacked functions like less-than-dollar and past-due date logic that had to be added. Substantial rework was required, but bugs were fixed, gaps were closed, function was refined, and testing finally validated code functionality and integrity.

What a Difference a Mistake Makes

So what’s this have to do with this article’s headline? Shortly before this code reached a point of substantial stability and was judged ready for production, a software freeze was enacted. When the freeze was lifted, production preparations began, including one full-scale test before pulling the trigger. Perhaps because of elapsed time, unique transactions, or they just hadn’t been caught, new errors cropped up. The first few fixes were easy and quick, then a new, rather unique situation arose. A/R and QA experts gathered to determine correct values, with A/R championing one set and QA advocating another, but deferring to A/R as the final authority.

I made the changes necessary to produce A/R’s results. But when QA testing revealed new errors, and I made further corrections to fix those glitches, other code broke. Thus began a frustrating and exhausting series of fix-something-break-something-else sequences that got worse and worse. An application on the verge of production now seemed riddled with errors, contradictions and inconsistencies. Finally, everyone gathered in a room to follow the code through various scenarios, one of which coincidentally turned out to be the one that A/R and QA had disagreed on. Prior to taking the code through a scenario, A/R would calculate the correct answer, helpful in following the code. This time, on this rarified scenario, A/R’s answer turned out to be what QA had maintained all along was correct. A/R conceded they had been wrong.

All of a sudden the fix-something-break-something-else phenomenon made sense. Code written to produce an erroneous but A/R-blessed result set off a cascade of events: every code change designed to fix errors caused by the misguided code propagated inconsistencies that created errors. Essentially, the bad code caused good code to go bad, and resultant fixes caused errors because they were based on bad code, ultimately based on a bad answer. Until the initial problem was fixed, any fix for propagated errors caused further errors. When the base code was changed to produce the correct answer (and previous bad fixes backed out) good code started working again.


How could have this been prevented? In this case an informal discussion was the method of results determination. Maybe QA should have taken a firmer stance. Having a third-party or formal resolution process might help. Putting it in writing might have been helpful, because there might have been more cross-checking. But more than anything else this experience taught me the value of heavy investment in time and up-front analysis and design. Not all scenarios could have been identified, but many would have, probably including this one. In this case, many, perhaps most scenarios, were only discovered through testing. Correct results, relevant accounts, simulated dates and everything involved in this business process should have gotten up-front consensus of the entire development team (A/R, QA, and IT).

Jim Schesvold can be reached at

Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.

comments powered by Disqus



2019 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store
IBMi News Sign Up Today! Past News Letters