Making Changes: Prudence and Discipline Always
IT centers are evolution in fast forward; technology rapidly advances, applications constantly mutate—partly due to technological advancement; demands for new function; or legal requirements, security issues, changing society and transaction volumes. Demand drives change to unexpected heights. Central to this is the interrelationship of parts. Changing one part cannot be performed in isolation; what enhances or fixes one part might break another. All this change must be managed and controlled; if it is not, the result can be disaster.
I’ve seen a few disasters: Most are avoidable, although some will happen regardless of how much care or planning is taken. But it’s the ones where the same misguided approach is used time after time that are the worst. It’s the embodiment of Einstein’s definition: “insanity is doing the same thing over and over again and expecting different results.” I was in an engagement where a company worked with a mainframe outsourcer and supported the IT operation. I was retained to investigate how an outsourcer’s staff member could make a minor change that deleted thousands of business files—it was due to a misunderstanding of how a software feature worked. That change was never properly vetted or tested. During the next couple months, the same individual stalled the computer by submitting a massive number of jobs, then inadvertently varied a production drive offline, both times bypassing change control and without testing. These weren’t the first mistakes this individual made, but they were the last, costing him his job. While the mistakes were quite different in nature, the underlying cause was the same—avoidance of change management discipline.
Change management is a discipline, a set of rules and methods designed to minimize mistakes and their impact that can occur when making changes. While rules and methods may vary between organizations, there are underlying principles and components that are vital to a successful, effective change management system. Here’s what I believe are the most important ones:
Think Things Through
The terms “transactional” and “analytical” come to mind when making changes. A transactional approach is focused solely on getting the change done. An analytical approach not only involves the change, but what other components may be affected, how the change might go wrong, at interdependencies between the change and other components. Transactional thinking is atomistic, only considering the change and assuming nothing can go wrong, while analytic thinking is holistic, looking at all aspects of the change and identifying where it could go wrong and how to avoid it.
Backout Plans Rule
Backout plans are more vital to a successful change than the change plan itself. That’s because a good backout plan can mitigate the damage of a change gone awry, and can reduce the length of an outage—often a malfeasant change’s most damaging effect—by orders of magnitude. A backout procedure consists of tasks reversing the change, so the change plan in reverse can be used for designing backout instructions. However, other steps or jobs may also be needed.
Test Before Execution
Test systems are extremely useful for testing changes and backout plans. When I’d install a new product release, there was a rename process—changing the names of the runtime data sets to a “.old” suffix and the new-release data sets to a “.new” suffix—and I’d run it in both directions to reverse the names back to original values. This would test both the rename to the new product and rename to old release. It wasn’t unusual to have an error or two the first time, which was no big deal and easy to fix, sometimes revealing a data set naming mistake rather than a typo. But it would have been a big deal if that error happened during the upgrade.
Jobs Eliminate Typos
When a task can be performed either by online command or via batch, I prefer batch. Command syntax can be carefully checked, via eyeball and via products that validate the Job Control Language. The commands usually can be run in test to validate them. The output goes into a print data set, reviewable as necessary, but there’s no consistent form of review for online commands, especially from Time Sharing Option.
I’ve worked with numerous clients whose primary measure of a programmer is how fast they can get something done; they often attempt to do that by overloading their staff with work. I’ve never seen that approach work and, especially with changes, it’s a recipe for disaster. The odds of making a mistake are greatly magnified when a short window and excessive pressure is inherent in a change. Deliberation and thoroughness are critical to a successful change.
Document, Document, Document
Having a thorough plan isn’t enough; it has to be documented. The plan should be complete and concise and include all details, decision and completion criteria, issues, contact information, contingencies and measurement metrics. The plan should be written such that someone else could do by following the directions. It should be saved in a library, as an audit trail and for use with a future changes. Clear and complete documentation can mean the difference between success and failure of a change.
Double-Check and Peer Review
After all aspects of a change plan have been completed and tested, it’s a great idea to step away from it, take a break and do something else—assuming it wasn’t done at the last minute, which is never a good idea. After refreshing one’s mind, run through the change plan again, testing only where something was missed. It’s amazing the things that emerge in a second review, how things become obvious. And have a peer review the plan; another person brings a second set of experience validating the steps and refining and adding vital details.
Nobody likes to be on the wrong end of a change gone badly. Sometimes a mistake can pass unknown or without consequence, but that can have consequences, too, because it encourages sloth and carelessness. And when something does go wrong, the consequences can be severe. So can the consequences of regularly botching changes. It doesn’t have to be that way. By thinking a change through, having rigorous backout plans, carefully testing, pre-built jobs, carefully documenting every step and procedure, having a fellow member’s review, changes can be performed smoothly and correctly. Deliberation, prudence, thoroughness and discipline mean success.
More in the Series
Fourteenth in my how-not-to series looks at the value of change management, and is based on my more than 35 years of work in mainframe IT, reviewing mistakes I’ve seen or participated in and helping others to avoid them. Find previous stories in the series on my author bio page.
Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.
comments powered by