Change Management: Approval Must Be Earned
Once a change has been envisioned, formulated, tested and quality-assured, it’s time for the rubber to hit the road. Well, not quite. In any professional organization that values high availability, change control is a rigorous, formalized, scrutinizing discipline that screens changes for readiness. A lot of variables need evaluation in determining when, how and if a change should be implemented in a business-critical IT system. The famous “W” list—who, what, when, where and why, plus misfit how—are good questions to ask in a situation like this.
Change approval is vital no matter what the change or how dire the situation. Even when a system’s down and everyone’s screaming bloody hell, the adage “while things are never so good they can’t get better, they’re never so bad they can’t get worse” holds true. Major outages are the worst because desperation leads to bad decisions. The answer to “What do we have to lose?” is almost invariably “A lot!” I’ve seen where someone bypassed change control while running file maintenance and, because of insufficient testing, thousands of databases and/or files were lost. Many had to be manually repopulated—an overwhelming effort—and some client and financial data were gone. Change control would have enforced testing and saved business-critical data.
According to dictionary.com, change is “to make the form, nature, content, structure or any constituent component of something [i.e., an information system] different from what it is or what it would be if left alone.” Normally we think of a change as a program or vendor software modification, but changes can be procedural, hardware, network, scheduling, data structure, security-related, etc.
A good change approval process encompasses all changes, evaluates every aspect, determines whether standards and scheduling requirements are met, and assesses risk and impact; in every way it vets whether a change is ready. If so, it’s approved; if not, it’s pended, rescheduled or rejected. Below is what it takes.
Sometimes a change meeting gets too large, so attendees should be limited to those either involved with a change or members of change control, including:
- Owner of change and possibly manager.
- Change manager/approver. May be delineated by platform (mainframe, midrange, LAN-based, etc.).
- Secretary to take minutes and maintains change calendar.
- Peer reviewer (fellow technician) who reviewed the change in detail.
The change calendar keeps track of approved changes and what time and date they’re scheduled. This serves to ensure:
- Scheduling conflicts are quickly identified. In general, only a single change should be performed in a given system or subsystem at one time. There will be valid exceptions; they should be carefully vetted.
- There are certain times of the day, week or month when changes can be performed. Scheduled system IPLs or boots usually occur late night or early morning on weekends, and certain changes require that. Conversely, weekdays during business hours are times when changes are avoided if possible.
- Special circumstances always require special handling, exceptions to the rules.
Degree of Difficulty
The scope, extent and effort of a change are key factors in assessing the risk of a change because:
- Changes vary in the time they take to perform, and is based on more than just effort. IPLs/reboots, jobs— especially tape jobs—and other similar processes can involve significant wait time when making a change.
- The more steps—manual changes, jobs to run, hardware to move or cables to pull—the greater the degree of difficulty.
- The amount of testing necessary to validate the change influences difficulty. Sometimes only a single test of a single function is needed to verify a change, other times it may take a full quality assurance staff.
- Other factors like time limits, concurrent changes, staff availability, etc., can affect a change’s difficult.
Not only does a change’s difficulty need to be assessed; the effect of a change going wrong needs evaluation and a cost estimate. Questions to be considered include:
- How many users will be impacted?
- What functions will be affected?
- If an outage occurs, what is the likely duration?
- What secondary effects could occur to other systems? Batch delays? Missing input/files?
- Will there be impacts to staffing during data recovery or manual processing (e.g., taking orders by hand)?
- What is the estimated time to back out a change during prime time?
Based on the answers to questions like these, the change should be rated on a sliding scale from low to high.
The change plan should be carefully inspected for completeness and clarity, and should have been distributed to all interested parties one to three days before change approval so members can review the plan and provide feedback for corrections or additions. The change plan should at least:
- Be written in clear and easy-to-understand instructions such that someone with only a modicum of knowledge could execute the plan.
- Be thorough and document every step involved in every aspect of the change.
- Reference any supporting documentation and where it may be found.
- Include a back out plan that addresses every required step, including manual or automated steps.
- Include a testing and validation plan that documents every step and the results that should be observed to verify that the change is working as expected.
- Include any cleanup steps that need to be taken, and should identify cleanup steps that play a role in backup and/or recovery, as well as the time to wait until the change is considered complete.
- Include a time period after which back out will be impossible or inadvisable, including reasons and considerations.
Peer Review and Risk Assessment
After reviewing all previously discussed items and soliciting peer opinion, a risk on a scale from low to high is assigned, and the change is either approved onto the change calendar, pended or rejected. After the meeting, the calendar and change plans—or links to them—are distributed to relevant departments and managers for informational purposes and feedback.
Change Control Mandatory
Nobody, absolutely nobody should be able to bypass change control. That’s where I’ve seen the worst disasters; someone thinks a change is too minor, they’ve done this a million times and can’t make a mistake, it’s urgent, they’re repairing a mistake and don’t want anyone to know, the reasons are rampant. It doesn’t matter, it has to go through change control. The stakes are just too high.
More in the Series
Fifteenth in my how-not-to series looks at the value of change control, and is based on my more than 35 years of work in mainframe IT, reviewing mistakes I’ve seen or participated in and helping others to avoid them. Find previous stories in the series on my author bio page.
Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.
comments powered by