Upgrading Your HMC from 8.8.2 or Later
Performing the upgrade and tackling the issues involved with moving to 8.8.6 SP2 from 8.8.2 or 8.8.3.
By Jaqui Lynch02/01/2018
In a previous article, I’d discussed general maintenance for your HMCs. Since then, I’ve run into some issues relating to upgrading to 8.8.6 SP2 from 8.8.2 or 8.8.3. In this article, I’ll go over what is involved in performing that upgrade and will also cover the checks that should be performed along the way. Many of the steps can be performed using the GUI, however, in this article I’ll be using SSH (secure shell) to the HMC and will use the command line.
Process for Upgrades
In order to upgrade from 8.8.2 or 8.8.3 to 8.8.6 sp2 MH01743 (this is the latest patch level) there is a multi-step process unless you want to reinstall the HMC. Each upgrade requires a reboot. The reboot takes about 20 minutes and the upgrades about 30 minutes. The backups to USB take between 30 and 45 minutes. I would allow 8 hours for these upgrades.
You’ll need 3 USB sticks for the backups. You should also make sure you have a copy of the recovery media in case you have to do a full reinstall. Throughout, I assume you will be doing the upgrades across the network. If you’re unable to do this, then you can also download the upgrades to USB sticks and perform the upgrades from those USB sticks. If you’re using redundant HMCs then they’ll both need to be updated the same day to avoid issues.
The upgrade process is:
- Check your current level
- Backup the server profiles
- Get an HMC Scanner report and fully document your HMC including networking, userids, groups, etc.
- Backup the HMC
- Clear out old upgrades and traces to ensure enough space
- Upgrade to 8.8.4 (MH01559)
- Upgrade to MH01560 (mandatory) – this must be done using the commandline updhmc not the GUI
- Upgrade to MH01729
- Backup the HMC
- Upgrade to 184.108.40.206 (MH01654)
- Reboot again once it comes up
- Backup again
- Upgrade to 8.8.6 sp2 (MH01690)
- Upgrade to MH01743
- Backup the HMC again
First, ensure your HMC can be upgraded to 8.8.6. This level requires that the HMC must be at least a C08 deskside or a CR5 rack mount. It could also be a virtual HMC which is a VM on an Intel server. The new POWER based HMC requires 8.8.7. Earlier HMCs are not supported on v8. Additionally, POWER5 servers are not supported on v8.
The next step is to determine the level you are currently running. This is done by logging into the HMC and issuing the lshmc -V command. You can also get this information from the GUI by hovering over the HMC version on the home page.
lshmc -V "version= Version: 8 Release: 8.3.0 Service Pack: 0 HMC Build level 20150611.1 MH01514: Required fix for HMC V8R8.3.0 (05-21-2015) MH01530: Fix for HMC V8R8.3.0 (06-11-2015) ","base_version=V8R8.3.0 "
The above system is at 220.127.116.11 SP0 with two patches applied.
Prior to upgrading it is important to read the description file for every planned upgrade. This is where you will find the list of any prerequisites for the upgrade. You should also note that the HMC must currently be at v8.8.4+mandatory fix MH01560 or v8.8.5+mandatory fix MH01617 prior to upgrading to v8.8.6. The 18.104.22.168 description file also makes the following recommendations:
Your VIO servers may need to be upgraded if they are not at 2.2.3 yet. AIX, Linux and VIOS LPARs should be at RSCT 22.214.171.124 or higher Lastly, verify the server firmware compatibility with the HMC level. This is a critical step and helps ensure that you don’t upgrade to a level of the HMC that is not supported for the firmware your servers are currently running. You can find the compatibility list here.
Backups Prior to Upgrading
You’re now ready to start the upgrade process and it’s important to take a backup prior to upgrading. That way if things go wrong you can still recover. There are two parts to the backup. The first is to take a local backup of the critical console data. This is done to ensure that local customizations are saved. Then you take a backup of the HMC itself. A backup will be needed prior to upgrading to 8.8.4.
I typically also backup the system profiles.
lssyscfg -r sys - gets managed system names
Now we back up their profiles:
bkprofdata -m systemname -f backup1-feb0418
This backs up the system profile data to a file called backup1-feb0418 to the /var/hsc/profiles directory on the HMC. It will create a subdirectory for each managed system you decide to backup. You should backup each managed system.
The next step is to save upgrade data to disk.
saveupgdata -r disk
And then finally you can backup to the USB stick. Remember to format the USB stick prior to backup. It must be formatted as FAT or FAT32.
bkconsdata -r usb
We now check we have enough memory and disk space for the upgrade. If an upgrade fails with an error E212E13A then you need to free up some disk space.
ls -al /dump total 1595256 -rw-r--r-- 1 root hmc 1580681889 Feb 13 2016 HMClogs.JAQUIHMC0213H41.zip
This deletes the log files collected in /dump
Updating to 8.8.4
Updating from 8.8.2 or 8.8.3 to 8.8.4 is a three-step process. This consists of upgrading to 8.8.4, then applying MH01560 and then MH01729. This is required before upgrading to 8.8.6.
During the update to MH1560 it is possible that service code E2FF4304 may be reported. This can be ignored as it indicates a temporary file access issue during installation. It should also be noted that MH01560 must be installed using the updhmc command—it cannot be installed using the GUI.
The upgrade to 8.8.4 is a major upgrade and we use the getupgfiles command to download the upgrade files. In order to do this, we open two SSH sessions to the HMC. On one I would be running the following script to monitor the upgrade:
while true ; do date ls -la /hmcdump ls -la /hmcboot sleep 60 done
The above monitors the /hmcdump filesystem where the files get downloaded to. It also monitors /hmcboot which is where the files get moved to after they are downloaded.
On the second session you perform the download:
getupgfiles -h public.dhe.ibm.com -u anonymous --passwd anonymous -d /software/server/hmc/network/v8840
Typically, you will see
base.img, disk1.img, hmcnetworkfiles.sum, img2a and img3a being downloaded
The above downloads the actual upgrade files.
Once the software is downloaded the download session will return to the prompt and you will no longer see the files in /hmcdump and /hmcboot. You can now exit the monitoring script with ctrl-c and logoff that session. Then tell the HMC to perform the upgrade by issuing the following two commands:
chhmc -c altdiskboot -s enable --mode upgrade
The above tells it to set up to upgrade on boot
hmcshutdown -r -t now
It takes about 20-30 minutes to actually perform the upgrade and reboot. Once it’s done you then you need to install the other fixes. Each upgrade or fix will take at least 20 minutes as a reboot of the HMC is required for each. If you have redundant HMCs you should do them both the same day as problems can occur when they are out of sync. This is explained in the description files. While installing MH01560 (the first update), the HMC may do a call home with an E35F0012, which is a known issue. This is documented as a known issue in the HMC update documentation.The following 2 updates will stop it happening again
This has to be done using the updhmc command. It cannot be done from the GUI.
updhmc -t ftp -h ftp.software.ibm.com -u anonymous -f /software/server/hmc/fixes/MH01560.iso
You will get a message once it is downloaded saying the service was applied and reminding you to reboot. Don’t put any other patch on till after you reboot.
hmcshutdown -r -t now
Now you can apply fix MH01729
updhmc -t ftp -h ftp.software.ibm.com -u anonymous -f /software/server/hmc/fixes/MH01729.iso And again, you will need to reboot hmcshutdown -r -t now
After this reboot I recommend taking another backup to the USB stick. Once you get to 8.8.6 you won’t be able to restore from any pre 8.8.4 backups. Run the saveupgdata and bkconsdata again to a new USB stick.
Updating to 8.8.6
Now that the system is at 8.8.4 MH01729 you can start the 8.8.6 upgrade. This consists of upgrading to 8.8.6 (MH01654), then applying SP2 (MH01690) and then MH01743.
First check it is really at 8.8.4 MH01729: lshmc -V "version= Version: 8 Release: 8.4.0 Service Pack: 0 HMC Build level 20151112.2 MH01560: Required fix for HMC V8R8.4.0 (11-12-2015) MH01729: Fix for HMC V8R8.4.0 (11-07-2017) ","base_version=V8R8.4.0 "
Clear out the temp files from the previous upgrade:
monhmc -r disk -n 0
Now update to base 126.96.36.199 (MH01654). Set up the same monitoring loop on one SSH session and do the upgrade on the other.
getupgfiles -h public.dhe.ibm.com -u anonymous --passwd anonymous -d /software/server/hmc/network/v8860
When the download completes tell it to do an upgrade on reboot and then reboot
chhmc -c altdiskboot -s enable --mode upgrade hmcshutdown -r -t now
After 188.8.131.52 boots you’ll need to reboot it again. There’s a known problem where the HMC will report event E212E11C after an upgrade from 8.8.4 or 8.8.5 as it needs the second reboot to initialize the swap space.
After the second reboot you can install 8.8.6 sp2 (MH01690)
updhmc -t ftp -h ftp.software.ibm.com -u anonymous -f /software/server/hmc/ updates/HMC_Update_V8R860_SP2.iso
You will need to reboot it
hmcshutdown -r -t now
Finally put on ifix MH01743
updhmc -t ftp -h ftp.software.ibm.com -u anonymous -f /software/server/hmc/fixes/MH01743.iso
Again, you will need to reboot it
hmcshutdown -r -t now
The HMC is now at the latest level of 8.8.6. lshmc -V should now show:
"version= Version: 8 Release: 8.6.0 Service Pack: 2 HMC Build level 20171211.1 MH01743: Fix for HMC V8R8.6.0 SP2 (01-12-2018) ","base_version=V8R8.6.0 "
Take another backup, and don’t forget the saveupgdata.
Doing an upgrade from 8.8.2 or 8.8.3 to the latest HMC levels is not a small task. You may want to evaluate whether it is better to just reinstall your HMC. Staying up to date is a time-consuming but necessary process. If you update your HMC regularly you’ll avoid having to go through the lengthy process described here. I typically do quarterly updates of my HMCs in order to ensure that I am able to move forward easily with new servers and functionality. Regular backups and updates should be built into your system maintenance plans.
8.8.5 requires 184.108.40.206 or higher as a starting point
Upgrade to 220.127.116.11 (MH01615)
Mandatory fix MH01617
Then 18.104.22.168 (MH01704)
Then 22.214.171.124 Mandatory PTF MH01706
Then 126.96.36.199 SP1 (MH01725)
Finally, the latest fix is MH01748
There will be backups and additional reboots throughout this process.
Also note: 8.8.7 only supports the following HMC models: 7042-cr7, cr8, cr9, OE1, OE2. And the new Power 7063-CR1.
Jaqui Lynch has over 38 years of experience working with a projects and OSes across vendor platforms, including IBM Z, UNIX systems and more.More →