PowerVM Installation and Upgrade Experience
PowerVM 3.1 is proving to be a reliable, high-performing virtualization environment.
By Jaqui Lynch05/01/2019
Prerequisites for Upgrading
The minimums for memory (4GB) and disk space (30GB) listed in the readme documents are just that—minimums. I normally provide at least 8GB of memory (if no SSPs) and 16GB (if using SSPs) plus 1 core. Additional memory and cores may be required depending on the number of clients, high performance adapters and workloads. I also provide 2 x 100GB disk luns (1 for rootvg and 1 for alternate disk install) plus extra disk LUNs if I plan to use file backed optical (FBO). Thus, my typical configuration is at least 2 x 100GB luns for the operating system and a 100GB or 200GB lun for the file backed optical.
Power VM 3.1 is supported on POWER7+, POWER8 and POWER9. For the POWER7+ servers, this means the D model servers and some of the Flex nodes. It doesn’t support the Bladecenter blades. If you have older servers or blades, I’d recommend upgrading them to at least PowerVM 22.214.171.124 to make sure they are at a fully supported level, even though they may not be able to run PowerVM 3.1. All levels prior to PowerVM 2.2.6 will be out of support by the end of 2019, so this is the time to upgrade.
If you plan to use NIM to maintain your VIO servers, then your NIM server will need to be at AIX 7.2 TL3 SP2 or higher. This is the minimum level required for NIM if you are running PowerVM v126.96.36.199 or v3.1.
Finally, if you use logical volumes (LVs) in your VIO server to provide rootvg volumes to client LPARs, these will need to be backed up prior to any upgrades as they are not backed up or recovered using either version of viosupgrade (NIM or VIO).
Getting the Code
The base code is downloaded from the IBM entitled software site (ESS). When you go to download it from your entitled software you will see it listed as 5765-VE3 PowerVM Enterprise ED V3 or 5765-VS3 PowerVM Standard Edition or 5765-VL3 PowerVM Linux Edition. Technology levels and service packs are downloaded from Fix Central as per normal and, as of November 15, 2018, there is a service pack for 188.8.131.52 available. I went to ESS and just downloaded the flash image as it is a full PowerVM 3.1 image that has been updated to the latest service pack 184.108.40.206. This saves an update step later.
Installing the Code
A fresh install can be done in multiple ways. You can burn the code to DVD (if you have a DVD drive on your server). If you use the flash image then you only need to burn one DVD. A second option is to upload the code to the HMC and install from there over the network. To install from a USB stick you will need to download the flash image from the ESS website. This is a single volume image designed specifically for Flash drives (USB sticks). The flash drive must have at least 16GB free. The v3.1.0 readme has instructions for writing the image out to the USB drive. Using the USB stick for the install has the advantage that you can DLPAR it to and from VIO LPARs, whereas on some servers you cannot do that with the DVD or the servers (POWER9) do not come with an internal DVD drive. In all cases I used the flash image as it was a single image at the latest service pack.
If you plan to use NIM there are three mksysb images across the two base 3.1.0 DVDs that will need to be concatenated together into one large mksysb image which will become the NIM resource, or you can use the flash image which has a single mksysb and is at the latest service pack. Note: In order to extract the mksysb images from the iso file you normally loopmount the iso on the VIO or an AIX system and then copy the mksysb image or images into a directory where you can work with them. I was able to mount the iso image for the flash iso on AIX, but was unable to read anything off it. In order to get the mksysb image off the flash iso I had to mount it on a Linux system or my windows system, then extract the mksysb and copy it up to the system I wanted to use it on.
The installation method I chose for the fresh install was from USB using the flash image (burned the flash iso to a USB), and for the upgrade I uploaded the mksysb I extracted from the flash iso to my VIO and used the VIO server viosupgrade command.
Once the installation method has been chosen, the VIO LPAR profile is built and resources are assigned. The VIO LPAR then gets booted into SMS mode and is installed the same way you would normally install a VIO server.
After it’s installed and it reboots (takes about 30 minutes), you then log in as padmin and it will require you to set the password and then to reply a to accept the licenses.
Once you are logged in you should see the $ prompt
I then type in: license –accept
ioslevel should now show 220.127.116.11
I then type in oem_setup_env to become root and run a couple of checks:
instfix -I | grep ML
shows all filesets on
All filesets for 18.104.22.168_AIX_ML were found.
All filesets for 7200-00_AIX_ML were found.
All filesets for 7200-01_AIX_ML were found.
All filesets for 7200-02_AIX_ML were found.
All filesets for 7200-03_AIX_ML were found.
# oslevel -s
Based on the above the operating system installed cleanly and I can now go on and do my regular customizations. IBM is still installing two page spaces on the same LUN, so this needs correcting first.
# lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum
paging00 hdisk0 rootvg 1024MB 1 yes yes lv 0
hd6 hdisk0 rootvg 512MB 0 yes yes lv 0
I always remove paging00 and make hd6 at least 4GB.I also increase the size of several filesystems. Filesystems that I usually increase are / to 2GB, /usr to 10GB, /var to 2GB, /tmp to 5GB, /admin to 1GB, and /opt to 5GB. If I plan to use file backed optical a lot, then I create a user (typically jaqui) and increase /home to 20GB so I can easily upload files and then move them into the repository (/var/vio/VMLibrary). I also setup a separate filesystem called /usr/local (about 10GB) and a logging filesystem called /usr/local/logs (1GB). I edit /etc/syslog.conf and add the following lines to the end:
Then I create the logs and stop and start logging.
touch mailog syslog infolog messages
stopsrc -s syslogd
startsrc -s syslogd
This means I have proper logging on the system. If I run out of space in the new logging filesystem then it just stops logging. If I do the logging into /var then running out of space can take the system down.
I perform a number of other customizations such as upgrading SSH, SSL and Java to the latest levels as well as installing any patches identified by FLRTVC (IBM’s fix level recommendation tool vulnerability checker). These patches are installed using emgr. As of today, the updated levels for Java, etc are:
SSL is at: 22.214.171.1241
SSH is at: 126.96.36.1990
Java is at: 32 bit and 64 bit at 188.8.131.520
Java6 and Java5 are no longer used as of PowerVM 2.2.6.
The patches installed (as of 4/20/2019) based on FLRTVC recommendations are:
# emgr -l
ID STATE LABEL INSTALL TIME UPDATED BY ABSTRACT
=== ===== ========== ================= ========== ======================================
1 S IJ13334s2a 04/16/19 12:07:16 ifix for apar IJ13334
2 S IJ12983m2a 04/16/19 12:07:48 Fix Vulnerabilities in tcpdump
3 S IJ09625s2a 04/16/19 12:08:08 IJ09624 184.108.40.206
4 S IJ12809s2b 04/16/19 12:08:47 ifix for apar IJ12809
5 S IJ11550s0a 04/16/19 12:09:13 Xorg Security Vulnerability fix
These final steps will also need to be performed after the upgrade if you do an upgrade to PowerVM 3.1 rather than a fresh install.
Upgrades and Alternate Disk
For the upgrade I chose to use the VIO server version of viosupgrade—this is different than the NIM command viosupgrade, which uses different flags. This allowed me to do the upgrade on the running system to an alternate disk.
If you plan to use an alternate disk (highly recommended) for your install, then you’ll need to have an extra disk in the VIO server that you can perform the install against. That disk needs to be available and not assigned to any other volume group. You also need to ensure you do not have a volume group named altinst_rootvg already.
When the viosupgrade is run it will try to create that volume group. If you do have a disk with that name and you need to keep it then you should rename it by using exportvg to export it and then importvg to import it with a different volume group name.
Preparing to Upgrade from PowerVM 2.2
Typically, updates have taken place on the VIO using the updateios command. However, updateios only supports updates for technology levels and service packs and other patches. In order to upgrade to PowerVM 3.1, it is necessary to use the viosupgrade tool. To do this your VIO has to be at a minimum of 220.127.116.11 (18.104.22.168 if using SSPs). I always upgrade to 22.214.171.124 prior to the PowerVM 3.1 upgrade.
These levels allow you to use VIO viosupgrade with alternate disk install and that means significantly less risk. Moving to PowerVM 3.1 is a version upgrade so it will create a clean rootvg, hence the preference for alternate disk install. It’s highly recommended that full backups are taken prior to the update including using viosbr to backup meta-data. I would also save any non-VIO server application data to a remote location. The metadata should be backed up using viosbr and those files should be copied to a remote location. An automatic backup can be set up to keep the last 7 copies by using:
viosbr -backup -file vioservername -frequency daily -numfiles 7
You perform this once and then never worry about it again. You can check those backups as follows:
ls -al /home/padmin/cfgbackups
viosbr -view -list
If you are using LVs (logical volumes) to provide disks to any client LPARs then do not forget to back these up as the upgrade will not do so.
There are specific steps to follow if you are using SSPs (shared storage pools) and these are detailed in the readme and also in Nigel Griffith’s AIXPert blog. The key point to note is that all upgrades to PowerVM 3.1 that involve VIO servers with SSPs require that the system to be upgraded is at PowerVM 126.96.36.199 prior to the upgrade.
The Actual Upgrade
Using the VIO server version of viosupgrade ensures that certain things are done for you that the NIM version doesn’t do. The VIO server viosupgrade performs the following:
· It does the config backup for you
· It builds vios 3.1 on the new disk
· It migrates the config
· It sets the bootlist
· It reboots
· It then restores the config backup
· It reboots again
I always take both a regular and a NIM backup regardless. /usr/local/backups is an NFS mounted filesystem from my NIM server which is where my backups go to. There seems to be a bug in PowerVM where it ignores the -nomedialib and I get huge backups because it includes my FBO library, even when it is not in rootvg. So my simple solution is to unmount it prior to the backup and this works.
su - padmin -c "ioscli backupios -file /usr/local/backups/vio1-previo31-apr1319.mksysb -mksysb -nomedialib"
#vio1 below is a directory where the nim resources will go
su - padmin -c "ioscli backupios -file /usr/local/backups/vio1 -nomedialib"
The next step is to perform the alternate disk upgrade. The current disk is hdisk0 and the alternate disk is hdisk1. The disk to be used cannot be owned by anyone else. You also cannot have an altinst_rootvg volume group anywhere. If you have a disk in that volume group then you need to export it. If you want to keep it you can import it with another name:
If hdisk2 is in altinst_rootvg
importvg -y hdisk2 rootvgcopy
To determine if the disk you plan to use is truly free as padmin you can use
You can also check it is big enough using
If the disk does not show up in the free list then you need to check it is not really in use (lsmap -all, etc). Once you are sure it is safe you need to clear the disk:
chpv -c hdisk1
chpv -C hdisk1
It should now show as free (the -C cleared ownership)
NAME PVID SIZE(megabytes)
hdisk1 00f95d3ad4c037cb 270648
In /home/padmin create a file called filestosave.txt. It should be a list of files you want the upgrade to save such as /etc/inetd.conf, etc.
My filestosave.txt consists of:
These will get saved to /home/padmin/backup_files and you can compare to them after the upgrade.
Change into the directory the mksysb image is in and then:
viosupgrade -l -i /usr/local/soft/vios31-flash-mksysb_image -a hdisk1 -g /home/padmin/filestosave.txt
You will see a Welcome, then a number of other messages ending with:
VIOS will be rebooted after '60' seconds to boot from the newly installed disk.
Press contrl+c to terminate.
If you hit ctrl-c don’t forget to reset the bootlist to your current image as the viosupgrade sets it to the new disk.
If you allow it to continue it will reboot and start the upgrade on the new disk. It may reboot a couple of times and it sits on multi-user initialization completed for a while (about 10 minutes in my case). After the final reboot, you get a login prompt and are then required to change your password and to accept the license. After that you type in license -accept even though you just accepted the license.
You then type in viosupgrade -l -q to check on the upgrade status. It will tell you the upgrade is in progress and will then start restoring from the viosbr backup it took. At the end of that it will reboot. It rebooted on me twice at this point and then came up complete.
At this point I checked the system status and had to perform all my normal customizations as if it was a fresh install (paging, filesystem increases, log filesystem, etc.). I also had to rerun my scripts to tune the virtual buffers. Finally, I installed the updates to java, ssh, ssl and the FLRTVC identified patches.
There are a number of logs updated during the process but the key one if you have any problems is /var/adm/ras/ioslogs/viosupg_global.log.
Don’t forget to run another viosbr and mksysb backup after the install or upgrade is complete. Since we used alternate disk install the fall back is to reboot from the original untouched disk (in this case hdisk0). It’s also important to upgrade both VIO servers the same day, if you’re pairing them. Before upgrading the second VIO, check the clients to make sure they have all their paths back then go ahead and do the upgrade.
Now’s the Time to UpgradePowerVM 3.1 is almost 6 months old now so the time is right to start planning those upgrades. You should allow the same amount of time you would allow for a fresh install as you will have to redo any customizations that you normally include in your VIO server installations. It’s a very clean install and the upgrade process using VIO viosupgrade works very well. The key to success is to ensure your current VIO is healthy and to bring your VIO up to 188.8.131.52 prior to the upgrade. This significantly reduces potential issues and helps ensure a clean upgrade. Don’t forget to read the readme documents for the v3.1.0 base and the v184.108.40.206 service pack. Careful preparation and lots of backups should make this a very smooth installation and/or upgrade.
For more information on PowerVM 3.1, check out the following reading materials:
What's new in Virtual I/O Server commands
Virtual I/O Server release notes – include USB Memory/Flash key install
VIOS viosupgrade command in VIOS 220.127.116.11
NIM viosupgrade command on the NIM AIX 7.2 TL3 + sp
IBM Powervm 3.1 Announcement
PowerVM 3.1.0 Base Readme
PowerVM 18.104.22.168 Service Pack readme
Upgrading the VIO server (non-SSP)
VIOS 3.1 Upgrade presentation handout (Raleigh User Group April 2019)
Jaqui’s Presentations and handouts online
PowerVM UK User Group – Nigel Griffiths Presentation on Upgrading to VIOS 3.1 in action
FLRT (Fix level reporting tool)
Jaqui Lynch has over 38 years of experience working with a projects and OSes across vendor platforms, including IBM Z, UNIX systems and more.More →