AIX > Administrator > Virtualization

PowerVM Installation and Upgrade Experience

Upgrade to PowerVM 3.1
PowerVM 3.1 has now been generally available for almost six months and is proving to be a reliable, high-performing virtualization environment. In this article I’ll recap my recent experiences doing a fresh install as well as an upgrade to PowerVM 3.1. Planning is critical as there are some prerequisites for PowerVM 3.1 that need to be noted.
 
Prerequisites for Upgrading
The minimums for memory (4GB) and disk space (30GB) listed in the readme documents are just that—minimums. I normally provide at least 8GB of memory (if no SSPs) and 16GB (if using SSPs) plus 1 core. Additional memory and cores may be required depending on the number of clients, high performance adapters and workloads. I also provide 2 x 100GB disk luns (1 for rootvg and 1 for alternate disk install) plus extra disk LUNs if I plan to use file backed optical (FBO). Thus, my typical configuration is at least 2 x 100GB luns for the operating system and a 100GB or 200GB lun for the file backed optical.
 
Power VM 3.1 is supported on POWER7+, POWER8 and POWER9. For the POWER7+ servers, this means the D model servers and some of the Flex nodes. It doesn’t support the Bladecenter blades. If you have older servers or blades, I’d recommend upgrading them to at least PowerVM 2.2.6.32 to make sure they are at a fully supported level, even though they may not be able to run PowerVM 3.1.  All levels prior to PowerVM 2.2.6 will be out of support by the end of 2019, so this is the time to upgrade.
 
If you plan to use NIM to maintain your VIO servers, then your NIM server will need to be at AIX 7.2 TL3 SP2 or higher. This is the minimum level required for NIM if you are running PowerVM v2.2.6.32 or v3.1. 
 
Finally, if you use logical volumes (LVs) in your VIO server to provide rootvg volumes to client LPARs, these will need to be backed up prior to any upgrades as they are not backed up or recovered using either version of viosupgrade (NIM or VIO).
 
Getting the Code
The base code is downloaded from the IBM entitled software site (ESS). When you go to download it from your entitled software you will see it listed as 5765-VE3 PowerVM Enterprise ED V3 or 5765-VS3 PowerVM Standard Edition or 5765-VL3 PowerVM Linux Edition. Technology levels and service packs are downloaded from Fix Central as per normal and, as of November 15, 2018, there is a service pack for 3.1.0.10 available. I went to ESS and just downloaded the flash image as it is a full PowerVM 3.1 image that has been updated to the latest service pack 3.1.0.10. This saves an update step later.
 
Installing the Code
A fresh install can be done in multiple ways. You can burn the code to DVD (if you have a DVD drive on your server). If you use the flash image then you only need to burn one DVD. A second option is to upload the code to the HMC and install from there over the network. To install from a USB stick you will need to download the flash image from the ESS website. This is a single volume image designed specifically for Flash drives (USB sticks). The flash drive must have at least 16GB free. The v3.1.0 readme has instructions for writing the image out to the USB drive. Using the USB stick for the install has the advantage that you can DLPAR it to and from VIO LPARs, whereas on some servers you cannot do that with the DVD or the servers (POWER9) do not come with an internal DVD drive. In all cases I used the flash image as it was a single image at the latest service pack.
 
If you plan to use NIM there are three mksysb images across the two base 3.1.0 DVDs that will need to be concatenated together into one large mksysb image which will become the NIM resource, or you can use the flash image which has a single mksysb and is at the latest service pack. Note: In order to extract the mksysb images from the iso file you normally loopmount the iso on the VIO or an AIX system and then copy the mksysb image or images into a directory where you can work with them. I was able to mount the iso image for the flash iso on AIX, but was unable to read anything off it. In order to get the mksysb image off the flash iso I had to mount it on a Linux system or my windows system, then extract the mksysb and copy it up to the system I wanted to use it on.
 
The installation method I chose for the fresh install was from USB using the flash image (burned the flash iso to a USB), and for the upgrade I uploaded the mksysb I extracted from the flash iso to my VIO and used the VIO server viosupgrade command.
 
Fresh Install
Once the installation method has been chosen, the VIO LPAR profile is built and resources are assigned. The VIO LPAR then gets booted into SMS mode and is installed the same way you would normally install a VIO server.  
 
After it’s installed and it reboots (takes about 30 minutes), you then log in as padmin and it will require you to set the password and then to reply a to accept the licenses. 
 
Once you are logged in you should see the  $ prompt
I then type in: license –accept
ioslevel should now show 3.1.0.10
$ ioslevel
3.1.0.10

I then type in oem_setup_env to become root and run a couple of checks:
instfix -I | grep ML 
            shows all filesets on
    All filesets for 7.2.0.0_AIX_ML were found.
    All filesets for 7200-00_AIX_ML were found.
    All filesets for 7200-01_AIX_ML were found.
    All filesets for 7200-02_AIX_ML were found.
    All filesets for 7200-03_AIX_ML were found.
 
# oslevel -s
7200-03-02-1846
 

Based on the above the operating system installed cleanly and I can now go on and do my regular customizations. IBM is still installing two page spaces on the same LUN, so this needs correcting first.
# lsps -a
Page Space      Physical Volume   Volume Group    Size %Used  Active    Auto    Type  Chksum
paging00        hdisk0            rootvg        1024MB     1    yes     yes      lv      0
hd6                 hdisk0           rootvg         512MB     0    yes     yes      lv      0
 

I always remove paging00 and make hd6 at least 4GB.I also increase the size of several filesystems. Filesystems that I usually increase are / to 2GB, /usr to 10GB, /var to 2GB, /tmp to 5GB, /admin to 1GB, and /opt to 5GB. If I plan to use file backed optical a lot, then I create a user (typically jaqui) and increase /home to 20GB so I can easily upload files and then move them into the repository (/var/vio/VMLibrary). I also setup a separate filesystem called /usr/local (about 10GB) and a logging filesystem called /usr/local/logs (1GB). I edit /etc/syslog.conf and add the following lines to the end:
mail.debug                /usr/local/logs/mailog
*.emerg                     /usr/local/logs/syslog
*.alert                        /usr/local/logs/syslog
*.crit                           /usr/local/logs/syslog
*.err                           /usr/local/logs/syslog
auth.notice                /usr/local/logs/infolog
*.info                           /usr/local/logs/messages

 
Then I create the logs and stop and start logging.
cd /usr/local/logs
touch mailog syslog infolog messages
stopsrc -s syslogd
startsrc -s syslogd
 

This means I have proper logging on the system. If I run out of space in the new logging filesystem then it just stops logging. If I do the logging into /var then running out of space can take the system down.
 
I perform a number of other customizations such as upgrading SSH, SSL and Java to the latest levels as well as installing any patches identified by FLRTVC (IBM’s fix level recommendation tool vulnerability checker). These patches are installed using emgr.  As of today, the updated levels for Java, etc are:
SSL is at:       1.0.2.1601
SSH is at:      7.5.102.1600
Java is at:      32 bit and 64 bit at 7.0.0.640
Java6 and Java5 are no longer used as of PowerVM 2.2.6.
 
The patches installed (as of 4/20/2019) based on FLRTVC recommendations are:
# emgr -l
 
ID  STATE LABEL      INSTALL TIME      UPDATED BY ABSTRACT
=== ===== ========== ================= ========== ======================================
1    S   IJ13334s2a 04/16/19 12:07:16           ifix for apar IJ13334
2    S   IJ12983m2a 04/16/19 12:07:48           Fix Vulnerabilities in tcpdump
3    S   IJ09625s2a 04/16/19 12:08:08           IJ09624 7.2.3.2
4    S   IJ12809s2b 04/16/19 12:08:47           ifix for apar IJ12809
5    S   IJ11550s0a 04/16/19 12:09:13           Xorg Security Vulnerability fix
 

These final steps will also need to be performed after the upgrade if you do an upgrade to PowerVM 3.1 rather than a fresh install.
 
Upgrades and Alternate Disk
For the upgrade I chose to use the VIO server version of viosupgrade—this is different than the NIM command viosupgrade, which uses different flags. This allowed me to do the upgrade on the running system to an alternate disk. 
 
If you plan to use an alternate disk (highly recommended) for your install, then you’ll need to have an extra disk in the VIO server that you can perform the install against. That disk needs to be available and not assigned to any other volume group. You also need to ensure you do not have a volume group named altinst_rootvg already. 
 
When the viosupgrade is run it will try to create that volume group. If you do have a disk with that name and you need to keep it then you should rename it by using exportvg to export it and then importvg to import it with a different volume group name.
 
Preparing to Upgrade from PowerVM 2.2
Typically, updates have taken place on the VIO using the updateios command. However, updateios only supports updates for technology levels and service packs and other patches. In order to upgrade to PowerVM 3.1, it is necessary to use the viosupgrade tool. To do this your VIO has to be at a minimum of 2.2.6.30 (2.2.6.32 if using SSPs). I always upgrade to 2.2.6.32 prior to the PowerVM 3.1 upgrade.
 
These levels allow you to use VIO viosupgrade with alternate disk install and that means significantly less risk. Moving to PowerVM 3.1 is a version upgrade so it will create a clean rootvg, hence the preference for alternate disk install. It’s highly recommended that full backups are taken prior to the update including using viosbr to backup meta-data. I would also save any non-VIO server application data to a remote location. The metadata should be backed up using viosbr and those files should be copied to a remote location. An automatic backup can be set up to keep the last 7 copies by using:
viosbr -backup -file vioservername -frequency daily -numfiles 7
You perform this once and then never worry about it again. You can check those backups as follows:
ls -al /home/padmin/cfgbackups
Or

viosbr -view -list

If you are using LVs (logical volumes) to provide disks to any client LPARs then do not forget to back these up as the upgrade will not do so.
 
There are specific steps to follow if you are using SSPs (shared storage pools) and these are detailed in the readme and also in Nigel Griffith’s AIXPert blog.  The key point to note is that all upgrades to PowerVM 3.1 that involve VIO servers with SSPs require that the system to be upgraded is at PowerVM 2.2.6.32 prior to the upgrade.
 
The Actual Upgrade
 
Using the VIO server version of viosupgrade ensures that certain things are done for you that the NIM version doesn’t do. The VIO server viosupgrade performs the following:
·     It does the config backup for you
·     It builds vios 3.1 on the new disk
·     It migrates the config
·     It sets the bootlist
·     It reboots
·     It then restores the config backup
·     It reboots again
 
I always take both a regular and a NIM backup regardless. /usr/local/backups is an NFS mounted filesystem from my NIM server which is where my backups go to.  There seems to be a bug in PowerVM where it ignores the -nomedialib and I get huge backups because it includes my FBO library, even when it is not in rootvg. So my simple solution is to unmount it prior to the backup and this works.
umount /var/vio/VMLibrary
su - padmin -c "ioscli backupios -file /usr/local/backups/vio1-previo31-apr1319.mksysb -mksysb -nomedialib"
#vio1 below is a directory where the nim resources will go
su - padmin -c "ioscli backupios -file /usr/local/backups/vio1 -nomedialib"
mount /var/vio/VMLibrary
 

The next step is to perform the alternate disk upgrade. The current disk is hdisk0 and the alternate disk is hdisk1. The disk to be used cannot be owned by anyone else. You also cannot have an altinst_rootvg volume group anywhere. If you have a disk in that volume group then you need to export it. If you want to keep it you can import it with another name:
            If hdisk2 is in altinst_rootvg
exportvg altinst_rootvg
importvg -y hdisk2 rootvgcopy

 
To determine if the disk you plan to use is truly free as padmin you can use 
lspv -free

You can also check it is big enough using
lspv -size

If the disk does not show up in the free list then you need to check it is not really in use (lsmap -all, etc).  Once you are sure it is safe you need to clear the disk:
chpv -c hdisk1
chpv -C hdisk1
 

It should now show as free (the -C cleared ownership)
lspv -free
NAME            PVID                                SIZE(megabytes)
hdisk1          00f95d3ad4c037cb                    270648

 
In /home/padmin create a file called filestosave.txt. It should be a list of files you want the upgrade to save such as /etc/inetd.conf, etc.
My filestosave.txt consists of:
/etc/motd
/etc/netsvc.conf
/etc/inetd.conf
/etc/hosts
/etc/environment
/etc/profile
/etc/inittab
/etc/resolv.conf
/etc/ntp.conf
/etc/group
/etc/passwd
/etc/security/passwd
/etc/security/limits
/etc/security/login.cfg
/etc/tunables/nextboot
/etc/tunables/rc-tunevio.sh
These will get saved to /home/padmin/backup_files and you can compare to them after the upgrade.
 
Change into the directory the mksysb image is in and then:
viosupgrade -l -i /usr/local/soft/vios31-flash-mksysb_image -a hdisk1 -g /home/padmin/filestosave.txt
 
You will see a Welcome, then a number of other messages ending with:
VIOS will be rebooted after '60' seconds to boot from the newly installed disk.
Press contrl+c to terminate.
If you hit ctrl-c don’t forget to reset the bootlist to your current image as the viosupgrade sets it to the new disk.
If you allow it to continue it will reboot and start the upgrade on the new disk. It may reboot a couple of times and it sits on multi-user initialization completed for a while (about 10 minutes in my case). After the final reboot, you get a login prompt and are then required to change your password and to accept the license.  After that you type in license -accept even though you just accepted the license.
 
You then type in viosupgrade -l -q to check on the upgrade status. It will tell you the upgrade is in progress and will then start restoring from the viosbr backup it took. At the end of that it will reboot. It rebooted on me twice at this point and then came up complete.
 
At this point I checked the system status and had to perform all my normal customizations as if it was a fresh install (paging, filesystem increases, log filesystem, etc.). I also had to rerun my scripts to tune the virtual buffers. Finally, I installed the updates to java, ssh, ssl and the FLRTVC identified patches.
 
There are a number of logs updated during the process but the key one if you have any problems is /var/adm/ras/ioslogs/viosupg_global.log.
 
Don’t forget to run another viosbr and mksysb backup after the install or upgrade is complete. Since we used alternate disk install the fall back is to reboot from the original untouched disk (in this case hdisk0). It’s also important to upgrade both VIO servers the same day, if you’re pairing them. Before upgrading the second VIO, check the clients to make sure they have all their paths back then go ahead and do the upgrade.
 

Now’s the Time to Upgrade

PowerVM 3.1 is almost 6 months old now so the time is right to start planning those upgrades. You should allow the same amount of time you would allow for a fresh install as you will have to redo any customizations that you normally include in your VIO server installations. It’s a very clean install and the upgrade process using VIO viosupgrade works very well. The key to success is to ensure your current VIO is healthy and to bring your VIO up to 2.2.6.32 prior to the upgrade. This significantly reduces potential issues and helps ensure a clean upgrade. Don’t forget to read the readme documents for the v3.1.0 base and the v3.1.0.10 service pack. Careful preparation and lots of backups should make this a very smooth installation and/or upgrade. 
 
References
For more information on PowerVM 3.1, check out the following reading materials:
 
What's new in Virtual I/O Server commands
https://www.ibm.com/support/knowledgecenter/en/9040-MR9/p9hcg/p9hcg_whatsnew.htm
 
Virtual I/O Server release notes – include USB Memory/Flash key install
https://www.ibm.com/support/knowledgecenter/en/9040-MR9/p9eeo/p9eeo_ipeeo_main.htm
 
 
VIOSUPGRADE command
https://www.ibm.com/support/knowledgecenter/POWER9/p9hb1/p9hb1_vios_migrate_viosupgrade.htm?view=embed
 
VIOS viosupgrade command in VIOS 2.2.6.30 
https://www.ibm.com/support/knowledgecenter/en/9009-42A/p9hcg/p9hcg_viosupgrade.htm
 
NIM viosupgrade command on the NIM AIX 7.2 TL3 + sp
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.cmds6/viosupgrade.htm
 
IBM Powervm 3.1 Announcement
https://www-01.ibm.com/common/ssi/rep_ca/6/897/ENUS218-346/ENUS218-346.PDF
 
PowerVM 3.1.0 Base Readme
https://www.ibm.com/support/knowledgecenter/POWER9/p9eeo/p9eeo_ipeeo_main.htm
 
PowerVM 3.1.0.10 Service Pack readme
https://www-01.ibm.com/support/docview.wss?uid=ibm10738523
 
Fix Central
https://www-945.ibm.com/support/fixcentral/
 
Entitled Software
https://www-304.ibm.com/servers/eserver/ess/ProtectedServlet.wss
 
Upgrading the VIO server (non-SSP)
https://www.ibm.com/support/knowledgecenter/POWER9/p9hb1/p9hb1_vios_migrate_nonssp_cluster.htm
 
VIOS 3.1 Upgrade presentation handout (Raleigh User Group April 2019)
http://www.circle4.com/ptechu/vios31.pdf
 
Jaqui’s Presentations and handouts online
http://www.circle4.com/movies/
 
AIXPERT Blog
https://www.ibm.com/developerworks/community/blogs/aixpert?lang=en_us
 
PowerVM UK User Group – Nigel Griffiths Presentation on Upgrading to VIOS 3.1 in action
http://public.dhe.ibm.com/systems/power/community/aix/PowerVM_webinars/80_Upgrading_to_VIOS31_in_Action.pdf
http://public.dhe.ibm.com/systems/power/community/aix/PowerVM_webinars/80_Upgrading_to_VIOS31_in_Action.mp4
 
FLRT (Fix level reporting tool)
http://www14.software.ibm.com/webapp/set2/flrt/home
 

Jaqui Lynch is an independent consultant, focusing on enterprise architecture, performance and delivery on Power Systems with AIX and Linux.



Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.


comments powered by Disqus

Advertisement

Advertisement

2019 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.

AIX > ADMINISTRATOR > VIRTUALIZATION

A Closer Look at the IBM PowerVM 3.1 Update

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store
IBMi News Sign Up Today! Past News Letters