AIX > Trends > Cloud Computing

AIX Live Update in a PowerVC Managed Cloud

PowerVC

Starting with AIX Version 7.2, the AIX operating system provides the Live Update function that eliminates downtime required when patching the AIX operating system kernel. Your application workloads are not stopped, yet they can take advantage of the new fixes, immediately after the Live Update operation, without a reboot. Administrators can use the Live Update feature to install AIX interim fixes, service packs and technology levels.

In a nutshell, Live Update allows AIX administrators to install AIX updates in-place, but instead of reboot, we clone the root volume group (rootvg), boot a surrogate LPAR with the updated kernel, live migrate the processes to the new surrogate LPAR and then remove the original LPAR. If you’re not familiar with the Live Update (LU) concepts or would like to learn more about how it works, please read my blog on this topic. I encourage you to read this (and other related material on Live Update) before you attempt to implement LU.

Here’s how to live update your AIX system without rebooting the server.

If you’ve used LU before, you’ll know that there is a requirement for several spare disks (at a minimum two disks) to be available, on the original partition. This usually requires the AIX admin to perform several administrative storage steps prior to the operation. With AIX 7.2 TL2 we now can simplify this process by utilizing IBM PowerVC Virtualization Center (version 1.3.3.1).

PowerVC is an advanced virtualization and cloud management offering, built on OpenStack, that provides simplified virtualization management and cloud deployments for IBM AIX (and IBM i and Linux virtual machines) running on IBM Power Systems.

Leveraging this enhancement allows Live Update to use PowerVC, instead of the HMC, to create the surrogate LPAR. It also means that the storage, required by Live Update, will be provisioned automatically (without administrator intervention) by PowerVC. Also, using PowerVC, it is no longer necessary to pre-configure the lvupdate.data configuration file (still required in non-PowerVC HMC managed environments).

This new capability works with both HMC and NovaLink managed PowerVC Cloud environments, as shown in the following diagram.

Figure 1

For a complete list of the supported and required versions to support AIX Live Update in a PowerVC managed environment, please refer here

I’ll now demonstrate how to use this new capability. Before I start, first I confirm that my AIX and PowerVC systems are at the correct versions to support this operation.

PowerVC Management Server

[root@cgpvc ~]# cat /opt/ibm/powervc/version.properties
…
[1.3.3.1 Install]
name = IBM PowerVC
version = 1.3.3.1
build = 20170901-1542
install-date = 2017-09-13

AIX LPAR

root@orion / # oslevel -s
7200-02-01-1731

Please note, the AIX LPAR (named orion), is already managed by PowerVC. The LPAR must already be managed by PowerVC for the Live Update operation to succeed. I can view the LPAR details from the PowerVC UI.

Figure 2

The LPAR is configured with a single disk only (for rootvg), as shown in the lspv output from inside the LPAR.

root@orion / # lspv
hdisk0     00f94f587d8b0203           rootvg          active

I can also view the disk from the “Attached Volumes” tab under the VM details in PowerVC.

Figure 3

The lvupdate.data file has not been configured and does not exist prior to starting the LU operation.

root@orion / # ls -tlr /var/adm/ras/liveupdate/lvupdate.data
ls: 0653-341 The file /var/adm/ras/liveupdate/lvupdate.data does not exist.

To use this new capability, you first employ the new pvcauth tool on your AIX system to authenticate with the PowerVC management server. This command is used to obtain a token, required to use the PowerVC services for Live Update. In the example below, we have authenticated with a PowerVC server named cgpvc, using the PowerVC admin user named pvcadmin.

root@orion / # pvcauth -u pvcadmin -p abc1234 -a cgpvc
root@orion / # pvcauth -l
Address  : 10.1.50.232
User name: pvcadmin
Project  : ibm-default
Port     : 5000
TTL      : 5:59:58

Note: Before authenticating with PowerVC, the "Administrator" role should be assigned to the PowerVC user, otherwise the following message will be reported by Live Update: "1430-175 FAILED: User does not have PowerVC permissions (admin role) for Live Update processing”. You can review and change assigned roles from the PowerVC User Interface, as shown in the figure below.

Figure 4

With the token in place, next we perform a Live Update preview operation with geninstall –k –p. Note that the output below has been shortened for brevity. You’ll notice that, along with the usual checks, there are several new tests performed to ensure the PowerVC environment is ready for Live Update.

root@orion / # geninstall -k -p
Validating live update input data.

Computing the estimated time for the live update operation:
-------------------------------------------------------
LPAR: orion
Blackout time(in seconds): 12
Total operation time(in seconds): 1559
...
Checking lpar minimal memory size:
------------------------------------------
Required memory size: 2048 MB
...
Checking other requirements:
------------------------------------------
...
PASSED: PowerVC token is valid.
PASSED: PowerVC is at a supported level.
PASSED: User has PowerVC permissions for Live Update processing.
PASSED: Host is not in maintenance mode.
PASSED: PowerVC token expiration date is valid.
PASSED: PowerVC network devices match those present on partition.
PASSED: PowerVC volumes match hdisks present on partition.
...
INFO: Any system dumps present in the current dump logical

If the LPAR is not currently managed by PowerVC, you would see the following failure during the geninstall preview, ”1430-190 FAILED: Live Update initialization of PowerVC failed.”

After a successful preview, we can initiate the actual LU process now, using geninstall –k. Note, the output is identical to that of a “non-PowerVC enabled” Live Update operation. Again, the output has been shortened for brevity. We aren’t applying any AIX updates here; instead we are simply performing the Live Update operation, which will provision a new LPAR and migrate the workload, without updating AIX. This is a great way to test Live Update.

root@orion / # geninstall –k
…
Non-interruptable live update operation begins in 10 seconds.

Broadcast message from root@orion (pts/0) at 02:22:09 ...

Live AIX update in progress.

Initializing live update on original LPAR.

Validating original LPAR environment.

Beginning live update operation on original LPAR.

Requesting resources required for live update.
........................
Notifying applications of impending live update.
....
Creating rootvg for boot of surrogate.
................................................................................................
Starting the surrogate LPAR.
....................................................................................................
Creating mirror of original LPAR's rootvg.
....................................................................................................................................................................
Moving workload to surrogate LPAR.
............
        Blackout Time started.

        Blackout Time end.

Workload is running on surrogate LPAR.
....................................................................................................
Shutting down the Original LPAR.
....................................................................
The live update operation succeeded.

Broadcast message from root@orion (pts/0) at 02:47:20 ...
Live AIX update completed.

Monitoring the Live Update process, from the PowerVC UI we observe several events and actions taking place. For example, we notice the automatic creation and allocation of new disks. These disks are used by Live Update for creating and booting the surrogate partition. You’ll see something similar to the following, in the UI console.


New disk automatically assigned to original LPAR for creation of surrogate boot disk Figure 5

New disk automatically assigned for mirror of rootvg Figure 6

PowerVC Messages indicating the creation & attachment of new volumes for Live Update Figure 7


Before the Live Update process starts, we find only one Virtual Machine (VM), named orion (the original LPAR). But once the process starts, we eventually see another VM with the same name but different IP address. This is the surrogate LPAR, which will ultimately house the migrated workload from the original LPAR. The IP address, for the surrogate, is automatically assigned from the PowerVC IP pool. Once the operation completes we are left with only the new surrogate LPAR, which is re-assigned the same IP address as the original LPAR. We also observe that the VMs have different instance IDs in PowerVC.


Original LPAR (IP address 10.1.50.191) Figure 8

Original LPAR details (existing instance ID) Figure 9

PowerVC deploy of surrogate VM Figure 10

Surrogate LPAR deployed by PowerVC – in state of “Building”. Figure 11

Surrogate LPAR deployed with new IP address 10.1.50.192, from PowerVC IP Pool. Figure 12

Surrogate LPAR after workload migrated to it. Original IP assigned to surrogate VM. Figure 13

Surrogate LPAR details (new instance ID) Figure 14

Removal of original LPAR and cleanup of resources after Live Update Figure 15


The surrogate (initial/temporary) boot disk remains assigned to the VM, after the LU operation. It can be reused (or removed) after a reboot or another live update via a different disk in the VM. If you wish to remove the Live Update surrogate boot disk (lvup_rootvg) after a reboot, you can use the clvupdate command. This command will contact the PowerVC management server (requires a pvcauth token) and automatically remove the volume group and disk from the LPAR, as shown below.

; reboot LPAR
root@orion / # uptime
  04:42AM   up 1 mins,  1 user,  load average: 1.08, 0.40, 0.15

root@orion / # lspv
hdisk0          00f94f587d8b0203                    rootvg          active
hdisk1          00f94f584f60b1c1                    lvup_rootvg

root@orion / # pvcauth -u pvcadmin -p heathrow -a cgpvc
root@orion / # pvcauth -l
Address  : 10.1.50.232
User name: pvcadmin
Project  : ibm-default
Port     : 5000
TTL      : 5:59:53

root@orion / # clvupdate -v
The temporary Live Update storage was successfully removed.
Cleanup action succeeded.

root@orion / # lspv
hdisk0          00f94f587d8b0203                    rootvg          active

A message will appear in the PowerVC UI, indicating that the volume has been detached and deleted from the VM.

The lvup_rootvg disks removed after a reboot with clvupdate –v (requires a PowerVC token) Figure16

I should also note that you can monitor the status of the PowerVC VMs and storage volumes using the OpenStack nova and cinder commands from the PowerVC management server. For example:

List VMs

[root@cgpvc ~]# nova list | grep -i orion
| a1c908b0-70c1-4e25-8762-57c3b3eba173 | orion | ACTIVE | - | Running | VLAN50=10.1.50.191 |
| a1c908b0-70c1-4e25-8762-57c3b3eba173 | orion | ACTIVE | - | Running | VLAN50=10.1.50.192 |

List Volumes

[root@cgpvc ~]# cinder --service-type volume list | grep orion
| 4bd87b45-7bad-40a0-8dbd-e593d075005e | in-use    | volume-orion-07dfdb3e-00000016-boot-0-2aaf7ffb-da75 | 20   | - | false 
| a1c908b0-70c1-4e25-8762-57c3b3eba173 |

While Live Update might still be new to many AIX administrators, it promises to be a much used and lauded feature. The integration with PowerVC will only help to further simplify and automate an already innovative process. For customers that have already started managing their private clouds using IBM PowerVC, this is another welcome extension to automating their AIX cloud environments.

Chris Gibson is an AIX and PowerVM specialist located in Melbourne, Australia. He is an IBM Champion for Power Systems, IBM CATE (Power Systems and AIX), and a co-author of several IBM Redbooks publications.



Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.


comments powered by Disqus

Advertisement

Advertisement

2018 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store
IBMi News Sign Up Today! Past News Letters
not mf or hp