Skip to main content

2019 AIX Systems Management Tips

AIX expert Jaqui Lynch runs through tips and tricks she's learned from conferences and reading.

Blue and green illustrated icon of lightening bolts on white background.

2019 has been a busy year but I have still been able to attend a couple of conferences as well as catching up on my reading. During that time, I picked up a few tips and tricks that may be helpful to you.
 

loopmount to access media

The first useful command, which I have used for some time now, is loopmount. This lets you mount an iso image file as if it was a local DVD. Here’s an example:
 
On the NFS client:
Assuming the DVD ripped was the AIX v7 DVD ½ and is now called aixv7-base-1of2.iso you cd into the directory it is in and:

 
mkdir /cdrom
loopmount –i aixv7-base-1of2.iso  –o “-V cdrfs –o ro” -m /cdrom

The above mounts the AIX 7 base iso as a filesystem called /cdrom. You can now create an lpp_source or spot from the iso or you can simply read the files and process them.

If you are on the latest version of AIX (7.2.3) you can also mount the flash image for the VIO server as follows:

loopmount –i vios31021-flash-image.iso  –o “-V udfs –o ro” -m /cdrom

HMC Scanner

I've written up HMC Scanner before but it's such a critical tool that I want to mention it again. I also want to note that the latest level is v42. This level is required for accessing HMCs that are at v9r1m930 or higher but can also be used for older levels of the HMC. The HMC Scanner is a Java based tool that connects to your HMC and documents everything on it that the HMC can see. It will also connect to an FSM (Flex Manager) and document the PureFlex environment. It’s easy to install and can be run from Windows or AIX. On Windows I change into the directory and type in:

hmcScanner.bat  hmcname hscroot -p password

You substitute your HMC name for hmcname and the correct username and password.

The end result is a spreadsheet organized in tabs that documents the HMC, servers, LPARs, physical slots, virtual ethernet and SEAs, virtual SCSI configuration and mappings, virtual fiber mappings, LPAR and virtual processor pool configurations. 

After I build a VIO server or make any changes I always run this to redocument the environment. It also helps me check that the SEA and aggregates were built correctly.

FixCentral and Software Updates

IBM changed the environment for fix Central and moved it from their fixed servers into the cloud. This has led to significantly faster downloads.

To get Java patches you go to Fix Central, then click on find product and type in java in the product selector prompt. Then select Runtimes for Java Technology. After that it will give you choices regarding version, 32 or 64 bit, etc. You will need to download 32 and 64 bit separately. These updates are installed with either install (AIX) or updateios (VIO server).

To get the latest SSH and SSL patches you will need to visit this website (which requires an IBM login).

You select either OpenSSH or OpenSSL and then continue and it takes you to the latest downloads.  

Finally, products for the Linux tool box can be found here.

Note that the levels of ssh, ssl and java that are on the current AIX and PowerVM DVDs have security holes in them so they should be updated to the latest levels.

Fix Level Recommendation Tool (FLRT)

There were a couple of changes made to FLRT. The first was in the HMC checking. When running FLRT it asks you for your HMC level but it then also asks you for the type and model of HMC that you have. This means they now also check if your hardware needs to be replaced for the HMC. They also added a version timeline so you can see when versions of software, etc were released and when they go out of service.

FLRTVC is the associated vulnerability checker that you run on your system. It’s constantly being updated to ensure that known problems can be addressed as quickly as possible.

Dealing With Missing Paths; reusing WWPNs

Recently I had to migrate an LPAR without LPM to a new server. I learned two things during this process:

  1. How to reuse the old WWPNs so we did not have to rezone everything
  2. How to deal with the missing paths due to adapter number changes.

In order to the use the old WWPNs I ran an HMCScanner report which showed me the NPIV WWPNs for the LPAR. I then created a profile on the new server with everything except the two virtual fiber adapters that I wanted to use. I then added them both as follows:

chsyscfg -r prof -m Servername  -i "name=default,lpar_id=6,\"virtual_fc_adapters=\"\"3/client/2/vio1/12/c0507604d635000c,c0507604d635000d/0\"\",\"\"5/client/3/vio2/12/c0507604d635000a,c0507604d635000b/0\"\"\""

The order for adapters is: client adapter id, client (says it’s a client), VIO LPAR  id, VIO name, VIO adapter id, WWPN1 and 2 for that adapter.

The above added the default and LPM WWPN as adapter 3 on the client as vio1 adapter 12. It added the vio2 WWPNs as adapter 5 on the client as vio2 adapter 12.

After the profile was completely built, we shut down the old LPAR and booted it on the new server with no issues. However, lspath showed we had the correct number of paths but there were a bunch of missing paths. The following script identifies missing paths and builds a remove script for them:

vi badpaths.sh
#!/bin/ksh
# badpaths - create cleanup scripts
>removepaths.sh
disks=$(lspv | awk '{print $1}')
for loop in $disks
do
lspath -l $loop -H -F "name:parent:connection:status" |grep Missing| awk -F: '{print "rmpath -dl",$1,"-p", $2, "-w", $3}'>>removepaths.sh
done
exit 0

If you replace Missing in the above with Defined it will also show you the paths that have fallen into a defined state.

You can check an individual disk as follows:

lspath -l hdisk15 -H -F "name:parent:connection:status" |grep Missing

The script creates a new script called removepaths.sh.  You should check the contents and, once you are happy with it, you can then make it executable and it will clean up your paths. Here’s a link to the original article where I found this.

LPM Tips

If you are using IBM i with NPIV and you want to use LPM then you have to set your IBM i LPAR into restricted I/O mode. This means the LPAR can have no direct attached adapters. It’s very common in the IBM i world to use tape drives that are DLPAR’d in and out of the various LPARs. You’ll need to switch these tapes over to NPIV as they will no longer be able to be attached to the LPAR if it is in restricted I/O mode. This also means that those tape drives are no longer available to your VIO servers to be used for backups there so you will need to find an alternative method for backing them up (NIM or an extra tape drive on a different adapter, or to an NFS share). 

Backing up and restoring VIO servers

Today, it’s possible to install your VIO server from the HMC, from NIM, from DVD and from a USB stick. Because of this, many of us thought that meant you could back it up to a USB stick and restore it from one. The IBM website is clear that this isn’t supported.  Just in case it was out of date, I opened a case with IBM and they were very clear that backup and restore of VIO servers to USB sticks is not supported at this time. You can restore from tape, DVD, NIM, NFS and from the HMC.

My second question was about support for aggregated networks. I’d noticed that I could never install my VIO server if the network ports were set up as aggregated. I checked with IBM and they informed me that any network-based install or restore does not support aggregated ports—they only work over access ports (regular unaggregated ports). This applies not just to installing your VIO from the HMC or NIM, but also to installing or restoring AIX over the network as well. My solution so far has been to do one of the following:

  1. Install my VIO or AIX and then aggregate the ports after the install
  2. Have a second management network with a single access port and use it for installs and restores

If you do aggregate after the installation keep in mind that if you need to restore later the switch port will need to be changed to an access port (removed from the aggregate) for the restores.

Gareth Coate’s Tips and Tricks Session

I picked up some good tips from this session and wanted to share them with you. There are many more tips in his presentation than the ones I share below so please be sure to download it from the Virtual User Group website.

Systems

The first tip is how to list all the servers that the HMC sees:

lssyscfg -r sys -F name

WWPNs

The second is how to list all the WWPNs on the AIX system: 

lsdev -Cc adapter|grep fcs|awk '{print $1,$3}'|while read fcs slot
do
wwpn=` lscfg -vl $fcs | grep -i network | cut -c 37-60`
wwpn1=` lscfg -vl $fcs | grep -i Hardware | cut -c 47-64`
echo $fcs $slot $wwpn1 $wwpn
done

You can also get similar information on WWPNs from the HMC:

lshwres -r io --rsubtype slotchildren -m Systemname –F phys_loc,description,mac_address,wwpn,microcode_version | grep Fibre
lshwres -r io --rsubtype slotchildren -m Systemname -F phys_loc,description,mac_address,wwpn | grep PCI

Replace systemname above with the actual server name on the HMC

Checking Memory Region Size

If you are planning to use LPM then it is required that you have the LMB (also called memory region size) set to the same on the LPARs.  You can check them all on the HMC by using the following:

To check it for all systems on the HMC

lssyscfg -r sys -F name | while read M
> do
> echo -e $M\\t\\c
> lshwres -r mem -m $M  -- level sys -F mem_region_size
> done


More on WWPNs

When you go to zone WWPNs at the switch or map them at the storage the storage administrators typically want them to be lit up so they can see them. If the LPAR is not up or is going to boot from SAN then those WWPNs won’t be alive. Additionally, the second WWPN that is used for LPM won’t be lit up even if the LPAR is up.


There are a couple of options for lighting these up:

  • Login to the HMC and use the lsnportlogin and chnportlogin commands
lsnportlogin -m Servername --filter "profile_names=normal"

Login the WWPNs via the HMC GUI

  • Edit the LPAR profile, select the virtual fibre adapters, click actions, then advanced, then login or logout fibre channel

You will see one of three status’ for the WWPN:

0 - WWPN is not activated (it is logged out)

1 - WWPN is activated (it is logged in)

2 - WWPN status is unknown

New POWER HMC

The new POWER HMC (7063-cr1) doesn’t have a DVD drive; it also requires three network connections. The first connection is to the public network, which is how you access it via SSH or HTTPS to perform HMC functions. The second connection is to the private network that connects to the service processors on the attached servers. The third connection is the new one—this goes on the same public network and requires an IP address. This connects the BMC (base management controller) so that the actual firmware on the HMC can be updated, even if the HMC is shutdown. It also allows you to power the HMC on and off remotely. If you connect the public IP to ETH0 on the server and the private network to one of the other ports (ETH1, 2 or 3) then you can configure your BMC in failover mode. It will still need an IP, but it will not need a separate physical connection. Even if the HMC is shutdown you should be able to connect to the BMC using https://bmc-ip

Miscellaneous Tips

You can always backup your VIO lpar or AIX LPAR by making a clone as follows;

Assume rootvg is on hdisk0 and hdisk1 is free:

alt_disk_copy -B -d hdisk1

The above copied the running rootvg to hdisk1 and leaves the bootlist as is.  If you leave out the -B it will set the bootlist to the new copy.

On the vio as padmin

lspv  -free

The above lists all the disks on the VIO LPAR and whether they are available or not.  Even if they are not in a volume group they may not be free.  Typically I will exportvg my clone (altinst_rootvg) so I can do an upgrade but it does not clear the owning flag. If I'm absolutely certain this is the case then you can clear the owner ship using:

chpv -C hdisk?

You can also clear the boot record from that disk using:

chpv -c hdisk?
lspv -size

The above shows the size for each of the disks.

Rebooting the VIO that is the Primary on the SEA

If rebooting the VIO that is in the PRIMARY state, it is recommended to failover to the BACKUP state prior to performing any maintenance or reboot. Otherwise, the connectivity of the client partitions that are using the SEA in PRIMARY state may lose connectivity temporarily.  This is done as follows:

As root on the primary VIO:

# chdev -l entX -a ha_mode=standby 
Or as padmin on the primary VIO:
$ chdev -dev entX -attr ha_mode=standby 

After the primary VIO is back up reset it back to auto and the SEA should fail back to the primary VIOS: 

# chdev -l entX -a ha_mode=auto 

or 

$ chdev -dev entX -attr ha_mode=auto

As time goes by we all pick up lots of tips and tricks that we include in our administration arsenal. I have many more of these in my useful things list but thought these would be helpful ones to share.

IBM Systems Webinar Icon

View upcoming and on-demand (IBM Z, IBM i, AIX, Power Systems) webinars.
Register now →