AIX > Administrator > LPAR

Implementing a Three-Node Redundant GPFS Cluster


Working on LPAR gpfs1, create a file /usr/local/etc/gpfs-nodesinit.txt that contains:

gpfs1.abc.com:quorum-manager
gpfs2.abc.com:quorum-manager
gpfs3.abc.com:quorum-manager

Now create the cluster with gpfs1 as primary and gpfs2 as secondary.

mmcrcluster -C GPFSCLUST1 -p gpfs1.abc.com -s gpfs2.abc.com -r /usr/bin/ssh -R /usr/bin/scp -N /usr/local/etc/gpfs-nodesinit.txt -A

Then you must accept the licenses.

mmchlicense server --accept -N gpfs1.abc.com,gpfs2.abc.com,gpfs3.abc.com

I normally check the cluster at this point using mmlscluster and mmlsconfig.

Now, create the network-shared disks (NSDs). In this case, we plan to use two failure groups. The hdisks from subsystem A will be in failure group 2 and those from subsystem B will be in failure group 3. We create a file /usr/local/etc/gpfsdisks.txt that contains:

%nsd:  nsd=vahdisk1 device=/dev/hdisk1 usage=dataAndMetadata failuregroup=2 
%nsd:  nsd=vahdisk2 device=/dev/hdisk2 usage=dataAndMetadata failuregroup=2 
%nsd:  nsd=vbhdisk3 device=/dev/hdisk3 usage=dataAndMetadata failuregroup=3 
%nsd:  nsd=vbhdisk3 device=/dev/hdisk4 usage=dataAndMetadata failuregroup=3

This defines the NSDs and names them. I made the names match the hdisk names.

On gpfs1, run the following to create the NSDs:

mmcrnsd -F /usr/local/etc/gpfsdisks.txt

You can now run lspv and mmlspv to see how the disks are mapped. At this point, you can start GPFS on each node from gpfs1 using mmstartup –a. mmlsnsd will show you the NSD mappings.

Final steps include creating the filesystem. Create /gpfsa with a blocksize of 512 across the two disks in failure group 2 and replicated across the two disks in failure group 3:

mmcrfs /gpfsa  gpfsa -F /usr/local/etc/gpfsdisks.txt -m2 -M2 -r 2 -R 2

Note: -m2 says two replicas for metadata, -M2 says max of two metadata replicas, -r2 says two replicas for data, -R2 says max of two data replicas.

On gpfs1, run mmmount all –a. This causes the filesystem to mount on all the nodes. You can see it with df or mmdf /gpfsa and should now be able to create and delete files within the filesystem. Because we’re using replicas, a du –sg will show the filesystem as twice the size it really is but an ls –al will show the real size that you’re using. The df will also show the total space across all the disks but the %used will be correct.

Testing, Testing

At this point, you’re ready to test redundancy by unzoning or disconnecting some of the disks.

To support network-based client nodes, convert your current NSDs to support network connectivity. The default order of access will be local block interfaces for SAN and then NSD servers. So the first step is to unmount the filesystem and convert the current primary and secondary NSDs to support network connectivity:

mmumount gpfsa -a
mmchnsd "vahdisk1:gpfs1.abc.com,gpfs2.abc.com,gpfs3.abc.com"
mmchnsd "vahdisk2:gpfs1.abc.com,gpfs2.abc.com,gpfs3.abc.com"
mmchnsd "vbhdisk3:gpfs1.abc.com,gpfs2.abc.com,gpfs3.abc.com"
mmchnsd "vbhdisk4:gpfs1.abc.com,gpfs2.abc.com,gpfs3.abc.com"

mmlsnsd will show the changes and you can now remount the filesystem from gpfs1 using mmmount gpfsa –a.

Add network client nodes by installing GPFS on the clients, setting up ssh and licensing them as clients. You’ll need to update the gpfs-nodes.txt file on all the nodes. Adding a network client (gpfs4) is done from gpfs1 as follows:

mmaddnode –N gpfs4.abc.com:client
mmchlicense client --accept -N gpfs4.abc.com

On the client, run mmstartup and then set the client up to mount gpfsa as network only:

mmmount gpfsa -o useNSDserver=always

This causes it to always expect to do a network mount.

A Simple Alternative

This is a fairly simple implementation but it does provide redundancy and is very durable. GPFS is a great way to provide shared disks in a redundant fashion with better performance than that provided by NFS. It’s well worth testing out as an alternative.

Jaqui Lynch is an independent consultant, focusing on enterprise architecture, performance and delivery on Power Systems with AIX and Linux.


comments powered by Disqus

Advertisement

Advertisement

2019 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store
IBMi News Sign Up Today! Past News Letters