AIX > Administrator > Systems Management

Architecting Power Systems for I/O

Architecting I/O

Whenever I architect Power Systems infrastructure for a client, I do two things. The first is to ensure the backplane on the server (where offered) is split. This gives the client more flexibility should they want to run two VIO servers or LPARs booting from an internal disk. The cost is minimal and it is much easier to do it upfront than to try and back-port it. But additionally, I pay attention to where the adapters get installed in the server and I/O drawers. Architecting the system correctly upfront saves a significant amount of time, money and effort. It also ensures that you don’t run into issues where adapters require certain slots that other adapters may also require.

Whenever you purchase a new system you should always download two manuals: The Technical Overview Redbook and the PCI Adapter Placement Guide. These two manuals provide a wealth of information on the server and on the I/O connections. Using the E850C as an example, I will look at designing for I/O below. Remember, that for every high-performance port that is active (anything 8GB and above), you will need to add extra memory to the system. If using NPIV there is additional memory needed for every VFC (virtual fibre client). Details on this are provided in the VIO server presentation in the references.

CEC (The Server Itself)

Processor Modules		PCIe X16 slots	PCIe x8 slots
2				4			3
3				6			3
4				8			3

The slot numbering is not intuitively obvious. In documentation IBM sometimes refers to the slot numbers and slot addresses interchangeably. The table below provides both for translation purposes:

Module	Slot No	Slot Address		Type	
0		9		C10			x16
0		10		C11			x8
0		11		C12			x16
1		5		C6			x8
1		6		C7			x8
1		7		C8			x16
1		8		C9			x16

2		3		C3			x16
2		4		C4			x16

3		1		C1			x16
3		2		C2			x16

At least one PCIe ethernet adapter must be installed in the CEC. The slot identified for this is C11 (aka slot 10). Slots C6 and C7 share bandwidth and are also shared with the USB controller. For every adapter, there is a slot priority list provided in the adapter placement guide. The best performance is obtained for an adapter by ensuring it is in the highest priority adapter slot. Redundancy across processor modules can be obtained by putting adapters in the highest priority slot in two separate modules.

As an example: In the E850C the EN0A 16GB 2 port fibre card lists the following slot priority for a four-module server: 9, 7, 4, 2, 11, 8, 3, 1, 5, 6. So, with two cards, the best performance and redundancy would be obtained by putting one card in slot 9 (C10) and the second in slot 7 (C8). I normally map out all the cards based on which are the highest priority. This depends on workload—for some workloads the network cards are more important.

Slot assignment is also affected by external I/O drawers and the specific cards selected to support those drawers. As an example, the EJ14 RAID card that is used to connect to the external disk drawer is only supported in slots 9, 7, 4, 2 in the CEC. They are supported in different slots if you are using a PCIe3 adapter drawer. Gathering this information ahead of time and mapping it out in a spreadsheet can ensure a much smoother installation and implementation.

PCIe3 I/O Drawer

The EMX0 PCIe3 adapter I/O drawer is typically connected using two EJ08 cable adapters. These can go in any slot in the CEC, however the priority order is 9, 7, 3, 1, 11, 8, 4, 2. I would put these in slots 9 and 7 to provide the best performance for the adapters in the I/O drawer. If you are adding an EXP24S disk drawer you may need to use different slots for this drawer as the disk drawer has more limited options. The EMX0 drawer consists of the drawer itself plus one or two fanout modules. Each fanout module provides six PCIe3 slots as follows:

Fanout		Slot	Slot Address	Type
1		1	P1-C1		x16
1		2	P1-C2		x8
1		3	P1-C3		x8
1		4	P1-C4		x16
1		5	P1-C5		x8
1		6	P1-C6		x8

2		1	P2-C1		x16
2		2	P2-C2		x8
2		3	P2-C3		x8
2		4	P2-C4		x16
2		5	P2-C5		x8
2		6	P2-C6		x8



Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.


comments powered by Disqus

Advertisement

Advertisement

2017 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.

AIX > ADMINISTRATOR > SYSTEMS MANAGEMENT

How to Download Fixes

Drawer Configuration

Understand your options for 12X PCIe I/O drawers

clmgr: A Technical Reference

PowerHA SystemMirror 7.1 introduces a robust CLI utility

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store
AIX News Sign Up Today! Past News Letters