https://docs.oracle.com/cd/E18476_01/doc.220/e18478/GUID-070B1E3D-38F2-41E3-BD12-A0D505A67654.htm

Partition Key (P_Key)

Definition of a Partition

A partition is defined as a collection of CA, router, and switch ports that are permitted to communicate with one another. A port may be a member of multiple partitions simultaneously.

Who Enrolls a Port in a Partition?

Refer to Figure 16-1 on page 322. A partition is represented by a 15-bit partition ID. Theoretically, this permits up to 32K partitions to be set up. Partition values assigned to a port are stored in the port's P_KeyTable attribute. The attribute consists of a series of P_Key entries.

Figure 16-1. P_Key Format

演示环境介绍

Subnet Manager节点 ： Node IO
无Partition节点： H3，H4
Partition pz节点： H1，H2
如下图

OpenSM节点Partition配置

修改parition.conf配置文件后，需要重启opensm进程，使配置生效

Partition.conf配置文件

写法1

[root@IO ~]# cat /etc/opensm/partitions.conf
pz=0x1234,indx0,ipoib,defmember=full:0xe41d2d0300cac9f6,0xe41d2d0300cac9f7,0xe41d2d0300cac9fa ;

写法2

[root@IO ~]# cat /etc/opensm/partitions.conf
pz=0x1234,indx0,ipoib:0xe41d2d0300cac9f6=full,0xe41d2d0300cac9f7=full,0xe41d2d0300cac9fa=full ;

节点间连通性

默认分区

在partitions.conf存在的时候，隐含以下default partition配置

Default=0x7fff,indx0,ipoib:ALL=limited,SELF=full ;

• Default partition: pkey=0x7fff
• OpenSM节点默认权限为full
• 在partitions.conf存在的时候，默认分区的成员权限为limited
  使得：

1. H3间H4不通
2. IO到H3/4互通

PZ分区

通过以下条目明文配置

pz=0x1234,indx0,ipoib,defmember=full:0xe41d2d0300cac9f6,0xe41d2d0300cac9f7,0xe41d2d0300cac9fa ;

• pz partition: pkey=0x1234
• 因为配置indx0，使得pkey table中index为0的默认pkey为pz partition的pkey 0x1234
• 定义三个full成员，分别是H1的两个端口，和H2的端口的GUID
  使得：

1. H1到H2间互通
2. H1/2到H3/4不通
3. IO到H1/2不通，因为H1/2默认用0x1234了，而IO用的是SM SELF 0x7fff

PZ分区到OpenSM 节点通过PKEY 0x7fff通信

PZ分区成员可以通过Default partition的pkey和Subnet Manager也就是IO节点通信

查询PKEY INDEX

查询PZ分区成员的默认pkey 0x7fff，看到7fff被分配在table中第19项

[root@h2 ~]# grep 7fff /sys/class/infiniband/mlx5_0/ports/1/pkeys/ -r
/sys/class/infiniband/mlx5_0/ports/1/pkeys/19:0x7fff
[root@h2 ~]# smpquery pkeytable 10
   0: 0x9888 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
   8: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  16: 0x0000 0x0000 0x0000 0x7fff 0x0000 0x0000 0x0000 0x0000
  24: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  32: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  40: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  48: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  56: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  64: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  72: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  80: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  88: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
  96: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
 104: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
 112: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
 120: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
128 pkeys capacity for this port

用PKEY 0x7fff连接并测试ib_write_bw

Subnet Manager节点

[root@IO ~]# ib_write_bw 

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF      Device         : mlx4_0
 Number of qps   : 1        Transport type : IB
 Connection type : RC       Using SRQ      : OFF
 CQ Moderation   : 100
 Mtu             : 2048[B]
 Link type       : IB
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x01 QPN 0x02ad PSN 0x686c48 RKey 0x30010200 VAddr 0x007fd305410000
 remote address: LID 0x0a QPN 0x00b5 PSN 0xecb0d6 RKey 0x0137a9 VAddr 0x007ff0caf50000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      5000             6152.22            6152.04        0.098433
---------------------------------------------------------------------------------------

H2节点，ib_write_bw增加--pkey_index=19参数，指定pkey

[root@h2 ~]# ib_write_bw -d mlx5_0 io --pkey_index=19
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF      Device         : mlx5_0
 Number of qps   : 1        Transport type : IB
 Connection type : RC       Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 2048[B]
 Link type       : IB
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x0a QPN 0x00b5 PSN 0xecb0d6 RKey 0x0137a9 VAddr 0x007ff0caf50000
 remote address: LID 0x01 QPN 0x02ad PSN 0x686c48 RKey 0x30010200 VAddr 0x007fd305410000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 1200.000000 != 2299.951000. CPU Frequency is not max.
 65536      5000             6152.22            6152.04        0.098433
---------------------------------------------------------------------------------------

PZ分区上的InfiniBand端口绑定

配置文件

[root@h1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ib0
DEVICE=ib0
TYPE=InfiniBand
NM_CONTROLLED=no
ONBOOT=yes
MASTER=bbbbond0
SLAVE=yes
BOOTPROTO=none
PRIMARY=no
[root@h1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ib1
DEVICE=ib1
TYPE=InfiniBand
NM_CONTROLLED=no
ONBOOT=yes
MASTER=bbbbond0
SLAVE=yes
BOOTPROTO=none
PRIMARY=yes
[root@h1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bbbbond0 
DEVICE=bbbbond0
IPADDR=66.66.66.80
NETMASK=255.255.255.0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
NM_CONTROLLED=no
BONDING_OPTS="mode=active-backup primary=ib0 miimon=100 updelay=100 downdelay=100"
MTU=2044

配置验证

[root@h1 ~]# cat /proc/net/bonding/bbbbond0 
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup) (fail_over_mac active)
Primary Slave: None
Currently Active Slave: ib1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 100
Down Delay (ms): 100

Slave Interface: ib1
MII Status: up
Speed: 56000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 20:00:18:8d:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:ca:c9:f7
Slave queue ID: 0