Aruba DRNI+RDMA Configuration Guide

Contents

Example: Deploying DRNI and RDMA ··························································· 1

Network configuration ········································································································································ 1

Connectivity models ··········································································································································· 4

Applicable product matrix ··································································································································· 4

Restrictions and guidelines ································································································································ 5

Configuring 5945 switches as leaf nodes ·········································································································· 5

Procedure summary ··································································································································· 5

Configuring global RoCE settings ·············································································································· 6

Configuring the links towards the spine tier ····························································································· 14

Configuring RoCE settings for the links towards the spine tier ································································ 15

Configuring DRNI ····································································································································· 18

Configuring RoCE settings for the IPL ····································································································· 20

Configuring the links towards the servers ································································································ 22

Configuring RoCE settings for the links towards the servers ··································································· 24

Configuring an underlay BGP instance ···································································································· 27

Configuring VLAN interfaces and gateways (dual-active gateways)························································ 28

Configuring VLAN interfaces and gateways (VRRP gateways) ······························································· 29

Configuring spine devices ································································································································ 29

Spine device configuration tasks at a glance ··························································································· 29

Configuring global RoCE settings ············································································································ 30

Configuring the links towards the leaf tier ································································································ 32

Configuring RoCE settings for the links towards the leaf tier ··································································· 32

Configuring routing policies to replace the original AS numbers with the local AS number ····················· 35

Configuring an underlay BGP instance ···································································································· 36

Influence of adding or deleting commands on traffic ······················································································· 37

Traffic model ···················································································································································· 39

Convergence performance test results ············································································································ 40

Failure test results ···································································································································· 40

Verifying the configuration································································································································ 40

Verification commands ····························································································································· 40

Procedure ················································································································································· 41

Tuning the parameters ····································································································································· 43

Recommended PFC settings ··················································································································· 43

Recommended ECN settings ··················································································································· 44

NIC settings for reference ························································································································ 44

Guidelines for tuning parameters ····················································································································· 45

Restrictions and guidelines ······················································································································ 45

Identifying whether packets are dropped ································································································· 46

Identifying whether the latency meets the requirements ·········································································· 47

Upgrading the devices ····································································································································· 49

Upgrading a leaf device ··························································································································· 49

Upgrading a spine device ························································································································· 50

Expanding the network····································································································································· 51

Adding a leaf device ································································································································· 51

Replacing hardware ········································································································································· 51

Replacing an interface module················································································································· 51

Replacing a switching fabric module ········································································································ 52

1

Example: Deploying DRNI and RDMA

Network configuration

As shown in Figure 1, build a Layer 2 network within the data center to implement traffic forwarding

across leaf devices and spine devices.

The following is the network configuration:

•

Deploy the leaf devices as access switches. Use DRNI to build two pairs of access switches as

DR systems. The DR interfaces in a DR system form a multichassis aggregate link for

improved link usage. Configure dual-active-gateway or VRRP for the leaf devices to provide

connectivity for servers.

•

Deploy the spine devices as aggregation switches. Configure them as route reflectors (RRs) to

reflect BGP routes among leaf devices.

•

Configure intelligent lossless network features such as PFC and data buffer to provide zero

packet loss for RDMA. In this example, zero packet loss is implemented for packets with

802.1p priority 3.

•

Configure DHCP relay on leaf devices to help servers obtain IP addresses.

Figure 1 Network diagram

Device

Interface

IP address

Remarks

Leaf 1

WGE1/0/1 N/A Member port of a DR interface.

Connected to Server 1.

WGE1/0/2 N/A Member port of a DR interface.

Connected to Server 2.

HGE1/0/29 N/A Member port of the IPP.

Connected to HGE 1/0/29 on Leaf 2.

HGE1/0/30 N/A

Member port of the IPP.

Spine 1 Spine 2

Leaf 1 Leaf 2 Leaf 3 Leaf 4

Server 1 Server 2 Server 3 Server 4

HGE1/1/1~HGE1/1/4 HGE1/1/1~HGE1/1/4

Keepalive Keepalive

Peer-link Peer-link

BAGG 100

WGE1/0/1

WGE1/0/2

WGE1/0/1

HGE1/0/25

HGE1/0/26

HGE1/0/25

HGE1/0/26

WGE1/0/1

WGE1/0/2

WGE1/0/1

HGE1/0/25

HGE1/0/26

HGE1/0/25

HGE1/0/26

Access layer

Aggregation

layer

Master Backup

Backup Master

IPL IPL

2

Device

Interface

IP address

Remarks

Connected to HGE 1/0/30 on Leaf 2.

HGE1/0/31 N/A Member port of the IPP.

Connected to HGE 1/0/31 on Leaf 2.

HGE1/0/32 N/A Member port of the IPP.

Connected to HGE 1/0/32 on Leaf 2.

WGE1/0/55 RAGG 1000:

1.1.1.1/30

Keepalive link between DR member

devices.

Connected to WGE 1/0/55 on Leaf 2.

WGE1/0/56 Keepalive link between DR member

devices.

Connected to WGE 1/0/56 on Leaf 2.

HGE1/0/25 172.16.2.154/30 Connected to WGE 1/1/1 on Spine 1.

HGE1/0/26 172.16.3.154/30 Connected to WGE 1/1/1 on Spine 2.

Loopback1 50.50.255.41/32 N/A

Leaf 2

WGE1/0/1 N/A Member port of a DR interface.

Connected to Server 1.

WGE1/0/2 N/A Member port of a DR interface.

Connected to Server 2.

HGE1/0/29 N/A Member port of the IPP.

Connected to HGE 1/0/29 on Leaf 1.

HGE1/0/30 N/A Member port of the IPP.

Connected to HGE 1/0/30 on Leaf 1.

HGE1/0/31 N/A Member port of the IPP.

Connected to HGE 1/0/31 on Leaf 1.

HGE1/0/32 N/A Member port of the IPP.

Connected to HGE 1/0/32 on Leaf 1.

WGE1/0/55 RAGG 1000:

1.1.1.2/30

Keepalive link between DR member

devices.

Connected to WGE 1/0/55 on Leaf 1.

WGE1/0/56 Keepalive link between DR member

devices.

Connected to WGE 1/0/56 on Leaf 1.

HGE1/0/25 172.16.2.158/30 Connected to WGE 1/1/2 on Spine 1.

HGE1/0/26 172.16.3.158/30 Connected to WGE 1/1/2 on Spine 2.

Loopback1 50.50.255.42/32 N/A

Leaf 3

WGE1/0/1 N/A

Interface connected to a

single-homed device.

Connected Server 3.

WGE1/0/2 N/A

Interface connected to a

single-homed device.

Connected Server 4.

HGE1/0/29 N/A Member port of the IPP.

Connected to HGE 1/0/29 on Leaf 4.

3

Device

Interface

IP address

Remarks

HGE1/0/30 N/A Member port of the IPP.

Connected to HGE 1/0/30 on Leaf 4.

HGE1/0/31 N/A Member port of the IPP.

Connected to HGE 1/0/31 on Leaf 4.

HGE1/0/32 N/A Member port of the IPP.

Connected to HGE 1/0/32 on Leaf 4.

WGE1/0/55 RAGG 1000:

1.1.1.1/30

Keepalive link between DR member

devices.

Connected to WGE 1/0/55 on Leaf 1.

WGE1/0/56 Keepalive link between DR member

devices.

Connected to WGE 1/0/56 on Leaf 1.

HGE1/0/25 172.16.2.82/30 Connected to WGE 1/1/2 on Spine 1.

HGE1/0/26 172.16.3.82/30 Connected to WGE 1/1/2 on Spine 2.

Loopback1 50.50.255.23/32 N/A

Leaf 4

WGE1/0/1 N/A

Interface connected to a

single-homed device.

Connected Server 3.

WGE1/0/2 N/A

Interface connected to a

single-homed device.

Connected Server 4.

HGE1/0/29 N/A Member port of the IPP.

Connected to HGE 1/0/29 on Leaf 3.

HGE1/0/30 N/A Member port of the IPP.

Connected to HGE 1/0/30 on Leaf 3.

HGE1/0/31 N/A Member port of the IPP.

Connected to HGE 1/0/31 on Leaf 3.

HGE1/0/32 N/A Member port of the IPP.

Connected to HGE 1/0/32 on Leaf 3.

WGE1/0/55 RAGG 1000:

1.1.1.2/30

Keepalive link between DR member

devices.

Connected to WGE 1/0/55 on Leaf 1.

WGE1/0/56 Keepalive link between DR member

devices.

Connected to WGE 1/0/56 on Leaf 1.

HGE1/0/25 172.16.2.86/30 Connected to WGE 1/1/2 on Spine 1.

HGE1/0/26 172.16.3.86/30 Connected to WGE 1/1/2 on Spine 2.

Loopback1 50.50.255.24/32 N/A

Spine 1

HGE1/1/1 172.16.2.153/30 Connected to HGE 1/0/25 on Leaf 1.

HGE1/1/2 172.16.2.157/30 Connected to HGE 1/0/25 on Leaf 2.

HGE1/1/3 172.16.2.81/30 Connected to HGE 1/0/25 on Leaf 3.

HGE1/1/4 172.16.2.85/30 Connected to HGE 1/0/25 on Leaf 4.

4

Device

Interface

IP address

Remarks

Loopback1 50.50.255.1/32 N/A

Spine 2

HGE1/1/1 172.16.3.153/30 Connected to HGE 1/0/26 on Leaf 1.

HGE1/1/2 172.16.3.157/30 Connected to HGE 1/0/26 on Leaf 2.

HGE1/1/3 172.16.3.81/30 Connected to HGE 1/0/26 on Leaf 3.

HGE1/1/4 172.16.3.85/30 Connected to HGE 1/0/26 on Leaf 4.

Loopback1 50.50.255.2/32 N/A

Connectivity models

The following are the types of connectivity between servers:

•

Layer 2 connectivity between servers attached to the same DR system at the leaf tier.

•

Layer 3 connectivity between servers attached to the same DR system at the leaf tier.

•

Layer 3 connectivity between servers attached to different DR systems at the leaf tier.

Applicable product matrix

Role

Devices

Software version

Spine HPE FlexFabric 12900E Switch Series Not recommended.

Leaf

HPE FlexFabric 5940 Switch Series Not recommended.

HPE FlexFabric 5710 Switch Series Not recommended.

HPE FlexFabric 5944 & 5945 Switch

Series

This example uses 5945 switches as leaf

nodes.

R6710

Server NIC

Mellanox ConnectX-6 Lx

•

Driver version:

MLNX_OFED_LINUX-5.4-3.2.7.2.3-

rhel8.4-x86_64

• Firmware version:

 Driver: mlx5_core

 Version: 5.4-3.2.7.2.3

 firmware-

version: 26.31.2006

(MT_0000000531)

Mellanox ConnectX-5

•

Driver version:

MLNX_OFED_LINUX-5.4-3.2.7.2.3-

rhel8.4-x86_64

• Firmware version:

 Driver: mlx5_core

 Version: 5.4-3.2.7.2.3

 firmware-version: 16.31.2006

(MT_0000000080)

Mellanox ConnectX-4 Lx

•

Driver version:

MLNX_OFED_LINUX-5.4-3.2.7.2.3-

rhel8.4-x86_64

• Firmware version:

 Driver: mlx5_core

5



Version: 5.4-3.2.7.2.3

 firmware-version: 14.31.2006

(MT_2420110034)

Restrictions and guidelines

Determine the appropriate convergence ratio according to actual business needs. As a best

practice, use two or four high-speed interfaces for the IPL to meet the east-west traffic forwarding

requirements among servers in the same leaf group. Use other high-speed interfaces as uplink

interfaces. Use common-rate interfaces to connect to servers. If the convergence ratio cannot meet

requirements, attach less servers to leaf devices.

When you enable PFC for an 802.1p priority, the device will set a default value for each threshold.

As a best practice, use the default threshold values. For more information, see "Tuning the

parameters" and "Guidelines for tuning parameters."

You can configure WRED by using one of the following approaches:

•

Interface configuration—Configure WRED parameters on an interface and enable WRED.

You can configure WRED parameters for only the RoCE queue. You can configure WRED

parameters on different interfaces. This approach is recommended.

•

WRED table configuration—Create a WRED table, configure WRED parameters for a queue,

and then apply the WRED table to an interface. If you do not configure WRED parameters for a

queue, the default values are used for that queue. As a best practice, configure WRED

parameters for each queue because the default values are small and might not be applicable.

Configuring 5945 switches as leaf nodes

This example describes the procedure to deploy nodes Leaf 1 and Leaf 2. The same procedure

applies to nodes Leaf 3 and Leaf 4 except the single-homed device configuration.

Procedure summary

•

Configuring global RoCE settings

•

Configuring the links towards the spine tier

•

Configuring RoCE settings for the links towards the spine tier

Choose one option as needed

 Configuring a WRED table

 Configuring WRED on an interface

•

Configuring DRNI

•

Configuring RoCE settings for the IPL

•

Configuring the links towards the servers

•

Configuring RoCE settings for the links towards the servers

Choose one option as needed

 Configuring a WRED table

 Configuring WRED on an interface

•

Configuring an underlay BGP instance

•

Configuring VLAN interfaces and gateways. Choose one option as needed:

 Configuring VLAN interfaces and gateways (dual-active gateways)

 Configuring VLAN interfaces and gateways (VRRP gateways)

6

Configuring global RoCE settings

Configuring global PFC settings and data buffer settings

Leaf 1

Leaf 2

Description

Remarks

priority-flow-control

poolID 0 headroom

131072

priority-flow-control

poolID 0 headroom

131072

S

et the maximum

number of cell

resources that can

be used in a

headroom storage

space.

Set the maximum value

allowed.

priority-flow-control

deadlock cos 3 interval

10

priority-flow-control

deadlock cos 3 interval

10

Set the PFC

deadlock detection

interval for the

specified CoS value.

N/A

priority-flow-control

deadlock precision high

priority-flow-control

deadlock precision

high

Set the precision for

the PFC deadlock

detection timer.

Set the high precision for the

PFC deadlock detection

timer.

b

uffer egress cell queue

3 shared ratio 100

buf

fer egress cell

queue 3

shared ratio

100

Set the maximum

shared-

area ratio to

100% for the RoCE

queue.

N/A

buffer

egress cell queue

6 shared ratio 100

buffer

egress cell

queue 6 shared ratio

100

Set the maximum

shared-

area ratio to

100% for the CNP

queue.

N/A

buffer apply buffer apply

Apply manually

configured data

buffer settings. N/A

Configuring WRED tables

Leaf 1

Leaf 2

Description

Remarks

qos wred queue table

100G-WRED-Template

qos wred queue table

100G-WRED-Template

Create a

queue-based WRED

table. N/A

queue 0 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 0 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

20

queue 0 drop-level 2

low-limit 37999 high-limit

38000 discard-probability

20

queue 0 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

queue 0 drop-level 1

low-limit 37999

high-limit 38000

discard-probability 20

queue 0 drop-level 2

low-limit 37999

high-limit 38000

discard-probability 20

Configure WRED

table settings for

queue 0. N/A

queue 1 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 1 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

queue 1 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

queue 1 drop-level 1

low-limit 37999

high-limit 38000

Configure WRED

table settings for

queue 1. N/A

7

Leaf 1

Leaf 2

Description

Remarks

20

queue 1 drop-level 2

low-limit 37999 high-limit

38000 discard-probability

20

discard-probability 20

queue 1 drop-level 2

low-limit 37999

high-limit 38000

discard-probability 20

queue 2 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 2 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

20

queue 2 drop-level 2

low-limit 37999 high-limit

38000 discard-probability

20

queue 2 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

queue 2 drop-level 1

low-limit 37999

high-limit 38000

discard-probability 20

queue 2 drop-level 2

low-limit 37999

high-limit 38000

discard-probability 20

Configure WRED

table settings for

queue 2. N/A

queue 3 drop-level 0

low-limit 1000 high-limit

2000 discard-probability

20

queue 3 drop-level 1

low-limit 1000 high-limit

2000 discard-probability

20

queue 3 drop-level 2

low-limit 1000 high-limit

2000 discard-probability

20

queue 3

weighting-constant 0

queue 3 ecn

queue 3 drop-level 0

low-limit 1000 high-limit

2000

discard-probability 20

queue 3 drop-level 1

low-limit 1000 high-limit

2000

discard-probability 20

queue 3 drop-level 2

low-limit 1000 high-limit

2000

discard-probability 20

queue 3

weighting-constant 0

queue 3 ecn

Configure WRED

table settings for

queue 3.

Set small values for the low

limit and high limit for the

RoCE queue and large

values for other queues.

queue 4 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 4 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

20

queue 4 drop-level 2

low-limit 37999 high-limit

38000 discard-probability

20

queue 4 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

queue 4 drop-level 1

low-limit 37999

high-limit 38000

discard-probability 20

queue 4 drop-level 2

low-limit 37999

high-limit 38000

discard-probability 20

Configure WRED

table settings for

queue 4. N/A

queue 5 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 5 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

20

queue 5 drop-level 2

low-limit 37999 high-limit

38000 discard-probability

20

queue 5 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

queue 5 drop-level 1

low-limit 37999

high-limit 38000

discard-probability 20

queue 5 drop-level 2

low-limit 37999

high-limit 38000

discard-probability 20

Configure WRED

table settings for

queue 5. N/A

8

Leaf 1

Leaf 2

Description

Remarks

queue 6 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 6 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

20

queue 6 drop-level 2

low-limit 37999 high-limit

38000 discard-probability

20

queue 6 ecn

queue 6 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

queue 6 drop-level 1

low-limit 37999

high-limit 38000

discard-probability 20

queue 6 drop-level 2

low-limit 37999

high-limit 38000

discard-probability 20

queue 6 ecn

Configure WRED

table settings for

queue 6. N/A

queue 7 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 7 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

20

queue 7 drop-level 2

low-limit 37999 high-limit

38000 discard-probability

20

queue 7 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

queue 7 drop-level 1

low-limit 37999

high-limit 38000

discard-probability 20

queue 7 drop-level 2

low-limit 37999

high-limit 38000

discard-probability 20

Configure WRED

table settings for

queue 7. N/A

qos wred queu

e table

25G-WRED-Template

qos wred queue table

25G-WRED-Template

Create a

queue-based WRED

table. N/A

queue 0 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 0 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

20

queue 0 drop-level 2

low-limit 37999 high-limit

38000 discard-probability

20

queue 0 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

queue 0 drop-level 1

low-limit 37999

high-limit 38000

discard-probability 20

queue 0 drop-level 2

low-limit 37999

high-limit 38000

discard-probability 20

Configure WRED

table settings for

queue 0. N/A

queue 1 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 1 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

20

queue 1 drop-level 2

low-limit 37999 high-limit

38000 discard-probability

20

queue 1 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

queue 1 drop-level 1

low-limit 37999

high-limit 38000

discard-probability 20

queue 1 drop-level 2

low-limit 37999

high-limit 38000

discard-probability 20

Configure WRED

table settings for

queue 1. N/A

queue 2 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 2 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

Configure WRED

table settings for

queue 2. N/A

9

Leaf 1

Leaf 2

Description

Remarks

queue 2 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

20

queue 2 drop-level 2

low-limit 37999 high-limit

38000 discard-probability

20

queue 2 drop-level 1

low-limit 37999

high-limit 38000

discard-probability 20

queue 2 drop-level 2

low-limit 37999

high-limit 38000

discard-probability 20

queue 3 drop-level 0

low-limit 400 high-limit

1625 discard-probability

20

queue 3 drop-level 1

low-limit 400 high-limit

1625 discard-probability

20

queue 3 drop-level 2

low-limit 400 high-limit

1625 discard-probability

20

queue 3

weighting-constant 0

queue 3 ecn

queue 3 drop-level 0

low-limit 400 high-limit

1625

discard-probability 20

queue 3 drop-level 1

low-limit 400 high-limit

1625

discard-probability 20

queue 3 drop-level 2

low-limit 400 high-limit

1625

discard-probability 20

queue 3

weighting-constant 0

queue 3 ecn

Configure WRED

table settings for

queue 3.

Set small values for the low

limit and high limit for the

RoCE queue and large

values for other queues.

queue 4 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 4 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

20

queue 4 drop-level 2

low-limit 37999 high-limit

38000 discard-probability

20

queue 4 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

queue 4 drop-level 1

low-limit 37999

high-limit 38000

discard-probability 20

queue 4 drop-level 2

low-limit 37999

high-limit 38000

discard-probability 20

Configure WRED

table settings for

queue 4. N/A

queue 5 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 5 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

20

queue 5 drop-level 2

low-limit 37999 high-limit

38000 discard-probability

20

queue 5 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

queue 5 drop-level 1

low-limit 37999

high-limit 38000

discard-probability 20

queue 5 drop-level 2

low-limit 37999

high-limit 38000

discard-probability 20

Configure WRED

table settings for

queue 5. N/A

queue 6 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 6 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

20

queue 6 drop-level 2

queue 6 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

queue 6 drop-level 1

low-limit 37999

high-limit 38000

discard-probability 20

queue 6 drop-level 2

Configure WRED

table settings for

queue 6. N/A

10

Leaf 1

Leaf 2

Description

Remarks

low-limit 37999 high-limit

38000 discard-probability

20

queue 6 ecn

low-limit 37999

high-limit 38000

discard-probability 20

queue 6 ecn

queue 7 drop-level 0

low-limit 37999 high-limit

38000 discard-probability

20

queue 7 drop-level 1

low-limit 37999 high-limit

38000 discard-probability

20

queue 7 drop-level 2

low-limit 37999 high-limit

38000 discard-probability

20

queue 7 drop-level 0

low-limit 37999

high-limit 38000

discard-probability 20

queue 7 drop-level 1

low-limit 37999

high-limit 38000

discard-probability 20

queue 7 drop-level 2

low-limit 37999

high-limit 38000

discard-probability 20

Configure WRED

table settings for

queue 7. N/A

Configuring priority mapping

NOTE:

•

To configure PFC on an interface connecting a leaf device to a server, you must configure the

interface to trust the 802.1p or DSCP priority in packets. To configure PFC on a Layer 3

interface connecting a leaf device to a spine device, you must configure the interface to trust the

DSCP priority in packets.

•

As a best practice, configure outgoing packets of servers to carry the DSCP priority and to be

enqueued by DSCP priority. In this case, configure the interface connecting a leaf device to a

server to trust the DSCP priority in packets. If the server does not support carrying the DSCP

priority, configure the interface connecting a leaf device to a server to trust the 802.

1p priority in

packets and configure DSCP-to-802.1p mappings for all involved packets.

•

This section describes the procedure to configure DSCP-to-

802.1p mappings when an interface

is configured to trust 802.1p priority in packets. This section does not need

to be configured if an

interface is configured to trust the DSCP priority in packets.

Leaf 1

Leaf 2

Description

Remarks

qos map-table dot1p-lp

import 0 export 0

import 1 export 1

import 2 export 2

qos map-table dot1p-lp

import 0 export 0

import 1 export 1

import 2 export 2

Configure the dot1p-lp

mapping table. N/A

traffic classifier dot1p0

operator and traffic classifier dot1p0

operator and

Create a traffic class

with the AND operator.

If the downlink

interface is

configured with

the command,

th

e QoS policy

does not need to

be configured.

if-match service-dot1p 0 if-match service-dot1p 0

Configure an outer

802.1p priority match

criterion. N/A

traffic classifier dot1p1

operator and traffic classifier dot1p1

operator and

Create a traffic class

with the AND operator. N/A

if-match service-dot1p 1 if-match service-dot1p 1

Configure an outer

802.1p priority match

N/A

11

Leaf 1

Leaf 2

Description

Remarks

criterion.

traffic classifier dot1p2

operator and traffic classifier dot1p2

operator and

Create a traffic class

with the AND operator. N/A

if-match service-dot1p 2 if-match service-dot1p 2

Configure an outer

802.1p priority match

criterion. N/A

traffic classifier dot1p3

operator and traffic classifier dot1p3

operator and

Create a traffic class

with the AND operator. N/A

if-match service-dot1p 3 if-match service-dot1p 3

Configure an outer

802.1p priority match

criterion. N/A

traffic classifier dot1p4

operator and traffic classifier dot1p4

operator and

Create a traffic class

with the AND operator. N/A

if-match service-dot1p 4 if-match service-dot1p 4

Configure an outer

802.1p priority match

criterion. N/A

traffic classifier dot1p5

operator and traffic classifier dot1p5

operator and

Create a traffic class

with the AND operator. N/A

if-match service-dot1p 5 if-match service-dot1p 5

Configure an outer

802.1p priority match

criterion. N/A

traffic classifier dot1p5

operator and traffic classifier dot1p5

operator and

Create a traffic class

with the AND operator. N/A

if-match service-dot1p 6 if-match service-dot1p 6

Configure an outer

802.1p priority match

criterion. N/A

traffic classifier dot1p7

operator and traffic classifier dot1p7

operator and

Create a traffic class

with the AND operator. N/A

if-match service-dot1p 7 if-match service-dot1p 7

Configure an outer

802.1p priority match

criterion. N/A

traffic classifier dscp0 operator

and traffic classifier dscp0

operator and

Create a traffic class

with the AND operator. N/A

if-match dscp default if-match dscp default Con

figure an outer

802.1p priority match

criterion. N/A

traffic classifier dscp10

operator and traffic classifier dscp10

operator and

Create a traffic class

with the AND operator. N/A

if-match dscp af11 if-match dscp af11

Configure a DSCP

priority match criterion. N/A

traffic classifier dscp18

operator and traffic classifier dscp18

operator and

Create a traffic class

with the AND operator. N/A

if-match dscp af21 if-match dscp af21

Configure a DSCP

priority match criterion. N/A

traffic classifier dscp26

operator and traffic classifier dscp26

operator and

Create a traffic class

with the AND operator. N/A

if-match dscp af31 if-match dscp af31

Configure a DSCP

priority match criterion. N/A

12

Leaf 1

Leaf 2

Description

Remarks

traffic classifier dscp34

operator and traffic classifier dscp34

operator and

Create a traffic class

with the AND operator. N/A

if-match dscp af41 if-match dscp af41

Configure a DSCP

priority match criterion. N/A

traffic classifier dscp40

operator and traffic classifier dscp40

operator and

Create a traffic class

with the AND operator. N/A

if-match dscp cs5 if-match dscp cs5

Configure a DSCP

priority match criterion. N/A

traffic classifier dscp48

operator and traffic classifier dscp48

operator and

Create a traffic class

with the AND operator. N/A

if-match dscp cs6 if-match dscp cs6

Configure a DSCP

priority match criterion. N/A

traffic classifier dscp56

operator and traffic classifier dscp56

operator and

Create a traffic class

with the AND operator. N/A

if-match dscp cs7 if-match dscp cs7

Configure a DSCP

priority match criterion. N/A

traffic behavior dot1p0 traffic behavior dot1p0

Create a traffic

behavior. N/A

remark dot1p 0 remark dot1p 0

Configure an 802.1p

priority marking action. N/A

traffic behavior dot1p1 traffic behavior dot1p1

Create a traffic

behavior. N/A

remark dot1p 1 remark dot1p 1

Configure an 802.1p

priority marking action. N/A

traffic behavior dot1p2 traffic behavior dot1p2

Create a traffic

behavior. N/A

remark dot1p 2 remark dot1p 2

Configure an 802.1p

priority marking action. N/A

traffic behavior dot1p3 traffic behavior dot1p3

Create a traffic

behavior. N/A

remark dot1p 3 remark dot1p 3

Configure an 802.1p

priority marking action. N/A

traffic behavior dot1p4 traffic behavior dot1p4

Create a traffic

behavior. N/A

remark dot1p 4 remark dot1p 4

Configure an 802.1p

priority marking action. N/A

traffic behavior dot1p5 traffic behavior dot1p5

Create a traffic

behavior. N/A

remark dot1p 5 remark dot1p 5

Configure an 802.1p

priority marking action. N/A

traffic behavior dot1p5 traffic behavior dot1p5

Create a traffic

behavior. N/A

remark dot1p 6 remark dot1p 6

Configure an 802.1p

priority marking action. N/A

traffic behavior dot1p7 traffic behavior dot1p7

Create a traffic

behavior. N/A

13

Leaf 1

Leaf 2

Description

Remarks

remark dot1p 7 remark dot1p 7 Configure

an 802.1p

priority marking action. N/A

traffic behavior dscp-0 traffic behavior dscp-0

Create a traffic

behavior. N/A

remark dscp default remark dscp default

Configure a DSCP

priority marking action. N/A

traffic behavior dscp-af11 traffic behavior dscp-af11

Create a traffic

behavior. N/A

remark dscp af11 remark dscp af11

Configure a DSCP

priority marking action. N/A

traffic behavior dscp-af21 traffic behavior dscp-af21

Create a traffic

behavior. N/A

remark dscp af21 remark dscp af21 Configure a DSCP

priority marking action. N/A

traffic behavior dscp-af31 traffic behavior dscp-af31

Create a traffic

behavior. N/A

remark dscp af31 remark dscp af31

Configure a DSCP

priority marking action. N/A

traffic behavior dscp-af41 traffic behavior dscp-af41

Create a traffic

behavior. N/A

remark dscp af41 remark dscp af41

Configure a DSCP

priority marking action. N/A

traffic behavior dscp-cs5 traffic behavior dscp-cs5

Create a traffic

behavior. N/A

remark dscp cs5 remark dscp cs5

Configure a DSCP

priority marking action. N/A

traffic behavior dscp-cs6 traffic behavior dscp-cs6

Create a traffic

behavior. N/A

remark dscp cs6 remark dscp cs6

Configure a DSCP

priority marking action. N/A

traffic behavior dscp-cs7 traffic behavior dscp-cs7

Create a traffic

behavior. N/A

remark dscp cs7 remark dscp cs7

Configure a DSCP

priority marking action. N/A

qos policy dot1p-dscp qos policy dot1p-dscp

Create a generic QoS

policy. N/A

classifier dot1p0 behavior

dscp-0 classifier dot1p0 behavior

dscp-0 Associate a traffic class

with a traffic behavior. N/A

classifier dot1p3 behavior

dscp-af31 classifier dot1p3 behavior

dscp-af31 Associate a traffic class

with a traffic behavior. N/A

classifier dot1p1 behavior

dscp-af11 classifier dot1p1 behavior

dscp-af11 Associate a traffic class

with a traffic behavior. N/A

classifier dot1p2 behavior

dscp-af21 classifier dot1p2 behavior

dscp-af21 Associate a traffic class

with a traffic behavior. N/A

classifier dot1p4 behavior

dscp-af41 classifier dot1p4 behavior

dscp-af41 Associate a traffic class

with a traffic behavior. N/A

14

Leaf 1

Leaf 2

Description

Remarks

classifier dot1p5 behavior

dscp-cs5 classifier dot1p5 behavior

dscp-cs5 Associate a traffic class

with a traffic behavior. N/A

classifier dot1p5 behavior

dscp-cs6 classifier dot1p5 behavior

dscp-cs6 Associate a traffic class

with a traffic behavior. N/A

classifier dot1p7 behavior

dscp-cs7 classifier dot1p7 behavior

dscp-cs7 Associate a traffic class

with a traffic behavior. N/A

qos policy dscptodot1p qos policy dscptodot1p Create a generic QoS

policy. N/A

classifier dscp0 behavior

dot1p0 classifier dscp0 behavior

dot1p0 Associate a traffic class

with a traffic behavior. N/A

classifier dscp26 behavior

dot1p3 classifier dscp26 behavior

dot1p3 Associate a traffic class

with a traffic behavior. N/A

classifier dscp10 behavior

dot1p1 classifier dscp10 behavior

dot1p1 Associate a traffic class

with a traffic behavior. N/A

classifier dscp18 behavior

dot1p2 classifier dscp18 behavior

dot1p2 Associate a traffic class

with a traffic behavior. N/A

classifier dscp34 behavior

dot1p4 classifier dscp34 behavior

dot1p4 Associate a traffic class

with a traffic behavior. N/A

classifier dscp40 behavior

dot1p5 classifier dscp40 behavior

dot1p5 Associate a traffic class

with a traffic behavior. N/A

classifier dscp48 behavior

dot1p5 classifier dscp48 behavior

dot1p5 Associate a traffic class

with a traffic behavior. N/A

classifier dscp56 behavior

dot1p7 classifier dscp56 behavior

dot1p7 Associate a traffic class

with a traffic behavior. N/A

Configuring the links towards the spine tier

Leaf 1

Leaf 2

Description

monitor-link group 1 monitor-link group 1 Configure a monitor link group.

interface

HundredGigE1/0/25 interface HundredGigE1/0/25 Configure the interface connected to

Spine 1.

port link-mode route port link-mode route Configure the interface to operate in

route mode as a Layer 3 interface.

ip address 172.16.2.154

255.255.255.252 ip address 172.16.2.158

255.255.255.252 Configure an IP address.

port monitor-link group 1

uplink port monitor-link group 1 uplink Configure the interface as the uplink

interface of the monitor link group.

interface

HundredGigE1/0/26 interface HundredGigE1/0/26 Configure the interface connected to

Spine 1.

port link-mode route port link-mode route Configure the interface to operate in

route mode as a Layer 3 interface.

ip address 172.16.3.154

255.255.255.252 ip address 172.16.3.158

255.255.255.252 Configure an IP address.

port monitor-link group 1

uplink port monitor-link group 1 uplink Configure the interface as the uplink

interface of the monitor link group.

15

Configuring RoCE settings for the links towards the spine tier

Configuring a WRED table

Leaf 1

Leaf 2

Description

Remarks

interface range

HundredGigE1/0/25

HundredGigE1/0/26

interface range

HundredGigE1/0/25

HundredGigE1/0/26

Configure the

interfaces connected

to Spine 1. N/A

priority-flow-control

deadlock enable priority-flow-control

deadlock enable

Enable PFC deadlock

detection on an

interface. N/A

priority-flow-control

enable priority-flow-control enable

Enable PFC on all

Ethernet interfaces. N/A

priority-flow-control

no-drop dot1p 3 priority-flow-control

no-drop dot1p 3

Enable PFC for the

queue

of RoCE

packets. N/A

priority-flow-

control dot1p

3 headroom 491 priority-flow-

control dot1p

3 headroom 491

Set the headroom

buffer threshold to 491

for 802.1p priority 3.

After you enable PFC

for the specified 802.1p

priority, the device

automatically deploys

the PFC threshold

settings. For more

information, see

"

Recommended PFC

settings."

priority-flow-

control dot1p

3 reserved-buffer 17 priority-flow-

control dot1p

3 reserved-buffer 17 S

et the PFC reserved

threshold.

priority-flow-

control dot1p

3 ingress-

buffer dynamic

5

priority-flow-

control dot1p

3 ingress-buffer dynamic 5

Set the dynamic back

pressure frame

triggering threshold.

priority-flow-

control dot1p

3 ingress-threshold-offset

12

priority-flow-

control dot1p

3 ingress-threshold-offset

12

Set the offset between

the back pressure

frame stopping

threshold and

triggering threshold.

qos trust dscp qos trust dscp Configure the

interface to trust the

DSCP priority N/A

qos wfq byte-count qos wfq byte-count Enable byte-count

WFQ. N/A

qos wfq be group 1

byte-count 15 qos wfq be group 1

byte-count 15

Configure the weight

for queue 0.

Adjust the weight

according to your

business needs.

qos wfq af1 group 1

byte-count 2 qos wfq af1 group 1

byte-count 2

Configure the weight

for queue 1.

Adjust the weight

according to your

business needs.

qos wfq af2 group 1

byte-count 2 qos wfq af2 group 1

byte-count 2

Configure the weight

for queue 2.

Adjust the weight

according to your

business needs.

qos wfq af3 group 1

byte-count 60 qos wfq af3 group 1

byte-count 60

Configure the weight

for queue 3.

Adjust the weight

according to your

business needs.

This example

configures the weights

of the RoCE queue and

other queues at a 4:1

ratio.

qos wfq cs6 group sp qos wfq cs6 group sp Assign queue 6 to the

Adjust the setting

according to your

16

Leaf 1

Leaf 2

Description

Remarks

SP group.

business needs.

qos wfq cs7 group sp qos wfq cs7 group sp Assign queue 7 to the

SP group.

Adjust the setting

according to your

business needs.

qos wred apply

100G-WRED-Template

qos wred apply

100G-WRED-Template Apply the WRED table

to the interface. N/A

qos gts queue 6 cir

50000000 cbs 16000000 qos gts queue 6 cir

50000000 cbs 16000000

Configure GTS with a

CIR of 50 Gbps for the

CNP queue. N/A

Configuring WRED on an interface

Leaf 1

Leaf 2

Description

Remarks

interface range

HundredGigE1/0/25

HundredGigE1/0/26

interface range

HundredGigE1/0/25

HundredGigE1/0/26

Configure the

interfaces connected

to Spine 1. N/A

priority-flow-control

deadlock enable priority-flow-control

deadlock enable

Enable PFC deadlock

detection on an

interface. N/A

priority-flow-control

enable priority-flow-control enable

Enable PFC on all

Ethernet interfaces. N/A

priority-flow-control

no-drop dot1p 3 priority-flow-control

no-drop dot1p 3

Enable PFC for the

queue of RoCE

packets. N/A

priority-flow-

control dot1p

3 headroom 491 priority-flow-

control dot1p

3 headroom 491

Set the headroom

buffer threshold to 491

for 802.1p priority 3.

After you enable PFC

for the specified 802.1p

priority, the device

automatically deploys

the PFC threshold

settings. For more

information, see

"

Recommended PFC

settings."

priority-flow-control dot1p

3 reserved-buffer 17 priority-flow-

control dot1p

3 reserved-buffer 17 S

et the PFC reserved

threshold.

priority-flow-

control dot1p

3 ingress-

buffer dynamic

5

priority-flow-

control dot1p

3 ingress-buffer dynamic 5

Set the dynamic back

pressure frame

triggering threshold.

priority-flow-

control dot1p

3 ingress-threshold-offset

12

priority-flow-

control dot1p

3 ingress-threshold-offset

12

Set the offset between

the back pressure

frame stopping

threshold and

triggering threshold.

qos trust dscp qos trust dscp Configure the

interface to trust the

DSCP priority N/A

qos wfq byte-count qos wfq byte-count Enable byte-count

WFQ. N/A

qos wfq be group 1

byte-count 15 qos wfq be group 1

byte-count 15

Configure the weight

for queue 0.

Adjust the weight

according to your

business needs.

qos wfq af1 group 1

byte-count 2 qos wfq af1 group 1

byte-count 2

Configure the weight

for queue 1.

Adjust the weight

according to your

business needs.

qos wfq af2 group 1

byte-count 2 qos wfq af2 group 1

byte-count 2 Co

nfigure the weight

for queue 2.

Adjust the weight

according to your

business needs.

17

Leaf 1

Leaf 2

Description

Remarks

qos wfq af3 group 1

byte-count 60 qos wfq af3 group 1

byte-count 60

Configure the weight

for queue 3.

Adjust the weight

according to your

business needs.

This example

configures the weights

of the RoCE queue and

other queues at a 4:1

ratio.

qos wfq cs6 group sp qos wfq cs6 group sp Assign queue 6 to the

SP group.

Adjust the setting

according to your

business needs.

qos wfq cs7 group sp qos wfq cs7 group sp Assign queue 7 to the

SP group.

Adjust the setting

according to your

business needs.

qos wred queue 3

drop-level 0 low-limit

1000 high-limit 2000

discard-probability 20

qos wred queue 3

drop-level 1 low-limit

1300 high-limit 2100

discard-probability 20

qos wred queue 3

drop-level 2 low-limit

1300 high-limit 2100

discard-probability 20

qos wred queue 3

drop-level 0 low-limit 1000

high-limit 2000

discard-probability 20

qos wred queue 3

drop-level 1 low-limit 1300

high-limit 2100

discard-probability 20

qos wred queue 3

drop-level 2 low-limit 1300

high-limit 2100

discard-probability 20

Set the drop-related

parameters

for the

RoCE queue.

Set small values for the

low limit and high limit

for the RoCE queue

and large values for

other queues.

qos wred queue 3

weighting-constant 0 qos wred queue 3

weighting-constant 0

Set the WRED

exponent for average

queue size calculation

for the RoCE queue.

N/A

qos wred queue 3 ecn qos wred queue 3 ecn Enable

ECN for the

RoCE queue. N/A

qos wred queue 6

drop-level 0 low-limit

37999 high-limit 38000

discard-probability 20

qos wred queue 6

drop-level 1 low-limit

37999 high-limit 38000

discard-probability 20

qos wred queue 6

drop-level 2 low-limit

37999 high-limit 38000

discard-probability 20

qos wred queue 6

drop-level 0 low-limit

37999 high-limit 38000

discard-probability 20

qos wred queue 6

drop-level 1 low-limit

37999 high-limit 38000

discard-probability 20

qos wred queue 6

drop-level 2 low-limit

37999 high-limit 38000

discard-probability 20

Set the drop-related

parameters

for the

CNP queue. N/A

qos wred queue 6 ecn qos wred queue 6 ecn

Set the WRED

exponent for average

queue size calculation

for the CNP queue.

N/A

qos gts queue 6 cir

50000000 cbs 16000000 qos gts queue 6 cir

50000000 cbs 16000000 Enable

ECN for the

CNP queue. N/A

18

Configuring DRNI

Leaf 1

Leaf 2

Description

Remarks

interface

Bridge-Aggregation1000 interface

Bridge-Aggregation1000

Create the Layer 2

aggregate interface to be

used as the IPP and enter

its interface view.

N/A

link-

aggregation mode

dynamic link-

aggregation mode

dynamic

Configure the aggregate

interface to operate in

dynamic mode. N/A

quit quit Return to system view. N/A

interface range

HundredGigE1/0/29 to

HundredGigE1/0/32

interface range

HundredGigE1/0/29 to

HundredGigE1/0/32

Enter

interface range

view. N/A

port link-aggregation

group 1000 port link-aggregation

group 1000

Assign the ports

to the

link aggregation group for

the IPP (aggregation

group 1000).

N/A

quit quit Return to system view. N/A

interface

Bridge-Aggregation1000 interface

Bridge-Aggregation1000

Enter the interface view

for the port to be used as

the IPP. N/A

port drni intra-portal-port

1 port drni intra-portal-port 1

Specify the aggregate

interface as the IPP. N/A

quit quit Return to system view. N/A

ip vpn-instance

DRNIKeepalive ip vpn-instance

DRNIKeepalive C

reate a VPN instance

used for DR keepalive. N/A

quit quit Return to system view. N/A

interface

Route-Aggregation1000 interface

Route-Aggregation1000

Create the Layer 3

aggregate interface to be

used as the keepalive

interface.

N/A

ip binding vpn-instance

DRNIKeepalive ip binding vpn-instance

DRNIKeepalive Associate the interface

with the VPN instance. N/A

ip address 1.1.1.1

255.255.255.252 ip address 1.1.1.2

255.255.255.252

Assign an IP address to

the keepalive interface. N/A

link-

aggregation mode

dynamic link-

aggregation mode

dynamic

Configure the aggregate

interface to operate in

dynamic mode. N/A

quit quit Return to system view. N/A

interface

Twenty-FiveGigE1/0/55 interface

Twenty-FiveGigE1/0/55

Configure the interface as

a member port

of the

keepalive interface. N/A

port link-mode route port link-mode route

Configure the interface for

keepalive detection to

operate in route mode as

a Layer 3 interface.

N/A

port link-aggregation

group 1000 port link-aggregation

group 1000 Assign the port to the link

aggregation group for the

IPP (aggregation group

N/A

19

Leaf 1

Leaf 2

Description

Remarks

1000).

quit quit Return to system view. N/A

interface

Twenty-FiveGigE1/0/56 interface

Twenty-FiveGigE1/0/56

Configure the interface as

a member port

of the

keepalive interface. N/A

port link-mode route port link-mode route

Configure the interface for

keepalive detection to

operate in route mode as

a Layer 3 interface.

N/A

port link-aggregation

group 1000 port link-aggregation

group 1000

Assign the port to the link

aggregation group for the

IPP

(aggregation group

1000).

N/A

quit quit Return to system view. N/A

drni role priority 50 drni role priority 100 Set the DR role priority. N/A

drni system-mac

2001-0000-0018 drni system-mac

2001-0000-0018 Configure the DR system

MAC address.

You must configure

the same DR

system MAC

address for all DR

member devices.

drni system-number 1 drni system-number 2 Set the DR system

number.

You must assign

different DR system

numbers to the

member devices in

a DR system.

drni system-priority 110 drni system-priority 110 (Optional.) Set the DR

system priority.

You must set the

same DR system

priority on the

member devices in

a DR system.

The lower the value,

the higher the

priority.

drni

keepalive ip

destination 1.1.1.2 source

1.1.1.1 vpn-instance

DRNIKeepalive

drni

keepalive ip

destination 1.1.1.1 source

1.1.1.2 vpn-instance

DRNIKeepalive

Configure DR keepalive

packet parameters.

If the interface of an

IP address is not an

interface excluded

from the MAD

shutdown action,

configure it as an

interface excluded

from the MAD

shutdown action.

drni

mad exclude

interface LoopBack1 drni mad exclude interface

LoopBack1

Exclude Loopback1 from

the shutdown acti

on by

DRNI MAD. N/A

drni

mad exclude

interface

Route-Aggregation1000

drni mad exclude interface

Route-Aggregation1000

Exclude

Route-Aggregation 1000

from the shutdown action

by DRNI MAD.

N/A

drni

mad exclude

interface

Twenty-FiveGigE1/0/55

drni mad exclude interface

Twenty-FiveGigE1/0/55

Exclude Twenty-FiveGigE

1/0/55 from the shutdown

action by DRNI MAD. N/A

drni

mad exclude

drni mad exclude interface Exclude Twenty-FiveGigE N/A

Page is loading ...

Aruba DRNI+RDMA Configuration Guide

Aruba DRNI+RDMA Configuration Guide

Related papers

Other documents