Asus PEM-FDR User manual

Brand: Asus

Model: PEM-FDR

Type: User manual

www.mellanox.com

Mellanox WinOF VPI User Manual

Rev 4.3

Mellanox Technologies

350 Oakmead Parkway Suite 100

Sunnyvale, CA 94085

U.S.A.

www.mellanox.com

Tel: (408) 970-3400

Fax: (408) 970-3403

Mellanox Technologies, Ltd.

Beit Mellanox

PO Box 586 Yokneam 20692

Israel

www.mellanox.com

Tel: +972 (0)74 723 7200

Fax: +972 (0)4 959 3245

Mellanox®, Mellanox logo, BridgeX®, ConnectX®, CORE-Direct®, InfiniBridge®, InfiniHost®, InfiniScale®,

MLNX-OS®, PhyX®, SwitchX®, UFM®, Virtual Protocol Interconnect® and Voltaire® are registered trademarks of

Mellanox Technologies, Ltd.

Connect-IB™, FabricIT™, Mellanox Open Ethernet™, Mellanox Virtual Modular Switch™, MetroX™, MetroDX™,

ScalableHPC™, Unbreakable-Link™ are trademarks of Mellanox Technologies, Ltd.

All other trademarks are property of their respective owners.

NOTE:

THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED

DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY

KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE

THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT

HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE

PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND

DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST

QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT

ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES

FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND

(INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;

LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY

THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE

OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED

DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Document Number: 3280

Rev 4.3

Mellanox Technologies

WinOF VPI for Windows User Manual Rev 4.3

Mellanox Technologies 3

Table of Contents

Document Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Supported Network Adapter Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Documentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Common Abbreviations and Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Chapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.1 Mellanox VPI Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.2 Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.3 Supported Network Adapter Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.3.1 Downloading the Firmware Tools Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

1.3.2 Downloading the Firmware Image of the Adapter Card. . . . . . . . . . . . . . . . . . . . 12

1.3.3 Updating Adapter Card Firmware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Chapter 2 Driver Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.1 RDMA over Converged Ethernet (RoCE). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.1.1 RoCE Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.1.2 Ported Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.1.3 Reading Port Counters Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.1.4 Setting RoCE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.1.5 Setting RoCE MTU. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.2 A Detailed Example of Using InfiniBand over Ethernet (RoCE) . . . . . . . . . . . . 14

2.2.1 Installing and Loading the Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.2.2 Running an Example Test - ib_send_bandwidth . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.2.3 Using an ND Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.3 Hyper-V with VMQ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.4 Header Data Split. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.5 Receive Side Scaling (RSS). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.6 Port Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.6.1 Auto Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.6.2 Port Protocol Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.7 Load Balancing, Fail-Over (LBFO) and VLAN . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.7.1 Adapter Teaming. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.7.2 Creating a Load Balancing and Fail-Over (LBFO) Bundle . . . . . . . . . . . . . . . . . 22

2.7.3 Creating a Port VLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.7.4 Removing a Port VLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Chapter 3 Deploying Windows Server 2012 with SMB Direct . . . . . . . . . . . . . . . . . . . 29

3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.2 Hardware and Software Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.3 SMB Configuration Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Rev 4.3

Mellanox Technologies

3.3.1 Verifying SMB Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.3.2 Verifying SMB Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.4 Verifying SMB Events that Confirm RDMA Connection . . . . . . . . . . . . . . . . . 30

Chapter 4 Driver Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1 Configuring the InfiniBand Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1.1 Modifying IPoIB Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1.2 Displaying Adapter Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.2 Configuring the Ethernet Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Chapter 5 Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

5.1 General Performance Optimization and Tuning . . . . . . . . . . . . . . . . . . . . . . . . . 36

5.1.1 Registry Tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

5.1.2 Enable RSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

5.1.3 Tuning the IPoIB Network Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

5.1.4 Tuning the Ethernet Network Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

5.2 Application Specific Optimization and Tuning. . . . . . . . . . . . . . . . . . . . . . . . . . 41

5.2.1 Ethernet Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

5.2.2 IPoIB Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

5.3 Tunable Performance Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Chapter 6 OpenSM - Subnet Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Chapter 7 InfiniBand Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

7.1 Network Direct Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

7.2 part_man - Virtual IPoIB Port Creation Utility . . . . . . . . . . . . . . . . . . . . . . . . . . 47

7.3 InfiniBand Fabric Diagnostic Utilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

7.3.1 Utilities Usage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

7.3.2 ibdiagnet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

7.3.3 ibportstate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

7.3.4 ibroute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

7.3.5 smpquery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

7.3.6 perfquery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

7.3.7 ibping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

7.3.8 ibnetdiscover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

7.3.9 ibtracert . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

7.3.10 sminfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

7.3.11 ibclearerrors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

7.3.12 ibstat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

7.3.13 vstat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

7.3.14 osmtest. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

7.4 InfiniBand Fabric Performance Utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

7.4.1 ib_read_bw . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

7.4.2 ib_read_lat. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

7.4.3 ib_send_bw . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

7.4.4 ib_send_lat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

7.4.5 ib_write_bw. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

7.4.6 ib_write_lat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

7.4.7 ibv_read_bw . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

WinOF VPI for Windows User Manual Rev 4.3

Mellanox Technologies 5

7.4.8 ibv_read_lat. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

7.4.9 ibv_send_bw . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

7.4.10 ibv_send_lat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

7.4.11 ibv_write_bw. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

7.4.12 ibv_write_lat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

7.4.13 ibaddr. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

7.4.14 ibcacheedit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

7.4.15 iblinkinfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

7.4.16 ibqueryerrors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

7.4.17 ibsysstat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

7.4.18 perfquery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

7.4.19 saquery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

7.4.20 smpdump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

Chapter 8 Software Development Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Chapter 9 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

9.1 InfiniBand Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

9.2 Ethernet Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

Chapter 10 Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

Rev 4.3

Mellanox Technologies

List of Tables

Table 1: Document Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

Table 2: Documentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Table 3: Abbreviations and Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Table 4: Registry Keys Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Table 5: ibdiagnet (of ibutils) Output Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Table 6: ibportstate Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Table 7: ibroute Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Table 8: smpquery Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58

Table 9: perfquery Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Table 10: ibping Flags and Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Table 11: ibnetdiscover Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Table 12: ibtracert Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Table 13: sminfo Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71

Table 14: ibclearerrors Flags and Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Table 15: ibstat Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Table 16: vstat Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Table 17: osmtest Flags and Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

Table 18: ib_read_bw Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77

Table 19: ib_read_lat Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .78

Table 20: ib_send_bw Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79

Table 21: ib_send_lat Flags and Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .80

Table 22: ib_write_bw Flags and Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81

Table 23: ib_write_lat Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

Table 24: ibv_read_bw Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83

Table 25: ibv_read_lat Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .84

Table 26: ibv_send_bw Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85

Table 27: ibv_send_lat Flags and Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

Table 28: ibv_write_bw Flags and Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .88

Table 29: ibv_write_lat Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90

Table 30: ibaddr Flags and Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Table 31: ibcacheedit Flags and Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

Table 32: iblinkinfo Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .95

Table 33: ibqueryerrors Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .97

Table 34: ibsysstat Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .100

Table 35: perfquery Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

Table 36: saquery Flags and Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Table 37: smpdump Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .108

WinOF VPI for Windows User Manual Rev 4.3

Mellanox Technologies 6

Document Revision History

Table 1 - Document Revision History

Document

Revision Date Changes

Rev 4.3 January 6, 2013 • Changed the options description in

Appendix 5.1.4.3,“Options,” on page 39

• Added “perf_tuning”Appendix 5.1.4.2,“Synopsys,” on page

• Added “-st” flag to Appendix 5.1.4.3,“Options,” on page 39

• Added ‘ibdiagnet’ utility:

Appendix 7.3.2.1,“ibdiagnet Options,” on page 50

Appendix 7.3.2.2,“ibdiagnet Output Files,” on page 50

Appendix 7.3.2.3,“ibdiagnet Error Codes,” on page 52

• Minor changes

Rev 4.2 October 20, 2012 • Added Section 3, “Deploying Windows Server 2012

with SMB Direct”, on page 29, and its subsections

• Updated Section 5, “Performance Tuning”, on

page 36

• Added Section 2.4, “Header Data Split”, on page 17

• Added Section 7.2, “part_man - Virtual IPoIB Port

Creation Utility”, on page 47

Rev 3.2.0 July 23, 2012 • No changes

Rev 3.1.0 May 21, 2012 • Added section Tuning the IPoIB Network Adapter

• Added section Tuning the Ethernet Network Adapter

• Added section Performance tuning tool application

• Removed section Tuning the Network Adapter

• Removed section part_man

• Removed section ibdiagnet

Rev 3.0.0 February 08, 2012 • Added section RDMA over Converged Ethernet (RoCE) and

its subsections

• Added section Hyper-V with VMQ

• Added section Network Driver Interface Specification (NDIS)

• Added section Header Data Split

• Added section Auto Sensing

• Added section Adapter Teaming

• Added section Port Protocol Configuration

• Added section Advanced Configuration for InfiniBand Driver

• Added section Advanced Configuration for Ethernet Driver

• Added section Updated section Tunable Performance Parame-

ters

• Added section Merged Ethernet and InfiniBand features sec-

tions

• Removed section Sockets Direct Protocol and its subsections

• Removed section Winsock Direct and Protocol and its subsec-

tions

• Removed section Added ConnectX®-3 support

• Removed section IPoIB Drivers Overview

• Removed section Booting Windows from an iSCSI Target

Rev 2.1.3 January 28. 2011 Complete restructure

Rev 4.3

Mellanox Technologies

Rev 2.1.2 October 10, 2010 • Removed section Debug Options.

• Updated Section 3, “Uninstalling Mellanox VPI Driver,” on

page 11

• Added Section 6, “InfiniBand Fabric,” on page 38 and its sub-

sections

• Added Section 6.3, “InfiniBand Fabric Performance Utilities,”

on page 71 and its subsections

Rev 2.1.1.1 July 14, 2010 Removed all references of InfiniHost® adapter since it is

not

supported starting with WinOF VPI v2.1.1

Rev 2.1.1 May 2010 First release

Table 1 - Document Revision History

Document

Revision Date Changes

WinOF VPI for Windows User Manual Rev 4.3

Mellanox Technologies 8

About this Manual

Scope

The document describes WinOF Rev 4.3 features, performance, InfiniBand diagnostic, tools con-

tent and configuration. Additionally, this document provides information on various performance

tools supplied with this version.

Intended Audience

This manual is intended for system administrators responsible for the installation, configuration,

management and maintenance of the software and hardware of VPI (InfiniBand, Ethernet)

adapter cards. It is also intended for application developers.

Supported Network Adapter Cards

Mellanox WinOF Rev 4.3 for Windows 2012, Windows server 2008 and Windows server 2008

R2 supports the following Mellanox network adapter cards:

• ConnectX®-2 SDR/DDR/QDR

• ConnectX®-3 FDR/SDR/QDR

Rev 4.3

Mellanox Technologies

Documentation Conventions

Common Abbreviations and Acronyms

Table 2 - Documentation Conventions

Description Convention Example

File names file.extension

Directory names directory

Commands and their parameters command param1 mts3610-1 > show hosts

Required item < >

Optional item [ ]

Mutually exclusive parameters { p1, p2, p3 } or {p1 | p2 |

p3}

Optional mutually exclusive parame-

ters

[ p1 | p2 | p3 ]

Variables for which users supply spe-

cific values

Italic font enable

Emphasized words Italic font These are emphasized words

Note <text> This is a note..

Warning <text> May result in system instabil-

ity.

Table 3 - Abbreviations and Acronyms (Sheet 1 of 2)

Abbreviation /

Acronym Whole Word / Description

B (Capital) ‘B’ is used to indicate size in bytes or multiples of bytes

(e.g., 1KB = 1024 bytes, and 1MB = 1048576 bytes)

b (Small) ‘b’ is used to indicate size in bits or multiples of bits (e.g.,

1Kb = 1024 bits)

FW Firmware

HCA Host Channel Adapter

HW Hardware

IB InfiniBand

WinOF VPI for Windows User Manual Rev 4.3

Mellanox Technologies 10

LSB Least significant byte

lsb Least significant bit

MSB Most significant byte

msb Most significant bit

NIC Network Interface Card

SW Software

VPI Virtual Protocol Interconnect

IPoIB IP over InfiniBand

PFC Priority Flow Control

PR Path Record

RDS Reliable Datagram Sockets

RoCE RDMA over Converged Ethernet

SL Service Level

MPI Message Passing Interface

EoIB Ethernet over Infiniband

QoS Quality of Service

ULP Upper Level Protocol

VL Virtual Lane

Table 3 - Abbreviations and Acronyms (Sheet 2 of 2)

Abbreviation /

Acronym Whole Word / Description

IntroductionRev 4.3

Mellanox Technologies

1 Introduction

This User Manual addresses the Mellanox WinOF driver Rev 4.3 package distributed for:

• Windows Server 2012 (64 bit only)

• Windows Server 2008 (32 bit, and 64 bit)

• Windows Server 2008 R2 (64 bit only)

Mellanox WinOF is composed of several software modules that contain an InfiniBand and Ether-

net driver. The Mellanox WinOF driver supports Infiniband and 40/10GB Ethernet ports. The port

type is determined upon boot based on card’s capability and user setting.

1.1 Mellanox VPI Package Contents

The Mellanox WinOF for Windows package contains the following components:

• Core and ULPs

• IB network adapter cards low-level drivers (mlx4)

• IB Access Layer (IBAL)

• Ethernet driver (ETH)

• IP over InfiniBand (IPoIB)

• Upper Layer Protocols (ULPs):

• NetworkDirect (ND)

• Utilities

• SW Development Kit (SDK)

• Documentation

1.2 Hardware and Software Requirements

• Administrator privileges on your machine(s)

• Disk Space for installation: 100MB

1.3 Supported Network Adapter Cards

Mellanox MLNX_WinOF_4.3_<OS_type>_<arch>.exe supports the following Mellanox net-

work adapter cards:

ConnectX®-3 Pro QDR/FDR10/FDR and ConnectX®-3 Pro EN - For the firmware

version, please contact Mellanox support.

ConnectX®-3 QDR/FDR10/FDR and ConnectX®-3 EN - Firmware v2.11.0500 and

above

ConnectX®-2 SDR/DDR/QDR and ConnectX®-2 EN - Firmware v2.9.1200 and

above

The adapter card may not have been shipped with the latest firmware version. This section

describes how to update firmware.

WinOF VPI for Windows User Manual Rev 4.3

Mellanox Technologies 12

1.3.1 Downloading the Firmware Tools Package

1. Download Mellanox Firmware Tools

Please download the current firmware tools package (MFT) from http://www.mellanox.com > Prod-

ucts > Software/Drivers > InfiniBand & VPI SW/Drivers > Firmware Tools.

The tools package to download is "MFT Software for Windows_x64" for x64 architecture.

2. Install and Run WinMFT

To install the WinMFT package, double click the MSI or run it from the command prompt.

Enter:

3. Check the Device Status

• start/stop mst is automatically done by the tools > C:\Users\herod\Desktop>mst start

• To check device status run > mst status

If no card installation problems occur, the status command should produce the following output:

where device ID will be one of the supported PCI device IDs.

1.3.2 Downloading the Firmware Image of the Adapter Card

• To download the correct card firmware image, please visit

http://www.mellanox.com > Support > Firmware Download

• To identify your adapter card, please visit

http://www.mellanox.com > Support > Firmware Downloads > Identifying Adapter Cards

1.3.3 Updating Adapter Card Firmware

Using a card specific binary firmware image file, enter the following command:

For additional details, please check the MFT user's manual under

http://www.mellanox.com > Products > Adapter IB/VPI SW

Install the WinMFT package from the command line with administrator privileges.

msiexec.exe /i WinMFT_<arch>_<version>.msi

mt<device id>_pciconf0

mt<device id>_pci_cr0

> flint -d mt<device id>_pci_cr0 -i <image_name.bin> burn

Driver FeaturesRev 4.3

Mellanox Technologies

2 Driver Features

The Mellanox VPI WinOF driver release introduces the following capabilities:

• One or two ports

• Up to 16 Rx queues per port

• Rx steering mode (RSS)

• Hardware Tx/Rx checksum calculation

• Large Send Offload (i.e., TCP Segmentation Offload)

• Hardware multicast filtering

• Adaptive interrupt moderation

• MSI-X support

• Auto Sensing

•RoCE

Ethernet Only:

• HW VLAN filtering

• Header Data Split

For the complete list of Ethernet and InfiniBand Known Issues and Limitations, see

MLNX_WinVPI_ReleaseNotes.txt.

2.1 RDMA over Converged Ethernet (RoCE)

2.1.1 RoCE Overview

Remote Direct Memory Access (RDMA) is the remote memory management capability that

allows server to server data movement directly between application memory without any CPU

involvement. RDMA over Ethernet (RoCE) is a mechanism to provide this efficient data transfer

with very low latencies on loss-less Ethernet networks. With advances in data center convergence

over reliable Ethernet, ConnectX®-2/ConnectX®-3 EN with RoCE uses the proven and efficient

RDMA transport to provide the platform for deploying RDMA technology in mainstream data

center application at 10GigE and 40GigE link-speed. ConnectX®-2/ConnectX®-3 EN with its

hardware offload support takes advantage of this efficient RDMA transport (InfiniBand) services

over Ethernet to deliver ultra low latency for performance-critical and transaction intensive

applications such as financial, data base, storage, and content delivery networks.

RoCE encapsulates IB transport and GRH headers in Ethernet packets bearing a dedicated ether

type. While the use of GRH is optional within IB subnets, it is mandatory when using RoCE.

Applications written over IB verbs should work seamlessly, but they require provisioning of

GRH information when creating address vectors. The library and driver are modified to provide

mapping from GID to MAC addresses required by the hardware.

2.1.2 Ported Applications

The following applications are ported with RoCE:

• Network Direct (ND) applications work without any change

WinOF VPI for Windows User Manual Rev 4.3

Mellanox Technologies 14

• Performance tests

2.1.3 Reading Port Counters Statistics

RoCE port statistics are not shown in the Windows network counters associated with Etherent

interface. It is possible to read port statistics in the same way it is done for regular InfiniBand

ports. The information is available by running vstat -c.

2.1.4 Setting RoCE

» To set the RoCE, please perform the following:

Step 1. Open Device Manager and expand System devices display pane.

Step 2. Right-click the Mellanox ConnectX VPI (MT26428) - PCIe 2.0 5GT/s, IB QDR / 10GigE Net-

work Adapter

entry and left-click Properties.

Step 3. Click the Port Protocol tab and check RoCE check box.

Step 4. Click OK. It's will cause to driver restart

2.1.5 Setting RoCE MTU

Ethernet packet uses the general MTU value whereas the RoCE packet uses the RoCE MTU.

All devices that run the RoCE protocol must have the same MTU, otherwise packets larger than

the minimum MTU are dropped and not transferred.

When RoCE is enabled, you can configure the MTU that can be sent by the RoCE protocol.

• The valid RoCE MTU values are: 256, 512, 1024, 2048

When using MTU 2048, the administrator should configure the switches to support MTU 2048 or

higher.

• The default MTU is 1024

To set the RoCE MTU, please perform the following:

Step 1. Open Device Manager and expand Network Adapters in the device display pane.

Step 2. Right-click the Mellanox ConnectX 10Gb Ethernet Adapter entry and left-click Properties.

Step 3. Click the Advanced tab and modify the desired properties.

Step 4. Select RoCE Options and click Properties to modify the settings as needed.

Step 5. Click OK

2.2 A Detailed Example of Using InfiniBand over Ethernet (RoCE)

This section provides a step-by-step example of using InfiniBand over Ethernet (RoCE).

2.2.1 Installing and Loading the Driver

The Mellanox WinOF VPI installation installs RoCE as part its package..

Driver FeaturesRev 4.3

Mellanox Technologies

Enter the following command to display the current status run of Mellanox WinOF VPI.

Notes regarding the command output:

1. The InfiniBand port (port 1) is in PORT_DOWN state, and the Ethernet port (port 2) is in

PORT_ACTIVE state.

2. Look at the transport parameter of each port. In this case port 1 is IB and port 2 is RoCE.

Nevertheless, port 2 appears in the list of the HCA's ports.

3. The firmware version is 2.10.1170 (appears at the top).

4. The IB over Ethernet’s Port MTU is 2K byte at maximum. Please note that RoCE’s MTU are

subject to IB MTU restrictions. The RoCE’s MTU values are, 256 byte, 512 byte, 1024 byte

and 2K. To modify MTU settings, please go to Device Manager > Network Adapter > right

click on the required adapter > Properties >Advanced tab > Settings > RoCE Options and

click Properties.

vstat.exe

hca_idx=0

uplink={BUS=PCI_E, SPEED=5.0 Gbps, WIDTH=x4, CAPS=2.5*x8}

MSI-X={ENABLED=1, SUPPORTED=128, GRANTED=10, ALL_MASKED=N}

vendor_id=0x02c9

vendor_part_id=4099

hw_ver=0x0

fw_ver=2.10.1170

PSID=MT_1020110019

node_guid=0002:c902:004a:0110

num_phys_ports=2

port=1

port_guid=0002:c902:004a:0111

port_state=PORT_DOWN (1)

link_speed=NA

link_width=NA

rate=NA

port_phys_state=DISABLED (3)

active_speed=10.00 Gbps

sm_lid=0x0000

port_lid=0x0000

port_lmc=0x0

transport=IB

max_mtu=4096 (5)

port=2

port_guid=0202:c9ff:fe2e:9d01

port_state=PORT_ACTIVE (4)

link_speed=10.00 Gbps

link_width=4x (2)

rate=40.00 Gbps

port_phys_state=CFG_TRAINING (4)

active_speed=10.00 Gbps

sm_lid=0x0000

port_lid=0x0000

port_lmc=0x0

transport=RoCE

max_mtu=2048 (4)

WinOF VPI for Windows User Manual Rev 4.3

Mellanox Technologies 16

2.2.2 Running an Example Test - ib_send_bandwidth

Step 1. Start the server first:

Step 2. Start the client:

2.2.3 Using an ND Test

Step 1. Start the server first:

ib_send_bw -a

------------------------------------------------------------------

Send BW Test

Inline data is used up to 400 bytes message

Connection type : RC

test

max inline size 28

local address: LID 0000, QPN 0x5a020000, PSN 0xf50e0000, RKey 0x120008 VAddr 0x00000

001140040

remote address: LID 0000, QPN 0x5a020000, PSN 0x36090000, RKey 0x120000 VAddr 0x00000

001080040

Using grh is forced due to the use of RoCE

Mtu : 2048

------------------------------------------------------------------

#bytes #iterations BW peak[MB/sec] BW average[MB/sec]

ib_send_bw.exe -a 15.4.12.128

------------------------------------------------------------------

Send BW Test

Inline data is used up to 400 bytes message

Connection type : RC

max inline size 28

local address: LID 0000, QPN 0x5a020000, PSN 0x36090000, RKey 0x120000 VAddr 0x00000001080040

remote address: LID 0000, QPN 0x5a020000, PSN 0xf50e0000, RKey 0x120008 VAddr 0x00000001140040

Using grh is forced due to the use of RoCE

Mtu : 2048

------------------------------------------------------------------

#bytes #iterations BW peak[MB/sec] BW average[MB/sec]

2 1000 4.95 4.26

4 1000 9.91 7.76

8 1000 19.82 15.36

16 1000 39.64 31.09

32 1000 79.28 76.59

64 1000 158.55 154.08

128 1000 317.10 311.80

256 1000 634.20 557.29

512 1000 1268.40 857.03

1024 1000 2536.80 1389.27

2048 1000 1691.20 1452.51

ndpingpong.exe s 15.4.12.128 2222 p1

Driver FeaturesRev 4.3

Mellanox Technologies

Step 2. Start the client:

2.3 Hyper-V with VMQ

Mellanox WinOF Rev 4.3 includes a virtual machine queue (VMQ) interface to support Micro-

soft Hyper-V network performance improvements and security enhancement.

VMQ interface supports:

• Classification of received packets by using the destination MAC address to route the

packets to different receive queues

• NIC ability to use DMA to transfer packets directly to a Hyper-V child-partition's

shared memory

• Scaling to multiple processors, by processing packets for different virtual machines on

different processors.

2.4 Header Data Split

The header-data split feature improves network performance by splitting the headers and data in

received Ethernet frames into separate buffers. The feature is disabled by default and can be

enabled in the Advanced tab (Performance Options) from the Properties sheet.

For further information, please refer to the MSDN library:

http://msdn.microsoft.com/en-us/library/windows/hardware/ff553723(v=VS.85).aspx

ndpingpong.exe c 15.4.12.128 2222 p1

Using 8 processors. Sender Frequency is 2597685

1 100000 3.80 98.56 262824.1

2 100000 3.80 98.68 526268.5

4 100000 3.81 102.53 1049904.8

8 100000 3.95 98.81 2023532.8

16 100000 3.97 98.29 4025846.5

32 100000 4.05 100.25 7896576.7

64 100000 4.18 101.02 15324826.5

128 100000 4.66 100.65 27483686.0

256 100000 6.46 99.14 39617564.3

512 100000 7.08 101.52 72318742.7

1024 100000 7.70 99.41 132958061.2

2048 100000 8.99 99.10 227871399.7

4096 100000 11.56 101.35 354235590.7

8192 100000 16.76 99.77 488844578.1

16384 100000 26.91 99.88 608904254.0

32768 64000 47.69 101.37 687158577.7

65536 32000 88.75 100.14 738457065.3

131072 16000 171.46 100.24 764467556.7

262144 8000 334.70 99.79 783225451.6

524288 4000 665.06 99.85 788330687.1

1048576 2000 1323.22 100.37 792441466.4

2097152 1000 2629.31 99.84 797606523.6

4194304 500 5272.73 100.16 795470318.4

When using MTU 2048, the administrator should configure the switches to support

MTU 2048 or higher

WinOF VPI for Windows User Manual Rev 4.3

Mellanox Technologies 18

2.5 Receive Side Scaling (RSS)

Mellanox WinOF Rev 4.3 IPoIB and Ethernet drivers use NDIS 6.30 new RSS capabilities. The

main changes are:

• Supports unlimited number of processors (previously 64)

• Individual network adapter RSS configuration usage

To set the RSS capability for individual adapter instead of global setting, and to improve RSS on

Windows 2012 server, set the registry keys listed in the table below:

2.6 Port Configuration

After MLNX_VPI installation, it is possible to modify the network protocol that runs on each

port of VPI adapter cards. Each port can be set to run as InfiniBand, Ethernet or Auto Sensing.

2.6.1 Auto Sensing

Auto Sensing enables the NIC to automatically sense the link type (InfiniBand or Ethernet) based

on the cable connected to the port and load the appropriate driver stack (InfiniBand or Ethernet).

Auto Sensing is performed only when rebooting the machine or after disabling/enabling the

mlx4_bus interface from the Device Manager. Hence, if you replace cables during the runtime,

the NIC will not perform Auto Sensing.

For further information on how to configure it, please refer to Section 2.6.2, “Port Protocol Con-

figuration”, on page 19.

Table 4 - Registry Keys Setting

Sub-key Description

HKLM\SYSTEM\CurrentControlSet\Con-

trol\Class\{XXXXX72-XXX}\<network adapter num-

ber>\*MaxRSSProcessors

Maximum number of CPUs allotted. Sets

the desired maximum number of processors

for each interface. The number can be differ-

ent for each interface.

Note: Restart the network adapter when you

change this registry key.

HKLM\SYSTEM\CurrentControlSet\Con-

trol\Class\{XXXXX72-XXX}\<network adapter num-

ber>\*RssBaseProcNumber

Base CPU number. Sets the desired base

CPU number for each interface. The number

can be different for each interface. This

allows partitioning of CPUs across network

adapters.

Note: Restart the network adapter when you

change this registry key.

HKLM\SYSTEM\CurrentControlSet\Con-

trol\Class\{XXXXX72-XXX}\<network adapter num-

ber>\*NumaNodeID

NUMA node affinitization

HKLM\SYSTEM\CurrentControlSet\Con-

trol\Class\{XXXXX72-XXX}\<network adapter num-

ber>\*RssBaseProcGroup

Sets the RSS base processor group for sys-

tems with more than 64 processors.

Driver FeaturesRev 4.3

Mellanox Technologies

2.6.2 Port Protocol Configuration

Step 1. Display the Device Manager and expand “System devices”.

Step 2. Right-click on the Mellanox ConnectX VPI network adapter and left-click Properties. Select

the Port Protocol tab from the Properties sheet.

The “Port Protocol” tab is displayed only if the NIC is a VPI (IB and ETH).

Page is loading ...

Asus PEM-FDR User manual

Asus PEM-FDR User manual

Other documents