Mellanox Technologies MCX4412-ACAN User manual

Brand: Mellanox Technologies

Model: MCX4412-ACAN

Type: User manual

This manual is also suitable for

Mellanox Technologies

www.mellanox.com

ConnectX®-4 Lx Single and Dual 10/25 Gb/s

Ethernet SFP28 Port Adapter Card User Manual for

Open Compute Project

P/N: MCX4421A-XCQN, MCX4411A-ACAN, MCX4412-ACAN, MCX4411A-ACQN, MCX4421A-

ACQN

Rev 2.5

Mellanox Technologies

350 Oakmead Parkway Suite 100

Sunnyvale, CA 94085

U.S.A.

www.mellanox.com

Tel: (408) 970-3400

Fax: (408) 970-3403

Mellanox®, Mellanox logo, Accelio®, BridgeX®, CloudX logo, CompustorX®, Connect-IB®, ConnectX®,

CoolBox®, CORE-Direct®, EZchip®, EZchip logo, EZappliance®, EZdesign®, EZdriver®, EZsystem®,

GPUDirect®, InfiniHost®, InfiniBridge®, InfiniScale®, Kotura®, Kotura logo, Mellanox CloudRack®, Mellanox

CloudXMellanox®, Mellanox Federal Systems®, Mellanox HostDirect®, Mellanox Multi-Host®, Mellanox Open

Ethernet®, Mellanox OpenCloud®, Mellanox OpenCloud Logo®, Mellanox PeerDirect®, Mellanox ScalableHPC®,

Mellanox StorageX®, Mellanox TuneX®, Mellanox Connect Accelerate Outperform logo, Mellanox Virtual Modular

Switch®, MetroDX®, MetroX®, MLNX-OS®, NP-1c®, NP-2®, NP-3®, NPS®, Open Ethernet logo, PhyX®,

PlatformX®, PSIPHY®, SiPhy®, StoreX®, SwitchX®, Tilera®, Tilera logo, TestX®, TuneX®, The Generation of

Open Ethernet logo, UFM®, Unbreakable Link®, Virtual Protocol Interconnect®, Voltaire® and Voltaire logo are

registered trademarks of Mellanox Technologies, Ltd.

All other trademarks are property of their respective owners.

For the most updated list of Mellanox trademarks, visit http://www.mellanox.com/page/trademarks

NOTE:

THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT(S) ) AND ITS RELATED

DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-ISﺴ WITH ALL FAULTS OF ANY

KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT

USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST

ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY

QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES

CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE

HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES , INCLUDING, BUT NOT LIMITED TO, THE

IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND

NONINFRINGEMENT ARE DISCLAIMED . IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR

ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL

DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF

SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)

HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,

OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE

PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH

DAMAGE.

Doc #: MLNX-15-5504

2Mellanox Technologies

Rev 2.5 3Mellanox Technologies

Table of Contents

Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Related Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Document Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Chapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

1.1 Product Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

1.1.1 OCP 2.0 Spec Stacking Height . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.1.1.1 OCP Spec Type 1 Stacking Height . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.2 Operating Systems/Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.3 Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

1.4 Features and Benefits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Chapter 2 Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.1 Ethernet SFP28 Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.2 PCI Express Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.3 LEDs Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Chapter 3 Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.1 System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.1.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.1.2 Operating Systems/Distributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.1.3 Software Stacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.2 Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.3 Pre-installation Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.4 Adapter Card Installation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.5 Cables and Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.5.1 Cable Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.6 Adapter Card Un-installation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.6.1 Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.6.2 Card Un-installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.7 Identify the Card in Your System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.7.1 On Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.7.2 On Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Chapter 4 Driver Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.1 Linux Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.1.1 Hardware and Software Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Rev 2.5 4Mellanox Technologies

4.1.2 Downloading Mellanox OFED. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.1.3 Installing Mellanox OFED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.1.3.1 Installation Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.1.3.2 Installation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1.3.3 Installation Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1.3.4 openibd Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1.3.5 Driver Load Upon System Boot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.1.3.6 mlnxofedinstall Return Codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.1.4 Uninstalling Mellanox OFED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.1.5 Installing MLNX_OFED using YUM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.1.5.1 Setting up MLNX_OFED YUM Repository . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.1.5.2 Installing MLNX_OFED using the YUM Tool. . . . . . . . . . . . . . . . . . . . . . . . 34

4.1.5.3 Uninstalling Mellanox OFED using the YUM Tool . . . . . . . . . . . . . . . . . . . 35

4.1.6 Installing MLNX_OFED using apt-get . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.1.6.1 Setting up MLNX_OFED apt-get Repository . . . . . . . . . . . . . . . . . . . . . . . 35

4.1.6.2 Installing MLNX_OFED using the apt-get Tool. . . . . . . . . . . . . . . . . . . . . . 36

4.1.6.3 Uninstalling Mellanox OFED using the apt-get Tool . . . . . . . . . . . . . . . . . 37

4.1.7 Updating Firmware After Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.1.7.1 Updating the Device Online . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.1.7.2 Updating the Device Manually . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.1.7.3 Updating the Device Firmware Automatically upon System Boot . . . . . 38

4.1.8 UEFI Secure Boot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

4.1.8.1 Enrolling Mellanox's x.509 Public Key On your Systems . . . . . . . . . . . . . 38

4.1.8.2 Removing Signature from Kernel Modules . . . . . . . . . . . . . . . . . . . . . . . . 39

4.1.9 Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.2 Windows Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.2.1 Hardware and Software Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.2.2 Downloading Mellanox WinOF-2 Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.2.3 Installing Mellanox WinOF-2 Driver. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.2.3.1 Attended Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.2.3.2 Unattended Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

4.2.4 Installation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

4.2.5 Extracting Files Without Running Installation. . . . . . . . . . . . . . . . . . . . . . . 48

4.2.6 Uninstalling Mellanox WinOF-2 Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.2.6.1 Attended Uninstallation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.2.7 Firmware Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.2.8 Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.3 VMware Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.3.1 Hardware and Software Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.3.2 Installing VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.3.3 Removing Previous Mellanox Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

4.3.4 Loading/Unloading Driver Kernel Modules. . . . . . . . . . . . . . . . . . . . . . . . . 53

Rev 2.5 5Mellanox Technologies

4.3.5 Firmware Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Chapter 5 Updating Adapter Card Firmware. . . . . . . . . . . . . . . . . . . . . . . . . . 55

5.1 Firmware Update (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Chapter 6 Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6.1 General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6.2 Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

6.3 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Chapter 7 Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

7.1 MCX4421A-XCQN Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

7.2 MCX4411A-ACAN Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

7.3 MCX4421A-ACAN Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

7.4 MCX4411A-ACQN Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

7.5 MCX4421A-ACQN Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

7.6 Adapter LEDs Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

7.7 Board Mechanical Drawing and Dimensions . . . . . . . . . . . . . . . . . . . . . . . 65

Appendix A Interface Connectors Pinouts . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

A.1 SFP28 Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

A.2 PCI Express x8 Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Appendix B Finding the MAC and Serial Number on the Adapter Card . . . . 70

Appendix C Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Appendix D Avertissements de sécurité d’installation (Warnings in French) 74

Appendix E Sicherheitshinweise (Warnings in German) . . . . . . . . . . . . . . . . 76

Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) 78

Rev 2.5 6Mellanox Technologies

List of Tables

Table 1: Revision History Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Table 2: Related Documents List. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Table 3: Dual Port 10Gb/s Ethernet Adapter Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Table 4: Single and Dual Port 25Gb/s Ethernet Adapter Cards . . . . . . . . . . . . . . . . . . . . . 13

Table 5: Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Table 6: Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Table 7: mlnxofedinstall Return Codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Table 8: Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Table 9: Hardware and Software Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Table 10: MCX4421A-XCQN Specifications Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

Table 11: MCX4411A-ACAN Specifications Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Table 12: MCX4421A-ACAN Specifications Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Table 13: MCX4411A-ACQN Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

Table 14: MCX4421A-ACQN Specifications Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Table 15: Physical and Logical Link Indications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Table 16: SFP28 Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Rev 2.5 7Mellanox Technologies

List of Figures

Figure 1: Type 1 Vertical Stack Front View - Single Port Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Figure 2: Type 1 Vertical Stack Front View - Dual Port Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Figure 3: PCI Device (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Figure 4: Installation Results (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Figure 5: Mechanical Drawing of Single-port Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Figure 6: Mechanical Drawing of Dual-port Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Figure 7: Rear View of Module With Pin Placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Figure 8: PCIe x8 Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Figure 9: MCX4421A-XCQN Board Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Figure 10: MCX4411A-ACAN Board Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Figure 11: MCX4421A-ACAN Board Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Figure 12: MCX4411A-ACQN Board Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Figure 13: MCX4421A-ACQN Board Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Rev 2.5 8Mellanox Technologies

Revision History

This document was printed on January 30, 2018.

Table 1 - Revision History Tables

Date Rev Comments/Changes

January 2018 2.5 Added a note to System Requirements on page 19

December 2017 2.4 Updated the following specifications:

• MCX4411A-ACQN Specifications on page 62

• MCX4421A-ACQN Specifications Table on page 63

August 2017 2.3 • Updated Related Documentation on page 10

• Updated Product Overview on page 12

• Updated OCP 2.0 Spec Stacking Heigh

t on page 14

April 2017 2.2 • Updated Adapter Card Installation Instructions on page 20 and Adapter

Card Un-installation Instructions on page 22.

September 2016 2.1 • Updated the following tables:

• MCX4421A-XCQN Specifications on page 59

• MCX4411A-ACAN Specifications on page 60

• MCX4421A-ACAN Specifications on page 61

• MCX4411A-ACQN Specifications on page 62

• MCX4421A-ACQN Specifications on p

age 63.

August 2016 2.0 • Updated Ethernet Protocol Support in the following tables:

• MCX4421A-XCQN Specifications on page 59

• MCX4411A-ACAN Specifications on page 60

• MCX4421A-ACAN Specifications on p

age 61

• MCX4411A-ACQN Specifications on page 62

• MCX4421A-ACQN Specifications on page 63

• Updated Adapter LEDs Operations on

page 64

• Added Mechanical Drawing of Single-port Cards on page 65.

July 2016 1.9 • Added MCX4421A-XCQN and updated necessary sections accordingly.

• Updated Features and Benefits on page 16.

• Updated PCI Express Interface on page 18.

• Updated Adapter Card Installation Instructions on page 20.

• Updated PCI Express x8 Connector Pinout on page 69.

June 2016 1.8 • Added Mellanox logo "25 is the New 10™

" in the cover page.

• Updated Safety Warnings on page 72.

April 2016 1.7 • Updated Product Overview on page 12.

• Updated Section 1.1.1, “OCP 2.0 Spec Stacking Height,” on page 14.

March 2016 1.6 • Removed I2C-compatible Interface from Interfaces on page 18.

• Updated Features and Benefits on page 16.

February 2016 1.5 • Updated Linux Driver on page 26.

• Updated Windows Driver on page 41.

• Updated VMware Driver on page 52.

• Updated power numbers in the following specif

ication tables:

• MCX4421A-XCQN Specifications on p

age 59

• MCX4411A-ACAN Specifications on page 60

• MCX4421A-ACAN Specifications on p

age 61

• MCX4411A-ACQN Specifications on page 62.

January 2016 1.4 • Added a note to “Finding the MAC and Serial Number on the Adapter

Card” on page 62.

Rev 2.5 9Mellanox Technologies

December 2015 1.3 • Updated Related Documentation on page 9.

• Updated Product Overview on page 12.

• Updated speed specifications in the following sections:

• Features and Benefits on page 13.

• Connectivity on page 14.

• Updated For further information on Linux performance, please refer to the

Performance Tuning Guide for Mellanox Network

Adapters. on page 33.

• Updated Specifications on page 54.

• Updated dimensions in Mechanical Drawing of the Dual-Port Adapter

Cards on page 58.

• Added SFP28 Connector Pinout on page 60.

• Added MCX4421A-ACQN Board Label on page 63 and MCX4411A-

ACQN Board Label on page 63.

December 2015 1.2 • Added MCX4411A-ACQN and updated necessary sections accord

ingly.

• Updated Related Documentation on page 9.

• Updated On Windows on page 21.

• Added VMware Driver on page 47.

• Updated the following Spec

ifications tables:

• MCX4411A-ACAN Specification

s Table on page 53.

• MCX4421A-ACAN Specifications Table on page 54.

• MCX4421A-ACQN Specifications Table on page 56.

• Updated Adapter LEDs Operations on

page 64.

• Updated Updating Adapter Card Firmware on page 50.

November 2015 1.1 Added MCX4421A-ACQN and updated necessary sections accordingly.

November 2015 1.0

First Release

Table 1 - Revision History Tables

Date Rev Comments/Changes

Rev 2.5 10Mellanox Technologies

About this Manual

This User Manual describes Mellanox Technologies ConnectX®-4 Lx 10/25 Gb/s Ethernet Sin-

gle and Dual SFP28 port PCI Express x8 adapter cards for Open Compute Project, spec 2.0. It

provides details as to the interfaces of the board, specifications, required software and firmware

for operating the board, and relevant documentation.

Intended Audience

This manual is intended for the installer and user of these cards.

The manual assumes basic familiarity with Ethernet network and architecture specifications.

Related Documentation

Table 2 - Related Documents List

Mellanox Firmware Tools (MFT) User

Manual

Document no. 2204UG

User Manual describing the set of MFT firmware management

tools for a single node.

See http://www.mellanox.com => Products => Software => Firm-

ware Tools

Mellanox Firmware Utility (mlxup) User

Manual and Release Notes

Mellanox firmware update and query utility used to update the

firmware.

See http://www.mellanox.com => Products => Software => Firm-

ware Tools => mlxup Firmware Utility

Mellanox OFED for Linux

User Manual

Document no. 2877

User Manual describing OFED features, performance, tools con-

tent and configuration.

See http://www.mellanox.com => Products => Software => Ether-

net Drivers => Mellanox OpenFabrics Enterprise Distribution for

Linux (MLNX_OFED)

WinOF-2 for Windows

User Manual

Document no: MLNX-15-3280

User Manual describing WinOF-2 features, performance, tools

content and configuration.

See http://www.mellanox.com => Products => Software => Ether-

net Drivers

WinOF-2 for Windows

Release Notes

Release notes for Mellanox Technologies WinOF-2 for Ethernet

drivers for Mellanox adapter cards:

See http://www.mellanox.com => Products => Software => Ether-

net Drivers

Mellanox MLX4_EN Driver for VMware

README

Document no. 3527

User Manual describing MLX4_EN driver for VMware features,

performance, diagnostic, tools content and configuration.

See http://www.mellanox.com => Products => Software => Ether-

net Drivers => VMware Drivers

Mellanox EN for Linux Driver Release

Notes

Release notes for Mellanox Technologies' MLNX_EN for Linux

driver kit for Mellanox adapter cards:

See http://www.mellanox.com => Products => Software => Ether-

net Drivers => Mellanox EN for Linux

Rev 2.5 11Mellanox Technologies

Document Conventions

When discussing memory sizes, MB and MBytes are used in this document to mean size in mega

Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is

used to mean PCI Express

Mellanox VMware for Ethernet User Man-

ual

Document no. MLNX-15-4896

User Manual describing the various components of the Mellanox

ConnectX®-4 NATIVE ESXi stack:

See http://www.mellanox.com => Products => Software => Ether-

net Drivers => VMware Driver => User Manual v4.15.2.0

Mellanox VMware for Ethernet Release

Notes

Release notes for Mellanox ConnectX®-4 NATIVE ESXi stack:

See http://www.mellanox.com => Products => Software => Ether-

net Drivers => VMware Driver => Release Notes

IEEE Std 802.3 Specification This is the IEEE Ethernet specification

http://standards.ieee.org/getieee802

PCI Express 3.0 Specifications Industry Standard PCI Express 3.0 Base and Card Electromechan-

ical Specifications

https://pcisig.com/specifications

Open Compute Project 2.0 Specification OCP Spec 2.0

Table 2 - Related Documents List

Introduction

Rev 2.5

12Mellanox Technologies

1 Introduction

This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the Con-

nectX®-4 Lx integrated circuit device. These adapters connectivity provide the highest perform-

ing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data

Centers, High-Performance Computing, and Embedded environments.

This chapter covers the following topics:

• Section 1.1, “Product Overview,” on page 12

• Section 1.2, “Operating Systems/Distributions,” on page 14

• Section 1.3, “Connectivity,” on page 15

• Section 1.4, “Features and Benefits,” on page 16

1.1 Product Overview

The following section provides the ordering part number, port speed, number of ports, and PCI

Express speed.

Table 3 - Dual Port 10Gb/s Ethernet Adapter Card

Ordering Part Number (OPN)

MCX4421A-XCQN - Dual-port card with Host Management

support complying with OCP type1 stacking height

a. See Section 1.1.1, “OCP 2.0 Spec Stacking Height,” on page 14

Data Transmission Rate

Ethernet: 1/10 Gb/s

Network Connector Types

Dual-port SFP28

PCI Express (PCIe) SerDes Speed

PCIe 3.0 x8 8GT/s

RoHS

Adapter IC Part Number

MT27712A0-FDCF-XE

Device ID (decimal)

4117 for Physical Function (PF)

4118 for Virtual Function (VF)

Introduction

Rev 2.5

13Mellanox Technologies

Table 4 - Single and Dual Port 25Gb/s Ethernet Adapter Cards

Ordering Part Number (OPN)

Cards complying with OCP Type1 stacking heighta:

MCX4411A-ACAN - Single-port card

MCX4421A-ACAN - Dual-port card

MCX4411A-ACQN - Single-port card with Host Management support

MCX4421A-ACQN - Dual-port card with Host Management support

a. See OCP 2.0 Spec Stacking Height on page 14

Data Transmission Rate

Ethernet: 1/10/25 Gb/s

Network Connector Types

Single and dual-port SFP28

PCI Express (PCIe) SerDes

Speed

PCIe 3.0 x8 8GT/s

RoHS

Adapter IC Part Number

MT27711A0-FDCF-AE - single-port cards

MT27712A0-FDCF-AE - dual-port cards

Device ID (decimal)

4117 for Physical Function (PF)

4118 for Virtual Function (VF)

Introduction

Rev 2.5

14Mellanox Technologies

1.1.1 OCP 2.0 Spec Stacking Height

1.1.1.1 OCP Spec Type 1 Stacking Height

Single and dual port 10/25Gb/s Ethernet adapter cards comply with OCP 2.0 Type 1 with 8mm

stacking height. See

Figure 1 for the single port front view and Figure 2 for the dual port card

front view.

Figure 1: Type 1 Vertical Stack Front View - Single Port Cards

Figure 2: Type 1 Vertical Stack Front View - Dual Port Cards

1.2 Operating Systems/Distributions

• RHEL/CentOS

• FreeBSD

•VMware

• OpenFabrics Enterprise Distribution (OFED)

• OpenFabrics Windows Distribution (WinOF-2)

Introduction

Rev 2.5

15Mellanox Technologies

1.3 Connectivity

• Interoperable with 1/10/25 Gb/s Ethernet switches

• Passive copper cable with ESD protection

• Powered connectors for optical and active cable support

Introduction

Rev 2.5

16Mellanox Technologies

1.4 Features and Benefits

Table 5 - Features

PCI Express (PCIe)

Uses PCIe Gen 3.0 (1.1 and 2.0 compatible) through an x8 FCI B2Bconnector up to 8GT/s

Up to 25/50 Gigabit

Ethernet

Mellanox adapters comply with the following IEEE 802.3 standards:

– 25GbE / 10GbE / 1GbE

– 25G Ethernet Consortium 25

– IEEE 802.3by 25 Gigabit Ethernet

– IEEE 802.3ae 10 Gigabit Ethernet

– IEEE 802.3az Energy Ef

ficient Ethernet

– IEEE 802.3ap based auto-negotiation and KR startup

– IEEE 802.3ad, 802.1AX Link

Aggregation

– IEEE 802.1Q, 802.1P VLAN tags and priority

– IEEE 802.1Qau (QCN)

– Congestion Notification

– IEEE 802.1Qaz (ETS)

– IEEE 802.1Qbb (PFC)

– IEEE 802.1Qbg

– IEEE 1588v2

– Jumbo frame support (9.6KB)

Memory

PCI Express - stores and accesses Ethernet fabric connection information and packet data.

SPI - includes one 16MB SPI Flash device (W25Q128FVSIG device by WINBOND-

NUVOTON).

Overlay Networks

In order to better scale their networks, data center operators often create overlay networks

that carry traffic from individual virtual machines over logical tunnels in encapsulated for-

mats such as NVGRE and VXLAN. While this solves network scalability issues, it hides

the TCP packet from the hardware offloading engines, placing higher loads on the host

CPU. ConnectX-4 Lx effectively addresses this by providing advanced NVGRE and

VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay proto-

col header as well as offloads TCP stateless activities on the encapsulated packet.

RDMA and RDMA over

Converged Ethernet (RoCE)

ConnectX-4 Lx, utilizing IBTA RoCE (RDMA over Converged Ethernet) technology,

delivers low-latency and high-performance over Ethernet networks. Leveraging data center

bridging (DCB) capabilities as well as ConnectX-4 Lx advanced congestion control hard-

ware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and

Layer 3 networks.

Mellanox PeerDirect™

PeerDirect™ communication provides high efficiency RDMA access by eliminating

unnecessary internal data copies between components on the PCIe bus (for example, from

GPU to CPU), and therefore significantly reduces application run time. ConnectX-4 Lx

advanced acceleration technology enables higher cluster efficiency and scalability to tens

of thousands of nodes.

CPU Offload

Adapter functionality enabling reduced CPU overhead allowing more available CPU for

computation tasks.

Quality of Service (QoS)

Support for port-based Quality of Service enabling various application requirements for

latency and SLA.

Hardware-based I/O

Virtualization

ConnectX-4 Lx provides dedicated adapter resources and guaranteed isolation and protec-

tion for virtual machines within the server.

Storage Acceleration

A consolidated compute and storage network achieves significant cost-performance advan-

tages over multi-fabric

networks. Standard block and

file access protocols can lev

erage RDMA for high-performance storage access.

Introduction

Rev 2.5

17Mellanox Technologies

SR-IOV

ConnectX-4 Lx SR-IOV technology provides dedicated adapter resources and guaranteed

isolation and protection for virtual machines (VM) within the server.

NC-SI

The adapter supports a slave Network Controller Sideband Interface (NC-SI) that can be

connected to a BMC. Enabled for MCX4421A-XCQN, MCX4411A-ACQN and MCX-

4421A-ACQN.

Wake-on-LAN (WoL)

Supported in MCX4421A-XCQN, MCX4411A-ACQN and MCX4421A-ACQN.

Reset-on-Lan (RoL)

Supported in MCX4421A-XCQN, MCX4411A-ACQN and MCX4421A-ACQN.

a. This section describes hardware features and capabilities. Please refer to the driver release notes for feature availability. See

Table 2, “Related Documents List,” on page 10

Table 5 - Features

Interfaces

Rev 2.5

18Mellanox Technologies

2 Interfaces

Each adapter card includes the following interfaces:

• “Ethernet SFP28 Interface”

• “PCI Express Interface”

• “LEDs Interface”

The adapter cards include special circuits to protect from ESD shocks to the card/server when

plugging copper cables.

2.1 Ethernet SFP28 Interface

The network ports of the ConnectX®-4 Lx adapter cards are compliant with the IEEE 802.3

Ethernet standards listed in

Table 5, “Features,” on page 16.The SFP28 port has one Tx/Rx pair

of SerDes. Ethernet traffic is transmitted through the cards' SFP28 connectors.

2.2 PCI Express Interface

The ConnectX®-4 Lx adapter cards support PCI Express 3.0 (1.1 and 2.0 compatible) through an

x8connector. The device can be either a master initiating the PCI Express bus operations or a

slave responding to PCI bus operations. The following lists PCIe FCI B2B connector interface

features:

• PCIe Gen 3.0 compliant, 1.1 and 2.0 compatible

• 2.5, 5.0, or 8.0GT/s link rate x8

• Auto-negotiates to x8, x4, x2, or x1

• Support for MSI/MSI-X mechanisms

2.3 LEDs Interface

There are two I/O LEDs per port. For LEDs specifications, please refer to Section 7.6,

“Adapter LEDs Operations,” on page 64.

Hardware Installation

Rev 2.5

19Mellanox Technologies

3 Hardware Installation

3.1 System Requirements

3.1.1 Hardware

A system with a PCI Express x8 FCI B2B connector is required for installing the card.

3.1.2 Operating Systems/Distributions

Please refer to Section 1.2, “Operating Systems/Distributions,” on page 14.

3.1.3 Software Stacks

Mellanox OpenFabric software package - MLNX_OFED for Linux, WinOF-2 for Windows and

VMware. See

Chapter 4,“Driver Installation” on page 26.

3.2 Safety Precautions

1. Remove any metallic objects from your hands and wrists.

2. Make sure to use only insulated tools.

3. Verify that the system is p

owered off and is unplugged.

4. It is strongly recommended to us

e an ESD strap or other antistatic devices.

3.3 Pre-installation Checklist

1. Verify that your system meets the hardware and software requirements stated above.

2. Shut down your system if active.

3. After shutting down the system, turn of

f power and unplug the cord.

4. Remove the c

ard from its package.

5. Please note that the card must be placed on an antistatic surface.

6. Check the card for visible signs of damage. Do not attempt to install the card if damaged.

Unless otherwise specified, Mellanox products are designed to work in an environ-

mentally controlled data center with low levels of gaseous and du

st (particulate) con-

tamination.

The operation environment should meet severity level G1 as per ISA

71.04 for gaseous

contamination and ISO 14644-1 cl

ass 8 for cleanliness level.

The adapter is being installed in a system that operates with voltages that can be lethal.

Before opening the case of the system, observe the following precautions to avoid

injury and prevent damage to system components.

Hardware Installation

Rev 2.5

20Mellanox Technologies

3.4 Adapter Card Installation Instructions

1. Before installing the card, make sure that the system is off and the power cord is not con-

nected to the server. Please follow proper electrical grounding procedures.

2. Open the system case.

3. Locate an available PCI Express slot for the adapter card.Make sure the adapter clips or

screws are open.

4. Place the adapter card on the clips without applying any pressure.

Please note that the following images are for illustration purposes only.

A lesser width adapter can be seated into a greater width slot (x8 in a x16), but a

greater width adapter cannot be seated into a lesser width slot (x16 in a x8). Align the

adapter connector edge with the PCI Express connector slot.

Open Clips

Page is loading ...

Mellanox Technologies MCX4412-ACAN User manual

Brand: Mellanox Technologies

Model: MCX4412-ACAN

Type: User manual

This manual is also suitable for

Download PDF

report

Mellanox Technologies MCX4412-ACAN User manual

This manual is also suitable for

Mellanox Technologies MCX4412-ACAN User manual

Related papers

Other documents