Nvidia Mellanox ConnectX-6 Dx Series User manual

Brand: Nvidia

Model: Mellanox ConnectX-6 Dx Series

Type: User manual

Exported onOct/22/2020 06:45 PM

https://docs.mellanox.com/x/esV-AQ

NVIDIA Mellanox ConnectX-6 Dx PCIe HHHL

Ethernet Adapter Cards User Manual

2
Table of Contents
Table of Contents .......................................................................................................2
Introduction................................................................................................................9
Product Overview .................................................................................................................... 9
Features and Benefits........................................................................................................... 11
Operating Systems/Distributions ......................................................................................... 13
Connectivity ........................................................................................................................... 13
Manageability ........................................................................................................................ 14
Interfaces .................................................................................................................15
ConnectX-6 Dx IC Interface.................................................................................................. 16
Encryption......................................................................................................................... 16
PCI Express Interface ........................................................................................................... 16
Networking Ports LEDs Interface ........................................................................................ 16
Scheme 1: One Bi-Color LED .......................................................................................... 17
Scheme 3: One Bi-Color LED and one Single Color LED............................................... 17
Voltage Regulators................................................................................................................ 18
Heatsink................................................................................................................................. 18
Hardware Installation ..............................................................................................19
Safety Warnings..................................................................................................................... 19
Installation Procedure Overview........................................................................................... 19
System Requirements........................................................................................................... 20
Hardware Requirements ................................................................................................. 20
Airflow Requirements .................................................................................................20
Software Requirements ................................................................................................... 20
Safety Precautions ................................................................................................................ 20
Unpacking the Package ........................................................................................................ 21
Pre-Installation Checklist..................................................................................................... 21
Bracket Replacement Instructions ...................................................................................... 22
Installation Instructions........................................................................................................ 22
Uninstalling the Card ............................................................................................................ 24
Cables and Modules.............................................................................................................. 25
Identifying the Card in Your System ..................................................................................... 25
Uninstalling the Card ............................................................................................................ 26

3
Driver Installation ....................................................................................................28
Linux Driver Installation........................................................................................................ 28
Prerequisites .................................................................................................................... 28
Downloading Mellanox OFED .......................................................................................... 28
Installing Mellanox OFED ................................................................................................ 29
Installation Script ........................................................................................................ 29
Installation Procedure ................................................................................................30
Installation Results .....................................................................................................32
Installation Logs..........................................................................................................33
openibd Script..............................................................................................................33
Driver Load Upon System Boot ..................................................................................33
mlnxofedinstall Return Codes....................................................................................34
Uninstalling MLNX_OFED...........................................................................................34
Installing MLNX_OFED Using YUM ................................................................................. 34
Setting up MLNX_OFED YUM Repository................................................................... 34
Installing MLNX_OFED Using the YUM Tool ..............................................................35
Uninstalling MLNX_OFED Using the YUM Tool .........................................................36
Installing MLNX_OFED Using apt-get Tool..................................................................... 36
Setting up MLNX_OFED apt-get Repository ..............................................................37
Installing MLNX_OFED Using the apt-get Tool..........................................................37
Uninstalling MLNX_OFED Using the apt-get Tool.....................................................38
Updating Firmware After Installation.............................................................................. 38
Updating the Device Online.........................................................................................38
Updating the Device Manually ....................................................................................38
Updating the Device Firmware Automatically upon System Boot ............................39
UEFI Secure Boot............................................................................................................. 39
Enrolling Mellanox's x.509 Public Key on Your Systems...........................................39
Removing Signature from kernel Modules ................................................................ 40
Performance Tuning ........................................................................................................ 41
Windows Driver Installation.................................................................................................. 41
Software Requirements .................................................................................................. 41
Downloading Mellanox WinOF-2 Driver .......................................................................... 41
Installing Mellanox WinOF-2 Driver ............................................................................... 42
Attended Installation...................................................................................................42

4
Unattended Installation...............................................................................................47
Installation Results .....................................................................................................48
Uninstalling Mellanox WinOF-2 Driver............................................................................ 48
Attended Uninstallation ..............................................................................................48
Unattended Uninstallation..........................................................................................48
Extracting Files Without Running Installation ................................................................ 49
Firmware Upgrade ........................................................................................................... 51
VMware Driver Installation ................................................................................................... 51
Hardware and Software Requirements........................................................................... 51
Installing Mellanox NATIVE ESXi Driver for VMware vSphere........................................ 52
Removing Earlier Mellanox Drivers................................................................................. 52
Firmware Programming .................................................................................................. 53
Updating Adapter Firmware ....................................................................................54
Troubleshooting .......................................................................................................55
GeneralTroubleshooting ...................................................................................................... 55
LinuxTroubleshooting .......................................................................................................... 55
WindowsTroubleshooting..................................................................................................... 56
Specifications ...........................................................................................................57
MCX621102A[C/E/N]-ADAT Specifications........................................................................... 57
MCX623102A[C/N]-ADATSpecifications .............................................................................. 58
MCX623102A[C/N/S/E]-GDAT Specifications ...................................................................... 59
MCX623106A[C/N/S/E]-CDATSpecifications ....................................................................... 60
MCX623106P[C/N/E]-CDATSpecifications........................................................................... 61
MCX623105A[N/E]-CDATSpecifications .............................................................................. 62
MCX623105A[C/N/S/E]-VDATSpecifications........................................................................ 63
Dual-Port SFP28/SFP56 x8 Adapter Cards Mechanical Drawing................................. 64
Dual-PortSFP28/SFP56 x16 Adapter Cards Mechanical Drawing............................... 64
Single-Port QSFP56 x16 Adapter Cards Mechanical Drawing...................................... 65
Dual-Port QSFP56 x16 Adapter Cards Mechanical Drawing ........................................ 65
Bracket Mechanical Drawing ............................................................................................... 65
Dual-Port SFP28/SFP56 Adapter Card ........................................................................... 66
Short Bracket ..............................................................................................................66
Tall Bracket ................................................................................................................. 66
Single-Port QSFP56 Adapter Card .................................................................................. 66

5
Short Bracket ..............................................................................................................66
Tall Bracket ................................................................................................................. 66
Dual-Port QSFP56 Adapter Card..................................................................................... 67
Short Bracket ..............................................................................................................67
Tall Bracket ................................................................................................................. 67
Finding the MAC on the Adapter Card......................................................................68
Document Revision History ......................................................................................71

About This Manual

This User Manual describes NVIDIA® Mellanox®ConnectX®-6 Dx Ethernet adapter cards. It provides

details as to the interfaces of the board, specifications, required software and firmware for operating

the board, and relevant documentation.

Ordering Part Numbers

The table below provides the ordering part numbers (OPN) for the available ConnectX-6 Dx Ethernet

adapter cards.

Configuration OPN Marketing Description

25GbE PCIe

OPNs

MCX621102AC-

ADAT

ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0 x8,

Crypto and Secure Boot, Tall Bracket

MCX621102AN-

ADAT

ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0 x8,

No Crypto, Tall Bracket

MCX621102AE-

ADAT

ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0 x8,

Crypto, No Secure Boot, Tall Bracket

MCX623102AC-

ADAT

ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0

x16, Crypto and Secure Boot, Tall Bracket

MCX623102AN-

ADAT

ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0

x16, No Crypto, Tall Bracket

50GbE OPNs MCX623102AC-

GDAT

ConnectX®-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe 4.0

x16, Crypto and Secure Boot, Tall Bracket

MCX623102AN-

GDAT

ConnectX®-6 Dx EN adapter card, 50GbE , Dual-port SFP56, PCIe 4.0

x16, No Crypto, Tall Bracket

MCX623102AS-

GDAT

ConnectX®-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe 4.0

x16, Secure Boot, No Crypto, Tall Bracket

MCX623102AE-

GDAT

ConnectX®-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe 4.0

x16, Crypto, No Secure Boot, Tall Bracket

100GbE OPNs MCX623105AN-

CDAT

ConnectX®-6 Dx EN adapter card, 100GbE, Single-port QSFP56, PCIe

4.0 x16, No Crypto, Tall Bracket

MCX623105AE-

CDAT

ConnectX®-6 Dx EN adapter card, 100GbE, Single-port QSFP56, PCIe

4.0 x16, Crypto, No Secure Boot, Tall Bracket

MCX623106AC-

CDAT

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0

x16, Crypto and Secure Boot, Tall Bracket

MCX623106AN-

CDAT

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0

x16, No Crypto, Tall Bracket

MCX623106AS-

CDAT

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0

x16, Secure Boot, No Crypto, Tall Bracket

MCX623106AE-

CDAT

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0

x16, Crypto, No Secure Boot, Tall Bracket

100GbE with PPS

In/Out OPNs

MCX623106PC-

CDAT

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with PPS

In/Out, PCIe 4.0 x16, Crypto and Secure Boot, Tall Bracket

MCX623106PN-

CDAT

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with PPS

In/Out, PCIe 4.0 x16, No Crypto, Tall Bracket

MCX623106PE-

CDAT

ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with PPS

In/Out, PCIe 4.0 x16, Crypto, No Secure Boot, Tall Bracket

•

Configuration OPN Marketing Description

200GbE OPNs MCX623105AC-

VDAT

ConnectX®-6 Dx EN adapter card, 200GbE, Single-port QSFP56, PCIe

4.0 x16, Crypto and Secure Boot, Tall Bracket

MCX623105AN-

VDAT

ConnectX®-6 Dx EN adapter card, 200GbE , Single-port QSFP56, PCIe

4.0 x16, No Crypto, Tall Bracket

MCX623105AS-

VDAT

ConnectX®-6 Dx EN adapter card, 200GbE, Single-port QSFP56, PCIe

4.0 x16, Secure Boot, No Crypto, Tall Bracket

MCX623105AE-

VDAT

ConnectX®-6 Dx EN adapter card, 200GbE, Single-port QSFP56, PCIe

4.0 x16, Crypto, No Secure Boot, Tall Bracket

Intended Audience

This manual is intended for the installer and user of these cards.The manual assumes basic familiarity

with Ethernet network and architecture specifications.

Technical Support

Customers who purchased Mellanox products directly from Mellanox are invited to contact usthrough

the following methods:

URL:http://www.mellanox.com> Support

E-mail:[email protected]

Tel: +1.408.916.0055

Customers who purchased Mellanox M-1 Global Support Services, please see your contract fordetails

regarding Technical Support.

Customers who purchased Mellanox products through a Mellanox approved reseller should first

seekassistance through their reseller.

Related Documentation

Mellanox OFED for

Linux User Manual

and Release Notes

User Manual describing OFED features, performance, band diagnostic, tools

content and configuration. SeeMellanox OFED for Linux Documentation.

WinOF-2 for

WindowsUser Manual

and Release Notes

User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools

content and configuration. SeeWinOF-2 for Windows Documentation.

Mellanox VMware for

Ethernet User Manual

and Release Notes

User Manual describing the various components of the Mellanox ConnectX®

NATIVE ESXi stack. See http://www.mellanox.comProducts > Software > Ethernet

Drivers > VMware Driver

Mellanox Firmware

Utility (mlxup) User

Manual and Release

Notes

Mellanox firmware update and query utility used to update the firmware. Seehttp:/

/www.mellanox.com> Products > Software > Firmware Tools >mlxup Firmware

Utility

Mellanox Firmware

Tools (MFT) User

Manual

User Manual describing the set of MFT firmware management tools for a single

node. SeeMFT User Manual.

IEEE Std 802.3

Specification

IEEE Ethernet specification at http://standards.ieee.org/

PCI Express

Specifications

Industry Standard PCI Express Base and Card Electromechanical Specifications at

https://pcisig.com/specifications

Mellanox LinkX

Interconnect

Solutions

Mellanox LinkX Ethernet cables and transceivers are designed to maximize the

performance of High-Performance Computing networks, requiring high-

bandwidth, low-latency connections between compute nodes and switch nodes.

Mellanox offers one of industry’s broadest portfolio 40Gb/s, 56Gb/s, 100Gb/s and

200Gb/s cables, including Direct Attach Copper cables (DACs), copper splitter

cables, Active Optical Cables (AOCs) and transceivers in a wide range of lengths

from 0.5m to 10km.Mellanox tests every product in an end-to-end environment

ensuring a Bit Error Rate of less than 1E-15.Read more at https://

www.mellanox.com/products/interconnect/infiniband-overview.php

Document Conventions

When discussing memory sizes, MB and MBytes are used in this document to mean size in mega

Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to

mean PCI Express.

Revision History

A list of the changes made to this document are provided inDocument Revision History.

Introduction

Product Overview

This is the

User Manual

for Mellanox Technologies Ethernet adapter cards based on the ConnectX®-6

Dx integrated circuit device.

As the world's most advanced cloud SmartNIC, ConnectX-6 Dx provides up to two ports of 25, 50 or

100Gb/s or a single-port of 200Gb/s Ethernet connectivity, powered by 50Gb/s PAM4 SerDes technology

and PCIe Gen 4.0 host connectivity.ConnectX-6 Dx continues among Mellanox's innovation path in

scalable cloud fabrics, delivering unparalleled performance and efficiency at every scale. ConnectX-6

Dx's innovative hardware offload engines, including IPsec and TLS inline data-in-motion encryption, are

ideal for enabling secure network connectivity in modern data-center environments. Please refer toFe

ature and Benefits for more details.

Mellanox ConnectX-6 Dx 25GbE Adapter Cards

Part Number MCX621102AC-

ADAT

MCX621102AE-

ADAT

MCX621102AN-

ADAT

MCX623102AC-

ADAT

MCX623102AN-

ADAT

Form Factor/

Dimensions

5.59in. x 2.71in (142.00mm x 68.90mm)

Data Transmission

Rate

Ethernet: 1/10/25 Gb/s

Network Connector

Type

Dual-port

SFP28

Dual-port

SFP28

Dual-port SFP28 Dual-port

SFP28

Dual-port SFP28

PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x8 PCIe Gen 4.0 SERDES @ 16.0GT/s

x16

Crypto √ √ - √ -

Secure Boot √ - - √ -

Adapter IC Part

Number

MT28928A0-

CCCF-CE

MT28928A0-

NCCF-CE

MT28928A0-

NCCF-CE

MT28928A0-

CCCF-CE

MT28928A0-

NCCF-CE

RoHS RoHS Compliant

Mellanox ConnectX-6 Dx 50GbE Adapter Cards

Part Number MCX623102AC-GDAT MCX623102AN-GDAT MCX623102AS-GDAT

Form Factor/Dimensions 5.59in. x 2.71in (142.00mm x 68.90mm)

Data Transmission Rate Ethernet: 1/10/25/40/50 Gb/s

Network Connector Type Dual-port SFP56

PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x16

Crypto √ √ - -

Secure Boot √ - - √

Adapter IC Part Number MT28928A0-CCCF-CE MT28928A0-

CCCF-CE

MT28928A0-NCCF-CE MT28928A0-NCCF-CE

RoHS RoHS Compliant

Mellanox ConnectX-6 Dx 100GbE Adapter Cards

Part Number MCX623105A

N-CDAT

MCX623105A

E-CDAT

MCX623106A

E-CDAT

MCX623106A

C-CDAT

MCX623106A

N-CDAT

MCX623106A

S-CDAT

Form Factor/

Dimensions

5.59in. x 2.71in (142.00mm x 68.90mm)

Data

Transmission

Rate

Ethernet: 1/10/25/40/50/100 Gb/s

Network

Connector Type

Single-port

QSFP56

Single-port

QSFP56

Dual-port

QSFP56

Dual-port

QSFP56

Dual-port

QSFP56

Dual-port

QSFP56

PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x16

Crypto - √ √ √ - -

Secure Boot - - - √ - √

Adapter IC Part

Number

MT28928A0-

XCCF-CE

MT28928A0-

XCCF-CE

MT28928A0-

CCCF-CE

MT28928A0-

CCCF-CE

MT28928A0-

NCCF-CE

MT28928A0-

NCCF-CE

RoHS RoHS Compliant

Mellanox ConnectX-6 Dx 100GbE Adapter Cards withPPS In/Out SMAs

Part Number MCX623106PN-CDAT MCX623106PE-CDAT MCX623106PC-CDAT

Form Factor/Dimensions 5.59in. x 2.71in (142.00mm x 68.90mm)

Data Transmission Rate Ethernet: 1/10/25/40/50/100 Gb/s

Network Connector Type Dual-port QSFP56 Dual-port QSFP56 Dual-port QSFP56

PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x16

Crypto - √ √

Secure Boot - - √

PPS In/Out SMAs √ √ √

Adapter IC Part Number MT28928A0-CCCF-CE MT28928A0-NCCF-CE MT28928A0-NCCF-CE

RoHS RoHS Compliant

Mellanox ConnectX-6 Dx 200GbE Adapter Cards

Part Number MCX623105AC-

VDAT

MCX623105AE-

VDAT

MCX623105AN-

VDAT

MCX623105AS-

VDAT

Form Factor/

Dimensions

5.59in. x 2.71in (142.00mm x 68.90mm)

Data Transmission

Rate

Ethernet: 1/10/25/40/50/100/200 Gb/s

Network Connector

Type

Single-port QSFP56

PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x16

Crypto √ √ - -

Secure Boot √ - - √

Adapter IC Part

Number

MT28928A0-CCCF-

MT28928A0-NCCF-

RoHS RoHS Compliant

For more detailed information seeSpecifications.

Features and Benefits

Feature Description

PCI

Express

(PCIe)

PCIe Gen 4.0 SERDES@ 8.0GT/s / 16.0GT/s through x8/x16 Edge Connector

This section describes hardware features and capabilities. Please refer to the relevant driver

and/or firmware release notes for feature availability.



•

Feature Description

Mellanox adapters comply with the following IEEE 802.3 standards:

200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE

– IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet

– IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes

– IEEE 802.3ba 40 Gigabit Ethernet

– IEEE 802.3by 25 Gigabit Ethernet

– IEEE 802.3ae 10 Gigabit Ethernet

– IEEE 802.3ap based auto-negotiation and KR startup

– IEEE 802.3ad, 802.1AX Link Aggregation

– IEEE 802.1Q, 802.1P VLAN tags and priority

– IEEE 802.1Qau (QCN)

– Congestion Notification

– IEEE 802.1Qaz (ETS)

– IEEE 802.1Qbb (PFC)

– IEEE 802.1Qbg

– IEEE 1588v2

– Jumbo frame support (9.6KB)

Memory PCI Express - stores and accesses Ethernet fabric connection information and packet data.

SPI Quad - includes 256Mbit SPI Quad Flash device (MX25L25645GXDI-08G device by

Macronix)

FRU EEPROM - Stores the parameters and personality of the card. The EEPROM capacity

is 128Kbit. FRU I2C address is (0x50) and is accessible through the PCIe SMBus(Available

only in QSFP cards).

Overlay

Networks

In order to better scale their networks, data center operators often create overlay networks that

carry traffic from individual virtual machines over logical tunnels in encapsulated formats such as

NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the

hardware offloading engines, placing higher loads on the host CPU. Mellanox ConnectX-6 Dx

effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines

that encapsulate and de-capsulate the overlay protocol.

RDMA

overConve

rged

Ethernet

(RoCE)

Mellanox ConnectX-6 Dx, utilizing RoCE (RDMA over Converged Ethernet) technology, delivers

low-latency and high-performance over Band and Ethernet networks. Leveraging data center

bridging (DCB) capabilities as well as Mellanox ConnectX-6 Dx advanced congestion control

hardware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and

Layer 3 networks.

Mellanox

PeerDirect

Mellanox PeerDirect® communication provides high-efficiency RDMA access by eliminating

unnecessary internal data copies between components on the PCIe bus (for example, from GPU to

CPU), and therefore significantly reduces application run time. Mellanox ConnectX-6 Dx advanced

acceleration technology enables higher cluster efficiency and scalability to tens of thousands of

nodes.

CPU

Offload

Adapter functionality enabling reduced CPU overhead allowing more available CPU for

computation tasks.

Open vSwitch (OVS) offload using ASAP

- Accelerated Switch and Packet

• Flexible match-action flow tables

• Tunnelingencapsulation / decapsulation

Quality of

Service

(QoS)

Support for port-based Quality of Service enabling various application requirements for latency

and SLA.

Hardware-

based I/

OVirtualiza

tion

Mellanox ConnectX-6 Dx provides dedicated adapter resources and guaranteed isolation and

protection for virtual machines within the server.

•

Feature Description

Storage

Acceleratio

A consolidated compute and storage network achieves significant cost-performance advantages

over multi-fabric networks. Standard block and file access protocols can leverage

• RDMA for high-performance storage access

• NVMe over Fabric offloads for target machine

• Erasure Coding

• T10-DIF Signature Handover

SR-IOV Mellanox ConnectX-6 Dx SR-IOV technology provides dedicated adapter resources and guaranteed

isolation and protection for virtual machines (VM) within the server.

High-

Performan

ceAccelera

tions

• Tag Matching and Rendezvous Offloads

• Adaptive Routing on Reliable Transport

• Burst Buffer Offloads for Background Checkpointing

PPS In/Out

SMAs

Mellanox offers a full IEEE 1588v2 PTP software solution, as well as time-sensitive related

features called “5T”. Mellanox PTP and 5T software solution are designed to meet the most

demanding PTP profiles. ConnectX-6 Dx incorporates an integrated Hardware Clock (PHC) that

allows ConnectX-6 Dx to achieve sub 20u Sec accuracy and also offers many timing related

functions such as time-triggered scheduling or time-based SND accelerations (time-based

ASAP²) . Furthermore, the 5T technology enables the software application to transmit fronthaul

(ORAN) compatible in high bandwidth. The PTP part supports slave clock, master clock, and

boundary clock.

ConnectX-6 Dx PTP solution allows you to run any PTP stack on your host.

With respect to testing and measurements, selected Mellanox adapters allow you to use the PPS-

out signal from onboard SMA connecter, ConnectX-6 Dx also allows to measure PTP in scale, with

PPS-In signal. The PTP HW clock on the Network adapter will be sampled on each PPS-In signal,

and the timestamp will be sent to the SW.

Enabled in MCX623106PN-CDAT, MCX623106PE-CDAT and MCX623106PC-CDAT.

Operating Systems/Distributions

RHEL/CentOS

Windows

FreeBSD

VMware

OpenFabrics Enterprise Distribution (OFED)

OpenFabrics Windows Distribution (WinOF-2)

Connectivity

Interoperable with 1/10/25/40/50/100/200 Gb/s Ethernet switches

Passive copper cable with ESD protection

Powered connectors for optical and active cable support

Manageability

Mellanox ConnectX-6 Dx technology maintains support for manageability through a BMC. Mellanox

ConnectX-6 Dx PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP

over PCIe protocols as if it is a standard Mellanox PCIe stand-up adapter. For configuring the adapter

for the specific manageability solution in use by the server, please contact Mellanox Support.

Interfaces

The below figures show the component side of the Mellanox ConnectX-6 Dx adapter card. Each

numbered interface that is referenced in the figures is described in the following table with a link to

detailed information.

Callout Item Description

1 “ConnectX-6 Dx IC” Mellanox ConnectX-6 Dx IC on the board.

2 “PCI Express Interface" PCIe Gen 3.0/4.0 through an x8/x16 edge connector.

3 “Ethernet SFP28/SFP56/QSFP56

Interface”

Ethernet traffic is transmitted through the adapter's

SFP28/SFP56/QSFP56 connectors. The networking

connectors allow for the use of modules, optical and

passive cable interconnect solutions.

4 “Networking Ports LEDs Interface” There are two I/O LEDs per port to indicate speed

and link status.

“SMBus Interface” Allows BMC connectivity using MCTP over SMBus or

MCTP over PCIe protocols.

“Voltage Regulators” Voltage supply pins that feed onboard regulators.

The below figures are for illustration purposes only and might not reflect the current revision

of the adapter card.



•

ConnectX-6 Dx IC Interface

The ConnectX®-6 Dx EN family of adapter IC devicesdelivers two ports of 10/25/40/50/100Gb/s or a

single-port of 200Gb/s Ethernet connectivity paired with best-in-class hardware capabilities that

accelerate and secure cloud and data-center workloads.

Encryption

ConnectX-6 Dx brings security to every end-point, including:

Purpose-built inline acceleration engines that offload IPsec and TLS data-in-motion and XTS-

AES data-at-rest cryptographic operations.

Stateful firewall solution acceleration, powered by Open vSwitch connection tracking and

Mellanox’s ASAP2 technology.

Embedded hardware root-of-trust and support for RSA-based secure firmware update and

secure boot, providing guaranteed integrity of the network adapter.

Ethernet SFP28 / SFP56 / QSFP56 Interfaces

The network ports of the ConnectX-6 Dx adapter card are compliant with the IEEE 802.3 Ethernet

standards listed in Features and Benefits. Ethernet traffic is transmitted through the SFP28 / SFP56 /

QSFP56connector on the adaptercard.

PCI Express Interface

ConnectX-6 Dx adapter cards support PCI Express Gen 3.0/4.0 (1.1 and 2.0 compatible) through x8/16

edge connectors. The device can be either a master initiating the PCI Express bus operations, or a

slave responding to PCI bus operations.

The following lists PCIe interface features:

PCIe Gen 4.0 and 3.0 compliant, 2.0 and 1.1 compatible

2.5, 5.0, 8.0, or 16.0 GT/s link rate x8 or x16 lanes

Auto-negotiates to x16, x8, x4, x2, or x1

Support for MSI/MSI-X mechanisms

Networking Ports LEDs Interface

For the networking ports LEDs description, follow the below table depending on the OPN you have

purchased.

Applies to Crypto OPNs only.



The adapter card includes special circuits to protect from ESD shocks to the card/server when

plugging copper cables.



OPN LEDs Scheme

MCX621102AC-ADAT, MCX621102AN-ADAT,

MCX621102AE-ADAT

Scheme 1: One Bi-Color LED

MCX623102A[C/N]-ADAT, MCX623102A[C/N]-

GDAT,MCX623102A[S/E/N/C]-GDAT

MCX623105A[N/E]-CDAT,MCX623106A[C/N/S/E]-

CDAT,MCX623106P[C/N/E]-CDAT,

MCX623105A[C/N/S/E]-VDAT

Scheme 3: One Bi-Color LED and one Single Color

LED

Scheme 1: One Bi-Color LED

There is one bicolor(Yellow and Green) I/O LED per port to indicate speed and link status.

Physical and Logical Link Indications

State Bi-Color LED (Yellow/Green)

Physical link speed

Beacon command for locating the adapter card 1Hz blinking Yellow

Error 4Hz blinking Yellow indicates a problem with the link

Physical Activity • A constant Green indicates a physical link with the

maximum networking speed.

• A constant Yellow indicates a physical link with less than

the maximum networking speed.

Link Up • A constant Green indicates a physical link with

themaximum networking speed.

• A constant Yellow indicates a physical link with less than

the maximum networking speed.

Scheme 3: One Bi-Color LED and one Single Color LED

There are two I/O LEDs per port to indicate speed and link status. LED1 is a bicolor LED (Yellow and

green) and LED2 is a single color LED (green).

Physical and Logical Link Indications (Ethernet Mode)

State Bi-Color LED (Yellow/Green)

Physical link speed

Single Color LED (Green)

Logical link/activity

Beacon command for locating the

adapter card

1Hz blinking Yellow OFF

Error 4Hz blinking Yellow indicates a problem

with the link

State Bi-Color LED (Yellow/Green)

Physical link speed

Single Color LED (Green)

Logical link/activity

Physical Activity • A constant Green indicates a physical

link with the maximum networking speed.

• A constant Yellow indicates a physical

link with less than the maximum

networking speed.

Blinking

Link Up • A constant Green indicates a physical

link with themaximum networking speed.

• A constant Yellow indicates a physical

link with less than the maximum

networking speed.

SMBus Interface

ConnectX-6 Dx technology maintains support for manageability through a BMC. ConnectX-6 Dx PCIe

stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as

if it is a standard Mellanox PCIe stand-up adapter. For configuring the adapter for the specific

manageability solution in use by the server, please contact Mellanox Support.

Voltage Regulators

The voltage regulator power is derived from the PCI Express edge connector 12V supply pins. These

voltage supply pins feed on-board regulators that provide the necessary power to the various

components on the card.

Heatsink

The heatsink is attached to the ConnectX-6 Dx IC in order to dissipate the heat. It is attached either by

using four spring-loaded push pins that insert into four mounting holes, or by screws. ConnectX-6 Dx

IC has a thermal shutdown safety mechanism that automatically shuts down the ConnectX-6 Dx card in

cases of high-temperature event, improper thermal coupling or heatsink removal.

Hardware Installation

Installation and initialization of ConnectX-6 Dx adapter cards require attention to the mechanical

attributes, power specification, and precautions for electronic equipment.

Safety Warnings

Please observe all safety warnings to avoidinjury and prevent damage to system components. Note

that not all warnings are relevant to all models.

Unable to render include or excerpt-include. Could not retrieve page.

Installation Procedure Overview

The installation procedure of ConnectX-6 Dx adapter cards involves the following steps:

Step Procedure Direct Link

1 Check the system’s hardware and software

requirements.

Refer to System Requirements

2 Pay attention to the airflow consideration within the

host system

Refer to Airflow Requirements

3 Follow the safety precautions Refer to Safety Precautions

4 Unpack the package Refer to Unpack the package

5 Follow the pre-installation checklist Refer to Pre-Installation Checklist

6 (Optional) Replace the full-height mounting bracket

with the supplied short bracket

Refer to Bracket Replacement

Instructions

7 Install the ConnectX-6 Dx and PCIe Auxiliary

connection cards in the system

Refer to Installation Instructions

8 Connect cables or modules to the card Refer to Cables and Modules

9 Identify ConnectX-6 Dx in the system Refer to Identifying Your Card

Safety warnings are provided here in the English language. For safety warnings in other

languages, refer to the Adapter Installation Safety Instructions document available on

mellanox.com.



•

System Requirements

Hardware Requirements

A system with a PCI Express slot of corresponding bus width is required for installing the card.

Airflow Requirements

ConnectX-6 Dx adapter cards are offered with two airflow patterns: from the heatsink to the network

ports.

Please refer to the Specificationssection for airflow numbers for each specific card model.

Software Requirements

See Operating Systems/Distributionssection under the Introduction section.

Software Stacks - Mellanox OpenFabric software package MLNX_OFED for Linux, WinOF-2 for

Windows, and VMware. See the Driver Installationsection.

Safety Precautions

The adapter is being installed in a system that operates with voltages that can be lethal. Before

opening the case of the system, observe the following precautions to avoid injury and prevent damage

to system components.

Remove any metallic objects from your hands and wrists.

Make sure to use only insulated tools.

Verify that the system is powered off and is unplugged.

It is strongly recommended to use an ESD strap or other antistatic devices.

Unless otherwise specified, Mellanox products are designed to work in an environmentally

controlled data center with low levels of gaseous and dust (particulate) contamination.

The operation environment should meet severity level G1 as per ISA 71.04 for gaseous

contamination and ISO 14644-1 class 8 for cleanliness level.



For proper operation and performance, please make sure to use a PCIe slot with a

corresponding bus width and that can supply sufficient power to your card. Refer to the

Specifications section of the manual for more power requirements.



All cards in the system should be planned with the same airflow direction.



Page is loading ...

Nvidia Mellanox ConnectX-6 Dx Series User manual

Nvidia Mellanox ConnectX-6 Dx Series User manual

Nvidia Mellanox ConnectX-6 Dx Series User manual

Related papers

Other documents