2. Nvidia
  3. Mellanox ConnectX-6 Dx Series

Nvidia Mellanox ConnectX-6 Dx Series User manual

  • Hello! I am an AI chatbot trained to assist you with the Nvidia Mellanox ConnectX-6 Dx Series User manual. I’ve already reviewed the document and can help you find the information you need or explain it in simple terms. Just ask your questions, and providing more details will help me assist you more effectively!
Exported onOct/22/2020 06:45 PM
https://docs.mellanox.com/x/esV-AQ
NVIDIA Mellanox ConnectX-6 Dx PCIe HHHL
Ethernet Adapter Cards User Manual
2
Table of Contents
Table of Contents .......................................................................................................2
Introduction................................................................................................................9
Product Overview .................................................................................................................... 9
Features and Benefits........................................................................................................... 11
Operating Systems/Distributions ......................................................................................... 13
Connectivity ........................................................................................................................... 13
Manageability ........................................................................................................................ 14
Interfaces .................................................................................................................15
ConnectX-6 Dx IC Interface.................................................................................................. 16
Encryption......................................................................................................................... 16
PCI Express Interface ........................................................................................................... 16
Networking Ports LEDs Interface ........................................................................................ 16
Scheme 1: One Bi-Color LED .......................................................................................... 17
Scheme 3: One Bi-Color LED and one Single Color LED............................................... 17
Voltage Regulators................................................................................................................ 18
Heatsink................................................................................................................................. 18
Hardware Installation ..............................................................................................19
Safety Warnings..................................................................................................................... 19
Installation Procedure Overview........................................................................................... 19
System Requirements........................................................................................................... 20
Hardware Requirements ................................................................................................. 20
Airflow Requirements .................................................................................................20
Software Requirements ................................................................................................... 20
Safety Precautions ................................................................................................................ 20
Unpacking the Package ........................................................................................................ 21
Pre-Installation Checklist..................................................................................................... 21
Bracket Replacement Instructions ...................................................................................... 22
Installation Instructions........................................................................................................ 22
Uninstalling the Card ............................................................................................................ 24
Cables and Modules.............................................................................................................. 25
Identifying the Card in Your System ..................................................................................... 25
Uninstalling the Card ............................................................................................................ 26
3
Driver Installation ....................................................................................................28
Linux Driver Installation........................................................................................................ 28
Prerequisites .................................................................................................................... 28
Downloading Mellanox OFED .......................................................................................... 28
Installing Mellanox OFED ................................................................................................ 29
Installation Script ........................................................................................................ 29
Installation Procedure ................................................................................................30
Installation Results .....................................................................................................32
Installation Logs..........................................................................................................33
openibd Script..............................................................................................................33
Driver Load Upon System Boot ..................................................................................33
mlnxofedinstall Return Codes....................................................................................34
Uninstalling MLNX_OFED...........................................................................................34
Installing MLNX_OFED Using YUM ................................................................................. 34
Setting up MLNX_OFED YUM Repository................................................................... 34
Installing MLNX_OFED Using the YUM Tool ..............................................................35
Uninstalling MLNX_OFED Using the YUM Tool .........................................................36
Installing MLNX_OFED Using apt-get Tool..................................................................... 36
Setting up MLNX_OFED apt-get Repository ..............................................................37
Installing MLNX_OFED Using the apt-get Tool..........................................................37
Uninstalling MLNX_OFED Using the apt-get Tool.....................................................38
Updating Firmware After Installation.............................................................................. 38
Updating the Device Online.........................................................................................38
Updating the Device Manually ....................................................................................38
Updating the Device Firmware Automatically upon System Boot ............................39
UEFI Secure Boot............................................................................................................. 39
Enrolling Mellanox's x.509 Public Key on Your Systems...........................................39
Removing Signature from kernel Modules ................................................................ 40
Performance Tuning ........................................................................................................ 41
Windows Driver Installation.................................................................................................. 41
Software Requirements .................................................................................................. 41
Downloading Mellanox WinOF-2 Driver .......................................................................... 41
Installing Mellanox WinOF-2 Driver ............................................................................... 42
Attended Installation...................................................................................................42
4
Unattended Installation...............................................................................................47
Installation Results .....................................................................................................48
Uninstalling Mellanox WinOF-2 Driver............................................................................ 48
Attended Uninstallation ..............................................................................................48
Unattended Uninstallation..........................................................................................48
Extracting Files Without Running Installation ................................................................ 49
Firmware Upgrade ........................................................................................................... 51
VMware Driver Installation ................................................................................................... 51
Hardware and Software Requirements........................................................................... 51
Installing Mellanox NATIVE ESXi Driver for VMware vSphere........................................ 52
Removing Earlier Mellanox Drivers................................................................................. 52
Firmware Programming .................................................................................................. 53
Updating Adapter Firmware ....................................................................................54
Troubleshooting .......................................................................................................55
GeneralTroubleshooting ...................................................................................................... 55
LinuxTroubleshooting .......................................................................................................... 55
WindowsTroubleshooting..................................................................................................... 56
Specifications ...........................................................................................................57
MCX621102A[C/E/N]-ADAT Specifications........................................................................... 57
MCX623102A[C/N]-ADATSpecifications .............................................................................. 58
MCX623102A[C/N/S/E]-GDAT Specifications ...................................................................... 59
MCX623106A[C/N/S/E]-CDATSpecifications ....................................................................... 60
MCX623106P[C/N/E]-CDATSpecifications........................................................................... 61
MCX623105A[N/E]-CDATSpecifications .............................................................................. 62
MCX623105A[C/N/S/E]-VDATSpecifications........................................................................ 63
Dual-Port SFP28/SFP56 x8 Adapter Cards Mechanical Drawing................................. 64
Dual-PortSFP28/SFP56 x16 Adapter Cards Mechanical Drawing............................... 64
Single-Port QSFP56 x16 Adapter Cards Mechanical Drawing...................................... 65
Dual-Port QSFP56 x16 Adapter Cards Mechanical Drawing ........................................ 65
Bracket Mechanical Drawing ............................................................................................... 65
Dual-Port SFP28/SFP56 Adapter Card ........................................................................... 66
Short Bracket ..............................................................................................................66
Tall Bracket ................................................................................................................. 66
Single-Port QSFP56 Adapter Card .................................................................................. 66
5
Short Bracket ..............................................................................................................66
Tall Bracket ................................................................................................................. 66
Dual-Port QSFP56 Adapter Card..................................................................................... 67
Short Bracket ..............................................................................................................67
Tall Bracket ................................................................................................................. 67
Finding the MAC on the Adapter Card......................................................................68
Document Revision History ......................................................................................71
6
About This Manual
This User Manual describes NVIDIA® Mellanox®ConnectX®-6 Dx Ethernet adapter cards. It provides
details as to the interfaces of the board, specifications, required software and firmware for operating
the board, and relevant documentation.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-6 Dx Ethernet
adapter cards.
Configuration OPN Marketing Description
25GbE PCIe
OPNs
MCX621102AC-
ADAT
ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0 x8,
Crypto and Secure Boot, Tall Bracket
MCX621102AN-
ADAT
ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0 x8,
No Crypto, Tall Bracket
MCX621102AE-
ADAT
ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0 x8,
Crypto, No Secure Boot, Tall Bracket
MCX623102AC-
ADAT
ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0
x16, Crypto and Secure Boot, Tall Bracket
MCX623102AN-
ADAT
ConnectX®-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0
x16, No Crypto, Tall Bracket
50GbE OPNs MCX623102AC-
GDAT
ConnectX®-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe 4.0
x16, Crypto and Secure Boot, Tall Bracket
MCX623102AN-
GDAT
ConnectX®-6 Dx EN adapter card, 50GbE , Dual-port SFP56, PCIe 4.0
x16, No Crypto, Tall Bracket
MCX623102AS-
GDAT
ConnectX®-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe 4.0
x16, Secure Boot, No Crypto, Tall Bracket
MCX623102AE-
GDAT
ConnectX®-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe 4.0
x16, Crypto, No Secure Boot, Tall Bracket
100GbE OPNs MCX623105AN-
CDAT
ConnectX®-6 Dx EN adapter card, 100GbE, Single-port QSFP56, PCIe
4.0 x16, No Crypto, Tall Bracket
MCX623105AE-
CDAT
ConnectX®-6 Dx EN adapter card, 100GbE, Single-port QSFP56, PCIe
4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623106AC-
CDAT
ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0
x16, Crypto and Secure Boot, Tall Bracket
MCX623106AN-
CDAT
ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0
x16, No Crypto, Tall Bracket
MCX623106AS-
CDAT
ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0
x16, Secure Boot, No Crypto, Tall Bracket
MCX623106AE-
CDAT
ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0
x16, Crypto, No Secure Boot, Tall Bracket
100GbE with PPS
In/Out OPNs
MCX623106PC-
CDAT
ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with PPS
In/Out, PCIe 4.0 x16, Crypto and Secure Boot, Tall Bracket
MCX623106PN-
CDAT
ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with PPS
In/Out, PCIe 4.0 x16, No Crypto, Tall Bracket
MCX623106PE-
CDAT
ConnectX®-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with PPS
In/Out, PCIe 4.0 x16, Crypto, No Secure Boot, Tall Bracket
7
•
•
•
Configuration OPN Marketing Description
200GbE OPNs MCX623105AC-
VDAT
ConnectX®-6 Dx EN adapter card, 200GbE, Single-port QSFP56, PCIe
4.0 x16, Crypto and Secure Boot, Tall Bracket
MCX623105AN-
VDAT
ConnectX®-6 Dx EN adapter card, 200GbE , Single-port QSFP56, PCIe
4.0 x16, No Crypto, Tall Bracket
MCX623105AS-
VDAT
ConnectX®-6 Dx EN adapter card, 200GbE, Single-port QSFP56, PCIe
4.0 x16, Secure Boot, No Crypto, Tall Bracket
MCX623105AE-
VDAT
ConnectX®-6 Dx EN adapter card, 200GbE, Single-port QSFP56, PCIe
4.0 x16, Crypto, No Secure Boot, Tall Bracket
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic familiarity
with Ethernet network and architecture specifications.
Technical Support
Customers who purchased Mellanox products directly from Mellanox are invited to contact usthrough
the following methods:
URL:http://www.mellanox.com> Support
E-mail:[email protected]
Tel: +1.408.916.0055
Customers who purchased Mellanox M-1 Global Support Services, please see your contract fordetails
regarding Technical Support.
Customers who purchased Mellanox products through a Mellanox approved reseller should first
seekassistance through their reseller.
Related Documentation
Mellanox OFED for
Linux User Manual
and Release Notes
User Manual describing OFED features, performance, band diagnostic, tools
content and configuration. SeeMellanox OFED for Linux Documentation.
WinOF-2 for
WindowsUser Manual
and Release Notes
User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools
content and configuration. SeeWinOF-2 for Windows Documentation.
Mellanox VMware for
Ethernet User Manual
and Release Notes
User Manual describing the various components of the Mellanox ConnectX®
NATIVE ESXi stack. See http://www.mellanox.comProducts > Software > Ethernet
Drivers > VMware Driver
Mellanox Firmware
Utility (mlxup) User
Manual and Release
Notes
Mellanox firmware update and query utility used to update the firmware. Seehttp:/
/www.mellanox.com> Products > Software > Firmware Tools >mlxup Firmware
Utility
Mellanox Firmware
Tools (MFT) User
Manual
User Manual describing the set of MFT firmware management tools for a single
node. SeeMFT User Manual.
8
IEEE Std 802.3
Specification
IEEE Ethernet specification at http://standards.ieee.org/
PCI Express
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications at
https://pcisig.com/specifications
Mellanox LinkX
Interconnect
Solutions
Mellanox LinkX Ethernet cables and transceivers are designed to maximize the
performance of High-Performance Computing networks, requiring high-
bandwidth, low-latency connections between compute nodes and switch nodes.
Mellanox offers one of industry’s broadest portfolio 40Gb/s, 56Gb/s, 100Gb/s and
200Gb/s cables, including Direct Attach Copper cables (DACs), copper splitter
cables, Active Optical Cables (AOCs) and transceivers in a wide range of lengths
from 0.5m to 10km.Mellanox tests every product in an end-to-end environment
ensuring a Bit Error Rate of less than 1E-15.Read more at https://
www.mellanox.com/products/interconnect/infiniband-overview.php
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to
mean PCI Express.
Revision History
A list of the changes made to this document are provided inDocument Revision History.
9
Introduction
Product Overview
This is the
User Manual
for Mellanox Technologies Ethernet adapter cards based on the ConnectX®-6
Dx integrated circuit device.
As the world's most advanced cloud SmartNIC, ConnectX-6 Dx provides up to two ports of 25, 50 or
100Gb/s or a single-port of 200Gb/s Ethernet connectivity, powered by 50Gb/s PAM4 SerDes technology
and PCIe Gen 4.0 host connectivity.ConnectX-6 Dx continues among Mellanox's innovation path in
scalable cloud fabrics, delivering unparalleled performance and efficiency at every scale. ConnectX-6
Dx's innovative hardware offload engines, including IPsec and TLS inline data-in-motion encryption, are
ideal for enabling secure network connectivity in modern data-center environments. Please refer toFe
ature and Benefits for more details.
Mellanox ConnectX-6 Dx 25GbE Adapter Cards
Part Number MCX621102AC-
ADAT
MCX621102AE-
ADAT
MCX621102AN-
ADAT
MCX623102AC-
ADAT
MCX623102AN-
ADAT
Form Factor/
Dimensions
5.59in. x 2.71in (142.00mm x 68.90mm)
Data Transmission
Rate
Ethernet: 1/10/25 Gb/s
Network Connector
Type
Dual-port
SFP28
Dual-port
SFP28
Dual-port SFP28 Dual-port
SFP28
Dual-port SFP28
PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x8 PCIe Gen 4.0 SERDES @ 16.0GT/s
x16
Crypto √ √ - √ -
Secure Boot √ - - √ -
Adapter IC Part
Number
MT28928A0-
CCCF-CE
MT28928A0-
NCCF-CE
MT28928A0-
NCCF-CE
MT28928A0-
CCCF-CE
MT28928A0-
NCCF-CE
RoHS RoHS Compliant
Mellanox ConnectX-6 Dx 50GbE Adapter Cards
Part Number MCX623102AC-GDAT MCX623102AN-GDAT MCX623102AS-GDAT
Form Factor/Dimensions 5.59in. x 2.71in (142.00mm x 68.90mm)
Data Transmission Rate Ethernet: 1/10/25/40/50 Gb/s
Network Connector Type Dual-port SFP56
PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x16
10
Crypto √ √ - -
Secure Boot √ - - √
Adapter IC Part Number MT28928A0-CCCF-CE MT28928A0-
CCCF-CE
MT28928A0-NCCF-CE MT28928A0-NCCF-CE
RoHS RoHS Compliant
Mellanox ConnectX-6 Dx 100GbE Adapter Cards
Part Number MCX623105A
N-CDAT
MCX623105A
E-CDAT
MCX623106A
E-CDAT
MCX623106A
C-CDAT
MCX623106A
N-CDAT
MCX623106A
S-CDAT
Form Factor/
Dimensions
5.59in. x 2.71in (142.00mm x 68.90mm)
Data
Transmission
Rate
Ethernet: 1/10/25/40/50/100 Gb/s
Network
Connector Type
Single-port
QSFP56
Single-port
QSFP56
Dual-port
QSFP56
Dual-port
QSFP56
Dual-port
QSFP56
Dual-port
QSFP56
PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x16
Crypto - √ √ √ - -
Secure Boot - - - √ - √
Adapter IC Part
Number
MT28928A0-
XCCF-CE
MT28928A0-
XCCF-CE
MT28928A0-
CCCF-CE
MT28928A0-
CCCF-CE
MT28928A0-
NCCF-CE
MT28928A0-
NCCF-CE
RoHS RoHS Compliant
Mellanox ConnectX-6 Dx 100GbE Adapter Cards withPPS In/Out SMAs
Part Number MCX623106PN-CDAT MCX623106PE-CDAT MCX623106PC-CDAT
Form Factor/Dimensions 5.59in. x 2.71in (142.00mm x 68.90mm)
Data Transmission Rate Ethernet: 1/10/25/40/50/100 Gb/s
Network Connector Type Dual-port QSFP56 Dual-port QSFP56 Dual-port QSFP56
PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x16
Crypto - √ √
Secure Boot - - √
PPS In/Out SMAs √ √ √
Adapter IC Part Number MT28928A0-CCCF-CE MT28928A0-NCCF-CE MT28928A0-NCCF-CE
RoHS RoHS Compliant
11
Mellanox ConnectX-6 Dx 200GbE Adapter Cards
Part Number MCX623105AC-
VDAT
MCX623105AE-
VDAT
MCX623105AN-
VDAT
MCX623105AS-
VDAT
Form Factor/
Dimensions
5.59in. x 2.71in (142.00mm x 68.90mm)
Data Transmission
Rate
Ethernet: 1/10/25/40/50/100/200 Gb/s
Network Connector
Type
Single-port QSFP56
PCIe Interface PCIe Gen 4.0 SERDES @ 16.0GT/s x16
Crypto √ √ - -
Secure Boot √ - - √
Adapter IC Part
Number
MT28928A0-CCCF-
CE
MT28928A0-CCCF-
CE
MT28928A0-NCCF-
CE
MT28928A0-NCCF-
CE
RoHS RoHS Compliant
For more detailed information seeSpecifications.
Features and Benefits
Feature Description
PCI
Express
(PCIe)
PCIe Gen 4.0 SERDES@ 8.0GT/s / 16.0GT/s through x8/x16 Edge Connector
This section describes hardware features and capabilities. Please refer to the relevant driver
and/or firmware release notes for feature availability.
12
•
•
•
Feature Description
Mellanox adapters comply with the following IEEE 802.3 standards:
200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
– IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
– IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
– IEEE 802.3ba 40 Gigabit Ethernet
– IEEE 802.3by 25 Gigabit Ethernet
– IEEE 802.3ae 10 Gigabit Ethernet
– IEEE 802.3ap based auto-negotiation and KR startup
– IEEE 802.3ad, 802.1AX Link Aggregation
– IEEE 802.1Q, 802.1P VLAN tags and priority
– IEEE 802.1Qau (QCN)
– Congestion Notification
– IEEE 802.1Qaz (ETS)
– IEEE 802.1Qbb (PFC)
– IEEE 802.1Qbg
– IEEE 1588v2
– Jumbo frame support (9.6KB)
Memory PCI Express - stores and accesses Ethernet fabric connection information and packet data.
SPI Quad - includes 256Mbit SPI Quad Flash device (MX25L25645GXDI-08G device by
Macronix)
FRU EEPROM - Stores the parameters and personality of the card. The EEPROM capacity
is 128Kbit. FRU I2C address is (0x50) and is accessible through the PCIe SMBus(Available
only in QSFP cards).
Overlay
Networks
In order to better scale their networks, data center operators often create overlay networks that
carry traffic from individual virtual machines over logical tunnels in encapsulated formats such as
NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the
hardware offloading engines, placing higher loads on the host CPU. Mellanox ConnectX-6 Dx
effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines
that encapsulate and de-capsulate the overlay protocol.
RDMA
overConve
rged
Ethernet
(RoCE)
Mellanox ConnectX-6 Dx, utilizing RoCE (RDMA over Converged Ethernet) technology, delivers
low-latency and high-performance over Band and Ethernet networks. Leveraging data center
bridging (DCB) capabilities as well as Mellanox ConnectX-6 Dx advanced congestion control
hardware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and
Layer 3 networks.
Mellanox
PeerDirect
®
Mellanox PeerDirect® communication provides high-efficiency RDMA access by eliminating
unnecessary internal data copies between components on the PCIe bus (for example, from GPU to
CPU), and therefore significantly reduces application run time. Mellanox ConnectX-6 Dx advanced
acceleration technology enables higher cluster efficiency and scalability to tens of thousands of
nodes.
CPU
Offload
Adapter functionality enabling reduced CPU overhead allowing more available CPU for
computation tasks.
Open vSwitch (OVS) offload using ASAP
2
- Accelerated Switch and Packet
• Flexible match-action flow tables
• Tunnelingencapsulation / decapsulation
Quality of
Service
(QoS)
Support for port-based Quality of Service enabling various application requirements for latency
and SLA.
Hardware-
based I/
OVirtualiza
tion
Mellanox ConnectX-6 Dx provides dedicated adapter resources and guaranteed isolation and
protection for virtual machines within the server.
13
•
•
•
•
•
•
•
•
•
Feature Description
Storage
Acceleratio
n
A consolidated compute and storage network achieves significant cost-performance advantages
over multi-fabric networks. Standard block and file access protocols can leverage
• RDMA for high-performance storage access
• NVMe over Fabric offloads for target machine
• Erasure Coding
• T10-DIF Signature Handover
SR-IOV Mellanox ConnectX-6 Dx SR-IOV technology provides dedicated adapter resources and guaranteed
isolation and protection for virtual machines (VM) within the server.
High-
Performan
ceAccelera
tions
• Tag Matching and Rendezvous Offloads
• Adaptive Routing on Reliable Transport
• Burst Buffer Offloads for Background Checkpointing
PPS In/Out
SMAs
Mellanox offers a full IEEE 1588v2 PTP software solution, as well as time-sensitive related
features called “5T”. Mellanox PTP and 5T software solution are designed to meet the most
demanding PTP profiles. ConnectX-6 Dx incorporates an integrated Hardware Clock (PHC) that
allows ConnectX-6 Dx to achieve sub 20u Sec accuracy and also offers many timing related
functions such as time-triggered scheduling or time-based SND accelerations (time-based
ASAP²) . Furthermore, the 5T technology enables the software application to transmit fronthaul
(ORAN) compatible in high bandwidth. The PTP part supports slave clock, master clock, and
boundary clock.
ConnectX-6 Dx PTP solution allows you to run any PTP stack on your host.
With respect to testing and measurements, selected Mellanox adapters allow you to use the PPS-
out signal from onboard SMA connecter, ConnectX-6 Dx also allows to measure PTP in scale, with
PPS-In signal. The PTP HW clock on the Network adapter will be sampled on each PPS-In signal,
and the timestamp will be sent to the SW.
Enabled in MCX623106PN-CDAT, MCX623106PE-CDAT and MCX623106PC-CDAT.
Operating Systems/Distributions
RHEL/CentOS
Windows
FreeBSD
VMware
OpenFabrics Enterprise Distribution (OFED)
OpenFabrics Windows Distribution (WinOF-2)
Connectivity
Interoperable with 1/10/25/40/50/100/200 Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support
14
Manageability
Mellanox ConnectX-6 Dx technology maintains support for manageability through a BMC. Mellanox
ConnectX-6 Dx PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP
over PCIe protocols as if it is a standard Mellanox PCIe stand-up adapter. For configuring the adapter
for the specific manageability solution in use by the server, please contact Mellanox Support.
15
Interfaces
The below figures show the component side of the Mellanox ConnectX-6 Dx adapter card. Each
numbered interface that is referenced in the figures is described in the following table with a link to
detailed information.
Callout Item Description
1 “ConnectX-6 Dx IC” Mellanox ConnectX-6 Dx IC on the board.
2 “PCI Express Interface" PCIe Gen 3.0/4.0 through an x8/x16 edge connector.
3 “Ethernet SFP28/SFP56/QSFP56
Interface”
Ethernet traffic is transmitted through the adapter's
SFP28/SFP56/QSFP56 connectors. The networking
connectors allow for the use of modules, optical and
passive cable interconnect solutions.
4 “Networking Ports LEDs Interface” There are two I/O LEDs per port to indicate speed
and link status.
“SMBus Interface” Allows BMC connectivity using MCTP over SMBus or
MCTP over PCIe protocols.
“Voltage Regulators” Voltage supply pins that feed onboard regulators.
The below figures are for illustration purposes only and might not reflect the current revision
of the adapter card.
16
•
•
•
•
•
•
•
ConnectX-6 Dx IC Interface
The ConnectX®-6 Dx EN family of adapter IC devicesdelivers two ports of 10/25/40/50/100Gb/s or a
single-port of 200Gb/s Ethernet connectivity paired with best-in-class hardware capabilities that
accelerate and secure cloud and data-center workloads.
Encryption
ConnectX-6 Dx brings security to every end-point, including:
Purpose-built inline acceleration engines that offload IPsec and TLS data-in-motion and XTS-
AES data-at-rest cryptographic operations.
Stateful firewall solution acceleration, powered by Open vSwitch connection tracking and
Mellanox’s ASAP2 technology.
Embedded hardware root-of-trust and support for RSA-based secure firmware update and
secure boot, providing guaranteed integrity of the network adapter.
Ethernet SFP28 / SFP56 / QSFP56 Interfaces
The network ports of the ConnectX-6 Dx adapter card are compliant with the IEEE 802.3 Ethernet
standards listed in Features and Benefits. Ethernet traffic is transmitted through the SFP28 / SFP56 /
QSFP56connector on the adaptercard.
PCI Express Interface
ConnectX-6 Dx adapter cards support PCI Express Gen 3.0/4.0 (1.1 and 2.0 compatible) through x8/16
edge connectors. The device can be either a master initiating the PCI Express bus operations, or a
slave responding to PCI bus operations.
The following lists PCIe interface features:
PCIe Gen 4.0 and 3.0 compliant, 2.0 and 1.1 compatible
2.5, 5.0, 8.0, or 16.0 GT/s link rate x8 or x16 lanes
Auto-negotiates to x16, x8, x4, x2, or x1
Support for MSI/MSI-X mechanisms
Networking Ports LEDs Interface
For the networking ports LEDs description, follow the below table depending on the OPN you have
purchased.
Applies to Crypto OPNs only.
The adapter card includes special circuits to protect from ESD shocks to the card/server when
plugging copper cables.
17
OPN LEDs Scheme
MCX621102AC-ADAT, MCX621102AN-ADAT,
MCX621102AE-ADAT
Scheme 1: One Bi-Color LED
MCX623102A[C/N]-ADAT, MCX623102A[C/N]-
GDAT,MCX623102A[S/E/N/C]-GDAT
MCX623105A[N/E]-CDAT,MCX623106A[C/N/S/E]-
CDAT,MCX623106P[C/N/E]-CDAT,
MCX623105A[C/N/S/E]-VDAT
Scheme 3: One Bi-Color LED and one Single Color
LED
Scheme 1: One Bi-Color LED
There is one bicolor(Yellow and Green) I/O LED per port to indicate speed and link status.
Physical and Logical Link Indications
State Bi-Color LED (Yellow/Green)
Physical link speed
Beacon command for locating the adapter card 1Hz blinking Yellow
Error 4Hz blinking Yellow indicates a problem with the link
Physical Activity • A constant Green indicates a physical link with the
maximum networking speed.
• A constant Yellow indicates a physical link with less than
the maximum networking speed.
Link Up • A constant Green indicates a physical link with
themaximum networking speed.
• A constant Yellow indicates a physical link with less than
the maximum networking speed.
Scheme 3: One Bi-Color LED and one Single Color LED
There are two I/O LEDs per port to indicate speed and link status. LED1 is a bicolor LED (Yellow and
green) and LED2 is a single color LED (green).
Physical and Logical Link Indications (Ethernet Mode)
State Bi-Color LED (Yellow/Green)
Physical link speed
Single Color LED (Green)
Logical link/activity
Beacon command for locating the
adapter card
1Hz blinking Yellow OFF
Error 4Hz blinking Yellow indicates a problem
with the link
ON
18
State Bi-Color LED (Yellow/Green)
Physical link speed
Single Color LED (Green)
Logical link/activity
Physical Activity • A constant Green indicates a physical
link with the maximum networking speed.
• A constant Yellow indicates a physical
link with less than the maximum
networking speed.
Blinking
Link Up • A constant Green indicates a physical
link with themaximum networking speed.
• A constant Yellow indicates a physical
link with less than the maximum
networking speed.
ON
SMBus Interface
ConnectX-6 Dx technology maintains support for manageability through a BMC. ConnectX-6 Dx PCIe
stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as
if it is a standard Mellanox PCIe stand-up adapter. For configuring the adapter for the specific
manageability solution in use by the server, please contact Mellanox Support.
Voltage Regulators
The voltage regulator power is derived from the PCI Express edge connector 12V supply pins. These
voltage supply pins feed on-board regulators that provide the necessary power to the various
components on the card.
Heatsink
The heatsink is attached to the ConnectX-6 Dx IC in order to dissipate the heat. It is attached either by
using four spring-loaded push pins that insert into four mounting holes, or by screws. ConnectX-6 Dx
IC has a thermal shutdown safety mechanism that automatically shuts down the ConnectX-6 Dx card in
cases of high-temperature event, improper thermal coupling or heatsink removal.
19
Hardware Installation
Installation and initialization of ConnectX-6 Dx adapter cards require attention to the mechanical
attributes, power specification, and precautions for electronic equipment.
Safety Warnings
Please observe all safety warnings to avoidinjury and prevent damage to system components. Note
that not all warnings are relevant to all models.
Unable to render include or excerpt-include. Could not retrieve page.
Installation Procedure Overview
The installation procedure of ConnectX-6 Dx adapter cards involves the following steps:
Step Procedure Direct Link
1 Check the system’s hardware and software
requirements.
Refer to System Requirements
2 Pay attention to the airflow consideration within the
host system
Refer to Airflow Requirements
3 Follow the safety precautions Refer to Safety Precautions
4 Unpack the package Refer to Unpack the package
5 Follow the pre-installation checklist Refer to Pre-Installation Checklist
6 (Optional) Replace the full-height mounting bracket
with the supplied short bracket
Refer to Bracket Replacement
Instructions
7 Install the ConnectX-6 Dx and PCIe Auxiliary
connection cards in the system
Refer to Installation Instructions
8 Connect cables or modules to the card Refer to Cables and Modules
9 Identify ConnectX-6 Dx in the system Refer to Identifying Your Card
Safety warnings are provided here in the English language. For safety warnings in other
languages, refer to the Adapter Installation Safety Instructions document available on
mellanox.com.
20
•
•
•
•
•
•
System Requirements
Hardware Requirements
A system with a PCI Express slot of corresponding bus width is required for installing the card.
Airflow Requirements
ConnectX-6 Dx adapter cards are offered with two airflow patterns: from the heatsink to the network
ports.
Please refer to the Specificationssection for airflow numbers for each specific card model.
Software Requirements
See Operating Systems/Distributionssection under the Introduction section.
Software Stacks - Mellanox OpenFabric software package MLNX_OFED for Linux, WinOF-2 for
Windows, and VMware. See the Driver Installationsection.
Safety Precautions
The adapter is being installed in a system that operates with voltages that can be lethal. Before
opening the case of the system, observe the following precautions to avoid injury and prevent damage
to system components.
Remove any metallic objects from your hands and wrists.
Make sure to use only insulated tools.
Verify that the system is powered off and is unplugged.
It is strongly recommended to use an ESD strap or other antistatic devices.
Unless otherwise specified, Mellanox products are designed to work in an environmentally
controlled data center with low levels of gaseous and dust (particulate) contamination.
The operation environment should meet severity level G1 as per ISA 71.04 for gaseous
contamination and ISO 14644-1 class 8 for cleanliness level.
For proper operation and performance, please make sure to use a PCIe slot with a
corresponding bus width and that can supply sufficient power to your card. Refer to the
Specifications section of the manual for more power requirements.
All cards in the system should be planned with the same airflow direction.
/