Nvidia Mellanox ConnectX-6 MCX654106A-ECAT User manual

Brand: Nvidia

Model: Mellanox ConnectX-6 MCX654106A-ECAT

Type: User manual

This manual is also suitable for

Exported onOct/22/2020 07:16 PM

https://docs.mellanox.com/x/NCQuAQ

NVIDIA Mellanox ConnectX-6 InfiniBand/VPI

Adapter Cards User Manual

2
Table of Contents
Table of Contents .......................................................................................................2
Introduction................................................................................................................9
Product Overview .................................................................................................................... 9
ConnectX-6 PCIe x8 Card...................................................................................................... 10
ConnectX-6 PCIe x16 Card.................................................................................................... 11
ConnectX-6 Socket Direct™ Cards....................................................................................... 11
ConnectX-6 Dual-slot Socket Direct Cards (2x PCIe x16) .............................................. 12
ConnectX-6 Single-slot Socket Direct Cards (2x PCIe x8 in a row) ............................... 13
Package Contents ................................................................................................................. 14
ConnectX-6 PCIe x8/x16 Adapter Cards.......................................................................... 14
ConnectX-6 Socket Direct Cards (2x PCIe x16)............................................................... 14
Features and Benefits........................................................................................................... 15
Operating Systems/Distributions .................................................................................... 17
Connectivity ...................................................................................................................... 17
Manageability ................................................................................................................... 17
Interfaces .................................................................................................................18
InfiniBand Interface............................................................................................................... 18
Ethernet QSFP56 Interfaces ................................................................................................. 18
PCI Express Interface ........................................................................................................... 18
LED Interface......................................................................................................................... 18
Heat Sink Interface................................................................................................................ 19
SMBus Interface.................................................................................................................... 20
Voltage Regulators................................................................................................................ 20
Hardware Installation ..............................................................................................21
Safety Warnings..................................................................................................................... 21
Installation Procedure Overview........................................................................................... 21
System Requirements........................................................................................................... 22
Hardware Requirements ................................................................................................. 22
Airflow Requirements ........................................................................................................... 22
Software Requirements ................................................................................................... 23
Safety Precautions ................................................................................................................ 23
Pre-Installation Checklist..................................................................................................... 24

3
Bracket Replacement Instructions ...................................................................................... 24
Installation Instructions........................................................................................................ 25
Cables and Modules......................................................................................................... 25
Identifying the Card in Your System ..................................................................................... 26
ConnectX-6 PCIe x8/16 Installation Instructions................................................................. 27
Installing the Card............................................................................................................ 27
Uninstalling the Card ....................................................................................................... 29
ConnectX-6 Socket Direct (2x PCIe x16) Installation Instructions ...................................... 30
Installing the Card............................................................................................................ 31
Uninstalling the Card ....................................................................................................... 37
Driver Installation ....................................................................................................39
Linux Driver Installation........................................................................................................ 39
Prerequisites .................................................................................................................... 39
Downloading Mellanox OFED .......................................................................................... 39
Installing Mellanox OFED ................................................................................................ 41
Installation Script ........................................................................................................41
Installation Procedure ................................................................................................42
Installation Results .....................................................................................................44
Installation Logs..........................................................................................................44
openibd Script..............................................................................................................44
Driver Load Upon System Boot ..................................................................................45
mlnxofedinstall Return Codes....................................................................................45
Uninstalling MLNX_OFED...........................................................................................46
Installing MLNX_OFED Using YUM ................................................................................. 46
Setting up MLNX_OFED YUM Repository...................................................................46
Installing MLNX_OFED Using the YUM Tool .............................................................. 47
Uninstalling MLNX_OFED Using the YUM Tool .........................................................48
Installing MLNX_OFED Using apt-get Tool..................................................................... 48
Setting up MLNX_OFED apt-get Repository ..............................................................48
Installing MLNX_OFED Using the apt-get Tool..........................................................48
Uninstalling MLNX_OFED Using the apt-get Tool.....................................................49
Updating Firmware After Installation.............................................................................. 49
Updating the Device Online.........................................................................................49
Updating the Device Manually ....................................................................................50

4
Updating the Device Firmware Automatically upon System Boot ............................ 50
UEFI Secure Boot............................................................................................................. 51
Enrolling Mellanox's x.509 Public Key on Your Systems...........................................51
Removing Signature from kernel Modules ................................................................51
Performance Tuning ........................................................................................................ 52
Windows Driver Installation.................................................................................................. 52
Software Requirements ................................................................................................... 53
Downloading Mellanox WinOF-2 Driver .......................................................................... 53
Attended Installation...................................................................................................54
Unattended Installation...............................................................................................58
Installation Results .....................................................................................................59
Uninstalling Mellanox WinOF-2 Driver............................................................................ 60
Attended Uninstallation ..............................................................................................60
Unattended Uninstallation..........................................................................................60
Extracting Files Without Running Installation ................................................................ 60
Firmware Upgrade ........................................................................................................... 63
VMware Driver Installation ................................................................................................... 63
Hardware and Software Requirements........................................................................... 63
Installing Mellanox NATIVE ESXi Driver for VMware vSphere........................................ 64
Removing Earlier Mellanox Drivers................................................................................. 64
Firmware Programming .................................................................................................. 65
Updating Adapter Firmware ....................................................................................66
Troubleshooting .......................................................................................................67
GeneralTroubleshooting ...................................................................................................... 67
LinuxTroubleshooting .......................................................................................................... 67
WindowsTroubleshooting..................................................................................................... 68
Specifications ...........................................................................................................69
MCX651105A-EDAT Specifications ...................................................................................... 69
MCX653105A-HDAT Specifications....................................................................................... 70
MCX653106A-HDAT Specifications....................................................................................... 71
MCX653105A-ECAT Specifications ....................................................................................... 73
MCX653106A-ECAT Specifications ....................................................................................... 74
MCX654105A-HCAT Specifications....................................................................................... 75
MCX654106A-HCAT Specifications....................................................................................... 77

5
MCX654106A-ECAT Specifications ....................................................................................... 78
MCX653105A-EFAT Specifications........................................................................................ 80
MCX653106A-EFAT Specifications........................................................................................ 81
Adapter Card and Bracket Mechanical Drawings and Dimensions.................................... 82
ConnectX-6 PCIe x16 Adapter Card................................................................................ 83
ConnectX-6 PCIe x8 Adapter Card.................................................................................. 83
Auxiliary PCIe Connection Card...................................................................................... 84
Tall Bracket ...................................................................................................................... 84
Short Bracket ................................................................................................................... 84
PCIe Express Pinouts Description for Single-Slot Socket Direct Card....................85
Finding the GUID/MAC on the Adapter Card ............................................................86
Document Revision History ......................................................................................88

•

About This Manual

This User Manual describes NVIDIA® Mellanox® ConnectX®-6 InfiniBand/VPI adapter cards. It

provides details as to the interfaces of the board, specifications, required software and firmware for

operating the board, and relevant documentation.

Ordering Part Numbers

The table below provides the ordering part numbers (OPN) for the available ConnectX-6 InfiniBand/VPI

adapter cards.

OPN Marketing Description

MCX654106A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR

InfiniBand and 100GbE), dual-port QSFP56, Socket Direct 2x

PCIe 3.0/4.0 x16, tall bracket

MCX653105A-EFAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and

100GbE), single-port QSFP56, PCIe3.0/4.0 Socket Direct 2x8 in

a row, tall bracket

MCX653106A-EFAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR

IBand100GbE), dual-port QSFP56, PCIe3.0/4.0 Socket Direct

2x8 in a row, tall bracket

MCX651105A-EDAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and

100GbE, single-port QSFP56, PCIe4.0 x8, tall bracket

MCX653105A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and

100GbE), single-port QSFP56, PCIe3.0/4.0 x16, tall bracket

MCX653106A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and

100GbE), dual-port QSFP56, PCIe3.0/4.0 x16, tall bracket

MCX653105A-HDAT ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE,

single-port QSFP56, PCIe3.0/4.0 x16, tall bracket

MCX653106A-HDAT ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE,

dual-port QSFP56, PCIe3.0/4.0 x16, tall bracket

MCX654105A-HCAT ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE,

single-port QSFP56, Socket Direct 2x PCIe3.0/4.0x16, tall

bracket

MCX654106A-HCAT ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE,

dual-port QSFP56, Socket Direct 2x PCIe3.0/4.0x16, tall

bracket

Intended Audience

This manual is intended for the installer and user of these cards.

The manual assumes basic familiarity with InfiniBand and Ethernet network and architecture

specifications.

Technical Support

Customers who purchased Mellanox products directly from Mellanox are invited to contact usthrough

the following methods:

URL:http://www.mellanox.com> Support

E-mail:[email protected]

• Tel: +1.408.916.0055

Customers who purchased Mellanox M-1 Global Support Services, please see your contract fordetails

regarding Technical Support.

Customers who purchased Mellanox products through a Mellanox approved reseller should first

seekassistance through their reseller.

Related Documentation

Mellanox OFED

for Linux User

Manual and

Release Notes

User Manual describing OFED features, performance, band diagnostic, tools content and

configuration. SeeMellanox OFED for Linux Documentation.

WinOF-2 for

WindowsUser

Manual and

Release Notes

User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools

content and configuration. SeeWinOF-2 for Windows Documentation.

Mellanox

VMware for

Ethernet User

Manual

User Manual describing the various components of the Mellanox ConnectX® NATIVE

ESXi stack. See http://www.mellanox.comProducts > Software > Ethernet Drivers >

VMware Driver > User Manual

Mellanox

VMware for

Ethernet Release

Notes

Release notes for Mellanox ConnectX® NATIVE ESXi stack. See http://

www.mellanox.comSoftware > Ethernet Drivers > VMware Driver > Release Notes

Mellanox

Firmware Utility

(mlxup) User

Manual and

Release Notes

Mellanox firmware update and query utility used to update the firmware. Seehttp://

www.mellanox.com> Products > Software > Firmware Tools > mlxup Firmware Utility

Mellanox

Firmware Tools

(MFT) User

Manual

User Manual describing the set of MFT firmware management tools for a single node.

SeeMFT User Manual.

IEEE Std 802.3

Specification

IEEE Ethernet specification at http://standards.ieee.org

PCI Express

3.0/4.0

Specifications

Industry Standard PCI Express Base and Card Electromechanical Specifications at

https://pcisig.com/specifications

Mellanox LinkX

Interconnect

Solutions

Mellanox LinkX InfiniBand cables and transceivers are designed to maximize the

performance of High-Performance Computing networks, requiring high-bandwidth, low-

latency connections between compute nodes and switch nodes. Mellanox offers one of

industry’s broadest portfolio of QDR/FDR10 (40Gb/s), FDR (56Gb/s), EDR/HDR100

(100Gb/s) and HDR (200Gb/s) cables, including Direct Attach Copper cables (DACs),

copper splitter cables, Active Optical Cables (AOCs) and transceivers in a wide range of

lengths from 0.5m to 10km. In addition to meeting IBTA standards,Mellanox tests every

product in an end-to-end environment ensuring a Bit Error Rate of less than 1E-15.Read

more at https://www.mellanox.com/products/interconnect/infiniband-overview.php

Document Conventions

When discussing memory sizes, MB and MBytes are used in this document to mean size in mega

Bytes. The use of Mb or Mbits (small b) indicates size in mega bits.IB is used in this document to mean

InfiniBand. In this document PCIe is used to mean PCI Express.

Revision History

A list of the changes made to this document are provided inDocument Revision History.

Introduction

Product Overview

This is the user guide for Mellanox technologies VPI adapter cards based on the ConnectX®-6

integrated circuit device. ConnectX-6 connectivity provides the highest performing low latency and

most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in enterprise datacenters

and high-performance computing environments.

ConnectX-6 Virtual Protocol Interconnect® adapter cards provide up to two ports of 200Gb/s for

InfiniBand and Ethernet connectivity, sub-600ns latency and 200 million messages per second,

enabling the highest performance and most flexible solution for the most demanding High-

Performance Computing (HPC), storage, and datacenter applications.

ConnectX-6 is a groundbreaking addition to the Mellanox ConnectX series of industry-leading adapter

cards. In addition to all the existing innovative features of past ConnectX versions, ConnectX-6 offers a

number of enhancements that further improve the performance and scalability of datacenter

applications.

ConnectX-6 adapter cards are offered in a variety of PCIe configurations, as described in the below

table.

Configuration OPN Marketing Description

ConnectX-6 PCIe x8 Card MCX651105A-EDAT ConnectX®-6 VPI adapter card, 100Gb/s

(HDR100, EDR IB and 100GbE, single-port

QSFP56, PCIe4.0 x8, tall bracket

ConnectX-6 PCIe x16 Card

MCX653105A-HDAT ConnectX®-6 VPI adapter card, HDR IB

(200Gb/s) and 200GbE, single-port QSFP56,

PCIe4.0 x16, tall bracket

MCX653106A-HDAT ConnectX®-6 VPI adapter card, HDR IB

(200Gb/s) and 200GbE, dual-port QSFP56,

PCIe3.0/4.0 x16, tall bracket

MCX653105A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s

(HDR100, EDR IB and 100GbE), single-port

QSFP56, PCIe3.0/4.0 x16, tall bracket

MCX653106A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s

(HDR100, EDR IB and 100GbE), dual-port

QSFP56, PCIe3.0/4.0 x16, tall bracket

ConnectX-6 Dual-slot Socket Direct

Cards (2x PCIe x16)

MCX654105A-HCAT ConnectX®-6 VPI adapter card kit, HDR IB

(200Gb/s) and 200GbE, single-port QSFP56,

Socket Direct 2x PCIe3.0 x16, tall brackets

MCX654106A-HCAT ConnectX®-6 VPI adapter card, HDR IB

(200Gb/s) and 200GbE, dual-port QSFP56,

Socket Direct 2x PCIe3.0/4.0x16, tall bracket

Make sure to use a PCIe slot that is capable of supplying the required power and airflow to the

ConnectX-6 as stated in Specifications.



Configuration OPN Marketing Description

MCX654106A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s

(HDR100, EDR InfiniBand and 100GbE), dual-

port QSFP56, Socket Direct 2x PCIe3.0/4.0

x16, tall bracket

ConnectX-6 Single-slot Socket Direct

Cards (2x PCIe x8 in a row)

MCX653105A-EFAT ConnectX®-6 VPI adapter card, 100Gb/s

(HDR100, EDR IB and 100GbE), single-port

QSFP56, PCIe3.0/4.0 Socket Direct 2x8 in a

row, tall bracket

MCX653106A-EFAT ConnectX®-6 VPI adapter card, 100Gb/s

(HDR100, EDR IBand100GbE), dual-port

QSFP56, PCIe3.0/4.0 Socket Direct 2x8 in a

row, tall bracket

ConnectX-6 PCIe x8 Card

ConnectX-6 with a single PCIe x8 slot can support a bandwidth of up to 100Gb/s in a PCIe Gen 4.0 slot.

Part Number MCX651105A-EDAT

Form Factor/Dimensions PCIe Half Height, Half Length / 167.65mm x 68.90mm

Data Transmission Rate Ethernet: 10/25/40/50/100 Gb/s

InfiniBand: SDR, DDR, QDR, FDR, EDR, HDR100

Network Connector Type Single-port QSFP56

PCIe x8 through Edge Connector PCIe Gen 3.0 / 4.0 SERDES @ 8.0GT/s / 16.0GT/s

RoHS RoHS Compliant

Adapter IC Part Number MT28908A0-XCCF-HVM

ConnectX-6 PCIe x16 Card

ConnectX-6 with a single PCIe x16 slot can support a bandwidth of up to 100Gb/s in a PCIe Gen 3.0 slot,

or up to 200Gb/s in a PCIe Gen 4.0 slot.

Part Number MCX653105A-

ECAT

MCX653106A-

ECAT

MCX653105A-

HDAT

MCX653106A-

HDAT

Form Factor/Dimensions PCIe Half Height, Half Length / 167.65mm x 68.90mm

Data Transmission Rate Ethernet: 10/25/40/50/100 Gb/s

InfiniBand: SDR, DDR, QDR, FDR,

EDR, HDR100

Ethernet: 10/25/40/50/100/200 Gb/s

InfiniBand: SDR, DDR, QDR, FDR,

EDR, HDR100, HDR

Network Connector Type Single-port

QSFP56

Dual-port

QSFP56

Single-port

QSFP56

Dual-port QSFP56

PCIe x16 through Edge Connector PCIe Gen 3.0 / 4.0 SERDES @ 8.0GT/s / 16.0GT/s

RoHS RoHS Compliant

Adapter IC Part Number MT28908A0-XCCF-HVM

ConnectX-6 Socket Direct™ Cards

The Socket Direct technology offers improved performance to dual-socket servers by enabling

direct access from each CPU in a dual-socket server to the network through its dedicated PCIe

interface.

Please note thatConnectX-6 Socket Direct cards do not support Multi-Host functionality (i.e.

connectivity to two independent CPUs). For ConnectX-6 Socket Direct card with Multi-Host

functionality, please contact Mellanox.

ConnectX-6 Socket Direct cards are available in two configurations: Dual-slot Configuration (2x PCIe

x16) and Single-slot Configuration (2x PCIe x8).

ConnectX-6 Dual-slot Socket Direct Cards (2x PCIe x16)

In order to obtain 200Gb/s speed, Mellanox offers ConnectX-6 Socket Direct that enable 200Gb/s

connectivity also for servers with PCIe Gen 3.0 capability. The adapter’s 32-lane PCIe bus is split into

two 16-lane buses, with one bus accessible through a PCIe x16 edge connector and the other bus

through an x16 Auxiliary PCIe Connection card. The two cards should be installed into two PCIe x16

slots and connected using two Cabline SA-II Plus harnesses, as shown in the below figure.

Part Number MCX654105A-

HCAT

MCX654106A-

HCAT

MCX654106A-ECAT

Form Factor/Dimensions Adapter Card: PCIe Half Height, Half Length / 167.65mm x 68.90mm

Auxiliary PCIe Connection Card: 5.09 in. x 2.32 in. (129.30mm x 59.00mm)

Two 35cm Cabline CA-II Plus harnesses

Data Transmission Rate Ethernet: 10/25/40/50/100/200

Gb/s

InfiniBand: SDR, DDR, QDR, FDR,

EDR, HDR100, HDR

Ethernet: 10/25/40/50/100 Gb/s

InfiniBand: SDR, DDR, QDR, FDR, EDR,

HDR100

Network Connector Type Single-port

QSFP56

Dual-port QSFP56

PCIe x16 through Edge Connector PCIe Gen 3.0 / 4.0SERDES@ 8.0GT/s / 16.0GT/s

PCIe x16 through Auxiliary Card PCIe Gen 3.0SERDES@ 8.0GT/s

RoHS RoHS Compliant

Adapter IC Part Number MT28908A0-XCCF-HVM

ConnectX-6 Single-slot Socket Direct Cards (2x PCIe x8 in a

row)

The PCIe x16 interface comprises two PCIe x8 in a row, such that each of the PCIe x8 lanes can be

connected to a dedicated CPU in a dual-socket server. In such a configuration, Socket Direct brings

lower latency and lower CPU utilization as the direct connection from each CPU to the network means

the interconnect can bypass a QPI (UPI) and the other CPU, optimizing performance and improving

latency. CPU utilization is improved as each CPU handles only its own traffic and not traffic from the

other CPU.

A system with a custom PCI Express x16 slot that includes special signals is required for installing the

card. Please refer toPCIe Express Pinouts Description for Single-Slot Socket Direct Card



for pinout

definitions.

Part Number MCX653105A-EFAT MCX653106A-EFAT

Form Factor/Dimensions PCIe Half Height, Half Length / 167.65mm x 68.90mm

Data Transmission Rate Ethernet: 10/25/40/50/100 Gb/s

InfiniBand: SDR, DDR, QDR, FDR, EDR, HDR100

Network Connector Type Single-port QSFP56 Dual-port QSFP56

PCIe x16 through Edge Connector PCIe Gen 3.0 / 4.0 SERDES @ 8.0GT/s / 16.0GT/s Socket Direct 2x8 in a row

RoHS RoHS Compliant

Adapter IC Part Number MT28908A0-XCCF-HVM

Package Contents

ConnectX-6 PCIe x8/x16 Adapter Cards

Category Qty Item

Cards 1 ConnectX-6 adapter card

Accessories 1 Adapter card short bracket

1 Adapter card tall bracket (shipped assembled on the

card)

ConnectX-6 Socket Direct Cards (2x PCIe x16)

Category Qty. Item

Cards 1 ConnectX-6 adapter card

1 PCIe Auxiliary Card

Harnesses 1 35cm Cabline CA-II Plus harness (white)

1 35cm Cabline CA-II Plus harness (black)

Accessories

2 Retention Clip for Cablline harness (optional accessory)

1 Adapter card short bracket

1 Adapter card tall bracket (shipped assembled on the

card)

1 PCIe Auxiliary card short bracket

1 PCIe Auxiliary card tall bracket (shipped assembled on

the card)

Applies to MCX651105A-EDAT, MCX653105A-ECAT, MCX653106A-ECAT, MCX653105A-HDAT,

MCX653106A-HDAT, MCX653105A-EFAT, MCX653106A-EFAT.



Applies to MCX654105A-HCAT, MCX654106A-HCAT and MCX654106A-ECAT.



•

Features and Benefits

PCI

Express

(PCIe)

Uses the following PCIe interfaces:

PCIe x8/x16 configurations:

PCIe Gen 3.0 (8GT/s) and Gen 4.0 (16GT/s) through an x8/x16 edge connector.

2x PCIe x16 configurations:

PCIe Gen 3.0/4.0 SERDES @ 8.0/16.0 GT/s through Edge Connector

PCIe Gen 3.0 SERDES @ 8.0GT/s through PCIe Auxiliary Connection Card

200Gb/s

Virtual

Protocol

Interconne

ct (VPI)

Adapter

ConnectX-6 offers the highest throughput VPI adapter, supporting HDR 200b/s InfiniBand and

200Gb/s Ethernet and enabling any standard networking, clustering, or storage to operate

seamlessly over any converged network leveraging a consolidated software stack.

InfiniBand

Architectu

Specificati

on v1.3

compliant

ConnectX-6 delivers low latency, high bandwidth, and computing efficiency for performance-driven

server and storage clustering applications. ConnectX-6 is InfiniBand Architecture Specification

v1.3 compliant.

Up to 200

Gigabit

Ethernet

Mellanox adapters comply with the following IEEE 802.3 standards:

200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE

- IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet

- IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes

- IEEE 802.3ba 40 Gigabit Ethernet

- IEEE 802.3by 25 Gigabit Ethernet

- IEEE 802.3ae 10 Gigabit Ethernet

- IEEE 802.3ap based auto-negotiation and KR startup

- IEEE 802.3ad, 802.1AX Link Aggregation

- IEEE 802.1Q, 802.1P VLAN tags and priority

- IEEE 802.1Qau (QCN)

- Congestion Notification

- IEEE 802.1Qaz (ETS)

- IEEE 802.1Qbb (PFC)

- IEEE 802.1Qbg

- IEEE 1588v2

- Jumbo frame support (9.6KB)

InfiniBand

HDR100

A standard InfiniBand data rate, where each lane of a 2X port runs a bit rate of 53.125Gb/s with a

64b/66b encoding, resulting in an effective bandwidth of 100Gb/s.

InfiniBand

HDR

A standard InfiniBand data rate, where each lane of a 4X port runs a bit rate of 53.125Gb/s with a

64b/66b encoding, resulting in an effective bandwidth of 200Gb/s.

Memory

Componen

SPI Quad - includes 256Mbit SPI Quad Flash device (MX25L25645GXDI-08G device by

Macronix)

FRU EEPROM - Stores the parameters and personality of the card. The EEPROM capacity is

128Kbit. FRU I2C address is (0x50) and is accessible through the PCIe SMBus.

Make sure to use a PCIe slot that is capable of supplying the required power and airflow to the

ConnectX-6 cards as stated in Specifications.



•

Overlay

Networks

In order to better scale their networks, datacenter operators often create overlay networks that

carry traffic from individual virtual machines over logical tunnels in encapsulated formats such as

NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the

hardware offloading engines, placing higher loads on the host CPU. ConnectX-6 effectively

addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that

encapsulate and de-capsulate the overlay protocol.

RDMA and

RDMA over

Converged

Ethernet

(RoCE)

ConnectX-6, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged

Ethernet) technology, delivers low-latency and high-performance over InfiniBand and Ethernet

networks. Leveraging datacenter bridging (DCB) capabilities as well as ConnectX-6 advanced

congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services

over Layer 2 and Layer 3 networks.

Mellanox

PeerDirect

™

PeerDirect™ communication provides high efficiency RDMA access by eliminating unnecessary

internal data copies between components on the PCIe bus (for example, from GPU to CPU), and

therefore significantly reduces application run time. ConnectX-6 advanced acceleration technology

enables higher cluster efficiency and scalability to tens of thousands of nodes.

CPU

Offload

Adapter functionality enabling reduced CPU overhead allowing more available CPU for

computation tasks.

Flexible match-action flow tables

Open VSwitch (OVS) offload using ASAP2(TM)

Tunneling encapsulation / decapsulation

Quality of

Service

(QoS)

Support for port-based Quality of Service enabling various application requirements for latency

and SLA.

Hardware-

based I/O

Virtualizati

ConnectX-6 provides dedicated adapter resources and guaranteed isolation and protection for

virtual machines within the server.

Storage

Accelerati

A consolidated compute and storage network achieves significant cost-performance advantages

over multi-fabric networks. Standard block and file access protocols can leverage:

RDMA for high-performance storage access

NVMe over Fabric offloads for target machine

Erasure Coding

T10-DIF Signature Handover

SR-IOV ConnectX-6 SR-IOV technology provides dedicated adapter resources and guaranteed isolation and

protection for virtual machines (VM) within the server.

High-

Performan

Accelerati

ons

Tag Matching and Rendezvous Offloads

Adaptive Routing on Reliable Transport

Burst Buffer Offloads for Background Checkpointing

•

Operating Systems/Distributions

OpenFabrics Enterprise Distribution (OFED)

RHEL/CentOS

Windows

FreeBSD

VMware

OpenFabrics Enterprise Distribution (OFED)

OpenFabrics Windows Distribution (WinOF-2)

Connectivity

Interoperable with 1/10/25/40/50/100/200 Gb/s InfiniBand/VPI and Ethernet switches

Passive copper cable with ESD protection

Powered connectors for optical and active cable support

Manageability

ConnectX-6 technology maintains support for manageability through a BMC. ConnectX-6 PCIe stand-up

adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a

standard Mellanox PCIe stand-up adapter. For configuring the adapter for the specific manageability

solution in use by the server, please contact Mellanox Support.

ConnectX-6 Socket Direct cards 2x PCIe x16 (OPNs: MCX654105A-HCAT, MCX654106A-HCAT

and MCX654106A-ECAT) are not supported in Windows and WinOF-2.



•

Interfaces

InfiniBand Interface

The network ports of the ConnectX®-6 adapter cards are compliant with the

InfiniBand Architecture

Specification, Release 1.3.

InfiniBand traffic is transmitted through the cards' QSFP56 connectors.

Ethernet QSFP56 Interfaces

The network ports of the ConnectX-6 adapter card are compliant with the IEEE 802.3 Ethernet

standards listed in Features and Benefits. Ethernet traffic is transmitted through the QSFP56

connectors on the adaptercard.

PCI Express Interface

ConnectX®-6 adapter cards support PCI Express Gen 3.0/4.0 (1.1 and 2.0 compatible) through x8/x16

edge connectors. The device can be either a master initiating the PCI Express bus operations, or a

slave responding to PCI bus operations.

The following lists PCIe interface features:

PCIe Gen 3.0 and 4.0 compliant, 2.0 and 1.1 compatible

2.5, 5.0, 8.0, or 16.0 GT/s link rate x16/x32

Auto-negotiates to x32, x16, x8, x4, x2, or x1

Support for MSI/MSI-X mechanisms

LED Interface

There are two I/O LEDs per port:

LED 1 and 2: Bi-color I/O LED which indicates link status. LED behavior is described below for

Ethernet and InfiniBand port configurations.

The adapter card includes special circuits to protect from ESD shocks to the card/server when

plugging copper cables.



The adapter card includes special circuits to protect from ESD shocks to the card/server when

plugging copper cables.



• LED 3 and 4: Reserved for future use.

LED1 and LED2 Link Status Indications (Physical and Logical) - Ethernet Protocole:

LED Color and State Description

Off A link has not been established

Blinking amber 1 Hz Blinking amber occurs due to running a beacon command

for locating the adapter card

4 Hz blinking amber indicates a problem with the physical link

Solid green Indicates a valid link with no active traffic

Blinking green Indicates a valid logical link with active traffic

LED1 and LED2Link Status Indications(Physical and Logical) - InfiniBand Protocole:

LED Color and State Description

Off A physical link has not been established

Solid amber Indicates an active physical link

Blinking amber 1 Hz Blinking amber occurs due to running a beacon command

for locating the adapter card

4 Hz blinking amber indicates a problem with the physical link

Solid green Indicates a valid logical (data activity) link with no active traffic

Blinking green Indicates a valid logical link with active traffic

Heat Sink Interface

The heatsink is attached to the ConnectX-6 IC in order to dissipate the heat from the ConnectX-6 IC. It

is attached either by using four spring-loaded push pins that insert into four mounting holes, or by

screws.

ConnectX-6 IC has a thermal shutdown safety mechanism which automatically shuts down the

ConnectX-6 card in cases of high-temperature event, improper thermal coupling or heatsink removal.

For the required airflow (LFM) per OPN, please refer toSpecifications.

SMBus Interface

ConnectX-6 technology maintains support for manageability through a BMC. ConnectX-6 PCIe stand-up

adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a

standard Mellanox PCIe stand-up adapter. For configuring the adapter for the specific manageability

solution in use by the server, please contact Mellanox Support.

Voltage Regulators

The voltage regulator power is derived from the PCI Express edge connector 12V supply pins. These

voltage supply pins feed on-board regulators that provide the necessary power to the various

components on the card.

Page is loading ...

Nvidia Mellanox ConnectX-6 MCX654106A-ECAT User manual

Brand: Nvidia

Model: Mellanox ConnectX-6 MCX654106A-ECAT

Type: User manual

This manual is also suitable for

Download PDF

report

Nvidia Mellanox ConnectX-6 MCX654106A-ECAT User manual

This manual is also suitable for

Nvidia Mellanox ConnectX-6 MCX654106A-ECAT User manual

Related papers

Other documents