Mellanox Technologies MT51336-VPI-EVB User manual

www.mellanox.com

SwitchX Evaluation Board Hardware User

Manual

PN: MT51336-EN-EVB, MT51336-VPI-EVB

Rev 1.0

Mellanox Technologies

350 Oakmead Parkway Suite 100

Sunnyvale, CA 94085

U.S.A.

www.mellanox.com

Tel: (408) 970-3400

Fax: (408) 970-3403

Mellanox Technologies, Ltd.

PO Box 586 Hermon Building

Yokneam 20692

Israel

Tel: +972-4-909-7200

Fax: +972-4-959-3245

Mellanox®, BridgeX®, ConnectX®, CORE-Direct®, InfiniBridge®, InfiniHost®, InfiniScale®, PhyX®, Virtual Protocol Interconnect and

Voltaire are registered trademarks of Mellanox Technologies, Ltd.

FabricIT, MLNX-OS and SwitchX are trademarks of Mellanox Technologies, Ltd.

All other trademarks are property of their respective owners.

SwitchX Switch EVB Hardware User Manual

Document Number: 3493

Mellanox Technologies

2

NOTE:

THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PRO-

VIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF

AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUS-

TOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES

TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CAN-

NOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY.

ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MER-

CHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT

SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEM-

PLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCURE-

MENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)

HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT

(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED

DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

SwitchX Switch EVB Hardware User Manual Rev 1.0

Mellanox Technologies

3

Pr

e

liminary

Dr

a

ft

Table of Contents

Revision 8

Chapter 1 Overview 9

1.1 Management CPU 9

1.2 PCIE/1GbE Extender Kit 10

1.2.1 PCIE Extender Kit 10

1.2.2 1GbE Extender Kit 10

1.3 Compliance Ports 11

1.4 Reference Traces 11

1.5 Features 12

1.6 Serial Number and Product Version Information 13

Chapter 2 40 Gb/s Ethernet 14

Chapter 3 FDR 15

Chapter 4 Basic Operation and Installation 16

4.1 Switch Platform Hardware Overview 16

4.1.1 Status LEDs 17

4.1.2 Reset Button 22

4.2 QSFP Cable Power Budget Classification 23

4.3 SFP+ Cable Power Budget Classification 23

4.4 Interfaces 23

4.4.1 Port Connector Interfaces 23

4.4.2 Management and Firmware Updating Interfaces 24

4.5 Switch Platform Installation and Operation 26

4.5.1 Installation Safety Warnings 26

4.5.2 Mechanical Installation 28

4.6 Package Contents and Installation 29

4.6.1 Installing the Switch in the Rack 29

4.6.2 Grounding the Switch 33

4.6.3 Power Connections and Initial Power On 34

4.6.4 Extracting and Inserting the Power Supply Unit 34

4.6.5 Cable Installation 36

4.6.6 Extracting and Inserting the Fan Unit 36

Chapter 5 Installing the PCIE/1GbE Server Side board 38

5.1 PCIE/1GbE Extender Kit parts 38

5.2 Removing the Top Cover 40

5.3 Removing the Management CPU 42

5.4 Installing the PCIE/1GbE Extender 43

5.5 Installing the Server Side Board 45

5.6 Board Connectors Numbering and Locations 46

5.7 Switch Shut Down Procedure 49

5.8 Disassembly of the Switch from the Rack 49

5.9 Disposal 50

5.10 Resetting the Switch 50

Chapter 6 Management and Tools Overview 51

6.1 Chassis Management Using the MLNX-OS Stack 51

6.2 Network Management and Clustering Software 51

6.3 Chassis Management 52

6.3.1 Open SM 52

6.4 Fabric Management with UFM 53

6.5 Configuring the Switch for the First Time 54

Chapter 7 Connecting to the Switch Platform 60

Rev 1.0

Mellanox Technologies

4

Pre

limin

a

r

y

D

raft

7.1 Starting an SSH Connection to the Switch (CLI) 60

7.2 Starting a WebUI Connection to the Switch 60

7.2.1 Managing the System 62

7.2.2 I2C Connector 62

7.2.3 How to Get Mellanox Firmware Tools (MFT) 62

7.3 Updating Firmware 63

Chapter 8 Troubleshooting 64

SwitchX Switch EVB Hardware User Manual Rev 1.0

Mellanox Technologies

5

Pr

e

liminary

Dr

a

ft

List of Figures

Figure 1: Connector and Power Side Panels 9

Figure 2: Pull Out Tab 13

Figure 3: Connector and Power Side Panel 16

Figure 4: Power Side Panel 16

Figure 5: Power, Fan, and System LEDs 18

Figure 6: System Health LED 18

Figure 7: Power Side Panel 19

Figure 8: PSU Status LEDs 20

Figure 9: Fan Status LED Connector Side 20

Figure 10: Fan Status LED Power side 21

Figure 11: Reset Button 22

Figure 12: Port Numbering 23

Figure 13: Top and Bottom Ports 24

Figure 14: Management Interfaces 24

Figure 15: Rack Rail Kit Parts 30

Figure 16: Mounting Options 31

Figure 17: Screwing on the Rail 32

Figure 18: Inserting the Caged Nuts 32

Figure 19: Slide the Rail into the Rail Slide 33

Figure 20: Power Supply Unit Extraction 35

Figure 21: PSU Pulled Out 35

Figure 22: Top and Bottom Ports 36

Figure 23: Air Flow Labels 37

Figure 24: Fan Module Latches 37

Figure 25: Server Side Card 39

Figure 26: PCIE/1GbE Extender 40

Figure 27: Screw Locations 41

Figure 28: Slide the Top 41

Figure 29: Management Board Installed 42

Figure 30: Unscrew the Management Board 42

Figure 31: Remove the Management Board 43

Figure 32: Holes for the Extender Board Interfaces 43

Figure 33: Install the Extender Board 44

Figure 34: Connecting the EVB to the Server Side Board 45

Figure 35: EVB Board Connections View 1 46

Rev 1.0

Mellanox Technologies

6

Pre

limin

a

r

y

D

raft

Figure 36: Reset Button 50

Figure 37: Management Interfaces 54

Figure 38: Transfer Power Cord 71

Figure 39: Install the Power Cord Clip 72

Figure 40: Transfer Power Cord Finished 72

Figure 41: Connector Pinout 73

SwitchX Switch EVB Hardware User Manual Rev 1.0

Mellanox Technologies

7

Pr

e

liminary

Dr

a

ft

List of Tables

Table 1: Revision History 8

Table 2: System Status LED Configurations 18

Table 3: Connector Physical and Logical Link Indications 19

Table 4: PSU Status LED Configurations 20

Table 5: Fan Status LED Configurations 21

Table 6: Bad Port LED Configurations 22

Table 7: Port Connector Numbering 47

Table 8: Serial Terminal Program Configuration 54

Table 9: Configuration Wizard Session - IP Configuration by DHCP 55

Table 10: Configuration Wizard Session - IP Zeroconf Configuration 57

Table 11: Configuration Wizard Session - Static IP Configuration 58

Table 12: Adapter Cards Certification Status 67

Table 13: Specifications for MT51336-NUDT 69

Table 14: Connector Pinout 73

Rev 1.0

Mellanox Technologies

8

Pre

limin

a

r

y

D

raft

Revision

Table 1 - Revision History

Revision Description

1.0 Initial Release

0.7 Added the correct H x W x D and weight

Added Power numbers

Added verified compliance port to use.

0.6 Added the word “Hardware” to the name.

Added correct graphics of the server side card

0.5 Added tie down the cables

Added Caution Note do not use as a shelf.

0.4 fixed graphics

0.3 More fixes

0.2 Added Initial configuration section

0.1 Preliminary Release

SwitchX Switch EVB Hardware User Manual Rev 1.0

Mellanox Technologies

9

Pr

e

liminary

Dr

a

ft

1Overview

This EVB is intended to be used as a platform to evaluate the SwitchX silicon. SW engineers can

use it as a means for SW developement using the SwitxchX SDK. SW developers can implement

initial driver development/integration into non PPC CPU environments.

Signal integrity engineers can use it to evaluate SwitchX SI performance and compliance to spec.

This EVB can be used to:

• evaluate and qualify



Ethernet SerDes



IB SerDes



FC SerDes

• Initial Ethernet and IB switch testing for L2/L2+

This EVB can be used both while rack mounted (standard 19” rack) and in a tabletop setup(for

Signal Integrity evaluation).

Figure 1: Connector and Power Side Panels

1.1 Management CPU

The EVB comes with a Power PC 460E CPU for management. It also comes with a PCIE/1GbE

Extender that allows the user bypass the management CPU and to connect directly to the chip.

When the PCIE/1GbE Extender is used the Power PC is removed from the system

and none of the management interfaces on the connector side will work.

When the Power PC is removed the switch/ EVB becomes unmanaged and does

NOT need Initial configuration.

MT51336-EN-EVB

CONSOLE

MGT

RST

UID

PS1

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

35

36

33

34

1

2

4

3

Mellanox

!

OK

!

OK

I2C

Overview

Rev 1.0

Mellanox Technologies

10

Pre

limin

a

r

y

D

raft

1.2 PCIE/1GbE Extender Kit

The Power PC management module can be removed and in its place a PCIE/1GbE extender can be

installed. When the Management module is removed the EVB becomes externally managed. The

PCIE/1GbE extender kit contains the system side board and the server side board.

The board has four connectors:

The server side board in the kit, gets inserted in a server. Connect the card into a 8X or 16X PCI

slot. This card has 3 ports:

1.2.1 PCIE Extender Kit

This option allows the EVB to be managed through the server. This is done with three cables con-

necting the EVB to the Server. One cable is for clocking and reset and two cables are for data TX

and RX.

1.2.2 1GbE Extender Kit

The PCIE extender allows for remote management through the 10/100/1000Mb RJ45 Ethernet

connection on the EVB Extender side board. This is connected to any standard Ethernet port fabric

management port.

Interface Description

CX4 connector for clocking and reset

2 QSFP connectors one is blue, one is green

• The green one goes from the green labeled connector on the EVBduaghter board to the

green label on the server side card included in the kit.

• The blue one goes from the blue labeled connector on the EVBduaghter board to the blue

label on the server side card included in the kit.

Server Side board a card that gets inserted into the server X8 PCI slot.

Use only one option the PCIE or the Ethernet, do not use both

RJ 45 connector for management through an ethernet port

Interface Description

CX4 connector for clocking and reset

2 QSFP connectors one is blue, one is green

• The green one goes from the green labeled connector on the EVBduaghter board to the

green label on the server side card included in the kit.

• The blue one goes from the blue labeled connector on the EVBduaghter board to the blue

label on the server side card included in the kit.

SwitchX Switch EVB Hardware User Manual Rev 1.0

Mellanox Technologies

11

Pr

e

liminary

Dr

a

ft

1.3 Compliance Ports

Ports 3 and 4 and 33 and 34 are connected to the chip. These are compliance ports and go to the

SMA/mini-SMP interfaces. Ports 3 and 4 are SMA ports and ports 33 and34 are mini-SMP ports.

Compliance ports are the worst case loss (for all of the on board connectors) through the channel.

They are designed to have an insertion loss of 5.5dB as defined in SFF-8431.

See Figure 12 on page 23 for identifying the trace and compliance ports. See Figure 35 on page 46

and Table 7 on page 47 for locations and numbering of the trace and compliance port board con-

nections.

1.4 Reference Traces

Ports 1 and 2 and 35 and 36 are not connected to the chip. These are reference ports and go to the

SMA/mini-SMP interfaces. See Figure 35 on page 46 and Table 7 on page 47 For locations and

numbering of the board connections. Ports 1 and 2 are SMAports and ports 35 and36 are mini-

SMP ports.

These reference traces are designed to have identical insertion loss as the corresponding compli-

ance ports, and those traces are routed in the same geometry and length as the traces of the compli-

ance ports.

As of this revision only port 4 is verified at the proper designed insertion loss.

Use port 4 for compliance.

Overview

Rev 1.0

Mellanox Technologies

12

Pre

limin

a

r

y

D

raft

1.5 Features

Full Feature List

• Reference traces SMA and mini-SMP

• Compliance ports SMA and mini-SMP

• 2 ports connect to mini SMP connectors for SI testing

• 24 QSFP ports

• 1 PSU

• 12 SFP+ ports

• RADIUS

• TACACS+

• LDAP

• SSH

• DHCP

•CLI

• IPv6 MNG

•Telnet

• FTP

• RSTP

• PFC

• Static MAC

• TFTP

•SCP

• SNTP

• AAA

•Syslog

•Dual SW image

• Auto temprature control

• Upload/download configuration

• System alarms

• VLAN aware

• ETS

• Flow Control

• 36 FDR (56Gb/s) ports in a 1U switch

• IBTA 1.3 and 1.21 complaint

• SDR/DDR/QDR/FDR10/FDR link speed

• Congestion control

• Adaptive routing

SwitchX Switch EVB Hardware User Manual Rev 1.0

Mellanox Technologies

13

Pr

e

liminary

Dr

a

ft

• Port mirroring

• FDR10 supports 20% more bandwidth over regular QDR using the same cables/connectors**

• Up to 8 multiple switch partitions*

• All ports can be set to gateway ports connecting to 40GigE (over QSFP) or 1/10GigE (over

QSA)*

• 4.032 Tb/s aggregate data switching capacity with ultra low latency

• CBF *

• DCBX *

• LAG/LACP *

• QCN *

1.6 Serial Number and Product Version Information

The Serial number and GUID for the switch and the MAC for the Management PC are found on

the pull out tab below the USB interface connection.

Figure 2: Pull Out Tab

S/N: MT1117X00014

P/N: MT51336-EN-EVB

MAC: 0002C927051C

Made in IL

MGT 1:

00:02:C9:11:A2:01

MGT 2:

00:02:C9:11:A2:02

Rev:A1

40 Gb/s Ethernet

Rev 1.0

Mellanox Technologies

14

Pre

limin

a

r

y

D

raft

2 40 Gb/s Ethernet

Each QSFP port is capable of up to 40GigE and with a QSA the ports can run at 10GigE/1GigE.

Up to 24 QSFP ports are capable of 1/2/4/8G Fibre Channel with a QSA.

All ports are capable of virtual switching (VEPA+). All usable virtual switching ports are grouped

together.

The port protocol can only be obtained from the management interface.

40 Gb ETH is only guaranteed to work with approved Mellanox Cables.

SwitchX Switch EVB Hardware User Manual Rev 1.0

Mellanox Technologies

15

Pr

e

liminary

Dr

a

ft

3FDR

The SwitchX EVB supports FDR, a pre-standard InfiniBand data rate, where each lane of a 4X

port runs a bit rate of 14.0625Gb/s with a 64b/66b encoding, resulting in an effective bandwidth of

54.54Gb/s. The FDR physical layer is an IBTA specified physical layer using different block

types, deskew mechanism and framing rules.

SwitchX™ also supports FDR10, a non-standard InfiniBand data rate, where each lane of a 4X

port runs a bit rate of 10.3125Gb/s with a 64b/66b encoding, resulting in an effective bandwidth of

40Gb/s.

FDR10 supports 20% more bandwidth over regular QDR using the same QSFP cables/connectors.

Both FDR and FDR10 support Forward Error Correction (FEC), as described in IEEE 802.3ap

chapter 74.

FDR10 is only guaranteed to work with approved Mellanox ConnectX-3 adapters.

Basic Operation and Installation

Rev 1.0

Mellanox Technologies

16

Pre

limin

a

r

y

D

raft

4 Basic Operation and Installation

4.1 Switch Platform Hardware Overview

Figure 4 shows the power side panel and connector side panel views of the switch EVB.

Figure 3: Connector and Power Side Panel

The figure shows port configurations for the switch systems, including:

• 2 – Ethernet RJ45 connectors

• 1 – RJ45 connector for connecting to a host PC (CONSOLE)

• 1 – USB connector

• 24 QSFP ports

• 12 SFP+ ports

• various switch, port, fan and power status LEDs

Figure 4: Power Side Panel

• 1 – hot-swap fan module

• 1 – power supply unit

• 1 – I2C banana connector

• various switch, port, fan and power status LEDs

All connectivity except for power cords is via the connector side panel. All connectors can support

active cables.

MT51336-EN-EVB

CONSOLE

MGT

RST

UID

PS1

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

35

36

33

34

1

2

4

3

Mellanox

!

OK

!

OK

I2C

SwitchX Switch EVB Hardware User Manual Rev 1.0

Mellanox Technologies

17

Pr

e

liminary

Dr

a

ft

4.1.1 Status LEDs

4.1.1.1 System Status Indicators

The System Status Indicators( ) are located to the left of the connectors on the connector side

panel, and on the power side at the far right. Both of these LEDs give identical information.

The system status indicators should display as follows:

• When the switch is plugged in, within three minutes the STATUS LED should light up green.

• The PSU LED should light up green.

• The FAN LED should light up green.

If the FAN LED shows red, troubleshoot the fan module.

If the PSU LED is not green, this indicates a problem with the power supply. The switch is opera-

tional only if the PSU LED is green.

As long as there is power to the switch and the switch is booted up and running, the

status LED should be green.

If the STATUS LED shows red after three minutes, unplug the switch and call your

Mellanox representative for assistance.

If the switch shuts down due to over temperature, unplug the switch, wait 5 minutes

and replug in the switch. For more information See “Troubleshooting” on page 64.

OK

RST

UID

PS1

Power side

status LED

Connector side

status LED

Green – OK

Yellow – a fault in the sys-

tem

Red – Major Fault or

Fatal Error

Led Color shows the

switch status

Basic Operation and Installation

Rev 1.0

Mellanox Technologies

18

Pre

limin

a

r

y

D

raft

Figure 5: Power, Fan, and System LEDs

4.1.1.2 Power Side Panel System LED

On the right side of the power side panel is a single system LED that displays the health of the

switch. This indicator is the same as the system status indicator on the other side of the switch. The

LED assignment is as in Table 2.

Figure 6: System Health LED

4.1.1.3 Port Connector LED Assignment

Above the ports are two LEDs one for the upper port and one for the lower port . The fol-

lowing table shows the port status according to the LED indication.

Table 2 - System Status LED Configurations

LED Configuration STATUS/ System Health LED

Green OK – The system is up and running.

Flashing Green The system is booting up.

Yellow Error – A fault in the system, most likely

the firmware did not BOOT properly.

Red Major Error – Possible damage can result to the switch. Turn off immediately.

e.g. bad firmware, can’t boot, overheated

Off Off – The system has no power.

RST

UID

PS1

OK

This is a three color LED

Green — OK

Yellow — Fault in the system

Red — Major fault

SwitchX Switch EVB Hardware User Manual Rev 1.0

Mellanox Technologies

19

Pr

e

liminary

Dr

a

ft

4.1.1.4 Power Supply Status Indicators

The switch EVB is only available with one installed Power Supply Unit.

Figure 7: Power Side Panel

The power supply unit (PSU) is located on the right side of the power side panel. The PSU has a

single 2 color LED on the right side of the PSU, that indicates the internal status of the unit.

Table 3 - Connector Physical and Logical Link Indications

LED Status LED Description

Off No power to the port

Solid Green Logical link up

Flashing Green Data activity flashing speed ≈ data transfer speed

Orange Physical link up

Flashing Orange A problem with the physical link. Usually the SM is down or not enabling the port.

Fan Unit

Mellanox

!

OK

!

OK

I2C

Power Supply Unit

PSU1 status LED

Fan status LED

Basic Operation and Installation

Rev 1.0

Mellanox Technologies

20

Pre

limin

a

r

y

D

raft

Figure 8: PSU Status LEDs

4.1.1.5 Fan Status Indicators

The indicator is located to the left of the connectors on the connector side panel.

Figure 9: Fan Status LED Connector Side

The LED indicator on the Fan Module is on the right side of the module.

Table 4 - PSU Status LED Configurations

LED Color Status

Green OK – The Power supply is delivering the correct voltage. 12VDC

Red Error – The PSU is not operational

Off Off – There is no power to the system .

!

OK

This is a two color LED

Green — OK

Red — Major fault

Off — No power to the system

Fan air flow indicator

RST

UID

PS1

This is a two color LED

Green — OK

Red — Major fault

Off — No power to the system

Page is loading ...

Mellanox Technologies MT51336-VPI-EVB User manual

This manual is also suitable for

Mellanox Technologies MT51336-VPI-EVB User manual

Related papers

Other documents