Mellanox Technologies sRB-20210G User manual

www.mellanox.com

Mellanox Technologies

10 GbE Gateway

(sRB-20210G Module)

User Manual

Version 2.0

2

Document Number: DOC-00288 A01

Mellanox Technologies

NOTE:

THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE

PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF

AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE

CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX

TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX

TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE

HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON INFRINGEMENT ARE

DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,

INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO,

PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS

INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,

OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND

RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Mellanox Technologies, Inc.

350 Oakmead Parkway Suite 100

Sunnyvale, CA 94085

U.S.A.

www.mellanox.com

Tel: (408) 970-3400

Fax: (408) 970-3403

Mellanox Technologies Ltd

PO Box 586 Hermon Building

Yokneam 20692

Israel

Tel: +972-4-909-7200

Fax: +972-4-959-3245

Mellanox®, BridgeX®, ConnectX®, SwitchX®, CORE-Direct®, InfiniBridge®, InfiniHost®, InfiniScale®, PhyX®, Virtual Protocol

Interconnect and Voltaire are registered trademarks of Mellanox Technologies, Ltd.

FabricIT and MLNX-OS are trademarks of Mellanox Technologies, Ltd.

All other trademarks are property of their respective owners.

Contents

Version 2.0

3

Mellanox Technologies

Contents

Chapter 1. Introduction ........................................................................................................... 10

1.1 Overview ............................................................................................................................... 10

1.2 Physical Description ............................................................................................................. 10

1.2.1 Plug-in Module ........................................................................................................ 10

1.2.2 LED Indicators ........................................................................................................ 11

1.3 Features ................................................................................................................................ 12

1.3.1 Basic Features ........................................................................................................ 12

1.3.2 New in Version 2.0 .................................................................................................. 12

1.4 Functional Description .......................................................................................................... 12

1.4.1 High Performance InfiniBand-to-GbE Connectivity ................................................ 12

1.4.2 Comprehensive Management................................................................................. 13

1.4.3 sRB-20210G Architecture ....................................................................................... 13

1.4.4 Supported Protocols ............................................................................................... 13

1.4.5 sRB-20210G Application Diagram .......................................................................... 14

1.4.6 Traffic Flow Management Overview ....................................................................... 14

1.5 Technical Specifications ....................................................................................................... 14

1.5.1 General ................................................................................................................... 14

1.5.2 10 Gigabit Ethernet (GbE) ...................................................................................... 14

1.5.3 InfiniBand ................................................................................................................ 15

1.5.4 IETF Protocols ........................................................................................................ 15

1.5.5 Management Interfaces .......................................................................................... 15

1.5.6 Physical ................................................................................................................... 15

1.5.7 Environmental ......................................................................................................... 15

1.5.8 Certifications ........................................................................................................... 15

Chapter 2. Initial Configuration .............................................................................................. 16

2.1 Prerequisite Installation Tasks .............................................................................................. 16

2.2 Configuring the sRB-20210G Gateway in the ISR 2XXX ..................................................... 16

2.2.1 Managing the Unit over Fast Ethernet (Out-of-Band) or InfiniBand (Inband) ......... 16

2.2.2 Handling Different Subnets ..................................................................................... 18

2.3 Configuring an sRB-20210G Module (10 GbE to InfiniBand Bridge) ................................... 20

2.4 Setting Time and Date .......................................................................................................... 21

2.5 Default Settings .................................................................................................................... 21

2.6 Users and Passwords ........................................................................................................... 22

2.7 Verifying the Configuration ................................................................................................... 23

Chapter 3. Mellanox Device Manager (VDM) ........................................................................ 25

3.1 Overview ............................................................................................................................... 25

Version 2.0

Contents

4

Mellanox Technologies

3.2 Device Manager Description ................................................................................................ 25

3.3 Logging-in to the Device Manager........................................................................................ 26

3.4 The Switch Device Manager Main Window .......................................................................... 27

3.4.1 The Main Menu ....................................................................................................... 29

3.4.2 Device or Navigation Tree (Left Panel) .................................................................. 29

3.4.3 sRB-20210G Gateway Graphical Representation .................................................. 30

3.4.4 Right-Click Menus ................................................................................................... 31

3.4.5 Left-Click Option ..................................................................................................... 33

3.5 Color-Coding Table .............................................................................................................. 33

3.6 Trap Table ............................................................................................................................ 34

3.6.1 Resizing the Trap Table .......................................................................................... 34

3.6.2 Rearranging Columns ............................................................................................. 34

3.6.3 Viewing and Hiding Events ..................................................................................... 34

3.6.4 Acknowledging Events ............................................................................................ 35

3.6.5 Displaying Additional Data for an Event ................................................................. 35

Chapter 4. Device Information ............................................................................................... 37

4.1 Getting Started with the sRB-20210G Device Manager ....................................................... 37

4.2 Device Information via the CLI ............................................................................................. 39

4.3 sRB-20210G Port Information .............................................................................................. 40

4.4 Viewing the System Time and Date (CLI only) ..................................................................... 41

4.5 Viewing sRB-20210G Temperature (Switch CLI only) ......................................................... 42

4.6 Setting the Uptime ................................................................................................................ 42

4.7 Checking the Device Software Version ................................................................................ 42

4.7.1 Device Version via the GUI ..................................................................................... 42

4.7.2 Device Version via the CLI ..................................................................................... 43

Chapter 5. Device Configuration ............................................................................................ 44

5.1 IP Settings ............................................................................................................................ 44

5.1.1 Overview ................................................................................................................. 44

5.1.2 Setting the I/O IP via the GUI ................................................................................. 44

5.1.3 Configuring the sRB-20210G IP via the Switch CLI ............................................... 45

5.2 Resetting the sRB-20120G ................................................................................................... 46

5.2.1 Resetting the I/O via the GUI .................................................................................. 46

5.2.2 Resetting the I/O via the CLI................................................................................... 46

5.3 User Settings (Setting Passwords) ....................................................................................... 47

5.3.1 User Settings via the GUI ....................................................................................... 47

5.3.2 User Settings via the CLI ........................................................................................ 48

Contents

Version 2.0

5

Mellanox Technologies

Chapter 6. Advanced Configuration ...................................................................................... 49

6.1 Updating the Software .......................................................................................................... 49

6.1.1 Identify the File Location for Software Update........................................................ 49

6.1.2 Updating the VDM (GUI) - Gateway Version Automatic Upgrade .......................... 51

6.2 Remote Management Settings (SNMP) ............................................................................... 51

6.2.1 Setting Remote Management via the GUI .............................................................. 51

6.2.2 Setting Remote Management via the CLI ............................................................... 53

6.3 Remote Configuration Settings (FTP/SCP) .......................................................................... 56

6.3.1 Setting Remote Configuration via the GUI ............................................................. 57

6.3.2 Setting Remote Configuration via the CLI .............................................................. 57

6.3.3 Backing up the Configuration via the GUI and the CLI ........................................... 58

6.4 Setting Link Aggregation (LAG) ............................................................................................ 59

6.4.1 Setting Link Aggregation via the GUI ..................................................................... 60

6.4.2 Setting Link Aggregation via the CLI ...................................................................... 61

6.5 Configuring I/O Interfaces ..................................................................................................... 62

6.6 Cluster Configuration ............................................................................................................ 63

6.6.1 Configuring a Cluster via the GUI ........................................................................... 63

6.6.2 Configuring a Cluster via the CLI ............................................................................ 65

6.6.3 Cluster Mode (sRB-20210G High Availability) ........................................................ 65

6.6.4 Setting sRB-20210G Algorithms ............................................................................. 66

6.6.5 Loop Prevention ...................................................................................................... 68

6.7 Setting the Event Policy Filters ............................................................................................. 70

6.8 Port MTU Configuration (Jumbo Frames) ............................................................................ 72

6.8.1 Setting the MTU via the GUI ................................................................................... 73

6.8.2 Setting the MTU via the CLI.................................................................................... 74

6.9 MIB Support .......................................................................................................................... 75

6.10 Security Support ................................................................................................................... 75

6.10.1 Overview ................................................................................................................. 75

6.10.2 Enabling/Disabling Non-secure Protocols .............................................................. 76

Chapter 7. Setting-up Networks ............................................................................................. 78

7.1 Setting up a Network/Interface via the GUI .......................................................................... 78

7.1.1 Network Table Right-click Menu ............................................................................. 78

7.1.2 Defining (Adding) a New Network .......................................................................... 79

7.1.3 Viewing Network Information .................................................................................. 80

7.1.4 Removing Networks ................................................................................................ 80

7.1.5 Updating the Default Gateway ................................................................................ 81

7.1.6 Removing a Default Gateway ................................................................................. 81

7.1.7 Refresh Table ......................................................................................................... 82

Version 2.0

Contents

6

Mellanox Technologies

7.2 Setting up a Network/Interface via the CLI ........................................................................... 82

7.2.1 Viewing Networks ................................................................................................... 82

7.2.2 Adding Networks ..................................................................................................... 82

7.2.3 Removing Networks ................................................................................................ 83

7.2.4 Setting the IP and Broadcast Address .................................................................... 83

7.2.5 Setting the Default Gateway ................................................................................... 83

Chapter 8. Traffic Flow Management ..................................................................................... 85

8.1 Overview ............................................................................................................................... 85

8.2 Setting TFM .......................................................................................................................... 85

8.2.1 Setting TFM via the GUI ......................................................................................... 85

Chapter 9. Static Route Configuration .................................................................................. 98

9.1 Overview ............................................................................................................................... 98

9.2 Routing Configuration ........................................................................................................... 98

9.2.1 Setting Routing via the GUI .................................................................................... 98

9.2.2 Routing Configuration via the CLI ......................................................................... 100

Chapter 10. Logs and Statistics ............................................................................................. 102

10.1 Using ’exportLog’ ................................................................................................................ 102

10.2 Viewing Port Counters Statistics ........................................................................................ 102

10.2.1 Viewing Port Counters Statistics via the GUI ....................................................... 102

10.2.2 Viewing Port Status and Statistics via the CLI ...................................................... 104

10.3 Statistics Logs ..................................................................................................................... 107

10.4 Viewing Traps/Events ......................................................................................................... 108

10.4.1 Viewing Events ..................................................................................................... 110

10.4.2 Viewing Traps ....................................................................................................... 110

Chapter 11. Multicast Settings ............................................................................................... 111

11.1 Overview ............................................................................................................................. 111

11.2 Definitions ........................................................................................................................... 111

11.3 Multicast Settings................................................................................................................ 111

11.4 MC-Table Session .............................................................................................................. 112

Chapter 12. Mellanox Fabric Manager (VFM) ........................................................................ 113

12.1 Overview ............................................................................................................................. 113

12.2 Viewing I/O System Information via the GridVision Fabric Manager ................................. 113

12.3 I/O System Overview Parameter Description ..................................................................... 114

12.4 Viewing an I/O via the CLI .................................................................................................. 114

12.5 Connecting to an I/O Device via the CLI ............................................................................ 115

12.6 Viewing I/O Port Properties ................................................................................................ 115

12.7 Viewing I/O Network (Partitioning) Information .................................................................. 117

Contents

Version 2.0

7

Mellanox Technologies

Chapter 13. Partitioning .......................................................................................................... 118

13.1 Overview ............................................................................................................................. 118

13.2 Brief Functional Description ................................................................................................ 118

13.3 Mechanisms Used to Create and Enforce Partitions .......................................................... 119

13.4 Quick Settings ..................................................................................................................... 120

13.4.1 Switch Settings (GUI) ........................................................................................... 120

13.4.2 sRB-20210G Settings (GUI) ................................................................................. 120

13.5 Setting Partitioning via the GridVision Fabric Manager GUI .............................................. 121

13.5.1 Creating a Network ............................................................................................... 122

13.5.2 Viewing Network Information ................................................................................ 124

13.5.3 Assigning Server Ports to a Network .................................................................... 125

13.5.4 Deleting Server Ports from a Network .................................................................. 125

13.5.5 Assigning Network Membership ........................................................................... 126

13.5.6 Mapping an InfiniBand Partition to an Ethernet VLAN via the 10 GbE Gateway . 126

13.5.7 Multiple Selection for Partitioning Configuration ................................................... 128

13.6 Setting Partitioning via the CLI ........................................................................................... 129

13.6.1 Quick Steps to set up a Network and Perform Partitioning .................................. 129

13.6.2 Creating a Network ............................................................................................... 129

13.6.3 Deleting a Network ............................................................................................... 130

13.6.4 Viewing Network Details ....................................................................................... 130

13.6.5 Viewing the Network Error Log File ...................................................................... 131

13.6.6 Binding a Port GUID to a Network ........................................................................ 131

Chapter 14. Command Line Interface (CLI) Management ................................................. 139

14.1 Starting a CLI Management Session .................................................................................. 139

14.2 Mellanox CLI Model ............................................................................................................ 139

14.3 CLI Operating Conventions ................................................................................................ 140

14.3.1 Keyboard Usage in CLI Operation ........................................................................ 140

14.3.2 CLI Command Structure ....................................................................................... 141

14.3.3 Case-Sensitivity .................................................................................................... 142

14.3.4 CLI Modes............................................................................................................. 142

14.4 Exec Mode Commands ...................................................................................................... 143

14.5 Privileged Mode Commands .............................................................................................. 150

14.5.1 Statistics Mode ..................................................................................................... 157

14.6 Configuration (Config) Commands ..................................................................................... 159

14.6.1 Cluster Options ..................................................................................................... 162

14.6.2 Diagnostics ........................................................................................................... 168

14.6.3 Interface Options .................................................................................................. 168

14.6.4 NTP Options ......................................................................................................... 171

Version 2.0

Contents

8

Mellanox Technologies

14.6.5 Port Options .......................................................................................................... 173

14.6.6 Remote Options .................................................................................................... 177

14.6.7 Route Options ....................................................................................................... 183

14.6.8 Security Options ................................................................................................... 184

14.6.9 SNMP Options ...................................................................................................... 186

14.6.10 Traffic Flow Manager (TFM) ................................................................................. 192

14.7 Safe Mode CLI Commands ................................................................................................ 200

14.8 Software Upgrade Procedure in Safe Mode ....................................................................... 202

14.9 CLI Safe Mode .................................................................................................................... 203

10 GbE Gateway

(sRB-20210G Module) User Manual

Version 2.0

9

Mellanox Technologies

About this Manual

This manual describes how to operate the sRB-20210G module, also referred to as Mellanox’s

10 GbE gateway. The manual describes the tasks commonly required to configure and use the

device.

Information regarding the sRB-20210G hardware installation can be found in the Mellanox

ISR 2004/2012 Grid Director Installation Manual.

NOTE:

Refer to the release notes for last minute updates and restrictions.

Audience

This manual is intended primarily for:

 System administrators who are familiar with the fundamentals of IP based networking,

but might not be familiar with the specifics of Mellanox products or the routing

protocols supported by Mellanox products

 System administrators who are responsible for configuring TCP/IP to InfiniBand based

connectivity.

Document Conventions

NOTE:

Text presented in this format indicates clarifying information, specific instructions,

commentary, sidelights, or interesting points of information.

IMPORTANT:

Text presented in this format indicates important information regarding a specific feature.

CAUTION:

Text presented in this format indicates that failure to follow directions could result in

damage to equipment or loss of information.

Text presented in this manner indicates terminal/CLI screen/command

format.

Version 2.0

Introduction

10

Mellanox Technologies

Chapter 1. Introduction

This chapter introduces the Mellanox InfiniBand-to-10 GbE Gateway (sRB-20210G).

1.1 Overview

The sRB-20210G module (see Figure 1-1 and Figure 1-2) is an InfiniBand-to-10 GbE gateway

module, designed as a plug-in unit for installation in the Mellanox ISR 2004/2012 Grid

Director.

The sRB-20210G module enables multiple virtual networks to span across InfiniBand and

Ethernet-based networks providing network-agnostic enterprise-class connectivity. With this

solution, InfiniBand can be used to accelerate applications without the need to replace

Ethernet-based services and devices while allowing the provision of multiple virtual Ethernet

and storage networks on from same director class switch.

IMPORTANT:

The sRB-20210G module is used as a bridge in the system. It does not run on several

subnets. However, it can be used to interface several subnets using partitions. For more

information, refer to Chapter 13, Partitioning.

1.2 Physical Description

1.2.1 Plug-in Module

The Mellanox sRB-20210G is a plug-in module, which consists of the following physical

components:

 1U-high line card for the Mellanox Grid Director 2012/2004

 Twenty-two (22) CX4 20Gbps 4x InfiniBand ports

 Two (2) 10 GbE XFP cages supporting the following XFPs:

 10GBASE-SR and -LR optic XFP MSA modules, LC connector

 10GBASE-CX4 Copper XFP MSA modules, CX4 connector.

Figure 1-1 shows a Mellanox sRB-20210G with empty cages for installation in the

ISR 2004/2012. The empty cages can accommodate optic or copper XFP modules.

Figure 1-1. sRB-20210G – Empty Cages

10 GbE Gateway

(sRB-20210G Module) User Manual

Version 2.0

11

Mellanox Technologies

Figure 1-2 shows a Mellanox sRB-20210G with optical XFP modules installed.

Figure 1-2. sRB-20210G – With Pluggable Optic XFP Modules

1.2.2 LED Indicators

Three LEDs are located on the face of the module, indicating link status of the combined

interfaces in the integrated Management port (mgmt) and overall system status.

The LEDs of both the InfiniBand and 10 GbE ports have the same indication types, as follows:

Table 1-1. LED Indicators

LED Type

Function

Status

Description

Power [green]

System Power

ON ,

green

Internal power is OK

OFF

Internal power failed

Info [green]

General purpose

LED for

management use

Blinking

OFF

Sets the Info LED on a desired module. When you enable this LED

through the GUI or CLI, you can go on site and immediately physically

locate the module.

System [green]

System state

(controlled by

SW)

In the GUI, note

that the state is

also shown by a

tooltip when the

mouse is over

this LED.

ON,

green

10 GbE Gateway is active

Blinking

slow

10 GbE Gateway is initializing

Blinking

fast

10 GbE Gateway fault – Software in

safe mode

Logical link

indication (per

port) [amber]

Logical link (24

Log for the whole

sRB-20210G)

ON

Subnet Manager has configured and

activated the link

Blinking,

amber

Data traffic over the logical link.

OFF

The Subnet Manager has not yet

configured the link.

Physical link

indication (per

port) (green)

Physical link

(24 Phy for the

whole

ON,

green

Activated upon physical link

Blinking

Unstable physical link.

Version 2.0

Introduction

12

Mellanox Technologies

LED Type

Function

Status

Description

sRB-20210G)

OFF

The port is not connected or there is

no physical link.

1.3 Features

1.3.1 Basic Features

The basic functional features of the sRB-20210G module include:

 High performance InfiniBand-to-10 GbE connectivity using Mellanox-developed silicon

 Extremely low latency using cut-thru architecture (under two [2] microseconds)

 IBTA 1.1 compliant

 Standard IPoIB support

 VLAN support according to 802.1q (up to 64 IB partitions; see New in Version 2.0)

 IP Multicast support – up to 1024 multicast groups

 Layer 2/3/4, traffic filtering, and classification

 Link aggregation, load sharing, and High Availability

 CLI and GUI management via the Mellanox Fabric Manager (VFM) and the Mellanox

Device Manager (VDM).

1.3.2 New in Version 2.0

Version 2.0 includes the following improvements and enhancements:

 Support for different MTU rates on the Ethernet and IB side. This allows the gateway to

work in a Jumbo frame environment on the Ethernet side. It also reduces the burden of

decreasing the default MTU of all IB hosts in the fabric from 2 Kbytes to 1.5 Kbytes.

 A loop prevention mechanism has been added for cases when multiple gateways are

deployed in a specific application, so that the user would be able to configure them

without creating loops on the Ethernet network, while preserving the high-availability

requirements.

 Support for up to 64 partitions/VLANs

 Support for up to 3,000 multicast groups

 Enhanced monitoring capabilities.

1.4 Functional Description

1.4.1 High Performance InfiniBand-to-GbE Connectivity

The sRB-20210G core capabilities rely on the Mellanox in-house developed silicon used for

forwarding traffic between InfiniBand and Ethernet infrastructures. This custom silicon

enables the Mellanox sRB-20210G to achieve wire-speed protocol processing with minimal

latency, while transparently providing upper-layer traffic classification capabilities. In

addition, multiple 10 GbE ports can be aggregated to scale connectivity and meet various

application requirements.

10 GbE Gateway

(sRB-20210G Module) User Manual

Version 2.0

13

Mellanox Technologies

1.4.2 Comprehensive Management

The rich functionality provided by the Mellanox sRB-20210G is fully integrated into

GridVision™ Grid Interconnect Management Software that is embedded in Mellanox

InfiniBand switches and gateways.

GridVision is an easy-to-use management suite that provides access through a Web-based

GUI, a command line interface (CLI) and SNMP.

Out-of-band management includes InfiniBand, Ethernet and EIA/TIA-232 connections. The

Mellanox sRB-20210G Gateway management is also integrated with leading third party grid

and on-demand management platforms.

1.4.3 sRB-20210G Architecture

 All data traffic is handled by the sRB-20210G hardware (Silicon) to achieve the highest

possible throughput, with the lowest possible latency.

 Management traffic is handled in software by the CPU.

Software

Ethernet InfiniBand

Silicon

Fast Path

Slow Path

Figure 1-3. sRB-20210G Conceptual Block Diagram

1.4.4 Supported Protocols

The sRB-20210G supports the following protocols:

 10 Gigabit Ethernet over fiber - IEEE 802.3ae - 10GBASE-SR and 10GBASE-LR.

 10 Gigabit Ethernet over twin-axial cable – IEEE 820.3ak – 10GBASE-CX4

 InfiniBand 1.1 compliant

 IETF RFC 4391 Transmission of IP over InfiniBand (IPoIB)

 IETF RFC 4392 IP over InfiniBand (IPoIB) Architecture

 SNMP v2c: IETF-RFC190x

 Java 1.6 for the GUI.

IPoIB (IP-over-InfiniBand) is a protocol for encapsulating data requests and responses over

InfiniBand. The sRB-20210G uses IP over InfiniBand (IPoIB) protocols in order to provide IP

link between Ethernet network and InfiniBand based network.

The tunneling functionality is achieved by Layer II address translation of an InfiniBand

address to an Ethernet address (MAC) and vice versa.

Version 2.0

Introduction

14

Mellanox Technologies

1.4.5 sRB-20210G Application Diagram

Figure 1-4 shows an example of an IPoIB network.

Figure 1-4. IPoIB Network Response/Request Routing

The sRB-20210G provides transparent bridging of Ethernet traffic to servers on the InfiniBand

fabric and vice versa, as if they were directly attached to the clients over a local network. This

network behavior resembles a bridge methodology.

1.4.6 Traffic Flow Management Overview

The Traffic Flow Management (TFM) feature allows a network administrator to determine

rules for handling IP packets arriving at the sRB-20210G, based on their source or destination

IP address and ports. The administrator can define services and rules to control how the

sRB-20210G processes these packets. This processing includes:

 Forwarding – the packet is forwarded to a destination host either on the Ethernet side

or the InfiniBand side

 Drop – the packet is discarded.

The CLI

add

and

remove

CLI commands are used for managing the packet handling rules in

the sRB-20210G. The user can define rules or use the default rules. If no new rule was added

or if there was no change to the default rules, the packets are handled by a default method. For

further details, refer to the Traffic Flow Manager (TFM), Chapter 8.

1.5 Technical Specifications

1.5.1 General

 Line board for Mellanox Grid Director ISR 2012 and ISR 2004 chassis

 Power and info LED indicators

 Full non-blocking connectivity.

1.5.2 10 Gigabit Ethernet (GbE)

 IEEE 802.3ab, and 802ad (link aggregation) compliant

 Two 10GbE interconnect ports, XFP interfaces

 IEEE P802.3ak and IEEE P802.3ae 10GBASE-SR, 10GBASE-LR, 10GBASE-CX4

Ethernet NAS - NFS,

CIFS

Router – LAN/WAN

Clustered File System

Ethernet SAN - iSCSI

10 GbE Gateway

(sRB-20210G Module) User Manual

Version 2.0

15

Mellanox Technologies

 Port LED indicators: Physical connectivity and traffic indication

 Support for up to 64 partitions/VLAN groups

 Support for Jumbo frames.

1.5.3 InfiniBand

 22, auto-sensing 20/10 Gbps (4X) InfiniBand ports –

› Connectors: 8-pair MicroGigaCN / LANELINK F CONN

› Support hot-pluggable media converters for optical InfiniBand connectivity.

 Port LED indicators: Physical connectivity and logical connectivity.

1.5.4 IETF Protocols

 TCP/IP: IETF RFC-793, IETF RFC-791, IETF RFC-768, RFC-926, RFC-1812,

RFC-1027

 IPoIB: IETF RFC 4391, 4392

 VLAN support according IEEE 802.1q (up to 64 VLANS)

 IP multicast (IETF RFC 3171) and IGMPv2 (up to 4000 multicast groups)

 SNMP v2c: IETF — RFC190x.

1.5.5 Management Interfaces

 Full management access is available via the Grid Director chassis management or the

Mellanox GridVision management software.

1.5.6 Physical

 Mellanox Grid Director ISR 2012 and ISR 2004 Line Board compatible

 Supports any combination of sLB-2024 and sRB-20210G Line Boards

 Power and cooling through ISR 2012 or ISR 2004 chassis.

1.5.7 Environmental

 Operating –

› Temperature: 32° to 104°F (0° to 45°C)

› Humidity: 15 to 80%, non-condensing

› Altitude: 0 to 9843 ft (3000m)

 Storage –

› Temperature: -13° to 158°F (-25° to 70°C)

› Humidity: 5 to 90% non-condensing

› Altitude: 0 to 15,000 ft (4570m)

1.5.8 Certifications

 Full safety and EMC as a part of the Mellanox Grid Director ISR 2012 and ISR 2004.

Version 2.0

Initial Configuration

16

Mellanox Technologies

Chapter 2. Initial Configuration

This chapter provides instructions on how to perform the basic initial configuration settings for

the sRB-20210G. These steps are typically performed when installing a new device.

2.1 Prerequisite Installation Tasks

IMPORTANT:

Before configuring the Mellanox sRB-20210G for the first time, make sure you have

completed the hardware installation according to the relevant Mellanox Installation Manual.

The sRB-20210G is managed through the Switch chassis management, via either inband

(through the InfiniBand interface) or out-of-band (through the Ethernet interface)

connection. For more information on Inband/out-of-band management, refer to the switch

(ISR 2xxx) user manual.

2.2 Configuring the sRB-20210G Gateway in the ISR 2XXX

NOTES:

Do not physically connect any of the sRB-20210G GbE ports to the same physical network

as the switch’s Ethernet management port. In order to prevent broadcast packet loops, the

switch’s Ethernet management port must be physically-separated from the GbE network

and InfiniBand fabric.

The management interface of the sRB-20120G (smb-mng) must be configured with a

unique IP address on the same network with the Ethernet/InfiniBand management

interface of the switch. Refer to the GridVision Integrated Documentation Suite (First-Time

Configuration) for further details.

2.2.1 Managing the Unit over Fast Ethernet (Out-of-Band) or InfiniBand

(Inband)

When the system is configured to manage a chassis containing an sRB-20210G, and the Fast

Ethernet is the selected management interface in the ISR 2XXX Grid Director, the sMB

automatically enters into Proxy-ARP mode. This allows the connection between the manager

that resides on the ETH network and the sRB-20210G residing on the InfiniBand network.

IMPORTANT:

 Make sure to configure the sRB-20210G management (smb-mng) interface on the

same subnet that was defined for the switch management (Fast Ethernet) interface.

 The management interface of the sRB-20120G (smb-mng) must be configured with a

unique IP address on the same network with the Ethernet/InfiniBand management

interface of the switch. Refer to the GridVision Integrated Documentation Suite

(First-Time Configuration) for further details.

10 GbE Gateway

(sRB-20210G Module) User Manual

Version 2.0

17

Mellanox Technologies

Configure the interfaces and sRB-20210Gs as follows:

Step 1

Configure the Switch sMB Fast Ethernet interface (IP address, subnet mask), as described in

the Switch User Manual (First-Time Configuration Section).

Step 2

Configure the Switch InfiniBand interface to the same values of the Fast Ethernet interface,

the same IP address and same subnet mask, as described in the Switch User Manual

(First-Time Configuration Section).

Step 3

Make sure that the IP address of each I/O Gateway module is set on the same subnet

defined for the Fast Ethernet interface.

Step 4

Configure the gateways as detailed in the following procedure.

NOTE:

Run the following command to verify that the Switch default gateway is already set:

ISR2004-3200(config-if)# ip-address show

Example:

ISR2004-3200(config-if)# ip-address show

port selection Ethernet

ip 172.28.1.223

mask 255.255.0.0

broadcast 172.28.255.255

DHCP client disabled

default-gw Ip is: 172.28.0.1

Link-speed auto-negotiation.

To set the sRB-20210G IP address and Default Gateway (Switch side):

Use the following table to enter the appropriate commands:

Commands

Procedure

From the Switch prompt:

Step 1

guest

a. Enter the guest mode. The default password

for the guest user is

voltaire.

b. Enter the Privileged mode as follows:

Type enable and enter the admin password (the

default password for the admin user is

123456

.)

OR

admin

Enter the privileged mode directly.

The default password for the admin user is

123456

.

Note: When connecting to the sRB-20210G via the

switch using the

io connect

command, you are

already in admin mode and do not need a password.

Step 2

config

Enter the Configuration mode from the Exec mode.

Step 3

chassis

Enter Chassis Configuration mode.

Version 2.0

Initial Configuration

18

Mellanox Technologies

Commands

Procedure

Step 4

io show

Use the

io show

CLI command to view the

sRB-20210G gateways and mark the position of the

desired I/O to connect to it.

Step 5

io ip-address set slot-number

position-number ip-address

netmask

Sets the IP address of the I/O Gateway the ISR 2XXX

Switch.

Example:

switch(config-chassis)io ip-address set

1 2 192.168.70.60 255.255.255.0

Step 6

io connect

Connect to the selected interface (Enters as admin

user to the sRB-20210G Gateway).

Example:

switch(config-chassis)io connect 4 1

You will now get the sRB-20210G admin prompt.

If the switch and the host are in the same subnet, the sRB-20210G is now ready for operation.

2.2.2 Handling Different Subnets

This section explains how to set the sRB-2021G when the switch and the host are in different

subnets.

IMPORTANT:

In case the switch and the host are in different subnets, you must set the default gateways

for each I/O Module, as described in the following section.

To set the sRB-20210G IP address and Default Gateway on the sRB-20210G side:

From the sRB-20210G prompt:

Step 1

config

Enter the sRB-20210G Configuration mode from the

sRB-20210G Exec mode.

Step 2

interface

Enter the sRB-20210G Interface mode to view which

interfaces are available.

Step 3

ip-address set [alias]

[ip-address] [netmask]

[optional broadcast]

Set the IP address for the default interface (if0)

sRB-20210G-6044(config-if)# ip-address set

ip-address set [alias] [ip-address] [netmask]

[optional broadcast]

10 GbE Gateway

(sRB-20210G Module) User Manual

Version 2.0

19

Mellanox Technologies

Step 4

interface show all

View all the interfaces in the sRB-20210G

Example:

10G (config-if)# interface show all

entry alias ip mask broadcast vlan

pkey(hex) trunk

|-----|-----|----------|------------|-----

1 smb-mng 172.28.1.18 255.255.0.0 172.28.255.255 no

0x7fff No port

2 if0 192.168.2.6 255.255.0.0 192.168.255.255 no

0x7fff P1-2

Step 5

exit

Exits the Interface menu.

Step 6

route

Enter the sRB-20210G Configuration Route menu.

Step 7

default-gw set [row

[1..10]][Gateway][Iface]

If you are setting the management through fast Eth

(inband) then you must set the default gateway

address to the smb-mng management interface.

This command sets the default gateway as shown in

the example below. This will add the default gateway

to the Route Table.

Set the default gateway of the management as

shown in the following examples. This will add the

default gateway to the Static Route Table.

IMPORTANT:

If you are setting the management through fast

Ethernet interface (out-of-band), then you must set

the default gateway address to the smb-mng

management interface.

If you are setting the management through InfiniBand

(inband), then you must set the default gateway

address to the if0 management interface.

Example (Out-of-band – Ethernet Interface):

10G-Card(config-route)# default-gw set 1

172.30.0.1 smb-mng

Expected Result:

sRB-20210G-6044(config-route)# static-route show

entry destination netmask gateway interface

|-----|----------|---------|---------|--------

1 0.0.0.0 0.0.0.0 172.30.0.1 smb-mng

Example (Inband – InfiniBand Interface):

10G-Card(config-route)# default-gw set 1 172.25.0.1

if0

Expected Result:

sRB-20210G-6044(config-route)# static-route show

entry destination netmask gateway interface

|-----|----------|---------|---------|--------

1 0.0.0.0 0.0.0.0 172.25.0.1 if0

Step 8

end

Exits to the Switch chassis mode.

The sRB-20210G is now ready; traffic is set to be transferred.

Version 2.0

Initial Configuration

20

Mellanox Technologies

2.3 Configuring an sRB-20210G Module (10 GbE to InfiniBand

Bridge)

IMPORTANT:

This section is only relevant if you intend to use multiple VLANs.

You can separate traffic between different ports (no LAG) by adding bridges/VLANs. The

sRB-20210G supports up to 16 bridges located on different partitions/VLANs.

To configure a GbE/InfiniBand Bridge:

Commands

Procedure

From the Switch prompt:

Step 1

guest

a. Enter the guest mode. The default

password for the guest user is

voltaire.

b. Enter the Privileged mode as follows:

Type enable and enter the admin password (The

default password for the admin user is

123456

.)

OR

admin

Enter directly the privileged mode.

The default password for the admin user is

123456

.

Note: When connecting to the sRB-20210G via

the switch using the

io connect

command, you

are already in admin mode and do not need a

password.

Step 2

config

Enter the config mode (from the privileged

mode).

Step 3

chassis

Enter the chassis mode.

Step 4

io show

View (and note) the I/O (sRB-20120G) position in

the switch.

Step 5

io connect {slot-number

position-number}

Connect to the desired I/O (sRB-20120G). The

sRB-20210G prompt appears.

From the sRB-20210G prompt:

Step 6

[password]

At the sRB-20210G prompt, enter the password.

Default password is

123456

.

Step 7

config

Enter the config mode (from privileged mode).

Step 8

interface

Enter the interface configuration mode.

Note: Before going to the following step, you must first hand out this PKey to the

SRB-20210G from the Switch. Refer to the GridVision User Documentation for further

information.

Page is loading ...

Mellanox Technologies sRB-20210G User manual

Related papers

Other documents