Voltaire ISR 9024 User manual

Voltaire Switch User Manual

ISR 9024, ISR 9096, and ISR 9288 Switches

Business Headquarters

Voltaire Inc.

6 Fortune Drive, Suite 301

Billerica, MA USA 01821

Tel: 978-439-5400

Fax: 978-439-5401

Israel Office

Voltaire Ltd.

9 Hamenofim St.

Bldg. A Herzeliya

46725, Israel

Tel: +972 (9) 971-7666

Fax: +972 (9) 971-7660

Release AAA-BAA

May 2006

P/N 399Z00038

THIS DOCUMENT IS PROVIDED "AS IS" WITH NO WARRANTIES WHATSOEVER, INCLUDING ANY

WARRANTY OF MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR ANY PARTICULAR

PURPOSE, OR ANY WARRANTY OTHERWISE ARISING OUT OF ANY PROPOSAL, SPECIFICATION

OR EXAMPLES.

Voltaire disclaims all liability, including liability for infringement of any proprietary rights, relating to use of

information in this specification.

No license, expressed or implied, by written or otherwise, to any intellectual property rights is granted herein.

This document as well as the software described in it is furnished under license and may only be used or copied

in accordance with the terms of the license. The information in this manual is furnished for informational use

only, is subject to change without notice, and should not be construed as a commitment by Voltaire. Voltaire

Ltd. assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any

software that may be provided in association with this document. Except as permitted by such license, no part of

this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any means

without the express written consent of Voltaire Ltd.

Voltaire is a trademark or registered trademark of Voltaire Ltd. or its subsidiaries in the United States and other

countries.

Other names and brands may be claimed as the property of others and have been designated with an asterisk (*)

throughout this document.

Voltaire Part Number:

399Z00038

About this Manual

This preface describes the audience, organization and command syntax conventions

of the Voltaire Switch User Manual. It also provides information on how to obtain

related documentation and technical assistance. Information about hardware

installation of Voltaire switches can be found in ISR 9024 Installation Manual and

in the ISR 9XXX Installation Manual, which includes the ISR 9096 and the

ISR 9288.

NOTE

Consult Voltaire release notes for last minute updates and restrictions.

The Voltaire Technical Support Center (TSC) is at your service. You may access

Warranty Service through our Web Request Form by using the following link:

http://www.voltaire.com/support.html

Contact Us:

Please send your documentation-related comments and feedback or report mistakes

to [email protected].

We are committed to constant and never-ending improvement. Your input will

greatly help us in our endeavor.

Audience

This manual is primarily intended for system administrators who are familiar with

the fundamentals of router-based internetworking and network storage devices, but

who might not be familiar with the specifics of Voltaire products or the routing

protocols supported by Voltaire products

It is assumed that readers are familiar with InfiniBand technology and terminology.

Related Documentation

 ISR 9XXX Installation Manual, Voltaire

 ISR 9024 Installation Manual, Voltaire

About this Manual

ii

Document Conventions

NOTE

Text set off in this manner presents clarifying information, specific instructions,

commentary, sidelights, or interesting points of information.

IMPORTANT

Text set off in this manner indicates important information regarding a specific

feature.

CAUTION

Text set off in this manner indicates that failure to follow directions could result in

damage to equipment or loss of information.

About this Manual

iii

Document Organization

This guide contains the following chapters:



Chapter 1 – Introduction: Describes the InfiniBand technology and provides an

introduction to the ISR 9024, ISR 9096 and ISR 9288 switches.



Chapter 2 – CLI Overview: Provides a brief introduction on the Voltaire

Command Line Interface (CLI).



Chapter 3 – Initial System Setup: Provides instructions on how to perform an

initial setup of your system.



Chapter 4 – Basic CLI Maintenance Commands: This section explains how to

configure switch parameters and how to perform a software and firmware

upgrade.



Chapter 5 – Voltaire Device Manager (VDM): Describes the Voltaire Device

Manager, a Graphic User Interface (GUI) application for device management

and switch configuration and provides operation instructions.



Chapter 6 – Voltaire Fabric Manager (VFM): Describes the Voltaire Fabric

Manager (VFM), a Graphic User Interface (GUI) application that provides a

wide range of provisioning, configuration and maintenance functions for the

Switch, and provides operation instructions.



Chapter 7 – Names Settings: This section provides instructions on how to set

names to Switches and unmanaged ISR 9024.



Chapter 8 – Rack Grouping: Introduces the Rack grouping feature and

provides instructions on how to create, set, edit, view, and delete rack-groups.



Chapter 9 – Configuring Quality of Service (QoS): Provides a QoS overview,

instruction on how to perform SL to VL mapping, QoS arbitration, and an

example of SM parameter configuration.



Chapter 10 – Configuring Routing Algorithms: Lists the supported algorithms,

details the Balanced Routing, Rearrangeable Routing (Multi-Path with sFB

Consistency), and Up/Down Routing schemes, as well as providing

instructions on how to configure Routing via the VFM and the CLI.



Chapter 11 – Configuring Partitioning: Describes the Partitioning feature used

to enforce isolation among systems using an InfiniBand Fabric and provides

instructions on how to assign partition keys (Pkeys). It also describes how

Partitioning is used to associate hosts or devices that share common attributes.



Chapter 12 – Replacing Modules: Provides instructions on how to replace

sFBs, sLBs, sMBs and unmanaged ISR 9024.

About this Manual

iv

 Chapter 13 – Configuring Cluster Monitor: Allow you to view your entire

cluster in a browser using the Ganglia tool.



Chapter 14 – Fabric Diagnostic: Provides information on logging and

monitoring functions for advanced fabric debugging and failure detection

capabilities.



Appendix A – CLI Commands Reference: Explains the CLI basic concept and

details the CLI command menus. The CLI provides configuration and

management functions for Voltaire switches, as well as a wide range of

provisioning, configuration and maintenance functions for the InfiniBand

fabric.



Appendix B – Port Counters : For Advanced Diagnostics - Describes and

details the port counter fields of the CSV file, details the CSV counters

summary reports and lists the ports counters currently available.



Appendix C – Supported Traps and Events: Defines the structure of traps and

lists the traps, events and alarms supported in this version.



Appendix D – Glossary



Appendix E – Index

About this Manual

v

What's New in this Manual?

This switch software version includes the following new features:

 Subnet Management Enhancements: Enhanced “Bad-Ports” feature, Up-Down

Routing Scheme, Performance Improvement: Improve Fabric Configuration

Time, Subnet Manager Priority, Partitioning, Non-Intrusive Routing

Reconfiguration, and Subnet Prefix.

 Rack Grouping (Creating, renaming, deleting rack groups, and viewing rack

information via the GUI and the CLI; groups.csv file)

 The Voltaire switch software supports a new hardware module: sLB-8-12

 Diagnostics Enhancements: Enhanced Event Notification - extended

notification mode, New alarm category in current alarm mechanism: Fabric

Config Error, Enhanced Performance Manager (PM) [PortCounters files; Port-

verify Utility; Port-Manage Utility]

 Unique Switch Name

 CLI Enhancements (exportLogs, new madstat and vendstat options)

 New default firmware settings

 Router Version Automatic Upgrade

Contents

Audience........................................................................................................................................i

Related Documentation................................................................................................................i

Document Conventions ..............................................................................................................ii

Document Organization.............................................................................................................iii

What's New in this Manual? .......................................................................................................v

Chapter 1. Introduction..........................................................................................................1-1

1.1 About InfiniBand...........................................................................................................1-2

1.2 ISR 9024 System Overview ..........................................................................................1-3

1.3 ISR 9288/9096 System Overview.................................................................................1-4

1.4 Voltaire Switch Management......................................................................................1-10

Chapter 2. CLI Overview........................................................................................................2-1

2.1 Interfaces for CLI Connection.......................................................................................2-2

2.2 CLI Command Modes...................................................................................................2-2

2.3 CLI Usage Tips .............................................................................................................2-3

2.4 Keyboard Usage in CLI Operation................................................................................2-3

2.5 CLI Command Structure ...............................................................................................2-5

2.6 CLI Case Sensitivity......................................................................................................2-5

2.7 CLI Command Prompts.................................................................................................2-6

Chapter 3. Initial System Setup.............................................................................................3-1

3.1 Prerequisite Tasks..........................................................................................................3-2

3.2 First-Time Configuration...............................................................................................3-2

3.3 Collecting Required Information...................................................................................3-2

3.4 Connecting a Console....................................................................................................3-3

3.5 Starting a CLI Session...................................................................................................3-3

3.6 Configuring the Management Interface IP Address......................................................3-4

3.7 Configuring the Management IP Address in ISR 9288/9096/9024 Switches ...............3-5

3.8 Configuring the ISR 9288/9096 with I/O Modules.......................................................3-7

3.9 Configuring Time and Date...........................................................................................3-9

Contents

ii

3.10 Configuring Passwords..................................................................................................3-9

3.11 Verifying Configuration..............................................................................................3-10

Chapter 4. Basic CLI Maintenance Commands...................................................................4-1

4.1 Prerequisites ..................................................................................................................4-2

4.2 Updating Software.........................................................................................................4-2

4.3 Prerequisites for Updating Software .............................................................................4-3

4.4 Backing up the Switch Configuration ...........................................................................4-4

4.5 Software and Firmware Upgrade Process .....................................................................4-5

4.6 CLI Commands Reference ............................................................................................4-6

Chapter 5. Voltaire Device Manager (VDM)..........................................................................5-1

5.1 Device Manager Description.........................................................................................5-2

5.2 Getting Started with the Device Manager .....................................................................5-3

5.3 The Device Manager Main Window.............................................................................5-6

5.4 Voltaire Device Manager Operating Conventions ......................................................5-16

5.5 Viewing Switch Information.......................................................................................5-16

5.6 Viewing Module Information......................................................................................5-27

5.7 Using the Voltaire Device Manager for Switch Management.....................................5-40

5.8 Reset Operations..........................................................................................................5-45

Chapter 6. Voltaire Fabric Manager (VFM)...........................................................................6-1

6.1 Fabric Manager Description..........................................................................................6-2

6.2 Fabric Manager Installation...........................................................................................6-3

6.3 Getting Started with the Fabric Manager ......................................................................6-5

6.4 The Fabric Manager Main Window ..............................................................................6-8

6.5 Viewing Properties Data..............................................................................................6-28

6.6 Information Windows..................................................................................................6-31

6.7 Configuring the Fabric ................................................................................................6-41

6.8 Configuring Nodes ......................................................................................................6-61

Chapter 7. Names Settings....................................................................................................7-1

7.1 CLI Switch Name Settings............................................................................................7-2

7.2 GUI Node Name Settings..............................................................................................7-3

7.3 Naming an Unmanaged ISR 9024.................................................................................7-4

Chapter 8. Rack Grouping.....................................................................................................8-1

8.1 Rack Grouping Overview..............................................................................................8-2

8.2 Rack Grouping Settings via the VFM ...........................................................................8-3

8.3 Elements of the Group.csv File...................................................................................8-10

8.4 Backing up and Editing Groups Settings via the CLI .................................................8-12

Contents

iii

8.5 Rack Grouping Installation Suggestion.......................................................................8-14

Chapter 9. Configuring Quality of Service (QoS)................................................................9-1

9.1 QoS Overview...............................................................................................................9-2

9.2 SL to VL Mapping ........................................................................................................9-2

9.3 QoS Arbitration.............................................................................................................9-4

9.4 Example of SM Parameter Configuration.....................................................................9-7

Chapter 10. Configuring Routing Algorithms......................................................................10-1

10.1 Supported Algorithms .................................................................................................10-2

10.2 Balanced Routing Scheme...........................................................................................10-2

10.3 Rearrangeable Routing (Multi-Path with sFB Consistency).......................................10-2

10.4 Up/Down Routing Scheme..........................................................................................10-3

10.5 Configuring Routing via the VFM ..............................................................................10-4

10.6 Configuring Routing via the CLI ................................................................................10-7

Chapter 11. Configuring Partitioning....................................................................................11-1

11.1 Partitioning Overview .................................................................................................11-2

11.2 Mechanisms Used to Create and Enforce Partitions ...................................................11-3

11.3 Setting Partitioning......................................................................................................11-3

Chapter 12. Replacing Modules............................................................................................12-1

12.1 Replacing sFBs (Spines)..............................................................................................12-2

12.2 Replacing sLBs............................................................................................................12-6

12.3 Replacing sMBs ..........................................................................................................12-7

12.4 Replacing an Unmanaged ISR 9024............................................................................12-8

Chapter 13. Configuring Cluster Monitor.............................................................................13-1

13.1 What is Ganglia? .........................................................................................................13-2

13.2 Cluster Configuration..................................................................................................13-2

Chapter 14. Fabric Diagnostic...............................................................................................14-1

14.1 Fabric Diagnostic Overview........................................................................................14-2

14.2 Fabric Inspect Settings ................................................................................................14-7

14.3 Diagnostic Tools........................................................................................................14-10

14.4 Additional Tools and Examples ................................................................................14-20

Contents

iv

Appendix A CLI Commands Reference................................................................................. A-1

Appendix B Port Counters Fields........................................................................................... B-1

Appendix C Supported Traps and Events............................................................................. C-1

Appendix D Glossary............................................................................................................... D-1

Appendix E Index......................................................................................................................E-1

Figures

Figure 1-1. Voltaire ISR 9024 Grid Switch™ Functional Diagram................................... 1-4

Figure 1-2. Voltaire ISR 9288 Grid Backbone™ Functional Diagram.............................. 1-5

Figure 1-3. Voltaire ISR 9096 Grid Backbone™ Functional Diagram.............................. 1-5

Figure 1-4. ISR 9288 InfiniScale Switching Topology - Fat Tree Topology..................... 1-8

Figure 1-5. ISR 9096 InfiniScale Switching Topology - Fat Tree Topology..................... 1-9

Figure 1-6. Out-of-band Management.............................................................................. 1-11

Figure 1-7. In-band Management ..................................................................................... 1-11

Figure 5-1. Accessing the Voltaire Multi Protocol Switch Window.................................. 5-3

Figure 5-2. Login Dialog Box ............................................................................................ 5-4

Figure 5-3. Exit Warning Message..................................................................................... 5-6

Figure 5-4. ISR 9288 Voltaire Device Manager Main Window......................................... 5-7

Figure 5-5. ISR 9096 Voltaire Device Manager Main Window......................................... 5-8

Figure 5-6. Device Tree.................................................................................................... 5-10

Figure 5-7. ISR 9096 Switch VDM Graphical Representation (Front View) .................. 5-12

Figure 5-8. ISR 9096 Switch VDM Graphical Representation (Rear View) ................... 5-12

Figure 5-9. ISR 9288 Switch Graphical Representation (Front View)............................. 5-13

Figure 5-10. ISR 9288 Switch Graphical Representation (Rear View)............................ 5-14

Figure 5-11. ISR 9024 Switch Graphical Representation ................................................ 5-14

Figure 5-12. Device Information...................................................................................... 5-17

Figure 5-13. Port Information........................................................................................... 5-19

Figure 5-14. CPU Information.......................................................................................... 5-21

Figure 5-15. Temperature Readings ................................................................................. 5-23

Figure 5-16. Local SM Information ................................................................................. 5-24

Figures

v

Figure 5-17. Detailed Module Information ...................................................................... 5-26

Figure 5-18. sLB Information........................................................................................... 5-28

Figure 5-19. sLB-24 with Optical Adapters ..................................................................... 5-30

Figure 5-20. ISR 9288 Fabric Board Information ............................................................ 5-31

Figure 5-21. ISR 9096 Fabric Board Information ............................................................ 5-31

Figure 5-22. sMB Information.......................................................................................... 5-32

Figure 5-23. sMB Detailed Module Information Window............................................... 5-33

Figure 5-24. sCTRL Information...................................................................................... 5-34

Figure 5-25. sPSU Information ........................................................................................ 5-35

Figure 5-26. ISR 9288 Fan Information ........................................................................... 5-36

Figure 5-27. ISR 9096 Fan Information ........................................................................... 5-36

Figure 5-28. Faulty Fan Information ................................................................................ 5-37

Figure 5-29. Router Module Information ......................................................................... 5-39

Figure 5-30. IP Settings.................................................................................................... 5-40

Figure 5-31. Remote Manager Settings - SNMP Tab....................................................... 5-41

Figure 5-32. Remote Manager Settings - General tab ...................................................... 5-42

Figure 5-33. Configuring User Authorization .................................................................. 5-43

Figure 5-34. Configuring FTP.......................................................................................... 5-44

Figure 5-35. Warning popup ............................................................................................. 5-45

Figure 6-1. Voltaire Fabric Manager Initiate Window....................................................... 6-5

Figure 6-2. Login Window ................................................................................................. 6-6

Figure 6-3. Exit Warning Message..................................................................................... 6-7

Figure 6-4. Fabric Manager Main Window........................................................................ 6-8

Figure 6-5. Example of a System with 1104 Nodes ........................................................... 6-9

Figure 6-6. Topology Map ............................................................................................... 6-10

Figure 6-7. Right-Click a Node – Topology Map ............................................................ 6-16

Figure 6-8. Right-Click a Node – Information Area (Right Panel).................................. 6-16

Figure 6-9. Right-Click a Port – Information Area (Right Panel).................................... 6-17

Figure 6-10. Right-Click a Port – Table View ................................................................. 6-17

Figures

vi

Figure 6-11. Right-Click an Event ................................................................................... 6-17

Figure 6-12. Right-Click the Fabric Subnet ..................................................................... 6-17

Figure 6-13. Right-Click Racks........................................................................................ 6-18

Figure 6-14. Subnet Tree.................................................................................................. 6-18

Figure 6-15. Topology Table View .................................................................................. 6-22

Figure 6-16. Trap Table.................................................................................................... 6-23

Figure 6-17. Events Log Right-Click Menu..................................................................... 6-25

Figure 6-18. Events Data.................................................................................................. 6-25

Figure 6-19. Go to Node................................................................................................... 6-27

Figure 6-20. Node Properties ........................................................................................... 6-28

Figure 6-21. Port Properties.............................................................................................. 6-29

Figure 6-22. Link Properties............................................................................................. 6-31

Figure 6-23. List of Racks ................................................................................................ 6-31

Figure 6-24. Racks Components ...................................................................................... 6-32

Figure 6-25. Hosts List ..................................................................................................... 6-33

Figure 6-26. Hosts Ports ................................................................................................... 6-34

Figure 6-27. List of Switches ........................................................................................... 6-35

Figure 6-28. Switch/SLB Information.............................................................................. 6-36

Figure 6-29. Port Information........................................................................................... 6-37

Figure 6-30. Link Information.......................................................................................... 6-39

Figure 6-31. Grouping Information.................................................................................. 6-40

Figure 6-32. SM Details ................................................................................................... 6-41

Figure 6-33. Event Filters Settings................................................................................... 6-44

Figure 6-34. PM Settings.................................................................................................. 6-45

Figure 6-35. Event Log - Example ................................................................................... 6-46

Figure 6-36. Error Log – Example ................................................................................... 6-47

Figure 6-37. Bad Ports Log .............................................................................................. 6-48

Figure 6-38. PortCounters.csv File - Example ................................................................. 6-49

Figure 6-39. Cluster Monitor Dialog Box ........................................................................ 6-49

Figures

vii

Figure 6-40. Ganglia Cluster Monitor .............................................................................. 6-50

Figure 6-41. QoS SL to VL Mapping Table..................................................................... 6-51

Figure 6-42. QoS VL High Arbitration Table .................................................................. 6-52

Figure 6-43. Routing ........................................................................................................ 6-53

Figure 6-44. Routing Algorithm....................................................................................... 6-54

Figure 6-45. Add to Spine List ......................................................................................... 6-55

Figure 6-46. Current Alarms ............................................................................................ 6-56

Figure 6-47. Alarm Data................................................................................................... 6-57

Figure 6-48. Config FTP Dialog Box............................................................................... 6-59

Figure 6-49. Viewing Nodes ............................................................................................ 6-61

Figure 6-50. Nodes Grouping Info Window .................................................................... 6-62

Figure 6-51. Group Properties.......................................................................................... 6-63

Figure 6-52. Creating a New Rack ................................................................................... 6-63

Figure 6-53. Current Alarms of a Single Node ................................................................ 6-64

Figure 7-1. Node Properties ............................................................................................... 7-3

Figure 8-1. Rack-Groups (VFM)........................................................................................ 8-3

Figure 8-2. Rack Grouping – Topology Map..................................................................... 8-4

Figure 8-3. Rack Grouping – Table View .......................................................................... 8-4

Figure 8-4. Rack Grouping – Switches/Host Table............................................................ 8-5

Figure 8-5. Node Properties Dialog Box ............................................................................ 8-5

Figure 8-6. New Rack Dialog Box ..................................................................................... 8-6

Figure 8-7. New Rack......................................................................................................... 8-6

Figure 8-8. Group Properties Dialog Box .......................................................................... 8-7

Figure 8-9. Rack Group Contents....................................................................................... 8-8

Figure 8-10. Rack-Groups – Table View ........................................................................... 8-9

Figure 8-11. Absent Elements .......................................................................................... 8-10

Figure 9-1. QoS SL to VL Mapping Table......................................................................... 9-3

Figure 9-2. QoS Arbitration Table ..................................................................................... 9-5

Figure 10-1. Routing ........................................................................................................ 10-4

Figures

viii

Figure 10-2. Routing Algorithm....................................................................................... 10-5

Figure 10-3. Add to Spine List ......................................................................................... 10-6

Figure 11-1. Partitioning Block Diagram .......................................................................... 11-2

Figure 12-1. Resetting a Specific Board via the VDM..................................................... 12-4

Figure 13-1. Cluster Monitor Dialog Box ........................................................................ 13-2

Figure 13-2. Ganglia Cluster Monitor .............................................................................. 13-3

Figure 14-1. Diagnostic Block Diagram........................................................................... 14-3

Figure 14-2. PM Settings.................................................................................................. 14-9

Figure 14-3. Port Counters Information ......................................................................... 14-11

Figure 14-4. Port Counters Graph .................................................................................. 14-12

Figure 14-5. Trap Table – 1 X Link ............................................................................... 14-20

Figure 14-6. VFM Event Data Window ......................................................................... 14-20

Figure 14-7. Nodes Info Window................................................................................... 14-22

Figure 14-8. Bad Ports Log ............................................................................................ 14-25

1-1

1

Chapter 1. Introduction

In This Chapter

This chapter provides a brief introduction to InfiniBand technology and describes

Voltaire ISR 9024, ISR 9096 and ISR 9288 switches.

The following information is included in these pages:



About InfiniBand, on page 1-2



ISR 9024 System Overview, on page 1-3



ISR 9288/9096 System Overview, on page 1-4



Voltaire Switch Management, on page 1-10

Introduction About InfiniBand

1-2

1.1 About InfiniBand

InfiniBand technology is a high-performance channel-based interconnect

architecture that provides increased scalability and reliability for servers and other

Internet infrastructure equipment. InfiniBand architecture defines the entire stack,

from the physical to the application layers APIs and fabric management.

InfiniBand has unique capabilities for direct data placement implemented by

RDMA and for OS bypassing, discovery, fail-over, remote boot, I/O sharing, and

other advanced features.

InfiniBand is the IT industry's solution for expanding data network centers to

prepare for the next generation of communications. e-business, Internet use and the

demands of the communications age have created the need for increased processing

power, larger stores of information and a greater, more reliable flow of data. As the

present architecture struggled under the current load of information, it was painfully

clear that a solution was needed in order to manage the increasing demands of the

future. The giants of the IT industry (Compaq, Dell, IBM, Intel, HP, SUN and

Microsoft followed

by more than 230 companies) combined the best elements of

two competing technological initiatives to create InfiniBand, an industry standard

that has been adopted with confidence. Solving a great number of obstacles that

data centers were facing (scalability, reliability, manageability), InfiniBand enabled

the introduction of data clusters, an invaluable next step to data center construction.

In effect, InfiniBand is a fabric-based, switched network that allows devices such as

servers, storage and I/O to communicate at very high speeds. Until the advent of

InfiniBand, communication devices had to wait in line to send their information

through one shared bus. The ten-year-old architecture had reached its limits and did

not have enough bandwidth available to meet the rising demands of the information

load.

InfiniBand architecture is based upon a very different concept. Offering high-speed

interprocessor communication and memory sharing, it enables you to build server

clusters with performance comparable to that of large servers at a fraction of the

price. Instead of one bus through which every device communicates, InfiniBand

architecture offers a network fabric that incorporates numerous switches and router

modules, allowing devices to communicate simultaneously. Speed is no longer

restricted by the shortcomings of the infrastructure. InfiniBand operates at

2.5Gbps (1X), 10Gbps (4X), 20Gbps (4X DDR), 30Gbps (12X) to 60 Gbps (12X

DDR).

InfiniBand architecture is modular, highly scalable (it allows for virtually unlimited

network expansion) and reduces the workload of the operating system kernel and

the CPU, freeing their power to run applications. Offering advanced fault isolation

controls, nonexistent in previous protocols, InfiniBand provides a high level of fault

tolerance. And most importantly, because the InfiniBand system is modular,

processing power based on commodity servers can be added as needed.

Introduction ISR 9024 System Overview

1-3

InfiniBand System Elements

The following are the InfiniBand system elements:

 HCA (Host Channel Adapter), an adapter or silicon chip residing on the host

motherboard providing the host side functionality and advanced reliable

message passing capabilities.



Switch, connecting all the nodes in a switched point-to-point manner.



Router Modules, providing connectivity to other InfiniBand or IP subnets.



Storage (native and Fibre Channel access)

InfiniBand provides a reliable, low latency interconnect (with typical switch latency

of 100nS) with fault-tolerance mechanisms (Automatic Path Migration), physical

multiplexing (Virtual Lanes), physical link aggregation (1X, 4X, 12X) and

security/partitioning.

The InfiniBand fabric provides central management and configuration controlling

discovery, failures and resource allocation.

1.2 ISR 9024 System Overview

The Voltaire ISR 9024 Grid Switch is a high performance, low latency, fully non-

blocking switch for high performance computing (HPC) clusters and grids. Offering

available bandwidth of 480 Gbps, the Voltaire ISR 9024 is an alternative to

proprietary interconnect technologies. With twenty-four 10 Gbpsports in a 1U

chassis, the standards-based ISR 9024 delivers three times the port density of

proprietary offerings. Using the Voltaire ISR 9024 Grid Switch, you can build high

performance clusters and grids that scale from several to hundreds of nodes.

The ISR 9024 serves as a building block for Fat Tree topologies of hundred of

nodes for scientific, commercial HPC and enterprise grid applications. When

combined with the Voltaire ISR 9288/9096 high port density switch, the Voltaire

ISR 9024 enables the formation of clusters scaling to thousand of nodes.

The ISR 9024 architecture features a passive cooling device (with no fans) resulting

in a noise free, high MTBF unit.

Introduction ISR 9288/9096 System Overview

1-4

Figure 1-1 shows the ISR 9024 functional block diagram.

Figure 1-1. Voltaire ISR 9024 Grid Switch™ Functional Diagram

1.3 ISR 9288/9096 System Overview

The Voltaire ISR 9288/9096 Grid Backbone is a modular platform for building high

levels of performance and scalability for large high performance computing (HPC)

clusters. The Voltaire ISR 9288/9096 enables high performance applications to run

on distributed server, storage and network resources.

Elegantly designed with 10/30 Gbps of full bisectional bandwidth for up to 96 or

288 4X/12X ports in a single chassis, multiple ISR 9288/9096 can also be

interconnected to form very large clusters.

Possible configurations range from dozens to thousands of nodes with many

upgrade paths available.

Featuring fat tree (Clos) topology that provides full bisectional bandwidth for each

port, the Voltaire ISR 9288/9096 lead the InfiniBand industry with the highest

switching capacity.

Providing unmatched levels of IniniBand bandwidth, the ISR 9096 and the

ISR 9288 respectively support up to 96 and 288 InfiniBand ports, which can be

used in any combination of 10 Gbps 4X Ports and or 30 Gbps InfiniBand 12X

ports. The 12X port incorporates three 4X InfiniBand ports aggregated into a single

InfiniBand port.

Wire-speed non-blocking switch capacity combined with latency of less than

Page is loading ...

Voltaire ISR 9024, ISR 9096, ISR 9288 User manual

Other documents