Supporting Fabric OS 7.4.0
Supporting Fabric OS 7.4.1
Supporting Fabric OS 7.4.2
CONFIGURATION GUIDE
Brocade Fabric OS Extension
Conguration Guide, 7.4.x
53-1003507-09
24 August 2017
©
2017, Brocade Communications Systems, Inc. All Rights Reserved.
Brocade, the B-wing symbol, and MyBrocade are registered trademarks of Brocade Communications Systems, Inc., in the United States and in other
countries. Other brands, product names, or service names mentioned of Brocade Communications Systems, Inc. are listed at www.brocade.com/en/legal/
brocade-Legal-intellectual-property/brocade-legal-trademarks.html. Other marks may belong to third parties.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment,
equipment feature, or service oered or to be oered by Brocade. Brocade reserves the right to make changes to this document at any time, without
notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade
sales oce for information on feature and product availability. Export of technical data contained in this document may require an export license from the
United States government.
The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to the accuracy of this
document or any loss, cost, liability, or damages arising from the information contained herein or the computer programs that accompany it.
The product described by this document may contain open source software covered by the GNU General Public License or other open source license
agreements. To nd out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, and
obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.
Brocade Fabric OS Extension Conguration Guide, 7.4.x
2 53-1003507-09
Contents
Preface...................................................................................................................................................................................................................................7
Document conventions............................................................................................................................................................................................................................7
Notes, cautions, and warnings.....................................................................................................................................................................................................7
Text formatting conventions.........................................................................................................................................................................................................7
Command syntax conventions....................................................................................................................................................................................................8
Text formatting conventions.........................................................................................................................................................................................................8
Command syntax conventions....................................................................................................................................................................................................9
Notes, cautions, and warnings.....................................................................................................................................................................................................9
Brocade resources.....................................................................................................................................................................................................................................9
Contacting Brocade Technical Support......................................................................................................................................................................................... 10
Brocade customers.......................................................................................................................................................................................................................10
Brocade OEM customers.......................................................................................................................................................................................................... 10
Document feedback.............................................................................................................................................................................................................................. 10
About This Document..................................................................................................................................................................................................... 11
Supported hardware and software...................................................................................................................................................................................................11
What's new in this document............................................................................................................................................................................................................. 11
Changes made for this release (53-1003507-02).......................................................................................................................................................13
Changes made for this release (53-1003507-03).......................................................................................................................................................13
Changes made for this release (53-1003507-04).......................................................................................................................................................13
Changes made for this release (53-1003507-05).......................................................................................................................................................13
Changes made for this release (53-1003507-06).......................................................................................................................................................13
Changes made for this release (53-1003507-07).......................................................................................................................................................13
Changes made for this release (53-1003507-08).......................................................................................................................................................14
Changes made for this release (53-1003507-09).......................................................................................................................................................14
Brocade Extension Concepts and Features................................................................................................................................................................15
Brocade Extension concepts..............................................................................................................................................................................................................15
Brocade IP Extension concepts...............................................................................................................................................................................................15
VE_Ports and VEX_Ports..........................................................................................................................................................................................................16
Extension interfaces, circuits, and trunks............................................................................................................................................................................. 17
Extension Trunking ................................................................................................................................................................................................................................18
Redundancy and fault tolerance..............................................................................................................................................................................................18
Tunnel restrictions for Fibre Channel Protocol and FICON Acceleration...............................................................................................................19
IP WAN network considerations....................................................................................................................................................................................................... 19
Adaptive Rate Limiting ........................................................................................................................................................................................................................ 19
Brocade 7840 Switch support for ARL...............................................................................................................................................................................20
FSPF link cost calculation when ARL is used................................................................................................................................................................... 20
Conguring ARL............................................................................................................................................................................................................................ 20
ARL conguration limitations................................................................................................................................................................................................... 21
Compression options............................................................................................................................................................................................................................ 21
Brocade 7800 switch and FX8-24 blade..........................................................................................................................................................................21
Brocade 7840 switch compression options......................................................................................................................................................................22
FastWrite and Open Systems Tape Pipelining............................................................................................................................................................................22
FastWrite and OSTP congurations...................................................................................................................................................................................... 23
Support for IPv6 addressing..............................................................................................................................................................................................................24
IPv6 with embedded IPv4 addresses...................................................................................................................................................................................25
Brocade Fabric OS Extension Conguration Guide, 7.4.x
53-1003507-09 3
Memory use limitations for large-device tunnel congurations.......................................................................................................................................... 26
Control blocks created during FCP trac ow..................................................................................................................................................................26
Control blocks created during FICON trac ow............................................................................................................................................................ 26
Eect of conguration on tunnel control block memory...............................................................................................................................................27
Firmware downloads..............................................................................................................................................................................................................................28
Extension Features on Brocade Extension Switches and Blades..........................................................................................................................31
Extension platforms and supported features.............................................................................................................................................................................. 31
Brocade 7800 Extension Switch.....................................................................................................................................................................................................33
License options.............................................................................................................................................................................................................................. 33
Brocade FX8-24 Extension Blade..................................................................................................................................................................................................34
Removing Brocade FX8-24 blades......................................................................................................................................................................................37
License options.............................................................................................................................................................................................................................. 37
10 GbE port considerations......................................................................................................................................................................................................37
Multigigabit circuits....................................................................................................................................................................................................................... 38
Crossports........................................................................................................................................................................................................................................ 38
Bandwidth allocation and restrictions....................................................................................................................................................................................41
Brocade 7840 Extension Switch.....................................................................................................................................................................................................44
Brocade 7840 Fibre Channel port groups.........................................................................................................................................................................45
Brocade 7840 DP components and VE_Port distribution.........................................................................................................................................45
10 GbE and 40 GbE port considerations...........................................................................................................................................................................48
Ethernet port grouping................................................................................................................................................................................................................ 48
10VE and 20VE port modes...................................................................................................................................................................................................49
Extension Hot Code Load..........................................................................................................................................................................................................50
IP Extension..................................................................................................................................................................................................................................... 53
License options.............................................................................................................................................................................................................................. 56
Path Maximum Transmission Unit discovery..............................................................................................................................................................................57
Tunnel and circuit requirements........................................................................................................................................................................................................57
General tunnel, circuit, and port requirements...................................................................................................................................................................58
Brocade 7800 extension switches........................................................................................................................................................................................ 58
Brocade FX8-24 extension blades....................................................................................................................................................................................... 59
Brocade 7840 extension switches........................................................................................................................................................................................ 60
Circuit failover........................................................................................................................................................................................................................................... 61
Circuit Failover Grouping............................................................................................................................................................................................................ 62
10 GbE Lossless Link Loss (FX8-24 blade).....................................................................................................................................................................65
Failover in TI zones....................................................................................................................................................................................................................... 67
Bandwidth calculation during failover....................................................................................................................................................................................67
Conguring Extension Features....................................................................................................................................................................................69
Conguration preparation.................................................................................................................................................................................................................... 69
Conguration steps................................................................................................................................................................................................................................ 70
Setting VE_Ports to persistently disabled state......................................................................................................................................................................... 71
Disabling ports when FMS Mode is enabled.....................................................................................................................................................................71
Conguring VEX_Ports........................................................................................................................................................................................................................ 71
Conguring the media type for GbE ports 0 and 1 (Brocade 7800 switch).................................................................................................................72
Setting the GbE port operating mode on FX8-24 blade only.............................................................................................................................................72
Conguring switch and port modes (7840 switch)..................................................................................................................................................................73
Conguring port speed (Brocade 7840 switch)........................................................................................................................................................................ 74
Conguring an IPIF................................................................................................................................................................................................................................ 75
Conguring an IP route.........................................................................................................................................................................................................................76
Commands for conguring IP routes....................................................................................................................................................................................77
Brocade Fabric OS Extension Conguration Guide, 7.4.x
4 53-1003507-09
Commands for modifying IP routes......................................................................................................................................................................................78
Validating IP connectivity.....................................................................................................................................................................................................................78
Creating an Extension tunnel..............................................................................................................................................................................................................78
Creating an FX8-24 and Brocade 7800 tunnel..............................................................................................................................................................79
Creating Brocade 7840 tunnels............................................................................................................................................................................................. 80
Tunnel conguration options.....................................................................................................................................................................................................80
Keep-alive timeout option..........................................................................................................................................................................................................80
Creating additional circuits ................................................................................................................................................................................................................. 81
Verifying the tunnel conguration.....................................................................................................................................................................................................82
Conguring Extension HCL................................................................................................................................................................................................................82
DP and tunnel conguration..................................................................................................................................................................................................... 83
Conguring backup tunnels.......................................................................................................................................................................................................84
Enabling persistently disabled ports............................................................................................................................................................................................... 85
Disabling ports with FMS Mode enabled............................................................................................................................................................................86
Modifying a tunnel..................................................................................................................................................................................................................................86
Modifying a circuit...................................................................................................................................................................................................................................86
Deleting an IP interface ....................................................................................................................................................................................................................... 87
Deleting an IP route .............................................................................................................................................................................................................................. 87
Deleting a trunk........................................................................................................................................................................................................................................87
Deleting a circuit .....................................................................................................................................................................................................................................88
Conguring Per-Priority TCP QoS priorities over a trunk......................................................................................................................................................88
Modifying default priority values.......................................................................................................................................................................................................89
Using logical switches...........................................................................................................................................................................................................................90
Logical switch overview ............................................................................................................................................................................................................. 90
Considerations for logical switches........................................................................................................................................................................................92
Managing QoS, DSCP, and VLANs................................................................................................................................................................................................98
DSCP Quality of Service............................................................................................................................................................................................................ 99
VLANs and Layer 2 Quality of Service................................................................................................................................................................................ 99
Managing DSCP and VLAN support on circuits..............................................................................................................................................................99
When both DSCP and L2CoS are used........................................................................................................................................................................... 100
Managing the VLAN tag table...............................................................................................................................................................................................101
Implementing IPsec over tunnels..................................................................................................................................................................................................102
Limitations using IPsec over tunnels..................................................................................................................................................................................102
IPsec for the extension switches and blades...................................................................................................................................................................103
Enabling IPsec and IKE policies...........................................................................................................................................................................................103
Brocade 7840 IKE authentication failures.......................................................................................................................................................................105
Trac Isolation Zoning.......................................................................................................................................................................................................................105
Conguring IP Extension features............................................................................................................................................................................. 107
Conguration preparation for IP Extension features..............................................................................................................................................................107
Conguration steps for IP Extension features..........................................................................................................................................................................108
Conguring hybrid mode for IP Extension features.............................................................................................................................................................. 109
Conguring GbE port for IP Extension LAN features...........................................................................................................................................................110
Conguring LAG.................................................................................................................................................................................................................................. 111
Conguring switch virtual interface IPIF.....................................................................................................................................................................................113
Conguring a tunnel to support IP Extension..........................................................................................................................................................................114
Conguring bandwidth distribution...............................................................................................................................................................................................114
Conguring tunnel compression................................................................................................................................................................................................... 115
Conguring trac control lists........................................................................................................................................................................................................ 116
Conguring TCL for non-terminated trac.............................................................................................................................................................................. 117
Brocade Fabric OS Extension Conguration Guide, 7.4.x
53-1003507-09 5
Management and Troubleshooting............................................................................................................................................................................119
In-band management........................................................................................................................................................................................................................ 119
IP routing........................................................................................................................................................................................................................................119
Conguring IP addresses and routes.................................................................................................................................................................................120
VLAN tagging support.............................................................................................................................................................................................................124
IP forwarding support............................................................................................................................................................................................................... 124
WAN performance analysis tools..................................................................................................................................................................................................126
The tperf option...........................................................................................................................................................................................................................126
Using ping to test a connection............................................................................................................................................................................................ 127
Using traceroute..........................................................................................................................................................................................................................128
Using WAN Tool..........................................................................................................................................................................................................................128
Using the portshow command.......................................................................................................................................................................................................133
Displaying IP interfaces............................................................................................................................................................................................................133
Displaying IP routes...................................................................................................................................................................................................................133
Displaying switch mode information with the extncfg command.......................................................................................................................... 133
Displaying GbE port information with the portcfgge command.............................................................................................................................134
Listing the MAC addresses of LAN and GE ports....................................................................................................................................................... 134
Displaying LAG information...................................................................................................................................................................................................134
Displaying tunnel information................................................................................................................................................................................................135
Displaying tunnel HCL information.....................................................................................................................................................................................135
Displaying TCL information....................................................................................................................................................................................................135
Displaying IP Extension LAN statistics..............................................................................................................................................................................135
Displaying performance statistics........................................................................................................................................................................................136
Displaying QoS statistics.........................................................................................................................................................................................................136
Displaying details........................................................................................................................................................................................................................136
Displaying tunnel information (Brocade 7800)............................................................................................................................................................. 136
Displaying a tunnel with circuit information .................................................................................................................................................................... 136
Displaying tunnel performance ............................................................................................................................................................................................136
Displaying tunnel TCP statistics ..........................................................................................................................................................................................137
Displaying circuits.......................................................................................................................................................................................................................137
Displaying a single circuit........................................................................................................................................................................................................ 137
Displaying TCP statistics for circuits...................................................................................................................................................................................137
Displaying circuit performance ............................................................................................................................................................................................ 137
Displaying QoS prioritization for a circuit......................................................................................................................................................................... 137
Displaying tunnel information (FX8-24 blade)...............................................................................................................................................................138
Tunnel issues ........................................................................................................................................................................................................................................ 138
Tunnel does not come online................................................................................................................................................................................................ 138
Tunnel goes online and oine...............................................................................................................................................................................................139
Troubleshooting Extension links....................................................................................................................................................................................................139
Gathering additional information..........................................................................................................................................................................................140
Using FTRACE..................................................................................................................................................................................................................................... 141
FTRACE conguration.............................................................................................................................................................................................................141
Changing conguration settings...........................................................................................................................................................................................142
Displaying FTRACE status on a DP complex................................................................................................................................................................ 145
Brocade Fabric OS Extension Conguration Guide, 7.4.x
6 53-1003507-09
Preface
• Document conventions......................................................................................................................................................................................7
• Brocade resources............................................................................................................................................................................................... 9
• Contacting Brocade Technical Support....................................................................................................................................................10
• Document feedback.........................................................................................................................................................................................10
Document conventions
The document conventions describe text formatting conventions, command syntax conventions, and important notice formats used in
Brocade technical documentation.
Notes, cautions, and warnings
Notes, cautions, and warning statements may be used in this document. They are listed in the order of increasing severity of potential
hazards.
NOTE
A Note provides a tip, guidance, or advice, emphasizes important information, or provides a reference to related information.
ATTENTION
An Attention statement indicates a stronger note, for example, to alert you when trac might be interrupted or the device might
reboot.
CAUTION
A Caution statement alerts you to situations that can be potentially hazardous to you or cause damage to hardware,
rmware, software, or data.
DANGER
A Danger statement indicates conditions or situations that can be potentially lethal or extremely hazardous to you. Safety
labels are also attached directly to products to warn of these conditions or situations.
Text formatting conventions
Text formatting conventions such as boldface, italic, or Courier font may be used to highlight specic words or phrases.
Format Description
bold text Identies command names.
Identies keywords and operands.
Identies the names of GUI elements.
Identies text to enter in the GUI.
italic text Identies emphasis.
Identies variables.
Identies document titles.
Courier font
Identies CLI output.
Brocade Fabric OS Extension Conguration Guide, 7.4.x
53-1003507-09 7
Format Description
Identies command syntax examples.
Command syntax conventions
Bold and italic text identify command syntax components. Delimiters and operators
dene groupings of parameters and their logical
relationships.
Convention Description
bold text Identies command names, keywords, and command options.
italic text Identies a variable.
value In Fibre Channel products, a xed value provided as input to a command option is printed in plain text, for
example, --show WWN.
[ ] Syntax components displayed within square brackets are optional.
Default responses to system prompts are enclosed in square brackets.
{ x | y | z } A choice of required parameters is enclosed in curly brackets separated by vertical bars. You must select
one of the options.
In Fibre Channel products, square brackets may be used instead for this purpose.
x | y A vertical bar separates mutually exclusive elements.
< > Nonprinting characters, for example, passwords, are enclosed in angle brackets.
... Repeat the previous element, for example, member[member...].
\ Indicates a “soft” line break in command examples. If a backslash separates two lines of a command
input, enter the entire command at the prompt without the backslash.
Text formatting conventions
Text formatting conventions such as boldface, italic, or Courier font may be used in the ow of the text to highlight specic words or
phrases.
Format Description
bold text Identies command names
Identies keywords and operands
Identies the names of user-manipulated GUI elements
Identies text to enter at the GUI
italic text Identies emphasis
Identies variables
Identies document titles
Courier font
Identies CLI output
Identies command syntax examples
Document conventions
Brocade Fabric OS Extension Conguration Guide, 7.4.x
8 53-1003507-09
Command syntax conventions
Bold and italic text identify command syntax components. Delimiters and operators dene groupings of parameters and their logical
relationships.
Convention Description
bold text Identies command names, keywords, and command options.
italic text Identies a variable.
value In Fibre Channel products, a xed value provided as input to a command option is printed in plain text, for
example, --show WWN.
[ ] Syntax components displayed within square brackets are optional.
Default responses to system prompts are enclosed in square brackets.
{ x | y | z } A choice of required parameters is enclosed in curly brackets separated by vertical bars. You must select
one of the options.
In Fibre Channel products, square brackets may be used instead for this purpose.
x | y A vertical bar separates mutually exclusive elements.
< > Nonprinting characters, for example, passwords, are enclosed in angle brackets.
... Repeat the previous element, for example, member[member...].
\ Indicates a “soft” line break in command examples. If a backslash separates two lines of a command
input, enter the entire command at the prompt without the backslash.
Notes, cautions, and warnings
Notes, cautions, and warning statements may be used in this document. They are listed in the order of increasing severity of potential
hazards.
NOTE
A Note provides a tip, guidance, or advice, emphasizes important information, or provides a reference to related information.
ATTENTION
An Attention statement indicates a stronger note, for example, to alert you when trac might be interrupted or the device might
reboot.
CAUTION
A Caution statement alerts you to situations that can be potentially hazardous to you or cause damage to hardware,
rmware, software, or data.
DANGER
A Danger statement indicates conditions or situations that can be potentially lethal or extremely hazardous to you. Safety
labels are also attached directly to products to warn of these conditions or situations.
Brocade resources
Visit the Brocade website to locate related documentation for your product and additional Brocade resources.
White papers, data sheets, and the most recent versions of Brocade software and hardware manuals are available at www.brocade.com.
Product documentation for all supported releases is available to registered users at MyBrocade.
Click the Support tab and select Document Library to access product documentation on MyBrocade or www.brocade.com. You can
locate documentation by product or by operating system.
Brocade resources
Brocade Fabric OS Extension Conguration Guide, 7.4.x
53-1003507-09 9
Release notes are bundled with software downloads on MyBrocade. Links to software downloads are available on the MyBrocade landing
page and in the Document Library.
Contacting Brocade Technical Support
As a Brocade customer, you can contact Brocade Technical Support 24x7 online or by telephone. Brocade OEM customers should
contact their OEM/solution provider.
Brocade customers
For product support information and the latest information on contacting the Technical Assistance Center, go to www.brocade.com and
select Support.
If you have purchased Brocade product support directly from Brocade, use one of the following methods to contact the Brocade
Technical Assistance Center 24x7.
Online Telephone
Preferred method of contact for non-urgent issues:
• Case management through the MyBrocade portal.
• Quick Access links to Knowledge Base, Community, Document
Library, Software Downloads and Licensing tools
Required for Sev 1-Critical and Sev 2-High issues:
• Continental US: 1-800-752-8061
• Europe, Middle East, Africa, and Asia Pacic: +800-AT FIBREE
(+800 28 34 27 33)
• Toll-free numbers are available in many countries.
• For areas unable to access a toll-free number:
+1-408-333-6061
Brocade OEM customers
If you have purchased Brocade product support from a Brocade OEM/solution provider, contact your OEM/solution provider for all of
your product support needs.
• OEM/solution providers are trained and
certied by Brocade to support Brocade
®
products.
• Brocade provides backline support for issues that cannot be resolved by the OEM/solution provider.
• Brocade Supplemental Support augments your existing OEM support contract, providing direct access to Brocade expertise.
For more information, contact Brocade or your OEM.
• For questions regarding service levels and response times, contact your OEM/solution provider.
Document feedback
Quality is our rst concern at Brocade, and we have made every eort to ensure the accuracy and completeness of this document.
However, if you nd an error or an omission, or you think that a topic needs further development, we want to hear from you. You can
provide feedback in two ways:
• Through the online feedback form in the HTML documents posted on www.brocade.com
• By sending your feedback to documentation@brocade.com
Provide the publication title, part number, and as much detail as possible, including the topic heading and page number if applicable, as
well as your suggestions for improvement.
Contacting Brocade Technical Support
Brocade Fabric OS Extension Conguration Guide, 7.4.x
10 53-1003507-09
About This Document
• Supported hardware and software..............................................................................................................................................................11
• What's new in this document........................................................................................................................................................................11
Supported hardware and software
The following hardware platforms support Brocade Extension (Fibre Channel over IP features and IP Extension features) as described in
this manual:
• Brocade DCX, DCX-4S, DCX 8510-4, and DCX 8510-8 with one or more FX8-24 blades
• Brocade 7800 switch
• Brocade 7840 switch
What's new in this document
This document includes new and
modied information for the Fabric OS 7.4.0.
Major new additions or deletions in this document support the new IP Extension features related to the Brocade 7840 Extension Switch,
changes for Fabric OS v7.4.0, and corrections. An additional change to this document is the repositioning of the Brocade Fibre Channel
over IP (FCIP) features as the Brocade Extension features. Note that Brocade Fabric OS (FOS) commands that support Extension still
use "fcip".
Major sections of this publication aected by additions and corrections include the following:
• Extension Concepts and Features
– Added Tunnel restrictions for Fibre Channel Protocol and FICON Acceleration on page 19 information for the IP
Extension features.
– In the section IP WAN network considerations on page 19, claried that the network must allow ESP trac to pass when
using IPsec.
– In the section Memory use limitations for large-device tunnel congurations on page 26, updated memory information
for the Brocade 7840 from 512 MB to 1.3 GB.
– In the section Eect of conguration on tunnel control block memory on page 27, updated the memory threshold
information for FOS 7.4.0.
– In the section Firmware downloads on page 28, updated Extension HCL information for Fabric OS 7.4.0 operation.
• Extension Features on Brocade Extension Switches and Blades
– In the section Redundancy and fault tolerance on page 18, added a note about using Lossless Dynamic Load Sharing
(DLS) with multiple parallel tunnels.
– In the section Brocade FX8-24 Extension Blade on page 34, and in the section Crossports on page 38, corrected
information about XGE port numbering supported by Brocade FX8-24 blade and DP complexes.
– In the section Brocade 7840 switch considerations and limitations on page 98, restrictions are removed for using the
7840 switch as a base switch.
– Added IP Extension on page 53 to describe IP Extension features on the Brocade 7840 switch.
– In the section Removing Brocade FX8-24 blades on page 37, updated the instructions for blade removal.
– In the section Front-end and back-end bandwidth on page 41, corrected the illustration showing DP0 and DP1 port
associations.
Brocade Fabric OS Extension Conguration Guide, 7.4.x
53-1003507-09 11
– In the section Multigigabit circuits on page 48, added information on maximum number of circuits per tunnel.
– In the section Extension Hot Code Load on page 50, changed primary tunnel (PT) to main tunnel (MT).
– In the section Extension HCL Limitations and considerations on page 51, added information about using LLL when
parallel tunnels are congured.
– Added sections describing IP Extension, beginning with IP Extension on page 53.
– In the section Examples of circuit failover in groups on page 63, a conguration example was removed.
– In the section Conguring circuit failover groups on page 64, redundant information was removed.
• Conguring Extension
– In the 7.3.0 release, this information was known as Conguring FCIP.
– In the section Conguring switch and port modes (7840 switch) on page 73, add information for Brocade 7840 switch
modes to include FCIP mode and hybrid mode.
– In the section Conguration steps on page 70, updated information for the Brocade 7840.
– In the section Setting the GbE port operating mode on FX8-24 blade only on page 72, updated the output of the
bladecfggemode --show command.
– In the sectionConguring switch and port modes (7840 switch) on page 73, updated conguration modes.
– In the section Conguring port speed (Brocade 7840 switch) on page 74, updated the conguration steps and removed
an unsupported portCfgGe ge4 --set -speed auto command.
– In the section Commands for modifying IP routes on page 78, updated the information to indicate the command is for
the Brocade 7840 only.
– In the section Tunnel conguration options on page 80 , tables were removed describing tunnel and circuit options for the
portcfg fciptunnel create command and the portcfg fciptunnel modify command. You can refer to Brocade Fabric OS
Command Reference for information about the portcfg fciptunnel create command and the portcfg fciptunnel modify
command.
– The section "Creating a trunk (example)" showing an example of multi-circuit trunk creation has been removed.
– In the section Conguring Extension HCL on page 82, replaced the terms primary tunnel and PT with main tunnel and
MT. The ouptut example was updated for FOS 7.4.0.
– In the section Connecting logical switches on page 92, added support for Brocade 7840 switch.
– In the section Port sharing example on page 93, updated the command output example.
– In the section Brocade 7800 switch considerations and limitations on page 97, updated the allowed downgrade version
from v7.0.0 to v7.1.0.
– In the section Enabling XISL for VE_Ports (FX8-24 blade / 7840 switch) on page 98, removed the limits on using
Brocade 7800 and FX8-24 as base switches for XISL. XISL for Brocade 7840 was added.
– In the section Managing DSCP and VLAN support on circuits on page 99, removed the table of command options for
the portcfg fciptunnel create command and the portcfg fciptunnel modify command.
– In the section VLAN tagging examples on page 99, added a note about setting the VLAN ID for the Brocade 7840
switch.
– In the section Limitations using IPsec over tunnels on page 102, updated IPsec information for the Brocade 7840 switch.
– In the section IPsec for the extension switches and blades on page 103, updated information about the SA lifetime.
– In the section Enabling IPsec and IKE policies on page 103, added information for the Brocade 7840 IKE key, and
updated the example.
– Added the section Brocade 7840 IKE authentication failures on page 105.
• Conguring IP Extension
– Added the chapter Conguring IP Extension for the IP Extension feature conguration tasks. All content is new for 7.4.0.
• Extension Management and Troubleshooting
– In the section Redundant connections to the management stations example on page 122, updated the conguration
example.
What's new in this document
Brocade Fabric OS Extension Conguration Guide, 7.4.x
12 53-1003507-09
– In the section Using the portshow command on page 133, added the following new information for IP Extension on the
Brocade 7840.
› Displaying switch mode information with the extncfg command on page 133
› Displaying GbE port information with the portcfgge command on page 134
› Displaying LAG information on page 134
› Displaying tunnel HCL information on page 135
› Displaying TCL information on page 135
› Displaying IP Extension LAN statistics on page 135
› Listing the MAC addresses of LAN and GE ports on page 134
– Updated the section Gathering additional information on page 140 to include commands useful for IP Extension features.
Changes made for this release (53-1003507-02)
Minor changes have been made to the document to enhance presentation of the graphics and illustrations.
Changes made for this release (53-1003507-03)
In the section IP WAN network considerations on page 19, added information about not dropping the URG
ag from ports 3225 and
3226.
Changes made for this release (53-1003507-04)
In Firmware downloads on page 28 and Extension HCL Limitations and considerations on page 51, added information that when
Teradata emulation is congured on a Brocade 7840 switch, Extension HCL is not supported.
Changes made for this release (53-1003507-05)
In Brocade 7800 Extension Switch on page 33, corrected information to state total throughput is 4 Gbps.
In Keep-alive timeout option on page 80, added information that the default keep-alive timeout values (KATOV) for the portcfg
fciptunnel command will
dier depending on platform and FICON emulation status.
In Conguration steps for IP Extension features on page 108, added a note that IP Extension end points must be on dierent subnets.
Changes made for this release (53-1003507-06)
The following changes have been made:
• In Extension platforms and supported features on page 31, claried Fabric OS 7.0.1b only that supports Silver Lake
acceleration hardware.
• Added Conguring TCL for non-terminated trac on page 117.
Changes made for this release (53-1003507-07)
The following changes have been made:
• Added information about Fibre Channel (FC) port groups for the Brocade 7840 switch, Brocade 7840 Fibre Channel port
groups on page 45.
What's new in this document
Brocade Fabric OS Extension Conguration Guide, 7.4.x
53-1003507-09 13
Changes made for this release (53-1003507-08)
The following changes have been made:
• In VE_Ports and VEX_Ports on page 16, corrected a typo in the table header.
• In Tunnel restrictions for Fibre Channel Protocol and FICON Acceleration on page 19, removed a statement that concurrent
upgrade is supported.
• In Extension Hot Code Load on page 50, added a note that concurrent upgrade is not supported.
• In Brocade 7840 Fibre Channel port groups on page 45, corrected information about trunk port groups.
Changes made for this release (53-1003507-09)
The following changes have been made:
• In Managing the VLAN tag table on page 101, replaced portcfg with portshow to correct an error in the task list.
What's new in this document
Brocade Fabric OS Extension Conguration Guide, 7.4.x
14 53-1003507-09
Brocade Extension Concepts and
Features
• Brocade Extension concepts........................................................................................................................................................................15
• Extension Trunking .......................................................................................................................................................................................... 18
• IP WAN network considerations..................................................................................................................................................................19
• Adaptive Rate Limiting ...................................................................................................................................................................................19
• Compression options.......................................................................................................................................................................................21
• FastWrite and Open Systems Tape Pipelining...................................................................................................................................... 22
• Support for IPv6 addressing.........................................................................................................................................................................24
• Memory use limitations for large-device tunnel congurations.....................................................................................................26
• Firmware downloads........................................................................................................................................................................................28
Brocade Extension concepts
Brocade Extension products support FC/FICON based data ows as well as IP based storage data ows.
Brocade Extension enables you to use the existing IP wide are network (WAN) infrastructure to connect Fibre Channel and IP fabrics.
Brocade Extension supports applications such as remote data replication (RDR), centralized backup, and data migration over very long
distances that are impractical or very costly using native Fibre Channel or IP connections. Extension tunnels, built on a physical
connection between two extension switches or blades, allow Fibre Channel and IP I/O to pass through the IP WAN.
The extension tunnel and TCP connections ensure in-order delivery of Fibre Channel (FC) and IP frames and lossless transmission. The
Fibre Channel fabric and all targets and initiators, whether FC or IP, are unaware of the presence of the IP WAN.
Brocade IP Extension concepts
The Brocade 7840 Extension switch supports IP based storage data ows as well as FC/FICON based data ows. IP extensions provide
enterprise-class support for IP storage applications, using existing IP wide area network (WAN) infrastructure. IP Extension features are
oered only on the Brocade 7840 Extension Switch platform.
IP data ows across Brocade tunnels are referred to as IP Extension. IP Extension provides storage administrators with the means to
monitor and manage their IP storage ows across an IP infrastructure.
IP Extension enables you to use the existing IP WAN infrastructure to connect IP storage. Additionally, IP Extension gives you visibility
and control of ows using various diagnostic tools, IPsec, compression, QoS, Extension Trunking, and lossless tunnel resiliency. IP
Extension supports applications such as array native IP remote data replication (RDR), IP based centralized backup, VM replication; host
based and database replication over IP, NAS head replication between data centers, data migration between data centers and others.
Brocade WAN Optimized TCP (WO-TCP) ensures in-order lossless transmission of IP Extension data. IP Extension establishes a proxy
TCP endpoint for local devices. Local devices are unaware and unaected by the latency and quality of the IP WAN. This accelerates end
device native TCP. IP Extension data across the IP WAN uses WO-TCP, a highly ecient and aggressive TCP stack for moving data
between data centers.
IP Extension provides the following advantages:
• Data Center Interconnect (DCI) — Unied support and management of both FC/FICON and IP
• Storage Administrators — Provision once and over time connect many devices
• High performance for high speed WAN links (one or more 10 Gbps and 40 Gbps links)
Brocade Fabric OS Extension Conguration Guide, 7.4.x
53-1003507-09 15
• WAN bandwidth pooling — pool bandwidth from multiple links/providers
• Lossless link loss (LLL)
• Adaptive Rate Limiting (ARL)
• Network resiliency and high availability using Extension Trunking
• Ecient protocol transport — negligible added overhead
• TCP Acceleration with WAN Optimized TCP
• Streams — Virtual windows on WAN Optimized TCP to eliminate head of line blocking (HoLB)
• High speed compression using Deate
• High speed IPsec (AES 256)
• Diagnostic and troubleshooting tools — WAN Health and Wtool
• Separate QoS for both FCIP and IP Extension with DSCP and/or 802.1P marking and enforcement
• 9216 byte Jumbo Frames for both LAN and WAN networks
FIGURE 1 Extension tunnel concept and TCP/IP layers for FCIP and IP Extension
VE_Ports and VEX_Ports
Any FC communications between switches that include fabric services needs to communicate across an ISL using E_Ports. There are
various types of E_Ports:
• E_Port
• EX_Port
• VE_Port
• VEX_Port
VE and VEX ports are virtual because they are extension tunnel facing. These are the ports that enable communication across an
extension tunnel. EX and VEX ports are FC Routed ports. Router ports are the demarcation point of fabric services for a fabric. Fabric
services do not extend beyond an EX or VEX port.
Brocade Extension concepts
Brocade Fabric OS Extension Conguration Guide, 7.4.x
16 53-1003507-09
E port types No FCR FCR
Native FC E_Port EX_Port
Extended over tunnel VE_Port VEX_Port
NOTE
VEX_Ports are not supported on the Brocade 7840 switch.
Once the tunnels are congured and the WO-TCP connections are made for a circuit, a logical interswitch link (ISL) is established
between the switches. VE_Ports operate like E_Ports for all fabric services and Fabric OS operations, except that VE_Ports use TCP/IP
and Ethernet as the transport instead of FC.
A "virtual" EX_Port exposed by the extension tunnel to form an ISL connection allows you to congure a virtual EX_Port or VEX_Port to
support FCR demarcation. From the point of view of a switch in an edge fabric, a VEX_Port appears as a normal E_Port. It follows the
same Fibre Channel protocol as other E_Ports. However, VEX_Ports terminate the attached fabric at the port and do not allow fabrics to
merge by propagating fabric services or routing topology information beyond that edge fabric. This provides edge fabric,or remote edge
fabric isolation outward from the EX_Port or VEX_Port.
NOTE
VE_Ports or VEX_Ports cannot connect in parallel to the same domain at the same time as Fibre Channel E_Ports or
EX_Ports.
An extension tunnel is assigned to a VE_Port or VEX_Port on the switch or blade at each end of the tunnel. Because multiple VE_Ports
and VEX_Ports can exist on the extension switch or blade, you can create multiple tunnels.
Fibre Channel frames enter an extension tunnel through virtual E_Ports (VE_Ports) or virtual extension ports (VEX_Ports) and are
encapsulated and passed to TCP layer connections. A Data Processing (DP) complex on the switch or blade handles the FC frame
encapsulation, de-encapsulation, and transmission to the TCP link.
Extension interfaces, circuits, and trunks
A circuit is a connection between a pair of IP addresses that are associated with source and destination endpoints of an extension tunnel.
Circuits provide the links for trac ow between source and destination interfaces that are located on either end of the tunnel. For each
tunnel, you can congure a single circuit or a trunk consisting of multiple circuits. Multiple circuits can be congured per Ethernet port by
assigning them unique IP interfaces (IPIFs). When you congure a circuit, you provide the IP addresses for its source and destination
interfaces.
You must congure unique IPIFs at the endpoint of each circuit. An IPIF consists of an IP address for a circuit endpoint, subnet mask,
and an MTU size. For the Brocade 7840 switch, the IPIF can also contain a VLAN ID for VLAN tagging and the ability to enable PMTU.
If the remote IPIF is not on the same subnet as the local IPIF, you must congure an IP route to that destination specifying the gateway
to use. You can dene a specic number of routes per IPIF based on the extension platform. Refer to Tunnel and circuit requirements on
page 57 for specications. An Ethernet interface can contain multiple IPIFs.
NOTE
In this publication, the "source" or "local" is the switch you are conguring, while the "destination" or "remote" is the switch on the
other end of the tunnel.
Congure an extension tunnel by specifying a VE_Port for a source and destination interface. When you congure a circuit on the tunnel,
you will provide two IP addresses, one for the source and one for the destination IP interface.
For Extension Hot Code Load (HCL) tunnels, four IP addresses are congured per circuit, which includes both endpoints. The four
addresses are the local and remote IP addresses, and the local and remote HA IP addresses used by HCL. In most instances, the two
local IP addresses are in the same subnet and the two remote IP addresses are in the same subnet. All IP addresses must be able to
Brocade Extension concepts
Brocade Fabric OS Extension Conguration Guide, 7.4.x
53-1003507-09 17
communicate across the IP infrastructure. Extension HCL is supported only on the Brocade 7840. For additional information, refer to
Conguring Extension HCL on page 82.
FIGURE 2 Extension tunnel and circuits
An extension trunk is a tunnel consisting of multiple circuits.
For specications and restrictions on tunnels, circuits, and trunks for the Brocade 7800 Extension Switch, 7840 Extension Switch, and
the FX8-24 Extension Blade, refer to the Extension Features on Brocade Extension Switches and Blades chapter.
Extension Trunking
Extension Trunking is a method for managing the use of WAN bandwidth and providing redundant paths over the WAN that can protect
against transmission loss due to WAN failure. Extension Trunking also provides granular load balancing on a weighted round-robin basis
per batch. Trunking is enabled by creating multiple circuits within a tunnel so that the tunnel utilizes multiple circuits to carry
trac
between multiple source and destination addresses. For circuit capacities for Brocade extension switches and blades, refer to Tunnel and
circuit requirements on page 57.
Redundancy and fault tolerance
Multiple extension tunnels can be
dened between pairs of extension switches or blades, but doing so defeats the benets of a multiple-
circuit extension tunnel. Dening two tunnels between a pair of switches or blades is not as redundant or fault-tolerant as having multiple
circuits in one tunnel.
Extension Trunking provides lossless link loss (LLL). LLL ensures all data lost in ight is retransmitted and placed back in order prior to
being delivered to upper layer protocols. This is an essential feature to prevent interface control checks (IFCCs) on mainframes using
FICON and SCSI timeouts for open-system-based replication. For more information about LLL on specic Brocade extension switches
and blades, refer to Circuit failover on page 61.
NOTE
When you create multiple parallel tunnels between the same switch domains, you must enable Lossless Dynamic Load Sharing
(DLS). This is because there can be routing updates that will occur when tunnels come up or go down. Each routing update can
cause dropped, or unrouteable frames if the destination is via the peer tunnel connected switch domain.
Extension Trunking
Brocade Fabric OS Extension Conguration Guide, 7.4.x
18 53-1003507-09
Tunnel restrictions for Fibre Channel Protocol and FICON Acceleration
Multiple extension tunnels within the same switch are not supported between pairs of extension switches or blades when protocol
optimization features—FastWrite, OSTP, and FICON Acceleration—are enabled on the tunnel, unless Trac Isolation (TI) zones or logical
switch/logical fabric (LS/LF) congurations are used to provide deterministic ows between the switches. These features require
deterministic FC frame routing between initiators and targets when multiple tunnels or VE_Ports exist. Non-controlled, parallel (equal-
cost multi-path) tunnels are not supported between domains when protocol optimization is enabled on one or more tunnels without
controlling the routing of SID/DID pairs to a specic tunnel using TI Zones or Virtual Fabrics (VF) LS/LF congurations.
Note the following additional restrictions:
• FICON networks with or without FICON Acceleration do not support exchange-based routing (EBR) congurations.
• The recommended best practice is to have identical FOS versions at both ends of an extension tunnel.
• When planning Fabric OS upgrades or downgrades, it is recommended that you upgrade or downgrade both endpoints of an
extension tunnel with the same FOS version.
• When conguring tunnels to support large numbers of devices, consider memory limitations of the Brocade extension switch or
blade if you are enabling any type of protocol optimization feature. If too many devices are present or activated at one time,
protocol optimization such as FICON Acceleration can be negatively impacted. Refer to Memory use limitations for large-
device tunnel congurations on page 26.
IP WAN network considerations
Because Brocade Extension tunnels use TCP connections over an existing wide area network, consult with the WAN carrier and IP
network administrator to ensure that the network hardware and software equipment operating in the data path can properly support the
TCP connections. Keep the following considerations in mind:
• Routers and
rewalls that are in the data path must be congured to pass trac through a specic TCP port on the switch. If
IPsec is used, the network must allow Encapsulating Security Payload (ESP) trac to pass through (UDP port 500). The
Brocade WO-TCP implementation selects a port between 49152 and 65535 as the ephemeral (or initiating) port to open up
to port 3225 and 3226.
• On the Brocade 7840 switch, the TCP URG ag is frequently set. This ag indicates the beginning of a tunnel header in the
TCP packet. Make sure these ags are not dropped from ports 3225 and 3226.
• The Brocade 7800 switch and Brocade FX8-24 blade use TCP port 3225.
• The Brocade 7840 switch uses ports 3225 and 3226.
• To enable recovery from a WAN failure or outage, be sure that diverse, redundant network paths are available across the WAN.
• Be sure the underlying WAN infrastructure can support the redundancy and performance expected in your implementation.
Adaptive Rate Limiting
Adaptive Rate Limiting (ARL) is performed on circuits to change the rate in which the tunnel transmits data through the IP network. ARL
uses information from the TCP connections to determine and adjust the rate limit for the circuit dynamically. This allows connections to
utilize the maximum available bandwidth while providing a minimum bandwidth guarantee. ARL is
congured on a per-circuit basis
because each circuit may have available dierent amounts of bandwidth.
ARL is supported only if Fabric OS v7.0.0 and later is running on both ends of the tunnel. For Fabric OS v7.0.0 and later, you can
congure minimum and maximum rates for each circuit of a tunnel using the following ports:
• XGE (10 GbE) ports on the Brocade FX8-24 blade (xge0 and xge1).
Adaptive Rate Limiting
Brocade Fabric OS Extension Conguration Guide, 7.4.x
53-1003507-09 19
• 10G (1/10 GbE) ports on the Brocade 7840 switch (ge2-ge17).
• 40 GbE ports on the Brocade 7840 switch (ge0 and ge1).
• 1 GbE ports on the Brocade 7800 switch (ge0-ge5) and Brocade FX8-24 blade (ge0-ge9).
ARL applies a minimum and maximum trac rate, and allows the trac demand and WAN connection quality to determine the rate
dynamically. If trac is owing error-free over the WAN, the rate grows towards the maximum rate. If TCP reports an increase in
retransmissions, the rate reduces towards the minimum. ARL never attempts to exceed the maximum congured value and reserves at
least the minimum congured value.
For ARL limitations and features specic to supported products, refer to the Extension Features on Brocade Extension Switches and
Blades on page 31 chapter.
Brocade 7840 Switch support for ARL
ARL on the Brocade 7840 has been enhanced to react ten times faster to varying
trac patterns that compete for WAN bandwidth or
use shared interfaces.
ARL has always accommodated shared bandwidth; however, the amount of storage data using Extension connections continues to grow
and consume larger and faster links. On the Brocade 7840, the enhanced response time of ARL provides faster rate limiting adaptation,
which permits optimized throughput of not only Extension trac, but also the competing ows.
The back-o mechanism implemented by ARL is optimized to increase overall throughput. ARL dynamically preserves bandwidth and
evaluates network conditions to see whether additional back-os are required.
ARL maintains Round Trip Time (RTT) stateful information to better predict network conditions and to allow intelligent and granular
decisions about proper adaptive rate limiting. When ARL encounters a network error, it looks back at prior stateful information, which will
be dierent relative to the current state. Rate limit decisions are then modied using the ARL algorithm. When congured for automatic
selection, ARL will dynamically determine which algorithm to use based on the changing network conditions.
FSPF link cost calculation when ARL is used
Fabric Shortest Path First (FSPF) is a link state path selection protocol that directs
trac along the shortest path between the source and
destination based upon the link cost. When ARL is used, the link cost is equal to the sum of the maximum trac rates of all established,
currently active low metric circuits in the tunnel. The following formulas are used:
• If the bandwidth is greater than or equal to 2 Gbps, the link cost is 500.
• If the bandwidth is less than 2 Gbps, but greater than or equal to 1 Gbps, the link cost is 1,000,000 divided by the bandwidth in
Mbps.
• If the bandwidth is less than 1 Gbps, the link cost is 2000 minus the bandwidth in Mbps.
When running multiple parallel tunnels, set the static link costs if you want all tunnels used at all times. Otherwise, a circuit outage can
take a tunnel out of service because of a link cost change. If multiple parallel tunnels are used, congure Lossless DLS. This avoids circuit
bounce because of disruptive bandwidth updates.
Conguring ARL
To congure the minimum and maximum committed rates for ARL on a circuit, refer to Tunnel conguration options on page 80.
Adaptive Rate Limiting
Brocade Fabric OS Extension Conguration Guide, 7.4.x
20 53-1003507-09
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
Page is loading ...
/