Network Traffic
Engineering
Guidelines for Fibre
Channel Switches
Optimizing Performance When Designing Highly
Scalable SAN Solutions:
Guidelines for SAN Architects
Executive Summary
Before the advent of stackable switch solutions, storage area
network (SAN) architects had to trade scalability vs.
performance. Data traffic across the network was limited to
relatively few inter-switch links (ISL).
QLogic stackable switch solutions solve the
scalability/performance dilemma. Each model in the
SANbox
®
5000 series features a 4 pack of high-speed
(10Gb) stacking ports that enable a SAN architect to scale
seamlessly and maintain high performance. Connecting
switches together using the dedicated stacking trunks
preserves all 4Gb ports for use by servers and storage
devices.
QLogic’s highly scalable SAN solutions allow your network
traffic engineer to maximize overall performance and to minimize cost.
Key Findings
By following simple network engineering guidelines, SAN switch solutions can be created to balance
scalability and performance:
• Rule A: Localized Traffic. Connect critical server and storage devices connected to a single switch
to maximize throughput and minimize latency.
• Rule B: Remote/ISL Traffic. Avoid oversubscription by utilizing high bandwidth ISLs/trunks and
placing servers and storage across switches so the I/O operations load for remote traffic is less
than or equal to the capacity of inter-switch trunks. The QLogic SANbox 5200, 5600, and 5602
stackable switches also preserve 2Gb/4Gb ports for connections to server and storage devices.
A stackable solution and topology using SANbox 5000 series switches balance overall performance
and cost; whereas fixed-port solutions increase in cost faster and deliver less bandwidth between
switches.
2 Optimizing Performance When Designing Highly Scalable SAN Solutions:
Guidelines for SAN Architects
SPG-WP06004 SN0130924-00 Rev A
Scalability vs. Performance
This white paper provides guidelines for SAN architects to optimize performance when
designing highly scalable SAN solutions. Although this white paper focuses on Fibre
Channel switches, its concepts also apply to technology products like Ethernet switches
and will be familiar to technical staff in that field.
SAN solutions continue to grow in size and complexity to meet the ever-increasing
demands for storage capacity. Before the advent of stackable switch solutions, SAN
architects were faced with the design tradeoff of scalability versus performance.
Then: The Dilemma of Scalability vs. Performance
To provide additional network capacity, SAN administrators connected multiple fixed-port
switches together. A minimum number of inter-switch links (ISL) was used to preserve
switch ports for servers and storage. However, this approach reduced overall performance
because the data traffic across the network was limited to these few ISL connections.
An obvious solution would be to add further ISL connections between the switches to
support more data bandwidth, but this reduces the number of ports available to attach
servers and storage. This is the classic scalability versus performance dilemma with fixed-
port switch solutions.
Now: Stackable Switches Allow Both Scalability and Performance
Stackable switch solutions from QLogic were designed specifically to solve the
scalability/performance dilemma. Each model in the SANbox
®
5000 series of stackable
switches features a 4 pack of high-speed (10Gb) stacking ports that enable a SAN architect
to scale seamlessly and maintain high performance. Connecting switches together using
the dedicated stacking trunks preserves all 4Gb ports for use by servers and storage
devices.
Network Traffic Engineering
Network traffic engineering is the process of balancing the delivered bandwidth with the
cost of the overall solution. The key factor in this equation is architecture design because
this choice drives the following:
• Number of devices required to scale the solution
• Cost of interconnecting the devices
There are many architecture and topology choices available using stackable and fixed-port
switches, including cascade, partial mesh, full mesh, etc. As these solutions grow, the
network traffic balance must be engineered to maximize the overall performance and
minimize the cost.
3 Optimizing Performance When Designing Highly Scalable SAN Solutions:
Guidelines for SAN Architects
SPG-WP06004 SN0130924-00 Rev A
Network Traffic Engineering Guidelines
To maintain balanced performance across a SAN, the ratio of local to remote traffic on
switches must be considered.
Local and remote traffic are defined as follows:
• Local traffic
I/O operations between servers and storage located on the same switch.
• Remote traffic
I/O operations between servers and storage located on different switches.
For example, the following drawing depicts local data traffic over the hardware path from
servers to storage.
The following drawing depicts remote data traffic over the hardware path from servers to
storage.
Inter-switch
trunk (ISL)
4 Optimizing Performance When Designing Highly Scalable SAN Solutions:
Guidelines for SAN Architects
SPG-WP06004 SN0130924-00 Rev A
As switches are added to the network, it is important to place servers and storage
according to the following rules such that:
• Critical, high performance I/O operations between specific servers and
storage are connected locally (to/from the same switch).
• Total I/O operations load for remote traffic is less than or equal to the
capacity of inter-switch trunks. Oversubscription results if remote traffic
exceeds the trunk capacity.
Examples of Traffic Engineering Rules
This section gives examples of Rule A, for localized traffic, and Rule B, for remote/ISL
traffic, for stackable switch and traditional, fixed-port solutions.
Example of Rule A: Localized Traffic for Critical, High-Bandwidth I/O Operations
Following Rule A, for critical I/O operations, the respective server and storage devices
should be attached locally to a single switch, as shown below. This rule applies to all
switches (fixed or stackable). In this example, four QLogic SANbox 5602 stackable
switches are deployed to create a 64-port SAN.
Local connection of critical server and storage devices ensures maximum
performance by processing all I/O within a single on-board ASIC. In this manner,
data traffic is switched locally with minimum latency. Although inter-switch trunks
incur additional hops, this does not affect storage performance in most real-world
applications.
RULE A
RULE B
5 Optimizing Performance When Designing Highly Scalable SAN Solutions:
Guidelines for SAN Architects
SPG-WP06004 SN0130924-00 Rev A
Examples of Rule B: Remote/ISL Traffic
Per Rule B, the total amount of remote traffic must be less than or equal to the capacity of
the ISLs/trunks to avoid oversubscription. Therefore, network traffic must be balanced
across the SAN to ensure maximum performance using the ISLs.
Maximum ISL Traffic Load for 4Gb Stackable Switch
In the example shown below, using a SANbox 5602 stackable 4Gb switch, the 10Gb trunks
(actual bandwidth capability is 12Gb/sec) can support about 75% of the theoretical traffic
load to/from the attached server and storage devices. Due to the high capacity of these
ISLs/trunks, stackable solutions allow for scaling up to large port counts while
maintaining high performance.
ISLs support 75% of total
theoretical data traffic
(from attached server and
storage devices)
• Max theoretical bandwidth from devices is:
16 ports x 4Gb/sec speed x 2 (full duplex) = 128 Gb/sec
• Max theoretical bandwidth of ISLs/trunks is:
4 ports x 12Gb/sec x 2 (full duplex) = 96 Gb/sec
• ISL capacity (96 Gb/sec) is 75% of theoretical device bandwidth (128 Gb/sec).
Traffic Calculation
Traffic To “Remote”
Devices over ISLs
Server and Storage
Devices
Traffic To “Local”
Devices
6 Optimizing Performance When Designing Highly Scalable SAN Solutions:
Guidelines for SAN Architects
SPG-WP06004 SN0130924-00 Rev A
Maximum ISL Traffic Load for 16-Port Fixed Configuration Switch
The following drawing shows a fixed-port switch with 16 ports. To preserve ports for
devices, typically no more than two ports are used for ISLs to other switches. These ISLs
can support only about 14% of the theoretical traffic load of each switch
• Max theoretical data traffic from devices is:
14 ports x 4Gb/sec x 2 (full duplex) = 112 Gb/sec
• Max theoretical capacity of ISLs/trunks is:
2 ports x 4 Gb/sec x 2 (full duplex) = 16Gb
• ISL capacity (16 Gb/sec) is about 14% of total theoretical
data traffic from attached devices (112 Gb/sec).
Data Traffic Calculation:
Server and
Storage Devices
Data Traffic To “Remote”
Devices
Over ISLs
Data Traffic To
“Local” Devices
ISLs support 14% of total
theoretical data traffic
(from attached server and
storage devices)
7 Optimizing Performance When Designing Highly Scalable SAN Solutions:
Guidelines for SAN Architects
SPG-WP06004 SN0130924-00 Rev A
Maximum ISL Traffic Load for 32-Port Fixed Configuration Switch
The following drawing shows a fixed-configuration switch with 32 ports. Again, to preserve
ports for devices, typically no more than two ports are used for ISLs to other switches.
These ISLs can support only about 7% of the theoretical traffic load of each switch.
So by comparison, fixed-port switch solutions provide lower overall performance as
they scale due to the minimal number of ISL trunks available.
As discussed earlier, the obvious solution of adding more ISL links reduces the number
of ports available for server and storage devices, and therefore is counter-productive.
• Max theoretical data traffic from devices is:
30 ports x 4 Gb/sec x 2 (full duplex) = 240Gb/sec
• Max theoretical capacity of ISLs/trunks is:
2 ports x 4 Gb/sec x 2 (full duplex) = 16 Gb
• ISL capacity (16 Gb/sec) is about 7% of total theoretical
data traffic from attached devices (240Gb/sec).
Data Traffic Calculation:
Serve
r
And
Storage Devices
Data Traffic To “Remote”
Devices Over ISLs
Data Traffic to
“Local” Devices
ISLs support 7% of total
theoretical data traffic
(from attached server and
storage devices)
8 Optimizing Performance When Designing Highly Scalable SAN Solutions:
Guidelines for SAN Architects
SPG-WP06004 SN0130924-00 Rev A
Examples of Stackable Switch Configurations
The examples in this section are specific to QLogic stackable switches, like the SANbox
5200, 5600, and 5602. These examples are intended as guidelines to optimize scalability
and performance when choosing a topology design.
In general, ideal best practices when connecting switches via inter-switch links or trunks
include the following:
• A minimum of one link between every pair of switches to maximize switch-to-switch
data throughput and to minimize network hops (latency).
• A minimum of two paths between every pair of switches to provide fault tolerance in the
event of an ISL failure or forced removal.
• Additional ISLs can be added (where possible) to increase the remote traffic capability
between critical switches.
The following tables illustrate these guidelines.
9 Optimizing Performance When Designing Highly Scalable SAN Solutions:
Guidelines for SAN Architects
SPG-WP06004 SN0130924-00 Rev A
Baseline Configurations
The following table shows the topology summary of three baseline configurations.
Total
Switches
Total 2Gb/4Gb
Device Ports
Available
Total
10Gb
ISLs
Topology Summary
2 32 2
3 48 3
4 64 6
10 Optimizing Performance When Designing Highly Scalable SAN Solutions:
Guidelines for SAN Architects
SPG-WP06004 SN0130924-00 Rev A
Maximum Performance Configurations
By adding additional ISL trunks, higher remote traffic bandwidth can be supported between
switches, as indicated in the following table.
Total
Switches
Total 2Gb/4Gb
Device Ports
Available
Total
10Gb
ISLs
Topology Summary
2 32 4
3 48 6
4 64 8
11 Optimizing Performance When Designing Highly Scalable SAN Solutions:
Guidelines for SAN Architects
SPG-WP06004 SN0130924-00 Rev A
Conclusions
Network Engineering Guidelines
By following simple network engineering guidelines, SAN switch solutions can be created to
provide a balance between scalability and performance.
• Rule A: Localized Traffic. Connect critical server and storage devices connected to a
single switch to maximize throughput and minimize latency.
• Rule B: Remote/ISL Traffic. Avoid oversubscription by utilizing high bandwidth
ISLs/trunks and placing servers and storage across switches so the I/O operations load
for remote traffic is less than or equal to the capacity of inter-switch trunks. The QLogic
SANbox 5200, 5600, and 5602 stackable switches also preserve 2Gb/4Gb ports for
connections to server and storage devices.
Switch Architecture Choice
Architecture choice is a key factor in the network engineering equation. Per the guidelines
in the previous section, a stackable solution and topology using SANbox 5000 series
switches balance overall performance and cost; whereas fixed-port solutions increase in
cost faster and deliver less bandwidth between switches.
Small Large
Total Bandwidth
Solution
Cost
Stackable
Architecture
Fixed Port
Architecture
Small Large
No. of Devices in Solution
Average ISL
Bandwidth
Stackable
Architecture
Fixed Port
Architecture
For more information on QLogic and stackable switch solutions, visit www.qlogic.com
.
Stackable Solution:
Linear
cost as bandwidth/solution
scales.
Fixed Port Solution:
Rapidly increasing
cost as
bandwidth/solution scales.
Stackable Solution:
Consistent
ISL bandwidth as
solution scales.
Fixed Port Solution:
Decreasing
ISL bandwidth as
solution scales.
/