EMC WHITE PAPER
ATTO TECHNOLOGY 10 GBE ADAPTER
CONFIGURATION GUIDE
Using the ATTO FastFrame™ and ThunderLink™
Ethernet Adapters with EMC
®
Isilon
®
OneFS
®
ABSTRACT
This white paper reviews the integration requirements for the ATTO FastFrame and
ThunderLink 10 GbE network adapters when designing media and entertainment
workflows for EMC Isilon scale-out NAS.
September 2015
2
To learn more about how EMC products, services, and solutions can help solve your business and IT challenges, contact
your local
representative or authorized reseller, visit www.emc.com, or explore and compare products in the EMC Store
Copyright © 2015 EMC Corporation. All Rights Reserved.
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without
notice.
The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with
respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a
particular purpose.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.
All trademarks used herein are the property of their respective owners.
Part Number H10643.1
3
TABLE OF CONTENTS
EXECUTIVE SUMMARY .............................................................................. 4
Solution overview ............................................................................................. 4
GUIDELINES ............................................................................................. 4
CLIENT CONFIGURATION ......................................................................... 4
Mac OS X Performance Tuning ............................................................................ 5
Windows 8/2012 Performance Tuning .................................................................. 6
Linux Client Performance Tuning ......................................................................... 6
NETWORK DESIGN CONSIDERATIONS ...................................................... 7
PERFORMANCE ......................................................................................... 7
CONCLUSION ............................................................................................ 8
4
EXECUTIVE SUMMARY
The ATTO Technology FastFrame NS™ and ThunderLink™ 10 Gigabit Ethernet (GbE) network adapters are an ideal complement to
media and entertainment workflows featuring the EMC Isilon family of Network Attached Storage (NAS) devices. EMC Isilon scale-
out NAS is easily deployed into existing media and entertainment workflows utilizing standard file-sharing protocols over existing
1 GbE and 10 GbE network infrastructure. The ATTO FastFrame Ethernet network adapter provides high-performance 10 GbE
connectivity for Mac OS X, Windows, and Linux clients using standard file-sharing protocols.
Broadcasters, motion picture studios, and media professionals world-wide trust EMC Isilon to provide the storage foundation for their
media workflows. The Isilon X-series hardware platforms are the building blocks of a single scale-out NAS volume. EMC Isilon
customers leverage the power of a single OneFS volume to consolidate the storage requirements for editing, media asset
management, transcoding, and other workflows. A single EMC Isilon OneFS volume eliminates the need for costly silos of
proprietary, application-specific storage at each stage of the workflow. EMC Isilon clients use open protocols such as SMB, NFS, FTP,
and HTTP to access a OneFS volume, eliminating the need for costly per-client file-system licenses.
Media and entertainment workflows require flexible tools that protect your storage investment. EMC and ATTO Technology provide
flexible tools based on open protocols. A workflow based on open network file-sharing protocols and the effortless scalability of EMC
Isilon OneFS allows media and entertainment professionals to rapidly adapt to changing production requirements, ever-increasing
content storage requirements, and increasing complicated workflows. EMC Isilon easily adapts to your needs, without introducing
costly down-time to your tight production schedules.
SOLUTION OVERVIEW
The ATTO Technology FastFrame NS 10 GbE adapter is your gateway to high-performance EMC Isilon media storage. Properly tuned
10 GbE networks allow EMC Isilon users the ease of use of a shared network volume and throughput previously restricted to a
complicated Fibre Channel Storage Area Network (SAN). When combined with a low-latency 10 GbE switch such as the Arista 7050
series and EMC Isilon scale-out NAS, the ATTO Technology 10 GbE network adapters provides reliable performance for some of the
most demanding media and entertainment workflows.
The ATTO Technology FastFrame NS is a PCIe 2.0 compliant network interface card optimized for the high-performance access to
media storage workflows on a local area network. ATTO’s proprietary Advanced Data Streaming (ADS™) technology manages
network latency to provide high-performance media file transfers and reliable playback of streaming media.
The ATTO Technology ThunderLink NS interface builds on the success of the FastFrame network adapter by providing an external,
Thunderbolt 2 enabled interface for clients lacking PCIe expansion slots.
GUIDELINES
Client Configuration
For best performance, install the ATTO Technology FastFrame NS 10 GbE network adapter in a PCIe 2.0 slot that is electrically wired
for an 8x lane width or greater. Please consult your server or workstation documentation for additional information regarding the
host computer PCI slot configuration. The table below demonstrates the bandwidth limitations of PCIe 1.0 versus 2.0 at varying lane
widths.
Table 1 PCIe Bandwidth
LANES
BANDWIDTH PER DIRECT
ION PCIE 1.0
BANDWIDTH PER DIRECTION PCIE 2.0
1
250 MB/s
500 MB/s
2
500 MB/s
1 GB/s
4
1 GB/s
2 GB/s
8
2 GB/s
4 GB/s
The ATTO ThunderLink should be connected to a Thunderbolt 2 enabled port. For best performance when connecting the
ThunderLink adapter to a Mac Pro, ensure that the device is connected to a Thunderbolt port that is not sharing a bus with another
5
Thunderbolt device. Figure 1 shows the shared ports of the Apple Mac Pro. Note that bus 0 shares a port with the Mac Pro HDMI
connection. The 2 lower ports on Bus 0 should be avoided if a display is connected to the Mac Pro via the built-in HDMI interface.
Figure 1. Thunderbolt 2 Bus Configuration of Apple Mac Pro
Mac OS X Performance Tuning
Ensure that your clients are running current OS patch levels and the latest version of the ATTO Technology 10 GbE drivers.
For optimal 10 GbE performance, the following entries are added to the file /etc/sysctl.conf on clients running Linux or Mac OS 10.9
and greater.
net.inet.tcp.sendspace=1048576
net.inet.tcp.recvspace=1048576
net.inet.tcp.delayed_ack=3
net.inet.tcp.rfc1323=1
The net.inet.tcp.sendspace and net.inet.tcp.recvspace system control values determine the client OS transmit and receive
buffer sizes. The minimum recommended transmit and receive buffer sizes for a client’s network connectivity to EMC Isilon scale-out
NAS may be determined using the following equation.
Network bandwidth (bytes per second) x round-trip delay (seconds) = buffer size (bytes)
The net.inet.tcp.delayed_ack system control determines when TCP delayed acknowledgment is enabled. 0 will disable the
feature and responds to every packet. A setting of 1 implies that an ack will be sent for every 6 packets. A setting of 2 implies an
ack after every 2 packets (compatibility mode) which may be desirable in certain network topologies. The setting of 3, automatically
determine when to employ a delayed ack, is recommended in most cases.
The net.inet.tcp. rfc1323 system control should be set to 1 to enable window scaling
Jumbo frames support for the ATTO Technology 10 GbE network adapter is enabled via the Mac OS X network preferences pane.
Please see the “Network design considerations”
For more information on Mac OS X network performance tuning, please see the EMC Isilon document
Using Mac OS X Clients with
EMC Isilon OneFS 7.x.
6
Windows 8/2012 Performance Tuning
Little performance running is required when using the recommended SMB3 protocol to connect Windows 8/2012 clients to Isilon –
The SMB stack is largely auto-tuning. In the advanced settings tab of the Windows driver properties, select the Performance Options
settings. The Transmit and Receive Buffer sizes may be increased from 512 to 4,096 bytes, assuming the computer is appropriately
equipped with at least 16-32 GB of RAM to handle bandwidth intensive applications. In some cases, the Interrupt Moderation Rate
may need to be set to “Disabled”.
Figure 2. Windows Driver Performance Options
The ATTO network interface has Receive Side Scaling (RSS) enabled, making it an ideal client interface for the SMB3 Multichannel
implementation in EMC Isilon OneFS version 7.1.1 and greater. SMB3 multichannel will perform single stream load balancing and
fault tolerance over multiple 10 GbE interfaces, assuming you have a 2 or 4 port enabled network interface. SMB3 multichannel load
balancing is completely independent of hardware load balancing protocols such as Link Aggregation Control Protocol (LACP) and each
ATTO network interface should be configured independently with its own IP address.
SMB3 multichannel is enabled by default in OneFS version 7.1.1 and later, so you simply need to ensure that you are connecting
your Windows 8/2012 client to an Isilon node with dual 10 GbE interfaces configured for independent static IP addresses. SMB3
multichannel will automatically initiate a multi-10 GbE interface connection to the cluster once both 10 GbE interfaces are discovered
by the SMB3 multichannel capable client.
For optimal performance in media and entertainment workflows playing back 2K or 4K DPX or EXR image sequences, SMB3
multichannel may be combined with the EMC Isilon File Name Prefetch feature – enabled read speeds up to 1.4 GB/s to a single
Windows 8/2012 client from an X410 series cluster. For more information on the File Name Prefetch feature and accelerating the
RAM prefetch of sequentially named files, please see EMC Isilon KB article 202614
.
Linux Client Performance Tuning
The ATTO FastFrame NIC should be natively supported by the kernel of most modern Linux distributions. For recommended NFS
mount parameters when using Linux with Isilon, please see EMC Isilon KB article 90041,
OneFS: Best practices for NFS client
settings
7
NETWORK DESIGN CONSIDERATIONS
EMC Isilon scale-out NAS clients demanding the highest possible 10 gigabit Ethernet performance use two physically isolated
networks, as shown below in the example in Figure 3.
Figure 3. Dual homed network for Jumbo Frames Support
The primary network in this design is a standard gigabit Ethernet network, shown at the bottom of figure 1. The gigabit Ethernet
network is common to all devices on the LAN using each device’s on-board gigabit Ethernet port(s). All devices on the gigabit LAN
segment utilize a standard Maximum Transmission Unit (MTU) of 1500 bytes per packet. The gigabit Ethernet network is a layer 3
network and each host on the network is configured with a default gateway address. Clients on this network with ATTO Technology
FastFrame NS 10 gigabit Ethernet network adapters installed are configured for a dual-homed network configuration. The 10 gigabit
Ethernet network shown at the top of figure 1 is common to only 10 gigabit enabled clients.
Unlike the gigabit Ethernet network, the 10 gigabit Ethernet network is a layer 2 Ethernet switch configured to pass jumbo frames or
packets with an MTU size of 9000. Each host interface on the 10 gigabit Ethernet network is configured with an MTU of 9000. The
10 gigabit switch supporting this network is configured with an MTU of 9216 bytes in order to prevent the re-transmission of packets
exceeding 9000 bytes. Every client on the 10 gigabit LAN and every EMC Isilon 10 gigabit Ethernet interface is configured with an
MTU of exactly 9000 bytes per packet in order to prevent re-transmission of packets due to fragmentation.
EMC Isilon scale-out NAS clients utilizing a jumbo frame enabled LAN segment keep all interfaces using an MTU of 9000 bytes per
packet isolated to a dedicated jumbo frames network. The only interface that is allowed to route traffic to other subnets is the
gigabit Ethernet interface configured with an MTU of 1500 bytes per packet. The MTU 1500 interface is also configured as the
primary interface on each of the client operating systems. A “jumbo frames” enabled 10 gigabit Ethernet network may improve
workflow performance by more efficiently packetizing data for network file transfers.
EMC Isilon scale-out NAS customers wishing to simplify their network infrastructure configure their 10 gigabit Ethernet switches in a
layer 3/4 topology with an MTU of 1500 bytes per packet. While increasing MTU may yield a worthwhile performance increase in
some environments, the added complexity of dual-homing each client may negate the benefits of simplicity in a NAS architecture.
Careful consideration must be taken before enabling jumbo frame sizes, as MTU sizes must never be mixed on the same network
segment.
For optimal performance, 10 gigabit Ethernet clients are connected to the same switch as the EMC Isilon 10 gigabit Ethernet
connections, as each “hop” on a network of tiered Ethernet switches adds latency to file transfers.
PERFORMANCE
When validating performance for a 10 GbE enabled client, it is important to first set a baseline to determine if your performance is
being limited by a physical network or driver issue. To validate you will be able to saturate a network connection, make sure you run
an iperf test to read from each client network interface to each Isilon node network interface and write from each client network
interface to each Isilon node network interface. Iperf an open source tool for performing network performance measurement. Iperf
is pre-installed on every Isilon cluster. Windows binaries Linux install packages for iperf are available at http://iperf.sourceforge.net
.
To install iperf on Mac OS X, first install the Brew package manager from the Mac OS X command line interface:
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
8
After the Brew install, install iperf:
brew install iperf
To test upload speed from your client to an Isilon interface, run the following commands from the command line interface of the
Isilon node and client shell respectively:
isilon_node# iperf -s -w 2M -l 1M -N
client_shell# iperf -c <node 10G IP> -w 2M -l 1M -N -t 30
Run the command to test every combination of client network interface and Isilon node/network interface.
To test the download speed from your client to an Isilon interface, run the following commands from the command line interface of
the Isilon node and client shell respectively:
isilon_node# iperf -c <client 10G IP> -w 2M -l 1M -N -t 30
client_shell# iperf -s -w 2M -l 1M -N
Again, ensure that this test is run from every node in the cluster. This test provides a baseline for your potential network throughput
without the added bottlenecks of file sharing protocol overhead and disk input/output.
Expect at least 7-9 GB/s throughput in the results of your iperf tests. Real-world performance is highly dependent upon your
workflow, the Isilon node type, and your client configuration. In Figure 4, we see the results from the Isilon isi statistics
client command when comparing a Mac OS 10.11 client and a Windows 8.1 client.
Figure 4. Example Mac OS X and Windows 8 SMB3 performance
In Figure 4, we see the read and write performance results of an Apple Mac Pro equipped with a ThunderLink NS2102 and a Lenovo
D30 equipped with a FastFrame NS12. Both client computers are equipped with 32 GB of RAM and the performance tuning options
detailed in this document. The Isilon OneFS v7.2.1 cluster used in this test is a 3 node X410 cluster with SSD metadata read
acceleration and 64 GB or RAM per node. The read tests are the result of the Mac and Windows client simultaneously playing back a
4K DPX sequence in the BlackMagic Design Davinci Resolve 12 application. The fnprefetch data access pattern detailed in
EMC Isilon
KB article 202614 is applied recursively to the folder containing the DPX sequence. As you can see from the results, the Windows
8.1 client gains a significant performance advantage thanks to SMB3 Multichannel and the ability to balance traffic over dual 10GbE
interfaces.
The write performance results are generated from the simultaneous copy of identical folders containing a mix of large media files to
the Isilon cluster.
CONCLUSION
A network featuring ATTO 10GbE connectivity offers EMC Isilon users:
• An easily expandable and managed storage solution that can simplify many of today's complicated workflows
• High-speed 10GbE interconnects for collaborative workflows that include Windows, Linux, and Mac OS X
• An economical, simple-to-administer 10GbE solution for real-time access to shared volumes
• High performance and flexibility for scenarios such as remote editing and file access
• The ability to connect Thunderbolt-enabled laptops and all-in-one platforms to EMC Isilon storage
Media and entertainment workflows require high-performance storage solutions along with the ability to rapidly add new users and
content. A workflow based on open network file-sharing protocols and the effortless scalability of EMC Isilon OneFS allows media and
entertainment professionals to rapidly adapt to changing production requirements, ever-increasing content storage requirements,
and increasingly complicated workflows. EMC and ATTO Technology provide flexible tools based on open protocols that easily adapt
to your needs, while reliably supporting your tight production schedules.
/