PXH840 User’s Guide – Dolphin Interconnect Solutions Page 1
PCI Express PXH840 Adapter
PXH840 NTB and Transparent PCIe Adapter users guide
Version 1.0
Date: 19
th
June 2019
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 2
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 3
Table of Contents
DISCLAIMER ................................................................................................................................................................ 5
TERMS AND ACRONYMS ............................................................................................................................................. 5
PXH840 HIGH LEVEL SPECIFICATION ........................................................................................................................... 6
PRODUCT VARIANTS .............................................................................................................................................................. 6
PACKAGING ................................................................................................................................................................ 7
PRE-INSTALLATION QUESTIONS .................................................................................................................................. 7
PCIE SLOT DETERMINATION ................................................................................................................................................... 7
PXH840 CONFIGURATION..................................................................................................................................................... 7
OPERATING ENVIRONMENT .................................................................................................................................................... 7
CABLE CONNECTIONS ............................................................................................................................................................ 7
TRANSPARENT OPERATION. .................................................................................................................................................... 7
INSTALLATION ............................................................................................................................................................ 8
STEP 1 - UNPACK BOARD ....................................................................................................................................................... 8
STEP 3 - CONFIGURE THE BOARD FOR PROPER OPERATION ............................................................................................................ 8
NTB Configuration settings .......................................................................................................................................... 8
STEP 4 - INSTALL THE ADAPTER CARD ....................................................................................................................................... 8
STEP 5 - INSTALLING AND REMOVING CABLES ............................................................................................................................ 8
Connecting Cables ....................................................................................................................................................... 8
Disconnecting Cables ................................................................................................................................................... 8
STEP 6 – INSTALLING THE DOLPHIN EXPRESSWARE SOFTWARE ..................................................................................................... 9
STEP 5 - VERIFY INSTALLATION ................................................................................................................................................ 9
OPERATION AND MAINTENANCE ..............................................................................................................................10
CONFIGURATION AND DIP SWITCHES ..................................................................................................................................... 10
DIP Switch Bank – Configuration ............................................................................................................................... 10
NTB USE CASES SUMMARY AND SETTINGS ................................................................................................................11
USE CASE A - 2 NODE CONFIGURATION – NTB MODE .............................................................................................................. 11
Connecting the cables for 2 Node Configurations, x16 link ....................................................................................... 11
Connecting the cables for 2 Node Configurations, x8 link ......................................................................................... 11
Connecting the cables for 2 Node Configurations, x4 link ......................................................................................... 11
USE CASE B - 3 NODE CONFIGURATION – NTB MODE .............................................................................................................. 12
Connecting the cables for 3 Node Configurations, x8 link ......................................................................................... 12
TRANSPARENT USE CASES .........................................................................................................................................13
FIRMWARE UPGRADE .......................................................................................................................................................... 13
IDENTIFYING THE CARD ........................................................................................................................................................ 13
SUPPORT .......................................................................................................................................................................... 14
TECHNICAL INFORMATION ........................................................................................................................................15
PENDING COMPLIANCE AND REGULATORY TESTING .................................................................................................16
LIMITED WARRANTY..................................................................................................................................................17
WARRANTY PERIOD ............................................................................................................................................................ 17
COVERAGE ........................................................................................................................................................................ 17
SERVICE PROCEDURE .......................................................................................................................................................... 17
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 4
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 5
DISCLAIMER
DOLPHIN INTERCONNECT SOLUTIONS RESERVES THE RIGHT TO MAKE CHANGES WITHOUT FURTHER NOTICE TO ANY OF ITS
PRODUCTS TO IMPROVE RELIABILITY, FUNCTION, OR DESIGN.
TO THE FULLEST EXTENT PERMITTED BY LAW, DOLPHIN WILL NOT BE LIABLE FOR ANY INDIRECT, INCIDENTAL, SPECIAL OR
CONSEQUENTIAL DAMAGES (INCLUDING LOST PROFITS, LOST DATA, OR LOSS OF USE) ARISING OUT OF ANY USE OF DOLPHIN’S
PRODUCTS, SOFTWARE OR SERVICE PROVIDED. DOLPHIN’S MAXIMUM LIABILITY WILL NOT EXCEED THE TOTAL AMOUNT PAID FOR
THE PRODUCT BY THE PURCHASER.
LIFE SUPPORT POLICY
DOLPHIN INTERCONNECT SOLUTIONS’ PRODUCTS ARE NOT AUTHORIZED FOR USE AS CRITICAL COMPONENTS IN LIFE SUPPORT
DEVICES.
ENVIRONMENTAL POLICY
Dolphin is minimizing the amount of printed documentation and software CDs in its shipments; please download additional
documentation and software from www.dolphinics.com.
Terms and acronyms
Important terms and acronyms used in this manual
AOC Active Optical Cable. PCIe fiber cable assembly.
eXpressWare Dolphins software stack for PCIe clustering and IO. www.dolphinics.com for more information.
FireFly Optical fiber solution for PCIe over cable from Samtec Inc
Lane One PCI Express Lane contains a differential pair for transmission and a differential pair for reception.
Link A collection of one or more PCI Express Lanes providing the communication path between an Upstream and
Downstream Port.
Port The PXH840 has two MTP connectors providing four PCIe x4 ports, named P1, P2. The physical ports are
identified by text on the PCIe brackets. The PCIe chip only supports 2 NTB ports.
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 6
PXH840 high level specification
The PXH840 is a PCI Express Gen3 x16 adapter card supporting both NTB and Transparent operations. The card comes with
firmware and a license to use the eXpressWare PCIe networking software from Dolphin Interconnect Solutions.
Product variants
The card can be ordered with different FireFly optical engines mounted depending on your use case and requirements:
SKU / Model
#FireFly
Config
PXH840
2 x8
NTB x16 / NTB 2 x8
PXH841
1 x8
NTB x8
PXH843
2 x4
NTB x8 / NTB 2 x4
PXH844
1 x4
NTB x4
The PXH840 supports PCIe Gen1, Gen2 and Gen3 speeds and x4, x8 and x16 link-widths. The card will operate at the highest
common speed shared between the slot and the card (Gen3) and the widest common link-width (x16).
• PCI Express Base Specification, Rev. 3.0.
• PCI Express CEM Specification, Rev. 3.0.
• PCI Express Gen3 8.0 GT/s per lane signaling – 128 GT/s total signaling.
• PCI Express Gen3 x16 edge connector. The card installs in any PCI Express slot that has a physical x16
connector. The slot may have any electrical width x1 – x16.
• Compliant with PCI Express Gen1 through Gen3 computers and IO systems, auto detection.
• MTP cable connector
• The PXH840 in NTB mode supports to two or three other nodes.
• NTB Cable port configurations, up to
o One x16
o Two x8
• Broadcom / Avago / PLX PEX8733 PCI Express Gen3 chipset.
• 132 nanosecond cut-through latency port to port.
• Support for active optical fibers up to 100 meters.
• Low profile, Half length - PCI Express Electromechanical Specification, Rev 2.0.
• Dimensions 167.8mm x 68.9 mm
• Comes with both low profile and standard profile PCI Express bracket.
• Host clock isolation. Automatic support for host running CFC or SSC mode.
• VAUX powered board management controllers for flexible configuration and cable management.
• EEPROM recovery option.
• PCI Express power domain isolation.
• Power consumption:
o 12 Volt: Max 14 Watt with 4 FireFlys attached.
o 3.3 Volt: Max 3.3 Watt
o 3.3 Volt Aux: Max 1 Watt
o Port power (per cable port): 3.3 Volt +/- 5%, 0.6 A
• Operational Environment
o Operating Temperature: 0°C - 55°C (32°F - 131°F),
o Air Flow: 150 LFM
o Relative Humidity: 5% - 95% (non- condensing)
• Regulatory
o CE
o Compliant to EN-55022 (2010), EN 55024 (2010), Class A.
o RoHS
o FCC Class A.
o WEEE
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 7
Packaging
The PXH840 includes the following components.
• PXH840 Adapter Board with a full height PCIe bracket mounted
• Anti-static bag
• Getting started guide with serial number for quick download of Dolphins eXpressWare PCIe software.
Pre-Installation Questions
Certain steps should be taken prior to installing the PXH840. You should determine the following configuration requirements.
• Which PCIe slot and system will the card be installed in?
• What is the speed and link width of the slot that the card will be installed in?
• What is the operating environment in which the card will be installed?
• PCIe Transparent or NTB use?
• What width and length of cables will be used?
• Proper operational conditions, temperature, air-flow.
PCIe Slot determination
The PXH840 supports PCIe Gen1, Gen2 and Gen3 speeds and x1, x2, x4, x8 and x16 link-widths. The slot width and speed will affect
the performance of the card. The card can be physically installed in an x4, x8 or x16 connector. The card will auto configure to the
slot speed and width.
• Install the board in a slot connecting directly to the CPU for optimal shared memory performance.
• Install the board behind a PCIe switch in the IO system for optimal PCIe peer to peer performance (e.g. between GPUs,
FPGAs)
TIP: After installing the eXpressWare software, run the “sisci_benchmarks” benchmark suite to verify the expected performance.
Please contact Dolphin support if you need assistance to reach the expected performance.
PXH840 Configuration
The PXH840 has a DIP switch bank to control the main configuration of the card. The DIP switch labeled SW1 can be found close to
the upper edge of the board. The default DIP switch setting is single NTB link x16 operations. Alternative NTB configuration settings
is two x8 links. The PXH840 also supports Transparent configuraitons, the alternatives are one, dual or quad IO Expansion.
Operating environment
To maximize life time for the product and maintain the warranty, please honor the specified operating temperature and make sure
the specified air flow is present. Special care should be considered when PXH840 is used in office type cabinets in combination with
other high energy consuming PCIe devices, e.g. not active cooled GPUs:
Operating Temperature: 0°C - 55°C (32°F - 131°F), Air Flow: 150 LFM
Operating Temperature with AOC: 0°C - 45°C (32°F - 113°F), Air Flow: 150 LFM
Relative Humidity: 5% - 95% (non- condensing)
TIP: After installing the Dolphin eXpressWare software, you can use the tool dis_diag to determine the actual board temperatures.
Cable Connections
The PXH840 is designed for Samtec FireFly optical engines. The card support any length up to 100 meters without any special PCIe
tuning.
Transparent operation.
The PXH840 variants are designed for NTB operation. The PXH842 is a PCI Express Gen3 x16 Transparent Host and Target card for
PCIe IO expansion applications. Please consult the PXH842 users guide for details.
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 8
Installation
Step 1 - Unpack board
The PXH840 card is shipped in an anti-static bag to prevent static electricity damage. The card should only be removed from the
bag after ensuring that anti-static precautions are taken. Static electricity from your clothes or work environment can
damage your PCI Express adapter card or your PC. Always wear a grounded anti-static wrist strap while opening the
PC and when the PXH840 is removed from the anti-static bag.
Step 3 - Configure the board for proper operation
The PXH840 board support both NTB and Transparent use-cases and must be configured using the DIP Switch to meet your specific
use case.
NTB Configuration settings
DIP switch settings for the NTB use cases, two node x16 or three node x8. These settings must be applied to all cards in the PCIe
network.
NTB Configuration PXH840
DIP switch view
1 x16 NTB port
(Shipping Default)
2 x8 NTB ports
OPT3 set
Table 1: PXH840 SW1 NTB configuration settings.
Step 4 - Install the Adapter Card
Before installing the adapter card, make sure you are properly grounded to avoid static discharges that may destroy your computer
or the adapter card. Ensure you are properly grounded before opening your computer or the anti-static bag
containing the PXH840. Please follow your computer’s or expansion chassis manual on how to install a PCI Express
card.
The PXH840 Adapter card can be installed into any PCI Express x16 slot. The PXH840 supports PCI Express Gen1,
Gen2 and Gen3 signaling. NOTE: A Gen3 slot is recommended as it typically doubles the performance compared to a Gen2
slot. The PXH840 is an x16 card, so maximum performance will only be attained if the slot provides full electrical x16
signaling.
The PXH840 supports hosts using either spread spectrum or constant frequency clocking. The card implements clock isolation.
Step 5 - Installing and Removing Cables
Installing and removing cables in NTB mode can be done while the systems are running. Hot plugging / removing cables are fully
supported. In transparent mode, hot plugging and replacing cabels are normally not supported.
Connecting Cables
Please carefully install the fiber patch cable connector into the connector housing on the PXH840 adapter card. To install the cable,
match the cable house color with the MTP connector on the PXH840 adapter card. Use even pressure to insert the connector until
it is secure. Computer cables should always use strain relief to protect the connected equipment from excessive force on the cable.
This is especially important for cables between racks.
Disconnecting Cables
Please carefully pull the release tab to release the cable from the locking latches and gently pull the cable out of the connector
guides.
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 9
Step 6 – Installing the Dolphin eXpressWare software
The PXH840 adapter card in NTB mode is compliant with Dolphin’s eXpressWare software package for the PX product line. Please
visit
http://www.dolphinics.com/px.html
to register and download the latest documentation and software. It is recommended to follow the quick installation guide found
on the web page above to install the software for your operating system.
TIP: Dolphin provides software and documentation for several product families; please remember to select the PX product family
before downloading. The PXH840 requires Dolphin software version DIS 5.5.0 or higher to operate.
The software download requires a password to log in. The password will automatically be emailed to you if you follow the
instructions found on the getting started document bundled with the adapter card. If you fail to provide the correct serial number
found on the getting started document, your request will be managed manually.
Step 5 - Verify Installation
The PXH840 comes with a set of LEDs to verify proper configuration. Table 1: LED below is a chart of the LEDs and their
corresponding meaning.
The card has four bi-color Link-LEDs visible through the PCI Express front bracket.
LED color
Function
Off
No cable installed
Yellow
Cable installed, no link
Green
Link up, PCIe Gen 3 speed
Green blinking
Link up, PCIe Gen 1 or 2 speed
Table 1: LED behavior
Please install the Dolphin eXpressWare software and run the included verification tests as suggested by the software installation
procedure and documentation.
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 10
Operation and Maintenance
Configuration and DIP Switches
The PXH840 has one bank of 8 DIP switches. The default factory setting for the PXH840 is NTB mode, short cable, and single (up to
x16) link connection.
The PXH840 has DIP switches for setting special modes or operations. Please carefully read the documentation before modifying
any DIP switch settings. Please pay close attention to ON and OFF positions written on the DIP switch.
DIP Switch Bank – Configuration
Figure 1: DIP Switch shows the DIP switch for the PXH840. It is used to configure the adapter
card. Please leave all undocumented DIP switches in the default position. The table below
shows all the various DIP switch settings for the PXH840.
Note: Some DIP switch configuration options may be changed in the future versions. Please always consult the latest user guide for
details.
DIP no.
Name
Description
ON
OFF
Default
1
OPT1
Select between NTB and
Transparent operation
Transparent mode
NTB mode
OFF
2
OPT2
Configures a Transparent card to
Host or Target mode. Should be off
for NTB operations
Enable Transparent
Target operation.
Normal NTB operation or
Transparent Host (depends
on DIP 1)
OFF
3
OPT3
Enable dual x8 link configuration
Dual x8 link on cable
ports 1-2 and 3-4
Single x16 link, cable ports
1-4
OFF
4
OPT4
Enable quad link configuration for
Transparent.
Quad x4 link on cable
port 1,2,3,4
OFF
5
OPT5
TBD
TBD
Normal operation
OFF
6
OPT6
TBD
TBD
Normal operation
OFF
7
UTRES
Holds the management processors
in reset
Board management is
held in reset
Normal operation
OFF
8
SAFE
Enables the card to boot if the
EEPROM has been corrupted
Safe EEPROM
Normal operation
OFF
Figure 1: DIP Switch
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 11
NTB Use Cases Summary and Settings
The PXH840 card may be used in the following use cases as summarized in Table 2: Use Case DIP switch settings below.
Use
case
Description
DIP switch setting
A
Two node NTB configuration. Both
systems have PXH840 installed.
OPT2 on if long cables
All other off
B
Three node NTB configuration. All systems
have a PXH840 installed
OPT1 on to enable two x8 links
OPT2 on if long cables
All other off
Table 2: Use Case DIP switch settings
Use Case A - 2 Node Configuration – NTB Mode
Each node has a PXH840 adapter and a direct x4, x8 or x16 cable connection to the other system. This configuration is fully
supported by all Dolphin software, from version DIS 5.5.0 and newer. Both PXH840 adapter cards operate in NTB mode. The DIP-
Switch is set to all off.
Connecting the cables for 2 Node Configurations, x16 link
To establish an x16 link, a given port number should be
connected to the same port number on the other card.
Table 3: Required x16 cabling. Always connect a cable
from Port #x to Port #x
Port,
Host
A
Port,
Host
B
P1
P1
P2
P2
A failure connecting any of the cables will cause the link
to re-train to x8 or x4.
Connecting the cables for 2 Node Configurations, x8 link
To establish an x8 link, please connect a cable from port 1 to port 1 on the other card. If the card is populated with 4 FireFly’s, you
can connect the cable between any port.
Connecting the cables for 2 Node Configurations, x4 link
To establish an x4 link when in dual port mode, please connect port 1 to port 1. If the card is equipped with 4 FireFly’s, you can
connect the cable between any ports.
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 12
Use Case B - 3 Node Configuration – NTB Mode
Each node has a PXH840 adapter and a direct x4 or x8 cable connection is used between all systems. All PXH840 adapter cards
operate in NTB mode. The DIP-Switch OPT3 is ON to enable dual port x8 mode.
Connecting the cables for 3 Node Configurations, x8 link
Connect the cables between two nodes as described above, connecting two nodes x8. Connect the third node to the two other
cards using similar rules. Always connect P1 + P2 to P1 + P2 or P3 + P4
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 13
Transparent Use Cases
The PXH840 is designed for NTB operations. The PXH842 should be used for Transparent use cases. Please reference the PXH842
users guide for more information.
Firmware upgrade
The PXH840 design uses a microcontroller to implement configuration, management and maintenance functions. Dolphin may from
time to time publish updated firmware for the microcontroller or EEPROM data for the card. Please note that standard PLX
firmware tools cannot be used to upgrade the firmware. Please contact Dolphin for more information.
Identifying the Card
The card has a label-sticker with the serial number in the format ‘PXH840-YY-ZZZZZZ’, where YY denotes the card revision (e.g. CC)
and ZZZZZZ denotes the serialized production number (e.g. 012345) – this whole string makes up the serial number of the card (i.e.
PXH840-CC-012345).
You can also get this information using lspci in Linux:
First, identify the devices for the Dolphin Host card:
Than run lspci, and identify the card. It will show up as something like
Second, do
# lspci -s 1:0.0 -v
01:00.0 PCI bridge: PLX Technology, Inc. Device 8733 (rev ca) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 25
Memory at f7d00000 (32-bit, non-prefetchable) [size=256K]
Bus: primary=01, secondary=02, subordinate=03, sec-latency=0
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
Capabilities: [68] Express Upstream Port, MSI 00
Capabilities: [a4] Subsystem: Dolphin Interconnect Solutions AS Device 0832
Capabilities: [100] Device Serial Number 00-00-43-43-00-00-00-0a
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [138] Power Budgeting <?>
Capabilities: [10c] #19
Capabilities: [148] Virtual Channel
Capabilities: [e00] #12
Capabilities: [b00] Latency Tolerance Reporting
Capabilities: [b70] Vendor Specific Information: ID=0001 Rev=0 Len=010
Kernel driver in use: pcieport
Kernel modules: shpchp
# lspci | grep “Device 8733”
01:00.0 PCI bridge: PLX Technology, Inc. Device 8733 (rev ca)
02:08.0 PCI bridge: PLX Technology, Inc. Device 8733 (rev ca)
# lspci -s 1:0.0 -v | grep -E "Subsystem|Serial"
Capabilities: [a4] Subsystem: Dolphin Interconnect Solutions AS Device 0832
Capabilities: [100] Device Serial Number 00-00-43-43-00-00-00-0a
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 14
This shows the card as revision 0x4243 (hexadecimal values of the ‘CC’ letters in the ASCII table), with the production number
0x0000000A (0000010 in decimal).
Support
More information about the product, support and software download can be found at http://www.dolphinics.com/px.html. Please
email [email protected] if you have any questions.
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 15
Technical Information
Please reference the FireFly PCUO manual available from Samtec for more information.
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 16
Pending Compliance and Regulatory Testing
EMC Compliance
The Dolphin PCI Express PXH840 adapter is tested to the following relevant test standards for PCI Express
cards, telecommunication and industry equipment installed in a standard PC:
EN 55022 (2010), Class B
EN 55024 (2010), Class A
This does not ensure that it will comply with these standards in any random PC. It is the responsibility of the integrator to ensure
that their products are compliant with all regulations where their product will be used.
RoHS Compliance
The Dolphin PXH840 is RoHS compliant. A Compliance certificate issued by the manufacturer is
available upon request.
FCC Class A
This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to part 15 of
the FCC Rules.
PXH840 User’s Guide – Dolphin Interconnect Solutions Page 17
Limited Warranty
Dolphin Interconnect Solutions warrants this product to be free from manufacturing defects under the following terms:
Warranty Period
Dolphin warrants the product for one (1) year from the date of purchase. Extended warranties are available.
Coverage
To the extent permitted by applicable law, this warranty does not apply to:
• Damages caused by operator error or non-compliance with instructions available for the product.
• Use or attempt to use or program firmware not approved by Dolphin.
• Damage which results from accident, abuse, misuse, neglected improper handling or improper installation; moisture,
corrosive environments, high voltage surges, shipping or abnormal working conditions.
• Damages which results from violating the specified operating or storage temperatures and airflow.
• Damages caused by acts of nature, e.g. floods, storms, fire, or earthquakes.
• Damage caused by any power source out of range or not provided with the product.
• Normal wear and tear.
• Attempts to repair, modify, open or upgrade the product by personnel or agents not authorized by Dolphin.
• Products that have had the product serial number tampered with or removed.
• Damage to the product caused by products not supplied by Dolphin.
Service Procedure
If the product proves defective during the Warranty Period, you should contact the seller that supplied you with the product, or if
you purchased it directly from Dolphin, email [email protected] to obtain a valid RMA number and instructions.
Products returned to Dolphin without a proper RMA number will not be serviced under this warranty.
/
