Oracle EXADATA X5-8 User manual

Oracle

®

Server X5-8 Service Manual

Part No: E56311-03

December 2015

Oracle Server X5-8 Service Manual

Part No: E56311-03

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except

as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform,

publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is

prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable:

U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation,

delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental

regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the

hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government.

This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous

applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all

appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this

software or hardware in dangerous applications.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of

SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered

trademark of The Open Group.

This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are

not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement

between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content,

products, or services, except as set forth in an applicable agreement between you and Oracle.

Access to Oracle Support

Oracle customers that have purchased support have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/topic/lookup?

ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired.

Référence: E56311-03

Ce logiciel et la documentation qui l'accompagne sont protégés par les lois sur la propriété intellectuelle. Ils sont concédés sous licence et soumis à des restrictions d'utilisation et

de divulgation. Sauf stipulation expresse de votre contrat de licence ou de la loi, vous ne pouvez pas copier, reproduire, traduire, diffuser, modifier, accorder de licence, transmettre,

distribuer, exposer, exécuter, publier ou afficher le logiciel, même partiellement, sous quelque forme et par quelque procédé que ce soit. Par ailleurs, il est interdit de procéder à toute

ingénierie inverse du logiciel, de le désassembler ou de le décompiler, excepté à des fins d'interopérabilité avec des logiciels tiers ou tel que prescrit par la loi.

Les informations fournies dans ce document sont susceptibles de modification sans préavis. Par ailleurs, Oracle Corporation ne garantit pas qu'elles soient exemptes d'erreurs et vous

invite, le cas échéant, à lui en faire part par écrit.

Si ce logiciel, ou la documentation qui l'accompagne, est livré sous licence au Gouvernement des Etats-Unis, ou à quiconque qui aurait souscrit la licence de ce logiciel pour le

compte du Gouvernement des Etats-Unis, la notice suivante s'applique :

U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation,

delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental

regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the

hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government.

Ce logiciel ou matériel a été développé pour un usage général dans le cadre d'applications de gestion des informations. Ce logiciel ou matériel n'est pas conçu ni n'est destiné à être

utilisé dans des applications à risque, notamment dans des applications pouvant causer un risque de dommages corporels. Si vous utilisez ce logiciel ou ce matériel dans le cadre

d'applications dangereuses, il est de votre responsabilité de prendre toutes les mesures de secours, de sauvegarde, de redondance et autres mesures nécessaires à son utilisation dans

des conditions optimales de sécurité. Oracle Corporation et ses affiliés déclinent toute responsabilité quant aux dommages causés par l'utilisation de ce logiciel ou matériel pour des

applications dangereuses.

Oracle et Java sont des marques déposées d'Oracle Corporation et/ou de ses affiliés. Tout autre nom mentionné peut correspondre à des marques appartenant à d'autres propriétaires

qu'Oracle.

Intel et Intel Xeon sont des marques ou des marques déposées d'Intel Corporation. Toutes les marques SPARC sont utilisées sous licence et sont des marques ou des marques

déposées de SPARC International, Inc. AMD, Opteron, le logo AMD et le logo AMD Opteron sont des marques ou des marques déposées d'Advanced Micro Devices. UNIX est une

marque déposée de The Open Group.

Ce logiciel ou matériel et la documentation qui l'accompagne peuvent fournir des informations ou des liens donnant accès à des contenus, des produits et des services émanant de

tiers. Oracle Corporation et ses affiliés déclinent toute responsabilité ou garantie expresse quant aux contenus, produits ou services émanant de tiers, sauf mention contraire stipulée

dans un contrat entre vous et Oracle. En aucun cas, Oracle Corporation et ses affiliés ne sauraient être tenus pour responsables des pertes subies, des coûts occasionnés ou des

dommages causés par l'accès à des contenus, produits ou services tiers, ou à leur utilisation, sauf mention contraire stipulée dans un contrat entre vous et Oracle.

Accès aux services de support Oracle

Les clients Oracle qui ont souscrit un contrat de support ont accès au support électronique via My Oracle Support. Pour plus d'informations, visitez le site http://www.oracle.com/

pls/topic/lookup?ctx=acc&id=info ou le site http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs si vous êtes malentendant.

Contents

Using This Documentation ................................................................................ 11

About This Document ....................................................................................... 13

Service Notes .................................................................................................... 15

Intended Audience ......................................................................................... 15

Warning Label .............................................................................................. 15

Server and Components Overview ................................................................... 17

Server Overview ........................................................................................... 18

Chassis Overview .......................................................................................... 19

Chassis Front Side Components ............................................................... 19

Chassis Internal Components ................................................................... 20

Chassis Backside Components ................................................................. 22

CPU Module (CMOD) Overview ..................................................................... 23

Processor and Memory Overview ............................................................. 23

CMOD Configuration Options ................................................................. 23

CMOD Population Rules ........................................................................ 26

CMOD Layout ...................................................................................... 27

CMOD and Fan Module Power ............................................................... 28

System Module (SMOD) Overview .................................................................. 28

SMOD Overview .................................................................................. 29

Storage Drives ...................................................................................... 30

SMOD Motherboard .............................................................................. 31

Service Processor (SP) ........................................................................... 31

Storage Drive Backplanes ....................................................................... 32

SAS Host Bus Adapter (HBA) Card, Riser, and Cables ................................ 32

Internal USB Ports ................................................................................ 34

5

Contents

Energy Storage Module and Cable ........................................................... 34

Server Subsystems ......................................................................................... 34

Controls and Indicators ........................................................................... 34

Server Management Software .................................................................. 43

Storage and IO Subsystem ...................................................................... 45

Chassis Cooling Subsystem ..................................................................... 47

Power Subsystem .................................................................................. 50

Server Block Diagram .................................................................................... 50

Troubleshooting and Diagnostics ..................................................................... 53

Troubleshooting Server Hardware Component Faults ........................................... 53

Troubleshooting Server Hardware Faults ................................................... 54

Troubleshooting and Diagnostic Information .............................................. 58

Troubleshooting Indicators ...................................................................... 58

Indicator Blink Rates ............................................................................. 62

The CMOD Fault Remind Test Circuit ...................................................... 68

Troubleshooting System Cooling Issues ..................................................... 68

Troubleshooting Power Issues .................................................................. 69

Troubleshooting With Diagnostic Tools ............................................................. 70

Diagnostic Tools ................................................................................... 71

Diagnostic Tool Documentation ............................................................... 72

Attaching Devices to the Server ....................................................................... 72

▼ Attach Devices to the Server .............................................................. 73

Configuring Serial Port Sharing ............................................................... 73

Ethernet Port Device Naming .................................................................. 75

Getting Help ................................................................................................. 75

Contacting Support ................................................................................ 76

Locating the Chassis Serial Number ......................................................... 76

Servicing the Server ......................................................................................... 79

Component Locations, and Designations ............................................................ 79

Component Serviceability Requirements .................................................... 79

Component Locations ............................................................................. 80

Component Designations ........................................................................ 82

Component Network Access Control (NAC) Names .................................... 89

Performing Electrostatic Discharge and Static Prevention Measures ........................ 90

Using an Anti-static Wrist Strap ............................................................... 90

6 Oracle Server X5-8 Service Manual • December 2015

Contents

Using an Anti-static Mat ........................................................................ 90

Tools and Equipment ..................................................................................... 91

Component Filler Panels and Non-Powered Components ...................................... 91

▼ Clear Hardware Fault Messages .................................................................. 92

Preparing for Service ........................................................................................ 95

▼ Prepare the Server for Hot Service .............................................................. 95

▼ Prepare the Server for Warm Service ........................................................... 96

▼ Prepare the Server for Cold Service ............................................................. 98

Powering Off the Server ............................................................................... 100

▼ Power Off the Server Using the Server OS .......................................... 101

▼ Power Off, Graceful (Power Button) .................................................. 101

▼ Power Off, Immediate (Power Button) ................................................ 102

▼ Power Off, Remote (Oracle ILOM CLI) ............................................. 103

▼ Power Off, Remote (Oracle ILOM Web Interface) ................................ 104

▼ Remove Power ............................................................................... 105

Power Modes, Shutdowns, and Resets ..................................................... 107

Managing the Locator Indicator ..................................................................... 108

▼ Turn On the Locator Indicator Remotely (Oracle ILOM CLI) .................. 109

▼ Turn On the Locator Indicator Remotely (Oracle ILOM Web Interface) ..... 110

▼ Control the Locator Indicator Locally ................................................. 110

Servicing Components .................................................................................... 113

▼ Upgrade the Server from Four to Eight CMODs ........................................... 113

Servicing Fan Modules and Fan Frames .......................................................... 117

▼ Remove a Fan Module .................................................................... 117

▼ Install a Fan Module ....................................................................... 120

▼ Remove a Fan Frame ...................................................................... 122

▼ Install a Fan Frame ......................................................................... 124

Servicing Power Supply Units (PSUs) ............................................................. 126

▼ Remove a PSU .............................................................................. 126

▼ Install a PSU ................................................................................. 129

Servicing the Front Indicator Module (FIM) ..................................................... 132

▼ Remove the FIM ............................................................................ 133

▼ Install the FIM ............................................................................... 134

Servicing the CPU Module (CMOD) Components ............................................. 136

▼ Remove a CMOD ........................................................................... 137

7

Contents

▼ Remove and Install the CMOD Cover ................................................ 140

▼ Install a CMOD ............................................................................. 143

▼ Replace a Failed DIMM .................................................................. 146

▼ Install a DIMM .............................................................................. 152

▼ Remove a DIMM ........................................................................... 155

Memory and DIMM Reference .............................................................. 158

▼ Remove a Heatsink and Processor (FRU) ............................................ 161

▼ Install a Heatsink and Processor (FRU) .............................................. 172

Servicing Storage Drives ............................................................................... 180

▼ Remove a Storage Drive .................................................................. 180

▼ Install a Storage Drive ..................................................................... 182

Storage Drive Reference ....................................................................... 184

Servicing PCIe Cards and the Dual PCIe Card Carriers (DPCCs) .......................... 185

▼ Remove a DPCC ............................................................................ 185

▼ Remove a PCIe Card ...................................................................... 188

▼ Install a PCIe Card ......................................................................... 190

▼ Install a DPCC ............................................................................... 193

▼ Replace a DPCC ............................................................................ 196

PCIe Card and DPCC Reference ............................................................ 196

Servicing System Module (SMOD) Components ............................................... 197

▼ Remove the SMOD ........................................................................ 197

▼ Install the SMOD ........................................................................... 200

Servicing the Host Bus Adapter (HBA) Card ............................................ 202

Servicing the Energy Storage Module and Cables ...................................... 209

Servicing the SAS Cable ....................................................................... 214

Servicing the Internal USB Flash Drives .................................................. 215

▼ Replace the Real Time Clock (System) Battery .................................... 219

▼ Replace the Midplane Assembly ............................................................... 223

Returning the Server to Operation .................................................................. 235

▼ Prepare the Server for Operation ............................................................... 235

▼ Power On the Server ............................................................................... 236

BIOS Setup Utility ............................................................................................ 237

▼ Access the BIOS Setup Utility .................................................................. 237

BIOS Setup Utility Screens ........................................................................... 238

Main Screen (Legacy) .......................................................................... 238

8 Oracle Server X5-8 Service Manual • December 2015

Contents

Advanced Screen (Legacy) .................................................................... 244

Advanced - Processor Configuration ....................................................... 245

Advanced - CPU Power Management Configuration .................................. 252

Memory Configuration ......................................................................... 253

Advanced - USB Ports ......................................................................... 254

Advanced - Serial Port Console Redirection ............................................. 255

Advanced - Trusted Computing .............................................................. 259

Advanced - Network Stack .................................................................... 261

Advanced - Legacy iSCSI ..................................................................... 262

Advanced - BMC Network Configuration ................................................ 263

IO Screen ........................................................................................... 266

Boot Screens ....................................................................................... 272

Boot Screen in Legacy Boot Mode ......................................................... 272

Index ................................................................................................................ 275

9

10 Oracle Server X5-8 Service Manual • December 2015

Using This Documentation

This section describes how to get the latest firmware, software, and documentation for the

Oracle Server X5-8. It also provides feedback links and a document change history.

■

“Oracle Server X5-8 Model Naming Convention” on page 11

■

“Getting the Latest Firmware and Software” on page 11

■

“Documentation and Feedback” on page 12

■

“Contributors” on page 12

■

“Change History” on page 12

The information in this documentation set is presented in topic-based format (similar to online

help) and therefore does not include chapters, appendixes, or section numbering.

Oracle Server X5-8 Model Naming Convention

The Oracle Server X5-8 name identifies the following:

■

X identifies an x86 product.

■

The first number, 5, identifies the generation of the server.

■

The second number, 8, identifies the maximum number of processors.

Getting the Latest Firmware and Software

Firmware, drivers, and other hardware-related software for each Oracle x86 server are updated

periodically.

You can obtain the latest version in the following ways:

■

Oracle System Assistant: This is a factory-installed option for Oracle x86 servers. It has all

the tools and drivers you need and resides on a USB drive installed in most servers.

■

You can download updates from My Oracle Support: https://support.oracle.com

Using This Documentation 11

Documentation and Feedback

Documentation Link

All Oracle products

http://docs.oracle.com

Oracle Server X5-8

http://www.oracle.com/goto/X5-8/docs-videos

Oracle Integrated Lights Out Manager (ILOM). Refer to

the Oracle ILOM documentation.

http://www.oracle.com/goto/ILOM/docs

Oracle Hardware Management Pack. Refer to the

documentation for your supported version of Oracle

HMP as listed in the Product Notes.

http://www.oracle.com/goto/ohmp/docs

Provide feedback on this documentation at: http://www.oracle.com/goto/docfeedback

Contributors

Primary Authors: Michael Bechler, Cynthia Chin-Lee, Mark McGothigan.

Contributors: William Schweickert, Anthony Villamor, Mick Tabor, Richard Masoner, Ray

Angelo, Tamra Smith-Wasel, Denise Silverman.

Change History

The following lists the release history of this documentation set:

■

December 2015. Technical updates.

■

September 2015. Editorial improvements.

■

July 2015. Initial publication.

12 Oracle Server X5-8 Service Manual • December 2015

About This Document

This document provides service, maintenance, and component replacement procedures for the

Oracle Server X5-8.

The following table describes the major sections of this document.

Section Description Link

Important service information “Service Notes” on page 15

Server component and subsystem overviews “Server and Components Overview” on page 17

Troubleshooting procedures and information “Troubleshooting and Diagnostics” on page 53

General Information and procedures for

servicing the server

“Servicing the Server” on page 79

Information and procedures for preparing the

server for service

“Preparing for Service” on page 95

Procedures and information for removing

and installing components

“Servicing Components” on page 113

Procedures and information for returning the

server to operation after performing service

procedures

“Returning the Server to Operation” on page 235

Server BIOS Setup Utility information and

screen captures

“BIOS Setup Utility” on page 237

About This Document 13

14 Oracle Server X5-8 Service Manual • December 2015

Service Notes

This section contains preliminary service information:

Intended Audience

This guide is intended for trained technicians and authorized service personnel who have been

instructed on the hazards within the equipment and qualified to replace and install hardware.

Warning Label

The following warning label is visible from the front of the server when you remove a fan

module. It warns you to not insert your hands or any object into the space left vacant by the

removal of the fan module. Fan modules are hot-swap components. Removing a fan module

from a fully-powered server exposes open and active power connectors that can cause electric

shock.

Service Notes 15

16 Oracle Server X5-8 Service Manual • December 2015

Server and Components Overview

This section describes the server and its subsystems. It includes:

Section Description Link

List of server features “Server Overview” on page 18

Chassis front, internal, and back components “Chassis Overview” on page 19

Features and components of the CPU module

(CMOD)

“CPU Module (CMOD) Overview” on page 23

Features and components of the system

module (SMOD)

“System Module (SMOD) Overview” on page 28

Server subsystems, their functions, and

related components

“Server Subsystems” on page 34

Schematic-type block diagram of the server

interconnects

“Server Block Diagram” on page 50

Server and Components Overview 17

Server Overview

The Oracle Server X5-8 is a 5 rack-unit (RU) server with the following features:

■

Four and eight socket configurations that use Intel EX Xeon® E7-8895 v3 processors for a

total of 72 or 144 cores.

■

Maximum memory: 3 TB (four socket) and 6 TB (eight socket) of DDR3 1333 memory.

■

Eight backside accessible SAS3 or SATA storage drive bays.

■

Expandable IO: eight 16-lane and eight 8-lane PCIe Gen3 slots and one 4 lane PCIe Gen 2

HBA slot.

■

One Emulex Pilot 3 service processor (SP) with 256 MB DDR3 memory, 256 MB of flash

memory, and Oracle ILOM.

■

Four (N+N) hot-swap power supplies (PSUs).

■

Eight hot-swap redundant 100 watt cooling fan modules.

Note - For server specification information, see “Server Specifications” in Oracle Server X5-8

Installation Guide.

The following sections provide overviews of the main server components:

Component Link

Chassis “Chassis Overview” on page 19

CMODs “CPU Module (CMOD) Overview” on page 23

18 Oracle Server X5-8 Service Manual • December 2015

Chassis Overview

Component Link

SMOD “System Module (SMOD) Overview” on page 28

Chassis Overview

The chassis consists of the front accessible components, internal components, and components

accessible from the back of the server:

■

“Chassis Front Side Components” on page 19

■

“Chassis Internal Components” on page 20

■

“Chassis Backside Components” on page 22

Chassis Front Side Components

The following figure shows the front side components:

Server and Components Overview 19

Chassis Overview

The front-side components include:

Call Out Component Link

1 Front indicator module (FIM) “Controls and Indicators” on page 34

2 Four power supplies “Power Subsystem” on page 50.

3 and 4 Eight fan modules (FMs) in two fan frames “Chassis Cooling Subsystem” on page 47

5 Two internal CMOD bays “CPU Module (CMOD)

Overview” on page 23

Chassis Internal Components

The following figure shows the chassis internal components:

20 Oracle Server X5-8 Service Manual • December 2015

Page is loading ...

Oracle EXADATA X5-8 User manual

Oracle EXADATA X5-8 User manual

Related papers

Other documents