Cisco UCS C Series Troubleshooting Tips

UCS C−Series Servers Troubleshooting Tips

Document ID: 111508

Contents

Introduction

Prerequisites

Requirements

Components Used

Network Diagram

Conventions

Background Information

C−Series Troubleshooting Tips

Obtaining Showtech Support to TAC

Display of System Event Log Events

Display of Sensor Readings

Display of CIMC Log

Run Debug Firmware Utility (CLI)

Run Diagnostics (CLI)

Common Troubleshooting Scenarios − Power−On Related

Common Troubleshooting Scenarios − Host Does not Boot

Common Troubleshooting Scenarios − BMC

Verify

Troubleshoot

Related Information

Introduction

The Cisco Integrated Management Controller (CIMC) is the management service for the UCS C−Series

server. CIMC runs within the server.

You can use a web−based GUI or SSH−based CLI to access, configure, administer, and monitor the server.

Almost all tasks can be performed in either interface. The results of tasks performed in one interface are

automatically displayed in another.

This document provides some CIMC troubleshooting tips and common troubleshooting scenarios for UCS

C−Series servers.

Prerequisites

Requirements

Cisco recommends that you:

Have a working knowledge of the Cisco UCS C−Series Server Blade hardware and software

administration.

•

Be familiar with UCS C−Series Servers Cisco Integrated Management Controller.•

Understand the impact and implications of the different commands described in this document.•

Be familiar with the UCS components and topology.•

Components Used

The information in this document is based on Cisco UCS C−Series Servers.

The information in this document was created from the devices in a specific lab environment. All of the

devices used in this document started with a default configuration. If your network is live, make sure that you

understand the potential impact of any command.

Network Diagram

There is currently no specific network diagram available.

Conventions

Refer to the Cisco Technical Tips Conventions for more information on document conventions.

Background Information

There is currently no specific background information available.

C−Series Troubleshooting Tips

Common troubleshooting tips on C−Series servers are provided in this section.

Obtaining Showtech Support to TAC

Perform this task when requested by the Cisco Technical Assistance Center (TAC). This utility creates a

summary report containing configuration information, logs, and diagnostic data that will help TAC in

troubleshooting and resolving a technical issue.

This showtechsupport is available from GUI and CLI to tftp upload a techsupport file for offline analysis.

Complete these steps in order to obtain showtech via GUI:

In the Navigation pane, click the Admin tab.1.

From the Admin tab, click Utilities.2.

In the Actions area of the Utilities pane, click Export Technical Support Data.3.

In the Export Technical Support Data dialog box, complete these fields:

TFTP Server IP Address field − The IP address of the TFTP server on which the support data

file should be stored.

♦

Path and Filename field − The file name in which the support data should be stored on the

server. When you enter this name, include the relative path for the file from the top of the

TFTP tree to the desired location.

♦

4.

Click Export.

And via the CLI command:

Generate show techsupport

Generate show techsupport then provide the generated report file to Cisco TAC.

SanDiego# scope cimc

SanDiego /cimc # scope

firmware

log

network

tech−support

5.

SanDiego /cimc # scope tech−support

SanDiego /cimc/tech−support # set tftp−ip 192.168.1.1

SanDiego /cimc/tech−support *# set path \techsupport\showtech

SanDiego /cimc/tech−support *#commit

SanDiego /cimc/tech−support *#start

These are explanations of some of the key fields within the showtech:

var/ − Contains detailed logs, and status of all monitored services. It also contains services

information files such as the configuration of SOL and IPMI sensor alarms.

♦

var/log − This contains the rolling volatile log messages.♦

obfl/ − This contains the rolling non−volatile log messages.♦

met/ − Non−volatile configuration and SEL.♦

tmp/ − The show techsupport text files, along with BIOS techsupport text files.♦

Text files in tmp − These contain all process, network, system, mezzanine, and bios state

information.

♦

mctool − Gets basic information on the State of the CIMC to USC management API.♦

network − See current network configuration and socket information.♦

obfl − Live obfl♦

messages − Live /var/log/messages file♦

alarms − What sensors are in alarm.♦

sensors − Current sensor readings from IPMI.♦

power − The current power state of the x86.♦

Display of System Event Log Events

Complete these steps in order to display the System Event Log (SEL) events:

In the Navigation pane, click the Server tab.1.

From the Server tab, click System Event Log.2.

Review the following information for each system event in the log.3.

(Optional) From the Entries Per Page drop−down list, select the number of system events to display

on each page.

4.

(Optional) Click <Newer and Older> to move backward and forward through the pages of system

events, or click <<Newest to move to the top of the list. By default, the newest system events are

displayed at the top if the list. Cisco CIMC.

5.

Display of Sensor Readings

Complete these steps in order to display the sensor readings:

In the Navigation pane, click the Server tab.1.

From the Server tab, click Sensors.2.

View various sensors by clicking the desired sensor.3.

Display of CIMC Log

Complete these steps in order to display the CIMC log:

In the Navigation pane, click the Admin tab.1.

From the Admin tab, click CIMC Log.2.

From the Entries Per Page drop−down list, select the number of CIMC events to display on each page.3.

Run Debug Firmware Utility (CLI)

You can run Debug Firmware Utility (CLI) to view realtime CIMC debug status:

Access to read−only shell to view realtime CIMC Debug status.

CIMC Debug Utilities

!−−− enter debug shell

rtp−6100−A#

SanDiego /cimc # connect

debug−shell

diags

host

shell

SanDiego /cimc # connect debug−shell

<CR>

SanDiego /cimc # connect debug−shell

BMC Debug Firmware Utility Shell

[ help ]#

!−−− available debug options

[ help ]# ?

__________________________________________

Debug Firmware Utility

__________________________________________

Command List

__________________________________________

alarms

cores

exit

help [COMMAND]

images

mctools

memory

messages

network

obfl

post

power

sensors

sel

fru

tasks

top

update

users

version

__________________________________________

Notes:

"enter Key" will execute last command

"COMMAND ?" will execute help for that command

__________________________________________

[ help ]#

!−−− view how many alarms in realtime

[ help ]# alarms

StatusLedControl: Setting LED to AMBER

− Sensor[176] in ALARM Level[2]

[ alarms ]#

!−−− view all sensors in realtime

[ alarms ]# sensors

P3V_BAT_SCALED | 3.023 | Volts | ok | 2.706 | 2.798 | na | na | 3.089 | na

P12V_SCALED | 12.036 | Volts | ok | 11.269 | 11.623 | na | na | 12.331 | 12.685

P5V_SCALED | 5.037 | Volts | ok | 4.675 | 4.844 | na | na | 5.157 | 5.278

P3V3_SCALED | 3.302 | Volts | ok | 3.097 | 3.192 | na | na | 3.381 | 3.492

P5V_STBY_SCALED | 4.989 | Volts | ok | 4.675 | 4.844 | na | na | 5.157 | 5.278

VR_CPU1_IOUT | 10.680 | Amps | ok | na | na | na | 152.680 | 164.040 | 175.400

VR_CPU2_IOUT | 12.100 | Amps | ok | na | na | na | 152.680 | 164.040 | 175.400

PV_VCCP_CPU1 | 0.862 | Volts | ok | 0.706 | 0.725 | na | na | 1.392 | 1.431

PV_VCCP_CPU2 | 0.862 | Volts | ok | 0.706 | 0.725 | na | na | 1.392 | 1.431

P1V5_DDR3_CPU1 | 1.499 | Volts | ok | 1.411 | 1.450 | na | na | 1.548 | 1.588

P1V5_DDR3_CPU2 | 1.499 | Volts | ok | 1.411 | 1.450 | na | na | 1.548 | 1.588

P1V1_IOH | 1.088 | Volts | ok | 1.029 | 1.068 | na | na | 1.137 | 1.166

P1V8_AUX | 1.784 | Volts | ok | 1.695 | 1.744 | na | na | 1.852 | 1.911

IOH_THERMALERT_N | 0x0 | discrete | 0x0180| na | na | na | na | na | na

IOH_THERMTRIP_N | 0x0 | discrete | 0x0180| na | na | na | na | na | na

P2_THERMTRIP_N | 0x0 | discrete | 0x0180| na | na | na | na | na | na

P1_THERMTRIP_N | 0x0 | discrete | 0x0180| na | na | na | na | na | na

!−−− view power status in realtime

[ help ]# power

OP:[ status ]

Power−State: [ on ]

VDD−Power−Good: [ active ]

Power−On−Fail: [ inactive ]

Power−Ctrl−Lock: [ unlocked ]

OP−CCODE:[ Success ]

[ power ]#

!−−− view network status in realtime

[ power ]# network

eth1 Link encap:Ethernet HWaddr 02:44:67:84:09:1C

inet addr:172.25.183.109 Bcast:172.25.183.255 Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:42862 errors:0 dropped:0 overruns:0 frame:0

TX packets:26968 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:3786646 (3.6 MiB) TX bytes:12311980 (11.7 MiB)

Interrupt:1

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.255.0.0

UP LOOPBACK RUNNING MTU:16436 Metric:1

RX packets:8137 errors:0 dropped:0 overruns:0 frame:0

TX packets:8137 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:877033 (856.4 KiB) TX bytes:877033 (856.4 KiB)

Active Internet connections (servers and established)

Proto Recv−Q Send−Q Local Address Foreign Address State

tcp 0 0 0.0.0.0:3490 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:8195 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:2068 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:8195 127.0.0.1:2360 ESTABLISHED

tcp 0 0 127.0.0.1:8195 127.0.0.1:2361 ESTABLISHED

tcp 0 0 127.0.0.1:8195 127.0.0.1:2353 ESTABLISHED

tcp 0 0 127.0.0.1:2363 127.0.0.1:8195 ESTABLISHED

tcp 0 0 127.0.0.1:2360 127.0.0.1:8195 ESTABLISHED

tcp 0 0 127.0.0.1:2361 127.0.0.1:8195 ESTABLISHED

tcp 0 0 127.0.0.1:2367 127.0.0.1:8195 ESTABLISHED

tcp 0 0 127.0.0.1:8195 127.0.0.1:2354 ESTABLISHED

tcp 0 0 127.0.0.1:2354 127.0.0.1:8195 ESTABLISHED

tcp 0 0 127.0.0.1:2355 127.0.0.1:8195 ESTABLISHED

tcp 0 0 127.0.0.1:2353 127.0.0.1:8195 ESTABLISHED

tcp 0 0 127.0.0.1:2358 127.0.0.1:8195 ESTABLISHED

tcp 0 0 127.0.0.1:2359 127.0.0.1:8195 ESTABLISHED

tcp 0 0 127.0.0.1:2356 127.0.0.1:8195 ESTABLISHED

tcp 0 0 127.0.0.1:2357 127.0.0.1:8195 ESTABLISHED

tcp 0 0 127.0.0.1:8195 127.0.0.1:2363 ESTABLISHED

tcp 0 0 127.0.0.1:8195 127.0.0.1:2355 ESTABLISHED

tcp 0 4412 172.25.183.109:22 10.61.100.118:2632 ESTABLISHED

tcp 0 0 127.0.0.1:8195 127.0.0.1:2356 ESTABLISHED

tcp 0 0 127.0.0.1:8195 127.0.0.1:2357 ESTABLISHED

tcp 0 0 127.0.0.1:8195 127.0.0.1:2358 ESTABLISHED

tcp 0 0 127.0.0.1:8195 127.0.0.1:2367 ESTABLISHED

tcp 0 0 127.0.0.1:8195 127.0.0.1:2359 ESTABLISHED

netstat: no support for 'AF INET6 (tcp)' on this system

udp 0 0 127.0.0.1:9473 0.0.0.0:*

udp 0 0 0.0.0.0:623 0.0.0.0:*

netstat: no support for 'AF INET6 (udp)' on this system

netstat: no support for 'AF INET6 (raw)' on this system

Active UNIX domain sockets (servers and established)

Proto RefCnt Flags Type State I−Node Path

unix 2 [ ACC ] STREAM LISTENING 3330 /tmp/rpSocketCB25226

unix 2 [ ACC ] STREAM LISTENING 2112 /var/split_stream_RW

unix 2 [ ACC ] STREAM LISTENING 2114 /var/split_stream_RO

unix 2 [ ACC ] STREAM LISTENING 4437 /tmp/rpSocketSMCB536870913

unix 2 [ ACC ] STREAM LISTENING 2903 /tmp/rpSocket35003

Run Diagnostics (CLI)

You can run Diagnostics (CLI) to diagnose possible issue.

Note: Diagnostics while designed to be safe to a running server should not be run with a load you cannot

afford to lose. Please ensure critical server applications are off−line before running diagnostics.

Diagnostics (CLI)

To view realtime CIMC Diagnostics status.

CIMC Diags Shell

!−−− Enter diagnostics mode.

SanDiego# connect

debug−shell

diags

host

shell

SanDiego# connect diags

<CR>

SanDiego# connect diags

Diagnostics while designed to be safe to a running server should not

be run with a load you cannot afford to lose. Please ensure critical

server applications are off−line before running diagnostics.

Continue?[y|N]y

max_slots = 1

Registering hwmon commands

dsh_cmd_add: Adding hwmon 0x32658

Registering board commands

dsh_cmd_add: Adding sprom 0x129a8

dsh_cmd_add: Adding i2c 0x12a14

dsh_cmd_add: Adding spd 0x12db8

dsh_cmd_add: Adding tmp75 0x12c88

dsh_cmd_add: Adding clkbuf 0x132e4

dsh_cmd_add: Adding timhub 0x13368

dsh_cmd_add: Adding help 0x10778

dsh_cmd_add: Adding show 0x108f4

dsh_cmd_add: Adding run 0x108a0

dsh_cmd_add: Adding skip 0x10b2c

dsh_cmd_add: Adding unskip 0x10b64

dsh_cmd_add: Adding results 0x10b9c

dsh_cmd_add: Adding sp 0x10ce8

dsh_cmd_add: Adding reg 0x10db0

dsh_cmd_add: Adding version 0x10de8

dsh_cmd_add: Adding err 0x10e24

DIAG >

DIAG > q

SanDiego#

•

Common Troubleshooting Scenarios − Power−On Related

No Standby Power to UCS C250 M1 Extended−Memory Rack−Mount Server

Check that the AC power cord is ok.1.

Failure in Power Supply Unit.2.

•

Server Host does not power up

Check front I/O board connection.1.

Check Power Sequencer fault LEDs.2.

Power Supply unit failure (PS Failure LED blinking).3.

•

Server powers on with no video

Check that the front I/O dongle is properly seated.1.

Check the front I/O cable connection to Motherboard.2.

Memory subsystem failure.3.

•

BMC does not boot

Failure in standby Power rails.1.

Corrupt BMC BIOS.2.

•

Common Troubleshooting Scenarios − Host Does not Boot

Check for:

Verify front I/O dongle is seated correctly.1.

Check Front I/O cable connection.2.

Reseat/Replace Dimm(s).3.

Verify BIOS is not corrupt.4.

Verify host power rails are good.5.

Check CPU sockets for bent pins.6.

Verify Powerok signals are ok.7.

Verify Resets are good.8.

Common Troubleshooting Scenarios − BMC

BMC booted. Look for Blade health LED to come on which indicates that the BMC has started.

Check that the Standby power rails are ok.1.

Check that the BMC bios is not corrupt.2.

Check that the BMC clock is ok.3.

Check that standby power is ok and resets are valid.4.

BMC Ethernet cannot communicate

Check the flex cable connections to Mother Board and Rear I/O.

Verify

Refer to the above sub−sections for verifications.

Troubleshoot

There is currently no specific troubleshooting information available for this configuration.

Related Information

Technical Support & Documentation − Cisco Systems•

Contacts & Feedback | Help | Site Map

Cisco Systems, Inc.

Updated: Jan 08, 2010 Document ID: 111508

Cisco UCS C Series Troubleshooting Tips

Cisco UCS C Series Troubleshooting Tips

Related papers

Other documents