UCS C-Series Servers Troubleshooting Tips

Introduction

The Cisco Integrated Management Controller (CIMC) is the management service for the UCS C-Series server. CIMC runs within the server.

You can use a web-based GUI or SSH-based CLI to access, configure, administer, and monitor the server. Almost all tasks can be performed in either interface. The results of tasks performed in one interface are automatically displayed in another.

This document provides some CIMC troubleshooting tips and common troubleshooting scenarios for UCS C-Series servers.

Prerequisites

Requirements

Cisco recommends that you:

Have a working knowledge of the Cisco UCS C-Series Server Blade hardware and software administration.
Be familiar with UCS C-Series Servers Cisco Integrated Management Controller.
Understand the impact and implications of the different commands described in this document.
Be familiar with the UCS components and topology.

Components Used

The information in this document is based on Cisco UCS C-Series Servers.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a default configuration. If your network is live, make sure that you understand the potential impact of any command.

Network Diagram

There is currently no specific network diagram available.

Conventions

Refer to the Cisco Technical Tips Conventions for more information on document conventions.

Background Information

There is currently no specific background information available.

C-Series Troubleshooting Tips

Common troubleshooting tips on C-Series servers are provided in this section.

Obtaining Showtech Support to TAC

Perform this task when requested by the Cisco Technical Assistance Center (TAC). This utility creates a summary report containing configuration information, logs, and diagnostic data that will help TAC in troubleshooting and resolving a technical issue.

This showtechsupport is available from GUI and CLI to tftp upload a techsupport file for offline analysis. Complete these steps in order to obtain showtech via GUI:

In the Navigation pane, click the Admin tab.
From the Admin tab, click Utilities.
In the Actions area of the Utilities pane, click Export Technical Support Data.
In the Export Technical Support Data dialog box, complete these fields:
- TFTP Server IP Address field - The IP address of the TFTP server on which the support data file should be stored.
- Path and Filename field - The file name in which the support data should be stored on the server. When you enter this name, include the relative path for the file from the top of the TFTP tree to the desired location.

Click Export.

And via the CLI command:

Generate show techsupport

Generate show techsupport then provide the generated report file to Cisco TAC.
SanDiego# scope cimc
SanDiego /cimc # scope
  firmware
  log
  network
  tech-support

SanDiego /cimc # scope tech-support
SanDiego /cimc/tech-support # set tftp-ip 192.168.1.1
SanDiego /cimc/tech-support *# set path \techsupport\showtech
SanDiego /cimc/tech-support *#commit
SanDiego /cimc/tech-support *#start

These are explanations of some of the key fields within the showtech:

var/ - Contains detailed logs, and status of all monitored services. It also contains services information files such as the configuration of SOL and IPMI sensor alarms.
var/log - This contains the rolling volatile log messages.
obfl/ - This contains the rolling non-volatile log messages.
met/ - Non-volatile configuration and SEL.
tmp/ - The show techsupport text files, along with BIOS techsupport text files.
Text files in tmp - These contain all process, network, system, mezzanine, and bios state information.
mctool - Gets basic information on the State of the CIMC to USC management API.
network - See current network configuration and socket information.
obfl - Live obfl
messages - Live /var/log/messages file
alarms - What sensors are in alarm.
sensors - Current sensor readings from IPMI.
power - The current power state of the x86.

Display of System Event Log Events

Complete these steps in order to display the System Event Log (SEL) events:

In the Navigation pane, click the Server tab.
From the Server tab, click System Event Log.
Review the following information for each system event in the log.
(Optional) From the Entries Per Page drop-down list, select the number of system events to display on each page.
(Optional) Click <Newer and Older> to move backward and forward through the pages of system events, or click <<Newest to move to the top of the list. By default, the newest system events are displayed at the top if the list. Cisco CIMC.

Display of Sensor Readings

Complete these steps in order to display the sensor readings:

In the Navigation pane, click the Server tab.
From the Server tab, click Sensors.
View various sensors by clicking the desired sensor.

Display of CIMC Log

Complete these steps in order to display the CIMC log:

In the Navigation pane, click the Admin tab.
From the Admin tab, click CIMC Log.
From the Entries Per Page drop-down list, select the number of CIMC events to display on each page.

Run Debug Firmware Utility (CLI)

You can run Debug Firmware Utility (CLI) to view realtime CIMC debug status:

Access to read-only shell to view realtime CIMC Debug status.

CIMC Debug Utilities


!--- enter debug shell

rtp-6100-A#
SanDiego /cimc # connect
  debug-shell
  diags
  host
  shell

SanDiego /cimc # connect debug-shell
  <CR>

SanDiego /cimc # connect debug-shell
BMC Debug Firmware Utility Shell
[ help ]#

!--- available debug options



[ help ]# ?
__________________________________________
          Debug Firmware Utility
__________________________________________
Command List
__________________________________________

alarms
cores
exit
help [COMMAND]
images
mctools
memory
messages
network
obfl
post
power
sensors
sel
fru
tasks
top
update
users
version

__________________________________________
 Notes:
"enter Key" will execute last command
"COMMAND ?" will execute help for that command
__________________________________________
[ help ]#

!--- view how many alarms in realtime

[ help ]# alarms
  StatusLedControl: Setting LED to AMBER
   - Sensor[176] in ALARM Level[2]
[ alarms ]#

!--- view all sensors in realtime

[ alarms ]# sensors
P3V_BAT_SCALED   | 3.023      | Volts      | ok    | 2.706     | 2.798     | na        | na        | 3.089     | na
P12V_SCALED      | 12.036     | Volts      | ok    | 11.269    | 11.623    | na        | na        | 12.331    | 12.685
P5V_SCALED       | 5.037      | Volts      | ok    | 4.675     | 4.844     | na        | na        | 5.157     | 5.278
P3V3_SCALED      | 3.302      | Volts      | ok    | 3.097     | 3.192     | na        | na        | 3.381     | 3.492
P5V_STBY_SCALED  | 4.989      | Volts      | ok    | 4.675     | 4.844     | na        | na        | 5.157     | 5.278
VR_CPU1_IOUT     | 10.680     | Amps       | ok    | na        | na        | na        | 152.680   | 164.040   | 175.400
VR_CPU2_IOUT     | 12.100     | Amps       | ok    | na        | na        | na        | 152.680   | 164.040   | 175.400
PV_VCCP_CPU1     | 0.862      | Volts      | ok    | 0.706     | 0.725     | na        | na        | 1.392     | 1.431
PV_VCCP_CPU2     | 0.862      | Volts      | ok    | 0.706     | 0.725     | na        | na        | 1.392     | 1.431
P1V5_DDR3_CPU1   | 1.499      | Volts      | ok    | 1.411     | 1.450     | na        | na        | 1.548     | 1.588
P1V5_DDR3_CPU2   | 1.499      | Volts      | ok    | 1.411     | 1.450     | na        | na        | 1.548     | 1.588
P1V1_IOH         | 1.088      | Volts      | ok    | 1.029     | 1.068     | na        | na        | 1.137     | 1.166
P1V8_AUX         | 1.784      | Volts      | ok    | 1.695     | 1.744     | na        | na        | 1.852     | 1.911
IOH_THERMALERT_N | 0x0        | discrete   | 0x0180| na        | na        | na        | na        | na        | na
IOH_THERMTRIP_N  | 0x0        | discrete   | 0x0180| na        | na        | na        | na        | na        | na
P2_THERMTRIP_N   | 0x0        | discrete   | 0x0180| na        | na        | na        | na        | na        | na
P1_THERMTRIP_N   | 0x0        | discrete   | 0x0180| na        | na        | na        | na        | na        | na


!--- view power status in realtime

[ help ]# power
OP:[ status ]
Power-State:         [ on ]
VDD-Power-Good:      [ active ]
Power-On-Fail:       [ inactive ]
Power-Ctrl-Lock:     [ unlocked ]
OP-CCODE:[ Success ]
[ power ]#


!--- view network status in realtime

[ power ]# network
eth1      Link encap:Ethernet  HWaddr 02:44:67:84:09:1C
          inet addr:172.25.183.109  Bcast:172.25.183.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:42862 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26968 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3786646 (3.6 MiB)  TX bytes:12311980 (11.7 MiB)
          Interrupt:1

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.255.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:8137 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8137 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:877033 (856.4 KiB)  TX bytes:877033 (856.4 KiB)

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:3490            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:8195          0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:2068            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:23              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:8195          127.0.0.1:2360          ESTABLISHED
tcp        0      0 127.0.0.1:8195          127.0.0.1:2361          ESTABLISHED
tcp        0      0 127.0.0.1:8195          127.0.0.1:2353          ESTABLISHED
tcp        0      0 127.0.0.1:2363          127.0.0.1:8195          ESTABLISHED
tcp        0      0 127.0.0.1:2360          127.0.0.1:8195          ESTABLISHED
tcp        0      0 127.0.0.1:2361          127.0.0.1:8195          ESTABLISHED
tcp        0      0 127.0.0.1:2367          127.0.0.1:8195          ESTABLISHED
tcp        0      0 127.0.0.1:8195          127.0.0.1:2354          ESTABLISHED
tcp        0      0 127.0.0.1:2354          127.0.0.1:8195          ESTABLISHED
tcp        0      0 127.0.0.1:2355          127.0.0.1:8195          ESTABLISHED
tcp        0      0 127.0.0.1:2353          127.0.0.1:8195          ESTABLISHED
tcp        0      0 127.0.0.1:2358          127.0.0.1:8195          ESTABLISHED
tcp        0      0 127.0.0.1:2359          127.0.0.1:8195          ESTABLISHED
tcp        0      0 127.0.0.1:2356          127.0.0.1:8195          ESTABLISHED
tcp        0      0 127.0.0.1:2357          127.0.0.1:8195          ESTABLISHED
tcp        0      0 127.0.0.1:8195          127.0.0.1:2363          ESTABLISHED
tcp        0      0 127.0.0.1:8195          127.0.0.1:2355          ESTABLISHED
tcp        0   4412 172.25.183.109:22       10.61.100.118:2632      ESTABLISHED
tcp        0      0 127.0.0.1:8195          127.0.0.1:2356          ESTABLISHED
tcp        0      0 127.0.0.1:8195          127.0.0.1:2357          ESTABLISHED
tcp        0      0 127.0.0.1:8195          127.0.0.1:2358          ESTABLISHED
tcp        0      0 127.0.0.1:8195          127.0.0.1:2367          ESTABLISHED
tcp        0      0 127.0.0.1:8195          127.0.0.1:2359          ESTABLISHED
netstat: no support for 'AF INET6 (tcp)' on this system
udp        0      0 127.0.0.1:9473          0.0.0.0:*
udp        0      0 0.0.0.0:623             0.0.0.0:*
netstat: no support for 'AF INET6 (udp)' on this system
netstat: no support for 'AF INET6 (raw)' on this system
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ACC ]     STREAM     LISTENING     3330   /tmp/rpSocketCB25226
unix  2      [ ACC ]     STREAM     LISTENING     2112   /var/split_stream_RW
unix  2      [ ACC ]     STREAM     LISTENING     2114   /var/split_stream_RO
unix  2      [ ACC ]     STREAM     LISTENING     4437   /tmp/rpSocketSMCB536870913
unix  2      [ ACC ]     STREAM     LISTENING     2903   /tmp/rpSocket35003

Run Diagnostics (CLI)

You can run Diagnostics (CLI) to diagnose possible issue.

Note: Diagnostics while designed to be safe to a running server should not be run with a load you cannot afford to lose. Please ensure critical server applications are off-line before running diagnostics.

Diagnostics (CLI)

To view realtime CIMC Diagnostics status.

CIMC Diags Shell


!--- Enter diagnostics mode.

SanDiego# connect
  debug-shell
  diags
  host
  shell

SanDiego# connect diags
  <CR>

SanDiego# connect diags
Diagnostics while designed to be safe to a running server should not
be run with a load you cannot afford to lose. Please ensure critical
server applications are off-line before running diagnostics.
Continue?[y|N]y
 max_slots = 1
Registering hwmon commands
dsh_cmd_add: Adding hwmon 0x32658
Registering board commands
dsh_cmd_add: Adding sprom 0x129a8
dsh_cmd_add: Adding i2c 0x12a14
dsh_cmd_add: Adding spd 0x12db8
dsh_cmd_add: Adding tmp75 0x12c88
dsh_cmd_add: Adding clkbuf 0x132e4
dsh_cmd_add: Adding timhub 0x13368
dsh_cmd_add: Adding help 0x10778
dsh_cmd_add: Adding show 0x108f4
dsh_cmd_add: Adding run 0x108a0
dsh_cmd_add: Adding skip 0x10b2c
dsh_cmd_add: Adding unskip 0x10b64
dsh_cmd_add: Adding results 0x10b9c
dsh_cmd_add: Adding sp 0x10ce8
dsh_cmd_add: Adding reg 0x10db0
dsh_cmd_add: Adding version 0x10de8
dsh_cmd_add: Adding err 0x10e24
DIAG >
DIAG > q
SanDiego#

Common Troubleshooting Scenarios - Power-On Related

No Standby Power to UCS C250 M1 Extended-Memory Rack-Mount Server
1. Check that the AC power cord is ok.
2. Failure in Power Supply Unit.
Server Host does not power up
1. Check front I/O board connection.
2. Check Power Sequencer fault LEDs.
3. Power Supply unit failure (PS Failure LED blinking).
Server powers on with no video
1. Check that the front I/O dongle is properly seated.
2. Check the front I/O cable connection to Motherboard.
3. Memory subsystem failure.
BMC does not boot
1. Failure in standby Power rails.
2. Corrupt BMC BIOS.

Common Troubleshooting Scenarios - Host Does not Boot

Check for:

Verify front I/O dongle is seated correctly.
Check Front I/O cable connection.
Reseat/Replace Dimm(s).
Verify BIOS is not corrupt.
Verify host power rails are good.
Check CPU sockets for bent pins.
Verify Powerok signals are ok.
Verify Resets are good.

Common Troubleshooting Scenarios - BMC

BMC booted. Look for Blade health LED to come on which indicates that the BMC has started.

Check that the Standby power rails are ok.
Check that the BMC bios is not corrupt.
Check that the BMC clock is ok.
Check that standby power is ok and resets are valid.

BMC Ethernet cannot communicate

Check the flex cable connections to Mother Board and Rear I/O.

Technical Support & Documentation - Cisco Systems

Was this document helpful? Yes No

Open a Support Case Login Required (Requires a Cisco Service Contract.)

Related Cisco Support Community Discussions

The Cisco Support Community is a forum for you to ask and answer questions, share suggestions, and collaborate with your peers.

Refer to Cisco Technical Tips Conventions for information on conventions used in this document.

Updated: Jan 08, 2010

Document ID: 111508

Cisco UCS C-Series Rack Servers

UCS C-Series Servers Troubleshooting Tips

Hierarchical Navigation

Related Documents

Related Products

Contents

Related Cisco Support Community Discussions

Programs

Programs

Programs

Programs