Guest

Cisco Carrier Routing System

Cisco CRS-1 Carrier Routing System 16-Slot Line Card Chassis Hardware Operations and Troubleshooting Guide

Hierarchical Navigation

Downloads

 Feedback

Table Of Contents

Cisco CRS-1 Carrier Routing System
16-Slot Line Card Chassis Hardware
Operations and Troubleshooting Guide

Document Conventions

Warning Definition

Changes to This Document

Line Card Chassis Operations and Troubleshooting

DC Power Up Preparation and Procedures

Power-Related Site Preparation (DC System)

Conductor Characteristics and Low-Voltage Drops

Line Card Chassis Power-Up Procedures

Verifying the Installation of System Cards

Verifying the Installation of a Route Processor and Distributed Route Processor Card

Verifying the Installation of a Modular Services Card

Verifying the Installation of a Physical Layer Interface Module

Verifying the Installation of a Switch Fabric Card

Verifying the Installation of a Fan Controller Card

Troubleshooting the Installation

Troubleshooting Overview

Troubleshooting Using a Subsystem Approach

Normal Line Card Chassis Startup Sequence

Identifying Startup Issues

Troubleshooting System Cards

Troubleshooting a Route Processor and Distributed Route Processor

Troubleshooting a Modular Services Card

Troubleshooting a Physical Layer Interface Module

Troubleshooting a Switch Fabric Card

Troubleshooting a Fan Controller Card

Alphanumeric LED Descriptions

Troubleshooting the Power Subsystems

Troubleshooting an AC Rectifier

AC Rectifier Indicators

Troubleshooting a DC Power Entry Module

PEM Indicators

Troubleshooting the Power Distribution System

Troubleshooting the Cooling Subsystem

Line Card Chassis Airflow

Cooling System Operation

Fan Controller Redundancy

Line Card Chassis Fan Tray

Line Card Chassis Fan Controller Card

Isolating Cooling Subsystem Problems

Troubleshooting Using the Alarm Module

Monitoring Critical, Major, and Minor Alarm Status

Alarm Module Alphanumeric Messages

Obtaining Documentation, Obtaining Support, and Security Guidelines


Cisco CRS-1 Carrier Routing System
16-Slot Line Card Chassis Hardware
Operations and Troubleshooting Guide

This guide assists users of the Cisco CRS-1 16-Slot Line Card Chassis with properly operating, maintaining, and troubleshooting their equipment. For documentation on planning for and installing the chassis and its components, see the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section for more information.

Note Throughout the remainder of this guide, the Cisco CRS-1 16-slot line card chassis is referred to as the line card chassis.

This guide includes the following sections:

•Document Conventions

•Warning Definition

•Changes to This Document

•Line Card Chassis Operations and Troubleshooting

•Troubleshooting the Installation

•Obtaining Documentation, Obtaining Support, and Security Guidelines

Document Conventions

This guide uses the following conventions:

Caution Means reader be careful. You are capable of doing something that might result in equipment damage or loss of data.

Note Means reader take note. Notes contain helpful suggestions or references to materials not contained in this manual.

Warning Definition

Warning

IMPORTANT SAFETY INSTRUCTIONS

This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. Use the statement number provided at the end of each warning to locate its translation in the translated safety warnings that accompanied this device. Statement 1071

SAVE THESE INSTRUCTIONS


See Regulatory Compliance and Safety Information for the Cisco CRS-1 Carrier Routing System for translations of warnings and information about the regulatory, compliance, and safety standards with which the Cisco CRS-1 system conforms.

Changes to This Document

Table 1 lists the technical changes made to this document since it was first released.

Table 1 Changes to This Document

Revision
Date
Change Summary

OL-9166-06

April 2012

Corrected Temperature Alarm display in Table 12 to resolve an open caveat.

OL-9166-05

November 2010

Added details about fan speeds and the redundant architecture of the cooling system.

OL-9166-04

May 2010

Added details of a second fan tray with different operating range.

OL-9166-03

September 2007

Corrected values in step 4 of the Line Card Chassis Power-Up Procedures section.

OL-9166-02

October 2006

Corrected PLIM LED status in Verifying the Installation of a Physical Layer Interface Module and Troubleshooting a Physical Layer Interface Module.

Removed reference to alphanumeric LEDs on PLIM.

OL-9166-01

April 2006

Initial release of the document.


Line Card Chassis Operations and Troubleshooting

This section describes various operations and troubleshooting procedures related to the line card chassis. This section includes the following topics:

•DC Power Up Preparation and Procedures

•Power-Related Site Preparation (DC System)

•Conductor Characteristics and Low-Voltage Drops

•Line Card Chassis Power-Up Procedures

•Verifying the Installation of System Cards

•Troubleshooting Overview

•Troubleshooting System Cards

•Alphanumeric LED Descriptions

•Troubleshooting the Power Subsystems

•Troubleshooting the Cooling Subsystem

•Troubleshooting Using the Alarm Module

DC Power Up Preparation and Procedures

This section describes the preparation for and procedures required to properly and successfully power up a DC-powered line card chassis. This section includes excerpts from the line card chassis installation and site planning guides. All applicable power-up preparation information is included in this section.

Note This section includes general, recommended procedures to power up a line card chassis. However, the procedures can be modified to accommodate special requirements.

These procedures provide the following benefits:

•Promote personal safety and reduce potential fire hazards

•Provide better signalling for telecommunications operations

•Minimize potential equipment damage and service interruptions

•Minimize radiated and conducted electromagnetic emissions and susceptibility

•Power up the line card chassis with no service interruptions

•Comply with all relevant safety standards

•Provide visual confirmation of the functionality of field-replaceable unit (FRU) components during the power-up procedure

Power-Related Site Preparation (DC System)

The DC-powered line card chassis requires a total of 12 dedicated 60-A DC input power connections to the power entry modules (PEMs) to provide redundant DC power to all six power zones of the chassis midplane. Six 60-A DC inputs to power shelf 0 (zero) are connected to battery A, and the other six inputs to power shelf 1 are connected to battery B.

The line card chassis DC power system provides 13,200 watts to the chassis. Each DC-powered chassis contains two DC power shelves for 2N redundancy. The shelves contain the input power connectors. Each DC power shelf contains three PEMs. The power shelves and PEMs are field-replaceable units (FRUs). Each shelf has its own disconnect switch and each PEM also has its own disconnect switch.

All DC-equipped line card chassis observe the following guidelines:

•All power connection wiring should conform to the rules and regulations in the National Electrical Code (NEC) and any local codes.

•The installer connects two dedicated 60-A service to each PEM installed in the power shelf.

•Each PEM connection is rated at 60-A maximum. A dedicated, commensurately rated DC power source is required for each PEM connection.

•Each PEM requires two 48-VDC inputs to power both zones. This requirement equals four wires for each PEM (two pairs), or 12 total wires (six pairs) for each power shelf and one shelf grounding wire.

•For DC power cables, we recommend the use of commensurately rated, high-strand-count copper wire cable. Each PEM requires two 48-VDC inputs, with four wires for each PEM, or 12 total wires (six pairs), plus the grounding wire. The length of the wires depends on the line card chassis location. These wires are not available from Cisco Systems. However, they are available from commercial vendors.

DC power cables must be terminated by cable lugs at the power shelf end. The lugs should be dual hole and able to fit over M6 terminal studs at 0.625-in (15.88-mm) centers.

Conductor Characteristics and Low-Voltage Drops

To avoid signal degradation, a conductor must be large enough to prevent its impedance from creating a voltage drop equal to 2 percent of the reference voltage. In addition, the protective earth conductor must be large enough to carry all the current if the 48-VDC return fails. A large enough protective earth conductor is required for safety reasons. Full fault redundancy is achieved by having conductors of equal size for the protective earth ground and the 48-VDC return of the switch.

For site preparation, proper wire size and insulation must be selected. For planned power distribution, calculations must be done before installation to meet proper voltage drop and temperature rise contingencies.

For wire gauges that prevent unacceptable voltage drops over different lengths of copper wire, see Table 2. For the resistance of 1000 feet of copper wire for each gauge of wire, see Table 3. These references are for planning purposes and might be further subject to local laws and practices.

Table 2 Wire Gauges Preventing Unacceptable Voltage Drops over Different Wire Lengths1

DC Current
25 Feet
7 Meters
50 Feet
15 Meters
75 Feet
22 Meters
100 Feet
30 Meters
150 Feet
45 Meters
200 Feet
60 Meters
400 Feet
121 Meters

5 A

18 gauge

14 gauge

14 gauge

12 gauge

10 gauge

8 gauge

6 gauge

10 A

14 gauge

12 gauge

10 gauge

8 gauge

8 gauge

6 gauge

2 gauge

15 A

14 gauge

10 gauge

8 gauge

8 gauge

6 gauge

4 gauge

2 gauge

20 A

12 gauge

8 gauge

8 gauge

6 gauge

4 gauge

2 gauge

0 gauge

25 A

12 gauge

8 gauge

6 gauge

4 gauge

4 gauge

2 gauge

0 gauge

30 A

10 gauge

8 gauge

6 gauge

4 gauge

2 gauge

2 gauge

00 gauge

35 A

10 gauge

6 gauge

4 gauge

2 gauge

2 gauge

1 gauge

000 gauge

40 A

8 gauge

6 gauge

2 gauge

2 gauge

2 gauge

0 gauge

000 gauge

45 A

8 gauge

6 gauge

4 gauge

2 gauge

1 gauge

0 gauge

0000 gauge

50 A

8 gauge

4 gauge

4 gauge

2 gauge

1 gauge

00 gauge

—

55 A

8 gauge

4 gauge

2 gauge

2 gauge

0 gauge

00 gauge

—

60 A

8 gauge

4 gauge

2 gauge

2 gauge

0 gauge

00 gauge

—

65 A

6 gauge

4 gauge

2 gauge

1 gauge

0 gauge

000 gauge

—

70 A

6 gauge

4 gauge

2 gauge

1 gauge

00 gauge

000 gauge

—

75 A

6 gauge

4 gauge

2 gauge

1 gauge

00 gauge

000 gauge

—

100 A

4 gauge

2 gauge

1 gauge

00 gauge

000 gauge

—

—

1 Table 1 is for reference only. Industry methods are used to determine the circular mils per gauge.
Example formula is 11.1 x amps x length divided by voltage drop. For example: 11.1 x 60 amps x 100 feet divided by 2 voltage
drop = 33,300 circular mils.


Table 3 Resistance for Each Gauge of Copper Wire  

Gauge
Ohms per 1000 Feet
Ohms per 304 Meters
Gauge
Ohms per 1000 Feet
Ohms per 304 Meters

0000

0.0489

10

0.9968

000

0.0617

11

1.2570

00

0.0778

12

1.5849

0

0.0980

13

1.9987

1

0.1237

14

2.5206

2

0.1560

15

3.1778

3

0.1967

16

4.0075

4

0.2480

17

5.0526

5

0.3128

18

6.3728

6

0.3944

19

8.0351

7

0.4971

20

10.1327

8

0.6268

21

12.7782

9

0.7908

22

16.1059


Line Card Chassis Power-Up Procedures

To prevent FRU damage and minimize power line disturbance to other equipment in the facility during line card chassis power up, follow these steps:

Step 1 Confirm that the facility power is disconnected from the chassis. However, make sure that all the disconnect switches in the chassis are in the on position (all on-off levers are pushed in). Six disconnect switch levers exist for the PEMs and two levers exist for the two DC power shelves.

Step 2 Confirm that all system cards are removed from the backplane. These system cards include the route processor and distributed route processor (RP and DRP), switch fabric cards (SFCs), physical layer interface modules (PLIMs), and other removable cards.

Caution Make sure to use proper ESD-preventive equipment and procedures. See the Cisco CRS-1 Carrier Routing System 16-Slot Line Card Chassis Installation Guide for proper tools and grounding locations.

Step 3 Connect the safety ground wiring to each power shelf, if not already completed.

Step 4 Before connecting the power cables to the stud terminals, verify the following resistance measurements:

a. Measure the resistance between each -48 VDC and the chassis (GND). The resistance must be greater than 100 Kohm. However, with all disconnect switches in the off position, verify that the resistance between each power terminal and the chassis (GND) is open. This measurement should be done with the battery and return grounded and the battery and return opened.

b. Measure the resistance between the two power terminals (between positive and negative) of each input. The resistance must be greater than 1 Kohm. However, with all disconnect switches in the off position, verify that the resistance between the positive and negative terminals are open.

c. With all disconnect switches in the ON position, measure the resistance between the positive terminals of all the inputs. This verifies that the PEM online insertion and removal (OIR) diode is not shorted. Each measurement must be done twice by reversing the multi-meter probe. The resistance must be greater than 0.1 Mohm.

Step 5 Turn off all disconnect switches (eight of them) by pulling out the on-off levers.

Step 6 Switch off all 12 facility circuit breaker switches.

Step 7 Connect each input power one at a time. Energize the facility breaker to that particular input. Measure the voltage at the input terminal block. Verify that the DC voltage between the negative terminal with respect to positive terminal is between -42 VDC and -72 VDC. Record the voltage measurements of all 12 inputs using Table 4.

Table 4 Voltage at Input Terminal Block (No Board Inserted)

Input Power
Upper Power Shelf (PS0)
Lower Power Shelf (PS1)

1

   

2

   

3

   

4

   

5

   

6

   

Caution This system has a positive ground. Connect the positive lead to the +RTN terminal and connect the negative lead to the -48-V terminal.

Step 8 Switch on all 12 facility circuit breaker switches.

Step 9 Switch on the power shelf disconnect switch to power shelf 0 (PS0). Verify that the PEM is not powered up. All visual LEDs should be inactive.

Step 10 Switch on the PEM A0 disconnect switch in power shelf 0. Verify that the Power OK LED on the front panel of the PEM is green. Verify that no other LEDs are active on the PEM. Verify also that the CB trip LED lights with the disconnect switch of the PEM in the off position and while the master disconnect switch on power shelf 1 (PS1) is in the off position. Table 5 lists the PEM status indicators.

Table 5 PEM Status Indicators

Indicator
Color
Indicates that

Power OK

Green

The PEM is operating normally in a powered-up condition.

Fault

Yellow

A fault is detected within the PEM.

DC Input Fail

Yellow

DC input is out of range or is not being provided to the PEM.

OT

Yellow

The PEM is in an over-temperature condition and shutdown has occurred

Breaker Trip

Yellow

The input breaker is in the off position.


Step 11 Repeat Step 10 for PEMs A1 and A2.

Step 12 Switch on the power shelf disconnect switch to power shelf 1 (PS1). Verify that the PEM in power shelf 1 is not powered up. All visual LEDs should be inactive.

Step 13 Repeat Step 12 for PEMs B0, B1, and B2.

Caution Before proceeding, make sure that at least one fan controller is installed and operating.
Caution Before proceeding, make sure to use proper ESD-preventive equipment and procedures. See the
Cisco CRS-1 Carrier Routing System 16-Slot Line Card Chassis Installation Guide for proper tools and grounding locations.

Step 14 Insert all the system cards in the chassis in the following sequence:

a. Install the fan controller in the upper cage of the front of the chassis from left to right. Note that the fan controller must be installed in the designated slots FC0 and FC1.

b. Install the route processor in the lower cage of the front of the chassis from left to right. Note that the route processor must be installed in the designated slots RP0 and RP1.

c. Install the four switch fabric cards in the lower cage of the rear of the chassis from right to left. Note that the switch fabric cards must be installed in the designated slots SM70, SM16, SM25, and SM34.

d. Install the four switch fabric cards in the upper cage of the rear of the chassis from right to left. Note that the switch fabric cards must be installed in the designated slots SM34, SM25, SM16, and SM07.

e. Install the PLIMs into the front of the chassis. The PLIMs can be installed into line card slots in any order.

f. Install the MSCs into the rear of the chassis. The MSCs must be inserted into the slot corresponding to the slot number of PLIMs.

Verifying the Installation of System Cards

This section describes the steps needed to confirm the proper installation of the various system cards. This sections includes:

•Verifying the Installation of a Route Processor and Distributed Route Processor Card

•Verifying the Installation of a Modular Services Card

•Verifying the Installation of a Physical Layer Interface Module

•Verifying the Installation of a Switch Fabric Card

•Verifying the Installation of a Fan Controller Card

Verifying the Installation of a Route Processor and Distributed Route Processor Card

This section describes how to verify the installation of a route processor and distributed route processor (RP and DRP) card in the line card chassis. Figure 1 shows the front panel of the standard RP card.

Figure 1 RP Card Front Panel

1

Console port

6

Control Ethernet1 port

2

AUX port

7

Management Ethernet port

3

Hard disk drive slot

8

Alphanumeric LEDs

4

PC Card slot

9

Status LED

5

Control Ethernet0 port

   

To verify the installation of an RP and DRP card, follow these steps:

Step 1 Confirm that the card is seated firmly in the chassis slot. One easy way to verify physical installation is to see whether the front faceplate of the RP and DRP card is even with the front of the other cards installed in the card cage.

Step 2 Confirm that the ejector levers are latched and the captive screws are fastened properly. If you are uncertain, unlatch the levers, loosen the screws, and attempt to reseat the RP and DRP card.

Step 3 Examine the power shelves to see whether the chassis, as a whole, is receiving power.

Step 4 Use the Status LED, located on the RP and DRP card faceplate, to verify the correct installation of the card:

a. Green—The card is correctly installed and operating properly.

b. Blinking yellow—A problem exists on the card.

c. Off—The card status is unknown. Verify that the card is installed correctly. Verify also that there is power to the card by looking at the indicators on the power shelf.

d. When the RP is in primary mode (alphanumeric LED displays ACTV RP), the board is executing control processing functions and is not in a secondary or standby role.

e. The primary RP and DRP is functioning correctly when the alphanumeric LED displays "ACTV RP."

f. The secondary RP and DRP is functioning correctly when the alphanumeric LED displays "STBY RP."

Note Allow adequate time to boot up the RP and DRP. It takes 5 to 7 minutes to boot up each RP and DRP.

Verifying the Installation of a Modular Services Card

This section describes how to verify the installation of a modular services card (MSC) located in slots 0 through 15 on the rear of the line card chassis. Figure 2 shows the front panel of the MSC.

Figure 2 MSC Front Panel

1

Status LED

2

Alphanumeric LEDs


To verify the installation of an MSC, follow these steps:

Step 1 Confirm that the card is seated firmly in the line card chassis slot. One easy way to verify physical installation is to see whether the front faceplate of the MSC card is even with the front of the other cards installed in the card cage.

Step 2 Confirm that the ejector levers are latched and the captive screws are fastened properly. If you are uncertain, unlatch the levers, loosen the screws, and attempt to reseat the MSC.

Step 3 Examine the power shelves to see whether the chassis, as a whole, is receiving power.

Step 4 Use the status LED, located on the MSC faceplate, to verify the correct installation of the card:

a. Green—The card is correctly installed and operating properly.

b. Blinking yellow—A problem exists on the card.

c. Off—The card status is unknown. Verify that the card is installed correctly. Verify also that there is power to the card by looking at the indicators on the power shelf.

d. The MSC is functioning correctly if the alphanumeric LED displays "IOS-XR."

Verifying the Installation of a Physical Layer Interface Module

This section describes how to verify the installation of a physical layer interface module (PLIM) located in slots 0 through 15 on the front of the line card chassis. Figure 3 shows a sample PLIM front panel.

Figure 3 PLIM Front Panel (1-Port OC-768 Shown)

1

TX connector port

3

LED array (ACTIVE, CARRIER, and RX PKT)

2

RX connector port

4

Status LED


To verify the installation of a PLIM, follow these steps:

Step 1 Confirm that the card is seated firmly in the line card chassis slot. One easy way to verify physical installation is to see whether the front faceplate of the PLIM card is even with the front of the other cards installed in the card cage.

Step 2 Confirm that the ejector levers are latched and the captive screws are fastened properly. If you are uncertain, unlatch the levers, loosen the screws, and attempt to reseat the PLIM.

Step 3 Examine the power shelves to confirm that the entire chassis is receiving power.

Step 4 Using the status LED on the PLIM faceplate, verify the correct installation and operation of the card:

a. Green—The card is correctly installed and operating properly.

b. Yellow—A problem exists on the card.

c. Off—The card status is unknown. Verify that the card is installed correctly. Verify also that there is power to the card by looking at the indicators on the power shelf.

Verifying the Installation of a Switch Fabric Card

This section describes how to verify the installation of a switch fabric card (SFC). Figure 4 shows the SFC front panel.

Figure 4 SFC Front Panel

1

Status LED

2

Alphanumeric LEDs


To verify the installation of an SFC, follow these steps:

Step 1 Confirm that the card is seated firmly in the line card chassis slot. One easy way to verify physical installation is to see whether the front faceplate of the SFC is even with the front of the other cards installed in the card cage.

Step 2 Confirm that the ejector levers are latched and the captive screws are fastened properly. If you are uncertain, unlatch the levers, loosen the screws, and attempt to reseat the SFC.

Step 3 Examine the power shelves to confirm that the entire chassis is receiving power.

Step 4 Use the status LED, located on the front faceplate, to verify the correct installation of the card:

a. Green—The card is correctly installed and operating properly.

b. Blinking yellow—A problem exists on the card.

c. Off—The card status is unknown. Verify that the card is installed correctly. Verify also that there is power to the card by looking at the indicators on the power shelf.

d. The SFC is functioning correctly if the alphanumeric LED displays "IOS-XR."

Verifying the Installation of a Fan Controller Card

This section describes how to verify the installation of a fan controller (FC) card located in the front of the line card chassis. Figure 5 shows the FC card front panel.

Figure 5 FC Card Front Panel

1

EXT CLK 1 port (external clock)1

3

Alphanumeric LEDs

2

EXT CLK 2 port (external clock)1

4

Status LED

1 This feature is not currently supported.


To verify the installation of an FC card, follow these steps:

Step 1 Confirm that the card is seated firmly in the line card chassis slot. One easy way to verify physical installation is to see whether the front faceplate of the FC is even with the front of the other cards installed in the card cage.

Step 2 Confirm that the ejector levers are latched and the captive screws are fastened properly. If you are uncertain, unlatch the levers, loosen the screws, and attempt to reseat the FC card.

Step 3 Examine the power shelves to confirm that the entire chassis is receiving power.

Step 4 Use the status LED, located on the front faceplate, to verify the correct installation of the card:

a. Green—The card is correctly installed and operating properly.

b. Blinking yellow—A problem exists on the card.

c. Off—The card status is unknown. Verify that the card is installed correctly. Verify also that there is power to the card by looking at the indicators on the power shelf.

d. The FC card is functioning correctly if the alphanumeric LED displays "IOS-XR."

Troubleshooting the Installation

This section contains general troubleshooting information to help isolate the cause of difficulties you might encounter during the installation and initial startup of the line card chassis.

It is assumed with the procedures in this section that you are troubleshooting the initial startup of the line card chassis and that the system is running the original configuration. If you altered the original hardware configuration or changed any default configuration settings, the recommendations in this section may not apply.

Troubleshooting procedures are presented in the following sections:

•Troubleshooting Overview

•Troubleshooting System Cards

•Alphanumeric LED Descriptions

•Troubleshooting the Power Subsystems

•Troubleshooting the Cooling Subsystem

•Troubleshooting Using the Alarm Module

Troubleshooting Overview

This section describes the methods used in troubleshooting the line card chassis. The troubleshooting methods are organized according to the major subsystems in the line card chassis.

If you are unable to solve a problem on your own, you can consult a Cisco customer service representative for assistance. When you call, have the following information ready:

•Date you received the line card chassis and the chassis serial number (located on a label on the chassis).

•Installed system cards.

–Use the show hardware command to determine which cards are installed.

•Cisco IOS XR software release number.

–Use the show version command to determine this number.

•Brief description of the symptoms and steps you have taken to isolate and solve the problem.

•Maintenance agreement or warranty information.

Troubleshooting Using a Subsystem Approach

To solve a problem, try to isolate the problem to a specific subsystem. Compare current line card chassis behavior with expected line card chassis behavior. Because a startup issue is usually attributable to one component, it is most efficient to examine each subsystem rather than trying to troubleshoot each line card chassis component.

For troubleshooting purposes, the line card chassis consists of the following subsystems:

•Power subsystem—Includes the following components:

–AC-input rectifiers (Wye or Delta)

–DC-input power entry modules (PEMs)

•Processor subsystem—Includes redundant route processors (RPs and DRPs), modular services cards (MSCs) with physical layer interface modules (PLIMs), switch fabric cards (SFCs), and fan controller cards. The RPs and DRPs and MSCs are equipped with onboard processors. The RP and DRP downloads a copy of the Cisco IOS XR software image to each MSC processor. The system uses an alphanumeric display (on each MSC and RP and DRP) to display status and error messages, which can help in troubleshooting.

Note MSCs and PLIMs are interdependent devices. One cannot operate without the other. However, when troubleshooting you must look at each component for possible problems.

•Cooling subsystem—Consists of the fan tray, which circulates air through the card cages to cool the cards, and a fan in each of the power modules, which circulates cool air through the power module.

Normal Line Card Chassis Startup Sequence

You can generally determine when and where the line card chassis failed during the startup sequence by checking the status and alphanumeric LEDs located on all major components.

In a normal line card chassis startup sequence, the following sequence of events and conditions occurs:

1. The fans in the blower module receive power and begin drawing air through the chassis.

2. The fan in each power module (AC or DC) receives power and begins drawing air through the power module.

3. As the power-up and boot process progresses for the RP and DRP, SFC, FC card, and each installed MSC and PLIM, the status of each card appears on the status and alphanumeric LEDs on the front panel of each card.

Identifying Startup Issues

Table 6 shows the state of the status and alphanumeric LEDs after a successful system startup.

Table 6 Status and Alphanumeric LEDs at System Startup 

Component
Type of Indicator
Display Contents and LED Status and Meaning

RP and DRP

Alphanumeric display

Upper row: ACTV or STBY
Lower row: RP

The active or standby RP and DRP is enabled and recognized by the system. A valid Cisco IOS XR software image is running.

Status LED

Illuminated (Green)

PLIM

Status LED

Illuminated (Green)

MSC

Alphanumeric display

Upper row: IOS-
Lower row: XR

The MSC is enabled and ready for use.

Status LED

Illuminated (Green)

SFC

Alphanumeric display

Upper row: IOS-
Lower row: XR

The SFC is enabled and ready for use.

Status LED

Illuminated (Green)

Alarm module

Alphanumeric display

Upper row: IOS-
Lower row: XR

The alarm module is enabled and ready for use.

LEDs (two arrays)

Critical: Off
Major: Off
Minor: Off

Note: The Critical, Major, and Minor LED arrays are redundant. Two sets of LED arrays are present in case any of the LEDs malfunctions.

DC power entry module (PEM)

Power status

PWR OK: On
FAULT: Off
DC INPUT FAIL: Off
OT: Off
BREAKER TRIP: Off

The correct power module voltages are present and no faults are detected.

AC rectifier

Power status

PWR OK: On
FAULT: Off
AC INPUT FAIL: Off
OT: Off
BREAKER TRIP: Off
ILIM: Off

The correct power module voltages are present and no faults are detected.

Fan controller

Alphanumeric display

Upper row: IOS-
Lower row: XR

The fan controller is enabled and ready for use.

Status LED

Illuminated (Green)


Troubleshooting System Cards

If problems occur after verifying the installation of a system card, use this section to determine the cause and fix the problem. The following subsections are included:

•Troubleshooting a Route Processor and Distributed Route Processor

•Troubleshooting a Modular Services Card

•Troubleshooting a Physical Layer Interface Module

•Troubleshooting a Switch Fabric Card

•Troubleshooting a Fan Controller Card

Troubleshooting a Route Processor and Distributed Route Processor

If a RP and DRP fails to operate or power up, review this information:

•Make sure that the card is seated firmly in the slot. One easy way to verify physical installation is to see whether the front faceplate of the RP and DRP is even with the fronts of the other cards installed in the card cage.

•Check whether the ejector levers are latched and that the captive screws are fastened properly. If you are uncertain, unlatch the levers, loosen the screws, and attempt to reseat the RP and DRP.

•Examine the alarm module to see if there are any active alarm conditions.

•Examine the power shelves to see whether the chassis, as a whole, is receiving power.

Use the status LED on the RP and DRP faceplate to verify the correct installation of the card:

•When the card is properly installed, the Status LED turns green.

•When the Status LED is blinking yellow, a hardware or communications problem exists on the card.

•When the Status LED is off, the board status is unknown. Verify that there is power to the board by looking at the indicators on the power shelf.

•If there is a failure during the board boot sequence, the four-digit alphanumeric display indicates the current boot phase to assist you in debugging the board failure.

Troubleshooting a Modular Services Card

If a modular services card (MSC) fails to operate or power up, review this information:

•Make sure that the card is seated firmly in the slot. One easy way to verify physical installation is to see whether the front faceplate of the MSC is even with the fronts of the other cards installed in the card cage.

•Check whether the ejector levers are latched and the captive screws are fastened properly. If you are uncertain, unlatch the levers, loosen the screws, and attempt to reseat the MSC.

•Examine the alarm module to see if there are any active alarm conditions.

•Examine the power shelves to see whether the chassis, as a whole, is receiving power.

Use the status LED on the MSC faceplate to verify the correct installation of the MSC:

•When the card is properly installed, the Status LED turns green.

•When the Status LED is blinking yellow, a hardware or communications problem exists on the card.

•When the Status LED is off, the board status is unknown. Verify that there is power to the board by looking at the indicators on the power shelf.

•If there is a failure during the board boot sequence, the four-digit alphanumeric display indicates the current boot phase to assist you in debugging the board failure.

Troubleshooting a Physical Layer Interface Module

If a physical layer interface module (PLIM) fails to operate or power up, review this information:

•Make sure that the PLIM is seated firmly in the slot. One easy way to verify physical installation is to see whether the front faceplate of the PLIM is even with the fronts of the other PLIMs installed in the card cage.

•Check whether the ejector levers are latched and that the captive screws are fastened properly. If you are uncertain, unlatch the levers, loosen the screws, and attempt to reseat the PLIM.

•Examine the alarm module to see if there are any active alarm conditions.

•Examine the power shelves to see whether the chassis, as a whole, is receiving power.

Use the LEDs on the PLIM faceplate to verify the correct installation and operation of the card:

•During power up of the card, the Status LED is yellow.

•When the card is properly installed and the state changes to IOS-XR, the Status LED turns green.

•When the Status LED is yellow, a hardware or communications problem exists on the card.

•When the Status LED is off, the board status is unknown. Verify that there is power to the board by looking at the indicators on the power shelf.

•If there is a failure during the board boot sequence, the four-digit alphanumeric display on the MSC indicates the current boot phase to assist you in debugging the board failure.

•The LED array (three or five green LEDs) indicates the status of each port:

•ACTIVE—The port is logically active; the laser is on.

•CARRIER—The receive port (RX) is receiving a carrier signal.

•RX PKT—This LED blinks every time a packet is received.

•WRAP—The port is operating in DPT wrapped mode.

•PASS THRU—The port is operating in PoS mode (DPT pass-through).

Troubleshooting a Switch Fabric Card

If a switch fabric card (SFC) fails to operate or power up, review this information:

•Make sure that the card is seated firmly in the slot. One easy way to verify physical installation is to see whether the front faceplate of the SFC is even with the fronts of the other cards installed in the card cage.

•Check whether the ejector levers are latched and that the captive screws are fastened properly. If you are uncertain, unlatch the levers, loosen the screws, and attempt to reseat the SFC.

•Examine the alarm module to see if there are any active alarm conditions.

•Examine the power shelves to see whether the chassis, as a whole, is receiving power.

Use the status LED on the SFC faceplate to verify the correct installation of the card:

•When the card is properly installed, the Status LED turns green.

•When the Status LED is blinking yellow, a hardware or communications problem exists on the card.

•When the Status LED is off, the board status is unknown. Verify that there is power to the board by looking at the indicators on the power shelf.

•If there is a failure during the board boot sequence, the four-digit alphanumeric display indicates the current boot phase to assist you in debugging the board failure.

Troubleshooting a Fan Controller Card

If a fan controller (FC) card fails to operate or power up, review this information:

•Make sure that the card is seated firmly in the slot. One easy way to verify physical installation is to see whether the front faceplate of the FC card is even with the fronts of the other cards installed in the card cage.

•Check whether the ejector levers are latched and the captive screws are fastened properly. If you are uncertain, unlatch the levers, loosen the screws, and attempt to reseat the FC card.

•Examine the alarm module to see if there are any active alarm conditions.

•Examine the power shelves to see whether the chassis, as a whole, is receiving power.

Use the status LED on the FC card faceplate to verify the correct installation of the card:

•When the card is properly installed, the Status LED turns green.

•When the Status LED is blinking yellow, a problem exists on the board.

•When the Status LED is off, the board status is unknown. Verify that there is power to the board by looking at the indicators on the power shelf.

•If there is a failure during the board boot sequence, the two four-digit alphanumeric display indicates the current boot phase to assist you in debugging the board failure.

Alphanumeric LED Descriptions

Each system card includes an alphanumeric LED display consisting of four characters arranged in two rows (upper and lower). Table 7 lists the alphanumeric LED messages that may appear on any system card.

Table 7 Alphanumeric LED Messages 

Message
Description

Upper row: SP
Lower row: RMON

Indicates that the SP processor of the CPU is booting up from ROMMON. This is the first step in card bringup.

Upper row: ACTV
Lower row: RP

Indicates that this is the active RP.

Note: Only appears on the RP and DRP

Upper row: STBY
Lower row: RP

Indicates that this is the standby RP.

Note: Only appears on the RP and DRP

Upper row: IOS-
Lower row: XR

Indicates that the node is up and running.

Upper row: PRES
Lower row: ENT

Indicates that the card is plugged in but is not booted (probably due to a configuration or some other issue with the card).

Upper row: IN-R
Lower row: ESET

Indicates that the card either encountered critical alarms and therefore was shutdown or failed the boot process and hence was shut down. Manual intervention is required to recover the card.

Upper row: ROMM
Lower row: ON

Indicates the software state prior to the card booting.

Upper row: MBI-
Lower row: BOOT

Indicates that the initial minimum boot image is loading on the card.

Upper row: MBI-
Lower row: RUN

Indicates that the minimum boot image (MBI) is running and pulling the appropriate software packages from the active RP to the remote nodes.

Upper row: BRIN
Lower row: GDOW

Indicates that the node is down (due to a user configuration or some other error).

Upper row: IOS-
Lower row: XR F

Indicates an error that should shut down the card. However, the user has overridden the sequence through a configuration change. Hence, the fail state indicates an error on the card.


Troubleshooting the Power Subsystems

This section contains information to troubleshoot the power subsystems:

•Troubleshooting an AC Rectifier

•Troubleshooting a DC Power Entry Module

•Troubleshooting the Power Distribution System

Troubleshooting an AC Rectifier

Three AC rectifiers are mounted inside the AC power shelf. Figure 6 shows an AC rectifier.

Figure 6 AC Rectifier

A service processor (SP) module in the power shelf monitors the status of each AC rectifier. The service processor communicates with the system controller on the route processor (RP). The service processor circuitry monitors the following AC rectifier fault and alarm conditions:

•Fault—Indicates a failure in an AC rectifier, such as a failed bias supply, or an over-temperature or over-current limit condition. This indication also includes a warning that the DC output is beyond the allowable output range.

•AC input fail—Indicates that the AC input voltage is out of range.

•Circuit breaker trip—Indicates that the AC rectifier circuit breaker has tripped.

•Over temperature—Indicates that the AC rectifier has exceeded the maximum allowable operating temperature.

•AC rectifier present—Indicates that the rectifier is present and seated properly in the power shelf.

•Voltage and current monitor signals (Vmon, Imon)—Indicates that the output voltages and currents provided by the AC rectifier are within range.

Each AC rectifier contains an ID EEPROM that stores information used by control software (for example, part number, serial number, assembly deviation, special configurations, test history, and field traceability data).

AC Rectifier Indicators

Each AC rectifier has power and status indicators. The AC rectifier indicators receive power from both AC power shelves; therefore, the indicators are operational even when the AC rectifier is not powered from its input voltage.

Table 8 lists the AC rectifier status indicators and their functions. Table 9 lists the LED readings during failure conditions.

Table 8 AC Rectifier Status Indicators 

Name
Color
Function

PWR OK

Green

The AC rectifier is operating normally with power.

FAULT

Yellow

A fault has been detected in the AC rectifier.

AC INPUT FAIL

Yellow

AC input is out of range or not being provided to the AC rectifier.

OT

Yellow

The AC rectifier is overheated and has been shut down.

BREAKER TRIP

Yellow

The input circuit breaker is off (in the off position).

ILIM

Yellow

The AC rectifier is operating in a current-limiting condition.


Table 9 AC Rectifier LED Failure Conditions 

Condition
PWR OK LED
FAULT LED
AC INPUT FAIL LED
OT LED
BREAKER TRIP LED
ILIM LED

No fault
(power is on)

On

Off

Off

Off

Off

Off

Failed AC power

Off

Off

On

Off

Off

Off

Overheated temperature

Off

On

Off

On

Off

Off

Tripped breaker

Off

Off

Off

Off

On

Off

Current limit

Off

Off

Off

Off

Off

On


To troubleshoot an AC rectifier, follow these steps:

Step 1 Make sure that the AC rectifier is seated properly in the AC power shelf by removing and reseating the AC rectifier, using the instructions in the line card chassis installation guide.

Step 2 Make sure that the line card chassis is powered on and all power cords are connected properly:

•Power cords on the back panel of the power shelf are secured in place with their retention clips.

•Power cords at the power source end are securely plugged into their own AC power outlet.

•The source AC circuit breaker is switched on.

Step 3 Check the AC rectifier status LED indicators, using Table 8 and Table 9 references.

–Remove and then apply power to the AC rectifier by disconnecting its power cord.

–If the indicator remains on, reseat the AC rectifier.

–If the indicator still remains on, replace the AC rectifier.

•OT (yellow)—Indicates that the AC rectifier is in an over-temperature condition, causing a shutdown to occur.

Note When the OT indicator is on or flashing, the fault indicator also goes on.

–Verify that the AC rectifier fan is operating properly.

–Verify that the blower modules are operating properly.

–If the AC rectifier fan and blower modules are operating properly, replace the existing AC rectifier with a spare.

•ILIM (yellow)—Indicates the AC rectifier is operating in a current-limiting condition.

–Make sure that each power cord is connected to a dedicated AC power source.

–Make sure that each AC rectifier, which requires a minimum service of 20 A in North America (13 A internationally), is operating in the nominal range of 200 to 240 VAC.

Because redundant power supplies exist, a problem with the DC output voltage to the backplane from only one AC rectifier should not affect line card chassis operation. Because the line card chassis is equipped with multiple AC rectifiers, it powers on and operates even when one AC rectifier fails.

Troubleshooting a DC Power Entry Module

Three DC power entry modules (PEMs) are mounted inside the DC power shelf. Figure 7 shows a DC PEM.

Figure 7 DC PEM

A service processor module (in the power shelf) monitors each PEM and reports the status to the system controller function on the route processor. The service processor detects whether the PEM is present. It also monitors PEM output voltages and current and fault and alarm conditions.

Each PEM contains an ID EEPROM that stores information used by control software (for example, part number, serial number, assembly deviation, special configurations, test history, and field traceability data).

PEM Indicators

Each PEM has power and status indicators. PEM indicators are powered by both DC power shelves; therefore, the indicators are operational even when the PEM is not being powered from its input voltage. Table 10 lists the PEM status indicators and their functions. Table 11 lists the conditions of the LEDs under certain failure conditions.

Table 10 PEM Status Indicators  

Name
Color
Function

PWR OK

Green

The PEM is operating normally with power.

FAULT

Yellow

A PEM fault was detected (for example, failed bias supply, over-temperature or over-current, or out-of-range DC output).

DC INPUT FAIL

Yellow

No DC input exists to the PEM, or DC input is out of range.

OT

Yellow

The PEM is overheated and has been shut down.

BREAKER TRIP

Yellow

The circuit breaker has tripped and is in the off position.


Table 11 PEM LED Conditions

Condition
PWR OK LED
FAULT LED
DC INPUT FAIL LED
OT LED
BREAKER TRIP LED

No fault
(power is on)

On

Off

Off

Off

Off

Failed DC power

Off

Off

On

Off

Off

Overheated temperature

Off

On

Off

On

Off

Tripped breaker

Off

Off

Off

Off

On


To troubleshoot a DC PEM, follow these steps:

Step 1 Make sure that the DC PEM is seated properly in the DC power shelf by removing and reseating the DC PEM, using the instructions in the line card chassis installation guide.

Step 2 Make sure that the line card chassis is powered on and all power cords are connected properly:

•Power cords on the back panel of the power shelf are secured in place with their retention clips.

•Power cords at the power source end are securely plugged into their own DC power outlet.

•The source DC circuit breaker is switched on.

Step 3 Check the DC PEM status LED indicators, using Table 10 and Table 11 as references.

–Remove and then apply power to the DC PEM by disconnecting its power cord.

–If the indicator remains on, reseat the DC PEM.

–If the indicator still remains on, replace the DC PEM.

•OT (yellow)—Indicates that the DC PEM is in an over-temperature condition, causing a shutdown.

Note When the OT indicator is on or flashing, the fault indicator also goes on.

–Verify that the DC PEM fan is operating properly.

–Verify that the blower modules are operating properly.

–If the DC PEM fan and blower modules are operating properly, replace the existing DC PEM with a spare.

•ILIM (yellow)—Indicates the DC PEM is operating in a current-limiting condition.

–Make sure that each power cord is connected to a dedicated DC power source.

–Make sure that each DC PEM, which requires a minimum service of 20 A in North America (13 A internationally), is operating in the nominal range of 200 to 240 VDC

Because redundant power supplies exist, a problem with the DC output voltage to the backplane from only one PEM should not affect line card chassis operation. Because the line card chassis is equipped with multiple DC PEMs, it powers on and operates even when one PEM fails.

Troubleshooting the Power Distribution System

To troubleshoot the power distribution system, follow these steps.

Step 1 Check each AC rectifier or DC PEM to make sure that the:

•Ejector lever is fully closed and properly secured by its captive screw.

•PWR OK indicator is on.

•FAULT indicator is off.

•ILIM indicator is off

•OT indicator is off.

If the power supplies meet the preceding criteria, then the correct source power is present and within tolerance. The power supplies are functioning properly.

Step 2 Make sure that the blower modules are operating.

•If the blower modules are functioning, then the power to the blower modules is functioning properly.

•If a blower module is not functioning, a problem may exist with either the blower module itself or the power supplied to the blower module. Eject and reseat the blower module.

•If a blower module is still not operating, a problem could exist with the blower module controller card or cable. Replace the blower module. Also see the "Troubleshooting the Cooling Subsystem" section.

Troubleshooting the Cooling Subsystem

The cooling system dissipates the heat generated by the line card chassis and controls the temperature of chassis components. The cooling system has a fully redundant architecture that allows the chassis to continue operating with a single-fault failure (such as a single fan or fan tray failure). See the "Fan Controller Redundancy" section for more information. The architecture also supports a redundant load-sharing design.

The complete chassis cooling system includes:

•Two fan trays (each tray holds nine fans) and two fan controller cards

•Temperature sensors on cards and modules throughout the chassis

•Control software and logic

•Air filter, inlet and outlet air vents, and bezels

•Impedance carriers for empty chassis slots

The AC rectifiers and DC PEMs in the power shelves also have their own self-contained cooling fans.

All nine fans in a fan tray operate as a group. So, if it is necessary to increase or decrease airflow, all of the fans in the tray increase or decrease their rotation speed together. When two fan trays are operational in a chassis, the speed of the fans in both trays is adjusted together.

Thermal sensors (inlet, exhaust, and hot-spot) located throughout the line card chassis are used to monitor temperature readings and identify when the system is not cooling properly.

Software running on several types of service processor (SP) modules is used to control the operation of the fans. These SP modules are connected by internal Ethernet to the system controller on the route processor (RP).

Line Card Chassis Airflow

The airflow through the line card chassis is controlled by a push-pull configuration (see Figure 8). The bottom fan tray pulls in ambient air from the bottom front of the chassis, and the top fan pulls the air up through the card cages and exhausts the warm air from the top rear of the chassis.

Figure 8 Airflow Through Line Card Chassis

Note The line card chassis has a maximum airflow of 2050 cubic feet per minute.

The chassis has a replaceable air filter mounted in a slide-out tray above the lower fan tray. The line card chassis air filter, shown in Figure 9, plugs into the rear (MSC) side of the chassis.

You should change the air filter as often as necessary. In a dirty environment or when you start getting frequent temperature alarms, check the intake grills for debris and check the air filter to see if it needs to be replaced. Before removing the air filter for replacement, you should have a spare filter on hand. Then, when you remove the dirty filter, install the spare filter in the chassis.

Figure 9 Air Filter

Note A lattice of wire exists on both sides of the air filter with an arrow that denotes airflow and a pair of sheet metal straps on the downstream side of the filter assembly.

Cooling System Operation

The fan control software and related circuitry varies the DC input voltage to individual fans to control their speed. The fan control software and related circuitry increases or decreases the airflow needed to keep the line card chassis operating in a desired temperature range. The chassis cooling system uses multiple fan speeds to optimize cooling, acoustics, and power consumption. Four normal operating fan speeds exist, and one high-speed fan setting is used when a fan tray has failed.

At initial power up, control software powers on the fans to 4300 to 4500 RPM. The fans provide airflow during system initialization and software boot and ensures that there is adequate cooling for the chassis in case the software hangs during boot up. The fan control software is initialized after the routing system software boots up, which can take 3 to 5 minutes. The fan control software then adjusts the fan speeds appropriately.

During normal operation, the chassis averages the temperatures reported by inlet temperature sensors in the lower card cage (or in the upper card cage when the lower cage is empty). To determine the appropriate fan speed for the current temperature, the fan control software compares the average inlet temperature to a lookup table that lists the optimal fan speed for each temperature. The software then sets the fan speed to the appropriate value for the current temperature. The temperature ranges in the lookup table overlap to ensure a proper margin to avoid any type of fan speed oscillation occurring between states.

Note When there are no active alarms or failures, the fan control software checks temperature sensors every 1 to 2 minutes.

Thermal Alarms

Local thermal sensors on individual cards monitor temperatures and generate a thermal alarm when the cooling system is not cooling properly. A temperature sensor might be tripped in response to elevated ambient air temperature, a clogged air filter or other airflow blockage, or a combination of these causes. A fan failure causes a fault message, but if no thermal sensors have been tripped, the fan control remains unchanged.

When a thermal sensor reports a thermal alarm, the sensor passes the fault condition to its local service processor (SP), which then notifies the system controller on the route processor (RP). The system controller passes the fault condition to the SP on each fan controller board. The fan control software then takes appropriate action to resolve the fault.

When a thermal sensor is tripped, the fan control software tries to resolve the problem (for example, by increasing fan speed). The software performs a series of steps to prevent chassis components from getting anywhere near reliability-reducing, chip-destroying temperatures. If the fault continues, the software shuts down the card or module to save components.

Quick-Shutdown Mode

The fan controller cards and fan trays have a quick-shutdown mode that kills power when a card or fan tray is disengaged from the chassis midplane. The quick-shutdown mode minimizes inrush current during a hot swap or online insertion and removal (OIR). In normal maintenance conditions, the software gracefully shut downs the power to the failed part, allowing ample time for capacitors to discharge.

Fan Controller Redundancy

This section describes the redundant architecture of the cooling system, which allows the cooling system to continue operating even when certain components have failed.

Both fan controller cards work together to provide fully redundant input power and control logic for fan trays and fans. Each fan controller card receives its input power (-48-VDC) from both the A and B power shelves. The fan controller card then provides one fan tray with input power from the A bus and provides power to the other fan tray from the B bus. This feature ensures that the upper fan tray is powered from the A bus on one fan controller card and from the B bus on the second fan controller card.

In a fully redundant system-one that is equipped with dual power feeds, dual fan controller cards, and dual fan trays-the cooling system can withstand the failure of any one of the following components and still continue to properly cool the chassis:

•Fan tray—If one fan tray fails or is removed, the other fan tray automatically speeds up to the maximum limit and provides cooling for the entire chassis. (If multiple fans in a single fan tray fail, the remaining fans in the two fan trays provide cooling for the entire chassis.)

•Fan controller card—If one fan controller card fails, the other fan controller card provides all of the power to the fan trays. In this mode, the single remaining fan controller card provides a maximum of 24 VDC.

•Power shelf or power module (DC PEM or AC rectifier)—If one power feed fails, the other power feed provides all of the power to the fan trays.

In the single-failure cases described in this section, the rotational speed of the remaining operational fans changes automatically according to the cooling needs of the chassis.

A double-fault fan failure involves two fan trays, two fan tray boards, two fan controller cards, two power shelves, two power modules (DC PEMs or AC rectifiers), or any combination of two of these components. When a double-fault failure occurs, the system can automatically power down individual cards if the cooling power is insufficient to maintain them. The chassis remains powered on unless both fan trays have failed or thermal alarms indicate a problem serious enough to power down the entire chassis.

Note When a cooling system component fails, it should be replaced within a 24-hour period (or sooner).

For information on the rotational speeds of the fans in revolutions per minute (RPM), see the Line Card Chassis Fan Tray section.

Line Card Chassis Fan Tray

Figure 10 shows a fan tray, which plugs into the rear (MSC) side of the chassis. Each fan tray is hot-swappable and considered a field-replaceable unit. The chassis is designed to run with both fan trays in place.

Figure 10 Fan Tray

Each fan tray contains:

•Nine fans—Each fan uses a nominal +24 VDC as its input power. This voltage is adjusted to increase or decrease the speed of the fan. Two DC-to-DC converters, one on each fan controller card, provide input power to a single fan.

•Fan tray board—The board terminates signals to and from the fans, filters common-mode noise, and contains tracking and indicator parts.

•Front-panel status LED—The LED indicates the following:

–Green—The fan tray is operating normally.

–Yellow—The fan tray has failed and should be replaced.

–Off—An unknown state exists or the LED is faulty.

During normal operation, the fans in CRS-16-LCC-FAN-TR= operate in the range of 4000 to 5150 RPM and the fans in CRS-16-LCC-FNTR-B= operate in the range of 3300 to 5150 RPM. The system automatically adjusts the speed of the fans to meet the cooling needs of the entire chassis. If one fan controller card or one power feed fails, the fans continue to operate within the ranges specified above (up to 5150 RPM). If one fan tray fails completely, or is removed, the fans in the remaining fan tray automatically speed up to the maximum rotational limit, which is 6700 RPM for CRS-16-LCC-FAN-TR= and 6600 RPM for CRS-16-LCC-FNTR-B=.

Note The fan speed range limits listed in this document are nominal. These numbers have a tolerance of plus or minus 10 percent.

Line Card Chassis Fan Controller Card

A line card chassis contains two fan controller (FC) cards, shown in Figure 11.

Figure 11 FC Card Front Panel

1

EXT CLK 1 port (external clock)1

3

Alphanumeric LEDs

2

EXT CLK 2 port (external clock)1

4

Status LED

1 This feature is not currently supported.


The FC cards provide the following functions:

•Converts -48-VDC from the midplane to the DC voltages needed to operate the fans.

•System control and communications with the system controller function on the RPs, through a service processor module.

•Communicates inlet temperature and thermal alarms to the fan controller SP module from the system controller on the RP. The chassis uses three types of temperature sensors: inlet, exhaust, and hot spot. Any of these sensors can send a thermal alarm.

•Monitors individual fan tachometer signals from the fan tray.

•Status LED (good/bad) for each fan tray.

•Provides logic for hot-swappable online insertion and removal (OIR).

Isolating Cooling Subsystem Problems

The chassis cooling system dissipates heat generated by the routing system and controls the temperature of components in the fabric card chassis and line card chassis. The cooling system has a fully redundant architecture that allows the routing system to continue operating with a single-fault failure (such as a single fan or fan tray failure). Thermal sensors (on cards and modules, and inlet, exhaust, and hot-spot) are located throughout the chassis to monitor temperature readings.

A thermal alarm is generated when the system is not cooling properly. A temperature sensor might trip in response to any of the following causes:

•Elevated ambient air temperature

•Clogged air filter

•Blocked airflow

The fan controller attempts to maintain the temperature within the chassis by varying the speed of the fans. When the chassis temperature threshold is exceeded, a thermal alarm is generated. If the fault continues, the software shuts down the card or module to save components.

To isolate a problem with the chassis cooling system if you have an over-temperature condition, follow these steps.

Step 1 Make sure that the blower modules are operating properly when you power on the system.

•To determine if a blower module is operating, check the LED indicators on the blower module front panel:

–OK (green)—The blower module is functioning properly and receiving -48-VDC power, indicating that the cables from the chassis backplane to the blower module are good.

–Fail (red)—A fault is detected in the blower module. Replace the blower module.

–If neither indicator is on and the blower is not operating, a problem may exist with either the blower module or -48 VDC power supplied to the blower module. Go to Step 2.

Step 2 Eject and reseat the blower module making sure the captive screws are securely tightened.

If the blower module still does not function, go to Step 3.

Step 3 Check for -48 VDC power by looking at the LED indicators on each power module:

•If the Pwr OK indicator is on and the Fault indicator is off on each power module, the blower is receiving -48 VDC.

–If the blower module is still not functioning, a problem could exist with the blower module controller card or an undetected problem in the blower module cable. Replace the blower module.

–If the new blower module does not function, consult a Cisco customer service representative for assistance.

•If the Fault indicator is on, the power module is faulty. Replace the power module.

•If the Temp and Fault indicators are on, an over-temperature condition exists.

–Verify that the power module fan is operating properly.

–If the fan is not operating, replace the power module.

Consult your Cisco representative if replacing the power module does not fix the problem.

Troubleshooting Using the Alarm Module

The line card chassis processor subsystem consists of the RPs, PLIMs with MSCs, and alarm module. A minimally configured line card chassis must have an RP installed in slot 7 of the upper card cage. If the router is equipped with an optional, redundant RP, that RP must be installed in the far left slot in the lower card cage (slot 8).

Each AC or DC power shelf contains an alarm module that monitors the status of the power shelf and provides an external interface for system alarms. The same alarm module (see Figure 12) is used in all power shelves.

Note Only safety extra-low voltage (SELV) circuits can be connected to the Alarm connector on the alarm module faceplate. The maximum rating for the alarm circuit is 2 A, 50 V.

Figure 12 Alarm Module

1

External alarm connector

3

Alphanumeric LED display

2

Alarm LEDs

   

The alarm module performs the following functions:

•Outputs alarms to both LEDs and the relay:

–Alarm LEDs—Three large LEDs (Critical, Major, and Minor) indicate the status of the chassis. The LEDs are controlled by software on the RP system controller. For redundancy, each alarm indicator has two LEDs (to ensure that alarm status is visible even when one of the LEDs fails).

–Relay—The alarm module output function consists of a relay and its associated driver. As directed by the system controller (on the RP), the service processor module on the alarm module activates the relay. The alarm relay connector is a standard DA-15S connector.

•Monitors PEM or AC rectifier status—The alarm module monitors the performance and status of the DC PEMs or AC rectifiers. The module monitors Circuit Breaker Tripped conditions, Power Good, Power Fail, Internal Fault, Over Temp conditions, AC rectifier or PEM presence, and voltage and current output levels. Because the alarm module receives power from both power shelves, it can report the status of an unpowered shelf.

•Monitors alarms—An LED provides information about the status of the chassis.

–If the system is operating properly, "IOS-XR" appears in the LED.

–If an alarm occurs, this LED indicates the card or component that is having a problem. For example, if a fan tray is missing, the display indicates which fan tray is missing. A display such as "0 1 SP" indicates that the MSC in rack 0, slot 1 is having a problem.

Monitoring Critical, Major, and Minor Alarm Status

The alarms can warn of an over-temperature condition:

•On a component in the card cage

•A fan failure in a blower module

•An over-current condition in a power module

•An out-of-tolerance voltage on one of the cards

The RP continuously polls the system for the values of the temperature, voltage, current, and fan speed. If a threshold value is exceeded, the RP sets the appropriate alarm severity level on the alarm card, which lights the corresponding LED, and energizes the appropriate alarm display relays to activate any external audible or visual alarms wired to the alarm display. The RP also logs a message about the threshold violation on the system console. The status LEDs indicate the following:

•Critical—The card or system in not stable and is shutting down.

•Major—A high severity warning. Check the system and determine the cause of the problem. If ignored, the alarm can escalate to Critical.

•Minor—This is a basic warning indication. Often, no action is required.

Note As the temperature increases through minor, major, and critical thresholds, so does the fan speed. This fan speed counters the increase in temperature. If one critical threshold on a node is exceeded, that node is shut down.

Alarm Module Alphanumeric Messages

If an alarm occurs, the display indicates the card or component that is having a problem.

The Critical, Major, and Minor alarm LEDs are interpreted with the message displayed on the alphanumeric display. For example, if the Minor LED is lit and the alphanumeric display shows 0 SM0 SP, then a minor alarm exists on 0/SM0/SP. If the Major LED is lit and the alphanumeric display shows UPPR FAN-TRAY ALRM, then a major alarm is generated by the upper fan tray.

The status of the fans should be verified using the show environment fans command to check for failed fans. In both examples, checking the output of the show logging command provides details of the alarm (a message in the form of "MINOR alarm generated by <component_name>" or "MAJOR alarm generated by <component_name>." Table 12 lists the various states of the alarm module LEDs.

Table 12 Alarm Module LED Descriptions 

Alphanumeric Message
Condition
LED State

IOS XR

System is operating properly

—

UPPR SNGL FANx ALRM or
LWR SNGL FANx ALRM
(x = 1-9)

Single fan failure

Minor

MULT FAN FAIL ALRM

Multiple fan failure in one or more trays

Minor—fewer than 6 fans
Major—6 to 11 fans
Critical—12 or more fans

UPPR FAN- TRAY ALRM or
LWR FAN- TRAY ALRM

Fan tray removal or failure1

Major

FAN CTRL FAIL ALRM

Fan controller removal or failure

Major

SNGL PEM xx ALRM
(xx could be A0, A1, A2, B0, B1, or B2)

Single PEM failure or inserted PEM that is powered off
(power shelf is still on)

Minor

MULT PEM FAIL ALRM

Multiple PEM failure or inserted PEM that is powered off
(power shelf is still on)

Major

ZONE x FAIL ALRM2
(x = Zone number of failure)

Single zone failure due to multiple PEM failure or removal

Critical

MULT ZONE FAIL ALRM

Multiple zone failure due to multiple PEM failure or removal

Critical

PEM SHLF x ALRM (x = A or B)

PEM shelf failure or inserted power shelf that is powered off

Major

R/S/M

Single-node temperature alarm

Varies based on temperature level

MULT NODE TEMP ALRM

Multi-node temperature alarm

Varies based on temperature level

NODE R/S/M VLTG ALRM

Single-node voltage alarm

Varies based on how far the voltage is out of range

MULT NODE VLTG ALRM

Multi-node voltage alarm

Varies based on how far the voltage is out of range

1 If both fan trays are removed, the entire system shuts down immediately. This occurs as soon as the software detects that both fan trays are out of the chassis, irrespective of any other conditions. In other temperature-based cases, nodes are shut down one by one. Fan fray shutdown would also cause the fan trays to stop and cause a complete system shutdown over time.

2 This may be a rare condition because most PEMs have more than one zone attached to it.


Obtaining Documentation, Obtaining Support, and Security Guidelines

For information on obtaining documentation, obtaining support, providing documentation feedback, security guidelines, and also recommended aliases and general Cisco documents, see the monthly What's New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at:

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html