Cisco ASR 1000 Series Aggregation Services Routers SIP and SPA Software Configuration Guide - Troubleshooting the Cisco WebEx Node for ASR 1000 Series [Cisco Shared Port Adapters/SPA Interface Processors]

Table Of Contents

Troubleshooting the Cisco WebEx Node for the ASR 1000 Series Aggregation Services Routers

General Troubleshooting Information

Interpreting Console Error and System Messages

Using show Commands

show Commands to Display Cisco WebEx Node SPA Status

show Commands to Display Cisco WebEx Node SPA IDPROM

show Command to Display Cisco WebEx Node SPA Service Engine Status

show Command to Display Cisco WebEx Node SPA FPD version

show Command to Display Cisco WebEx Node SPA Error Messages on the RP Console

Accessing the SIP and SPA Consoles for Troubleshooting

Accessing the SIP Console

Accessing the Cisco WebEx Node SPA Console

Handling TraceBack and Core Dump Files on the Cisco WebEx Node SPA

Collecting Core Dump Files for the Cisco WebEx Node SPA

Collecting Traceback Information for the Cisco WebEx Node SPA

Performing Basic Troubleshooting

Troubleshooting Booting Failures

Troubleshooting Longer Booting Times in the Cisco WebEx Node SPA

Troubleshooting the Cisco WebEx Node SPA in an Out-of-Service State

Troubleshooting Hardware Failures

Troubleshooting Service Engine Application Status Errors

Using the ping Command from the Cisco ASR 1000 Series Aggregation Services Router to Verify Network Connectivity

Using the Cisco IOS Event Tracer to Troubleshoot Problems

Preparing for Online Insertion and Removal of a SPA

Troubleshooting the Cisco WebEx Node for the ASR 1000 Series Aggregation Services Routers

This chapter describes techniques that you can use to troubleshoot the operation of the Cisco WebEx Node for ASR 1000 Series on the Cisco ASR 1000 Series Aggregation Services Routers, also referred to in this document as the Cisco WebEx Node SPA.

•General Troubleshooting Information

•Performing Basic Troubleshooting

•Using the Cisco IOS Event Tracer to Troubleshoot Problems

•Preparing for Online Insertion and Removal of a SPA

For more information about troubleshooting your hardware installation, see also the Cisco ASR 1000 Series Aggregation Services Routers SIP and SPA Hardware Installation Guide.

General Troubleshooting Information

This section provides the following general information for troubleshooting the Cisco WebEx Node SPA:

•Interpreting Console Error and System Messages

•Using show Commands

•Accessing the SIP and SPA Consoles for Troubleshooting

•Handling TraceBack and Core Dump Files on the Cisco WebEx Node SPA

Interpreting Console Error and System Messages

To view the explanations and recommended actions for Cisco ASR 1000 Series Aggregation Services Routers error messages, including messages related to Cisco ASR1000-SIP10 and SPAs, refer to the System Messages for Cisco IOS XE document.

System error messages are organized in the documentation according to the particular system facility that produces the messages. The SIP and SPA error messages use the following facility names:

•Cisco ASR 1000 Series SIP:

–ASR1000_SIP

–ASR1000_SIP_SPA

•Cisco WebEx Node SPA:

–SPA_SE1

–SPA_WMA

•Cisco WebEx Node SPA Service Engine:

–SPA_SRVCS_ENGINE

–SPA_SRVCS_IF

•SPA Online Insertion and Removal—SPA_OIR

Using show Commands

There are several show commands, used in privileged EXEC mode, that you can use to monitor and troubleshoot the Cisco WebEx Node SPA on Cisco ASR 1000 Series Aggregation Services Routers. For more information on these commands, refer to the Cisco IOS command references and see also the "Performing Basic Troubleshooting" section, and the "Verifying the Service Engine Configuration" section of the "Configuring the Cisco WebEx Node for the ASR 1000 Series Aggregation Services Routers" chapter in this guide.

show Commands to Display Cisco WebEx Node SPA Status

•show platform—Displays the status of all modules installed in the Cisco ASR 1000 Series Router chassis. This command is not useful for out-of-service issues.

•show hw-module subslot all oir—Displays the OIR status of each Cisco WebEx Node SPA in the chassis. This command is useful for displaying out-of-service issues.

•show hw-module subslot oir—Displays the OIR status for a specified Cisco WebEx Node SPA.

•show hw-module subslot service-engine status—Displays the application and operational status and configuration of a specified Cisco WebEx Node SPA.

show Commands to Display Cisco WebEx Node SPA IDPROM

•show diag subslot eeprom—Displays main fields of the Cisco WebEx Node SPA IDPROM.

•show diag subslot detail—Display all the fields of the Cisco WebEx Node SPA IDPROM.

•show diag subslot dump—Dumps the content of the Cisco WebEx Node SPA IDPROM in hexadecimal format.

show Command to Display Cisco WebEx Node SPA Service Engine Status

The show hw-module subslot service-engine status command displays the operational status and configuration of the Cisco WebEx Node SPA. For successful configuration of the service-engine interface, the application status should be "Online." You can also confirm the provisioned operation mode for the CiscoWebEx Node, which is "Web Conferencing" in the following example:
Router# show hw-module subslot 0/0 service-engine status
Service Engine is Cisco SPA-WMA-K9
Service Engine state: Steady (0x300)
Service Engine OS Version: 1.0.0, Application Version: 1.0.0
Application: WebEx Node (Web Conferencing)
Application Status: Online
Configuration:
  Int ip address: 10.200.72.18 ,  mask: 255.255.255.252
  GW ip address: 10.200.72.17
  Nameserver 1: 10.100.4.10 ,  Nameserver 2: 10.100.4.20
  Hostname: spawma1, Domain name: cisco.com
  WMA URL - https://wmabts.webex.com/wmams
  WMA Token - 45484b3e-8ea5-41e5-b050-49409006d14e
  WMA Passcode Name - cisco_test, key:0552055C271A4B5C4D5D424A5B5E007F
show Command to Display Cisco WebEx Node SPA FPD version

The show hw-module all fpd command displays the FPD version of each Cisco WebEx Node SPA in the Cisco ASR 1000 Series Router. The following example shows some sample output:
Router# show hw-module all fpd
==== ====================== ====== =============================================
                             H/W   Field Programmable   Current   Min. Required
Slot Card Type               Ver.  Device: "ID-Name"    Version      Version
==== ====================== ====== ================== =========== ==============
 0/0 SPA-WMA-K9              0.169 4-ADM1066              0.5         0.5
                                   5-CPLD                 1.2         1.2
                                   1-Appl I/O FPGA        1.2         1.2
                                   2-ROMMON               1.1         1.1
---- ---------------------- ------ ------------------ ----------- --------------
 0/1 SPA-5X1GE-V2            1.2   1-GE I/O FPGA          1.10        1.10
---- ---------------------- ------ ------------------ ----------- --------------
 0/3 SPA-2X1GE-V2            0.22  1-2xGE V2 I/O FPGA     1.1         1.1
---- ---------------------- ------ ------------------ ----------- --------------
 1/0 SPA-WMA-K9              0.133 4-ADM1066              0.5         0.5
                                   5-CPLD                 1.2         1.2
                                   1-Appl I/O FPGA        1.2         1.2
                                   2-ROMMON               1.1         1.1
---- ---------------------- ------ ------------------ ----------- --------------
 1/1 SPA-WMA-K9              0.165 4-ADM1066              0.5         0.5
                                   5-CPLD                 1.2         1.2
                                   1-Appl I/O FPGA        1.2         1.2
                                   2-ROMMON               1.1         1.1
show Command to Display Cisco WebEx Node SPA Error Messages on the RP Console

The show logging command displays the logged messages that have appeared on the RP console. The following is an example of a software error message that might appear when you run the show logging command:
Router# show logging
*May 22 17:11:58.712: %SPA_SRVCS_ENGINE-3-APP_MSG_ERR: SIP0/0: SPA-WMA-K9[0/0]: Connect 
CWNMS server failed, check network availability to CWNMS server.
Accessing the SIP and SPA Consoles for Troubleshooting

Some troubleshooting for the Cisco WebEx Node SPA requires that you access the SIP or SPA consoles and run some commands to gather debug information or take other action. Under normal operation there is no need to access these consoles. However, sometimes it can be necessary to troubleshoot more complex problems and gather certain debug information.

Accessing the SIP Console

Use the request platform software console attach command to access the SIP console through the Cisco ASR 1000 Series Router RP console. The SIP console is used to collect debug information about the Cisco IOS code that is running on the SIP.

The following example shows how to access the SIP console and enter enable mode:
Router# request platform software console attach 0/0
#
# Connecting to the SPA console on 0/0.
# Enter Control-C to exit the console connection.
#
Router>enable
Router#
To exit the SIP console, enter Control-C.

Accessing the Cisco WebEx Node SPA Console

Use the hw-module subslot service-engine session command to access the Cisco WebEx Node SPA console through the Cisco ASR 1000 Series Router RP console. You will see a VEGAS Shell prompt and have access to the console commands.

Prerequisites

Before you can open a console session on a Cisco WebEx Node SPA, the SPA must first be configured with a minimum of the following commands and be in the "up" state:

•ip address

•service-engine ip address

•service-engine default-gateway

The service-engine ip address command must be configured before the service-engine default-gateway command.

Opening the Cisco WebEx Node SPA Console

The following example shows how to open the SPA console:

Note The SPA console default prompt is "service-spa." This prompt can be changed by configuring the service-engine hostname command.
Router# hw-module subslot 0/0 service-engine session
MontaVista(R) Linux(R) Carrier Grade Edition 5.0 (custom)
Linux/mips64 2.6.21_mvlcge500-octeon-mips64_octeon_v2_be
Vegas Shell  --  CGE 5.0 Version
        Copyright (c) 1985-2008 by Cisco Systems, Inc.
        All rights reserved.
service-spa#
Listing the Available Cisco WebEx SPA Console Commands

The Cisco WebEx SPA console supports a similar help function as the RP console. You can enter ? at the console to obtain a list of the available commands as shown in the following example:
service-spa# ?
Exec commands:
  clean       Clean commands
  copy        Copy files from src to destination
  delete      Remove files
  dir         Directory listing for files
  exit        Exit from the EXEC
  format      Format a device with ext3/fat16/dummy file system
  fsck        Perform file system check operation
  less        Shows content of file
  load        Load plug-in image
  mkdir       Create new directory
  move        Move files
  ping        Send echo messages
  rmdir       Remove existing directory
  show        Show system information
  terminal    Set terminal line parameters
  traceroute  Trace route to destination
  webex       Webex Application Commands
Some of the most useful commands include:

•show eventlog—Dumps the content of the /var/log/messages file.

•show disk partitions—Displays the partitions available on the hard disk drive (HDD).

•show disk smart—Displays S.M.A.R.T. output of the HDD.

•show tech-support—Gathers information for troubleshooting.

•ping—Sends echo messages.

•copy tftp: disk0:—Copies files from a TFTP server to the HDD first partition.

•load—Loads a plug-in image.

•exit—Exits from the Cisco WebEx Node SPA console.

Exiting the Cisco WebEx Node SPA Console

To exit the SPA console, use the exit command as shown in the following example:
service-spa# exit
Router#
Handling TraceBack and Core Dump Files on the Cisco WebEx Node SPA

This section describes how to gather traceback and core dump file information for troubleshooting by Cisco technical support.

Collecting Core Dump Files for the Cisco WebEx Node SPA

Linux utilities on the Cisco WebEx Node SPA and the WebEx application software can create core dump files when errors occur. When this happens an error message is logged on the RP console as shown in the following example:
*May 22 17:12:02.210:%SPA_SRVCS_ENGINE-2-APP_MSG_CRIT: SIP1/0: SPA-WMA-K9[1/0]: Core dump 
generated for 'smartd' (corefile:smartd-626-11-1242970631.core.gz). If the problem 
persists, contact your technical support representative for assistance.
The core file is saved on the hard disk on the Cisco WebEx Node SPA. Using commands from the Cisco WebEx Node SPA console, you can copy the file from the SPA to an external TFTP server. This file should be given to Cisco technical support as necessary for analysis of the problem.

To access the core dump file and copy it to a TFTP server, complete the following steps:

Step 1 From the RP console, access the Cisco WebEx Node SPA console as shown in the following example:
Router# hw-module subslot 0/0 service-engine session
MontaVista(R) Linux(R) Carrier Grade Edition 5.0 (custom)
Linux/mips64 2.6.21_mvlcge500-octeon-mips64_octeon_v2_be
        Vegas Shell  --  CGE 5.0 Version
        Copyright (c) 1985-2008 by Cisco Systems, Inc.
        All rights reserved.
service-spa#
Step 2 From the Cisco WebEx Node SPA console, run the dir core: command to display the smartd file as shown in the following example:

Note The core dump file is compressed in gzip format before it is saved to the SPA hard disk.
service-spa# dir core:
  1489726  May 22 05:37:27 2009 smartd-626-11-1242970631.core.gz
           Usage for core: filesystem
                  221794304 bytes total used
                 1925689343 bytes free
                 2147483647 bytes total
Step 3 Copy the core dump file from the SPA to a TFTP server as shown in the following example:

Note Copying the core file to a storage device on the RP is not supported.
service-spa# copy core:smartd-626-11-1242970631.core.gz tftp://dirt/tftpboot/username/
Trying to connect to tftp server......
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
(some output has been removed to make this more readable)
!!!!!!!!!!!!!!!!!!
TFTP put operation was successful
 1489726 bytes copied
service-spa# exit
Router#
Collecting Traceback Information for the Cisco WebEx Node SPA

Traceback information is generated during booting of the Cisco WebEx Node SPA and can be used to troubleshoot problems. You can access the traceback information from the SIP console.

No error messages are logged on the RP to indicate that a traceback event has occurred. These events are usually associated with other failures whose errors are reported on the RP console such as SYSINIT_FAILURE messages, as shown in the following example:
*May 1 07:46:32.650: %SPA_SE1-3-SYSINIT_FAILURE: SIP0/0: SPA-WMA-K9[0/0]: System init 
failure was detected during bootup - application installation error. SPA will be disabled 
because of this failure.
To collect traceback information for the Cisco WebEx Node SPA, complete the following steps:

Step 1 From the RP console, access the SIP console and enter enable mode as shown in the following example:
Router# request platform software console attach 0/0
#
# Connecting to the SPA console on 0/0.
# Enter Control-C to exit the console connection.
#
Router> enable
Router#
Step 2 From the SIP console, enter the show hw-module subslot bay bootlog cpu-prev or show hw-module subslot bay bootlog cpu-last commands to access the traceback from either the previous SPA boot (cpu-prev option) or last SPA boot (cpu-last option), as shown in the following examples:
Router# show hw-module subslot 0 bootlog cpu-last
% CPU boog log not available from last SPA-WMA-K9[0/0] bootup.
The following excerpt of a traceback shows a problem with the kernel that eventually causes the Cisco WebEx Node SPA to timeout. After 5 attempts to reload the Cisco WebEx Node SPA, the SPA enters an out-of-service state:

Note Some output has been removed to make the example more readable.
Router# show hw-module subslot 0 bootlog cpu-prev
--------- CPU boot log from previous SPA-WMA-K9[0/0] bootup ---------
Checking for the multiply/shift bug... no.
Checking for the daddi bug... no.
Checking for the daddiu bug... no.
CPU 0 Unable to handle kernel paging request at virtual address 0000000000000008,
epc == ffffffff8c029fb4, ra == ffffffff8c02a0bc
Oops[#1]:
Cpu 0
$ 0   : 0000000000000000 ffffffff8c4a3400 ffffffff8c4a5c00 ffffffff8c4a3400
$ 4   : 0000000000000028 0000000000000000 0000000000000000 0000000000000001
$ 8   : a800000414487e60 ffffffffffffffc0 0000000000000000 0000000000000000
$12   : 0000000000000000 0000000000000000 0000000000000000 0000000000000000
$16   : 0000000000000028 0000000000000000 0000000000000010 ffffffff8c4a5c70
$20   : 0000000000000001 0000000000000000 0000000000000000 0000000000000000
$24   : 0000000000000000 0000000000000000
$28   : a800000414484000 a800000414487e60 0000000000000000 ffffffff8c02a0bc
Hi    : 0000000000000000
Lo    : 0000000000000000
epc   : ffffffff8c029fb4 octeon_unmask_ciu_irq+0x24/0x30     Not tainted
ra    : ffffffff8c02a0bc octeon_irq_ciu_unmask+0x84/0xc8
Status: 10008ce2    KX SX UX KERNEL EXL
Cause : 00000008
BadVA : 0000000000000008
PrId  : 000d030b
Modules linked in:
Process swapper (pid: 1, threadinfo=a800000414484000, task=a800000414481440)
Stack : 0000000000000001 0000000000000000 0000000000000028 a800000414469b80
        ffffffff8c4a5c00 ffffffff8c02a164 0000000000000001 ffffffff8c046a20
        ffffffff8c095c7c ffffffff8c095afc 0000000000000028 0000000000000080
        ffffffff8c0068d8 ffffffff8c0068d8 ffffffff8c40fa18 a800000414469b80
        ffffffff8c095dec ffffffff8c095dc0 0000000000000000 0000000000000000
        0000000000000000 0000000000000000 0000000000000000 0000000000000000
        ffffffff8c006be8 0000000000000000 ffffffff8c4c68b0 0000000000000000
        ffffffff8c4b1b24 0000000000000000 0000000000000000 0000000000000000
        a800000414484000 a800000414487fe0 0000000000000000 0000000000000000
        0000000000000000 0000000000000000 0000000000000000 0000000000000000
Call Trace:
[<ffffffff8c029fb4>] octeon_unmask_ciu_irq+0x24/0x30
[<ffffffff8c02a0bc>] octeon_irq_ciu_unmask+0x84/0xc8
[<ffffffff8c02a164>] octeon_irq_ciu_startup+0x64/0x78
[<ffffffff8c095c7c>] setup_irq+0x234/0x2b0
[<ffffffff8c095dec>] request_irq+0xf4/0x118
[<ffffffff8c006be8>] plat_prepare_cpus+0x98/0xe0
[<ffffffff8c4c68b0>] smp_prepare_cpus+0xf8/0x140
[<ffffffff8c4b1b24>] init+0x94/0x748
[<ffffffff8c022808>] kernel_thread_helper+0x10/0x18
Code: 0061182d  0043102d  dc460020 <dcd90008> 03200008  00000000  7082fa32  3c030000  
3c018c4a
Kernel panic - not syncing: Attempted to kill init!
Call Trace:
[<ffffffff8c026570>] dump_stack+0x8/0x48
[<ffffffff8c052220>] panic+0xb0/0x350
[<ffffffff8c058a94>] do_exit+0xc7c/0xc80
[<ffffffff8c02647c>] die+0x1ac/0x1b0
[<ffffffff8c038034>] do_page_fault+0x1d4/0x480
[<ffffffff8c01ff84>] ret_from_exception+0x0/0x10
[<ffffffff8c029fb4>] octeon_unmask_ciu_irq+0x24/0x30
[<ffffffff8c02a0bc>] octeon_irq_ciu_unmask+0x84/0xc8
[<ffffffff8c02a164>] octeon_irq_ciu_startup+0x64/0x78
[<ffffffff8c095c7c>] setup_irq+0x234/0x2b0
[<ffffffff8c095dec>] request_irq+0xf4/0x118
[<ffffffff8c006be8>] plat_prepare_cpus+0x98/0xe0
[<ffffffff8c4c68b0>] smp_prepare_cpus+0xf8/0x140
[<ffffffff8c4b1b24>] init+0x94/0x748
[<ffffffff8c022808>] kernel_thread_helper+0x10/0x18
--------- End of CPU previous boot log ---------
Performing Basic Troubleshooting

This section includes the following topics:

•Troubleshooting Booting Failures

•Troubleshooting Hardware Failures

•Troubleshooting Service Engine Application Status Errors

•Using the ping Command from the Cisco ASR 1000 Series Aggregation Services Router to Verify Network Connectivity

Troubleshooting Booting Failures

You can perform most of the troubleshooting for booting failures using the show platform and show hw-module subslot all oir commands and examining the output to determine the source of an operational problem. Check the output of the show platform command to verify that the SIP and SPA are operational.

The following sections provide information about troubleshooting booting failures on the Cisco WebEx Node SPA:

•Troubleshooting Longer Booting Times in the Cisco WebEx Node SPA

•Troubleshooting the Cisco WebEx Node SPA in an Out-of-Service State

Troubleshooting Longer Booting Times in the Cisco WebEx Node SPA

Sometimes the Cisco WebEx Node SPA can remain in a booting state for longer-than-usual periods of time. There can be normal reasons for this, or it can be an indication of a problem.

Use the show hw-module subslot all oir command to display the operational status of the Cisco WebEx Node SPA. The following example shows sample output of the show hw-module subslot all oir command that shows the Cisco WebEx Node SPA in subslot 1/1 in the booting state:
Router#sh hw-module subslot all oir
Module        Model                Operational Status
------------- -------------------- ------------------------
subslot 0/0   SPA-2X1GE-V2         ok
subslot 0/1   SPA-WMA-K9           ok
subslot 1/1   SPA-WMA-K9           booting
To troubleshoot a Cisco WebEx Node SPA that is remaining in the boot state, complete the following steps:

Step 1 Review some of the possible causes for longer booting times for the Cisco WebEx Node SPA that are described in Table 23-1.

If none of the reasons described in Table 23-1 are found, and the SPA remains in a booting state for more than 2 minutes, then the SPA will eventually timeout in 6 minutes. The OIR process will automatically try to recover from the problem. The OIR process will make 5 attempts to bring up the Cisco WebEx Node SPA before powering off the SPA.

Step 2 Use the show logging command to look for the SPA_OIR-3-SPA_POWERED_OFF message as shown in the following example:
*Mar  3 23:27:39.884: %SPA_OIR-3-SPA_POWERED_OFF: subslot 1/0: SPA WMA SERVICES SPA 
powered off after 5 failures within 1200 seconds
Step 3 Reload the SIP as shown in the following example to try to correct the problem:
Router# hw-module subslot slot 1 reload
Table 23-1 describes the possible reasons why the Cisco WebEx Node SPA might remain in a booting state for a longer length of time than usual.
Table 23-1 Possible Reasons for Longer Booting Times in the Cisco WebEx Node SPA
Possible Problem

Observations and Comments

Solutions

An FPD upgrade is in progress.

When the system detects an incompatible FPD version on the Cisco WebEx Node SPA, it will attempt to perform an automatic FPD upgrade.

This operation can take up to 4 minutes to complete.

From EXEC mode, enter the show upgrade fpd progress command to check if a field-programmable devices (FPD) upgrade is in progress.

Tip You can also check if the STATUS LED is blinking in amber color.
Automatic file system checking and recovery of the hard disk drive is being performed.

This process can take around 7 minutes to complete.
Confirm that the following message appears on the RP console:
*Mar  3 23:29:29.763: 
%SPA_SE1-3-CHECKING_DISK: SIP1/0: 
SPA-WMA-K9[1/0]: Disk is being checked 
because of previous unclean shutdown of the 
SPA. Boot time might take longer because of 
this operation.
Troubleshooting the Cisco WebEx Node SPA in an Out-of-Service State

Both hardware and software problems can cause "out of service" states on the Cisco WebEx Node SPA.

To verify the out-of-service reason for the Cisco WebEx Node SPA, use the show hw-module subslot all oir command as shown in the following example:
Router# show hw-module subslot all oir
Module        Model                Operational Status
------------- -------------------- ------------------------
subslot 0/0   SPA-2X1GE-V2         ok
subslot 0/1   SPA-WMA-K9           ok
subslot 1/1   SPA-WMA-K9           out of service (Incompatible FPD)
Table 23-2 describes the possible problems and solutions for software-based out-of-service states. For more information about other hardware-based out-of-service states, refer to the troubleshooting chapter for the Cisco WebEx Node SPA in the Cisco ASR 1000 Series Aggregation Services Routers SIP and SPA Hardware Installation Guide.

Table 23-2 Possible Reasons for Software-Based Out-of-Service States on the Cisco WebEx Node SPA

Possible Reasons

Observations and Comments

Solutions

The FPD image was corrupted for some of the following possible reasons:

•SPA was removed during an FPD upgrade

•A reload of the router occurred during an FPD upgrade.

•A power failure occurred on the router during an FPD upgrade.

The show hw-module subslot all oir command displays a "failed too many time" reason code for the out of service operational status and the show logging command displays a HW-INIT-TIMEOUT failure.

The following is an example of a HW-INIT-TIMEOUT message:

*Mar 3 23:27:05.903: %SPA_OIR-6-ONLINECARD: SPA (SPA-WMA-K9) online in subslot 1/1
*Mar 3 23:27:16.488: %SPA_OIR-3-HW_INIT_TIMEOUT: subslot 1/0
*Mar 3 23:27:21.488: %SPA_OIR-3-RECOVERY_RELOAD: subslot 1/0: Attempting recovery by reloading SPA
*Mar 3 23:27:21.489: %SPA_OIR-6-OFFLINECARD: SPA (SPA-WMA-K9) offline in subslot 1/0

1. Enter the upgrade hw-module subslot fpd bundled command to start recovery of the FPD upgrade.

If the problem was due to an FPD image corruption problem, then the SPA should boot normally after the upgrade is complete.

2. If the FPD upgrade completes successfully but you still have an error, then the SPA probably has a hardware problem.

3. Refer to the troubleshooting steps in the the Cisco ASR 1000 Series Aggregation Services Routers SIP and SPA Hardware Installation Guide.

The FPD upgrade was unable to be performed.

The show hw-module subslot all oir command displays an "Incompatible FPD" reason code for the out of service operational status.

1. Enter the show hw-module all fpd command to verify the FPD versions.

2. Refer to the "Upgrading Field-Programmable Devices" chapter of this guide to get information about upgrading your FPD image.

The Cisco WebEx Node SPA software subpackage is not installed.

The show hw-module subslot all oir command displays a "not allowed online" reason code for the out of service operational status, and the show logging command displays the ASR1000_RP_SPA-3-MISSING_SPA_PKG_ERR as shown in the following example:

*Mar 24 22:39:49.832: %ASR1000_RP_SPA-3-MISSING_SPA_PKG_ERR: sipspawma package is not installed for slot = 0 and subslot = 0, SPA bootup failed.

The Cisco WebEx Node SPA requires you to install an independent software subpackage called sipspawmak9. To successfully install this subpackage you must complete the following steps:

1. Extract the individual system subpackages from the consolidated package for the router into a directory with the provisioning file (.conf file).

Note You cannot boot the router from a consolidated package when you are installing a Cisco WebEx Node SPA.

2. Download the optional sipspawmak9 subpackage to the same location as the individual subpackages and the provisioning file.

3. Configure the router to boot from a provisioning file.

4. Reload the router.

For more detailed information about installing the software for the Cisco WebEx Node SPA, refer to the Cisco ASR 1000 Series Aggregation Services Router Software Configuration Guide.

Troubleshooting Hardware Failures

Depending on the severity of the error, the SPA may initiate a reload to recover from the failure. Any error message that is of SPA_CPU_ERR error type is an indication of a hardware problem on the SPA.

For more information about hardware-related failures, refer to the troubleshooting chapter of the Cisco ASR 1000 Series Aggregation Services Routers SIP and SPA Hardware Installation Guide.

Troubleshooting Service Engine Application Status Errors

After you configure the virtual service-engine interface, you should verify the configuration using the show hw-module subslot service-engine status command, and look for the Application Status field to be "Online" as shown in the following example:
Router# show hw-module subslot 1/0 service-engine status
Service Engine is Cisco SPA-WMA-K9
Service Engine state: Steady (0x300)
Service Engine OS Version: 1.0.0, Application Version: 1.0.0
Application: WebEx Node (Web Conferencing)
Application Status: Online
Configuration:
If the Application Status is "Offline," refer to Table 23-3, which describes the possible reasons and solutions for this state.
Table 23-3 Possible Reasons for Application Status Problems in the Cisco WebEx Node SPA
Possible Reasons

Observations and Comments

Solutions
Provisioning error at the Cisco WebEx Data Center
The output of the show hw-module 
subslot service-engine status command 
shows "Online" in the Application Status 
field, and "Web Conferencing" as the 
mistaken operation mode, when the 
desired mode is "Voice and Video 
Conferencing" as shown in the following 
example:
Router# show hw-module subslot 1/0 
service-engine status
Service Engine is Cisco SPA-WMA-K9
Service Engine state: Steady (0x300)
Service Engine OS Version: 1.0.0, 
Application Version: 1.0.0
Application: WebEx Node (Web 
Conferencing) 
Application Status: Online
Contact Cisco WebEx Data Center support to verify that the Cisco WebEx Node for ASR 1000 Series has been provisioned for the expected "Web Conferencing" or "Voice and Video Conferencing" mode.

For more information, see the "Registering with the WebEx Data Center and the Cisco WebEx Node Management System" topic in Chapter 22 "Configuring the Cisco WebEx Node for the ASR 1000 Series Aggregation Services Routers."
Provisioning error at the Cisco WebEx Data Center
•The output of the show hw-module subslot service-engine status command shows "Offline" in the Application Status field.

•Confirm that a message similar to the following example appears on the RP console:
Nov  3 20:42:29.791: 
%SPA_SRVCS_ENGINE-3-APP_MSG_ERR: 
SIP0/1: SPA-WMA-K9[0/1]: 
Authentication failed with Cisco 
WebEx Data Center SSL server. 
Please contact with WebEx 
support for assistance
Contact Cisco WebEx support as directed by the system error message.
Network connectivity problem, such as the subnet assigned to the service-engine interface is not routable.
•The output of the show hw-module subslot service-engine status command shows "Offline" in the Application Status field.

•Confirm that a message similar to the following example appears on the RP console:
Nov 3 13:45:51.943: 
%SPA_SRVCS_ENGINE-3-APP_MSG_ERR: 
SIP0/0: SPA-WMA-K9[0/1]: Connect 
CWNMS server failed, check 
network availability to CWNMS 
server.
1. Check network connectivity to the Cisco WebEx Data Center, using the traceroute command from the Cisco WebEx SPA console as shown in the following example:
service-spa# traceroute wma.webex.com
traceroute to globalwatch.webex.com 
(10.114.169.228), 30 hops max, 38 byte 
packets
 1  10.1.1.1 (12.1.1.1)  0.277 ms  0.254 
ms  0.246 ms
 2  10.1.99.65 (10.1.99.65)  0.577 ms  
0.394 ms  0.387 ms
2. Check network connectivity from the clients that will be joining the meeting being hosted on the Cisco WebEx Node SPA. Use the ping command from the Cisco WebEx SPA console as shown in the following example, where 172.16.144.153 is address of the client:
service-spa# ping 172.16.144.153
PING 172.16.144.153 (172.16.144.153) 
100(128) bytes of data.
108 bytes from 172.16.144.153: 
icmp_seq=1 ttl=41 time=69.1 ms
108 bytes from 172.16.144.153: 
icmp_seq=2 ttl=41 time=68.9 ms
108 bytes from 172.16.144.153: 
icmp_seq=3 ttl=41 time=68.8 ms
108 bytes from 172.16.144.153: 
icmp_seq=4 ttl=41 time=68.9 ms
108 bytes from 172.16.144.153: 
icmp_seq=5 ttl=41 time=68.9 ms
--- 172.16.144.153 ping statistics ---
5 packets transmitted, 5 received, 0% 
packet loss, time 4000ms
rtt min/avg/max/mdev = 
68.882/68.976/69.133/0.299 ms
For more information about accessing the Cisco WebEx SPA console, see the "Accessing the Cisco WebEx Node SPA Console" section.
Using the ping Command from the Cisco ASR 1000 Series Aggregation Services Router to Verify Network Connectivity

Note The Cisco WebEx Node SPA might not activate properly if network access between the Cisco ASR Series 1000 Aggregation Services Router and certain Cisco WebEx servers is blocked, such as the Cisco WebEx Data Center SSL Gateway or Cisco WebEx Node Management server. Be sure to verify connectivity to these servers from the Cisco ASR 1000 Series Router.

The ping command is a convenient way to test the ability of an interface to send and receive packets over the network. The ping command sends ICMP echo request packets to a specified destination address, which should send an equal number of ICMP echo reply packets in reply. By measuring the numbering of packets that are successfully returned, as well as how long each packet takes to be returned, you can quickly obtain a rough idea of the Layer 3 to Layer 3 connectivity between two interfaces.

The IP ping command has the following syntax:

ping

or

ping ip-address [repeat count] [data hex] [size datagram-size]

If you enter just ping, the command interactively prompts you for all other parameters. Otherwise, you must specify at least a specific IP address as the destination for the ping. You can also optionally specify the following parameters:

•repeat count—Number of ICMP echo request packets to send. The default is five packets.

•data hex—The data pattern, in hexadecimal, to be sent in the ICMP echo request packets.

•size datagram-size—Specifies the size, in bytes, of the ICMP echo request packets to be sent. The range is 40 to 18024 bytes, with a default of 100 bytes.

Using the Cisco IOS Event Tracer to Troubleshoot Problems

Note This feature is intended for use as a software diagnostic tool and should be configured only under the direction of a Cisco Technical Assistance Center (TAC) representative.

The Event Tracer feature provides a binary trace facility for troubleshooting Cisco IOS software. This feature gives Cisco service representatives additional insight into the operation of the Cisco IOS software and can be useful in helping to diagnose problems in the unlikely event of an operating system malfunction or, in the case of redundant systems, route processor switchover.

Event tracing works by reading informational messages from specific Cisco IOS software subsystem components that have been preprogrammed to work with event tracing, and by logging messages from those components into system memory. Trace messages stored in memory can be displayed on the screen or saved to a file for later analysis.

The SPAs currently support the "spa" component to trace SPA OIR-related events.

Preparing for Online Insertion and Removal of a SPA

The Cisco ASR 1000 Series Aggregation Services Routers support online insertion and removal (OIR) of the SIP, in addition to each of the SPAs. Therefore, you can remove a SIP with its SPAs still intact, or you can remove a SPA independently from the SIP, leaving the SIP installed in the router.

This means that a SIP can remain installed in the router with one SPA remaining active, while you remove another SPA from one of the SIP subslots. If you are not planning to immediately replace a SPA into the SIP, then be sure to install a blank filler plate in the subslot. The SIP should always be fully installed with either functional SPAs or blank filler plates.

For more information about activating and deactivating SPAs in preparation for OIR, see Chapter 5 "Troubleshooting the SIP."