Cisco MDS 9500 Series Multilayer Directors

December 4, 2007

NOTICE:

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Products Affected

Problem Description

There is a small probability that Generation 2 modules DS-X9112, DS-X9124, DS-X9148 or DS-X9704 may reload when upgrading to SAN-OS 3.0(x), SAN-OS 3.1(x), or SAN-OS 3.2(1a).

Background

This issue has been identified as being caused by a software process reading unused memory locations at the time of the SAN-OS upgrade. Reading memory locations that have not been utilized may result in a memory read error and cause the module to reload.

Problem Symptoms

During a software upgrade to SAN-OS 3.0(x) or SAN-OS 3.1(x) or 3.2(1a), there is a small probability that the Generation 2 modules DS-X9112, DS-X9124, DS-X9148 or DS-X9704 may reload. In some cases, Install All aborts without upgrading the remaining modules.

From show logging log , the customer will see messages similar to the following:

%IMAGE_DNLD-SLOT4-2-IMG_DNLD_COMPLETE: Module image download process. Download successful. 
%MODULE-2-MOD_DIAG_FAIL: Module 4 (serial:XYZ) reported failure on ports 4/1-4/24 (Fibre Channel) due to Skyline 
fwd module experienced an error in device 58 (device error 0xc3a0065c) 
%MODULE-2-MOD_SOMEPORTS_FAILED: Module 4(serial:XYZ) reported failure on ports 4/1-4/24 (Fibre Channel) due to Skyline fwd 
module experienced an error in device 58 (error 0xc3a0065c) 
%PLATFORM-5-MOD_DETECT: Module 4 detected (Serial number XYZ) 
%PLATFORM-5-MOD_PWRUP: Module 4 powered up (Serial number XYZ) 


Or 


%IMAGE_DNLD-SLOT4-2-IMG_DNLD_COMPLETE: Module image download process. Download successful. 
%MODULE-2-MOD_DIAG_FAIL: Module 4 (serial:XYZ) reported failure on ports 4/1-4/24 (Fibre Channel) due to Tuscany fwd 
module experienced an error in device 53 (device error 0xc350060b) 
%MODULE-2-MOD_SOMEPORTS_FAILED: Module 4 (serial:XYZ) reported failure on ports 4/1-4/24 (Fibre Channel) due to Tuscany 
fwd module experienced an error in device 53 (error 0xc350060b) 
%PLATFORM-5-MOD_DETECT: Module 4 detected (Serial number XYZ) 
%PLATFORM-5-MOD_PWRUP: Module 4 powered up (Serial number XYZ) 


or 

2007 Oct 31 12:45:33 productdir1 %IMAGE_DNLD-SLOT4-2-IMG_DNLD_STARTED: Module image download process. Please wait until completion... 
2007 Oct 31 12:45:48 productdir1 %IMAGE_DNLD-SLOT4-2-IMG_DNLD_COMPLETE: Module image download process. Download successful. 
2007 Oct 31 12:46:34 productdir1 %MODULE-5-MOD_OK: Module 4 is online (serial: JAB103600CN) 
2007 Oct 31 12:46:37 productdir1 %PROC_MGR-SLOT4-2-ERR_MSG: ERROR: 
52(acltcam) - PID 410 abnormal exit, exit_code=0xff00(65280) 
2007 Oct 31 12:46:37 productdir1 %ACL-2-ACL_LINECARD_STATUS: Status from 
linecard: 3, for module: acltcam(ID: 0), status : 255, error_id: 
0x40320001, catastrophic: FALSE, restart count: 0. 
2007 Oct 31 12:46:40 productdir1 %PROC_MGR-SLOT4-2-ERR_MSG: ERROR: 
52(acltcam) - PID 448 abnormal exit, exit_code=0xff00(65280) 
2007 Oct 31 12:46:41 productdir1 %ACL-2-ACL_LINECARD_STATUS: Status from 
linecard: 3, for module: acltcam(ID: 0), status : 255, error_id: 
0x40320001, catastrophic: FALSE, restart count: 1. 
2007 Oct 31 12:46:43 productdir1 %PROC_MGR-SLOT4-2-ERR_MSG: ERROR: 
52(acltcam) - PID 451 abnormal exit, exit_code=0xff00(65280) 
2007 Oct 31 12:46:44 productdir1 %ACL-2-ACL_LINECARD_STATUS: Status from 
linecard: 3, for module: acltcam(ID: 0), status : 255, error_id: 
0x40320001, catastrophic: FALSE, restart count: 2. 
2007 Oct 31 12:46:44 productdir1 %ACL-2-ACL_LINECARD_STATUS: Status from 
linecard: 3, for module: acltcam(ID: 0), status : 255, error_id: 
0x40320001, catastrophic: TRUE, restart count: 2. 
2007 Oct 31 12:46:44 productdir1 %MODULE-5-MOD_REINIT: Re-initializing 
module 4 (serial: JAB103600CN) 
2007 Oct 31 12:46:43 productdir1 %PROC_MGR-SLOT4-2-ERR_MSG: (acltcam) 
crosses failed threshold, no restart 

Output of show logging onboard on the affected module has the following signature:

Skyline Fwd-Engine: Wed Nov 14 19:59:17 2007@ 68936 Skyline FWD module ACL PIO wr error. 
TCAM_STATUS_REG = 00240102, TCAM_PARITY_ERROR_FIFO_REG = 00000000 
00240102 

Skyline Fwd-Engine: Wed Nov 14 19:59:17 2007@ 69037 Skyline FWD module ACL PIO rd error. 
TCAM_STATUS_REG = 00240102, TCAM_PARITY_ERROR_FIFO_REG = 00000000 
00240102 

Skyline Fwd-Engine: Wed Nov 14 19:59:17 2007@ 69088 
fwd_tcam_pio_read():85: SKY ACL TCAM PIO read error detected at addr=0x00 00000000 00000000, data=00 00000000 00000000, tcam_status_reg = 00003800 

Or 

Tuscany Fwd-Engine: Wed Nov 14 19:59:17 2007@ 68936 Tuscany FWD module ACL PIO wr error. 
TCAM_STATUS_REG = 00240102, TCAM_PARITY_ERROR_FIFO_REG = 00000000 
00240102 

Tuscany Fwd-Engine: Wed Nov 14 19:59:17 2007@ 69037 Tuscany FWD module ACL PIO rd error. 
TCAM_STATUS_REG = 00240102, TCAM_PARITY_ERROR_FIFO_REG = 00000000 
00240102 

Tuscany Fwd-Engine: Wed Nov 14 19:59:17 2007@ 69088 
fwd_tcam_pio_read():85: TUS ACL TCAM PIO read error detected at addr=0x00 00000000 00000000, data=00 00000000 00000000, tcam_status_reg = 00003800 

Example of show module internal exceptionlog command output:

show module internal exceptionlog module 7 
********* Exception info for module 7 ******** 
exception information --- exception instance 1 ---- 
Module Slot Number: 7 
Device Id : 58 
Device Name : Skyline-fwd 
Device Errorcode : 0xc3a00606 
Device ID : 58 (0x3a) 
Device Instance : 00 (0x00) 
Dev Type (HW/SW) : 06 (0x06) 
ErrNum (devInfo) : 06 (0x06) 
System Errorcode : 0x40420037 Skyline fwd module experienced an error 
Error Type : Minor error 
PhyPortLayer : Fibre Channel 
Port(s) Affected : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24 
Error Description : Skyline FWD module interrupt threshold 
DSAP : 0 (0x0) 
UUID : 0 (0x0) 
Time : Sat Aug 18 13:25:22 2007 
(Ticks: 46C6F342 jiffies) 

Workaround/Solution

Workaround

After encountering this error the module is automatically reloaded. No further action is required.

In some rare cases, Install All aborts without upgrading the remaining modules. In such cases, customers can issue another Install All command to restart the upgrade for the remaining modules.

Solution

The software process causing this issue has been fixed in SAN-OS 3.2(2c) to correct unused memory locations. Upgrading to SAN-OS 3.2(2c) or a later software release will ensure Generation 2 modules DS-X9112, DS-X9124, DS-X9148 and DS-X9704 are not affected by this issue.

Note: Customers should not RMA the hardware as this is a software-only issue.

DDTS

To follow the bug ID link below and see detailed bug information, you must be a registered user and you must be logged in.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Product Alert Tool - Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.