Table Of Contents

Preface

Introduction

Objective

Audience

Organization

Conventions

Updates to this Document for Release 4.5.x

New Events and Alarms (Release 3.5 to Release 4.5.1)

Modified Events and Alarms (Release 3.5 to Release 4.5.1)

Deprecated Events and Alarms (Release 4.5 to Release 4.5.1)

New Events and Alarms (Release 4.5.13 to Release 4.5.1)

Modified Events and Alarms (Release 4.5.13 to Release 4.5.1)

Deprecated Events and Alarms (Release 4.5.13 to Release 4.5.1)

Obtaining Documentation and Submitting a Service Request

Preface

---

Revised: July 22, 2009, OL-8000-32

Introduction

This document provides detailed troubleshooting procedures for the Cisco BTS 10200 Softswitch. It provides the procedures for troubleshooting network, subscriber, billing, operations and maintenance, system administrative problems. It also includes details for aiding troubleshooting by utilizing diagnostic and trace procedures.

Objective

The maintenance personnel/troubleshooter of a Cisco BTS 10200 Softswitch can use this document to better understand how to troubleshoot the Cisco BTS 10200 Softswitch and quickly clear network problems.

Audience

This document is designed for engineers, technicians, and system administrators who troubleshoot and repair the Cisco BTS 10200 Softswitch.

Organization

This Troubleshooting Guide contains the following chapters:

•Chapter 1, "Troubleshooting Overview" - Provides an overview of troubleshooting the Cisco BTS 10200 Softswitch.

•Chapter 2, "Audit Troubleshooting" - Provides the information needed to monitor and troubleshoot Audit events and alarms.

•Chapter 3, "Billing Troubleshooting" - Provides the information needed to monitor and troubleshoot Billing events and alarms.

•Chapter 4, "Call Processing Troubleshooting" - Provides the information needed to monitor and troubleshoot Call Processing events and alarms.

•Chapter 5, "Configuration Troubleshooting" - Provides the information needed to monitor and troubleshoot Configuration events and alarms.

•Chapter 6, "Database Troubleshooting" - Provides the information needed to monitor and troubleshoot Database events and alarms.

•Chapter 7, "Maintenance Troubleshooting" - Provides the information needed to monitor and troubleshoot Maintenance events and alarms.

•Chapter 8, "Operations Support System Troubleshooting" - Provides the information needed to monitor and troubleshoot Operations Support System events and alarms.

•Chapter 9, "Security Troubleshooting" - Provides the information needed to monitor and troubleshoot Security events and alarms.

•Chapter 10, "Signaling Troubleshooting" - Provides the information needed to monitor and troubleshoot Signaling events and alarms.

•Chapter 11, "Statistics Troubleshooting" - Provides the information needed to monitor and troubleshoot Statistics events and alarms.

•Chapter 12, "System Troubleshooting" - Provides the information needed to monitor and troubleshoot System events and alarms.

•Chapter 13, "Network Troubleshooting" - Provides the information needed to conduct network troubleshooting on the Cisco BTS 10200 Softswitch.

•Chapter 14, "General Troubleshooting" - Provides the general troubleshooting information needed to conduct troubleshooting on the Cisco BTS 10200 Softswitch.

•Chapter 15, "Diagnostic Tests" - Describes the diagnostic tests that can be performed on media gateways, subscriber terminations, and trunk terminations.

•Chapter 16, "Disaster Recovery Procedures" - Describes how to recover the database in a disaster situation, how to recover the database from another database, and how to recover data from the Call Agent shared memory.

•Chapter 17, "Disk Replacement" - Describes how to manually recover a Cisco BTS 10200 Softswitch system.

•Appendix A, "Recoverable and Nonrecoverable Error Codes" - Lists normal, recoverable and nonrecoverable error codes for the Cisco BTS 10200 Softswitch.

•Appendix B, "System Usage of MGW Keepalive Parameters, Release 4.5.x" - Explains how the Cisco BTS 10200 Softswitch determines the connectivity status between itself and a media gateway (MGW)

Conventions

This document uses the following conventions:

---

Note Means reader take note. Notes contain helpful suggestions or references to material not covered in the manual.

---

---

Caution Means reader be careful. In this situation, you might do something that could result in equipment damage or loss of data.

---

Updates to this Document for Release 4.5.x

This document includes all of the information that was contained in the previous issue (the Release 4.5 Cisco BTS 10200 Softswitch Troubleshooting Guide, OL-8000-08), and has been updated for the Release 4.5 Cisco BTS 10200 Softswitch Troubleshooting Guide, OL-8000-09 as described in this Preface.

Updates are contained in the following chapters and appendixes of this document:

•Updated Chapter 16, "Disaster Recovery Procedures"

Additionally, this document has been updated to OL-8000-10. Updates are contained in the following chapters and sections:

•Updated Chapter 6, "Database Troubleshooting"

•Updated Chapter 6, "DATABASE (11)"

•Updated Chapter 6, "Element Management System Database Alert.log Alerts - Database (11)"

The updates to this document to OL-8000-11 include:

•Removed AUDIT (9) event information

•Added BILLING (60), MAINTENANCE (119), MAINTENANCE (120), SIGNALING (166), SIGNALING (167), and SYSTEM (15) event information.

The updates to this document for OL-8000-12 include:

•Adding Appendix B, "System Usage of MGW Keepalive Parameters, Release 4.5.x" to the document.

The updates to this document for OL-8000-13 include:

•Updated Chapter 16, "Disaster Recovery Procedures" to include the procedure for restoration of the Automatic Shared Memory Backup (ASMB)

The updates to this document for OL-8000-14 include:

•Updated Chapter 16, "Disaster Recovery Procedures"

The updates to this document for OL-8000-15 included:

•Updated DATABASE (11) primary cause and primary action

The updates to this document for OL-8000-16 included:

•Updated Chapters 14, 15, and 16 to include troubleshooting information for Release 4.5.1.

The updates to this document for OL-8000-17 included:

•Updated Chapter 7 to include the Maintenance 122 and Maintenance 123 alarms and events.

The update to this document for OL-8000-18 included:

•Removal of the Release Cause Code appendix, update of Chapter 14 and Chapter 15, and update of the System Usage of MGW Keepalive Parameters appendix.

The update of this document for OL-8000-19 included:

•Update of the threshold and throttle information in Chapter 1

The updates of this document for OL-8000-20 and OL-8000-21 included:

•Update of the keepalive information in Appendix B.

The update of this document from OL-8000-21 to OL-8000-22 include incorporating the delta alarm and event information from Release 3.5 to Release 4.5.x.

The update of this document from OL-8000-22 to OL-8000-23 included:

•Update of the keepalive information in Appendix B.

The update of this document from OL-8000-23 to OL-8000-24 included:

•Adding the event and alarm differences between release 4.5.13 and 4.5.1.

The update of this document from OL-8000-24 to OL-8000-25 included:

•Removing the Throttle/Threshold erroneous note

The update of this document from OL-8000-25 to OL-8000-26 included updating the disk replacement procedures in Chapter 17.

The update of this document from OL-8000-26 to OL-8000-27 included updating the recovering shared memory procedure and call agent database download and restore.

The update to OL-8000-28 included updating the information for the Audit 4 alarm.

The update to OL-8000-29 included adding the "PSTN Trunk Testing" section on page 15-80 to Chapter 15.

The update to OL-8000-30 included making changes to the "Automatic Shared Memory Backup Restore" section on page 16-30.

The update to OL-8000-31 included adding the "Oracle Database Tool Restart" section on page 13-8.

The update to OL-8000-32 included updating the "Recovering Shared Memory" section on page 16-26.

New Events and Alarms (Release 3.5 to Release 4.5.1)

lists the new events and alarms that were incorporated into the BTS 10200 system as part of the update from Release 3.5 to Release 4.5.1.

Table 1 New Events and Alarms (Release 3.5 to Release 4.5.1) 

Type and Number	Description
AUDIT (2)	Start or Stop of Signaling System 7-Circuit Identification Code Audit
AUDIT (3)	Signaling System 7 Circuit Identification Code Audit Terminated Before Successful Completion
AUDIT (4)	Call Exceeds a Long-Duration Threshold
AUDIT (6)	Major Internal Audit Failure
AUDIT (7)	Minor Internal Audit Failure
AUDIT (8)	Warning From Internal Audit
AUDIT (10)	Call Data Audit Complete
AUDIT (11)	Critical Network Time Protocol Service Failure
AUDIT (12)	Major Network Time Protocol Service Failure
AUDIT (13)	Minor Network Time Protocol Service Failure
AUDIT (14)	Network Time Protocol Service Warning
AUDIT (15)	Critical Index Shared Memory Error
AUDIT (16)	Process Heap Memory Usage Exceeds Minor Threshold Level
AUDIT (17)	Process Heap Memory Usage Exceeds Major Threshold Level
AUDIT (18)	Process Heap Memory Usage Exceeds Critical Threshold Level
AUDIT (19)	Recovered Memory of Stale Call
AUDIT (20)	Audit Found Lost Call Data Record
BILLING (52)	Bulk Data Management System Stopped Generating New Billing File
BILLING (57)	Prepaid Subscriber Call Attempt Failed Because of Balance
BILLING (58)	Signaling Prepaid Server Inaccessible
BILLING (60)	Bad File Detected During Startup
CALLP (38)	Release Cause 25 Exchange Routing Error Received
CALLP (39)	Test Call Blocked Due to Congestion or Isolation
CALLP (40)	Interactive Voice Response Real Time Transport Protocol Session Fail
CALLP (41)	INVITE Message From Unauthorized Call Agent
CONFIG (5)	Feature-Server Database and Command Line Host Mismatch
DATABASE (19)	Daily Database Backup Completed Successfully
DATABASE (20)	Replication Data Flush Timeout During Switchover
DATABASE (21)	Database Statistics Collection Exception
DATABASE (25)	Secure File Transfer Protocol Transfer Failed
DATABASE (26)	File Write Error
MAINTENANCE (86)	System Health Report Collection Error
MAINTENANCE (87)	Status Update Process Request Failed
MAINTENANCE (88)	Status Update Process Database List Retrieval Error
MAINTENANCE (89)	Status Update Process Database Update Error
MAINTENANCE (90)	Disk Partition Moderately Consumed
MAINTENANCE (91)	Internet Protocol Manager Configuration File Error
MAINTENANCE (92)	Internet Protocol Manager Initialization Error
MAINTENANCE (93)	Internet Protocol Manager Interface Failure
MAINTENANCE (94)	Internet Protocol Manager Interface State Change
MAINTENANCE (95)	Internet Protocol Manager Interface Created
MAINTENANCE (96)	Internet Protocol Manager Interface Removed
MAINTENANCE (102)	Thread Watchdog Counter Close to Expiry for a Thread
MAINTENANCE (103)	Central Processing Unit is Offline
MAINTENANCE (104)	Aggregration Device Address Successfully Resolved
MAINTENANCE (105)	Unprovisioned Aggregration Device Detected
MAINTENANCE (106)	Aggregration Device Address Resolution Failure
MAINTENANCE (107)	No Heartbeat Messages Received Through Interface From Router
MAINTENANCE (108)	A Log File Cannot be Transferred
MAINTENANCE (109)	Five Successive Log Files Cannot be Transferred
MAINTENANCE (110)	Access to Log Archive Facility Configuration File Failed or File Corrupted
MAINTENANCE (111)	Cannot Login to External Archive Server
MAINTENANCE (119)	Periodic Shared Memory Database Backup Failure
MAINTENANCE (120)	Periodic Shared Memory Database Backup Success
MAINTENANCE (122)	Northbound Provisioning Message is Retransmitted
MAINTENANCE (123)	Northbound Provisioning Message Dropped Due To Full Index Table
SECURITY (2)	Invalid Credentials Presented by a Session Initiation Protocol Phone
SECURITY (3)	Internet Protocol Security Connection Down
SECURITY (4)	Internet Protocol Security Media Terminal Adapter Key Establish Error
SECURITY (5)	Internet Protocol Security Outgoing Security Association Not Found
SECURITY (6)	Secure Session Initiation Protocol Endpoint Validation Failure
SIGNALING (109)	Stream Control Transmission Protocol Association Failure
SIGNALING (110)	Signaling Gateway Group is Out-of-Service
SIGNALING (111)	Stream Control Transmission Protocol Association Degraded (One of Two Internet Protocol Connections Down)
SIGNALING (112)	Stream Control Transmission Protocol Association Configuration Error
SIGNALING (113)	Signaling Gateway Failure
SIGNALING (114)	Signaling Gateway Process is Out-of-Service
SIGNALING (115)	Invalid Routing Context Received
SIGNALING (116)	Destination Point Code User Part Unavailable
SIGNALING (121)	Message Transfer Part 3 User Adapter Cannot Go Standby
SIGNALING (122)	Message Transfer Part 3 User Adapter Cannot Go Active
SIGNALING (124)	Remote Subsystem is Out Of Service
SIGNALING (125)	Signaling Connection Control Part Routing Error
SIGNALING (126)	Signaling Connection Control Part Binding Failure
SIGNALING (127)	Transaction Capabilities Application Part Binding Failure
SIGNALING (132)	Transaction Capabilities Application Part Reaches the Provisioned Resource Limit
SIGNALING (133)	Unable to Decode Generic Transport Descriptor Message
SIGNALING (134)	Signaling System 7 Message Encoding Failure
SIGNALING (135)	Signaling System 7 Message Decoding Failure
SIGNALING (136)	Signaling System 7 Message Invalid Received
SIGNALING (137)	Signaling System 7 Confusion Message Received
SIGNALING (138)	Number of Open Session Initiation Protocol Connections is Reaching Engineered Limit
SIGNALING (139)	Signaling System 7 Trunk was Found to be in Erroneous State
SIGNALING (140)	Unanswered Information Message
SIGNALING (141)	Address not Resolved by Domain Name System Server
SIGNALING (142)	Session Initiation Protocol Trunk Operationally Out-of-Service
SIGNALING (143)	Internet Protocol Interface Link to the Signaling System 7 Signaling Gateway is Down
SIGNALING (144)	All Internet Protocol Interface Links to Signaling System 7 Signaling Gateway are Down
SIGNALING (145)	One Internet Protocol Interface to Signaling System 7 Signaling Gateway is Down
SIGNALING (146)	All Retransmission Attempts of Session Initiation Protocol Request or Response Failed
SIGNALING (147)	Domain Name System Service Addresses Exhausted
SIGNALING (150)	Stream Control Transmission Protocol Association Congested
SIGNALING (151)	Termination Permanent Error Code Received
SIGNALING (152)	Termination Transient Error Received
SIGNALING (166)	No Routing Keys are Active
SIGNALING (167)	No Signaling Gateways are Active
STATISTICS (12)	Measurement Handshake Error - Schema Inconsistency
STATISTICS (13)	Traffic and Measurements Module Application Programming Interface Failure
SYSTEM (12)	Software Error
SYSTEM (13)	Multiple Readers and Multiple Writers Maximum Queue Depth Reached
SYSTEM (14)	Multiple Readers and Multiple Writers Queue Reached Low Queue Depth
SYSTEM (15)	Multiple Readers and Multiple Writers Throttle Queue Depth Reached

Modified Events and Alarms (Release 3.5 to Release 4.5.1)

lists the BTS 10200 modified events and alarms that were modified as part of the upgrade from Release 3.5 to Release 4.5.1.

Table 2 Modified Events and Alarms (Release 3.5 to Release 4.5.1) 

Type and Number	Description
BILLING (6)	File Transfer Protocol/Secure File Transfer Protocol Transfer Failed
BILLING (13)	Database Connection Error
BILLING (14)	File Open Error
BILLING (15)	File Write Error
BILLING (33)	File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid
BILLING (41)	Message Content Error
BILLING (44)	Record Keeping System Switch Occurred
BILLING (45)	Event Message Log File Opened
BILLING (46)	Event Message Log File Closed
BILLING (53)	Event Message Disk Space 50 Percent Full
BILLING (56)	Billing Data Corruption Detected
CALLP (9)	No Route Available for Carrier Dialed
DATABASE (5)	Element Management System Database HeartBeat Process Died
DATABASE (17)	Data Replication Failed
MAINTENANCE (41)	Administrative State Change Successful with Warning
MAINTENANCE (47)	Thread Watchdog Counter Expired for a Thread
MAINTENANCE (57)	Memory and Swap are Consumed at Critical Levels
MAINTENANCE (58)	Memory and Swap are Consumed at Abnormal Levels
MAINTENANCE (65)	Disk Partition Critically Consumed
MAINTENANCE (66)	Disk Partition Significantly Consumed
MAINTENANCE (71)	Local Domain Name System Server Response Too Slow
MAINTENANCE (72)	External Domain Name System Server Response Too Slow
MAINTENANCE (73)	External Domain Name System Server not Responsive
MAINTENANCE (82)	Average Outgoing Heartbeat Period Exceeds Critical Limit
OSS (6)	Structured Query Language Connection Error
SIGNALING (23)	Destination Point Code Unavailable
SIGNALING (25)	Unanswered Blocking
SIGNALING (26)	Unanswered Unblocking Message
SIGNALING (27)	Unanswered Circuit Group Blocking
SIGNALING (28)	Unanswered Circuit Group Unblocking
SIGNALING (29)	Unanswered Circuit Query Message
SIGNALING (30)	Unanswered Circuit Validation Test
SIGNALING (31)	Unanswered Reset Circuit
SIGNALING (32)	Unanswered Group Reset
SIGNALING (33)	Unanswered Release
SIGNALING (34)	Unanswered Continuity Check Request
SIGNALING (42)	Continuity Testing Message Received on the Specified Circuit Identification Code
SIGNALING (43)	Release Complete Received in Response to Reset Circuit Message on the Specified Circuit Identification Code
SIGNALING (44)	Continuity Recheck is Performed on Specified Circuit Identification Code
SIGNALING (45)	Circuit is UNEQUIPPED on Remote Side
SIGNALING (46)	Specified Circuit Identification Code is Invalid for the Operation
SIGNALING (49)	A General Processing Error Encountered
SIGNALING (50)	Unexpected Message for the Call State is Received: Clear Call
SIGNALING (51)	Set Trunk State as Remotely Unequipped
SIGNALING (52)	Set Trunk State as NOT Remotely Blocked
SIGNALING (53)	Set Trunk State as Remotely Blocked
SIGNALING (54)	Circuit Validation Test Aborted
SIGNALING (55)	Circuit Validation Successful
SIGNALING (57)	Continuity Recheck Failed
SIGNALING (58)	Continuity Recheck Successful
SIGNALING (68)	Media Gateway Endpoints are out of Service at Gateway
SIGNALING (69)	Call Agent and Feature Server Communication Message Timeout
SIGNALING (89)	Gatekeeper not Available/Reachable
SIGNALING (90)	Alternate Gatekeeper is not Responding
SIGNALING (107)	Logical Internet Protocol Addresses not Mapped Correctly
SIGNALING (108)	Simplex Only Operational Mode
SIGNALING (118)	Circuit Verification Response Received with Failed Indication

Deprecated Events and Alarms (Release 4.5 to Release 4.5.1)

lists the BTS 10200 events and alarms that were deprecated as part of the upgrade from Release 3.5 to Release 4.5.1.

Table 3 Deprecated Events and Alarms (Release 3.5 to Release 4.5.1) 

Type and Number	Description
BILLING(9)	Duplicate Event Message Received
BILLING (10)	Database Lock Error
BILLING (11)	Database Unlock Error
BILLING (12)	Table Pointer Read Error
BILLING(16)	Time Point Calculation Error
BILLING(17)	Shared Memory Add Failure
BILLING(18)	Error Calling MKTIME Function
BILLING(19)	Call Detail Block Database Write Error
BILLING(20)	Call Detail Block Database Read Error
BILLING(21)	Unknown Billing Event Message Received
BILLING(22)	Error Releasing Staging Area in Shared Memory
BILLING(23)	Error Releasing Busy List Entry in Shared Memory
BILLING(24)	Update Table Pointer Error
BILLING(26)	Shared Memory Update Failed
BILLING(27)	Shared Memory Entry Remove Failed
BILLING(28)	Wrong Port Number Specified
BILLING(34)	All Billing Links at CA Down
BILLING(39)	RADIUS Accounting Receive Failure
BILLING(43)	Invalid Billing Configuration—Program Exiting
MAINTENANCE(59)	Disk Reports Greater Than 70% Utilization
MAINTENANCE(60)	Disk Utilization has Returned to Normal Levels
SIGNALING(2)	Message Encoding Error
SIGNALING(3)	Message Decoding Error
SIGNALING(47)	Indicated Circuit is Out of Order
SIGNALING(48)	Indicated Circuit is Back in Service
SIGNALING(56)	Circuit Validation Failure
SIGNALING(67)	Gateway/Termination Returning Transient Error for the MGCP
SIGNALING(123)	Destination Point Code is Unavailable
SIGNALING(128)	INAP Binding Failure
SIGNALING(131)	INAP Reaches the Provisioned Resource Limit

New Events and Alarms (Release 4.5.13 to Release 4.5.1)

lists the new events and alarms that were incorporated into the BTS 10200 system as part of the update from Release 4.5.13 to Release 4.5.1.

Table 4 New Events and Alarms (Release 4.5.13 to Release 4.5.1) 

Type and Number	Description
BILLING (60)	Bad File Detected During Startup
DATABASE (25)	Secure File Transfer Protocol Transfer Failed
DATABASE (26)	File Write Error
MAINTENANCE (118)	Domain Name Server Zone Database does not Match Between the Primary Domain Name Server and the Internal Secondary Authoritative Domain Name Server (DNS Zone Database does not Match Between the Primary DNS and the ISADS)
MAINTENANCE (119)	Periodic Shared Memory Database Backup Failure
MAINTENANCE (120)	Periodic Shared Memory Database Backup Success
SYSTEM (9)	Timer Start Failure

Modified Events and Alarms (Release 4.5.13 to Release 4.5.1)

lists the BTS 10200 modified events and alarms that were modified as part of the upgrade from Release 3.5 to Release 4.5.1.

Table 5 Modified Events and Alarms (Release 4.5.13 to Release 4.5.1) 

Type and Number	Description
BILLING (6)	File Transfer Protocol/Secure File Transfer Protocol Transfer Failed
BILLING (14)	File Open Error
BILLING (15)	File Write Error
BILLING (56)	Billing Data Corruption Detected
SECURITY (6)	Secure Session Initiation Protocol Endpoint Validation Failure
SIGNALING (69)	Call Agent and Feature Server Communication Message Timeout
SIGNALING (148)	Softswitch Audit Released Stale Memory
SYSTEM (7)	Index List Remove Error

Deprecated Events and Alarms (Release 4.5.13 to Release 4.5.1)

There were no deprecated events or alarms between Release 4.5.13 and Release 4.5.1.

Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What's New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at:

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

Subscribe to the What's New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free service and Cisco currently supports RSS Version 2.0.