Table Of Contents

Operations Support System Troubleshooting

Introduction

Operations Support System Events and Alarms

OSS (1)

OSS (2)

OSS (3)

OSS (4)

OSS (5)

OSS (6)

OSS (7)

OSS (8)

OSS (9)

OSS (10)

OSS (11)

Monitoring Operations Support System Events

Test Report - Operations Support System (1)

Undefined Variable in Known Set - Operations Support System (2)

Undefined Data Column Identification - Operations Support System (3)

Request Handler Instantiation Error - Operations Support System (4)

Structured Query Language Error While Getting Statistics - Operations Support System (5)

Structured Query Language Connection Error - Operations Support System (6)

Simple Network Management Protocol File Read Error - Operations Support System (7)

No Reply Received from Destination - Operations Support System (8)

Simple Network Management Protocol Authentication Error - Operations Support System (9)

Queue Processing Module Database Management Index Failed with Error - Operations Support System (10)

Queue Processing Module Database Management Index Mismatch During Add or Delete - Operations Support System (11)

Troubleshooting Operations Support System Alarms

Undefined Variable in Known Set - Operations Support System (2)

Undefined Data Column Identification - Operations Support System (3)

Request Handler Instantiation Error - Operations Support System (4)

Structured Query Language Error While Getting Statistics - Operations Support System (5)

Structured Query Language Connection Error - Operations Support System (6)

No Reply Received from Destination - Operations Support System (8)

Simple Network Management Protocol Authentication Error - Operations Support System (9)

Queue Processing Module Database Management Index Failed with Error - Operations Support System (10)

Operations Support System Troubleshooting

---

Revised: July 22, 2009, OL-8000-32

Introduction

This chapter provides the information needed to monitor and troubleshoot Operations Support System events and alarms. This chapter is divided into the following sections:

•Operations Support System Events and Alarms - Provides a brief overview of each Operations Support System event and alarm.

•Monitoring Operations Support System Events - Provides the information needed to monitor and correct Operations Support System events.

•Troubleshooting Operations Support System Alarms - Provides the information needed to troubleshoot and correct Operations Support System alarms.

Operations Support System Events and Alarms

This section provides a brief overview of the Operations Support System events and alarms for the Cisco BTS 10200 Softswitch in numerical order. Table 8-1 lists all of the Operations Support System events and alarms by severity.

---

Note Click the Operations Support System message number in Table 8-1 to display information about the event or alarm.

---

Table 8-1 Operations Support System (OSS) Events and Alarms by Severity 

CRITICAL	MAJOR	MINOR	WARNING	INFO
	OSS (2)	OSS (5)	OSS (7)	OSS (1)
	OSS (3)	OSS (8)	OSS (9)
	OSS (4)		OSS (11)
	OSS (6)
	OSS (10)

OSS (1)

For additional information, refer to the "Test Report - Operations Support System (1)" section.

DESCRIPTION	Test Report
SEVERITY	Information (INFO)
THRESHOLD	10000
THROTTLE	0

OSS (2)

To troubleshoot and correct the cause of the alarm, refer to the "Undefined Variable in Known Set - Operations Support System (2)" section.

DESCRIPTION	Undefined Variable in Known Set
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Module Name - STRING [40] Field Name - STRING [40] Field Value - STRING [64]
PRIMARY CAUSE	There was no definition of a data column that could be found in the database.
PRIMARY ACTION	Contact Cisco Technical Assistance Center (TAC) for support.

Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

OSS (3)

To troubleshoot and correct the cause of the alarm, refer to the "Undefined Data Column Identification - Operations Support System (3)" section.

DESCRIPTION	Undefined Data Column Identification
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Noun - STRING [40] Data Column ID - STRING [40]
PRIMARY CAUSE	The database does not contain the required data column that was requested via the Simple Network Management Protocol (SNMP) interface.
PRIMARY ACTION	Contact Cisco TAC for support.

Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

OSS (4)

To troubleshoot and correct the cause of the alarm, refer to the "Request Handler Instantiation Error - Operations Support System (4)" section.

DESCRIPTION	Request Handler Instantiation Error
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	User Name - STRING [40] Host - STRING [40] Subsystem - STRING [64]
PRIMARY CAUSE	Resource limitation has prevented the creation of this object. This may come from a lack of memory or class path problem.
PRIMARY ACTION	Contact Cisco TAC for support.

Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

OSS (5)

To troubleshoot and correct the cause of the alarm, refer to the "Structured Query Language Error While Getting Statistics - Operations Support System (5)" section.

DESCRIPTION	Structured Query Language Error While Getting Statistics
SEVERITY	MINOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Statistics Category - STRING [40]
PRIMARY CAUSE	An error occurred in accessing the Structured Query Language (SQL) database for statistical information in the SNMP subsystem. This may result from a schema error.
PRIMARY ACTION	Contact Cisco TAC for support.

Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

OSS (6)

To troubleshoot and correct the cause of the alarm, refer to the "Structured Query Language Connection Error - Operations Support System (6)" section.

DESCRIPTION	Structured Query Language Connection Error
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Error Exception - STRING [64]
PRIMARY CAUSE	The connection to the database timed out or the database server is not running. This is generated in the SNMP subsystem.
PRIMARY ACTION	Contact Cisco TAC for support.

Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

OSS (7)

To monitor and correct the cause of the event, refer to the "Simple Network Management Protocol File Read Error - Operations Support System (7)" section.

DESCRIPTION	Simple Network Management Protocol File Read Error
SEVERITY	WARNING
THRESHOLD	100
THROTTLE	0
DATAWORDS	Filename - STRING [40]
PRIMARY CAUSE	The Management Information Base (MIB) file is missing or locked from access by the SNMP subsystem.
PRIMARY ACTION	Contact Cisco TAC for support.

Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

OSS (8)

To troubleshoot and correct the cause of the alarm, refer to the "No Reply Received from Destination - Operations Support System (8)" section.

DESCRIPTION	No Reply Received from Destination
SEVERITY	MINOR
THRESHOLD	100
THROTTLE	5
DATAWORDS	JMS Queue Name - STRING [40]
PRIMARY CAUSE	Received when there is no response to a command line interface (CLI) command from the Call Agent platform.
PRIMARY ACTION	If this Event Report is issued while the system is stable (i.e. no device failures) and the traffic is at or below the engineered level, then Cisco TAC technical support should be contacted to investigate the cause. (Contact Cisco TAC.)
SECONDARY ACTION	If components of the system are in the process of failing or restoring while CLI commands are being issued, then this event report is informational and no further action is required.
TERNARY ACTION	The traffic measurement reports can be checked to see if there is more traffic being handled than the engineered level. If this is the situation, then the traffic should be reduced or capacity should be added.

Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

OSS (9)

To troubleshoot and correct the cause of the alarm, refer to the "Simple Network Management Protocol Authentication Error - Operations Support System (9)" section.

DESCRIPTION	Simple Network Management Protocol Authentication Error
SEVERITY	MINOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	RemoteHost - STRING [64]
PRIMARY CAUSE	Network management system (NMS) issuing wrong community string.
PRIMARY ACTION	Ensure community string exists via CLI.

OSS (10)

To troubleshoot and correct the cause of the alarm, refer to the "Queue Processing Module Database Management Index Failed with Error - Operations Support System (10)" section.

DESCRIPTION	Queue Processing Module Database Management Index Failed with Error
SEVERITY	MAJOR
THRESHOLD	50
THROTTLE	0
DATAWORDS	Transaction ID - STRING [32] Sequence Number - EIGHT_BYTES Location Of Error - STRING [16] DBM Result - STRING [64]
PRIMARY CAUSE	Queue processing module (QAM) is receiving an error because of Data Mismatch between what is in the database management (DBM) index (IDX) and what is in the Element Management System (EMS) DB (Oracle).
PRIMARY ACTION	View Transaction Queue and execute Audit on the failed table.
SECONDARY CAUSE	DBM IDX is failing when attempting to perform SQL command.
SECONDARY ACTION	Contact Cisco Support.

Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

OSS (11)

To monitor and correct the cause of the event, refer to the "Queue Processing Module Database Management Index Mismatch During Add or Delete - Operations Support System (11)" section.

DESCRIPTION	Queue Processing Module Database Management Index Mismatch During Add or Delete
SEVERITY	WARNING
THRESHOLD	100
THROTTLE	0
DATAWORDS	Transaction ID - STRING [32] Sequence Number - EIGHT_BYTES Location Of Error - STRING [16] DBM Result - STRING [64]
PRIMARY CAUSE	QAM is received a warning that either an entry already exists in DBM IDX during Add or an entry is nonexistent during a Delete operation.
PRIMARY ACTION	None needed.

Monitoring Operations Support System Events

This section provides the information needed to monitor and correct Operations Support System events. Table 8-2 lists all Operations Support System events in numerical order and provides cross reference to each subsection in this section.

Table 8-2 Cisco BTS 10200 Softswitch OSS Events 

Event Type	Event Name	Event Severity
OSS(1)	Test Report - Operations Support System (1)	INFO
OSS(2)	Undefined Variable in Known Set - Operations Support System (2)	MAJOR
OSS(3)	Undefined Data Column Identification - Operations Support System (3)	MAJOR
OSS(4)	Request Handler Instantiation Error - Operations Support System (4)	MAJOR
OSS(5)	Structured Query Language Error While Getting Statistics - Operations Support System (5)	MINOR
OSS(6)	Structured Query Language Connection Error - Operations Support System (6)	MAJOR
OSS(7)	Simple Network Management Protocol File Read Error - Operations Support System (7)	WARNING
OSS(8)	No Reply Received from Destination - Operations Support System (8)	MINOR
OSS(9)	Simple Network Management Protocol Authentication Error - Operations Support System (9)	MINOR
OSS(10)	Queue Processing Module Database Management Index Failed with Error - Operations Support System (10)	MAJOR
OSS(11)	Queue Processing Module Database Management Index Mismatch During Add or Delete - Operations Support System (11)	WARNING

Test Report - Operations Support System (1)

The Test Report event is for testing the operations support system event category. The event is informational and no further action is required.

Undefined Variable in Known Set - Operations Support System (2)

The Undefined Variable in Known Set alarm (major) indicates that there was not a definition for a data column that could be found in the database. To troubleshoot and correct the cause of the Undefined Variable in Known Set alarm, refer to the "Undefined Variable in Known Set - Operations Support System (2)" section.

Undefined Data Column Identification - Operations Support System (3)

The Undefined Data Column Identification alarm (major) indicates that the database does not contain the required data column that was requested via the SNMP interface. To troubleshoot and correct the cause of the Undefined Data Column Identification alarm, refer to the "Undefined Data Column Identification - Operations Support System (3)" section.

Request Handler Instantiation Error - Operations Support System (4)

The Request Handler Instantiation Error alarm (major) indicates that the creation of the request handler object has failed. To troubleshoot and correct the cause of the Request Handler Instantiation Error alarm, refer to "Request Handler Instantiation Error - Operations Support System (4)" section.

Structured Query Language Error While Getting Statistics - Operations Support System (5)

The Structured Query Language Error While Getting Statistics alarm (minor) indicates that an error occurred while accessing the SQL database for statistical information in the SNMP subsystem. To troubleshoot and correct the cause of the Structured Query Language Error While Getting Statistics alarm, refer to the "Structured Query Language Error While Getting Statistics - Operations Support System (5)" section.

Structured Query Language Connection Error - Operations Support System (6)

The Structured Query Language Connection Error alarm (major) indicates that the connection to the database timed out or the database server is not running. To troubleshoot and correct the cause of the Structured Query Language Connection Error alarm, refer to the "Structured Query Language Connection Error - Operations Support System (6)" section.

Simple Network Management Protocol File Read Error - Operations Support System (7)

The Simple Network Management Protocol File Read Error event serves as a warning that the requested MIB file is missing or locked from access by the SNMP subsystem. To correct the primary cause of the event, contact Cisco TAC. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

No Reply Received from Destination - Operations Support System (8)

The No Reply Received from Destination alarm (minor) indicates that there no reply received from the destination. To troubleshoot and correct the cause of the No Reply Received from Destination alarm, refer to the "No Reply Received from Destination - Operations Support System (8)" section.

Simple Network Management Protocol Authentication Error - Operations Support System (9)

The Simple Network Management Protocol Authentication Error alarm (minor) indicates that the NMS is issuing the wrong community string. To troubleshoot and correct the cause of the Simple Network Management Protocol Authentication Error alarm, refer to the "Simple Network Management Protocol Authentication Error - Operations Support System (9)" section.

Queue Processing Module Database Management Index Failed with Error - Operations Support System (10)

The Queue Processing Module Database Management Index Failed with Error alarm (major) indicates that the QAM is receiving an error because of a data mismatch between the information that is in the DBM IDX and the information that is in the EMS database (Oracle). To troubleshoot and correct the cause of the Queue Processing Module Database Management Index Failed with Error alarm, refer to the "Queue Processing Module Database Management Index Failed with Error - Operations Support System (10)" section.

Queue Processing Module Database Management Index Mismatch During Add or Delete - Operations Support System (11)

The Queue Processing Module Database Management Index Mismatch During Add or Delete event serves as a warning that the QAM has received a warning that either an entry already exists in DBM IDX during an Add operation or an entry is nonexistent during a Delete operation. No further action is required.

Troubleshooting Operations Support System Alarms

This section provides the information needed to monitor and correct Operations Support System alarms. Table 8-3 lists all Operations Support System alarms in numerical order and provides cross reference to each subsection in this section.

Table 8-3 Cisco BTS 10200 Softswitch OSS Alarms 

Alarm Type	Alarm Name	Alarm Severity
OSS(2)	Undefined Variable in Known Set - Operations Support System (2)	MAJOR
OSS(3)	Undefined Data Column Identification - Operations Support System (3)	MAJOR
OSS(4)	Request Handler Instantiation Error - Operations Support System (4)	MAJOR
OSS(5)	Structured Query Language Error While Getting Statistics - Operations Support System (5)	MINOR
OSS(6)	Structured Query Language Connection Error - Operations Support System (6)	MAJOR
OSS(8)	No Reply Received from Destination - Operations Support System (8)	MINOR
OSS(9)	Simple Network Management Protocol Authentication Error - Operations Support System (9)	MINOR
OSS(10)	Queue Processing Module Database Management Index Failed with Error - Operations Support System (10)	MAJOR

Undefined Variable in Known Set - Operations Support System (2)

The Undefined Variable in Known Set alarm (major) indicates that there was not a definition for a data column that could be found in the database. The primary cause of the alarm is that there was no definition of a data column that could be found in the database. To correct the primary cause of the alarm, contact Cisco TAC for support. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

Undefined Data Column Identification - Operations Support System (3)

The Undefined Data Column Identification alarm (major) indicates that the database does not contain the required data column that was requested via the SNMP interface. The primary cause of the alarm is that the database does not contain the required data column that was requested via the SNMP interface. To correct the primary cause of the alarm, contact Cisco TAC for support. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

Request Handler Instantiation Error - Operations Support System (4)

The Request Handler Instantiation Error alarm (major) indicates that the creation of the request handler object has failed. The primary cause of the alarm is that a resource limitation has prevented the creation of this object. A lack of memory or a Class Path problem may cause the alarm. To correct the primary cause of the alarm, contact Cisco TAC for support. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

Structured Query Language Error While Getting Statistics - Operations Support System (5)

The Structured Query Language Error While Getting Statistics alarm (minor) indicates that an error occurred while accessing the SQL database for statistical information in the SNMP subsystem. The primary cause of the alarm is that an error occurred in accessing the SQL database for statistical information in the SNMP subsystem. The alarm may result from schema error. To correct the primary cause of the alarm, contact Cisco TAC for support. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

Structured Query Language Connection Error - Operations Support System (6)

The Structured Query Language Connection Error alarm (major) indicates that the connection to the database timed out or the database server is not running. The primary cause of the alarm is that the connection to the database timed out or the database server is not running. The alarm is generated in the SNMP subsystem. To correct the primary cause of the alarm, contact Cisco TAC for support. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

No Reply Received from Destination - Operations Support System (8)

The No Reply Received from Destination alarm (minor) indicates that there no reply received from the destination. The alarm is received when there is no response to a CLI command from the Call Agent platform. If this alarm is issued while the system is stable (i.e. no device failures) and the traffic is at or below the engineered level, then Cisco TAC technical support should be contacted to investigate the cause. If components of the system are in the process of failing or restoring while CLI commands are being issued, then this alarm is informational and no further action is required. The traffic measurement reports can be checked to see if there is more traffic being handled than the engineered level. If this is the situation, then the traffic should be reduced or capacity should be added. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

Simple Network Management Protocol Authentication Error - Operations Support System (9)

The Simple Network Management Protocol Authentication Error alarm (minor) indicates that the NMS is issuing the wrong community string. The primary cause of the alarm is that the NMS issuing the wrong community string. To correct the primary cause of the alarm, ensure the correct community string exists via CLI.

Queue Processing Module Database Management Index Failed with Error - Operations Support System (10)

The Queue Processing Module Database Management Index Failed with Error alarm (major) indicates that the QAM is receiving an error because of a data mismatch between the information that is in DBM IDX and the information that is in the EMS database (Oracle). The primary cause of the alarm is that the QAM is receiving an error because of a data mismatch between what is in the DBM IDX and what is in the EMS db (oracle). To correct the primary cause of the alarm, view the Transaction Queue and execute an Audit on the failed table. The secondary cause of the alarm is that the DBM IDX is failing when attempting to perform a SQL command. To correct the secondary cause of the alarm, contact Cisco TAC. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.