Table Of Contents
Troubleshooting Cisco IPICS Network Processes
Performing Tomcat Service Procedures
Checking the Status of the Tomcat Service
Manually Stopping the Tomcat Service
Manually Starting the Tomcat Service
Manually Restarting the Tomcat Service
Performing Database Server Procedures
Checking the Status of the Database Server
Manually Restarting the Database Server
Manually Starting the Database Server
Performing License Manager Procedures
Checking the Status of the License Manager
Restarting the License Manager
Manually Starting the License Manager
Performing Dial Engine Procedures
Checking the Status of the Dial Engine
Manually Stopping the Dial Engine
Manually Restarting the Dial Engine
Manually Starting the Dial Engine
Performing CSA Procedures
Viewing CSA Log Messages
Manually Stopping CSA
Manually Starting CSA
Troubleshooting Cisco IPICS Network Processes
When you boot up the Cisco IPICS server, the server software automatically starts the following network processes:
•
Tomcat service
•
Database server
•
License manager
•
Dial engine (if the policy engine is licensed for your server)
•
Cisco Security Agent (CSA)
This chapter provides information to help you to troubleshoot these processes and includes the following sections:
•
Performing Tomcat Service Procedures
•
Performing Database Server Procedures
•
Performing License Manager Procedures
•
Performing Dial Engine Procedures
•
Performing CSA Procedures
Note
This chapter documents procedures that use command-line interface (CLI) commands. If the tomcat service and database server are both running, you can check their status without using CLI by logging in to the Administration Console, navigating to the Serviceability > Diagnostics window, and viewing the information in the Diagnostic Summary pane.
Performing Tomcat Service Procedures
The tomcat service contains all of the Cisco IPICS web-based applications. The tomcat service runs processes that are required for the functional operation of Cisco IPICS, and must run continuously for you to access the Administration Console and other web applications.
Cisco IPICS includes a cron job that acts as a safeguard to make sure that the tomcat service continues to run. This cron job checks the status of the tomcat service every 60 seconds and can restart the service automatically, if the tomcat service stops.
This section includes information about the tomcat service in the following topics:
•
Checking the Status of the Tomcat Service
•
Manually Stopping the Tomcat Service
•
Manually Starting the Tomcat Service
•
Manually Restarting the Tomcat Service
Checking the Status of the Tomcat Service
You can check the status of the tomcat service by navigating to the Serviceability > Diagnostics window of the Administration Console and viewing the Cisco IPICS Tomcat Web Server Status field.
If the tomcat service or the database server is not running, you cannot check its status in the Administration Console. In this case, you must enter CLI commands to check its status. To check the status of the tomcat service by using CLI commands, perform the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
Check the status of the tomcat service by entering the following command:
[root]# service ipics_tomcat status
If the tomcat service is running properly, the command returns a process similar to the following example:
Tomcat process (pid: 24025) is running on the system
If the tomcat service is not running, you should see a response that is similar to the following example:
Tomcat is not running on the system.
If the tomcat service is not running, you can start it manually by entering the service ipics_tomcat start CLI command from the root user ID. For more information, see the "Manually Starting the Tomcat Service" section.
Manually Stopping the Tomcat Service
If you do not want any users to access the Administration Console when you perform system maintenance tasks, such as database-related activities, you can stop the tomcat service.
To stop the tomcat service, perform the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
The terminal console displays.
Step 2
To stop the tomcat service, enter the following command:
[root]# service ipics_tomcat stop
If the tomcat service stops, Cisco IPICS displays the message [OK].
If the tomcat service does not stop, an error message displays that indicates that Cisco IPICS cannot stop the service; if you receive this message, continue to Step 3.
Step 3
If the tomcat service fails to stop, you can terminate the processes that are running by performing the following procedure:
a.
To check which tomcat processes are still running, enter the following grep command, which returns information about the tomcat processes that continue to run:
[root]# ps -ef | grep tomcat
This command shows you a list of the running tomcat processes.
b.
Note the Process IDs, which display in the second column of the grep results.
c.
To stop the tomcat processes that are still running, enter the following command:
[root]# kill -9 <process-id>
where:
<process-id> specifies the Process IDs that you noted in Step b.
d.
Repeat Step c for every tomcat process that is running.
Step 4
Check the status of the tomcat service by entering the following command:
[root]# service ipics_tomcat status
If the tomcat service stops successfully, the following message displays:
Tomcat is not running on the system.
Step 5
If the tomcat service does not stop, check the ipics.log file to gather information on the nature of the problem by entering the following command:
[root]# tail -75 /root/tomcat/current/logs/ipics.log
Note
The ipics.log file contains messages regarding all transactions that occur in the Cisco IPICS server, including tomcat service transactions.
The last 75 lines of the ipics.log file displays.
Step 6
Attempt to fix the problem based on the information that you view in the log file.
The information might include information about a running process that could not stop.
Step 7
If you cannot resolve the problem by using the information in the ipics.log file, contact your Cisco technical support representative for further assistance.
Manually Starting the Tomcat Service
If the cron job fails to start the tomcat service successfully, or if you stop the tomcat service, you can start the service manually by using CLI commands.
To manually start the tomcat service, perform the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
A terminal window displays.
Step 2
To start the tomcat service, enter the following command:
[root]# service ipics_tomcat start
If the tomcat service starts successfully, Cisco IPICS displays the message [OK].
Note
After you start the tomcat service, there may be a delay of a few minutes before users can access the Administration Console.
Step 3
If the tomcat service does not successfully start, check the ipics.log file to gather information on the nature of the problem by entering the following command:
[root]# tail -75 /root/tomcat/current/logs/ipics.log
Note
The ipics.log file contains messages regarding all transactions that occur in the Cisco IPICS server, including tomcat service transactions.
The last 75 lines of the ipics.log file displays.
Step 4
Attempt to fix the problem based on the information that you view in the log file.
The information might contain a Java exception error or provide information about a process that did not start.
Step 5
If you cannot resolve the problem by using the information in the ipics.log file, contact your Cisco technical support representative for further assistance.
Manually Restarting the Tomcat Service
To restart the tomcat service, perform the following procedure:
Note
When you restart the tomcat service, the script logs out any users who are logged in to the Administration Console.
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
To restart the tomcat service, enter the following command:
[root]# service ipics_tomcat restart
After restarting the tomcat service, Cisco IPICS displays the message [OK].
Note
After you start the tomcat service, there may be a delay of a few minutes before users can access the Administration Console.
Performing Database Server Procedures
The database server performs all database-related activities in Cisco IPICS, such as backup and restore operations and database updates.
This section includes the procedures to start, stop, and check the status of the database server in the following topics:
•
Checking the Status of the Database Server
•
Manually Restarting the Database Server
•
Manually Starting the Database Server
Checking the Status of the Database Server
You can check the status of the database server via the Administration Console. To do so, navigate to the Serviceability > Diagnostics window and view the Diagnostic Summary area. Navigate to the Cisco IPICS Database Status area in this pane to view the database server status.
If the database server is stopped, you cannot log in to the Administration Console to check its status; however, you can check the status of the database server by entering CLI commands. To check the status of the database server, perform the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
To check the status of the database server, enter the following command:
[root]# service ipics_db status
If the database server is running properly, the server displays a message that is similar to the following example:
Ipics Database is running...
oninit (pid 21286 21285 21284 21283 21282 21281 21280) is running...
If the database server is not running, the server displays a message that is similar to the following example:
Ipics Database is stopped.
If the database server is not running, you can manually start the database server by performing the procedure in the "Manually Starting the Database Server" section.
Manually Restarting the Database Server
If you experience Cisco IPICS server performance issues, determine whether the database server is the cause of the problem by checking the system resources that the database is using. To check system resources, perform one of the following actions:
•
From the Administration Console, navigate to the Serviceability > Dashboard window and check the memory information that is displayed in the System Dashboard area.
•
Log in to a terminal console session by using the root user ID; then, enter the top command.
Note
The top command displays a listing of the server processes that are using the greatest amount of CPU memory.
If you determine that the Cisco IPICS processes are using a large amount of memory, you can restart the database server, which might speed up network processes.
To restart the database server, perform the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
To restart the database server, enter the following command:
[root]# service ipics_db restart
Cisco IPICS displays the message [OK] when the database server successfully stops and displays the message [OK] again when the database server successfully restarts.
Step 3
If you receive an error message after you attempt to restart the database server, contact your Cisco technical support representative for further assistance.
Manually Starting the Database Server
Cisco IPICS starts the database server when the server boots up. You can also start the database server manually if you determine that the database has stopped. To check whether the database is running, see the "Checking the Status of the Database Server" section.
To manually start the database server from a terminal console session, perform the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
To start the database server, enter the following command:
[root]# service ipics_db start
If the database server starts successfully, Cisco IPICS displays the message [OK].
Step 3
If the database does not successfully start, contact your Cisco technical support representative for further assistance.
Tip
To check the log file for the database, enter the following command:
[root]# more /opt/cisco/ipics/database/logs/diagnostics.log
The diagnostics.log file might provide you with additional details about the reason that the database could not start. Messages such as "Assert failed" or "PANIC" indicate a problem with the database that requires the assistance of your Cisco technical support representative.
Performing License Manager Procedures
The license manager is the network process that manages the Cisco IPICS licenses.
The license manager checks for new licenses every 24 hours. For a new license file to take effect immediately, you must restart the license manager.
This section includes the procedures to start, stop, and check the status of the license manager and includes the following topics:
•
Checking the Status of the License Manager
•
Restarting the License Manager
•
Manually Starting the License Manager
Checking the Status of the License Manager
To check the status of the license manager from the Cisco IPICS Administration Console, navigate to the Serviceability > Diagnostics window and view the Diagnostic Summary area. Navigate to the Cisco IPICS Tomcat Web Server Status area in this pane to view the license manager status.
Tip
Any field that includes the words lmgrd contains information about the license manager.
To manually check the status of the license manager, perform the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
To check the status of the license manager, enter the following command:
[root]# service ipics_lm status
If the license manager is running, the server displays a message that is similar to the following example:
ipics_lm is running (PID 20859).
If the license manager is not running, the server displays a message that is similar to the following example:
Step 3
If the license manager is not running, you can manually start the license manager by following the procedure in the "Manually Starting the License Manager" sectionl.
Restarting the License Manager
If you add files, or change the system date, you must restart the license manager for the license and date changes to take effect.
To restart the license manager from the Administration Console, navigate to the Administration > License Management window and click the Apply button.
To restart the license manager by using CLI commands, perform the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
To restart the license manager, enter the following command:
[root]# service ipics_lm restart
Cisco IPICS displays the message [OK] when the license manager successfully stops and displays the message [OK] again when the license manager successfully restarts.
Step 3
If you receive an error message that indicates that the license manager or lmgrd process could not start, contact your Cisco technical support representative for further assistance.
Manually Starting the License Manager
If the license manager has stopped, you should be able to restart it from the Administration Console by navigating to the Administration > License Management window and clicking the Apply button. You can also manually start the license manager from a terminal console session by performing the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
To start the license manager, enter the following command:
[root]# service ipics_lm start
If the license manager starts successfully, Cisco IPICS displays the message [OK].
Step 3
If the license manager does not start, check the status by performing the actions that are documented in the "Checking the Status of the License Manager" section.
Step 4
If you cannot start the license manager, contact your Cisco technical support representative for further assistance.
Performing Dial Engine Procedures
The dial engine controls the dial-in and dial-out functionality for the policy engine. For more information about the policy engine and dial engine, refer to the "Using the Cisco IPICS Policy Engine" chapter in the Cisco IPICS Server Administration Guide, Release 2.1(1).
Note
Your Cisco IPICS system must be licensed for the policy engine before you can perform dial engine procedures. To check whether you are licensed for the policy engine, navigate to the Administration > License Management > Summary tab in the Administration Console and check the Policy Engine Base License field. If your system is licensed for the policy engine, the field displays a status of Licensed. For more information about licenses, refer to the "Performing Cisco IPICS System Administrator Tasks" chapter in the Cisco IPICS Server Administration Guide, Release 2.1(1).
This section provides information about starting, stopping, restarting and checking the status of the dial engine and includes the following topics:
•
Checking the Status of the Dial Engine
•
Manually Stopping the Dial Engine
•
Manually Restarting the Dial Engine
•
Manually Starting the Dial Engine
Checking the Status of the Dial Engine
To check the status of the dial engine, perform the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
To check the status of the dial engine, enter the following command:
[root]# service ippe_dial_engine status
If the dial engine is running properly, the server displays a message that is similar to the following example:
CVD process (pid 11290) is running...
Engine process (pid 11670) is running...
If the dial engine processes are not running, the server displays a message that is similar to the following example:
CVD process is NOT running...
Engine process is NOT running...
Step 3
If the dial engine is not running, you can manually start the dial engine by performing the procedure in the "Manually Starting the Dial Engine" section.
Manually Stopping the Dial Engine
To stop the dial engine by using CLI commands, perform the following procedure:
Note
Cisco IPICS disconnects all active dial-in and dial-out calls when you stop the dial engine.
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
To stop the dial engine, enter the following command:
[root]# service ippe_dial_engine stop
Cisco IPICS displays the message [OK] when the dial engine processes successfully stop.
Manually Restarting the Dial Engine
To restart the dial engine by using CLI commands, perform the following procedure:
Note
Cisco IPICS disconnects all active dial-in and dial-out calls when you restart the dial engine.
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
To restart the dial engine, enter the following command:
[root]# service ippe_dial_engine restart
Cisco IPICS displays the message [OK] when the dial engine processes stop and restart.
Step 3
If you cannot restart the dial engine, perform the following steps:
a.
Check that the policy engine is licensed by navigating to the Administration > License Management > Summary tab.
b.
Check the status of your license in the Policy Engine Base License field.
The status displays as Licensed or Not Licensed.
c.
Perform one of the following actions based on the output that displays:
•
If the Policy Engine Base License field shows a status of Not Licensed, you are not licensed for the policy engine. To purchase a license that includes policy engine functionality, refer to the "Obtaining Your License File" section in the "Installing Cisco IPICS" chapter in the Cisco IPICS Server Installation and Upgrade Guide, Release 2.1(1).
•
If the Policy Engine Base License field shows a status of Licensed, contact your Cisco technical support representative for further assistance.
Manually Starting the Dial Engine
If the dial engine has stopped, you can manually start it by using CLI commands by performing the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
To start the dial engine, enter the following command:
[root]# service ippe_dial_engine start
If the dial engine processes starts successfully, Cisco IPICS displays the message [OK].
Step 3
If you cannot start the dial engine, perform the following steps:
a.
Check that the policy engine is licensed by navigating to the Administration > License Management > Summary tab.
b.
Check the status of your license in the Policy Engine Base License field.
The status displays as Licensed or Not Licensed.
c.
Perform one of the following actions based on the output that displays:
•
If the Policy Engine Base License field shows a status of Not Licensed, you are not licensed for the policy engine. To purchase a license and obtain dial engine and policy engine functionality, refer to the "Obtaining Your License File" "Installing Cisco IPICS" chapter of the Cisco IPICS Server Installation and Upgrade Guide, Release 2.1(1).
•
If the Policy Engine Base License field shows a status of Licensed, contact your Cisco technical support representative for further assistance.
Performing CSA Procedures
CSA provides threat protection for server and desktop computing systems. It also prevents users from performing unauthorized actions on the server. There may be times where stopping CSA is necessary to perform system-level functions, to debug an issue, or to edit protected system files.
This section describes how to perform certain CSA procedures and includes the following topics:
•
Viewing CSA Log Messages
•
Manually Stopping CSA
•
Manually Starting CSA
Viewing CSA Log Messages
If CSA denies a particular action, such as when a user or process attempts to modify or delete a protected file, the process generates a message similar to the following example:
Oct 15 04:02:02 [hostname] CiscoSecurityAgent[3480]: Event: The
process '/bin/cp' (as user root(0) group root(0)) attempted to access
'/var/cache/man/whatis'. The attempted access was an open. The
operation was denied.
You can view the CSA actions in the Security Event Log. To view the Security Event Log, perform the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
Navigate to the /var/log directory on the Cisco IPICS server by entering the following command:
[root]# cd /var/log
Step 3
To list the files that start with the csalog name in the directory, enter the following command:
[root]# ls -l csalog*
All files that begin with csalog display.
Note
The Security Event Log file is named csalog. If the csalog file has reached its maximum size, Cisco IPICS creates a new file called csalog.0, copies the information in the csalog file to the csalog.0 file, and removes the data in the csalog file. If the csalog file again reaches its maximum size, Cisco IPICS renames the csalog.0 file to csalog.1, copies the information in the csalog file to the csalog.0 file, and removes the data in the csalog file.
Step 4
To view the log file, enter the following command:
[root]# cat csalog.x
where:
x is the numeric extension of the file (if applicable).
A text viewer window displays the contents of the Security Event Log. This log provides a detailed list of security events that CSA has detected and the action, if any, that CSA performed as a result of those events.
Manually Stopping CSA
You can stop CSA by issuing a command in a terminal console session. To stop CSA from a terminal console session, perform the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
To stop CSA, enter the following command:
[root]# service ciscosec stop
Cisco IPICS displays the message [OK] after CSA stops.
Manually Starting CSA
If you installed CSA with the Cisco IPICS server software, CSA starts automatically when the Cisco IPICS server boots up. If you stop CSA or if CSA stops on its own for any reason, you can restart CSA in the CSA utility or by entering CLI commands.
To start the CSA process from a terminal console session, perform the following procedure:
Procedure
Step 1
Log in to the Cisco IPICS server by using the root user ID.
Step 2
To start CSA, enter the following command:
[root]# service ciscosec start
Cisco IPICS displays the message [OK] after CSA starts.