Table Of Contents
Tracing Application Data Paths
Setting Up Path Trace
Setting Up Path Trace on Networks With NAMs
Setting Up Path Trace on Networks Without NAMs
Configuring Routers for Medianet Performance Monitor and Mediatrace
Configuring Routers for Mediatrace and WSMA
Troubleshooting with Path Trace
Using the Path Trace Tables
Running Path Trace from Selected RTP Flows
Running Path Trace from Selected TCP Flows
Launching an Ad Hoc Path Trace From Endpoints
Troubleshooting Worst RTP Endpoints Using Dashlets
Comparing Flow Data From Multiple Sources
Tracing Application Data Paths
Prime Infrastructure supports tracing of RTP and TCP application traffic paths across endpoints and sites
Tracing data paths depends on Cisco Medianet and Web Services Management Agent (WSMA). Both are built-in features of Cisco IOS and Catalyst OS software images. They help isolate and troubleshoot problems with RTP and TCP data streams. Prime Infrastructure supports all versions of Medianet and WSMA and makes it easy to enable them on any router.
Where Cisco Network Analysis Module (NAM) traffic monitoring data is not available, Prime Infrastructure supports RTP path tracing using Medianet Performance Monitor and IOS NetFlow.
When properly configured, Path Trace can be your most valuable tool in troubleshooting RTP and TCP application problems.
This chapter contains the following sections:
•
Setting Up Path Trace
•Troubleshooting with Path Trace
Setting Up Path Trace
You cannot use Prime Infrastructure's Path Trace feature until you complete some prerequisite setup tasks. The prerequisite tasks vary according to whether you are monitoring router traffic using Cisco NAMs or not. The following sections explain both sets of tasks:
•Setting Up Path Trace on Networks With NAMs
•Setting Up Path Trace on Networks Without NAMs
Setting Up Path Trace on Networks With NAMs
If your network uses NAMs to monitor network traffic, you must complete the following tasks to enable path tracing for both RTP and TCP traffic:
1. Add NAMs to the system. You can do this either automatically, using Discovery, or manually, using bulk import or Device Work Center (see Discovering the Network).
2. Enable NAM Data collection. You can do this by choosing Administration > Data Sources > NAM Data Collector and then enabling data collection on each NAM (see Enabling NAM Data Collection).
3. Create a Site structure for your organization and use Device Work Center to assign your principal routers to the appropriate Sites. You can do this by choosing Design > Site Map Design and adding one or more Campuses (see Designing Sites).
4. Associate your Sites with Authorized Data Sources. You can do this by choosing Administration > System Settings > Data Deduplication, and assigning authoritative data sources for Voice/Video (for RTP data) and Application Response Time (for TCP data). For detailed steps, see Controlling Background Data Collection Tasks.
5. Associate your Sites with Endpoint subnets. You can do this by choosing Design > Endpoint-Site Association and then associating subnets with your Sites (see Associating Endpoints With Sites). If you fail to do this, then by default the data collected by NAMs for these endpoints will have their sites set to "Unassigned".
6. Configure your routers for Mediatrace and WSMA (see Configuring Routers for Mediatrace and WSMA).
Setting Up Path Trace on Networks Without NAMs
If your network is not equipped with Cisco NAMs, you can use Medianet Performance Monitor and NetFlow to enable path tracing of RTP flows. You can use this option only on the Cisco device platforms with at least the software images shown in Table 18-1.
.
Table 18-1 Routers Supporting Medianet Performance Monitor and Mediatrace
Platform
|
Minimum IOS Version
|
Cisco 2900 Series Integrated Services Routers
|
15.1(3)T
|
Cisco 3900 Series Integrated Services Routers
|
15.1(3)T
|
Cisco ASR 1000 Series Aggregation Services Routers
|
Cisco IOS XE Software Release 3.5 or later
|
You will also need to enable TCP path tracing on other routers using Mediatrace and WSMA.
To enable the Path Trace feature in a non-NAM environment:
1. Create a Site structure for your organization and use Device Work Center to assign your principal routers to the appropriate Sites. You can do this by choosing Design > Site Map Design and adding one or more Campuses (see Designing Sites).
2. Associate your Sites with Authorized Data Sources. You can do this by choosing Administration > System Settings > Data Deduplication, and assigning authoritative data sources for Voice/Video (for RTP data) and Application Response Time (for TCP data). For detailed steps, see Controlling Background Data Collection Tasks.
3. Associate your Sites with Endpoint subnets. You can do this by choosing Design > Endpoint-Site Association and then associating subnets with your Sites (see Associating Endpoints With Sites). If you fail to do this, then by default the data collected for these endpoints will have their sites set to "Unassigned".
4. Configure your compatible routers for Medianet Performance Monitor (see Configuring Routers for Medianet Performance Monitor and Mediatrace).
5. Configure your routers for Mediatrace and WSMA (see Configuring Routers for Mediatrace and WSMA).
Configuring Routers for Medianet Performance Monitor and Mediatrace
Prime Infrastructure supplies an out-of-the-box template that configures routers so that Medianet can be used to trace RTP paths in the absence of NAM data. This template configures compatible routers to use Medianet Performance Monitor and Cisco IOS NetFlow to export RTP flow metrics to the Prime Infrastructure server.
Note Before you begin this procedure, make sure you have completed all of the tasks in Setting Up Path Trace on Networks Without NAMs.
Step 1 Choose Design > Configuration Templates > My Templates > OOTB > Medianet-PerfMon.
Step 2 In the Template Basic section, enter a name and a description in the appropriate fields.
Step 3 In the Validation Criteria section. select one or more of the Medianet PerfMon-compatible device types listed in Table 18-1.
Step 4 In the Validation Criteria section, enter the OS Version for each of the selected device types. This must be at least the minimum IOS version shown in Table 18-1.
Step 5 In the Template Detail section, click the Form View tab and complete the fields as follows:
•Flow Exporter Name—Enter a name for the NetFlow exporter on the device types you selected. This can be any collection of characters (for example: EXPORTER-1).
•IP Address—Enter the IP address of the Prime Infrastructure server.
•Flow Exporter Port—Enter the port on which the NetFlow monitor will receive the exported data. Use the default 9991 port unless you have a special need to override it.
•Performance Monitor Name—Enter an arbitrary name for the Medianet Performance Monitor caching the data from the flow exporter (for example: MP-MONITOR-1).
•Interface—The name of the interface on the device whose NetFlow data you want to monitor (for example: ethernet 0/0).
•Flow Monitor Name—Enter an arbitrary name for the NetFlow monitor caching the data from the flow exporter (for example: FLOW-MONITOR-1).
Step 6 Click Save as New Template. After you save the template, deploy it to your routers using the procedures in Deploying Templates.
Configuring Routers for Mediatrace and WSMA
Prime Infrastructure supplies an out-of-the-box template that configures routers so that Medianet and WSMA can be run on them. It enables:
•Mediatrace Responder
•WSMA
•An HTTP server with local authentication policy.
You will want to apply this configuration to every router you want to use when tracing application paths. When applying this configuration, the HTTP user should have the highest privilege level (privilege =15) in order to run Mediatrace commands.
Note Before you begin this procedure, make sure you have completed all of the tasks in either Setting Up Path Trace on Networks With NAMs or Setting Up Path Trace on Networks Without NAMs.
Step 1 Choose Design > Configuration Templates > My Templates > OOTB > Mediatrace-WSMA-Configuration.
Step 2 In the Template Basic section, enter a name and a description in the appropriate fields.
Step 3 In the Validation Criteria section, leave "Routers" as the Device Type.
Step 4 In the Validation Criteria section, enter the OS Version.
Step 5 Click Save as New Template. After you save the template, deploy it to your routers using the procedures in Deploying Templates.
Troubleshooting with Path Trace
Path Trace is intended primarily as a troubleshooting tool. When data collection is configured properly for this tool (see Setting Up Path Trace), it shows a table listing all currently active RTP streams or TCP sessions. Using these Path Trace tables and their associated options, you can:
•Quickly identify and select RTP or TCP flows with problems (see Using the Path Trace Tables).
•Troubleshoot problems with RTP flows (see Running Path Trace from Selected RTP Flows).
•Troubleshoot problems with TCP flows (see Running Path Trace from Selected TCP Flows).
•Troubleshoot problems with RTP or TCP flows between any two arbitrary endpoints (see Launching an Ad Hoc Path Trace From Endpoints).
•Troubleshoot problems with RTP flows starting from the RTP Conversations dashlet (see Troubleshooting Worst RTP Endpoints Using Dashlets).
•Identify and compare flow performance indicators and data sources (see Comparing Flow Data From Multiple Sources).
Using the Path Trace Tables
The flow information shown in the RTP Streams and TCP Sessions tables is collected and aggregated from NAM and NetFlow data generated throughout the network.
Many rows in the RTP Streams table are arranged in a tree hierarchy. This will occur whenever an RTP application flow involves more than one data stream. In these cases, all the flows between the two application endpoints are aggregated into a single row with a triangle icon,
By default, Prime Infrastructure refreshes the RTP Streams table data every 60 seconds, automatically. It refreshes TCP Sessions data once every 300 seconds (5 minutes). You can also click either table's Refresh button at any time. You can turn off automatic refresh by unselecting Enable auto refresh.
To use the Path Trace tables:
Step 1 Choose Operate > Path Trace.
Step 2 In Application , choose RTP or TCP. The page shows the corresponding table: RTP Streams or TCP Sessions .
Step 3 Find the flow you want to troubleshoot:
•To review all flows with a particular type of issue: Click on the appropriate column heading to sort on that column.
For example, if you are monitoring RTP performance across the network and want to see the streams with the worst jitter or packet loss, click on the Jitter or Packet Loss column headings to sort the streams on these performance indicators. You can then select any of the streams for troubleshooting.
•To find a particular flow with a problem: Click the Quick Filter icon and enter a filter criterion under one or more row headings.
For example: An end user having trouble accessing an application may report to you his IP Address and the name of that application. You can do a quick filter on the TCP table for either the Client IP Address or Application ID and then select that session for troubleshooting.
•To spot issues in RTP subflows: Click the triangle icon next to any aggregated RTP flow.
For example: An RTP voice/video flow between any two endpoints will appear in the RTP Streams table as a single flow with a triangle icon. Clicking the icon will show you the four subflows: an incoming and outgoing video subflow, and an incoming and outgoing voice subflow.
Step 4 To troubleshoot the flow, see the related topics:
•Running Path Trace from Selected RTP Flows
•Running Path Trace from Selected TCP Flows
Running Path Trace from Selected RTP Flows
For each RTP flow, the RTP Streams table has a column showing that flow's:
•Type (e.g., voice or video)
•Source IP Address, Site, and User ID
•Destination IP Address Site, and User ID
•Jitter (in milliseconds)
•Packet Loss (in percent)
•Mean Opinion Score
•Traffic Volume
•Stream Start Time
To troubleshoot RTP flows using Path Trace:
Step 1 Choose Operate > Path Trace. In Application, choose RTP. Then find the flow you want using the steps in Using the Path Trace Tables.
Step 2 Select the flow and click Trace Service Path. Prime Infrastructure displays the RTP Stream Details page for the selected flow, with all the routers in the flow's path in the Troubleshooting Status table, in order of their distance from the flow's source endpoint. Routers that are Medianet-capable are indicated by a filmstrip icon.
Step 3 To run Mediatrace or Traceroute from a router in the flow's path, click on the Start Mediatrace or Start Traceroute link next to that router in the table.
Mediatrace can take a minute or more to run, depending on traffic, congestion and the total number of hops between the flow endpoints.
While running Mediatrace or Traceroute, click the Logs tab to see useful information, including:
•The progress of the operation.
•Errors encountered during the operation, including router response timeouts and other steps that did not complete.
•Where non-Medianet-capable routers where encountered and how they were processed.
•Medianet-capable routers on which Medianet is not configured.
Step 4 When the operation is complete, the Troubleshooting tab displays a topology map of the all the devices between the flow's two endpoints. Device icons in the map will be badged as follows:
•Alarm Severity: Indicates the most severe alarm currently recorded for the device.
•Flag: Indicates the device on which the Mediatrace or Traceroute was initiated.
•Filmstrip: Indicates the device is Medianet-capable.
•Minus sign on red background: Indicates the device is Medianet-capable but not configured as a Medianet responder. RTP/TCP performance statistics will not be available for the device. To remedy this situation, you must configure the device as a Medianet responder as explained in Configuring Routers for Mediatrace and WSMA.
•Minus sign: Indicates the device is unmanaged.
Step 5 To see key performance metrics for all Medianet-capable devices in the RTP flow's path, click the Medianet Path View tab. Click the subtabs in the Medianet Path View panel to see the performance metrics in numerical and graphic form.
Note The Medianet Path View tab is available only when you are able to start a Mediatrace operation from the Troubleshooting Status table. If you can only trigger Traceroute operations, it will not be shown.
Step 6 Use the appropriate links in the Troubleshooting Status table to launch a Mediatrace or Traceroute operation on a different router, restart a Mediatrace or Traceroute operation that is completed, or stop one in progress.
Running Path Trace from Selected TCP Flows
For each active TCP flow, the TCP Sessions table has a column showing that flow's:
•Client IP Address, Site, and User ID
•Server IP Address, and Site
•Application ID
•Average Network Time for the Client, WAN, and Server (in milliseconds)
•Maximum and Average Transition Time (in milliseconds)
•Traffic Volume on both the Client and Server (in bytes per second)
To troubleshoot TCP flows using Path Trace:
Step 1 Choose Operate > Path Trace. In Application, choose TCP. Then find the flow you want using the steps in Using the Path Trace Tables.
Step 2 Select the flow and click Start Path Trace. Prime Infrastructure displays the TCP Stream details page for the selected flow.
Step 3 To troubleshoot any listed flow: Select the flow and click Trace Service Path. Prime Infrastructure displays the TCP Stream Details page for the selected flow, with all the routers in the flow's path in the Troubleshooting Status table, in order of their distance from the flow's source endpoint. Routers with a "filmstrip" icon next to them are Medianet-capable.
Step 4 To run Mediatrace or Traceroute from a router in the flow's path, click on the Start Mediatrace or Start Traceroute link next to that router in the table.
Mediatrace can take a minute or more to run, depending on traffic, congestion and the total number of hops between the flow endpoints.
While running Mediatrace or Traceroute, click the Logs tab to see useful information, including:
•The progress of the operation.
•Errors encountered during the operation, including router response timeouts and other steps that did not complete.
•not complete.
•Where non-Medianet-capable routers where encountered and how they were processed.
•Medianet-capable routers on which Medianet is not configured.
Step 5 When the operation is complete, the Troubleshooting tab displays a topology map of the all the devices between the flow's two endpoints. Device icons in the map will be badged as follows:
•Alarm Severity: Indicates the most severe alarm currently recorded for the device.
•Flag: Indicates the device on which the Mediatrace or Traceroute was initiated.
•Filmstrip: Indicates the device is Medianet-capable.
•Minus sign on red background: Indicates the device is Medianet-capable but not configured as a Medianet responder. RTP/TCP performance statistics will not be available for the device. To remedy this situation, you must configure the device as a Medianet responder as explained in Configuring Routers for Mediatrace and WSMA.
•Minus sign: Indicates the device is unmanaged.
Step 6 To see key performance metrics for all Medianet-capable devices in the TCP flow's path, click the Medianet Path View tab. Click the subtabs in the Medianet Path View panel to see the performance metrics in numerical and graphic form.
Note The Medianet Path View tab is available only when you are able to start a Mediatrace operation from the Troubleshooting Status table. If you can only trigger Traceroute operations, it will not be shown.
Step 7 Use the appropriate links in the Troubleshooting Status table to launch a Mediatrace or Traceroute operation on a different router, restart a Mediatrace or Traceroute operation that is completed, or stop one in progress.
Launching an Ad Hoc Path Trace From Endpoints
You can quickly launch a Path Trace against all RTP or TCP flows between any two endpoints in the network. This can include either specific flows running between any two endpoints on the same or different sites, or between a pair of routers on two different sites.
This is handy if your network lacks NAM monitoring, or when you are in a hurry and you know at least the IP addresses of the two endpoints of the RTP or TCP flow. You must still navigate to and start the path trace from the appropriate RTP or TCP Path Trace table.
Step 1 Choose Operate > Path Trace. In Application, choose RTP or TCP.
Step 2 Click Specify Session for Path Trace.
Step 3 Specify the required flow endpoint information, as follows:
•For an RTP flow:
–Select the Source Site.
–Enter the Source Endpoint IP Address.
–Enter the Destination EndPoint IP Address.
•For a TCP flow:
–Select the Client Site.
–Enter the Client Endpoint IP Address.
–Enter Server Endpoint IP Address.
Step 4 Provide any additional endpoint information you have, as follows:
•For an RTP flow: Select or enter the Source Endpoint Port and Destination Endpoint Port.
•For a TCP flow: Select or enter the Server Endpoint Port,
Step 5 Click Trace Service Path (for an RTP flow) or OK (for a TCP flow). Prime Infrastructure displays the RTP or TCP Stream Details page for the specified flow, with all the routers in the flow's path in the Troubleshooting Status table, in order of their distance from the flow's source or client endpoint. Routers with a "filmstrip" icon next to them are Medianet-capable.
Step 6 To run Mediatrace or Traceroute from a router in the flow's path, click on the Start Mediatrace or Start Traceroute link next to that router in the table.
Mediatrace can take a minute or more to run, depending on traffic, congestion and the total number of hops between the flow endpoints.
While running Mediatrace or Traceroute, click the Logs tab to see useful information, including:
•The progress of the operation.
•Errors encountered during the operation, including router response timeouts and other steps that did not complete.
•Where and how non-Medianet-capable routers where encountered and processed.
•Medianet-capable routers on which Medianet is not configured.
Step 7 When the operation is complete, the Troubleshooting tab displays a topology map of the all the devices between the flow's two endpoints. Device icons in the map will be badged as follows:
•Alarm Severity: Indicates the most severe alarm currently recorded for the device.
•Flag: Indicates the device on which the Mediatrace or Traceroute was initiated.
•Filmstrip: Indicates the device is Medianet-capable.
•Minus sign on red background: Indicates the device is Medianet-capable but not configured as a Medianet responder. RTP/TCP performance statistics will not be available for the device. To remedy this situation, you must configure the device as a Medianet responder as explained in Configuring Routers for Mediatrace and WSMA.
•Minus sign: Indicates the device is unmanaged.
Step 8 To see key performance metrics for all Medianet-capable devices in the flow's path, click the Medianet Path View tab. Click the subtabs in the Medianet Path View panel to see the performance metrics in numerical and graphic form.
Note The Medianet Path View tab is available only when you are able to start a Mediatrace operation from the Troubleshooting Status table. If you can only trigger Traceroute operations, it will not be shown.
Step 9 Use the appropriate links in the Troubleshooting Status table to launch a Mediatrace or Traceroute operation on a different router, restart a Mediatrace or Traceroute operation that is completed, or stop one in progress.
Troubleshooting Worst RTP Endpoints Using Dashlets
You can quickly launch a Path Trace against the poorest performing RTP flows in your network using the Worst N RTP End Point Pairs. and RTP Conversation dashlets. This works only for RTP flows.
The RTP Conversations dashlet shows the complete history for a source endpoint, including flows that are no longer active. You will want to select only the most recent flows. If you launch Path Trace on such an inactive flow, you will receive an error message advising you of this fact.
Step 1 Choose Operate > Detail Dashboards > End User Experience.
Step 2 In the Worst N RTP End Point Pairs dashlet, note the Source Address for your worst performing RTP flows.
Step 3 In the RTP Conversations dashlet on the same page, find the most recent conversation for the same Source Address.
Step 4 Select that conversation in the RTP Conversations dashlet, then click Troubleshoot > Trace Service path. Prime Infrastructure displays the RTP Stream Details page for the selected flow, with all the routers in the flow's path in the Troubleshooting Status table, in order of their distance from the flow's source endpoint. Routers that are Medianet-capable are indicated by a filmstrip icon.
Step 5 To run Mediatrace or Traceroute from a router in the flow's path, click on the Start Mediatrace or Start Traceroute link next to that router in the table.
Mediatrace can take a minute or more to run, depending on traffic, congestion and the total number of hops between the flow endpoints.
While running Mediatrace or Traceroute, click the Logs tab to see useful information, including:
•The progress of the operation.
•Errors encountered during the operation, including router response timeouts and other steps that did not complete.
•Where and how non-Medianet-capable routers where encountered and processed.
•Medianet-capable routers on which Medianet is not configured.
Step 6 When the operation is complete, the Troubleshooting tab displays a topology map of the all the devices between the flow's two endpoints. Device icons in the map will be badged as follows:
•Flag: Indicates the device on which the Mediatrace or Traceroute was initiated.
•Filmstrip: Indicates the device is Medianet-capable.
•Minus sign: Indicates the device is unmanaged.
Step 7 To see key performance metrics for all Medianet-capable devices in the flow's path, click the Medianet Path View tab. Click the subtabs in the Medianet Path View panel to see the performance metrics in numerical and graphic form.
Note The Medianet Path View tab is available only when you are able to start a Mediatrace operation from the Troubleshooting Status table. If you can only trigger Traceroute operations, it will not be shown.
Step 8 Use the appropriate links in the Troubleshooting Status table to launch a Mediatrace or Traceroute operation on a different router, restart a Mediatrace or Traceroute operation that is completed, or stop one in progress.
Comparing Flow Data From Multiple Sources
When interpreting Path Trace performance data, you may find it helpful to:
•Identify the NAM, NetFlow, and other sources reporting this performance data.
•If you have multiple NAM or NetFlow data sources: Compare how those sources are reporting key performance indicators for a particular flow.
To do this, follow these steps.
Step 1 Choose Operate > Path Trace. In Application, choose RTP or TCP. Then find the flow you want using the steps in Using the Path Trace Tables
Step 2 Select the flow and then click Analyze Path (for an RTP flow) or Analyze on Multiple Data Sources (for a TCP flow). Prime Infrastructure displays the Multiple DAta Source KPI Analysis page. It provides a table presenting key performance indicators appropriate for the selected flow, and the data source for each such set of indicators.
Step 3 When you are finished, click OK.
Feedback
