Objectives
Objective of this white paper is to validate existing CFM (Connectivity Fault Management for Ethernet) Service diagnostics Scripts for Cisco® Integrated Services Routers (ISRs) and provide step-by-step instructions for troubleshooting Ethernet problems. This testing was implemented using the EEM (Embedded Event Manager) framework, which comes with Cisco IOS® Software IPBase images on the Cisco ISRs. This document also contains testbed topology, configuration, snapshots of script output recorded during testing, syslog and email notifications, and logging information.
Introduction and Summary:
Cisco CFM Service Diagnostics offers a collection of powerful onboard diagnostic tools to identify commonly encountered network problems, provide real-time alerts, automatically collect relevant information, and conduct root cause analysis. These Connectivity Fault Management Diagnostics Scripts run over the EEM (Embedded Event Manager) Infrastructure.
EEM brings the management capability inside the Cisco devices. Using EEM you can monitor events and take informational and corrective action when the monitored events occur or when a threshold is reached. The EEM framework comprises Event Detectors and Events Policy. Event Detectors detect events and notify the EEM Server. The EEM policies are configured using the Cisco IOS Software command-line interface (CLI) or TCL script; then recovery is implemented on the basis of the current state of the system and the actions specified in the policy for the given event. EEM provides very sophisticated and distributed control on automated configuration, event detection, remote monitoring, recovery, and device availability. Components of the EEM Architecture are shown in Figure 1.
Figure 1. EEM Architecture
With Ethernet moving toward the Service Provider Network, the requirement of a connectivity fault management tool becomes a high-demand requirement. High Industry interest in Ethernet connectivity fault management tools forced the IEEE to develop a standard 802.1ag (CFM). This standard specifies protocols, procedures, and managed objects to support transport fault management (detection and isolation of connectivity faults). CFM frames are distinguishable by Ether-Type 89-02 (and MAC Address for multicast messages). CFM employs regular Ethernet frames that travel in-band with the customer traffic. Devices that cannot interpret CFM Messages forward them as normal data frames. Connectivity Fault Management is end-to-end Fault Management, as shown in Figure 2.
Figure 2. CFM Positioning
This standard divides a network into administrative domains in the form of hierarchy levels. This division helps define the relationships among all entities from a maintenance perspective, to allow each entity to monitor the layers under its responsibility and easily localize problems. Basic components of CFM are shown in Figure 3.
Figure 3. CFM MEP/MIP (Maintenance Association End Points/Maintenance Association Intermediate Point)
CFM functions include Connectivity Check (for fault detection), Loopback (for fault verification), and Traceroute (for fault isolation).
Service Diagnostics tools are TCL scripts that use the capabilities of the CFM and EEM infrastructure to develop a framework that is ideal for automated fault diagnostics, analysis, and recovery. These diagnostic tools can help you increase network uptime, reduce time to repair, and improve service levels. These policies in the form of TCL scripts isolate the complex troubleshooting and manual fault diagnostics steps. TCL scripts also include an onboard menu system (EMM) to support interactive installation and deployment. Service Diagnostics Scripts cover all functions of CFM (i.e., Connectivity Check, Loopback, and Traceroute) and use EEM capabilities to use events generated by cctimeout to confirm connectivity breaks and verify the status of the network by performing a Loopback check and traceroute and notifying the service manager the status of the network through syslog/email/snmp notifications based on the configuration. The interaction between Service diagnostics with EEM and CFM is shown in Figure 4.
Figure 4. Service Diagnostics Interaction Feature
CFM and EEM Interworking
Ethernet is rapidly gaining acceptance in carrier networks and replacing traditional technologies. This rapid growth of Ethernet is also supported by IEEE 802.1ag (CFM) standard specifications. Because of its end-to-end reachability and hierarchical structures, CFM is very popular in Carrier Ethernet, and service managers like to use it for fault detection and isolation. However, Configuring CFM domains, MIPs, MEPs, and hierarchical levels, and looking at various databases and error status require in-depth understanding and knowledge of this new technology. Network Managers also need to deploy an OAM management infrastructure to monitor the network status. Most of event monitoring and management is performed by devices external to the network that add extra costs of hardware and networking.
Prior to CFM Diagnostics scripts, the following diagnostics scripts are already deployed and being used by the user community:
• Border Gateway Protocol
• Open Shortest Path First
• Quality-of-Service Scripts
For more details, please visit http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps9424/white_paper_cisco_ios_service_design_bgp_osp_qos.html.
CFM diagnostic scripts also use EEM infrastructure. EEM Infrastructure is embedded in Cisco IOS Software and widely used in distributed and customized environments for event detection and automated recovery. CFM Fault detection capabilities combined with EEM event detections and policy infrastructure, and Cisco developed an innovative solution of automated service diagnostics scripts that will simplify service managers' efforts. These automated scripts are deployed in Cisco Integrated Services Routers; they can be executed on demand or scheduled for any specific time. These scripts. when executed, collect all network status information and necessary logs and if necessary bounce the interfaces and send the status through email notification. These scripts also use SNMP notification and syslog messages and notify users accordingly. Figure 5 shows the CFM Service Diagnostics scripts currently employed.
Figure 5. Service Diagnostics: Carrier Ethernet Scenarios
Currently we have limited scripts ready for use on Cisco ISRs, and they can be found at CBeyond. You can create your own scripts as per your requirement by following EEM user guidelines and using this innovative technology on Cisco routers. Following is a list of scripts available at Cbeyond.
• cfm_cctimeout
• cfm_autotrace
• cfm_ondemand
• cfm_shut_noshut
• cfm_undeploy
• cfm_deploy
• cfmod
• cfm_display
Service Diagnostics scripts (policies) must be deployed and registered before use. You can deploy them manually by copying in the router memory (Please see http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps9424/whitepaper_c11-566741.html for more details) or you can deploy them using EMM (Embedded Menu Manager) after copying the mdf file in Cisco router memory (please see http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps9424/whitepaper_c11-566741.html for more details).
Supported Platform
Supported platforms include the Cisco 3945, 3945E, 3925, 3925E, 2951, 2921, 2911, 2901, 1941, 1921, 1905, 880, and 890 Integrated Services Routers and the Cisco 3800, 2800, and 1800 Series Integrated Services Routers.
Tested Cisco IOS Software Images
• Cisco IOS Software Release 15.1(1)T
• EEM Version 3.10
Testbed Topology
Logical Topology
In this logical topology (Figure 6), CPE1 and CPE2 are connected to provider network (PE1 and PE2).
Figure 6. Logical Topology
Physical Topology
Figure 7 shows the physical topology with the concepts of Maintenance Domain (MD), Maintenance Association (MA), Maintenance Association Endpoint (MEP), and Maintenance Association Intermediate Point (MIP). These components play a vital role in Connectivity Fault Management.
Figure 7. Physical Topology
MEP defines the boundaries of Maintenance Domain, and it is associated with the association of Maintenance with MEPID. MIPs are configured at intermediate point, and they support the discovery of paths among MEPs and location of faults along those paths. MIPs can add, check, and respond to received CFM PDUs. In the topology shown in Figure 8, MEPs are depicted with an arrow and MIPs are shown by small circles in Figure 7. Please note that Customer Domain is from end to end (CPE1 to CPE2) and configured with the highest level 7, whereas the Provider domain is from PE1 to PE2 configured with level 5. Connectiveity Check Messages are catalogued in Connectivity Check Message Database and errors are catalogued in Error DB. MEP connectivity check database (CCDB) contains only Active entries, and MIP CCDB contains both active and archived entries. Please note that MAs are end to end and defined by a set of Maintenance Endpoints (associated with different MEPIDs at the ends). Maintenance Total of 4 maintenance associations (MAs) are configured in this testbed as shown in Figure 8.
Figure 8. Maintenance Association Configurations
Configurations
Useful Show Commands
The following show command is used with Shut/No-Shut Script
agam-2821#show ethern cfm mpd domain CUST1 service SID_1
* = Can Ping/Traceroute to MEP
--------------------------------------------------------------------------------
MPID Domain Name MacAddress Version
Lvl Domain ID Ingress
Expd MA Name Type Id SrvcInst
EVC Name Age
--------------------------------------------------------------------------------
4 * CUST1 0017.95e4.4c71 IEEE-CFM
7 CUST1 Gi0/1.1000
- SID_1 Vlan 700 N/A
N/A 3s
4000* CUST1 0017.95e4.4c71 IEEE-CFM
7 CUST1 Gi0/1.1000
EXPD SID_1 Vlan 700 N/A
N/A 134s
400 * CUST1 0017.95e4.4c71 IEEE-CFM
7 CUST1 Gi0/1.1000
EXPD SID_1 Vlan 700 N/A
N/A 46s
Total Remote MEPs: 3
agam-3845#show event manager policy available
No. Type Time Created Name
1 system Thu Feb 7 01:28:15 2036 ap_perf_test_base_cpu.tcl
2 user Fri Apr 5 15:25:20 1940 cfm_autotrace.tcl
3 user Fri Apr 5 15:25:22 1940 cfm_cctimeout.tcl
4 user Fri Apr 5 15:25:22 1940 cfm_ondemand.tcl
5 user Mon Apr 8 14:34:22 1940 cfm_router_shut.tcl
6 user Fri Apr 5 15:25:24 1940 cfmod.tcl
7 system Thu Feb 7 01:28:15 2036 cl_show_eem_tech.tcl
8 user Fri Apr 5 15:25:24 1940 collectEmailParameters.tcl
9 system Thu Feb 7 01:28:15 2036 no_perf_test_init.tcl
10 user Fri Apr 5 15:25:18 1940 sdiag_router_cfm.tcl
11 user Fri Apr 5 15:25:20 1940 sdiag_router_cfm_display.tcl
12 user Fri Apr 5 15:25:20 1940 sdiag_router_cfm_undeploy.tcl
13 system Thu Feb 7 01:28:15 2036 sl_intf_down.tcl
14 system Thu Feb 7 01:28:15 2036 tm_cli_cmd.tcl
15 system Thu Feb 7 01:28:15 2036 tm_crash_reporter.tcl
16 system Thu Feb 7 01:28:15 2036 tm_fsys_usage.tcl
agam-3845#show event manager version
Embedded Event Manager Version 3.10
Component Versions:
eem: (v310_throttle)4.1.18
eem-gold: (v310_throttle)1.0.7
eem-call-home: (v310_throttle)1.0.6
Event Detectors:
Name Version Node Type
application 01.00 node0/0 RP
syslog 01.00 node0/0 RP
track 01.00 node0/0 RP
resource 01.00 node0/0 RP
routing 02.00 node0/0 RP
cli 01.00 node0/0 RP
counter 01.00 node0/0 RP
interface 01.00 node0/0 RP
ioswdsysmon 01.00 node0/0 RP
none 01.00 node0/0 RP
oir 01.00 node0/0 RP
snmp 01.00 node0/0 RP
snmp-notification 01.00 node0/0 RP
timer 01.00 node0/0 RP
ipsla 01.00 node0/0 RP
test 01.00 node0/0 RP
config 01.00 node0/0 RP
env 01.00 node0/0 RP
gold 01.00 node0/0 RP
nf 01.00 node0/0 RP
agam-3845#show event manager detector all
No. Name Version Node Type
1 application 01.00 node0/0 RP
2 syslog 01.00 node0/0 RP
3 track 01.00 node0/0 RP
4 resource 01.00 node0/0 RP
5 routing 02.00 node0/0 RP
6 cli 01.00 node0/0 RP
7 counter 01.00 node0/0 RP
8 interface 01.00 node0/0 RP
9 ioswdsysmon 01.00 node0/0 RP
10 none 01.00 node0/0 RP
11 oir 01.00 node0/0 RP
12 snmp 01.00 node0/0 RP
13 snmp-notification 01.00 node0/0 RP
14 timer 01.00 node0/0 RP
15 ipsla 01.00 node0/0 RP
16 test 01.00 node0/0 RP
17 config 01.00 node0/0 RP
18 env 01.00 node0/0 RP
19 gold 01.00 node0/0 RP
20 nf 01.00 node0/0 RP
agam-3845#
Prerequisites of CFM Diagnostics Deployment
Before deployment of Service Diagnostics Scripts, verify that:
• CFM is configured in ieee mode; i.e.,
ethernet cfm ieee
ethernet cfm global
• CFM MEP, MIP, MD, and MA are configured; verify remote connectivity through show ethernet cfm maintenance-association remote command.
• CFM Traceroute cache is enabled for CFM EEM policies (cfm_autotrace.tcl, cfm_ondemand.tcl)
• CFM alarm syslogs are enabled (ethernet cfm logging) for CFM EEM policies (cfm_cctimeout.tcl, cfm_shut.tcl, cfm_unshut.tcl)
• SNMP community/manager are configured
• EEM SNMP traps are enabled
• Domain name (ip domain-name "domainname.com") for email notification is configured; ensure that the correct email server IP or FQDN is known.
• CFM MA input file for CFM EEM policies (cfm_cctimeout.tcl, cfm_autotrace.tcl, cfm_shut.tcl) exists
CFM Service Diagnostics Deployment Considerations
Some of the known CFM SD deployment considerations follow:
• Execution time of CFM SD EEM policy is directly proportional to the number of remote MEPs and corresponding operations (e.g., ping/traceroute) performed. Ethernet ping / traceroute timeout is 5 seconds (Applicable to cfm_ondemand.tcl, cfm_cctimeout.tcl, cfm_autotrace.tcl).
• In case of concurrent syslogs triggering the same or different policies (i.e., cfm_cctimeout.tcl, cfm_shut.tcl), the execution is always done in sequence.
• Automatic Diagnostics and Shutdown Port policies (cfm_cctimeout.tcl, cfm_shut.tcl) have a common trigger (network failure event [remote mepTimeout]). This situation may cause both policies to trigger at the same time if input MA files have common entries. In this case, the order of policy deployment determines the execution order. Thus, it is recommended to always deploy first the Shutdown Port policy.
• On-demand Diagnostics policy parses out the content of CFM traceroute cache. Therefore, it is recommended to deploy and run the Autotrace policy in order to maintain the traceroute cache populated.
• Syslog and SNMP notifications are faster compared to Email. Email is sent at the end of processing.
Installing CFM SD (Service Diagnostics) Scripts in Cisco Access Router:
1. Installation Steps Using tclsh CLI parser mode
a) Download the CFM SD scripts zip file in local TFTP server.
b) Unzip the CFM SD zip file.
c) Create policy directory (i.e. svc_diag).
d) Copy policy scripts in newly created directory (i.e. svc_dir).
e) Create Library Directory (i.e. user_lib).
f) Copy library scripts in newly created library directory (i.e. lib_dir).
2. Installation Steps Using EMM
a) Download the mdf file (cfm_router.mdf) in local TFTP server.
b) Copy mdf file to Router disk system (flash).
c) Execute emm mdf flash:/cfm_router.mdf.
d) Steps follow:
agam-2821#emm mdf cfm_router.mdf
=======================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: 1
Enter ? for help
Enter a directory to store the CFM diagnostic policies in the form of a URL (excluding filename, e.g. disk0:/svc_diag
Enter value [flash:/svc_diag]: flash:/svc_diag
Enter ? for help
Enter a directory for the user library files in the form of a URL (excluding filename, e.g. disk0:/user_lib
Enter value [flash:/user_lib]: flash:/user_lib
Writing files....Done.
Press any key to continue...
===========================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: 7
Policy Directory contains following policy scripts:
1. sdiag_router_cfm.tcl
2. sdiag_router_cfm_undeploy.tcl
3. sdiag_router_cfm_display.tcl
4. cfm_autotrace.tcl
5. cfm_router_shut.tcl
6. cfm_ondemand.tcl
7. cfm_cctimeout.tcl
8. cfmod.tcl
9. collectEmailParameters.tcl
Service diagnostics Library contains following Library scripts:
1. cfm_lib.tcl
2. diag_lib.tcl
3. lib.tcl
4. lib_2.tcl
5. userlib.tcl
6. userlib_2.tcl
7. tclIndex
8. email_template_cmd
Carrier Ethernet Scenarios - Diagnostics
ON-DEMAND Diagnostics
Upon customer request to troubleshoot a given service, EEM script automatically verifies and isolates faults to remote MEPs affected by service failure. It could also search traceroute-cache. A diagnostics report is sent to NOC/Service Center (Figure 9).
Figure 9. On-Demand Diagnostics
EEM event detector: none
Script variables: CFM MD name, shortMA name, remote mep id (opt)
Working steps of ON-DEMAND Diagnostics are given as under. It is assumed that CFM DOMAIN name, MEP-ID and S_VLAN_ID is configured.
ParseCollectData
Verify CFM Domain Information
show ethernet cfm domain zzz
Inspect CFM Connectivity Check Messages Database
Examine MEP Connectivity Check Messages Database for entry logged
show ethernet cfm remote domain zzz
show ethernet cfm maintenace remote detail mpid xxx domain zzz vlan yyy
Inspect CFM ErrorDB
Look for errors logged
show ethernet cfm error domain-id zzz service www
Connectivity / Failure Verification (ping ethernet mac)
ping ethernet H.H.H domain zzz vlan xxx
ping ethernet multicast domain zzz vlan xxx
Path Discovery Isolate Failure (trace ethernet mac)
traceroute ethernet H.H.H domain zzz vlan xxx
Inspect Traceroute cache
Show ethernet cfm traceroute-cache
Generate Report
Place together the following information in report
• Number of active MEPs in CCDB with port state == UP
• Details of active MEPs in CCDB with port state <> UP
• Details of entries in Error DB
• Number of remote MEPs with verified connectivity
• Details of remote MEPs with connectivity problems
• Traceroute results for MEPs with connectivity problems
• Results of traceroute cache examination
Automatic Diagnostics
Upon a Service failure, EEM script automatically verifies and isolates faults to the remote MEPs affected. A diagnostics report is sent to NOC/Service Center (Figure 10).
Figure 10. Automatic Diagnostics
EEM event detector: CFM MEP Down (timeout) syslog
Script variables: Input file with list of shortMA names
Working steps of automatic diagnostics after even is detected are described as under.
Inspect CFM Connectivity Check Message DB
Examine MEP CCDB for entry logged against it.
show ethernet cfm maintenace remote detail mpid iii domain zzz vlan xxx
Inspect CFM ErrorDB
Look for errors logged against it.
show ethernet cfm error domain-id zzz service www
Connectivity / Failure Verification (ping ethernet mac)
Issue LBM towards it Record results.
ping ethernet H.H.H domain zzz vlan xxx
Path Discovery Isolate Failure (trace ethernet mac)
Issue LTM towards it Record results.
traceroute ethernet H.H.H domain zzz vlan xxx
Generate Report
Place together the following information:
• Details of entry in Error DB
• Details of failure verification
• Details of failure isolation
Auto-Traceroute
EEM event detector: timer expiration
Script variables: Input file with list of shortMA names
Timer value
Working steps of automatic diagnostics after even is detected are described as under.
Verify If Cache Is Enabled
Verify if traceroute cache is enabled. Also verify the size and the hold-time of traceroute cache.
show ethernet cfm traceroute-cache
Parse CFM CCM DB
Initialize these variables for the desired MAs:
• cfm Domains
• rmep mpids
• rmep MAC addresses
• vlan number
Verify with following show commands:
show ethernet cfm domain brief
show ethernet cfm domain <domain>
show ethernet cfm maintenance-points remote
show ethernet cfm maintenance-points remote domain <domain>
Path Discovery (trace ethernet mac)
Issue LTM for each of the rmeps discovered.
traceroute ethernet H.H.H domain zzz vlan xxx
Generate Report
Report. Gather the following information:
• Total number of traceroute operations performed
• rmep details
Action Scripts
Shut/No-shut Script
Shut/No-shut script is a combination of shut and no-shut script with DOWN MEPs. In point-to-point environments with NON-ELMI capable CEs, Shut/No-shut script bounces the interface, thus reducing CE blackholing of traffic when EVC or remote UNI failures occur (Figure 11).
Figure 11. Shut / No-shut operation
EEM event detector:
REMOTE_MEP_UP
REMOTE_MEP_DOWN
Script variables: Input file with list of shortMA names
Working steps of Shut/No-shut Diagnostics are given as under.
Assumptions: Single service (no multiplexing), CFM from UNI to UNI
Parse Collect Data:
Initialize CFM Domain name, S-VLAN ID, Mepid and verify with following command
show ethernet cfm maintenance-points remote
Inspect CFM CCM DB:
IF MEP UP IF state syslog
THEN verify if IF State == DOWN
Continue next step
Inspect CFM Local DB:
Examine the CFM maintenance point local DB to determine the interface that holds the MEP associated with the service under monitoring.
show ethernet cfm maintenance-points remote detail mpid <mpid> domain <zzz> vlan <xxx>
SHUT/No-Shut Port:
• Shutdown port
• Record remote mpid / shortMA name / MD name / Vlan information.
• No-shut port
Generate Report:
• Verify that interface is UP.
• Send message indicating the MA name and corresponding
• Interface that was unshut by the script.
Deploying CFM SD (Service Diagnostics) Scenarios in Cisco Access Router
Step-by-step instructions are shown to deploy CFM SD policy scripts in a Cisco Access Router.
Deployment of CFM SD scenarios using tclsh CLI parser mode
The following steps are required to deploy scenarios using tclsh CLI parser mode:
Configuring Policy and Library path:
event manager directory user policy "flash:/svc_diag"
event manager directory user library "flash:/user_lib"
Create Maintenance Association file:
agam-3845#more ma_list
SID_1
SID_2
SID_3
SID_4
Verify policy and library path is set using show run output:
event manager directory user policy "flash:/svc_diag"
event manager directory user library "flash:/user_lib"
Add email parameters:
agam-3845# tclsh flash:/svc_diag/collectEmailParameters.tcl c3845@cisco.com arshadm@cisco.com arshadm@cisco.com 64.102.124.15
Displaying policy deploying policy using tclsh:
agam-3845#tclsh flash:/svc_diag/sdiag_router_cfm.tcl
ERROR: Incorrect number of arguments
CFM CC-TIMEOUT POLICY Syntax:
Usage:tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_cctimeout.tcl <notification> <configurationHistory> <EventHistory> <CommandHistory> <prepend trigger message> <user_pol_dir> <user_lib_dir> <ma_list>
CFM ON-DEMAND POLICY syntax:
Usage:tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_ondemand.tcl <notification> <configurationHistory> <EventHistory> <CommandHistory> <prepend trigger message> <user_pol_dir> <user_lib_dir> <domain_name> <ma_name> <optional mep-id>
CFM SHUT POLICY Syntax:
Usage:tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_router_shut.tcl <notification> <configuration history> <event history> <command history> <prepend trigger message> <user_pol_dir> <user_lib_dir> <ma_list> <p2p>
CFM AUTO-TRACE POLICY Syntax:
Usage:tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_autotrace.tcl <notification> <configuration history> <event history> <commandHistory> <prepend trigger message> <user_pol_dir> <user_lib_dir> <ma_list> <timer>
For policy detailed parameters, use <policy> -help
For example: tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_router_shut.tcl -help
For policy environment variables, use <policy> -variables
For example: tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_router_shut.tcl -variables
Deploying cfm_cctimeout.tcl policy using tclsh CLI Parser:
agam-3845# tclsh flash:/svc_diag/sdiag_router_cfm.tcl cfm_cctimeout.tcl email,syslog,snmp true true true true flash:/svc_diag flash:/user_lib ma_test
WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
Policy cfm_cctimeout.tcl successfully registered
agam-3845#
Displaying parameters of deployed policies:
agam-3845#tclsh flash:/svc_diag/sdiag_router_cfm_display.tcl cfm_cctimeout.tcl
THE INPUTS GIVEN FOR CFM CC-TIMEOUT SCENARIO ARE:
Notification : EMAIL,SYSLOG,SNMP
Configuration history option : TRUE
Event history option : TRUE
Command history option : TRUE
Prepend option : TRUE
MA File name is : ma_test
Undeploying cfm_cctimeout.tcl policy:
agam-3845#tclsh flash:/svc_diag/sdiag_router_cfm_undeploy.tcl cfm_cctimeout.tcl
cfm_cctimeout.tcl is unregistered successfully
Deployment of CFM SD scenarios using emm
Step-by-step Instructions for deploying CFM SD scenario using EMM follow.
Setting global parameter using emm:
agam-3845#emm mdf flash:/cfm_router.mdf
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]:2
Enter the email address for the _email_from field
Enter value [c3845@cisco.com]:c3845@cisco.com
Enter the email address for the _email_to field
Enter value [arshadm@cisco.com]:arshadm@cisco.com
Enter the email address for the _email_cc field
Enter value [arshadm@cisco.com]:arshadm@cisco.com
Enter the IP address for the email server
Enter value [64.102.124.15]:64.102.124.15
Enter the directory where the CFM diagnostic policies are located
in the form of a URL (e.g. disk0:/svc-diag)
Enter value [flash:/svc_diag]:flash:/svc_diag
Enter the directory where the user library files are located
in the form of a URL (e.g. disk0:/user_library)
Enter value [flash:/user_lib]:flash:/user_lib
Press any key to continue...
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]:3
Deploying cfm_cctimeout.tcl policy using emm
agam-3845#emm mdf flash:/cfm_router.mdf
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: 3
================================================================================
Connectivity Fault Management Scenario Deployment
Enter ?# for item help
--------------------------------------------------------------------------------
1. Deploy CFM CC-Timeout Script
2. Deploy CFM On-Demand Script
3. Deploy Action Shut Script
4. Deploy CFM Action Auto Trace Script
5. Return to main CFM menu
Enter selection [5]: 1
Enter the path to the Maintenance Association name file (e.g. disk0:/)
Enter value [flash:]: <ENTER>
Please select the Maintenance Association name file or 'CREATE_FILE'
1. cert.pem
2. ma_list.txt
3. new_ma
4. ma_list
5. ma_test
6. CREATE_FILE
Enter choice: 5
Note: You may use an existing file or create a new one. We have mentioned procedure for creating a new file in router_shut scenario.
Select the type of notification to be sent when a problem is diagnosed
1. Email Only
2. SNMP Only
3. Syslog Only
4. Email and Syslog
5. Email and SNMP
6. Syslog and SNMP
7. Email, Syslog, and SNMP
Enter choice: 7
Do you want to log configuration history?
1. Yes
2. No
Enter choice: 1
Do you want to log event history?
1. Yes
2. No
Enter choice: 1
Do you want to log command history?
1. Yes
2. No
Enter choice: 1
Do you want to prepend trigger syslog messages to the diagnosis?
1. Yes
2. No
Enter choice: 1
WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
Policy cfm_cctimeout.tcl successfully registered
Press any key to continue...
================================================================================
Connectivity Fault Management Scenario Deployment
Enter ?# for item help
--------------------------------------------------------------------------------
1. Deploy CFM CC-Timeout Script
2. Deploy CFM On-Demand Script
3. Deploy Action Shut Script
4. Deploy CFM Action Auto Trace Script
5. Return to main CFM menu
Enter selection [5]: <ENTER>
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: <ENTER>
agam-3845#
Undeploying cfm_cctimeout.tcl script:
Use following steps to undeploy cctimeout scenario
agam-3845#emm mdf flash:/cfm_router.mdf
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: 6
Please select diagnostic scenario to remove from the running-configuration
1. CFM CC-Timeout
2. CFM On-Demand
3. CFM Action Shut
4. CFM Auto Trace
5. ALL CFM Policies
Enter choice: 1
cfm_cctimeout.tcl is unregistered successfully
1
Press any key to continue...
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: <ENTER>
agam-3845#
Deployment of On-Demand Scenario
Cfm_ondemand scenario can be deployed per ma (maintenance association) basis
Deploying cfm_ondemand.tcl policy using tclsh CLI Parser:
agam-3845#tclsh flash:/svc_diag/sdiag_router_cfm.tcl cfm_ondemand.tcl email,syslog true true true true flash:svc_diag flash:/user_lib CUST1 SID_1
WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
Policy cfm_ondemand.tcl successfully registered
agam-3845#
Undeploying cfm_ondemand.tcl policy using tclsh CLI Parser:
agam-3845#tclsh flash:/svc_diag/sdiag_router_cfm_undeploy.tcl cfm_ondemand.tcl
cfm_ondemand.tcl is unregistered successfully
Deploying cfm_ondemand.tcl policy using emm:
Following step by step instruction shows how to deploy cfm_ondemand scenario
agam-3845#emm mdf cfm_router.mdf
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: 3
===============================================================================
Connectivity Fault Management Scenario Deployment
Enter ?# for item help
--------------------------------------------------------------------------------
1. Deploy CFM CC-Timeout Script
2. Deploy CFM On-Demand Script
3. Deploy Action Shut Script
4. Deploy CFM Action Auto Trace Script
5. Return to main CFM menu
Enter selection [5]: 2
Please select a domain for the Maintenance Association of interest
1. CUST1
Enter choice: 1
Select the Maintenance Association Name for Domain CUST1
1. SID_3
2. SID_4
3. SID_1
4. SID_2
Enter choice: 1
Do you want to specify a Maintenance End Point (MEP)?
1. Yes
2. No
Enter choice: 1
Please select the Maintenance End Point for SID_3
1. 4200
Enter choice: 1
Select the type of notification to be sent when a problem is diagnosed
1. Email Only
2. SNMP Only
3. Syslog Only
4. Email and Syslog
5. Email and SNMP
6. Syslog and SNMP
7. Email, Syslog, and SNMP
Enter choice: 4
Do you want to log configuration history?
1. Yes
2. No
Enter choice: 1
Do you want to log event history?
1. Yes
2. No
Enter choice: 1
Do you want to log command history?
1. Yes
2. No
Enter choice: 1
Do you want to prepend trigger syslog messages to the diagnosis?
1. Yes
2. No
Enter choice: 1
WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
Policy cfm_ondemand.tcl successfully registered
Press any key to continue...
================================================================================
Connectivity Fault Management Scenario Deployment
Enter ?# for item help
--------------------------------------------------------------------------------
1. Deploy CFM CC-Timeout Script
2. Deploy CFM On-Demand Script
3. Deploy Action Shut Script
4. Deploy CFM Action Auto Trace Script
5. Return to main CFM menu
Enter selection [5]: 5
Undeploying cfm_ondemand.tcl script:
Use the following steps to undeploy ondemand script scenario:
agam-3845#emm mdf flash:/cfm_router.mdf
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: 6
Please select diagnostic scenario to remove from the running-configuration
1. CFM CC-Timeout
2. CFM On-Demand
3. CFM Action Shut
4. CFM Auto Trace
5. ALL CFM Policies
Enter choice: 2
cfm_ondemand.tcl is unregistered successfully
Press any key to continue...
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: <ENTER>
Deployment of autotrace Scenario
Deploying cfm_autotrace.tcl policy using tclsh CLI Parser:
agam-3845#tclsh flash:/svc_diag/ sdiag_router_cfm.tcl cfm_autotrace.tcl email/syslog true true true true flash:svc_diag flash:/user_lib ma_test 60
WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
Traceroute Cache Hold Time: 120 mins
CFM autotrace timer: 60 mins
Policy cfm_autotrace.tcl successfully registered
Undeploying cfm_autotrace.tcl policy using tclsh CLI Parser:
agam-3845# tclsh flash:/svc_diag/sdiag_router_cfm_undeploy.tcl cfm_autotrace.tcl
cfm_autotrace.tcl is unregistered successfully
Deploying cfm_autotrace.tcl action policy using emm
Please follow following steps to deploy cfm_autotrace scenario
agam-3845#emm mdf cfm_router.mdf
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: 3
================================================================================
Connectivity Fault Management Scenario Deployment
Enter ?# for item help
--------------------------------------------------------------------------------
1. Deploy CFM CC-Timeout Script
2. Deploy CFM On-Demand Script
3. Deploy Action Shut Script
4. Deploy CFM Action Auto Trace Script
5. Return to main CFM menu
Enter selection [5]: 4
Enter the path to the Maintenance Association name file (e.g. disk0:/)
Enter value [flash:]: <ENTER>
Please select the Maintenance Association name file or 'CREATE_FILE'
1. cert.pem
2. ma_list.txt
3. new_ma
4. ma_list
5. ma_test
6. CREATE_FILE
Enter choice: 5
Select the type of notification to be sent when a problem is diagnosed
1. Email Only
2. SNMP Only
3. Syslog Only
4. Email and Syslog
5. Email and SNMP
6. Syslog and SNMP
7. Email, Syslog, and SNMP
Enter choice: 4
Do you want to log configuration history?
1. Yes
2. No
Enter choice: 1
Do you want to log event history?
1. Yes
2. No
Enter choice: 1
Do you want to log command history?
1. Yes
2. No
Enter choice: 1
Do you want to prepend trigger syslog messages to the diagnosis?
1. Yes
2. No
Enter choice: 1
Enter the timer for this policy (minutes)
Enter a value in the range <60 - 80> [80]: 60
WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
Traceroute Cache Hold Time: 120 mins
CFM autotrace timer: 60 mins
Policy cfm_autotrace.tcl successfully registered
1
Press any key to continue...
================================================================================
Connectivity Fault Management Scenario Deployment
Enter ?# for item help
--------------------------------------------------------------------------------
1. Deploy CFM CC-Timeout Script
2. Deploy CFM On-Demand Script
3. Deploy Action Shut Script
4. Deploy CFM Action Auto Trace Script
5. Return to main CFM menu
Enter selection [5]: <ENTER>
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: <ENTER>
agam-3845#
Undeploying cfm_ondemand.tcl script:
Use the following steps to undeploy the ccondemand scenario:
agam-3845#emm mdf flash:/cfm_router.mdf
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: 6
Please select diagnostic scenario to remove from the running-configuration
1. CFM CC-Timeout
2. CFM On-Demand
3. CFM Action Shut
4. CFM Auto Trace
5. ALL CFM Policies
Enter choice: 4
cfm_autotrace.tcl is unregistered successfully
1
Press any key to continue...
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: <ENTER>
SHUT/NOSHUT Action Script
This is a Policy script to "bounce" corresponding Ethernet IF (sub-IF).
Deploying cfm_router_shut.tcl script using tclsh
Usage:tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_router_shut.tcl <notification> <configuration history> <event history> <command history> <prepend trigger message> <user_pol_dir> <user_lib_dir> <ma_list> <p2p>
agam-3845# tclsh flash:/svc_diag/sdiag_router_cfm.tcl cfm_router_shut.tcl email,syslog,sn mp true true true true flash:/svc_diag flash:/user_lib ma_list true
WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
Policy cfm_router_shut.tcl successfully registered
WARNING: User should specify MAs associated to P2P services with Down MEPs on local interface.
Undeploy cfm_router_shut.tcl policy:
agam-3845# tclsh flash:/svc_diag/sdiag_router_cfm_undeploy.tcl cfm_router_shut.tcl
cfm_router_shut.tcl is unregistered successfully
agam-3845#
Deploying cfm_router_shut.tcl policy using emm
agam-3845#emm mdf flash:/cfm_router.mdf
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]:2
Enter the email address for the _email_from field
Enter value [c3845@cisco.com]:c3845@cisco.com
Enter the email address for the _email_to field
Enter value [arshadm@cisco.com]:arshadm@cisco.com
Enter the email address for the _email_cc field
Enter value [arshadm@cisco.com]:arshadm@cisco.com
Enter the IP address for the email server
Enter value [64.102.124.15]:64.102.124.15
Enter the directory where the CFM diagnostic policies are located
in the form of a URL (e.g. disk0:/svc-diag)
Enter value [flash:/svc_diag]:flash:/svc_diag
Enter the directory where the user library files are located
in the form of a URL (e.g. disk0:/user_library)
Enter value [flash:/user_lib]:flash:/user_lib
Press any key to continue...
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]:3
================================================================================
Connectivity Fault Management Scenario Deployment
Enter ?# for item help
--------------------------------------------------------------------------------
1. Deploy CFM CC-Timeout Script
2. Deploy CFM On-Demand Script
3. Deploy Action Shut Script
4. Deploy CFM Action Auto Trace Script
5. Return to main CFM menu
Enter selection [5]:3
Enter the path to the Maintenance Association name file (e.g. disk0:/)
Enter value [flash:]:<ENTER>
Please select the Maintenance Association name file or 'CREATE_FILE'
1. cert.pem
2. ma_list.txt
3. new_ma
4. ma_list
5. CREATE_FILE
Enter choice: 5
Enter the file name for the Maintenance Association name file
Enter value [ma_list.txt]: ma_test
Please select a Maintenance Association name
1. SID_3
2. SID_4
3. SID_1
4. SID_2
Enter choice: 1
You have selected SID_3:
Write this entry to ma_test?
1. Yes
2. No
Enter choice: 1
Please select next operation
1. Add another record to the MA name file
2. Display current MA names file contents
3. Done with MA name file
Enter choice: 1
Please select a Maintenance Association name
1. SID_4
2. SID_1
3. SID_2
Enter choice: 1
You have selected SID_4:
Write this entry to ma_test?
1. Yes
2. No
Enter choice: 1
Please select next operation
1. Add another record to the MA name file
2. Display current MA names file contents
3. Done with MA name file
Enter choice: 1
Please select a Maintenance Association name
1. SID_1
2. SID_2
Enter choice: 1
You have selected SID_1:
Write this entry to ma_test?
1. Yes
2. No
Enter choice: 1
Please select next operation
1. Add another record to the MA name file
2. Display current MA names file contents
3. Done with MA name file
Enter choice: 1
Please select a Maintenance Association name
1. SID_2
Enter choice: 1
You have selected SID_2:
Write this entry to ma_test?
1. Yes
2. No
Enter choice: 1
Please select next operation
1. Add another record to the MA name file
2. Display current MA names file contents
3. Done with MA name file
Enter choice: 3
Select the type of notification to be sent when a problem is diagnosed
1. Email Only
2. SNMP Only
3. Syslog Only
4. Email and Syslog
5. Email and SNMP
6. Syslog and SNMP
7. Email, Syslog, and SNMP
Enter choice:7
Enter choice: 7
Do you want to log configuration history?
1. Yes
2. No
Enter choice: 1
Do you want to log event history?
1. Yes
2. No
Enter choice: 1
Do you want to log command history?
1. Yes
2. No
Enter choice: 1
Do you want to prepend trigger syslog messages to the diagnosis?
1. Yes
2. No
Enter choice: 1
Are the specified MAs associated to P2P services with DOWN MEPs on local interface ?
1. Yes
2. No
Enter choice: 1
WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
Policy cfm_router_shut.tcl successfully registered
WARNING: User should specify MAs associated to P2P services with Down MEPs on local interface.
Press any key to continue...
================================================================================
Connectivity Fault Management Scenario Deployment
Enter ?# for item help
--------------------------------------------------------------------------------
1. Deploy CFM CC-Timeout Script
2. Deploy CFM On-Demand Script
3. Deploy Action Shut Script
4. Deploy CFM Action Auto Trace Script
5. Return to main CFM menu
Enter selection [5]: 5
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: 7
agam-3845#
Un-deploying cfm_router_shut.tcl script using emm
agam-3845#emm mdf flash:/cfm_router.mdf
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]: 6
Please select diagnostic scenario to remove from the running-configuration
1. CFM CC-Timeout
2. CFM On-Demand
3. CFM Action Shut
4. CFM Auto Trace
5. ALL CFM Policies
Enter choice: 3
cfm_router_shut.tcl is unregistered successfully
1
Press any key to continue...
================================================================================
Connectivity Fault Management Diagnostics
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
3. Deploy CFM Scenarios
4. Display Registered Policies
5. Display Environment Variables
6. Remove Diagnostic Policies
7. Exit
Enter selection [7]:<ENTER>
Running CFM SD Scenarios
SHUT/NOSHUT Scenario
· Deploy cfm_router.tcl scenario (use either emm or tclsh CLI parser)
· Simulate fault in remote side by shutting down sub-interface .1000. Syslog and email notification is display on the screen. (debug snmp detail on).
agam-3845#
*Apr 15 16:43:33.593 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 16:43:33.593 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 16:43:34.325 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 16:43:34.361 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1
*Apr 15 16:43:34.361 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_2
*Apr 15 16:43:35.265 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 1
*Apr 15 16:43:35.265 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.1001
*Apr 15 16:43:35.841 EST: %SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:cfm_router_shut.tcl)
*Apr 15 16:43:35.953 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.1001 ...
*Apr 15 16:43:36.153 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 402 vlan 701 for service MA name SID_2 with the event code DefRemoteCCM.
*Apr 15 16:43:36.513 EST: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:cfm_router_shut.tcl)
*Apr 15 16:43:36.633 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 16:43:33.593 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 701, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.1001 with local MEP 420 has been shut and unshut.
*Apr 15 16:43:42.853 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:
Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details
Email details:
-----Original Message-----
From: c3845@cisco.com [mailto:c3845@cisco.com]
Sent: Thursday, April 15, 2010 5:48 PM
To: Arshad Mahmood (arshadm)
Cc: arsham@cisco.com
Subject: From router agam-3845:
TRIGGERED_BY "*Apr 15 16:47:54.481 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 701, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.1001 with local MEP 420 has been shut and unshut.
THE CONFIGURATION HISTORY is:
!Contextual Config Diffs:
+no logging buffered
+event manager environment _svcdiag_cfmShutNotif EMAIL,SYSLOG,SNMP
+event manager environment _svcdiag_cfmShutMAFile ma_list
+event manager environment _email_cc arshadm@cisco.com
-logging buffered 200000
-event manager environment _email_cc arsham@cisco.com
-event manager environment _svcdiag_cfmShutNotif ALL
-event manager environment _svcdiag_cfmShutMAFile flash:/ma_test
-event manager policy cfm_router_shut.tcl type user
agam-3845#
--------------------------------------------------------------
THE EVENT HISTORY is:
Syslog logging: enabled (0 messages dropped, 4 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
No Active Message Discriminator.
No Inactive Message Discriminator.
Console logging: level debugging, 1118 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 80 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled
No active filter modules.
Trap logging: level informational, 602 message lines logged
Log Buffer (200000 bytes):
*Apr 15 16:46:40.017 EST: %SYS-5-CONFIG_I: Configured from console by console
*Apr 15 16:47:09.309 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 16:47:09.309 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 420 vlan 701 MA name SID_2 domain CUST1 interface status Up event code Returning.
*Apr 15 16:47:54.481 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 16:47:54.481 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 16:47:55.217 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 16:47:55.241 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1
*Apr 15 16:47:55.241 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_2
*Apr 15 16:47:56.145 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 1
*Apr 15 16:47:56.145 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.1001
*Apr 15 16:47:56.721 EST: %SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:cfm_router_shut.tcl)
*Apr 15 16:47:56.833 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.1001 ...
*Apr 15 16:47:57.041 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 402 vlan 701 for service MA name SID_2 with the event code DefRemoteCCM.
*Apr 15 16:47:57.393 EST: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:cfm_router_shut.tcl)
*Apr 15 16:47:57.513 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 16:47:54.481 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 701, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.1001 with local MEP 420 has been shut and unshut.
agam-3845#
--------------------------------------------------------------
THE COMMAND HISTORY is:
Thu Apr 15 16:47:55 EST 2010 show event manager environ
Thu Apr 15 16:47:55 EST 2010 show ethernet cfm maintenance-points remote detail mpid 420 domain CUST1 vlan 701
Thu Apr 15 16:47:56 EST 2010 show ethernet cfm mpdb domain-id CUST1 service SID_2
Thu Apr 15 16:47:56 EST 2010 interface Gi0/1.1001
Thu Apr 15 16:47:56 EST 2010 shut
Thu Apr 15 16:47:57 EST 2010 interface Gi0/1.1001
Thu Apr 15 16:47:57 EST 2010 no shut
Syslogs and email notification after Shutting interface on remote 7604
agam-3845#
*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4200 vlan 800 MA name SID_3 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 17:02:35.377 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 720 level 7 VLAN 800 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 17:02:35.377 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 17:02:36.117 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 17:02:36.141 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1
*Apr 15 17:02:36.141 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_3
*Apr 15 17:02:37.049 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 0
*Apr 15 17:02:37.049 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.2000
*Apr 15 17:02:37.625 EST: %SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:cfm_router_shut.tcl)
*Apr 15 17:02:37.737 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.2000 ...
*Apr 15 17:02:37.937 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 720 vlan 800 for service MA name SID_3 with the event code DefRemoteCCM.
*Apr 15 17:02:37.937 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 402 vlan 701 for service MA name SID_2 with the event code DefRemoteCCM.
*Apr 15 17:02:38.193 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 17:02:38.193 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 17:02:38.297 EST: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:cfm_router_shut.tcl
*Apr 15 17:02:38.417 ES: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4200 vlan 800 MA name SID_3 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 800, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.2000 with local MEP 4200 has been shut and unshut.
*Apr 15 17:02:38.705 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4201 vlan 801 MA name SID_4 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 17:02:38.705 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 721 level 7 VLAN 801 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 17:02:40.925 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 7000 vlan 700 for service MA name SID_1 with the event code DefRemoteCCM.
*Apr 15 17:02:41.477 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 721 vlan 801 for service MA name SID_4 with the event code DefRemoteCCM.
*Apr 15 17:02:47.569 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:
Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details
--------------------------------------------------------------
*Apr 15 17:02:48.361 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 17:02:48.385 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1
*Apr 15 17:02:48.385 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_2
*Apr 15 17:02:49.285 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 1
*Apr 15 17:02:49.285 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.1001
*Apr 15 17:02:49.861 EST: %SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:cfm_router_shut.tcl)
*Apr 15 17:02:49.973 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.1001 ...
*Apr 15 17:02:50.533 EST: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:cfm_router_shut.tcl)
*Apr 15 17:02:50.653 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 701, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.1001 with local MEP 420 has been shut and unshut.
*Apr 15 17:02:59.809 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:
Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details
--------------------------------------------------------------
*Apr 15 17:03:00.609 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 17:03:00.637 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1
*Apr 15 17:03:00.637 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_1
*Apr 15 17:03:01.537 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 1
*Apr 15 17:03:01.537 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.1000
*Apr 15 17:03:02.113 EST: %SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:cfm_router_shut.tcl)
*Apr 15 17:03:02.225 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.1000 ...
*Apr 15 17:03:02.789 EST: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:cfm_router_shut.tcl)
*Apr 15 17:03:02.909 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 17:02:38.193 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 700, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.1000 with local MEP 42 has been shut and unshut.
*Apr 15 17:03:12.069 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:
Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details
--------------------------------------------------------------
*Apr 15 17:03:12.861 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 17:03:12.889 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1
*Apr 15 17:03:12.889 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_4
*Apr 15 17:03:13.793 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 0
*Apr 15 17:03:13.793 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.2001
*Apr 15 17:03:14.369 EST: %SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:cfm_router_shut.tcl)
*Apr 15 17:03:14.481 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.2001 ...
*Apr 15 17:03:15.041 EST: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:cfm_router_shut.tcl)
*Apr 15 17:03:15.161 EST %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 17:02:38.705 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4201 vlan 801 MA name SID_4 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 801, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.2001 with local MEP 4201 has been shut and unshut.
*Apr 15 17:03:23.329 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:
Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details
Email details with logging buffer 200000
-----Original Message-----
From: c3845@cisco.com [mailto:c3845@cisco.com]
Sent: Thursday, April 15, 2010 6:03 PM
To: Arshad Mahmood (arshadm)
Cc: arsham@cisco.com
Subject: From router agam-3845:
TRIGGERED_BY "*Apr 15 17:02:38.705 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4201 vlan 801 MA name SID_4 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 801, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.2001 with local MEP 4201 has been shut and unshut.
THE CONFIGURATION HISTORY is:
!Contextual Config Diffs:
+no logging buffered
+event manager environment _svcdiag_cfmShutNotif EMAIL,SYSLOG,SNMP
+event manager environment _svcdiag_cfmShutMAFile ma_list
+event manager environment _email_cc arshadm@cisco.com
-logging buffered 200000
ethernet cfm ais link-status global
-disable
-event manager environment _email_cc arsham@cisco.com
-event manager environment _svcdiag_cfmShutNotif ALL
-event manager environment _svcdiag_cfmShutMAFile flash:/ma_test
-event manager policy cfm_router_shut.tcl type user
agam-3845#
--------------------------------------------------------------
THE EVENT HISTORY is:
Syslog logging: enabled (0 messages dropped, 4 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
Console logging: level debugging, 1659 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 621 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled
No active filter modules.
Trap logging: level informational, 753 message lines logged
Log Buffer (200000 bytes):
*Apr 15 17:01:12.505 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 721 level 7 VLAN 801 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:01:12.505 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4201 vlan 801 MA name SID_4 domain CUST1 interface status Up event code Returning.
*Apr 15 17:01:13.001 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 17:01:13.025 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1
*Apr 15 17:01:13.025 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_4
*Apr 15 17:01:13.477 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Interface status associated with remote MEP with mpid 4201 vlan 801 MA name CUST1 is already Up. Aborting CFM shut script for this instance.
*Apr 15 17:01:14.241 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 17:01:14.265 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1
*Apr 15 17:01:14.265 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_3
*Apr 15 17:01:14.717 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Interface status associated with remote MEP with mpid 4200 vlan 800 MA name CUST1 is already Up. Aborting CFM shut script for this instance.
*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4200 vlan 800 MA name SID_3 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 17:02:35.377 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 720 level 7 VLAN 800 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 17:02:35.377 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 17:02:36.117 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 17:02:36.141 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1
*Apr 15 17:02:36.141 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_3
*Apr 15 17:02:37.049 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 0
*Apr 15 17:02:37.049 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.2000
*Apr 15 17:02:37.625 EST: %SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:cfm_router_shut.tcl)
*Apr 15 17:02:37.737 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.2000 ...
*Apr 15 17:02:37.937 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 720 vlan 800 for service MA name SID_3 with the event code DefRemoteCCM.
*Apr 15 17:02:37.937 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 402 vlan 701 for service MA name SID_2 with the event code DefRemoteCCM.
*Apr 15 17:02:38.193 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 17:02:38.193 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 17:02:38.297 EST: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:cfm_router_shut.tcl)
*pr 15 17:02:38.417 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4200 vlan 800 MA name SID_3 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 800, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.2000 with local MEP 4200 has been shut and unshut.
*Apr 15 17:02:38.705 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4201 vlan 801 MA name SID_4 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 17:02:38.705 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 721 level 7 VLAN 801 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 17:02:40.925 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 7000 vlan 700 for service MA name SID_1 with the event code DefRemoteCCM.
*Apr 15 17:02:41.477 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 721 vlan 801 for service MA name SID_4 with the event code DefRemoteCCM.
*Apr 15 17:02:47.569 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:
Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details
--------------------------------------------------------------
*Apr 15 17:02:48.361 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 17:02:48.385 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1
*Apr 15 17:02:48.385 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_2
*Apr 15 17:02:49.285 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 1
*Apr 15 17:02:49.285 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.1001
*Apr 15 17:02:49.861 EST: %SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:cfm_router_shut.tcl)
*Apr 15 17:02:49.973 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.1001 ...
*Apr 15 17:02:50.533 EST: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:cfm_router_shut.tcl)
*Apr 15 17:02:50.653 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 701, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.1001 with local MEP 420 has been shut and unshut.
*Apr 15 17:02:59.809 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:
Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details
-----------------------------------------------------------------------------------
THE COMMAND HISTORY is:
Thu Apr 15 17:03:12 EST 2010 show event manager environ
Thu Apr 15 17:03:13 EST 2010 show ethernet cfm maintenance-points remote detail mpid 4201 domain CUST1 vlan 801
Thu Apr 15 17:03:13 EST 2010 show ethernet cfm mpdb domain-id CUST1 service SID_4
Thu Apr 15 17:03:14 EST 2010 interface Gi0/1.2001
Thu Apr 15 17:03:14 EST 2010 shut
Thu Apr 15 17:03:14 EST 2010 interface Gi0/1.2001
Thu Apr 15 17:03:15 EST 2010 no shut
Executing cctimeout scenario:
Create a fault on the remote side to trigger cctimeout event. Following syslogs and email notification is displayed (debug SNMP detail enabled)
agam-3845#
*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 17:22:22.017 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 17:22:22.749 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 17:22:24.577 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 7000 vlan 700 for service MA name SID_1 with the event code DefRemoteCCM.
*Apr 15 17:22:56.765 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: TRIGGERED_BY "*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_timeout: Diagnostics for rmep 0024.14f6.80c1 on vlan 700, CFM MD CUST1.CCDB: 0 found; ErrorDB: 1 found; Reasons: Timeout; Ping: fail; Trace: fail
*Apr 15 17:23:04.917 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl:
Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details
--------------------------------------------------------------
agam-3845#
User will receive following email:
-----Original Message-----
From: c3845@cisco.com [mailto:c3845@cisco.com]
Sent: Thursday, April 15, 2010 6:23 PM
To: Arshad Mahmood (arshadm)
Cc: arsham@cisco.com
Subject: From router agam-3845:
TRIGGERED_BY "*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_timeout: Diagnostics for rmep 0024.14f6.80c1 on vlan 700, CFM MD CUST1.CCDB: 0 found; ErrorDB: 1 found; Reasons: Timeout; Ping: fail; Trace: fail
DETAILED MESSAGE:
Diagnostics for rmep 0024.14f6.80c1 on vlan 700, CFM MD CUST1, level = 7 ;
(1) Details of remote MEP was not found in CCDB.
(2) Details of remote MEP collected from Error DB
(a) MEP ID - 42
(b) MAC Address - 0024.14f6.80c1
(c) Reason - "Lifetime Timer Expired"
(3) Output of loopback operation performed to remote MEP :
Ping to "0024.14f6.80c1" failed.
(4) Output from Traceroute operation to remote MEP : Traceroute failed.
(5) TRACEROUTE OUTPUT:
Type escape sequence to abort. TTL 64. Linktrace Timeout is 5 seconds
Tracing the route to 0024.14f6.80c1 on Domain CUST1, Level 7, vlan 700
Traceroute sent via Gi0/1
B = Intermediary Bridge
! = Target Destination
* = Per hop Timeout
--------------------------------------------------------------------------------
MAC Ingress Ingr Action Relay Action
Hops Host Forwarded Egress Egr Action Previous Hop
--------------------------------------------------------------------------------
B 1 0017.0fad.9a50 RlyMPDB
Forwarded Gi2/48 EgrOK 0017.95e4.4c71
B 2 0017.0fad.9290 Gi2/48 IngOk RlyMPDB
Forwarded 0017.0fad.9a50
*
*
agam-3845#
THE CONFIGURATION HISTORY is:
!Contextual Config Diffs:
+no logging buffered
+event manager environment _svcdiag_cfmShutNotif EMAIL,SYSLOG,SNMP
+event manager environment _svcdiag_cfmShutConfig TRUE
+event manager environment _svcdiag_cfmShutEvent TRUE
+event manager environment _svcdiag_cfmShutCommand TRUE
+event manager environment _svcdiag_cfmShutPrepend TRUE
+event manager environment _svcdiag_cfmShutMAFile ma_list
+event manager environment _svcdiag_cfmShutP2P TRUE
+event manager environment _email_cc arshadm@cisco.com
-logging buffered 20000
ethernet cfm ais link-status global
-disable
-event manager environment _email_cc arsham@cisco.com
-event manager environment _svcdiag_cfmCctimeoutNotif ALL
-event manager environment _svcdiag_cfmCctimeoutConfig TRUE
-event manager environment _svcdiag_cfmCctimeoutEvent TRUE
-event manager environment _svcdiag_cfmCctimeoutCommand TRUE
-event manager environment _svcdiag_cfmCctimeoutPrepend TRUE
-event manager environment _svcdiag_cfmCctimeoutMAFile flash:/ma_test
-event manager policy cfm_cctimeout.tcl type user
agam-3845#
--------------------------------------------------------------
THE EVENT HISTORY is:
Syslog logging: enabled (0 messages dropped, 4 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
Console logging: level debugging, 1741 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 79 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled
No active filter modules.
Trap logging: level informational, 770 message lines logged
Log Buffer (20000 bytes):
*Apr 15 17:15:31.913 EST: %SYS-5-CONFIG_I: Configured from console by console
*Apr 15 17:20:55.697 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 721 level 7 VLAN 801 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:20:55.697 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4201 vlan 801 MA name SID_4 domain CUST1 interface status Up event code Returning.
*Apr 15 17:20:55.953 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:20:55.953 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 42 vlan 700 MA name SID_1 domain CUST1 interface status Up event code Returning.
*Apr 15 17:20:58.513 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 720 level 7 VLAN 800 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:20:58.513 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4200 vlan 800 MA name SID_3 domain CUST1 interface status Up event code Returning.
*Apr 15 17:20:59.281 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:20:59.281 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 420 vlan 701 MA name SID_2 domain CUST1 interface status Up event code Returning.
*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 17:22:22.017 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 17:22:22.749 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 17:22:24.577 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 7000 vlan 700 for service MA name SID_1 with the event code DefRemoteCCM.
*Apr 15 17:22:56.765 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: TRIGGERED_BY "*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_timeout: Diagnostics for rmep 0024.14f6.80c1 on vlan 700, CFM MD CUST1.CCDB: 0 found; ErrorDB: 1 found; Reasons: Timeout; Ping: fail; Trace: fail
agam-3845#
--------------------------------------------------------------
THE COMMAND HISTORY is:
Thu Apr 15 17:22:22 EST 2010 show event manager environ
Thu Apr 15 17:22:23 EST 2010 show ethernet cfm domain CUST1
Thu Apr 15 17:22:23 EST 2010 show ethernet cfm maintenance-points remote detail mpid 42 domain CUST1 vlan 700
Thu Apr 15 17:22:24 EST 2010 show ethernet cfm errors domain-id CUST1 service SID_1
Thu Apr 15 17:22:50 EST 2010 ping ethernet 0024.14f6.80c1 domain CUST1 vlan 700
Thu Apr 15 17:22:56 EST 2010 traceroute ethernet 0024.14f6.80c1 domain CUST1 vlan 700
Executing cfm_ondemand scenario:
After cfm_ondemand policy is deployed, you can use following command to trigger cfm_ondemand script. Syslog and email notification follows:
agam-3845#tclsh flash:/svc_diag/cfmod.tcl SID_1 CUST1 -n email,syslog
*Apr 15 18:08:03.785 EST: %HA_EM-6-LOG: cfm_ondemand.tcl: TRIGGERED_BY "Ondemand script run manually":: SERV-DIAG_CFM_ondemand: Diagnostics for vlan 700, CFM MD CUST1, rmep:all. CCDB: 1 found, 1 with UP IF state; ErrorDB: 0 found; Reasons: Ping: 1/0 total/fail;
agam-3845#
*Apr 15 18:08:12.025 EST: %HA_EM-6-LOG: cfm_ondemand.tcl:
Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details
Email detail:
-----Original Message-----
From: c3845@cisco.com [mailto:c3845@cisco.com]
Sent: Thursday, April 15, 2010 7:08 PM
To: Arshad Mahmood (arshadm)
Cc: arsham@cisco.com
Subject: From router agam-3845:
TRIGGERED_BY "Ondemand script run manually":: SERV-DIAG_CFM_ondemand: Diagnostics for vlan 700, CFM MD CUST1, rmep:all. CCDB: 1 found, 1 with UP IF state; ErrorDB: 0 found; Reasons: Ping: 1/0 total/fail;
DETAILED MESSAGE:
Diagnostics for vlan 700, CFM MD CUST1, rmep:all, level = 7 .
(1) Details of remote MEPs collected from CCDB:
(a) MEP ID - 42
(b) MAC Address - 0024.14f6.80c1
(c) Interface status - Up
(d) Port status - Up
(e) RDI state - FALSE
(f) CC packet statistics - 1139/0(Received/Error)
(2) Details of remote MEP collected from Error DB
(3) Output of loopback operation performed:
Ping to "0024.14f6.80c1" was successful.
Number of successful pings: 1
Number of failed pings: 0
(4) No MEPs with "Lifetime Timer Expired" reason is found in Error DB. Hence no traceroute operation is performed.
THE CONFIGURATION HISTORY is:
!Contextual Config Diffs:
+no logging buffered
+ethernet cfm traceroute cache hold-time 60
+event manager environment _svcdiag_cfmShutNotif EMAIL,SYSLOG,SNMP
+event manager environment _svcdiag_cfmShutConfig TRUE
+event manager environment _svcdiag_cfmShutEvent TRUE
+event manager environment _svcdiag_cfmShutCommand TRUE
+event manager environment _svcdiag_cfmShutPrepend TRUE
+event manager environment _svcdiag_cfmShutMAFile ma_list
+event manager environment _svcdiag_cfmShutP2P TRUE
+event manager environment _email_cc arshadm@cisco.com
-logging buffered 20000
-ethernet cfm traceroute cache hold-time 120
ethernet cfm ais link-status global
-disable
-event manager environment _email_cc arsham@cisco.com
-event manager environment _svcdiag_cfmOndemandConfig TRUE
-event manager environment _svcdiag_cfmOndemandEvent TRUE
-event manager environment _svcdiag_cfmOndemandCommand TRUE
-event manager environment _svcdiag_cfmOndemandPrepend TRUE
-event manager environment _svcdiag_cfmAutotraceNotif EMAIL,SYSLOG
-event manager environment _svcdiag_cfmAutotraceConfig TRUE
-event manager environment _svcdiag_cfmAutotraceEvent TRUE
-event manager environment _svcdiag_cfmAutotraceCommand TRUE
-event manager environment _svcdiag_cfmAutotracePrepend TRUE
-event manager environment _svcdiag_cfmAutotraceMAFile ma_test
-event manager environment _svcdiag_cfmAutotraceTimer 3600
-event manager environment _svcdiag_cfmOndemandMAname SID_1
-event manager environment _svcdiag_cfmOndemandDomain CUST1
-event manager environment _svcdiag_cfmOndemandMpid NULL
-event manager environment _svcdiag_cfmOndemandNotif email,syslog
-event manager policy cfm_ondemand.tcl type user
-event manager policy cfm_autotrace.tcl type user
agam-3845#
--------------------------------------------------------------
THE EVENT HISTORY is:
Syslog logging: enabled (0 messages dropped, 4 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
No Active Message Discriminator.
No Inactive Message Discriminator.
Console logging: level debugging, 1748 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 86 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled
No active filter modules.
Trap logging: level informational, 777 message lines logged
Log Buffer (20000 bytes):
*Apr 15 17:15:31.913 EST: %SYS-5-CONFIG_I: Configured from console by console
*Apr 15 17:20:55.697 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 721 level 7 VLAN 801 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:20:55.697 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4201 vlan 801 MA name SID_4 domain CUST1 interface status Up event code Returning.
*Apr 15 17:20:55.953 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:20:55.953 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 42 vlan 700 MA name SID_1 domain CUST1 interface status Up event code Returning.
*Apr 15 17:20:58.513 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 720 level 7 VLAN 800 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:20:58.513 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4200 vlan 800 MA name SID_3 domain CUST1 interface status Up event code Returning.
*Apr 15 17:20:59.281 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:20:59.281 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 420 vlan 701 MA name SID_2 domain CUST1 interface status Up event code Returning.
*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 17:22:22.017 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 17:22:22.749 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 17:22:24.577 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 7000 vlan 700 for service MA name SID_1 with the event code DefRemoteCCM.
*Apr 15 17:22:56.765 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: TRIGGERED_BY "*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_timeout: Diagnostics for rmep 0024.14f6.80c1 on vlan 700, CFM MD CUST1.CCDB: 0 found; ErrorDB: 1 found; Reasons: Timeout; Ping: fail; Trace: fail
*Apr 15 17:23:04.917 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl:
Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details
--------------------------------------------------------------
*Apr 15 17:23:13.937 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:23:13.941 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 42 vlan 700 MA name SID_1 domain CUST1 interface status Up event code Returning.
*Apr 15 18:04:13.657 EST: %SYS-5-CONFIG_I: Configured from console by console
*Apr 15 18:08:03.785 EST: %HA_EM-6-LOG: cfm_ondemand.tcl: TRIGGERED_BY "Ondemand script run manually":: SERV-DIAG_CFM_ondemand: Diagnostics for vlan 700, CFM MD CUST1, rmep:all. CCDB: 1 found, 1 with UP IF state; ErrorDB: 0 found; Reasons: Ping: 1/0 total/fail;
agam-3845#
--------------------------------------------------------------
THE COMMAND HISTORY is:
Thu Apr 15 18:07:53 EST 2010 show event manager environ
Thu Apr 15 18:07:55 EST 2010 show ethernet cfm domain CUST1
Thu Apr 15 18:07:55 EST 2010 show ethernet cfm domain CUST1
Thu Apr 15 18:07:56 EST 2010 show ethernet cfm maintenance-points remote domain CUST1
Thu Apr 15 18:07:57 EST 2010 show ethernet cfm errors domain-id CUST1 service SID_1
Thu Apr 15 18:07:58 EST 2010 ping ethernet 0024.14f6.80c1 domain CUST1 vlan 700
Thu Apr 15 18:08:03 EST 2010 ping ethernet multicast domain CUST1 vlan 700
--------------------------------------------------------------
Autotrace Scenario
Autotrace scenario automatically triggers the cfm_autotrace policy after a specified interval. This action is very useful and it does not require any human intervention. You are notified through syslog or email notification after regular time intervals. Following are details that are sent as an email notification every hour (60 Min):
-----Original Message-----
From: c3845@cisco.com [mailto:c3845@cisco.com]
Sent: Friday, April 16, 2010 1:05 PM
To: Arshad Mahmood (arshadm)
Cc: arsham@cisco.com
Subject: From router agam-3845:
TRIGGERED_BY "auto-trace watchdog timer every 60 minutes.":: SERV-DIAG_CFM_autotrace: A total of 4 MEPS for valid MAs specified in ma_test have been automatically tracerouted. Results stored in the CFM traceroute cache.
CFM Domain CUST1
MA SID_3 - vlan 800
mpid 4200, mac address 0024.14f6.80c1
MA SID_4 - vlan 801
mpid 4201, mac address 0024.14f6.80c1
MA SID_1 - vlan 700
mpid 42, mac address 0024.14f6.80c1
MA SID_2 - vlan 701
mpid 420, mac address 0024.14f6.80c1
THE CONFIGURATION HISTORY is:
!Contextual Config Diffs:
+no logging buffered
+ethernet cfm traceroute cache hold-time 60
+event manager environment _svcdiag_cfmShutNotif EMAIL,SYSLOG,SNMP
+event manager environment _svcdiag_cfmShutConfig TRUE
+event manager environment _svcdiag_cfmShutEvent TRUE
+event manager environment _svcdiag_cfmShutCommand TRUE
+event manager environment _svcdiag_cfmShutPrepend TRUE
+event manager environment _svcdiag_cfmShutMAFile ma_list
+event manager environment _svcdiag_cfmShutP2P TRUE
+event manager environment _email_cc arshadm@cisco.com
-logging buffered 20000
-ethernet cfm traceroute cache hold-time 120
ethernet cfm ais link-status global
-disable
-event manager environment _email_cc arsham@cisco.com
-event manager environment _svcdiag_cfmAutotraceNotif EMAIL,SYSLOG
-event manager environment _svcdiag_cfmAutotraceConfig TRUE
-event manager environment _svcdiag_cfmAutotraceEvent TRUE
-event manager environment _svcdiag_cfmAutotraceCommand TRUE
-event manager environment _svcdiag_cfmAutotracePrepend TRUE
-event manager environment _svcdiag_cfmAutotraceMAFile ma_test
-event manager environment _svcdiag_cfmAutotraceTimer 3600
-event manager policy cfm_autotrace.tcl type user
agam-3845#
--------------------------------------------------------------
THE EVENT HISTORY is:
Syslog logging: enabled (0 messages dropped, 4 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
Console logging: level debugging, 1856 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 194 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled
No active filter modules.
Trap logging: level informational, 885 message lines logged
Log Buffer (20000 bytes):
*Apr 15 17:15:31.913 EST: %SYS-5-CONFIG_I: Configured from console by console
*Apr 15 17:20:55.697 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 721 level 7 VLAN 801 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:20:55.697 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4201 vlan 801 MA name SID_4 domain CUST1 interface status Up event code Returning.
*Apr 15 17:20:55.953 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:20:55.953 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 42 vlan 700 MA name SID_1 domain CUST1 interface status Up event code Returning.
*Apr 15 17:20:58.513 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 720 level 7 VLAN 800 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:20:58.513 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4200 vlan 800 MA name SID_3 domain CUST1 interface status Up event code Returning.
*Apr 15 17:20:59.281 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:20:59.281 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 420 vlan 701 MA name SID_2 domain CUST1 interface status Up event code Returning.
*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut.
*Apr 15 17:22:22.017 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 enters AIS defect condition
*Apr 15 17:22:22.749 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification
*Apr 15 17:22:24.577 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 7000 vlan 700 for service MA name SID_1 with the event code DefRemoteCCM.
*Apr 15 17:22:56.765 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: TRIGGERED_BY "*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_timeout: Diagnostics for rmep 0024.14f6.80c1 on vlan 700, CFM MD CUST1.CCDB: 0 found; ErrorDB: 1 found; Reasons: Timeout; Ping: fail; Trace: fail
*Apr 15 17:23:04.917 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl:
Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details
--------------------------------------------------------------
*Apr 15 17:23:13.937 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 exited AIS defect condition
*Apr 15 17:23:13.941 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 42 vlan 700 MA name SID_1 domain CUST1 interface status Up event code Returning.
*Apr 15 18:04:13.657 EST: %SYS-5-CONFIG_I: Configured from console by console
*Apr 15 18:08:03.785 EST: %HA_EM-6-LOG: cfm_ondemand.tcl: TRIGGERED_BY "Ondemand script run manually":: SERV-DIAG_CFM_ondemand: Diagnostics for vlan 700, CFM MD CUST1, rmep:all. CCDB: 1 found, 1 with UP IF state; ErrorDB: 0 found; Reasons: Ping: 1/0 total/fail;
*Apr 15 18:08:12.025 EST: %HA_EM-6-LOG: cfm_ondemand.tcl:
Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details
THE COMMAND HISTORY is:
Fri Apr 16 12:04:41 EST 2010 show event manager environ
Fri Apr 16 12:04:41 EST 2010 show ethernet cfm traceroute-cache
References
Cisco Embedded Event Manager Overview: http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_eem_overview_ps6441_TSD_Products_Configuration_Guide_Chapter.html
Cisco EEM Cisco.com webpage: http://www.cisco.com/en/US/products/ps6815/products_ios_protocol_group_home.html
Cisco Service Diagnostics Cisco.com webpage : http://www.cisco.com/en/US/products/ps9424/products_ios_protocol_group_home.html
Cisco Service Diagnostics Q&A: http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps9424/qa_cisco_ios_service_diagnostics.html
Cisco Border Gateway Protocol, OSPF, and QoS Diagnostics Scripts: http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps9424/white_paper_cisco_ios_service_design_bgp_osp_qos.html
Cisco Service Diagnostics CFM 2.0 Scripts: http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps9424/whitepaper_c11-566741.html
CFM in Service Provider Network Configuration Guide: http://www.cisco.com/en/US/docs/ios/cether/configuration/guide/ce_cfm-ieee.html
Service Diagnostics Documentation and Scripts: http://www.cisco.com/go/iossd