Companywide migration to standard tools for services management and configuration enable enterprise integration, full support for solutions, and smooth path for future upgrades.
The mission-critical Cisco® technical support organization delivers a variety of revenue-generating customer services. Most support services contract customers, and customers covered by new product warranties, depend on the efficient and timely logging, tracking, and managing of customer service requests. Customers can request assistance from Cisco.com by sending e-mail to the technical support organization, or by phoning the customer call center. Two key application packages supported service delivery operations. The CARE system, based on Clarify application foundation technology, was used for tracking service requests. The Metrix system served as the foundation for all service logistics, including spare parts inventory management, accounting, returns/repair management, and order management. Both of these purchased applications had been in use for approximately a decade. Heavily customized by IT engineers, vendor-provided software upgrades and support were no longer available. CARE and Metrix were in end-of-support (EOS) status; neither package had been upgraded for several years.
A third software package, a product configuration tool based upon EOS Calico foundation technology, was used to configure and order replacement hardware. This Calico-based tool was considered part of the foundation systems architecture as well.
While the technical support engineering and service logistics organizations were deeply engaged in processes that heavily relied on these applications, the executive team was concerned. “The technical support organization of more than 3000 engineering and logistics personnel was doing business on a burning platform,” says Joe Mastropolo, the Cisco IT manager assigned to address the situation. “The software environment could not be upgraded, and support was becoming a real challenge. Key applications were in various states of disrepair, and the risk of affecting business had become very real. The executive team could not accept the increasing risk of a customer-impacting support process breakdown, so a decision was made to take action.”
David Murray
Director for Release Management, Oracle Projects, Cisco IT
Within the IT world, engineers are considered the most challenging user base when it comes to technology change. Years of customizations had resulted in a complex platform, but one that was familiar and comfortable for its resident users. CARE, Metrix, and Calico, while out of date and in danger of imminent failure, were getting the job done for the engineers. The biggest challenges associated with this project were user-focused. Convincing the hundreds of engineers that replacement software was vital to the success of the company was the first hurdle, and introducing the selected solutions would require unprecedented user preparation.
A sense of urgency drove the selection process forward. David Murray, the director of release management for the Oracle 11i initiative, described the challenge: “By operating with end-of-life software, the service organization was at great risk. The replacement project was crucial and time-sensitive. With the tech support organization, we had to carefully assess the impact of the change and manage user expectations. There was great attachment to the customized software and a brute-force replacement would doom our chances of successful user adoption.”
A detailed vendor package evaluation and proof of concept were required to ensure ultimate acceptance by the user base. With thorough evaluation criteria defined up front, the project team was able to objectively evaluate and score five vendors of relevant application packages. The choices were narrowed down to two vendors, each providing a solution demonstration as the final step in the selection process. The Oracle 11i application suite was selected as the best choice for customer call management and service logistics. For the product configurator, Selectica technology was chosen. This meant an implementation of many new modules for Cisco, as well as the use of relatively new technology from Oracle. While the Oracle 11i suite introduced risk, the team agreed it was the best solution overall.
In parallel with the technical support project, Cisco manufacturing and finance teams were upgrading to the Oracle 11i platform. The new corporate data foundation provided additional incentive for the adoption of Oracle 11i technical support solutions, but also introduced additional challenges since the new technical support tools would be integrated with a relatively new enterprise architecture (see the related Cisco IT case study, “ERP Manufacturing and Finance" at http://www.cisco.com/web/about/ciscoitatwork/business_of_it/erp_manufacturing_and_finance.html).
With the selection made, the project team prepared to implement the proof-of-concept phase. A “sandbox” or practice environment was established, and a sample set of users (engineers) was chosen to participate. These engineers were given extensive training - four to six weeks each - in preparation for the three-month evaluation phase. The efforts contributed to the overall solution implementation plan:
- Actual users were able to test out the new applications. They were able to assess the software limits and capabilities in reality, not just by evaluating documentation and product specifications.
- The lessons learned formed the foundation for a solid concept commit definition. Concept commitments were articulated based on factual observations and data, rather than best guesses.
- Precise implementation plans were formulated. In particular, the proof-of-concept team recommended the avoidance of any all-at-once implementation. The optimal plan included a phased approach to minimize risks, and to provide enough time for exhaustive data management (e.g., product configurator rule definitions) to be accomplished well in advance of actual deployment.
Phase One
In June 2003, the project team was ready for the first phase - setting up a production environment and configuration of the Oracle 11i application suite in preparation for the eventual transaction volume. This first minor release provided a good foundation for the replacements, and would serve as the reference environment for all subsequent test cycles. In parallel with the establishment of the production environment, the team began the process of replacing Calico with Selectica technology. Meeting the business needs would entail many months of defining data management rules, and this up-front work could only be done using the production environment.
Phase Two
By December, the foundation was ready for the next minor release - the loading of service data. Massive amounts of information residing on external systems - such as the service and sales marketing system—required months to load. With the data loaded, the project implementation team would be ready to use the new software.
Phase Three
The beginning of 2004 marked a major release and the first actual business transactions in the new production environment. Functions were phased in using a metered approach, with increasing amounts released each week over a four-week period. Initially, customer-facing functions were moved from the old CARE system to the new Oracle applications. Cisco’s call center traffic was gradually redirected, with e-mail requests being the first to be serviced by the new platform. E-mail requests represent the lower-priority requests, requiring responses in terms of days, not hours. Two weeks later, phone requests went live on the new platform. By this point, approximately one third of all service requests were being directed by the new solution. In the fourth week, the team activated the tool for handling Web-initiated requests, which represented the majority of all requests and the remainder of the transaction volume from the previous platform.
As new functions were introduced, the implementation team used previously defined criteria to evaluate them. The criteria were defined to mitigate risks and ensure that the new system was stable before continuing.
By the end of this phase, CARE was decommissioned. The team proceeded to phase in some back-office service logistics as well, relating to the service logistics system. Inventory management, spare parts planning, repair operations, finance, and other service-related functions were phased in gradually.
Phase Four
The next minor release focused on business intelligence. In May 2004, this minor release addressed reporting and business objects-related functions.
Phase Five
During a major release in December 2004, the final project steps were taken and Metrix was officially decommissioned. Customer-facing functions, including order entry, order management, and returns management, had also moved to the Oracle platform.
The staggered release plan, spread over an 18-month period, proved to be critical to the success of the project. With careful up-front planning and solid release management, risks were effectively mitigated. The main goal - avoiding incidents that would put the business at risk - was essentially met, although the program did encounter some minor challenges. Both of the major releases (March and December 2004) experienced some post-production stability issues. In each case, the project team was able to fully stabilize the platform within a couple of months without major impact to operations.
The resulting architectural changes have offered the technical support organization many benefits:
- Enterprise-class platform - Poorly implemented foundational software that evolved without any architectural direction has been replaced with integrated enterprise-class applications. The technical support solutions are in line with the corporate Oracle 11i initiative, and provide a single platform for product and service teams, optimizing cross-functional efficiencies and information sharing.
- Supportability and reliability - The new Oracle foundation gives the support organization the benefits of a three-tier design. Desktops require only a Web browser, with the applications themselves located on servers for simplified, centralized upgrades and maintenance. Back-end servers were previously configured in an active-passive configuration, with the backup server unused except during failover mode. The new architecture configures servers in active-active pairs, yielding much more compute power to the enterprise and providing hot-standby capabilities in the event of any server failures. Table 1 provides a before and after comparison of overall reliability, availability, and scalability (RAS) features. The standard software replaces highly customized software, providing a more stable foundation for future growth and enhancements, and introduces secure access capabilities to external partners where none was realistically available.
- Disaster recovery - Redundancy is built in at many levels. The main corporate database engine is hosted in a San Jose, California data center, using two servers in an active-active configuration. A second site in North Carolina acts as a disaster recovery site in the event of a site failure at the main data center.
- Stable platform for users - The chance of disruptions to users has been greatly minimized, and users can do their jobs with tools that provide stability and productivity. Achieving this stability required adopting standard products and giving up some previously customized alternatives, but the long-term benefits greatly outweigh the short-term inconveniences.
| Table 1. Upgrade Reliability, Availability, and Scalability. | ||
|---|---|---|
| RAS Features | Before Upgrade | After Upgrade |
| Network | Redundant Switches | Redundant Switches |
| Local Directors | None | Redundant Local Directors |
| Application Server | Single Node with fail-over | Multiple Nodes with Load Balancing. Scalable Architecture |
| Application Load Balancing | None | Apache, Forms and Reports Load balanced using Local Directors |
| Database | - Oracle 8i Parallel Server |
- Oracle 8i Parallel Server |
| Third Party Applications | Server Cluster for High-Availability | Server Cluster for High-Availability |
| OID (LDAP) for Authentication | None | High-Availability test is in progress |
Policy Server |
None | Multiple Nodes with Load Balancing |
“In hindsight,” says Mastropolo, “we understand that the Oracle CRM modules and DBMS version were somewhat new and relatively immature when we adopted them. Subsequent releases and patches eliminated many of the bug and stability issues that we encountered during our implementation. Our experience emphasized the benefit of being prepared for the unexpected. Anyone tackling a project of this magnitude needs to develop an implementation plan that mitigates business risk, and includes contingency plans.”
Beyond the lessons that the IT team learned about early release software, the users themselves point to lessons learned about teamwork and the setting of expectations. From the beginnings of the Oracle 11i initiative, the executive team recognized the need to involve representatives from every business team. This increased emphasis on communications with and training of end users has become a success factor for projects throughout Cisco. For the Oracle 11i initiative, a multilevel team was established to minimize risks, optimally prepare for the move to the new foundation and applications, and ensure successful adoption by all affected users. The team included:
- Executive team - This team met quarterly to define goals, allocate resources, assess progress, and adjust plans as needed.
- Steering committee - This multifunctional team met monthly to provide governance and guidance for the program management team, and to proactively address program risks and resource issues.
- Program management organization (PMO) - IT provided dedicated, full-time staff to manage the development and deployment work. This team also provided day-to-day management of the release management and business flow teams.
- Release management team - This group coordinated the process of planning and implementing rehearsals and “go-live” events.
- Business flow team - This team was made up of representatives from the affected groups in each of six critical Oracle-supported business flows, and served as the voice for business requirements combined with user perspectives during the move to Oracle 11i.
Mahesh Bhumralkar of the Cisco technical support team served as senior manager of the business flow team that represented the technical support business for the Oracle 11i deployment. “By involving users from every affected business area, the company made sure each business team was given a voice,” says Bhumralkar. “We were trained and prepared for critical business impacts in advance, and we had a way to raise our concerns and adjust the process to avoid pitfalls that our engineers could foresee.
“Changes of this magnitude will always cause some pain and be met with resistance,” added Bhumralkar. “But our organization adoption process helped to manage this resistance and increase user buy-in, even knowing that the changes would not all be positive for our users. By communicating the long-term benefits as well as the short-term sacrifices, such as losing some of our customized tools, we were able to more effectively manage the transition and the business impacts.”
End-user communications ultimately came from within individual organizations. The designation of inside advocates lessened the resistance, and ensured that users were given realistic information about the new solutions. The anticipated challenges were openly described. After the go-live events for the major releases, a network of ambassadors provided additional frontline end-user support. Each of the worldwide user constituents groups - more than 12 in all, for the technical support organization - had an identified onsite point of contact for any Oracle 11i issues. These ambassadors were kept up to date, and provided direct-line dissemination of status to the users.
Preparations for change were also necessary for customers, since some of the new software was customer-facing. Changes were inevitably introduced to the service request processes. While the service flows were changed for the better - matching best-in-class industry practices - some resistance to change was expected. The team learned the value of being prepared for customer feedback, both positive and negative, realizing that some negative feedback was inevitable and would require effective resolutions.
With the replacement accomplished, the technical support organization continues to address user concerns as they arise. The corporate goals for supportability and minimizing total cost of ownership have resulted in new attitudes regarding software customizations. To ease support and upgrades, Cisco relies on strong vendor relationships. User requests are fed back as enhancement requests, and the user base continues to adjust processes and workflow to maximize efficiencies. Murray summarizes, “The technical support systems replacement project exceeded our expectations - and purposeful focus on the user perspectives made all the difference. This complex project was vital to this large revenue-generating organization, and teamwork, meticulous planning, and comprehensive user preparedness have put us on solid ground for the future.”