Cisco on Cisco

Cisco IT is moving its production environment from San Jose, California, to Richardson, Texas. In Richardson, all applications will reside on a Cisco Unified Computing System, which combines compute, networking, storage access, and virtualization in a single system managed as a cohesive entity. The Cisco Unified Computing System is the foundation for Cisco's private cloud architecture.

Cisco IT's main objectives for the Richardson data center are minimizing total cost of ownership (TCO), making sure the company has adequate capacity for growth, ensuring business continuity, and supporting business innovation by accelerating IT provisioning. "To meet these goals, Cisco IT is consolidating to the x86 processor platform and two operating systems, Linux and Windows," says Sidney Morgan, data center architect, Cisco IT.

Cisco traditionally has used four operating systems: Linux and Windows as well as HP-UX and Solaris. Previously, approximately 200 HP Superdome and mid-range HP-UX servers in the U.S. and Europe hosted Oracle Real Application Clusters (RAC) and Oracle eBusiness Suite. In addition, the Cisco Scientific Atlanta division in Georgia used HP-UX Itanium servers to host SAP ERP Central Component (ECC) 6 and SAP Exchange Infrastructure (XI).

To continue the journey to the private cloud, Cisco IT needed to migrate these Oracle and SAP applications and databases to the Cisco Unified Computing System. One motivation was resiliency, because the RISC-based architecture did not allow Cisco to take full advantage of virtualization. The other motivation was reducing ongoing costs. "The cost of proprietary RISC platforms is very high, especially for maintenance and upgrades," says Jag Kahlon, IT architect at Cisco. "The x86 platform has matured, so by 2010, the time for Cisco to move entirely off the RISC platform had arrived."

Cisco IT is systematically moving Oracle and SAP applications and databases from HP-UX platforms to the Cisco Unified Computing System. "Cisco has been looking for ways to take advantage of the lower cost structure, greater flexibility, and rapid innovation possible with standard hardware platforms," says Brian Kennelly, IT manager for network and data services, Cisco. "That opportunity came when the x86 architecture began to support true 64-bit processing, which in turn supports large memory systems. The Cisco Unified Computing System takes advantage of these advances, enabling Cisco to operate our Oracle and SAP environments at lower cost."

The system in Richardson, used for the Oracle environment, contains four Cisco UCS B200 M1 Blade Servers, each with 48GB RAM. Currently, the Oracle RAC 11g database, Goal to Commission (G2C) applications, and Cisco channel partner database are in production on a Cisco Unified Computing System. The Quote to Cash (Q2C) database will be converted in March 2011, and the remaining Oracle databases will follow. SAP ECC 6 and SAP XI are also operating on a Cisco Unified Computing System in Georgia, in a test environment.

The process to migrate the Oracle and SAP environments is similar, with a few differences.

Migrating Oracle Environment from Superdome to Cisco Unified Computing System

Cisco IT began migrating the Oracle eBusiness Suite from HP Superdome to the Cisco Unified Computing System in October 2009. The actual conversion takes less than 24 hours. However, Cisco IT typically also performs application upgrades during the migration process, so the team completes multiple test cycles to test application functionality and performance in the new environment. Therefore, the full migration process takes about 6 to 12 months.

Cisco IT started with the G2C application, which has a 4TB database that previously resided on a two-node Superdome cluster with 32 cores. The web presentation layer and applications are virtualized, using VMware on the Cisco Unified Computing System. The Oracle database operates on bare metal in the same Cisco Unified Computing System chassis as the virtualized components.

The migration was somewhat more complicated for Cisco IT than it would be for most organizations because Cisco was moving to a new data center at the same time. To minimize downtime, Cisco IT performed the following steps prior to the migration:

• Set up a standby instance using Oracle Data Guard, on the Cisco Unified Computing System in the Richardson data center.
• Created the target database in Oracle RAC 10g.
• Imported metadata (read-only) from Data Guard to the target.
• Copied static data from the standby Richardson instance to the target, in read-only mode. Cisco migrated about 50 percent of the data in advance, using custom scripts.
• Copied the Oracle eBusiness Suite Release 12 (R12) code tree to the target Richardson hosts.
• Cloned the R12 target and prepared context files.

Then, during a one-hour downtime window on a weekend evening, Cisco IT completed the migration and the move to the new data center:

• Shut down the San Jose G2C production application after synching it with the Richardson Data Guard instance.
• Activated the Richardson Data Guard.
• Imported the remaining data. HP-UX uses the Big Endian system for data storage, while Linux uses the Little Endian system. (The naming refers to whether the big end or little end of a byte is stored first.) To convert database tables to the Little Endian format, Cisco IT uses the Oracle Data Pump utility or the Oracle Transportable Table Spaces (TTS) utility. "We could skip all the tables with zero rows because we had already imported the metadata," says Ramakrishna Nishtala, IT architect at Cisco.
• Created indexes, constraints, and triggers. To minimize conversion time, the team split the indexes into multiple scripts that executed in parallel across all four Cisco UCS blade servers. "Then we adjusted sequences because metadata had been imported earlier, so the sequences might have advanced," Nishtala says.
• Audited the import by comparing rows and objects.
• Executed autoconfig on the R12 front end to validate Apache, concurrent managers, forms, and workflow.
• Upgraded from Oracle RAC 10g to 11g.

Migrating SAP ERP from HP-UX to Cisco Unified Computing System

The Cisco Scientific Atlanta division uses SAP ERP software to support manufacturing processes. Previously, the SAP environment, used by 3000 employees, resided on five HP-UX Itanium servers. The SAP back end includes an Oracle RAC 10g database as well as a 3TB SAP production database.

The IT team has completed four major SAP upgrades since beginning to use the software in 1995. The last major upgrade was in 2008, when Cisco switched from ECC 4.6 to ECC 6 to acquire Unicode support for multiple languages. Because the software version was current, the migration from HP-UX to the Cisco Unified Computing System did not require a software upgrade. "It was a good time to migrate because we take a fresh look at platforms when the maintenance contract ends," says Chris Ellison, IT manager for application technology services, Cisco IT.

The first step in the migration to the Cisco Unified Computing System was to move SAP ECC 6 and SAP XI, used to manage application-to-application and business-to-business interfaces, such as inbound files from contract manufacturers. Figure 1 shows the Cisco Scientific Atlanta SAP landscape on the Cisco Unified Computing System.

Over one weekend in November 2010, Cisco IT converted the SAP ERP database in the test environment from HP Itanium to the Cisco Unified Computing System. The team used the following process:

• Built a copy of the entire production environment, including servers and storage, for parallel unloading and reloading. The test environment on the Cisco Unified Computing System consists of 20 Cisco UCS B200 M1 Blade Servers. Sixteen of the servers have 96 GB of RAM, for high-use applications, and the other four have 48 GB of RAM. Cisco IT configured multiple blade servers for unload and reload. "The more servers you use, the less time it takes to convert the data from Big Endian to Little Endian," Ellison says. "Therefore, the virtualization capabilities in Cisco Unified Computing System create a big advantage. After we completed the conversion to Little Endian, we were able to reclaim two servers for other uses."
• Moved applications from the HP-UX platform to the Cisco Unified Computing System. Organizations can migrate SAP applications and database to the Cisco Unified Computing System at the same time, or else migrate the applications first and temporarily keep the database on the HP-UX to migrate later. Cisco IT took the latter approach. "SAP can operate in a multi-platform environment," Ellison says. "We decided to replace the HP-UX servers gradually, to avoid disrupting other projects. We currently have seven SAP application servers on the Cisco Unified Computing System. Our end users can't tell which platform a given application runs on because the user experience is the same. In fact, application processes are actually faster on the Cisco Unified Computing System." (Performance improvements are shown in Figures 4 and 5.)
• Converted the back-end Oracle RAC database from Big Endian to Little Endian. Although Cisco IT is waiting to move the production database to the Cisco Unified Computing System until after the application servers have been moved, the team wanted to test the converted database beforehand. With practice, Cisco IT learned to perform the conversion faster. "We converted to Little Endian in 2008, to adopt Unicode," Ellison says. "After ten full-scale practice runs, during which we also fine-tuned the configuration, we improved our time from 42 hours to eight hours."
• Move applications from test to production. To move a SAP application on the Cisco Unified Computing System into production, Cisco IT simply points the SAP clients to the server's new location. Using the IP addresses and names of the original servers on the Cisco Unified Computing System avoids the need for any changes to the clients. "When our users opened the SAP GUI on the Monday following the conversion, they connected to the new servers without noticing any difference," says Ellison.

The next step will be converting the smaller development environment. "Then we'll start practicing the production conversion process," says Ellison. "The practice sessions will let us know the optimum number of servers we need to perform the unload-reload process." The team expects SAP ERP on the Cisco Unified Computing System in production in late 2011.

"Moving the Oracle and SAP environments to the Cisco Unified Computing System has been transformative," says Kahlon. "Performance increased, total cost of ownership decreased, and the conversion has been extremely smooth."

Lower Total Cost of Ownership

Factors in reducing the TCO include:

• Data center space, power, and cooling savings: The four-node Cisco Unified Computing System occupies one-tenth the space of the two-node HP Superdome it replaced, consumes less than one-third the power, and uses fewer than half of the cables (Figure 2).
• Lower licensing costs from migrating from RISC to x86 processors.
• Lower support and maintenance costs because of the x86 processor. "Platforms used for virtualization tend to run out of memory before they run out of compute capacity," says Kennelly. "The high memory capacity of the Cisco Unified Computing System lets us operate enterprise databases on blade servers with commodity-based hardware."
• Lower hardware costs by repurposing blade servers after the migration.

Increased Business Agility Through Automated Service Provisioning

By migrating business-critical applications to the internal private cloud, Cisco IT reduced deployment time from six to eight weeks to 15 minutes. As an example, consider a Cisco employee who needs an Oracle eBusiness application server. The employee visits a web-based ticket request system to specify:

• Number of processors
• Operating system
• Amount of memory
• Type of firewall security
• Load balancing
• SSL offload
• Storage requirements

The request generates a ticket that is sent automatically to the Cisco IT infrastructure implementation team. To fulfill the request, the engineer logs into a self-service web portal, is authenticated, sees a list of data centers where the server can be provisioned, and then clicks Submit. This action generates a message to the employee's manager for approval. "After approval, resources are deployed within 15 minutes," Morgan says. "This compares to six to eight weeks with our previous server platform, because of the manual configuration steps."

Morgan adds, "Cisco did not have to change any of our Six Sigma Lean Processes when we migrated Oracle databases and eBusiness applications to the Cisco Unified Computing System. We use the same processes, but deliver them using the IaaS [Infrastructure as a Service] model instead of manually. The result is lower TCO and increased business agility."

Resilience, for Business Continuity

If a Cisco UCS blade server fails, Cisco IT can provision another server in a few minutes by applying a Cisco UCS Manager service profile, and then use VMware tools to move the application. "With our previous hosting platform, we would have to call the vendor to install a new part, which would take hours at best, and possibly days," says Kahlon.

Better User Experience, for Increased Productivity

Moving applications that use the Oracle G2C database from HP Superdome to the Cisco Unified Computing System improved application performance, as shown in Figure 3.

Moving SAP applications from HP Itanium servers to Cisco Unified Computing System also improved the user experience (Figures 4 and 5). Times shown are averages, in milliseconds.

The Cisco IT team has already used the Cisco Unified Computing System to overcome a performance issue in the production SAP environment. As part of a Cisco global initiative with an aggressive schedule, the team had to load more than 300,000 part numbers into SAP, most of which required part-number descriptions in the simplified Chinese Unicode character set. The initial SAP Legacy System Migration Workbench (LSMW) parts load significantly slowed the SAP environment, degrading the user experience and causing several application servers to fail. By applying Cisco UCS Manager service profiles, the team could rapidly provision enough SAP application servers to meet the heavy demand. The additional SAP servers allowed the team to segregate user application servers from batch-load application servers. Segregating the massive parts loads on dedicated servers protected the performance of the other servers.

Cisco IT is planning to move all applications that reside on HP-UX servers, including mission-critical ERP applications, to the Cisco Unified Computing System. Examples include the Cisco Care Center, customer databases, ordering systems, and business intelligence.

Oracle Migration

• When Oracle certifies Cross-platform Transportable Table Spaces (XTTS) to migrate an R12 database from Big Endian to Little Endian, use XTTS to simplify the migration process.
• Migration is simpler if you use the same storage in the test and production environments.
• For subsequent migrations, Cisco IT will use the Cisco UCS B440 blade servers to take advantage of their higher capacity, and then remount the storage to Cisco UCS B200 Series blade servers.

SAP Migration

• Perform test runs before going live. "Practice lets you know how long the process will take so there are no surprises," says Ellison. "For SAP, if you're not doing an upgrade at the same time, it's pretty straightforward."
• For SAP migrations, use parallel unload-reload servers to save time. Use the SAP Distribution Monitor system copy tool in a way that takes advantage of parallel processing capabilities of the Cisco Unified Computing System. Experiment to find the optimum number of servers. Cisco used 10 unload-reload servers to convert the test environment.
• Create separate unload/reload network segments. Enable Jumbo Frames on both segments. "When we did this for SAP Unicode conversion, system copy time decreased by 25 to 30 percent," Ellison says.
• Take advantage of the storage array's on-disk backup capability to eliminate the need for long database restores.
• Consider converting SAP to Unicode during the migration. "Converting from non-Unicode to Unicode on RISC platforms is very time-consuming," says Ellison. "The virtualization capabilities in the Cisco Unified Computing System allow you to relatively quickly perform the conversion during the upgrade."