Cisco on Cisco

Cisco® IT began virtualizing servers in February 2005. By July 2007, more than 2000 virtualized servers on 127 physical machines had been deployed, new ones at a rate near 300 per quarter. A virtualized server is now Cisco IT’s default offering to internal clients, and clients who do not want a virtualized server must justify why their application will not work on one.

By any large enterprise measure, this is an aggressive implementation of virtualization technology—and a necessary one that has yielded Cisco IT beneficial results.

In 2005, Cisco faced the same operational challenges in its data centers as so many other enterprises. Rapid growth in applications (26 percent per year), servers (22 percent per year), and storage (roughly 50 percent per year) left Cisco IT with little to no available floor space in its data centers for new equipment such as servers. In addition to demand for space, this rapid growth fueled a greater need for expensive power, cooling, and related hardware resources, along with increasingly longer delays (12 weeks or more) for deploying new servers and rolling out new applications in response to market conditions and business needs.

“There was high demand on the infrastructure teams, and a lot of coordination and collaboration was needed to get applications into the data center,” says Ken Bulkin, senior manager in the Network and Data Center Services group at Cisco. “One of our biggest challenges in 2005 was getting infrastructure into the data center in a timely fashion. In some cases, our ability to meet client demands was restricted by unacceptable timeframes in acquiring hardware.”

What’s more, a predominant share of Cisco IT’s server space sat unused. Each new application required a new server to run it, yet most applications used only a fraction of the processing capacity of new servers. Typically, server utilization across the data center was only about 8 percent, according to Bulkin.

With server virtualization, operating systems can be installed on a single physical server. Instead of running one application on a machine, several applications can run, all isolated into virtual operating system images that do not affect each other. Virtualization effectively decouples the application environment from the hosting computing, network, and storage hardware. This decoupling allows logical partitioning of one device into many, consolidation of many devices into one logical resource pool, or both.

Running 10 or more virtualized servers on a single physical server would notably reduce the number of new servers required in the data center, and lower the space, power, heat, weight, hardware, operating system, and other costs for new installations. Most important to Cisco IT was the promise that virtualization would increase the group’s flexibility and responsiveness to the needs of the business. “We want IT to enable everything that the business might want to do, and that requires a very agile IT infrastructure,” says Bulkin.

To that end, Cisco IT formed a virtualization team composed primarily of system administrators from all of IT’s hosting groups. Gathering momentum for virtualization among application stakeholders was one of the biggest hurdles for Cisco IT. There was pushback from internal clients who had grown accustomed to dedicated servers supporting each of their applications. Many of these clients requested assurance that virtualization would not undermine their service-level objectives for application speed and reliability.

“We had to do a lot of communication,” says Mike Matthews, IT program manager at Cisco, “especially before we had a track record with virtualization and could show people data on how quickly we could provision their applications and how well applications run on virtualized servers. Being able to explain how virtualization cut costs for the company also helped.”

Crucial to these communication efforts was top-down support from not only IT upper management but management elsewhere in the company. In Cisco’s case, virtualization aligned neatly with strategic corporate initiatives focused on service agility and operational efficiencies in the data center.

Cisco IT’s server virtualization architecture would be straightforward: Cisco Catalyst® 6500 Series switches equipped with the Content Switching Module (CSM), and storage-area networks (SANs) based on the Cisco MDS 9500 Series Multilayer Director Switch to connect servers, storage devices, and other data center systems across the network.

The virtualization team had to develop a standard method of handling server virtualization and provisioning that would be applied consistently throughout the enterprise. A requirement is third-party virtualization software, and Cisco IT selected VMware’s ESX Server offering. VMware supports the creation of virtualized servers, each potentially using multiple CPUs and varying gigabytes of memory. The number of CPUs and amount of memory can be modified as applications grow, and virtualized servers can be relocated among the physical servers to accommodate fluctuating demands for computing resources, unexpected incidents, and planned downtime.

Cisco IT has deployed VMware in server farms that group four, eight, twelve, or sixteen physical servers. The server farms provide flexibility for distributing traffic loads and redundancy if a physical server fails.

Applications run on both the Windows and Linux operating systems, which co-exist on the same physical server running different virtual machines. The virtualized servers can run whichever operating system an application needs. A typical physical server supports between 10 and 20 virtual machines running applications.

Not all servers are suitable candidates for virtualization, and not all applications are suited to deployment on virtualized servers. Cisco IT determined that slightly more than half of its existing servers were viable candidates for reconfiguration as virtualized servers. Among the applications that might not operate on a virtualized server include those that use specialized devices (e.g., a dongle or software license key) and applications that require massive amounts of memory. Likewise, physical servers that require more than two CPUs or substantial memory and disk space, and servers that already have high CPU utilization levels, are not good candidates for virtualization with VMware.

Initially Cisco IT focused on providing virtualized servers for new applications. As the virtualization effort formalized, the group realized that migrating existing applications to virtualized servers is equally important, and requires collaboration with internal clients as well as vendors and partners. “These migrations are more challenging, but the paybacks are greater,” Bulkin says, “even though it can be harder to convince someone whose application is already doing just fine to make the switch.”

From a security standpoint, Bulkin emphasizes that a virtualized server is not that different from a physical server. “If an enterprise employs well-considered security policies and embeds security throughout its network, a virtualized server has all of the protection of a dedicated physical server, just as a VLAN [virtual LAN] is as secure as its hardwired counterpart,” he says.

Within a few months, Cisco IT started to reap the benefits of virtualization. Chief among these benefits:

• Reduced demand for data center resources. With fewer physical servers to install, Cisco IT
  can slow the demand for data center space even as the number of deployed server instances continues to grow. Fewer physical servers also eases the demand for burdensomely expensive power, air conditioning, floor space, hardware and software licensing costs, and backup resources in the data centers.
• Faster deployment of new servers. Delivering server capacity in response to a new request now takes a few days or less (many are done in under two hours) versus 12 weeks or more. In a recent quarter, 81 percent of the servers that Cisco IT deployed were delivered in less than three days, including the internal IT governance process.
  
  “Without a strong governance process, it is easy to find yourself with an unnecessary proliferation of virtualized servers because they are so easy and fast to deploy,” Bulkin says. Ultimately, Cisco IT wants to deliver 99 percent of all virtualized servers in less than 24 hours.
• Significant cost avoidance and savings. Of the roughly 2000 virtualized servers that Cisco IT currently maintains, about 70 percent were new deployments, and 30 percent existing physical servers that were reconfigured as virtualized servers to support more applications. Cisco IT estimates that this pool of virtualized servers produces cost avoidance and savings valued at a cumulative total of nearly US$16 million (calculated in the latter part of 2007).
  
  A physical server costs Cisco IT approximately US$11,000 to deploy for the server and software licensing costs alone (other costs, such as storage, are not factored in). Compare this to the average cost of a virtual machine. If, for example, 10 virtual machines are deployed on a single physical server, the average cost per virtual machine would be US$1100.
  
  “Every virtualized server represents one that we did not have to buy,” Bulkin says. “That is a lot of cost avoidance. Every application that we migrate onto a virtualized server represents savings in floor space, a reduction in hardware and associated maintenance, and a lower demand for power and cooling. We get both cost avoidance and clear savings.”
• Greater efficiency and flexibility of server resources. Virtualization has boosted server utilization manyfold. Among Cisco IT’s 127 physical servers, CPU utilization has climbed to 65 percent compared to roughly 8 percent in 2005.
  
  Virtualization is also fulfilling its promise of flexibility for the group, who can now organize and allocate server resources with great efficiency.
  
  Take the example of a tool used by Cisco’s human resources department and managers across the company. Formerly 22 physical servers were devoted to this application— most of them sitting idle except during yearly performance review cycles. Now this application is allocated the capacity that it needs only when salary adjustments are being processed and the tool is heavily used by managers. The rest of the time those 22 servers are freed up to support other applications. Then, too, there can be physical servers partitioned into 10 or more virtualized ones.
  
  “The human resources tool is an excellent example of the essential principle of virtualization,” says Bulkin. “The application, its usage pattern, and the operating system are most important, not the hardware that they run on.”

While benefiting from the immediate operational and efficiency advantages of server virtualization, Cisco IT is solidly on track toward reaping the longer-term benefits virtualization can bring, including deploying application services and provisioning server, storage, and network devices even faster. Integral to this track is an organizationally-supported shift in IT mindset. Server, storage, and computing devices must be treated as integrated, interdependent pools of resources in the data center.

Construction of a new Cisco data center in Richardson, Texas, is under way. It will eventually consolidate all of the company’s North American operations and be the first fully operational facility to adopt Cisco IT’s service-oriented data center model. Virtualization—of servers, storage, and network-based services—will be a cornerstone of this new facility.

Cisco IT has already started to segment its virtualized environment into pods, discrete amounts of server, storage, and computing capacity managed as a single entity. In Richardson and Cisco’s development data centers in San Jose, California, each pod will be managed together in a virtualized data center.

Successful integration in the data center also hinges on close collaboration among server, storage, and network staff.

“Each of the three areas of virtualization has its own level of maturity in terms of technology, and each has its own specific services and IT skills set,” Bulkin says. “In the future, we envision the potential to redefine roles, including that of a data center engineer who has a stronger breadth and depth of knowledge in all three functional areas. Increased collaboration skills might also be needed to bring these groups together.”

For Cisco IT, cross-functional integration within the data center provides the necessary means to an important end. According to Bulkin, “The overall goal is to move from running an IT infrastructure to delivering infrastructure as a service. Our implementation of virtualization and the service-oriented data center are enabling us to do that.”