Cisco on Cisco

Cisco has used Cisco Unity globally since 2004, and currently has approximately 72,000 voice mailboxes. “Voicemail remains the preferred form of messaging at Cisco,” says David Neustedter, IT network engineer.

Cisco’s Scientific Atlanta division began using Cisco Unity 7.0 in 2008, even before the first customer shipment. But most of the rest of the company was still using Cisco Unity 4.0. Cisco wanted to upgrade all global offices to the newest version to begin taking advantage of new features for anytime, anywhere collaboration and enhanced security.

One Unified Presence server cluster is deployed for each Unified Communications Manager cluster. Each server has a theoretical maximum of 5000 users, so Cisco IT deployed from two to six servers in each cluster, depending on the number of devices supported. “For clusters supporting 8000 or fewer users, we set up two servers with load balancing and failover,” says Chris Hartley, IT network engineer for unified communications, Cisco. “If one server goes down, then the client fails over to the second server.”

All Cisco Unified Presence servers were installed by June 2008, at about the same time that the Unified Communications Manager 6.1.1 upgrade was complete. They were initially installed with an early field trial (EFT) version of Cisco Unified Presence 7.0, which was upgraded in early September 2008 to the FCS version of Cisco Unified Presence 7.0 software (available as of late August 2008).

For example, new security features in Cisco Unity 7.0 would enforce Cisco’s retention policy for voicemail messages that were saved on employee PCs and laptops. Although integrated messaging was available in earlier Cisco Unity releases, Cisco IT had not yet enabled it because Cisco needed a way to help ensure that voicemail messages would not be retained in employees’ inboxes or hard drives after 30 days.

The challenge was how to plan the upgrade to minimize disruption to a mission-critical service. Cisco IT wanted to keep outages to no more than 30 minutes during nonbusiness hours, so that productivity and customer service would not be affected.

Cisco IT managed the upgrade with minimal disruption by prestaging the new server the week before upgrade, and then performed a live handoff to the new software during nonbusiness hours. The IT group completed the hardware a few servers at a time throughout 2007. “We performed the upgrade gradually instead of all at once so that we wouldn’t need to dedicate a large IT team to the effort,” Neustedter says. The team completed the upgrade in September 2008. Cisco IT managed the upgrade with minimal disruption by prestaging the new server the week before upgrade, and then performed a live handoff to the new software during nonbusiness hours. The IT group completed the hardware a few servers at a time throughout 2007. “We performed the upgrade gradually instead of all at once so that we wouldn’t need to dedicate a large IT team to the effort,” Neustedter says. The team completed the upgrade in September 2008.

The IBM servers that Cisco used for previous versions of Cisco Unity were nearing end of life, so Cisco IT decided to deploy Cisco Unity 7.0 on Cisco MCS 7845 H2 Media Convergence Servers. These servers have an onboard Redundant Array of Independent Disks (RAID), providing the high availability that Cisco needs for mission-critical messaging. The Cisco MCS platform also conserves data center space, cooling, and power because it uses only two rack units for each server, compared to seven rack units for the old servers.

Cisco maintains 87 Cisco Unity Unified Messaging servers in 11 global data centers. The week before a data center was scheduled for upgrade, a Cisco IT engineer prestaged the new Cisco MCS 7845 Server platforms by loading Windows 2003 Server and Structured Query Language (SQL) Server. After the hardware was ready, Cisco IT performed a live handoff upgrade during nonbusiness hours, as follows:

• Took the existing primary server offline, made the secondary active, and disabled failover.
• Installed Cisco Unity Unified Messaging 4.2.1 software (the previous version) on the new server. This took approximately 30 minutes.
• Restored the core database using Cisco Unity Disaster Recovery Tools (DRT), which took approximately 20 minutes. Cisco IT terminated the DRT syncher before it began synchronizing user accounts.
• Renamed the server. Depending on the location, Cisco IT assigned it the same IP address as the old primary server or changed the Domain Name System (DNS) entry.
• Executed a custom script to confirm that the SQL Server had correctly stored the server name. This step prevented it from storing two instances of the same server, a common problem.
• Brought the new server online as the primary server, connecting it over Gigabit Ethernet.
• Activated the new server and pointed all ports back to it.
• Repeated the same process for the secondary server. It is not necessary to perform a database restore for the secondary server.

The entire process took two to four hours per server, including prestaging the hardware. “Users were without voicemail for no more than 30 minutes, and usually just 5 to 15 minutes,” says Neustedter.

After deploying the new hardware platform, Cisco IT upgraded to Cisco Unity 7.0, as follows:

• Failed over the primary server to the secondary server.
• Disabled automatic failover/failback.
• Configured Cisco Unified Communications Manager so that the hunt list sent calls to the secondary server first.
• Upgraded the primary server, now offline, to Cisco Unity 7.0.
• Made the primary server active again, and configured Cisco Unified Communications Manager to route calls to the upgraded primary server first.
• Repeated the process for the secondary server.
• Re-enabled failover for both servers.
• Performed thorough testing (see next section).
• Updated configuration settings to take advantage of the new features in Cisco Unity 7.0 (see “Benefits for Cisco Employees”).

Typically, the software upgrade took approximately three hours: 30 to 45 minutes for the upgrade itself plus 45 minutes for testing, on both the primary and backup server. The IT person did not have to be present after starting the upgrade process.

Cisco IT performed the following tests immediately after reactivating the primary server:

• Logged in to a test mailbox to retrieve new or saved messages, send a message, and reply to a message.
• Called into different Cisco Unity servers from a nonsubscriber extension to check cross-server and cross-pod transfers.
• Checked Simple Network Management Protocol (SNMP) service on the primary server.
• Checked user accounts by logging in to a test account on the upgraded Cisco Unity system.

Cisco’s voicemail retention policy requires messages to be deleted after 30 days, which is easy to accomplish when all voicemail is stored on central Microsoft Exchange servers. Enforcing the policy would become more challenging, however, when Cisco IT enabled a Cisco Unity Unified Messaging feature that allows employees to access voicemail messages from an email inbox on PCs and laptops and save them to a hard drive. Therefore, Cisco IT previously blocked PCs from accessing the company’s Microsoft Exchange servers using common email protocols.

Now, new security features in Cisco Unity 7.0 automatically enforce Cisco’s retention policy so that the company can confidently allow employees to download voicemail messages to their email inboxes using Cisco Unified Personal Communicator (Figure 1). “Unity Unified Messaging 7.0 uses public-key encryption to help ensure that messages cannot be intercepted and read,” says Neustedter. “We set the encryption keys to expire after 30 days so that messages become unplayable after that time, even if employees retain them in their email inbox or copy them to their hard drive.”

Cisco Unity 7.0 benefits employees and the Cisco IT department.

New features in Cisco Unity 7.0 save time for Cisco employees who use voicemail extensively to stay in touch with customers and collaborate with coworkers. New features of the Cisco Unity 7.0 release that are especially useful for Cisco include:

• Integrated messaging. Employees save time every day by checking voicemail and email messages from one interface instead of two separate interfaces. Although integrated messaging was available before, the public-key encryption capabilities in Cisco Unity 7.0 enabled Cisco to adopt it for the first time.
• Dropped call recovery. Previously, if a cell phone connection dropped while an employee was recording or listening to an email, the employee had to start all over again after calling back. Now, if employees call back immediately after being disconnected, they hear “Welcome back” and are given the option to finish sending the message or listening to the message. The message begins 10 seconds before where it was when the connection dropped, an especially big time saver with long messages.
• Less time to address messages. Instead of spelling the recipient’s complete name with phone keys, employees can stop as soon as Cisco Unity 7.0 lets them know that it has found a unique match. And if numerous employees’ names match the entry, the system begins with the names the employee has selected before instead of saying the names in alphabetical order.
• Easier log in. In the previous version, employees who mistyped their passwords had to reenter their user ID, phone number, and password. Now they can just reenter their password.
• Alternate extension detection. If an employee regularly calls Cisco Unity 7.0 from a number other than the desk phone, the system asks whether it should add the number as an alternate extension. If employees answer yes, then Cisco Unity 7.0 recognizes them whenever they call from that number, saving them the time to enter their office number.
• Remembered message playback speed. Employees can press phone keys to speed up or slow down message playback and prompts. Cisco Unity 7.0 remembers the previously selected speed whenever employees dial in.
• Brief prompts. Employees can elect to hear brief prompts instead of longer, more descriptive prompts, another time saver.

Enhancements in Cisco Unity 7.0 also reduce Cisco IT’s workload and simplify the data center:

• Reduced support requirements. Cisco Unity 7.0 supports 10,000 subscribers on a single pod, double the number supported by the previous version. “This has reduced by half the number of servers that we need to support, power, and cool,” says Jon Heaton, IT architect. “The higher capacity of Cisco Unity 7.0 means that we can remove five of eight Unity server pods in Amsterdam; three of four in Boxborough, Massachusetts; and three of six in Research Triangle Park, North Carolina.” The number of Exchange servers will remain the same, so the ratio of Cisco Unity servers to Exchange servers has decreased from 1:1 to 1:4.
• Smaller footprint. Cisco data center space is at a premium, and the new Cisco MCS server uses only two rack units, compared to seven for the previous server. “The upgrade to Unity Unified Messaging 7.0 reduced our data center space by 75 percent, and also reduced power usage and support costs,” says Neustedter.

Cisco IT plans the following actions to gain even more business value from Cisco Unity:

• Begin using the voice user interface (VUI). Cisco currently does not use the VUI because of the effort required to tune the system for the diverse dialects and accents heard throughout Cisco’s global offices. The company plans to begin using the VUI when it upgrades to Cisco Unity Unified Messaging 8.0, which will have more tuning controls.
• Deploy Visual Voicemail, a widget that employees can use to view, listen, and respond to Cisco Unity messages from the Cisco Unified IP phone display, without even having to dial into their voice mailbox (Figure 2).
• Deploy Cisco Unity Unified Messaging 8.0 and Cisco Unity Connection 8.0. “Our goal is to be our own first customer for all future versions of Unity and Unity Connect, to gain the productivity advantages as soon as possible,” says Heaton.

Cisco IT offers the following advice for other enterprises planning to upgrade to Cisco Unity 7.0:

• Plan ahead to minimize impact on employees. Cisco IT prestaged the servers with all software other than Cisco Unity 7.0 so that the actual outage was no more than 30 minutes.
• Expose many people on the IT team to the patches and applications so that they can help out. “It’s a good idea to practice beforehand to identify potential mistakes,” says Heaton.
• If possible, perform the upgrade gradually, region by region: This approach minimizes risk.
• Develop and follow a detailed test plan. Typical errors include not updating all user accounts and not noticing that the service didn’t restart on the primary server after the secondary server was upgraded. “Verify all functionality, not just the core functionality,” Neustedter says.