Cisco on Cisco

Cisco currently has field sales offices in the following countries in the Middle East: Bahrain, Egypt, Jordan, Kuwait, Lebanon, Pakistan, Saudi Arabia, United Arab Emirates, and Qatar. These countries are part of Emerging Markets, one of Cisco's fastest growing revenue sources.

Until 2008, Cisco's WAN in the Middle East had only a fraction of the bandwidth available in other regions. This hindered competitiveness, for the following reasons.

Decreased Productivity

As Cisco has added employees to serve the rapidly growing Middle East market, limited bandwidth became a bigger problem. "We had invested in a high-speed WAN for the Dubai office, but employees in our other offices in the region had so little bandwidth that they struggled to use even the most basic productivity applications, such as email and calendaring," says Troy De Jong, IT project manager, Cisco. For example, in Cisco's Islamabad office, 40 employees shared 512 Kbps. In Riyadh, 200 employees had to share a 2 Mbps (E1) line. When customers visited an office for a demonstration, the country manager sometimes requested that employees work from home to conserve bandwidth.

High Voice Toll Charges

Most Middle East and Africa offices lacked sufficient bandwidth for VoIP. Therefore, Cisco phone bills in the Middle East were significantly higher than in other regions. "Typical annual toll charges for the
80-person Cairo office were US$360,000, or double the charge for the 230-person office in
Johannesburg, South Africa," says De Jong.

Inability to Use Advanced Collaboration Tools

Cisco strongly encourages employees to conduct face-to-face meetings with each other and customers using Cisco TelePresence, Cisco Unified Videoconferencing, and Cisco WebEx™ MeetingCenter to reduce travel time, costs, and environmental impact. However, employees in the Middle East could not take advantage of these advanced services because of low WAN bandwidth and high intraregional latency that exceeded the service-level agreement, or SLA (Figure 1). Cisco TelePresence and high-quality VoIP require round-trip latency of less than 300 milliseconds, and one-way latency of less than 150 milliseconds. Low latency is essential for the interaction to have a natural, in-person quality, where participants can interrupt but do not inadvertently talk over each other.

"The network had not been architected to support interactions within the region," says Roel Bernaerts, lead network architect, Cisco. "Therefore, even locations that were physically close, such as Amman and Beirut, were unable to use rich-media collaboration tools."

Opportunity to Provide a Backup Route from Asia to Europe

Cisco is also growing very quickly in India and China. The Cisco WAN in Asia connects eastward to North America over several WAN links. However, only a single lower bandwidth WAN link connected Asia and Europe. That link acted as a backup link if the primary WAN links failed during an earthquake or other event. Cisco IT wanted to improve the capacity and resiliency of the company’s global WAN backbone by providing an alternate route between India and Europe, through the Middle East.

"Additional connectivity between Europe and India, with a landing point in the Middle East, would also reduce intraregional latency," says Bernaerts.

IT projects in the Middle East require very long lead times. Cisco IT began defining the strategy for the Middle East WAN rearchitecture in June 2006 and issued a Request for Proposal (RFP) to global and preferred service providers in March 2007. The first circuit orders were placed in November 2007, and the network went into production just eight months later.

Requirements

During the planning phase, Cisco IT established the following requirements for the WAN rearchitecture:

• Quality of service (QoS) and multicast support. Some service providers in the Middle East and Africa cannot offer as many class of service (CoS) types as are available in other parts of the world, and also limit the number of multicast routes. Cisco requested that providers support at least four CoS and up to 3000 multicast routes.
• Redundancy and diversity. Preferably, each office would have two access paths, for redundancy and diversity. Path cuts are common in the region, as a result of acts of nature or construction. Therefore, Cisco IT specified that cabling paths should remain diverse up to the terminating equipment. Cisco IT made exceptions in two locations where the carrier could provide only one access path.
• Bandwidth. Where available and not cost-prohibitive, the smallest offices would receive a pair of E1 circuits (2048 Kbps upstream and downstream). Service providers would need to commit to usable bandwidth, the actual throughput for IP traffic, and not include bandwidth used for the encapsulation header. For example, sites that received only a single E1 circuit would use an Internet VPN for backup.
• Compliance with local and regional regulations. Cisco IT researched the complex local and country laws. Some countries currently prohibit VoIP for off-net calls because they do not want to decrease revenue for the service providers. However, some allow VoIP for on-net calls, such as calls between Cisco offices. "This discovery changed the entire design of the Cisco WAN in the Middle East," says Kees Gerritsen, lead voice architect, Cisco. (See "Selecting the Middle East Internet Data Center Location.")
• Service-level agreements. Cisco requires the SLAs shown in Table 1. "The availability SLAs might seem aggressive, but they do not include hours that offices are not open," says Jim Skilton, network operations engineer, Cisco. "This gives service providers ample opportunity to perform scheduled maintenance."

Table 1. Cisco IT Required SLAs for Carrier Availability, Carrier Packet Loss, and Jitter

Service Attribute	SLA
Carrier Availability QoS for IP VPN core Core sites with dual access Hub and engineering sites with dual access Satellite and partner sites with dual access Satellite and partner sites with single access	100% 99.99% 99.99% 99.95% 99.5%
Carrier Packet Loss	<0.1% for every QoS class
Jitter	15 ms for video and voice class

Selecting the Middle East Internet Data Center Location

When selecting the Internet Data Center (IDC) location to which all other Middle East offices would connect, Cisco considered United Arab Emirates, Turkey, Greece, Egypt, and Saudi Arabia. At first, Cisco IT selected Dubai, based on recommendations from carriers and other large companies in the region. But the plan changed when Cisco IT discovered that Dubai currently does not permit VoIP for off-net calls. Therefore, Cisco chose Manama, Bahrain.

"Bahrain's carrier offered significantly lower pricing for the global WAN, the Middle East regional WAN, and PSTN [public switched telephone network] connectivity, and imposed no restrictions on VoIP," says Bernaerts. In addition, the carrier’s IDC hosting facilities and cable infrastructure meet Cisco’s requirements. Latency is slightly higher than it would be if the IDC were located in Dubai, but the other advantages outweigh this fact.

Regional WAN Connectivity

"We were able to use our global WAN standards without modification," says Bernaerts. Cisco IT designed the Middle East regional WAN as a separate Multiprotocol Label Switching (MPLS) network. It connects to Cisco’s global WAN backbone through the IDC in Bahrain (Figure 2). Cisco IT selected MPLS instead of International Private Leased Circuit (IPLC) for regional WAN connectivity because of its attractive pricing, reduced operational complexity, scalability, and superior proactive and reactive monitoring.

Intraregional traffic is routed over the global backbone links connecting the Middle East and Europe. "We do not use MPLS as a transit network to avoid complicated failover between regional backbones," says Bernaerts.

The Middle East MPLS network connects to the Bahrain IDC with dual OC-3/STM-1 circuits. In addition, the global Cisco backbone network connects Asia to Europe using two separate circuits, for diversity. The Bahrain IDC connects Bangalore to the Amsterdam IDC with an OC-12/STM-4 circuit, providing a diverse and highly available connection to the Amsterdam Internet point of presence (POP) and data center. Another OC-12/STM-4 circuit connects Hong Kong to London, providing a second high-capacity path between Asia and Europe. This path supports low-latency communication between the two regions.

As shown in Figure 2, the two paths between Europe and Asia provide geographical diversity. One path is a terrestrial Europe Asia cable that connects through the Nordic countries and Russia. The other is a submarine cable that connects through the Mediterranean Sea and the Persian Gulf.

Global Connectivity

Cisco IT used the Middle East WAN rearchitecture project as an opportunity to provide a more direct path between Asia and Europe, reducing latency. The goal was to route traffic from Europe to India through Bahrain. Initially, Cisco IT had planned to lease high-capacity circuits with automatic protection, which is the ability to switch immediately to a hot spare circuit of equal bandwidth if the first circuit should fail. However, end-to-end automatic protection as a service requires reserving and paying for bandwidth on both the primary and the backup cable system, which is very expensive in the Middle East. Therefore, Cisco IT provisioned the second leased line between Hong Kong and London to provide Layer 3 diversity between Europe and the Middle East (Figure 3). If a submarine cable connecting Bahrain to London fails, traffic flows from Bahrain through India to Hong Kong, and then back to London. Failover takes only a few milliseconds.

Capacity Planning and WAN Sizing

Cisco typically allocates 56 Kbps per user in its field sales offices. For the new Middle East WAN, Cisco IT decided to allocate 68 Kbps, to accommodate a general increase in bandwidth usage as well as new technologies such as Cisco Unified Video Advantage. Bandwidth in the Middle East is very expensive, however, so Cisco IT wanted to avoid overprovisioning. At first, Cisco IT considered allocating bandwidth based on the percentage of time that employees typically spend in the office. For example, a 10-person office where employees spent 50 percent of time in their office would receive 0.5 x (10 x 68 K). In the end, Cisco IT decided to provide the full bandwidth allocation per person to help ensure full productivity during periods when all employees work in the office, such as fiscal quarter end.

When calculating bandwidth requirements for each office, Cisco IT added bandwidth for applications that apply to the entire office, such as Cisco TelePresence, content caching, and network services.

Interim Connectivity

Circuit delivery lead times in the Emerging Markets region generally range from several months to a full year. When Cisco adds a new office in this region, Cisco IT usually provisions an IP Security (IPsec) VPN link over the Internet to act as a temporary WAN link during the waiting period. VPN is preferable to satellite because it has lower latency and does not require dish installation costs and legal and customs complications. To avoid routing complexity, VPN backup circuits for each office terminate in the Bahrain IDC, the same location as the primary connection, and then travel across the backbone WAN to enter the Internet through Cisco IT’s European Internet POP in Amsterdam.

"We are still investigating opportunities to deploy a regional Internet POP to further reduce latency on our Internet VPN backup circuits," says Bernaerts.

Office Topology

Middle East and Africa offices connect to the WAN using one of three topologies, depending on bandwidth requirements and whether the carrier can provide diversity (Figure 4). If diversity is not available, VPN is used for backup connectivity. In regions with exceptionally high bandwidth costs, offices receive a single high-bandwidth circuit for rich-media collaboration tools and a backup solution with lower bandwidth to support critical applications, such as email and calendaring, if the primary connection fails. Offices with 30 or fewer people receive an E1 connection, and offices with more than 30 people receive an E3 connection.

All offices in the region, except those in Lebanon, Morocco, and Algeria, have physically diverse connections, and the Morocco and Algeria offices will have full diversity by August 2009. "Providing diversity required close collaboration with service providers," says De Jong. "We were very involved in detail work that we ordinarily would not get involved with, such as working with civil engineering teams to add a second duct into the opposite side of buildings that had only one circuit entry path. This extended lead times, but was essential to achieve the needed level of business continuity."

Load Balancing

Cisco IT load balances total bandwidth between two WAN gateways whenever circuit diversity is available, to maximize the return on investment from the available bandwidth. Otherwise, offices connect over a single WAN link and use International VPN (IVPN) for backup.

Routing Platform

Cisco IT used the WAN rearchitecture project as an opportunity to replace depreciated equipment, including Cisco 3640 Routers, Cisco Catalyst® Switches, and Cisco Aironet® Wireless Access Points. The new standard platform is the Cisco 3845 Integrated Services Router, which also acts as a H323 voice gateway.

Deployment Process

For the WAN rearchitecture project, Cisco IT followed its usual strategy of keeping core processes in-house and outsourcing contextual processes. Examples of contextual processes that Cisco outsources to the service provider include proactive link monitoring, capacity reports, and CoS reporting. "Language differences are a concern in the Middle East region, and in many locations support is only available in the local language," says Bernaerts. "Therefore, outsourcing support significantly reduces language challenges and the related operational overhead."

Testing

Before placing the WAN into production, Cisco performed the testing shown in Table 2.

Table 2. Cisco Thoroughly Tested the Middle East WAN

Type of Test	How Cisco IT Performed the Test
Core, hub, and engineering sites	Performed out-of-service throughput testing at the contracted rate, over a 24-hour period. The contracted rate is for raw IP bandwidth to the end location and does not include additional bandwidth for encapsulation headers.
Field sales offices	The service provider, not the local tail provider, certified circuits as clean. Cisco required the service provider to provide a detailed report of testing results. Throughput testing at field sales offices was not mandatory.
CoS offerings and transparency	During the cutover, Cisco and the service provider confirmed that the network honored CoS markings, performing to the contracted service levels. After implementation, the servicer provider began using Cisco IOS™ IP SLA to measure latency and jitter, proving that link efficiency mechanisms are working.
Multicast	Cisco IT started the internal Cisco IPTV viewer to confirm that content would appear. Cisco tested the maximum transmission unit (MTU), or largest size packet that could be sent over the network, to confirm that service provider could meet its agreements. Capacity testing of multicast streams was not required.
Routing	The Cisco VPN needs the ability to handle at least 3000 unicast routes. Cisco IT injected and removed routes from a VPN, recording the time required. This indicated how well the VPN could converge to new routes, which is critical when links fail.

 

The new E1 and E3 WAN access links provide a total of 140 Mbps bandwidth in the Middle East, an 800 percent increase over the previous design. The WAN rearchitecture project immediately increased productivity, began reducing telephony costs, and increased global backbone resiliency. It also provides the growing number of Cisco employees in the Middle East with the capability to use advanced collaboration tools and Cisco TelePresence, giving Cisco’s Middle East offices a competitive advantage.

Increased Productivity and Employee Satisfaction

Employees in the Middle East now experience the same fast performance for enterprise applications as their global counterparts. The WAN is meeting or exceeding requirements in all locations, and Cisco IT rarely receives complaints about network performance. "The network rearchitecture helps Cisco employees in the Middle East be productive and will let them use rich-media collaboration tools to reduce travel," says De Jong. "The timing is excellent because of current economic and environmental concerns."

Lower Voice Costs

Cisco has reduced its long-distance office telephony costs by eliminating toll charges for intracompany calls. Costs in Dubai decreased by 60 percent, and Cisco anticipates the same savings in other offices. "We expect to reduce our annual spending in our 80-person Cairo office from US$360,000 to $140,000," says De Jong. "The savings will increase when we enable WebEx MeetingCenter for conferencing."

Currently, Cisco can use VoIP only for calls to other Cisco offices, not because of network limitations, but because of legal regulations. For example, employees in the Morocco office can use Cisco IP Communicator or Cisco Unified Personal Communicator because they are connecting over Cisco's private WAN. However, tail-end hop off is not permitted in several countries, including United Arab Emirates, Saudi Arabia, Pakistan, and Kuwait. Therefore, employees in those countries currently cannot use a softphone at home. Cost savings will increase if regional carriers relax restrictions on VoIP. Cisco is actively working with regulators in Middle East countries.

Support for Cisco TelePresence

Previously, Cisco employees in the Middle East had to travel to Dubai to use Cisco TelePresence, enabling them to meet face-to-face with coworkers in other Cisco locations without the time, expense, and environmental impact of lengthier travel. Now Cisco TelePresence is available in Saudi Arabia, as well, and will become available in other offices after Cisco works out the legal issues in each country.

Sales Support

Sales engineers can now use and demonstrate Cisco unified communications technologies and Cisco TelePresence in a production setting. They can also invite customers to meet face to face with Cisco experts in any global location, with Cisco TelePresence, Cisco WebEx, or Cisco Unified MeetingPlace. Sales teams expect rich conferencing to help them close more sales, sooner.

Increased Resiliency

All Cisco offices in the Middle East and Africa, except those in Lebanon, Morocco, and Algeria, now have physically diverse network connections, supporting business continuance. In addition, Cisco now has a true global backbone, with connectivity from India to Europe and from Hong Kong to London, which can be used if earthquakes or other events take down the primary route. Cisco’s global backbone route from India to Europe is business critical because of Cisco’s growing presence in India.

"We have full diversity around the globe, which improves business resilience and also gives us the capacity for point-to-point Cisco and Intercompany Cisco TelePresence," says Bernaerts. In December 2008, an earthquake in the Mediterranean took down both circuits to Europe. Cisco employees still had global connectivity because of the path to India and back through Hong Kong.

Cisco IT shares the following suggestions with other companies that are building or rearchitecting a WAN in the Middle East and Africa.

• Plan for long lead times for circuits. Cisco IT assumed that the regional service providers would need more time than they estimated for circuit delivery, and built this assumption into the timeline. But the delays exceeded even Cisco's estimates. "If we did it again, we would ask service providers to provide empirical evidence of circuit delivery times for previous customers, so that we could base our estimates on real-world experience," says Bernaerts.
• Be very detailed in technical specifications. Specifications for IDC cages, for example, must be more detailed in the Middle East than they are in Europe, simply because service providers have less experience. "Be sure that the service provider signs off on documentation so that they are accountable," says De Jong.
• When requesting nonstandard circuits for the region, ensure that the correct circuit is provisioned: Cisco had requested DS-3 circuit in Riyadh. Although the paperwork showed the circuit was a DS, it was provisioned as an E3, the in-country standard. The error was discovered only after extensive troubleshooting.
• Be aware of regional business culture. "What we requested and what the service provider delivered were different," says De Jong. "We had to invest a lot of resources into adjusting requirements and resetting stakeholder expectations." In one country, for example, the service provider told Cisco that diversity would be achieved with two cable paths, both connected to the node in United Arab Emirates. "The diversity capability did not exist, however, so we had to reroute over an alternative path," says De Jong. "That required additional time and effort for discovery and analysis and resubmitting the stakeholder proposal."
• Become familiar with each country's legal requirements for VoIP and encryption. Cisco IT continues to work with local governments to clarify its understanding of their requirements. The goal is to meet their requirements while enabling Cisco offices and Cisco customers in the country to do business more easily.
• Include Cisco router pin layouts on technical specifications. Pin layouts are not usually included on technical specifications for local carriers. However, many local carriers did not have experience patching through buildings, which created issues that delayed progress. Therefore, Cisco IT recommends including pin layouts to help technicians configure the cable correctly.

Cisco IT has found several advantages in using service provider IP VPN networks in Europe and the Middle East. One is acquiring more WAN bandwidth at lower prices than the equivalent leased line. Based on this experience, Cisco IT has begun plans to migrate its extensive North American WAN to a similar service provider MPLS VPN solution.

Cisco IT will connect Nairobi to the MPLS network as soon as the country's submarine cable system is operational.

Cisco continues to meet with regulatory groups in the Middle East and Africa to learn how Cisco can provide more of its standard voice and video and remote access service offerings to its employees in the region, while supporting each government’s requirements.