Cisco on Cisco

Until early 2005, the Cisco® San Jose, California campus was connected to the Internet through four separate OC-3 links from four different providers. When one provider announced that it would cease providing enterprise support to place greater emphasis on the wholesale market, Cisco IT was tasked with identifying and implementing a replacement circuit. In addition, the existing circuits were terminated on Cisco 7500 Series Routers, which could no longer serve the growing Internet traffic at Cisco or the need for new features.

In evaluating new circuits, the Cisco IT project team defined the following goals:

• Implement a replacement circuit, but with the ability to add capacity in small increments as needed
• Migrate from the Cisco 7500 Series Routers to a next-generation platform that met Cisco IT's requirements for throughput and security telemetry
• Increase link diversification by installing the new circuit at a different location on the San Jose campus
• Evaluate alternative local-loop technologies for their ability to meet service requirements at a lower cost than SONET/TDM circuits

Cisco IT also specified several requirements for the new circuit, including:

• Service quality. Traffic between the San Jose campus and the local point of presence (POP) should exhibit no more than 15 ms latency and 5 ms jitter.
• Routing efficiency. The service provider must have adequate peering for directly delivering traffic to Internet endpoints.
• Usability. The circuit should be completely transparent to end users, including employees, customers, suppliers, and other visitors to Cisco Websites.
• Bandwidth. The circuit must support initial bandwidth of 200 Mbps and allow incremental upgrades to 1000 Mbps.

For the replacement circuit, Cisco IT chose Metro Ethernet services at the network edge – a first at Cisco. Deployed in late 2005, the Metro Ethernet service is delivered by AT&T, the local service provider for the San Jose campus. “Metro Ethernet services have many desirable attributes for our use of the Internet, including greater capacity, decreased provisioning times, and a lower cost per Mbps than traditional SONET/TDM circuit offerings,” says JJ Kim, a Cisco network engineer.

AT&T delivers the Metro Ethernet circuit through the AT&T OPT-E-MAN service, which is an Ethernet-over-MPLS (EoMPLS) Layer 2 solution for VPNs. Within the AT&T network, Cisco 7609 Routers use interface cards that connect optical services modules (OSMs) to the Gigabit Ethernet WAN to deliver Multiprotocol Label Switching (MPLS).

The Metro Ethernet service connects to the Cisco site by dark fiber that is terminated with a Cisco 12404 Router at the Cisco premises (Figure 1). That router is deployed in the DMZ area of the Cisco IT core backbone network.

The AT&T side of the service is implemented over two dark fiber pairs. The Port Aggregation Protocol (PAgP) provides Layer 2 load balancing that protects the service from failures that can occur in a single-pair design. With the double-pair design, if a failure occurs on one fiber pair, traffic continues on the other pair.

The physical interface to the circuit is a Gigabit Ethernet connection. The logical interface uses the Border Gateway Protocol (BGP) to select paths for Internet-bound traffic.

AT&T has deployed a Cisco Catalyst 3550 Series Switch to police traffic received from Cisco to ensure it does not exceed the 200-Mbps circuit capacity. Cisco IT has also implemented quality of service (QoS) policies at the egress interface on the Cisco 12404 Router to shape the outbound traffic at 200 Mbps (Figure 2).

Cisco IT monitors the Cisco 12404 Router using standard systems for network management. Cisco NetFlow information obtained from the router is analyzed for security monitoring and traffic patterns. In addition, AT&T monitors and manages the Metro Ethernet circuit as part of the OPT-E-MAN service.

AT&T was an attractive provider in part because of the large number of Cisco employees who telecommute over AT&T DSL service. Given the growth of telecommuting over the Cisco VPN, AT&T is now the primary source and destination for Internet traffic at the Cisco San Jose campus.

Implementing the Metro Ethernet circuit has produced both business and technical benefits for Cisco.

Improved performance. Data tracked by Cisco IT in early 2006 showed an average response time of 2 to 3 ms for the AT&T peer, and the average circuit availability level was 99.98 percent.

"When we first considered deploying Metro Ethernet at the network edge, we were concerned that it wouldn't be as reliable as the existing circuits," says Kim. "However, the AT&T Metro Ethernet service has delivered excellent availability and reliability."

As of January 2006, the AT&T circuit carried, on average, 45 percent of the total traffic handled by the four Internet circuits at the San Jose campus. This traffic volume is more than twice the amount carried by the previous OC-3 circuit, primarily because of the number of local telecommuters who are using the AT&T DSL service for accessing the Cisco VPN over the Internet.

Cost savings. The Metro Ethernet service delivers greater bandwidth—200 Mbps compared to the 60 Mbps configured on the previous OC-3 circuit—at approximately the same monthly cost. Cisco IT projections show a need to increase Internet service capacity by 2007. According to Kim, the choice of Metro Ethernet is clear: "Continuing to use TDM circuits would force us to replace the OC-3 circuits with OC-12 circuits, for a quadruple increment in capacity. This is more than what Cisco requires, and it comes with a very large cost increase." In contrast, the Metro Ethernet service offers an incremental increase in bandwidth, which reduces the impact of higher costs for serving greater traffic.

Investment protection. Cisco IT experienced no problems migrating from the Cisco 7500 Series Router to the Cisco 12404 Router. "Metro Ethernet allows us to make incremental capacity increases without additional hardware cost or service disruption during capacity changes," says Kim. "This simplifies deployment and improves the return on investment and quality of service for the circuit."

Network resilience. Terminating the AT&T Metro Ethernet service at a separate building on the San Jose campus allows for physical diversification of the Internet links, as well as load sharing and fault tolerance. The Cisco 12404 Router also offers increased traffic capacity and superior performance, making it easier to endure distributed denial of service (DDoS) attacks.

Network compatibility. Because the AT&T Metro Ethernet service operates over a Cisco Powered Network, both parties benefit from equipment interoperability and network features.

The Cisco IT project team offers the following lessons for other enterprises that are considering Metro Ethernet services:

• Network traffic was added to the AT&T Metro Ethernet circuit in phases. The project team started with a small amount of inbound and outbound traffic by controlling the BGP routes and advertisement. When the reliability of the service was confirmed, the BGP route and advertisement controls were removed, and the AT&T circuit started to carry a substantial amount of network traffic.
• The volume of network traffic on the AT&T circuit has been consistently high, with the peak rate frequently going over 195 Mbps. Because this level is too close to the 200 Mbps limit, some traffic on the AT&T service was sent to the other Internet circuits by manipulating the interface bandwidth and delay on the Cisco 12404 Router. Traffic shaping by the Cisco 12404 Router prevents outbound traffic from exceeding the 200-Mbps limit.
• Identifying problem sources can be more difficult in a Metro Ethernet service. For example, a link failure between the Cisco and AT&T endpoints might not be apparent to network management systems at Cisco. The Cisco side of the interface will appear to be active even if the BGP peering (the Layer 3 connection) is down, so long as the last-mile fiber connection to the AT&T switch is active. To overcome this limitation, Cisco network administrators monitor both the Cisco interface and availability of the peer.
• Because so many Cisco employees use AT&T Internet services to access the Cisco VPN from home, the Metro Ethernet link provides more direct paths that yield better performance for users.

Based on the improved performance gained from the Cisco 12404 Router implemented at the San Jose campus, Cisco IT plans to deploy Cisco 12000 Series Routers in the company's major Internet access hubs worldwide. And as the other OC-3 circuits in San Jose reach capacity, Cisco IT will consider deploying Metro Ethernet service as a replacement.

For more information about Cisco 12000 Series Routers, visit www.cisco.com\go\12000. For Cisco Metro Ethernet solutions, visit www.cisco.com/go/metro.