A Standards-Based Approach for Offering a Managed Security Service in a Multivendor Network Environment

by Kunjal Trivedi, Cisco Systems and Damien Holloway, Juniper Networks

As transport becomes a commodity, service providers are seeking new revenue sources and new ways to differentiate themselves. Managed security services address a growing market because business customers are struggling to comply with regulatory requirements such as the Payment Card Industry-Data Storage Standards (PCI-DSS), the Sarbanes-Oxley Act, the Gramm-Leach Bliley Act, Health Insurance Portability and Accountability Act (HIPAA), Directive 2002/58/EC, and the Asia-Pacific Economic Cooperation-Organization for Economic Cooperation and Development (APEC-OECD) initiative on regulatory reform. Increasingly, business customers recognize that outsourcing network security is less costly than staffing with highly specialized security personnel who can provide 24-hour incident detection and response. Another incentive for outsourcing is to free existing IT resources to focus on the core business.

A standards-based approach helps service providers take best advantage of the managed security service opportunity because it increases the potential breadth and depth of the service offering. Multivendor solutions are becoming the norm when deploying services on an integrated backbone. Therefore, standards simplify deployment and management, helping control operational costs and accelerating time to market.

Service providers are experiencing a growing need for skilled engineers who understand multivendor environments—the motivation for conducting a multivendor security workshop at the 2006 Asia Pacific Regional Internet Conference on Operational Technologies (APRICOT 2006) [15], held in Perth, Australia, in February 2006. During the workshop (which was repeated again at APRICOT 2007 in Bali), participants successfully deployed and tested a multivendor service environment using IP Security (IPsec)-based Layer 3 Virtual Private Networks (VPNs) [1, 2, 3] over a Border Gateway Protocol/Multiprotocol Label Switching (BGP/MPLS) core [4].

Technical Challenges

To offer managed security services, service providers need the following:

• A secure network infrastructure, including tools and techniques for risk mitigation
• Technical solutions for the customer's business needs, such as VPNs based on BGP/MPLS, IPsec, or both
• Web-based reporting tools that business customers can use to monitor the security service in accordance with Service-Level Agreements (SLAs). Service providers can scale cost-effectively by offering customers a secure, Web-based portal that shows open trouble tickets, security incident-handling detail, SLAs, and access reports that customers need to comply with regulations.

An effective managed security service requires tools and techniques to address the following challenges:

• More sophisticated threats, and less time between vulnerability and exploitation: In addition to worms and viruses, the service provider needs to protect its own and its customers' networks against Denial-of-Service (DoS) attacks. Today's botnets can launch thousands or even a million bots that carry out outbound DoS attacks. New varieties of worms have side effects similar to those of DoS attacks. These threats can take down the service provider infrastructure, thereby violating SLAs and eroding revenue.
• A need for proactive rather than reactive threat response: Many service provider security groups are stuck in reactive mode. Every network device and security system produces voluminous event logs every day, and vendors use different formats. Therefore, identifying security incidents in order to react to them can take hours or days—or not happen at all. The connection between two separate events in different parts of the network can easily escape human detection, especially when the clues are buried among tens of thousands of harmless events that took place around the same time.
• Multivendor networks: Network security and reporting are easier to achieve in single-vendor networks. Realistically, however, many service providers and business customers have multivendor networks, sometimes because of mergers and acquisitions. Even if the service provider itself has a single-vendor network, some of its customers will use other vendors' equipment.
• Slow progress toward adopting IP Next-Generation Networks (IP NGNs): When service providers complete the migration to IP NGN, they will achieve greater control, visibility, and operational efficiency. Until then, service providers will incur higher costs and labor requirements for support and migration.
• A need to comply with industry standards from IETF and ITU: Standards facilitate security in multivendor networks. MPLS helps ensure infrastructure security, whereas IPsec provides secure connectivity among the customer's branches and remote offices. By using industry standards, the service provider can select best-of-class products based on performance, features, or cost.
• Scalability challenges: The security operations center for a managed services provider cannot cost-effectively scale to process several million events for each customer. However, it can scale to process a few security-incident trouble tickets. Scalability hinges on the ability to minimize false positives. Products such as Cisco Security Monitoring, Analysis and Response System (MARS), IBM Micromuse, and NetIQ provide analysis and correlation of events from multiple elements in the IT infrastructure. They process events using consolidation, filtering, normalization, enrichment, correlation, and analysis techniques, and also notify IT staff about critical events.

Infrastructure Security in Multivendor Environments

Securing the service provider infrastructure requires the following common best practices:

• Point protection
• Edge protection
• Remote-triggered black-hole protection
• Source-address validation on all customer traffic
• Control-plane protection
• Total visibility into network activity

Point Protection

Before offering a managed security service, providers need to protect the backbone; security operations center or network operations center; Authentication, Authorization, and Accounting (AAA) [10, 11] server; and remote-access networks. Securing individual network devices requires enforcing AAA, controlling the type of packets destined to network devices, and performing regular configuration audits to ensure that no unauthorized changes have been made. Best common practices include:

• Protect the backbone by locking down the vty and console ports: This protection helps prevent unauthorized access to network devices.
• Encrypt management commands that staff send to devices: Use of the Secure Shell (SSH) protocol helps prevent hackers from obtaining passwords that they could later use to compromise the network. Service providers that use out-of-band management for device configuration should also encrypt this management traffic and restrict access to authorized personnel.
• Deploy a AAA server: Using a AAA server is preferable to relying on local authorization on the devices themselves because it enables centralized policy control. The AAA server controls a user's access to the device, or even the specific commands that the user is authorized to execute.

It is strongly recommended that service providers use TACACS+ [13] authentication rather than Remote Authentication Dial-In User Service (RADIUS) [12] authentication. With RADIUS, traffic is sent in the clear between the AAA servers and network devices using the User Datagram Protocol (UDP), which defeats the use of SSH to encrypt logins and passwords. Open-source implementations of TACACS+ are available.

• Use one-time passwords (OTPs): To distribute one-time passwords, service providers can provide authorized users with a token card, soft token, or soft key. One-time passwords ensure that the user was authorized at the time of login, and was not an attacker who used a packet-sniffer program to intercept a password.
• Protect the AAA infrastructure from DoS attacks: Some service providers set up local accounts on routers and switches so that staff can log in if the AAA infrastructure is down, creating vulnerability. If the service provider does not secure management-plane access to the device, hackers can use SSH or Telnet and attempt a brute-force attack to crack the local account. The local account is often not as secure as an OTP because it is changed only once every 30 days, providing a longer window of opportunity for hackers to gain device access. It is strongly advised to not use default or easy-to-guess passwords. To prevent attacks against the AAA infrastructure, service providers should harden the infrastructure and consider placing the server behind a firewall with stateful inspection. Use Access Control Lists (ACLs), which are packet filters, on the firewall to restrict traffic between the AAA server and network devices only. Also be sure to distribute the AAA servers so that they do not create a single point of failure.
• Regularly audit device configurations: Frequently, the first indications of an attack, often unnoticed, are unauthorized commands executed on routers that change the configuration. An easy way to monitor configurations is using RANCID (Really Awesome New Cisco Config Differ) [14], a UNIX or Linux freeware tool that logs into each of the devices in the device table file, runs various show commands, processes the output, and sends e-mail messages reporting any differences from the previous collection to staff. RANCID works with routers from Cisco and other vendors. Another tool for auditing device configurations, the Router Audit Toolkit (RAT) assigns security scores to ACLs and other security best practices to show the relative security of routers.

Traditionally, service providers enforced policy at the process level, using vty ACLs, Simple Network Management Protocol (SNMP) ACLs, and others. Some service providers used ingress ACLs when possible. Today, it is far preferable to stop DoS traffic at ingress points: the peer edge, downstream and upstream routers, colocated network devices, and the customer access edge, enabling central policy enforcement and more granular protection schemes.

In addition, many network devices at the network edge have hardware acceleration, which provides far more robust resistance to attack than the process level.

Edge Protection

In many service provider networks, each core router is individually secured but still accessible to outsiders using SNMP or Telnet. Now service providers can supplement individual router protection with infrastructure protection that prevents undesired traffic from ever touching the infrastructure.

The following steps help protect the network edge (Figure 1):

1. Classify the required protocols that are sourced from outside the Autonomous System (AS) access core routers, such as external BGP (eBGP) peering, Generic Routing Encapsulation (GRE) [5], and IPsec. (Examples of nonrequired protocols are SNMP and Telnet.) Classification can be performed using a classification packet filter or Cisco NetFlow telemetry. The classification packet filter comprises a series of permit statements that provide insight into required protocols. Gradually narrow down the list, keeping in mind that very few protocols need access to infrastructure equipment, and even fewer are sourced from outside the autonomous system. Summarize the IP address space as much as possible, for simpler and shorter ACLs. Be cautious: just because certain types of traffic appear in a classification packet filter or NetFlow telemetry data does not mean they should be permitted to pass through to the routers.
2. Begin filtering. Use an infrastructure packet filter to permit only the required protocols to access infrastructure-only address blocks, denying all other protocols. It is important to monitor the packet filter entry counters, because a high volume of hits, whether or not a protocol has been identified as required, might signal an attack. To permit transit traffic, use the following as the final line of the Infrastructure ACL (iACL): permit ip any any, protecting the core network with a basic iACL that admits only the required protocols. Note that iACLs also provide antispoof filtering by denying access to the space from external sources, denying the RFC 1918 space [6], and denying multicast source addresses. RFC 3330 [7] defines special-use IPv4 addressing.
3. Further protect the core by identifying legitimate source addresses for the required protocols, such as external BGP peers and tunnel endpoints.
4. Deploy destination filters when possible.

Infrastructure packet filters at the edge of the network protecting the infrastructure are an effective first layer of defense. Service providers need additional forms of infrastructure protection for their older routers that do not support infrastructure packet filters and for packets that cannot be filtered with infrastructure packet filters.

Remote Triggered Black Hole Filtering

Remote Triggered Black Hole Filtering (RTBH) is among the most effective reaction and mitigation tools for DoS, Distributed DoS (DDoS), and backscatter tracebacks. It enables service providers to quickly drop DoS traffic at the network edge (Figure 2). Rather than sending commands to every router to drop DoS or other problem traffic, the service provider can deploy a trigger router that uses BGP to signal all other routers—just as fast as iBGP can update the network. In destination-based RTBH, all traffic headed to the destination under attack is dropped—the good traffic as well as the bad. In source-based RTBH, traffic from all or certain sources are blocked. The advantage of sourced-based RTBH is that service providers can whitelist certain addresses, such as the Network Operations Center (NOC) or route-name servers, so that they can continue providing services.

Source Address Validation on all Customer Traffic

Source address validation, defined in Best Current Practices (BCP) 38 [8], prevents service provider customers from spoofing traffic—that is, sending IP packets out to the Internet with a source address other then the address allocated to them by the service provider. Best practices from BCP 38 are to filter as close to the edge as possible, filter precisely, and filter both the source and destination address when possible.

Every access technology has antispoofing mechanisms derived from BCP 38:

• Packet filters
• Dynamic packet filters that are provisioned to be AAA profiles; when a customer signs in with RADIUS, a packet filter is set up for the customer
• Unicast Reverse Path Forwarding (URPF)
• IP Source Verify and DHCP Snooping (Metro Ethernet)

To gain operational confidence in BCP 38, service providers can take a phased approach—for example, implementing it first on one port, then on a line card, then on an entire router, and then on multiple routers.

Control-Plane Protection

Protecting the infrastructure control plane helps prevent an attacker from taking down a BGP session and thereby causing denial of service. The exploits a service provider needs to prevent include saturating the receive-path queues so that BGP times out, saturating the link so that the link protocols time out, dropping the Transmission Control Protocol (TCP) session, and dropping the Interior Gateway Protocol (IGP), which causes a recursive loop-up failure.

Following are techniques for control-plane protection.

• Generalized Time-to-Live (TTL) Security Mechanism (GTSM): This technique protects BGP peers from multihop attacks. Routers are configured to transmit their packets with a TTL of 255, and to reject all packets with a TTL lower than 254 or 253. Therefore, a device that is not connected between the routers cannot generate packets that either router will accept.
• Configuring routing authentication: The Message Digest Algorithm 5 (MD5) peer authentication feature instructs the router to certify the authenticity of its neighbors and the integrity of route updates. MD5 peer authentication can also prevent malformed packets from tearing down a peering session, and unauthorized devices from transmitting routing information. Be aware that MD5 peer authentication does not protect the router if an attacker compromises the router and begins generating bogus routing updates. Although it is not a panacea, MD5 peer authentication does raise the level of protection.
• Customer ingress prefix filtering: Prefix hijacking is an exploit in which a service provider customer announces an address space that belongs to another customer. The remedy is customer ingress prefix filtering, which enables service providers to accept only those customer prefixes that have been assigned or allocated to their downstream customers. For example, if a downstream customer has a 220.50.0.0/20 block, customers can announce this block only to their peers, and upstream peers accept this prefix only. Service providers can apply ingress prefix filtering to and from customers, peers, and upstream routers.

Visibility into Network Activity

To gain visibility into the network for early detection of security incidents, service providers can use open-source tools to analyze flow-based telemetry data, which is retrieved from routers and switches. Open-source tools for visibility into security incidents include RRDTool, FlowScan, Stager, and NTOP Remote Monitoring (RMON).

These tools provide information such as packets per second, bits per second, and traffic types. For example, RRDTool shows the number of Domain Name System (DNS) queries per second, according to record type. A spike in Mail Exchange (MX) Record queries might indicate that a customer's router has been compromised and is being used as a spam proxy. Similarly, a sharp increase in round-trip-time latency might indicate a DoS attack.

MPLS Security in a Multivendor Environment

In addition to securing the infrastructure, managed security service providers need to secure packets as they travel from one customer-edge router to another—regardless of the equipment the customer uses at the edge. Layer 3 VPNs meet this need. RFC 4364, which replaced RFC 2547bis, defines a BGP/MPLS IP VPN that creates multiple virtual routers on a single physical router: one virtual router for each customer.

In BGP/MPLS VPNs, Customer Edge (CE) routers send their routes to the Service Provider Edge (PE) routers. Customer edge routers at different sites do not peer with each other, and the customer's routing algorithms are not aware of the overlay. Data packets are tunneled through the backbone so that the core routers do not need to know the VPN routes. BGP/MPLS IP VPNs support either full mesh or partial mesh, although full mesh is more cost-effective.

A unique advantage of BGP/MPLS VPNs is that two service provider customers with overlapping IP addresses can connect across the service provider backbone. The router distinguishes between traffic from different companies by examining the label at the beginning of the packet, and then instantly forwards the traffic based on the Label Switching Path (LSP) that has been established for each customer's VPN. Eliminating the need to look at the packet in depth enables faster forwarding. That is, the service provider core does not impose any latency as packets pass between the provider edge routers.

IPsec Security in a Multivendor Environment

In addition to or instead of deploying a BGP/MPLS IP VPN, the service provider can extend its service to other partner provider networks using IPsec. The options are to use MPLS alone, IPsec alone, or a combination (Figure 3 on page 12). A retail customer that needs to comply with PCI-DSS, for example, needs IPsec or Secure Sockets Layer (SSL) encryption for payment card transaction data as part of its managed security service.

Table 1 summarizes the process based on the option the service provider selects. In the table, VPNA refers to one customer's VPN on a router that hosts VPNs for multiple customers.

Table 1: Comparing Packet Flow in IPsec VPNs, BGP/MPLS VPNs, and Combination VPNs

IPsec	BGP/MPLS VPN	BGP/MPLS VPN and IPsec
Host A in site 1 of VPNA sends packets to host B in site 2 of VPNA. Routers A and B negotiate an Internet Key Exchange (IKE) [9] phase-one session in aggressive or main mode to establish a secure and authenticated channel between peers. Routers A and B negotiate an IKE phase-two session to establish security associations on behalf of IPsec services. Information is exchanged securely through an IPsec tunnel. The tunnel is terminated.	Host A in site 1 of VPNA sends packets to host B in site 2 of VPNA. Packet arrives on a VPN Route-Forwarding (VRF) VPNA interface on the PE1 router. The PE1 router performs an IP lookup, determines the label stack and the outgoing core-facing interface, and forwards the packet to the MPLS core. The packet is label-switched at each hop in the core until it reaches the penultimate hop router. At this point, the top label is popped before the packet is forwarded to the egress provider edge router. The egress PE2 router performs a MPLS lookup and determines that it should remove the label before forwarding the packet to host B in site 2. Router B in site 2 receives a regular IP packet and forwards it to host B.	Router A in site 1 and the associated PE1 router negotiate an IKE phase-one session in aggressive or main mode to negotiate a secure and authenticated channel between peers. Router A and the PE1 router negotiate an IKE phase-two session to establish security associations on behalf of IPsec services so that information is exchanged securely through an IPsec tunnel. Host A in site 1 of VPNA sends packets to host B in site 2 of VPNA. The PE1 router, which is enabled with VRF-aware IPsec, creates a direct association through the IPsec tunnel that connects site 1 and the corresponding VRF ID (VPNA) on the provider edge router over the Internet. Encrypted traffic arrives on an Internet-facing interface on the provider edge router A, which terminates the IPsec tunnel, decrypts the incoming packet, and forwards the plaintext packet to the VRF VPNA for further processing. The PE1 router performs an IP lookup, determines the label stack and the out-going core-facing interface, and forwards the packet to the MPLS core. The packet is label-switched at each hop in the core until it reaches the penultimate hop router. At this point, the top label is popped before the packet is forwarded to the egress provider edge router. The egress PE2 router performs a MPLS lookup and determines that it should remove the label before forwarding the packet to host B in site 2. Router B in site 2 receives a regular IP packet and forwards it to host B. If site 2 is also reachable over the Internet and the egress PE2 router is enabled with VRF-aware IPsec, the packet is encrypted and sent to site 2 across the Internet over an IPsec tunnel. Router B in site 2 terminates the IPsec tunnel, performs a regular IP lookup, and forwards the packet to host B.

Six-Step Methodology

Service providers can detect and mitigate attacks on the infrastructure using a six-step incident-response methodology (Figure 4).

• Preparation: The service provider needs to prepare the network, acquire the needed tools, develop and document a security plan, implement security procedures, and train NOC staff to use tools and procedures. It is vital that security be a practice; the first time that the NOC staff follows its incident-response procedures should not be during an actual attack.
• Identification: Unfortunately, service providers sometimes learn about a security incident from their customers. It is far better to be able to identify the threat before it becomes a problem, using NetFlow telemetry data and analysis tools, for example.
• Classification: The service provider needs to be able to quickly assess the nature of the threat and its scope: single customer, multiple customers, or entire infrastructure.
• Traceback: After classifying the threat, the IT staff needs to identify the point of ingress: peer, upstream server, downstream server, or compromised network device in the data center.
• Reaction: Following classification and traceback, the IT team applies the tools and processes needed to mitigate the attack. Success requires visibility into the network and well-defined procedures. Adherence to standard operating procedures helps prevent the service provider from inadvertently making the problem worse.
• Post-mortem: After the incident, the security team should analyze the root causes and integrate new insights into the security incident-handling proceduresfor use during the next incident.

Real-Life Observations About Interoperability from the APRICOT Workshops

Cisco and Juniper conducted a multivendor security workshop at APRICOT 2006 in Perth, Australia, and again at APRICOT 2007 in Bali, Indonesia. The workshops were offered in response to the fact that service providers often deploy a multivendor network for reasons ranging from financial to political.

Hands-on workshops were conducted in a lab using 12 routers running the Cisco IOS Software and another 12 running JUNOS software. Topics included:

• Password protection
• Packet filtering at the network edge
• Protecting the control plane
• Securing routing protocols
• Network monitoring techniques: NetFlow, syslog, SNMP, and Network Time Protocol (NTP)
• BGP MPLS Layer 3 VPNs
• IPsec VPNs

The goal of the workshops was to achieve a working configuration that interoperated with JUNOS and the Cisco IOS Software, resulting in consistent technology implementation, as well as common security policy enforcement. The workshops underscored the fact that interoperability is not automatic—even among standards-based network products. The reason is that standards bodies such as IETF, ITU, IEEE, and others define some aspects of protocols but leave others to vendor discretion. Standards do define protocol format, which is a syntactical structure identifying bit-field definition, length, and more. They also define protocol behavior, which specifies when actions occur, such as sending Hello and Keepalive timer probes and handling retransmission and reset packets. For purposes of analogy, a spoken language such as English is like a protocol format, and polite conversation conventions, such as beginning with a greeting and concluding with goodbye, is like a protocol behavior.

What standards do not cover are vendor-specific internal implementations, such as software coding techniques, hardware acceleration for performance, command-line interface (CLI) structure, and so on. Therefore, even though the APRICOT workshops involved deploying standards-based technology such as BGP-based MPLS VPNs and IPsec, vendor-specific differences had to be accounted for in the workshop materials and were noticed by participants. Following are examples noted at the APRICOT workshop:

• Label Distribution Protocol: With BGP MPLS VPN, JUNOS and Cisco IOS Software did not interoperate in their default configurations. However, routers from the same vendor did establish Label Distribution Protocol (LDP) sessions. The explanation, which participants found by troubleshooting with debug commands and referring to the manual, is that Cisco IOS Software uses the Tag Distribution Protocol (TDP) by default, whereas JUNOS uses LDP. After the Cisco IOS Software was changed to use LDP, the BGP-based MPLS VPN configuration succeeded.
• IPsec tunnel establishment: To simplify IPsec configuration, the workshop employed a Graphical User Interface (GUI) that prompted the user to choose source and destination IP addresses for the tunnel endpoints, a shared key, and the prefixes that defined the "interesting" traffic that was to use the IPsec tunnel. On the first attempt, the IPsec tunnel was not established. Workshop participants used the CLI to determine the problem, which was that the default encryption being negotiated was incompatible. The root cause for this mismatched encryption standard was that some routers were using an export version of software and needed an upgrade to support a higher encryption standard. Furthermore, even with common encryption capabilities, the two operating systems used different criteria to identify the interesting traffic that would be encrypted. Using the GUI, JUNOS defined interesting traffic as sourced from "ANY" network and destined to 192.168.1.0/24.
  
  In contrast, the Cisco IOS Software defined interesting traffic as sourced from 10.1.1.0/24 and destined to 192.168.1.0/24. Following a discussion about whether the JUNOS default was too permissive or the Cisco IOS Software default was too restrictive, workshop participants agreed to disallow traffic that did not require encryption in the IPsec tunnel. The consensus was that the customer's security policy would provide a more conclusive answer to how permissive the policy should be, and that it was reasonable to require use of the CLI to tweak the configuration because the GUI performed most of the more difficult parts of the configuration on both platforms.
• Loopback interface cost with Open Shortest Path First (OSPF): During the OSPF deployment, participants noticed that the OSPF cost associated with interfaces was the same for each vendor. The OSPF cost is based upon a reference bandwidth of 100 Mbps. However, the loopback interfaces had different values: a default OSPF cost of 1 for the Cisco IOS Software and 0 for JUNOS. It is advisable to change one of the defaults to make them the same.

Although these subtle differences in protocols are documented by the vendors, service provider operational teams often have little time to research them. Therefore, it can be valuable for them to participate in multivendor hands-on workshops. Anecdotal evidence suggests that operators who are comfortable with multiple vendors understand the protocols, helping them design networks that can support new, revenue-generating services.

It is hoped that events such as the APRICOT workshops will help build a community of professionals who can add value for their employers, each other, and the broader Internet community. The result will be a secure and trusted networking environment that people and industry can rely on and useto connect in new and innovative ways.

Summary

Managed security services represent a growing revenue opportunity for service providers. Most service providers operate in a multivendor environment, either because of mergers and acquisitions or because their customers use other vendors' equipment. Therefore, a standards-based approach positions providers to capitalize on the managed security service opportunity. Providers can secure their infrastructure in a multivendor environment by following best practices for point protection, edge protection, RTBH protection, source-address validation, control-plane protection, and total visibility into network activity.