by Kris Foster, TELUS

The Border Gateway Protocol (BGP) is the glue that binds networks and their individual policies together. Several attributes are passed along and possibly modified with each individual prefix, one of which is the community attribute. BGP communities are described poorly in most texts. The problem is not in explaining how they fit into the protocol, but in how to apply these to the real world. In this article I describe how they can be applied within a service provider network and between service provider networks. However, communities are not limited to service providers and can be applied creatively in enterprise networks.

The density of interconnection among service providers, and the various business agreements or political policies, means that controlling who can talk to whom over your network can become difficult. At a basic level there are two types of agreements between service providers: transit/customer and peers.

Customers pay to receive every prefix from a transit provider.
Customers advertise only the prefixes they own (along with their customers' prefixes) to the transit provider.
Peers agree to send only their customers' prefixes to each other, and not other peers' prefixes.

Several methods are available to implement these policies. They can include prefix filters, Autonomous System (AS) path filters, and communities. With only prefix and AS path filters, service providers must ensure that as a new customer or peer is added, the prefixes and AS Numbers (ASNs) associated with the customer (and potentially their customers) are added to the filters on all of the BGP edge routers. This can be automated with scripts, possibly in combination with a route registry database. Very small service providers may be able to manage such a scheme, but as they grow and customer churn begins, this can quickly get out of control. The more time network operators spend in router configurations, the greater likelihood of human error. Communities provide an elegant solution for these problems.

The BGP Community Attribute
Within an AS, all BGP-speaking routers run Internal BGP (iBGP) in a full mesh to prevent routing loops (route reflectors can be used to relax this rule). This means that every BGP-speaking router passes its prefixes to each of its iBGP neighbors. ASs that are adjacent typically run eBGP on directly connected routers. All BGP routers share their prefixes—that is, the network number, network mask, and BGP attributes with each other—allowing each to run its own best-path selection algorithm. As a prefix is passed between ASs, an attribute called the AS-PATH is updated with the corresponding ASN. The AS-PATH is used to prevent routing loops between eBGP neighbors.

A community is a BGP attribute that may be added to each prefix. Communities are transitive optional attributes [1], meaning BGP implementations do not have to recognize the attribute and at the network operator's discretion carry it through an AS or pass it on to another AS. The community attribute can be thought of as simply a flat, 32-bit value that can be applied to any set of prefixes. It can be read as a 32-bit value or split into two portions, the first 2 bytes representing an ASN and the last 2 bytes as a value with a predetermined meaning. The format of the community attribute is shown in Figure 1.

The values 0x00000000 through 0x0000FFFF and 0xFFFF0000 through 0xFFFFFFFF are reserved. Most modern router software displays communities as ASN:VALUE. In this format the communities 1:0 through 65534:65535 are available for use. The convention is to use the ASN of your own network as the leading 16 bits for your internal communities and communities that you accept from and send to your customers.

Three communities are defined in RFC 1997 [2] and are standard within BGP implementations: NO-EXPORT (0xFFFFFF01 ), NO-ADVERTISE (0xFFFFFF02 ), and NO-ADVERTISE-SUBCONFED (0xFFFFFF03 ). Additionally, NO-PEER (0xFFFFFF04 ) has been proposed in an Internet Draft [3].

NO-EXPORT is commonly used within an AS to instruct routers not to export a prefix to eBGP neighbors. For instance, subnets of a larger block can be advertised to influence external AS best-path selection, and those not required for this traffic engineering purpose may be tagged NO-EXPORT to prevent them from being leaked to the Internet (and thus contributing to unnecessary global routing table growth). If a neighboring AS accepts this community, it can be used to selectively leak more specifics for traffic engineering but limit their propagation to just one AS.

NO-ADVERTISE instructs a BGP-speaking router not to send the tagged prefix to any other neighbor, including other iBGP routers.

NO-ADVERTISE-SUBCONFED is used to prevent a prefix from being advertised to other members within a confederation. A confederation can be thought of as a single AS, broken down into sub-ASs. The use of confederations within service provider networks is rare or nonexistent, so they are not considered here.

Finally, NO-PEER is used in situations where traffic engineering control over a more specific prefix is required, but to constrain its propagation only to transit providers and not peers. That is, the prefix is advertised from AS to AS provided there is a transit/customer relationship, unlike NO-EXPORT, which restricts propagation of the prefix to only the adjacent AS. Because peers of the various upstream providers will not see this prefix, the larger prefix encompassing the more specific one is used for routing, thereby conserving an extra entry for some in the global routing table. At this time the community is not recognized by major vendors and requires manual implementation.

Adding Depth: The Extended Community
The current community attribute is getting an upgrade with a new transitive-optional attribute (Type 16) called the Extended Community [4]. Missing from regular communities was any real form of structure. The current Internet Draft defines the Extended Community as an 8-octet value as shown in Figure 1. The first octet specifies the type (and optionally the second value can specify a subtype). This value dictates the structure given to the remaining octets.

The Type field gives the community some immediate flexibility. The first is the use of bit 0 to represent whether the community is registered with the Internet Assigned Numbers Authority (IANA) or if it is specified by the Internet Engineering Task Force (IETF). The second bit gives the Extended Community a coarse scope, either Transitive, meaning it may be passed between ASs, or Non-Transitive, meaning it should be carried only within the local AS.

The Internet Draft also specifies numerous types available for use as templates.

The Route Target Community is already in popular use within Multiprotocol Label Switching Virtual Private Networks (MPLS VPNs). The Route Target Community identifies a set of routers that may receive this prefix. In the MPLS VPN context, this is necessary to limit the resources required to support individual VPN services; only routers that are part of the individual VPN need to hear about the routes within the VPN.

The Link Bandwidth Community gives the network operator additional control in influencing the best path selection. As prefixes are learned from eBGP neighbors, the local neighbor applies this community to specify in bytes per second the bandwidth of the link. It is a Non-Transitive Community, so its scope is limited to the local AS.

Figure 1: Community Formats



Intra-Autonomous System Communities
Policy control using communities within an AS can go farther than this, and their true value is evidenced when they are used to create new and complex policies. If we take our example of the three basic types of neighbor relationships, customers of a transit provider will want to send their customers' prefixes but not their peers' prefixes. To distinguish between a customer's prefix, a peer's prefix, and a transit provider's prefix, we can add a community to each as we learn it from the neighbor.

When advertising a prefix to a customer, peer, or transit provider, simply match all prefixes carrying the communities associated with the correct policy. As shown in Figure 2, all prefixes received from customers are tagged with 53:100, peers are tagged with 53:200, and transit is tagged with 53:300. Our basic definition of a customer is someone who expects to receive all prefixes, so each customer-facing BGP session is preconfigured to send all prefixes matching 53:100, 53:200, and 53:300. Again, from our definition of a peer being someone who wants to see only our customers, we would preconfigure all of our peers' BGP sessions to send only prefixes tagged with 53:100.

Figure 2: Internal Use of Communities for Applying a Basic Service Provider Policy



We can extend this community coding and turn it into a useful troubleshooting tool by adding more information such as where the route was learned geographically. Codes could be assigned per continent, country, state/province, city, or central office.

During redistribution from an Interior Gateway Protocol, a community can be used to specify the original protocol (for example, Intermediate System-to-Intermediate System [IS-IS], Open Shortest Path First [OPSF], or Routing Information Protocol [RIP]). These can be used to quickly determine where a prefix came from without tracing it back to the point of its origination.

It is possible to assign these additional properties in two different ways (or a combination). A single community value may represent a single meaning, such as 53:100, meaning a customer-learned prefix. We could then add additional communities such as 53:1 to mean a prefix learned on the east coast, 53:2 to mean central, and 53:3 to mean west coast. Alternatively, a single community could represent both a customer and a prefix learned on the west coast by tagging with the single tag 53:103. To support these complex values, most vendors allow for pattern matching of specific values, ranges of values, and logical operators such as OR and NOT, in the form of regular expressions. Using regular expressions and complex communities can help to make a router configuration more economical and easier to read.

Inter-Autonomous System Communities
We have some options for Inter-AS traffic engineering: we can prepend additional AS numbers onto a prefix path, use Multi-Exit Discriminators (if the provider supports this), announce more specific prefixes or not announce prefixes at all, modify the origin type, or use communities designed by the other service provider. Communities are clean and consistent with regard to the method of signaling to an adjacent AS how each prefix should be treated.

Of most concern to downstream customers is controlling their primary and backup circuits. Small service providers and enterprises may negotiate different rates on different circuits. Customers purchasing transit with a commitment to send a high amount of traffic with a lower cost per megabit on one circuit, and on a second circuit purchase transit with a very low commitment but at a higher cost per megabit can save some money, assuming they use only the second circuit during outages on the first. Two simple communities can be used to effectively influence a service provider into using the appropriate primary and backup circuits: one value to lower and another to raise the preference of specific prefixes during the transit provider's best-path selection.

An example of adjusting Local Preference with communities can be found in RFC 1998, "An Application of the BGP Community Attribute in Multi-home Routing" [5].

Some other traffic engineering signaling possibilities include:
Force the adjacent AS to prepend its ASN a certain number of times to a prefix sent to customers or peers.
Force the other side to selectively advertise a prefix to specific neighbors.
Request that the neighbor drop all traffic to a prefix.

The last example may seem a little strange; if you are paying someone to deliver traffic, you expect to receive that traffic. Here is where communities can play a role in network security. Denial-of-Service (DoS) attacks may take out an entire customer's service, but the attack may be focused on one or several hosts and not an entire network, as illustrated in Figure 3, allowing customers to tag individual host routes (a subnet consisting of a single address), the customer can signal to the provider to drop all traffic (black hole) for that specific address. To achieve this, the provider selects a single IP address and routes all traffic destined for it to the NULL interfaces on every BGP-speaking router. When a customer signals for a prefix to be blackholed, the service provider replaces the NEXT_HOP information in the BGP advertisement (which under normal circumstances is the edge router IP address) with the specific address that all other routers have statically routed to the NULL interface. When a packet arrives destined for the host under attack, the edge router performs a routing table lookup to find the BGP prefix; using the NEXT_HOP, it then performs a recursive lookup and ultimately sends the packet out the NULL interface. It is important to use other techniques such as prefix lists to prevent a third party from exploiting this technique to disrupt service for others in the Internet.

Figure 3: Customer-Initiated Black Hole to Defend Against a DoS Attack



A service provider may elect to send communities to its customers, leaving it up to the customers to decide for themselves which communities to act on. For a customer who is dual-homed to the same service provider in multiple states or countries, it may be helpful to know where a prefix was originated. A customer could use this community to prefer a connection in New York instead of a Los Angeles connection for European traffic. A single composite metric composed of all relevant geographical information is best, because this gives customers maximum flexibility in choosing the values that are meaningful to them.

Tagging the type of prefix may help other networks to selectively filter more specific addresses. Adding a community specifying if a block is a more specific part of a Classless Inter-Domain Routing (CIDR) block being advertised, the CIDR block itself, or if it is a more specific block but the CIDR block is not being advertised, can help the downstream network avoid incorrect filtering.

Example:
Network A announces
142.77.0.0/16 with a tag of 1:77
142.77.1.0/24 with a tag of 1:88
150.3.12.0/24 with a tag of 1:99
1:77 means it is a CIDR block
1:88 means it is a more specific block within a CIDR block
1:99 means that the full CIDR block is not being announced
Network B then has the option of accepting the more specific 142.77.1.0/24. It also knows that it must accept 150.3.12.0/24 because there is no other route to this network.

In extreme cases providers may find that a portion of their network has become severely degraded. Planned with customers in advance, the upstream provider manually sets a specific community on prefixes associated with the degradation to indicate that this path should be avoided. This could be helpful during natural disasters, fiber cuts, or other unanticipated network outages/degradation. The downstream customers' inbound filters would then match this community and lower the preference on the prefixes tagged with it, causing them to automatically shift traffic to an alternative source if it is available. The degradation signalling process can be seen in Figure 4.

Figure 4: Provided Initiated Signalling of Severe Route Degradation



Design Recommendations
The following are some suggestions if you are just starting out with using communities in your own network. Even the smallest network can benefit from starting early with a clean community design.

Choose a set of internal communities that best reflects the topology and characteristics of your network. For external communities some service providers offer none, others offer only enough to allow for the tagging of primary and backup circuits, and others provide a seemingly endless list.
Keep the set simple. Adding additional complexity typically requires changes to all the BGP-speaking edge routers. Router configurations can quickly grow to enormous proportions to accommodate the numerous community combinations. Troubleshooting a routing mess with a complex community structure can be difficult for those on the graveyard shift.
Avoid transiting communities received from neighboring ASs blindly through your network. This could be abused intentionally or unintentionally to influence traffic to use your costly transit over settlementfree peering and revenue-generating customer circuits. Problems can be created farther out in the Internet and can be very difficult to locate. Depending on the support of your router software, you may be able to selectively add and remove communities, or failing that, you may need to remove all communities and re-add what is acceptable.
Document your communities internally and externally. Your customers will appreciate the additional control, and your operations team will have an easier time troubleshooting.

Summary
Communities add power to BGP, changing it from a routing protocol to a tool for signaling and policy enforcement. If deployed correctly and consistently, communities can help make a network scale, easier to operate, easier to troubleshoot, and can give its customers what they want.

References
[1] Y. Rekhter and T. Li, "A Border Gateway Protocol 4 (BGP-4)," RFC 1771, March 1995.

[2] R. Chandra, P. Traina, and T. Li, "BGP Communities Attribute," RFC 1997, August 1996.

[3] G. Huston, "NOPEER Community for BGP Route Scope Control," Internet Draft, May 2003.

[4] S. Sangli, D. Tappan, and Y. Rekhter, "BGP Extended Communities Attribute," Internet Draft, May 2002.

[5] E. Chen and T. Bates, "An Application of the BGP Community Attribute in Multi-home Routing," RFC 1998, August 1996.

KRIS FOSTER, CCIE® #7749, currently lives in Calgary, Alberta, and spends his time in TELUS' IP backbone. His industry affiliations include the Association for Computing Machinery (ACM), the Internet Society (ISOC), and the North American Network Operators Group (NANOG). He can be reached at kris.foster@telus.com