This document is solution deployment guide for GDOI encryption in secondary network. Deployment of the following two solution test cases are covered in this document:
1. Use same GDOI policies for both primary and secondary Service Provider (SP) network paths in the Group Member (GM). PPPoE Secondary SP network path and DMVPN Secondary SP path are covered in this document.
2. Use different GDOI group for each SP network path with redundancy where even the crypto services are disjoint.
1. GETVPN Backup Network Solution Objectives
Following solution test cases are covered in this document:
• Same GDOI (Group Domain of Interpretation) group and policies for both networks: Use same GDOI policies for both primary and secondary Service Provider (SP) network paths in the Group Member (GM). This case is required if the enterprise VPN spans both providers (e.g. Inter-Provider Inter-AS MPLS VPN). When a GM's path via the primary MPLS network fails, routes are converged in the GM, and GM uses secondary (SP) path to reach other GMs. The GM's traffic may pass through the Inter-Provider Inter-AS link to reach other GM's attached to the primary SP network. In this case, all GM use same GDOI policies on all possible paths. Encrypted traffic flow high availability is achieved via route convergence. Generally, the shortest path between two GM dual-homed to two SP networks will be via the same SP network. There are cases where one GM has a failure to the primary network while another GM has a failure to the secondary network. Traffic between these GM must pass via the Inter-provider Inter-AS link.
• Failure: Outage of a GM link to the primary SP network.
• Recovery: Route convergence and traffic flow via Secondary SP network in the GM.
Solution test for this case is provided in section 2.
• Use different GDOI group for each SP network path: This case is needed if the user wants independent redundancy where even the crypto services are disjoint. The provider VPN networks MUST be disjoint in this case. Different GDOI group is used for each SP network path in the GM. GM High Availability is achieved by triggering EEM script to filter primary network routes if crypto failure is incurred so traffic may traverse through the secondary network.
• Failure: Primary SP routes are fine. But GM is unable to register to Key Server (KS) and unable to receive rekeys from the KS. These are typically operator errors. For example pre-shared keys are not matching between KS and GM or ACL is dropping rekey packets or rekeys are not received in GM due to multicast problem.
• Recovery: After GM registration failure to the last KS, trigger EEM script to filter all primary network routes except routes to the KS. This will cause route convergence to occur and induce GM traffic to be sent via secondary SP network with different keys. This is done in order to insure the routes are revoked on the primary provider network where the crypto has failed.
Solution test case for this case is provided in Section 3.
2. Using GETVPN in PPPoE Secondary SP Network
2.1 Objective
Following are objectives of the solution test.
• Test GDOI encryption of traffic via PPPoE network.
• Test failover of primary SP network. When primary SP network fails, traffic will flow through the secondary SP network. When the primary SP network is restored, traffic will switch back to primary SP network from the secondary SP network.
2.2 Solution Test Topology
GETVPN solution test setup consists of three GMs (Group Members) and two KS (Key Server) are included in the setup. "demo-pe1" simulates the MPLS primary SP network. One Key server is located in the Headquarters. Other Key server is connected behind GM in one of the branch.
Figure 1. Demo GETVPN Network Topology
GMs between branches and headquarters are also connected via secondary PPPoE service provider network. This secondary network will be used when there is network outage in primary SP network. GM between branches are connected to demo-lac via PPPoE interface. PPPoL2TP tunnel connects between demo-lac and demo-lns.
GDOI encryption is done on the customer network side in the GM routers. Traffic flowing through the interface connected to primary SP network and the interface connected to the secondary service provider network are GDOI encrypted.
2.3 PPPoE Secondary SP Network Configuration
Following sections provide PPPoE configuration commands needed for provisioning PPPoE secondary SP network in the solution test setup.
2.3.1 Configuration of GM in Branch 1 (demo-gm1)
Following lists PPPoE related configuration in demo-gm1:
aaa new-model
!
aaa authentication ppp default local
!
bba-group pppoe global
!
username demo password lab
!
interface FastEthernet0/1/0
switchport access vlan 10
spanning-tree portfast
!
interface FastEthernet1/0
description connected to demo-lac
no switchport
no ip address
ip pim sparse-mode
ip tcp adjust-mss 1452
pppoe enable group global
pppoe-client dial-pool-number 10
!
interface Vlan10
ip address 10.5.110.201 255.255.255.248
ip pim sparse-mode
ip igmp join-group 239.192.1.190 source 10.5.110.88
ip igmp join-group 239.192.1.190 source 10.5.110.99
no autostate
!
interface Dialer10
ip address negotiated
ip mtu 1492
ip pim sparse-mode
ip nat outside
ip virtual-reassembly
encapsulation ppp
no ip mroute-cache
dialer pool 10
ppp authentication pap
ppp pap sent-username demo@cisco.com password lab
crypto map gdoi
!
router eigrp 44
network 10.5.110.12 0.0.0.3
network 10.5.110.16 0.0.0.3
network 10.5.110.200 0.0.0.7
network 10.5.110.240 0.0.0.7
no auto-summary
!
ip nat inside source list 10 interface Dialer10 overload
!
access-list 10 permit 10.5.110.200 0.0.0.7
dialer-list 10 protocol ip list 10
2.3.2 Configuration of GM in Branch 2 (demo-gm2)
Following lists PPPoE related configuration in demo-gm2:
aaa new-model
!
aaa authentication ppp default local
!
bba-group pppoe global
!
username demo password lab
!
interface FastEthernet1/0
description connected to demo-lac
no switchport
no ip address
ip pim sparse-mode
ip tcp adjust-mss 1452
pppoe enable group global
pppoe-client dial-pool-number 10
!
interface Vlan10
ip address 10.5.110.209 255.255.255.248
ip pim sparse-mode
ip nat inside
ip virtual-reassembly
ip tcp adjust-mss 1452
ip igmp join-group 239.192.1.190 source 10.5.110.88
ip igmp join-group 239.192.1.190 source 10.5.110.99
no ip mroute-cache
no autostate
!
interface Dialer10
ip address negotiated
ip mtu 1492
ip pim sparse-mode
ip nat outside
ip virtual-reassembly
encapsulation ppp
no ip mroute-cache
dialer pool 10
ppp authentication pap
ppp pap sent-username demo@cisco.com password lab
crypto map gdoi
!
router eigrp 44
network 10.5.110.20 0.0.0.3
network 10.5.110.24 0.0.0.3
network 10.5.110.208 0.0.0.7
network 10.5.110.240 0.0.0.7
no auto-summary
!
ip nat inside source list 10 interface Dialer10 overload
!
access-list 10 permit 10.5.110.208 0.0.0.7
dialer-list 10 protocol ip list 10
2.3.3 Configuration of GM in Headquarters (demo-gm3)
Following lists PPPoE path related configuration in demo-gm3:
username demo password lab
!
interface FastEthernet0/1/8
switchport access vlan 20
!
interface Vlan20
ip address 10.5.110.46 255.255.255.252
ip pim sparse-mode
no autostate
crypto map gdoi
!
router eigrp 44
redistribute static
network 10.5.110.8 0.0.0.3
network 10.5.110.12 0.0.0.3
network 10.5.110.28 0.0.0.3
network 10.5.110.44 0.0.0.3
network 10.5.110.216 0.0.0.7
no auto-summary
2.3.4 Configuration of LAC in SP Network (demo-lac)
Following lists PPPoE related configuration in demo-lac:
aaa new-model
!
aaa authentication ppp default local
!
vpdn enable
!
vpdn-group demo-getvpn-pppoe
! Default L2TP VPDN group
request-dialin
protocol l2tp
domain cisco.com
initiate-to ip 10.5.110.41
local name demo-lac
l2tp tunnel password lab
!
bba-group pppoe demo-getvpn-pppoe
virtual-template 11
sessions auto cleanup
!
interface GigabitEthernet0/0
description connected to demo-gm1
ip address 10.5.110.34 255.255.255.252
ip pim sparse-mode
duplex auto
speed auto
media-type rj45
no negotiation auto
pppoe enable group demo-getvpn-pppoe
!
interface GigabitEthernet0/1
description connected to demo-gm2
ip address 10.5.110.37 255.255.255.252
ip pim sparse-mode
duplex auto
speed auto
media-type rj45
no negotiation auto
pppoe enable group demo-getvpn-pppoe
!
interface GigabitEthernet0/2
description connected to LNS
ip address 10.5.110.42 255.255.255.252
ip pim sparse-mode
duplex auto
speed auto
media-type rj45
no negotiation auto
!
interface Virtual-Template11
no ip address
ip mtu 1492
no ip route-cache cef
ppp authentication pap
ppp pap sent-username demo password lab
!
router eigrp 44
network 10.5.110.32 0.0.0.3
network 10.5.110.36 0.0.0.3
network 10.5.110.40 0.0.0.3
network 10.5.110.240 0.0.0.7
no auto-summary
2.3.5 Configuration of LNS in SP Network (demo-lns)
Following lists PPPoE related configuration in demo-lns:
aaa new-model
!
aaa authentication ppp default local
!
vpdn enable
!
vpdn-group demo-getvpn-pppoe
accept-dialin
protocol l2tp
virtual-template 10
terminate-from hostname demo-lac
local name demo-lns
l2tp tunnel password lab
!
username demo@cisco.com password lab
username demo password lab
!
interface GigabitEthernet0/0
ip address 10.5.110.41 255.255.255.252
ip pim sparse-mode
duplex auto
speed auto
media-type rj45
!
interface GigabitEthernet0/1
ip address 10.5.110.45 255.255.255.252
ip pim sparse-mode
duplex auto
speed auto
media-type rj45
!
interface Virtual-Template10
ip unnumbered GigabitEthernet0/0
ip mtu 1492
ip pim sparse-mode
no ip route-cache cef
peer default ip address pool l2tp-pool
ppp authentication pap
ppp pap sent-username demo password 7 060A0E23
!
router eigrp 44
network 10.5.110.40 0.0.0.3
network 10.5.110.44 0.0.0.3
network 10.5.110.240 0.0.0.7
no auto-summary
!
ip local pool l2tp-pool 10.5.110.242 10.5.110.246
2.4 PPPoE Secondary SP Network Verification
Following sections provide verification of PPPoE secondary SP network functionality in the solution test setup.
2.4.1 Verification of PPPoE Path Setup
On user router (client) enable the following debug commands:
Debug ppp negotiation
Debug ppp authentication
Debug pppoe events
Debug pppoe errors
On LAC and LNS enable the following debug commands:
3. Using GDOI Encryption in DMVPN Secondary SP Network
3.1 Objective
Following are objectives of the solution test.
• Test GDOI encryption of traffic via DMVPN Tunnel network path.
• Test failover of primary SP network. When primary SP network fails, traffic will flow through the secondary SP network. When the primary SP network is restored, traffic will switch back to primary SP network from the secondary SP network.
3.2 Solution Test Topology
GETVPN solution test setup consists of three GMs (Group Members) and two KSs (Key Servers). "demo-pe1" simulates the MPLS primary SP network.
Key servers are connected to both Primary and Secondary SP networks.
3.2.1 DMVPN Secondary Path Setup
Solution test setup consists of two DMVPN spoke routers demo-gm1 and demo-gm2 located in branches and one DMVPN hub router demo-gm3 located in the head quarters. "demo-dmvpn" simulates the IP VPN network. 3845 platform routers running 12.4(22)T IOS image are used.
For enterprise IPSec VPNs that traverse the public Internet, Group Encrypted Transport enhances Dynamic Multipoint VPN (DMVPN) and GRE-based site-to-site VPNs by providing manageable, highly scalable network meshing cost-effectively by using the group shared key. Hub-and-Spoke and Spoke-to-Spoke DMVPN (mGRE) tunnels established with IPSec protection. GDOI encryption is applied to the Tunnel Interface.
EIGRP routing protocol is used for DMVPN. Provider equipment uses BGP routing protocol.
GDOI encryption is done on the customer network side in the GM routers. Traffic flowing through the interface connected to primary SP network and the interface connected to the secondary service provider network are GDOI encrypted.
Figure 2. DMVPN Secondary SP Network with GDOI Encryption in the GMs
3.3 DMVPN Secondary SP Network Configuration
Following sections provide DMVPN configuration commands needed for provisioning DMVPN secondary SP network with GDOI encryption.
3.3.1 Configuration of GM in Branch 1 (demo-gm1)
Following includes DMVPN with GDOI encryption configuration in demo-gm1:
!
hostname demo-gm1
!
ip dhcp pool demo
network 10.5.110.200 255.255.255.248 ! Private local network
default-router 10.5.110.201
ip multicast-routing
ip igmp ssm-map enable
! GDOI IKE policy
crypto isakmp policy 1
encr 3des
authentication pre-share
group 2
! Pre-shared keys
crypto isakmp key dGvPnPsK address 10.5.110.88
crypto isakmp key dGvPnPsK address 10.5.110.99
! GM GDOI Group configuration
crypto gdoi group GETVPN-DEMO
identity number 1357924756
server address ipv4 10.5.110.88
server address ipv4 10.5.110.99
! IKE connection is registered via Vlan 10
crypto map demo-gdoi local-address Vlan10
! GDOI crypto map configuration
crypto map demo-gdoi 1 gdoi
set group GETVPN-DEMO
! DMVPN tunnel with GDOI encryption
interface Tunnel10
bandwidth 2000
ip address 64.0.0.2 255.255.255.0
no ip redirects
ip mtu 1400
ip pim sparse-dense-mode
ip nhrp map multicast 10.5.110.54
ip nhrp map 64.0.0.1 10.5.110.54
ip nhrp network-id 100000
ip nhrp holdtime 300
ip nhrp nhs 64.0.0.1
ip nhrp shortcut
ip nhrp redirect
ip tcp adjust-mss 1360
delay 2000
qos pre-classify
tunnel source FastEthernet1/1
tunnel mode gre multipoint
tunnel key 100000
crypto map demo-gdoi ! GDOI encryption applied on the Tunnel IF
!
interface GigabitEthernet0/0
description Connected to demo-pe1
ip address 10.5.110.17 255.255.255.252
ip flow ingress
ip pim sparse-mode
duplex auto
speed auto
crypto map demo-gdoi ! GDOI encryption applies
!
interface FastEthernet1/1
description connected to demo_dmvpn
ip address 10.5.110.34 255.255.255.252
ip pim sparse-mode
!
interface Vlan10
ip address 10.5.110.201 255.255.255.248
ip pim sparse-mode
ip igmp join-group 239.192.1.190 source 10.5.110.88
ip igmp join-group 239.192.1.190 source 10.5.110.99
no autostate
!
router eigrp 44
network 10.5.110.200 0.0.0.7 ! private local network
Both Primary path and secondary SP network interfaces are in active state.
Create the following on demo-pe1 to simulate primary network failure:
ip access-list standard block-demo-gm1
deny 10.5.110.17
Before ACL is applied, traffic flows through MPLS network.
Configure netflow in interface connecting to primary and secondary SP networks as follows for verifying traffic flow via each interface during primary SP network failover:
demo-pe1(config)#
interface GigabitEthernet0/0
ip route-cache flow
interface Tunnel10
ip route-cache flow
Verify number of packet encrypted before the primary SP network failover as follows:
Verify number of packets sent in each interface after the primary SP network failover as follows. Following output shows that packets are sent using the DMVPN tunnel interface after the primary SP network path failure in demo-gm1.
Gi0/0 10.5.110.18 Local 10.5.110.17 06 ECB5 00B3 3
Tu10 64.0.0.1 Null 224.0.0.10 58 0000 0000 365
Gi0/0 10.5.110.18 Local 10.5.110.17 01 0000 030D 13
Tu10 10.5.110.217 Local 10.5.110.201 01 0000 0000 7244
After the primary SP network failure, Branch 1 GM's (demo-gm1's) route to headquarters private network uses the DMVPN Tunnel interface. It is verified with the following command:
demo-gm1#show ip route 10.5.110.217
Routing entry for 10.5.110.216/29
Known via "eigrp 44", distance 90, metric 1794560, type internal
Redistributing via eigrp 44
Last update from 64.0.0.1 on Tunnel10, 00:00:16 ago
Routing Descriptor Blocks:
* 64.0.0.1, from 64.0.0.1, 00:00:16 ago, via Tunnel10
Route metric is 1794560, traffic share count is 1
Total delay is 20100 microseconds, minimum bandwidth is 2000 Kbit
Reliability 255/255, minimum MTU 1400 bytes
Loading 1/255, Hops 1
3.5.2 Verify Primary Network Recovery and Traffic Flow Switch to Primary Network
When ACL is removed, traffic flow switches to primary MPLS SP network.
Ping corporate private network address 10.5.110.217 as follows:
Success rate is 100 percent (10000/10000), round-trip min/avg/max = 1/2/52 ms
Verify Branch 1 GM's (demo-gm1) route to headquarters private network uses MPLS SP network path via interface Gi0/1 as follows:
demo-gm1#show ip route 10.5.110.217
Routing entry for 10.5.110.216/29
Known via "bgp 200", distance 20, metric 0
Tag 100, type external
Last update from 10.5.110.18 00:01:09 ago
Routing Descriptor Blocks:
* 10.5.110.18, from 10.5.110.18, 00:01:09 ago
Route metric is 0, traffic share count is 1
AS Hops 2
4. Co-op KS and GM High Availability
4.1 Objective
Following are objectives of the solution test.
• Test co-op KS high availability.
• Test GM high availability by using multiple GDOI groups.
4.2 Solution Test Topology
GETVPN solution test setup consists of three GMs (Group Members) and two KS (Key Server) are included in the setup. "demo-pe1" simulates the MPLS primary SP network. One Key server is located in the Headquarters. Other Key server is connected behind GM in one of the branch. Both KSs have path to primary MPLS SP network and secondary PPPoE SP network.
Figure 3. Demo GETVPN Network Topology with Multiple GDOI Groups
GMs between branches and headquarters are also connected via secondary PPPoE service provider network. This secondary network will be used when there is network outage in primary SP network. GM between branches are connected to demo-lac via PPPoE interface. PPPoL2TP tunnel connects between demo-lac and demo-lns.
GDOI encryption is done on the customer network side in the GM routers. Traffic flowing through the interface connected to primary SP network and the interface connected to the secondary service provider network are GDOI encrypted.
KS1 and KS2 are connected to both MPLS and PPPoE networks. GMs encrypt traffic using GDOI group GETVPN-DEMO-MPLS for MPLS network and encrypt traffic GETVPN-DEMO-PPPOE GDOI group for PPPoE network
Following table summarizes GDOI groups and IP addresses of GM interfaces.
Table 1. GDOI Groups and IP Addresses of GM Interfaces
GDOI Group
GDOI Encryption in demo-gm1 Interface
GDOI Encryption in demo-gm2 Interface
GDOI Encryption in demo-gm3 Interface
GETVPN-DEMO-MPLS GDOI Group for Primary MPLS Network (LAN)
Gi0/0 10.5.110.17
Gi0/0 10.5.110.22
Fa0/0 10.5.110.30
GETVPN-DEMO-PPPOE Group for Secondary SP Network (WAN)
Dialer 10 10.5.110.243
Dialer 10 10.5.110.242
Fa0/1/8 10.5.110.46
4.3 Co-op KS High Availability
If there is an outage in primary MPLS network, KS cannot send rekeys successfully to all GMs via MPLS network. This can result in Co-op KS split scenario and can cause encrypted traffic disruption between GMs. Co-op KS high availability can be improved by connecting the KSs to both primary MPLS network and secondary PPPoE network. If there is an outage in one network, rekeys will be sent via secondary network preventing disruption of encrypted traffic between GMs.
4.3.1 Verifying Co-op KS High Availability
4.3.1.1 Verifying Co-op KS Messages and Rekeys Sent via Secondary Network During MPLS Outage
To simulate outage in MPLS network, following is done.
In demo-ks2 check to make sure demo-ks1 uses route via MPLS path:
demo-ks2#show ip route 10.5.110.88
Routing entry for 10.5.110.88/32
Known via "eigrp 44", distance 90, metric 158720, type internal
Redistributing via eigrp 44
Last update from 10.5.110.25 on GigabitEthernet0/1, 00:00:23 ago
Routing Descriptor Blocks:
* 10.5.110.25, from 10.5.110.25, 00:00:23 ago, via GigabitEthernet0/1
Route metric is 158720, traffic share count is 1
Total delay is 5200 microseconds, minimum bandwidth is 100000 Kbit
Reliability 255/255, minimum MTU 1500 bytes
Loading 1/255, Hops 2
Make sure demo-gm1 receives rekeys from demo-gm1 by looking at the following log messages:
*Apr 17 12:52:50 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-MPLS from 10.5.110.88 to 239.192.1.190 with seq # 104
*Apr 17 12:52:50 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-PPPOE from 10.5.110.88 to 239.192.1.191 with seq # 104
*Apr 17 12:53:00 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-MPLS from 10.5.110.88 to 239.192.1.190 with seq # 105
*Apr 17 12:53:00 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-PPPOE from 10.5.110.88 to 239.192.1.191 with seq # 105
*Apr 17 12:53:10 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-MPLS from 10.5.110.88 to 239.192.1.190 with seq # 106
*Apr 17 12:53:10 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-PPPOE from 10.5.110.88 to 239.192.1.191 with seq # 106
Apply the following ACL to block traffic coming from demo-ks1 to demo-pe1 (MPLS network) in the ingress direction.
Add block-ks1-rekey ACL in demo-pe1.
ip access-list extended block-ks1-rekey
deny ip host 10.5.110.88 host 239.192.1.190
In demo-pe1 apply the following ACL configuration as follows:
demo-pe1(config)#int Fa1/0
demo-pe1(config-if)#ip access-group block-demo-ks1 in
demo-pe1(config-if)#exit
demo-pe1(config)#exit
*Apr 17 05:19:18: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 44: Neighbor 10.5.110.13 (FastEthernet1/0) is down: holding time expired
Now co-op KS demo-ks2 uses route via PPPoE path to reach demo-ks1:
demo-ks2#show ip route 10.5.110.88
Routing entry for 10.5.110.88/32
Known via "eigrp 88", distance 90, metric 158720, type internal
Redistributing via eigrp 88
Last update from 10.5.110.54 on GigabitEthernet0/2, 00:00:09 ago
Routing Descriptor Blocks:
* 10.5.110.54, from 10.5.110.54, 00:00:09 ago, via GigabitEthernet0/2
Route metric is 158720, traffic share count is 1
Total delay is 5200 microseconds, minimum bandwidth is 100000 Kbit
Reliability 255/255, minimum MTU 1500 bytes
Loading 1/255, Hops 2
demo-gm1 will continue to receive rekeys from demo-ks1 even during MPLS network outage via PPPoE path:
*Apr 17 13:06:00 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-MPLS from 10.5.110.88 to 239.192.1.190 with seq # 107
*Apr 17 13:06:00 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-PPPOE from 10.5.110.88 to 239.192.1.191 with seq # 107
*Apr 17 13:06:10 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-MPLS from 10.5.110.88 to 239.192.1.190 with seq # 108
*Apr 17 13:06:10 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-PPPOE from 10.5.110.88 to 239.192.1.191 with seq # 108
*Apr 17 13:06:20 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-MPLS from 10.5.110.88 to 239.192.1.190 with seq # 109
*Apr 17 13:06:20 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-PPPOE from 10.5.110.88 to 239.192.1.191 with seq # 109
4.3.1.2 Verifying Co-op KS Messages and Rekeys Sent via Primary Network When MPLS Network Is Restored
To simulate MPLS network restoration, following is done:
demo-pe1(config)#interface FastEthernet1/0
demo-pe1(config-if)#no ip access-group block-demo-ks1 in
demo-pe1(config-if)#end
*Apr 17 06:44:27: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 44: Neighbor 10.5.110.13 (FastEthernet1/0) is up: new adjacency
Now co-op KS demo-ks2 uses route via primary MPLS network path to reach demo-ks1:
demo-ks2#show ip route 10.5.110.88
Routing entry for 10.5.110.88/32
Known via "eigrp 44", distance 90, metric 158720, type internal
Redistributing via eigrp 44
Last update from 10.5.110.25 on GigabitEthernet0/1, 00:02:08 ago
Routing Descriptor Blocks:
* 10.5.110.25, from 10.5.110.25, 00:02:08 ago, via GigabitEthernet0/1
Route metric is 158720, traffic share count is 1
Total delay is 5200 microseconds, minimum bandwidth is 100000 Kbit
Reliability 255/255, minimum MTU 1500 bytes
Loading 1/255, Hops 2
demo-gm1 will continue to receive rekeys from demo-ks1 after MPLS network restoration without any disruption to encrypted traffic flow between GMs:
*Apr 17 13:02:23.661: %GDOI-5-KS_SEND_MCAST_REKEY: Sending Multicast Rekey for group GETVPN-DEMO-MPLS from address 10.5.110.88 to 239.192.1.190 with seq # 20
*Apr 17 13:02:23.685: %GDOI-5-KS_SEND_MCAST_REKEY: Sending Multicast Rekey for group GETVPN-DEMO-PPPOE from address 10.5.110.88 to 239.192.1.191 with seq # 20
*Apr 17 13:02:33.685: %GDOI-5-KS_SEND_MCAST_REKEY: Sending Multicast Rekey for group GETVPN-DEMO-MPLS from address 10.5.110.88 to 239.192.1.190 with seq # 21
*Apr 17 13:02:33.709: %GDOI-5-KS_SEND_MCAST_REKEY: Sending Multicast Rekey for group GETVPN-DEMO-PPPOE from address 10.5.110.88 to 239.192.1.191 with seq # 21
*Apr 17 13:02:43.709: %GDOI-5-KS_SEND_MCAST_REKEY: Sending Multicast Rekey for group GETVPN-DEMO-MPLS from address 10.5.110.88 to 239.192.1.190 with seq # 22
*Apr 17 13:02:43.733: %GDOI-5-KS_SEND_MCAST_REKEY: Sending Multicast Rekey for group GETVPN-DEMO-PPPOE from address 10.5.110.88 to 239.192.1.191 with seq # 22
4.4 GM High Availability by Using Multiple GDOI Groups
It is possible GM is not able to register to KSs or receive rekey from KSs due to network problem with MPLS network. GM without GDOI keys will not be able to send and receive packets from other GMs. When this situation happens, GM is not usable.
4.4.1 Create Separate GDOI Group for Each Network
To avoid this problem, following solution is deployed.
KS1 and KS2 are connected to both MPLS and PPPoE networks. Create GDOI group GETVPN-DEMO-MPLS for MPLS network and GETVPN-DEMO-PPPOE GDOI group for PPPoE network with following CLI commands in KS1. Similar commands are entered in KS2 also.
ip access-list extended dgvpn-rekey-multicast-group
permit ip any host 239.192.1.190
ip access-list extended dgvpn-rekey-multicast-group-new
permit ip any host 239.192.1.191
demo-gm1:
In GMs register to both GDOI groups and attach the GDOI group to respective interface as follows:
crypto gdoi group GETVPN-DEMO-MPLS
identity number 1357924756
server address ipv4 10.5.110.88
server address ipv4 10.5.110.99
!
crypto gdoi group GETVPN-DEMO-PPPOE
identity number 2
server address ipv4 10.5.110.88
server address ipv4 10.5.110.99
!
crypto map gdoi-mpls 1 gdoi
set group GETVPN-DEMO-MPLS
match address no-encryption-acl
!
crypto map gdoi-pppoe 1 gdoi
set group GETVPN-DEMO-PPPOE
match address no-encryption-acl
interface GigabitEthernet0/0
crypto map gdoi-mpls
!
interface Dialer10
crypto map gdoi-pppoe
4.4.2 Simulate GM Reachability Problem to Both KSs via MPLS Network
Following is done to simulate demo-gm1 reachability to KSs. Demo-gm1 gets GDOI keys from the KSs in two ways: GM gets key when it registers with KS. GM also gets key when KS sends rekey message.
4.4.2.1 Simulate GM Registration Problem with KS
Demo-gm1 registration problem with KS is simulated as follows. In demo-ks1 provide pre-share keys for all the registered interfaces except the demo-gm1 interface (10.5.110.17) connected to MPLS network as follows:
Similar pre-shared isakmp keys need to be provisioned in co-op KS demo-ks2 also.
demo-gm1 registration via Gi0/0 (10.5.110.17) will fail. GM will keep trying to register around TEK expiry (900 seconds for demo-gm1) forever.
*Apr 15 13:01:01 pst: %GDOI-4-GM_RE_REGISTER: The IPSec SA created for group GETVPN-DEMO-MPLS may have expired/been cleared, or didn't go through. Re-register to KS.
*Apr 15 13:01:01 pst: %CRYPTO-5-GM_REGSTER: Start registration to KS 10.5.110.88 for group GETVPN-DEMO-MPLS using address 10.5.110.17
*Apr 15 13:01:01 pst: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Informational mode failed with peer at 10.5.110.88
*Apr 15 13:01:41 pst: %CRYPTO-5-GM_CONN_NEXT_SER: GM is connecting to next key server from the list
*Apr 15 13:01:41 pst: %CRYPTO-5-GM_REGSTER: Start registration to KS 10.5.110.99 for group GETVPN-DEMO-MPLS using address 10.5.110.17
4.4.2.2 Simulate GM Not Receiving Rekeys from KS
Following is done to simulate GM not receiving rekeys from KS.
Add the following ACL in demo-pe1:
ip access-list extended block-ks1-rekey-to-gm1
deny ip host 10.5.110.88 host 239.192.1.190
deny ip host 10.5.110.99 host 239.192.1.190
permit ip any any
Attach this ACL to the demo-pe1 interface connecting to demo-gm1 in the MPLS network as follows:
interface GigabitEthernet0/1
description connected to demo-gm1
ip access-group block-ks1-rekey-to-gm1 out
This will block GETVPN-DEMO-MPLS keys sent via MPLS cloud to demo-gm1. demo-gm1 will not have GDOI keys for GETVPN-DEMO-MPLS network.
demo-gm1#show crypto gdoi
GROUP INFORMATION
Group Name : GETVPN-DEMO-MPLS
Group Identity : 1357924756
Rekeys received : 0
IPSec SA Direction : Both
Active Group Server : 10.5.110.88
Group Server list : 10.5.110.88
10.5.110.99
GM Reregisters in : 0 secs
Rekey Received(hh:mm:ss) : 1d19h
Rekeys received
Cumulative : 0
After registration : 1
ACL Downloaded From KS 10.5.110.88:
TEK POLICY:
GigabitEthernet0/0:
Demo-gm1 will receive rekeys for GETVPN-DEMO-PPPOE network:
*Apr 15 12:57:54 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-PPPOE from 10.5.110.88 to 239.192.1.191 with seq # 59
*Apr 15 12:58:04 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-PPPOE from 10.5.110.88 to 239.192.1.191 with seq # 60
*Apr 15 12:58:14 pst: %GDOI-5-GM_RECV_REKEY: Received Rekey for group GETVPN-DEMO-PPPOE from 10.5.110.88 to 239.192.1.191 with seq # 61
4.4.3 Resolve Routing Problems in demo-gm1 by Blocking Routes
Demo-gm1 routes are still pointing to the MPLS network. However demo-gm1 does not have GDOI keys to encrypt traffic over MPLS network. Traffic from demo-gm1 to other GMs will be dropped because there are no keys; however, the policy indicates encryption is required. Likewise, traffic from other GM's to demo-gm1 will fail since traffic arriving at demo-gm1 will be encrypted and demo-gm1 does not have the appropriate GDOI keys.
Ping demo-gm2 MPLS and PPPoE IP address as follows:
demo-gm1#ping 10.5.110.22 source vlan 10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.5.110.22, timeout is 2 seconds:
Packet sent with a source address of 10.5.110.201
....
Success rate is 0 percent (0/4)
demo-gm1#ping 10.5.110.242 source vlan 10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.5.110.242, timeout is 2 seconds:
Packet sent with a source address of 10.5.110.201
...
Success rate is 0 percent (0/3)
Following display shows route to demo-gm2 from demo-gm1. It forwards packet via MPLS PE address.
demo-gm1#show ip route 10.5.110.22
Routing entry for 10.5.110.20/30
Known via "eigrp 44", distance 90, metric 3072, type internal
Redistributing via eigrp 44
Last update from 10.5.110.18 on GigabitEthernet0/0, 00:05:57 ago
Routing Descriptor Blocks:
* 10.5.110.18, from 10.5.110.18, 00:05:57 ago, via GigabitEthernet0/0
Route metric is 3072, traffic share count is 1
Total delay is 20 microseconds, minimum bandwidth is 1000000 Kbit
Reliability 254/255, minimum MTU 1500 bytes
Loading 1/255, Hops
D 10.5.110.20/30 [90/3072] via 10.5.110.18, 00:07:20, GigabitEthernet0/0
demo-gm2 has the GDOI key and demo-gm1 does not have GDOI key. To verify you can do the following.
From demo-gm2, ping demo-gm1's MPLS LAN IP address as follows:
demo-gm2#ping 10.5.110.17 source vlan 10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.5.110.17, timeout is 2 seconds:
Packet sent with a source address of 10.5.110.209
.....
Success rate is 0 percent (0/5)
On demo-gm1 you will see the following log message indicating it does not have the GDOI key.
demo-gm1#
*Apr 16 11:57:07 pst: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=10.5.110.17, prot=50, spi=0x75D10654(1976632916), srcaddr=10.5.110.209
Following log message in demo-gm1 indicates failure of GM registration with last co-op KS:
*Apr 15 13:01:01 pst: %CRYPTO-5-GM_REGSTER: Start registration to KS 10.5.110.99 for group GETVPN-DEMO-MPLS using address 10.5.110.17
*Apr 15 13:01:01 pst: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Informational mode failed with peer at 10.5.110.99
When GM registration fails to last Co-op KS, one needs to block all routes to the other GMs while accepting routes to the KSs via MPLS network path.
One way to do this is as follows.
In the GM (demo-gm1) which is having registration problem, add the following ACL:
access-list 17 permit 10.5.110.88
access-list 17 permit 10.5.110.99
access-list 17 deny any
In demo-gm1 use following Embedded Event Manager (EEM) configuration CLIs to block all routes to the other GMs while accepting routes to the KSs via MPLS network path.
event manager applet block-lan-routes-except-ks
event syslog pattern "%CRYPTO-6-IKMP_MODE_FAILURE: Processing of Informational mode failed with peer at 10.5.110.99"
action 1 cli command "enable"
action 2 cli command "configure terminal"
action 3 cli command "router eigrp 44"
action 4 cli command "distribute-list 17 in"
The above EEM commands will be automatically executed during run-time to block routes whenever GM (demo-gm1) fails to register with KS (demo-ks1). This will result in blocking all the routes to the other GMs while accepting routes to KSs via MPLS network path. Now demo-gm1 can send and receive encrypted traffic from other GMs using GETVPN-DEMO-PPPOE GDOI keys using PPPoE network path.
After EEM applet runs, following is EIGRP related running-config of demo-gm1:
router eigrp 44
network 10.5.110.16 0.0.0.3
network 10.5.110.200 0.0.0.7
distribute-list 17 in
no auto-summary
!
router eigrp 88
network 10.5.110.200 0.0.0.7
network 10.5.110.240 0.0.0.7
no auto-summary
!
Verify successful blocking of routes via MPLS network as follows.
Ping demo-gm2 Dialer interface from demo-gm1:
demo-gm1#ping 10.5.110.242 source vlan 10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.5.110.242, timeout is 2 seconds:
Packet sent with a source address of 10.5.110.201
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
Ping demo-gm2 LAN interface from demo-gm1:
demo-gm1#ping 10.5.110.22 source vlan 10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.5.110.22, timeout is 2 seconds:
Packet sent with a source address of 10.5.110.201
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
Verify routes are forwarding traffic to PPPoE network:
demo-gm1#show ip route
Gateway of last resort is 10.5.110.41 to network 0.0.0.0
10.0.0.0/8 is variably subnetted, 15 subnets, 3 masks
D 10.5.110.99/32
[90/156416] via 10.5.110.18, 03:55:47, GigabitEthernet0/0
D 10.5.110.88/32
[90/156416] via 10.5.110.18, 04:25:45, GigabitEthernet0/0
D 10.5.110.32/30 [90/46228992] via 10.5.110.41, 04:43:53
D 10.5.110.36/30 [90/46228992] via 10.5.110.41, 04:43:53
C 10.5.110.41/32 is directly connected, Dialer10
D 10.5.110.40/30 [90/46226432] via 10.5.110.41, 04:43:53
D 10.5.110.44/30 [90/46228736] via 10.5.110.41, 04:43:54
D 10.5.110.48/30 [90/46228736] via 10.5.110.41, 04:41:32
D 10.5.110.52/30 [90/46228736] via 10.5.110.41, 04:39:15
C 10.5.110.16/30 is directly connected, GigabitEthernet0/0
C 10.5.110.243/32 is directly connected, Dialer10
D 10.5.110.242/32 [90/48786176] via 10.5.110.41, 04:44:00
C 10.5.110.200/29 is directly connected, Vlan10
D 10.5.110.208/29 [90/48788736] via 10.5.110.41, 00:00:14
D 10.5.110.216/29 [90/46231296] via 10.5.110.41, 00:00:14
D*EX 0.0.0.0/0 [170/46231296] via 10.5.110.41, 00:00:14
Every time around the TEK expiry time (~ 900 secs) you will see the following registration failure:
*Apr 15 13:57:01 pst: %GDOI-4-GM_RE_REGISTER: The IPSec SA created for group GETVPN-DEMO-MPLS may have expired/been cleared, or didn't go through. Re-register to KS.
*Apr 15 13:57:01 pst: %CRYPTO-5-GM_REGSTER: Start registration to KS 10.5.110.88 for group GETVPN-DEMO-MPLS using address 10.5.110.17
*Apr 15 13:57:01 pst: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Informational mode failed with peer at 10.5.110.88
All traffic from demo-gm1 will be encrypted and sent via PPPoE interface using GETVPN-MPLS-PPOE GDOI group due to invocation of EEM applet block-lan-routes-except-ks. demo-gm1 will be able to receive and send traffic from other GMs.
4.4.4 Unblock Block Routes When GM to KS Network Reachability Is Restored
Demo-gm1 to demo-ks1 network reachability is restored by undoing steps done in section 3.2 as follows.
Add following configuration to demo-ks1 and demo-ks2:
to allow demo-gm1 registration using pre-shared keys.
In demo-pe1 remove the access-list to block rekeys from KS1 to demo-gm1:
demo-pe1(config)#interface GigabitEthernet0/1
demo-pe1(config-if)#no ip access-group block-ks1-rekey-to-gm1 out
demo-pe1(config-if)#end
When the TEK time expires in demo-gm1, demo-gm1 will successfully register with the first KS.
*Apr 15 15:39:42 pst: %CRYPTO-5-GM_REGSTER: Start registration to KS 10.5.110.88 for group GETVPN-DEMO-MPLS using address 10.5.110.17
*Apr 15 15:39:51 pst: %GDOI-5-GM_REGS_COMPL: Registration to KS 10.5.110.88 complete for group GETVPN-DEMO-MPLS using address 10.5.110.17
Traffic from demo-gm1 to other GMs are still getting forwarded via PPPoE secondary network due to blocking of MPLS routes in demo-gm1 due to block-lan-routes-except-ksEEM applet.
Traffic from demo-gm1 to demo-gm2 goes via PPPoE secondary network.
demo-gm1#show ip route 10.5.110.22
Routing entry for 10.5.110.20/30
Known via "eigrp 44", distance 90, metric 48786432, type internal
Redistributing via eigrp 44
Last update from 10.5.110.41 02:04:44 ago
Routing Descriptor Blocks:
* 10.5.110.41, from 10.5.110.41, 02:04:44 ago
Route metric is 48786432, traffic share count is 1
Total delay is 120010 microseconds, minimum bandwidth is 56 Kbit
Reliability 254/255, minimum MTU 1492 bytes
Loading 1/255, Hops 2
Now demo-gm1 reachability with first co-op KS demo-ks1 via primary MPLS network is restored, unblock the routes filter applied in section 3.3 by applying the following EEM applet unblock-lan-routes CLIs in demo-gm1:
event manager applet unblock-lan-routes
event syslog pattern " %GDOI-5-GM_REGS_COMPL: Registration to KS 10.5.110.88 complete for group GETVPN-DEMO-MPLS using address 10.5.110.17"
action 1 cli command "enable"
action 2 cli command "configure terminal"
action 3 cli command "router eigrp 44"
action 4 cli command "no distribute-list 17 in"
Verify to make sure demo-gm1 traffic to other GMs goes through primary MPLS network.
Verify demo-gm2 route from demo-gm1:
demo-gm1#show ip route 10.5.110.22
Routing entry for 10.5.110.20/30
Known via "eigrp 44", distance 90, metric 3072, type internal
Redistributing via eigrp 44
Last update from 10.5.110.18 on GigabitEthernet0/0, 00:01:53 ago
Routing Descriptor Blocks:
* 10.5.110.18, from 10.5.110.18, 00:01:53 ago, via GigabitEthernet0/0
Route metric is 3072, traffic share count is 1
Total delay is 20 microseconds, minimum bandwidth is 1000000 Kbit
Reliability 254/255, minimum MTU 1500 bytes
Loading 1/255, Hops 1
Ping demo-gm2 Dialer10 interface from demo-gm1:
demo-gm1#ping 10.5.110.242 source vlan 10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.5.110.242, timeout is 2 seconds:
Packet sent with a source address of 10.5.110.201
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
Ping demo-gm2 MPLS LAN IF from demo-gm1:
demo-gm1#ping 10.5.110.22 source vlan 10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.5.110.22, timeout is 2 seconds:
Packet sent with a source address of 10.5.110.201
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
5. Glossary
The following list describes acronyms and definitions for terms used throughout this document:
GETVPN Group Encrypted Transport. A scalable VPN using group technology.
GDOI Group Domain of Interpretation, RFC 3547. A group key management system that is complimentary to IKE.
IKE Internet Key Exchange, RFC 2409. A pair-wise key management system used to negotiation IPSec tunnels.
IPSec IP Protocol Security, RFC 2401. The common name for a set of protocols that protect IP packets.
ISAKMP Internet Security Association and Key Management Protocol, RFC 2408. ISAKMP defines payloads for exchanging key generation and authentication data.
SA Security Association. SAs contain all the information required for execution of various network security services, such as the IP layer services (such as header authentication and payload encapsulation), transport or application layer services, or self-protection of negotiation traffic.