In my latest blog post i was ranting on how you should not do cloud connectivity, and specifically how you should stay miles away from whoever suggests the use of vxlan to “extend layer 2”.
Today i wanted to show you instead how you could actually extend your network into the cloud to allow workload mobility. It’s assumed that your application is “cloud ready” and won’t require a layer 2 adjacency with other components.
As part of a customer project i was supposed to design a cloud connectivity solution that would allow to extend several VRFs into AWS. The requirements were very clear, so let’s list them:
- It is required to extend around 15 VRFs into AWS to allow application migrations into the cloud.
- The solution needs to be ready for other clouds like Azure or IBM Cloud
- The solution needs to be scalable and be able to ensure support to additional VRFs without network redesign
The high level solution
Simply put, what we did was to extend VXLAN-EVPN Overlay into AWS, specifically by making the CSR 1000v a vtep.
In my specific use case, the customer is running a dual site VXLAN-EVPN DC with EVPN Multi-Site for the DCI so we went with Cisco CSRs.
To be honest though, the solution is pretty standard and can run with every vendor.
Building the Underlay
The picture below describes at High Level how the underlay connectivity to AWS would work. The only real requirement here is jumbo MTU on whatever WAN links you use:
Building the Ovelray
Once loopback reachability is achieved, all we need to do is to establish EVPN control plane between our border leaf and the CSR 1000 and we are basically done.
Please note that in this blog post i am only discussing conceptual connectivity and there is no reference to redundancy
Proof of concept and configurations
Of course, everything looks awesome on power point, but does it work? well, let’s demo this on GSN3:
In this scenario, we will assume that the spine/leaf environment is already working with OSPF in the underlay and iBGP EVPN as control plane. We will then do the following:
- Configure the WAN-RTR and extend OSPF underlay routing
- Configure and IPSec tunnel between the on prem WAN-RTR and the AWS-GW in the cloud
- Configure eBGP between the WAN-RTR and AWS-GW to exchange underlay VTEP prefixes (obviously redistribute those routes into OSPF)
- Configure eBGP EVPN between the border leaf and the AWAS-RTR to extend the EVPN Control plane
- Configure vrf “PROD” in the cloud and extend it via EVPN
OSPF Routing between Border Leaf and WAN-RTR
!
interface Loopback0
description Router-ID
ip address 10.254.180.13 255.255.255.255
ip ospf network point-to-point
!
interface GigabitEthernet1
mtu 9216
ip unnumbered Loopback0
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 MyPaSsWoRd
ip ospf network point-to-point
!
router ospf 1
router-id 10.254.180.13
log-adjacency-changes detail
area 0.0.0.10 authentication message-digest
network 10.254.180.13 0.0.0.0 area 0.0.0.10
!
Internet connectivity and IPSec tunnel
The following config is applied on the WAN-RTR
!
crypto keyring auth-keyring
local-address GigabitEthernet2
pre-shared-key address 1.1.1.2 key MYSECRET
!
crypto isakmp policy 200
encryption aes
authentication pre-share
group 2
lifetime 28800
crypto isakmp profile auth-isakmpprofile
keyring auth-keyring
match identity address 1.1.1.2 255.255.255.255
local-address GigabitEthernet2
!
crypto ipsec transform-set auth-ipsec esp-aes esp-sha-hmac
mode tunnel
!
crypto ipsec profile auth-ipsecprofile
set transform-set auth-ipsec
set pfs group2
!
interface Tunnel10
ip address 2.2.2.1 255.255.255.252
tunnel source GigabitEthernet2
tunnel destination 1.1.1.2
tunnel protection ipsec profile auth-ipsecprofile
ip virtual-reassembly
!
interface GigabitEthernet2
mtu 9216
ip address 1.1.1.1 255.255.255.252
!
I will not paste the config of the AWS-GW, suffice to say that is absolutely identical, just with swapped IP addresses.
Some show commands below just to verify internet connectivity:
WAN-RTR#show ip route ospf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
Gateway of last resort is not set
10.0.0.0/32 is subnetted, 7 subnets
O 10.254.180.1 [110/42] via 10.254.180.12, 00:24:40, GigabitEthernet1
O 10.254.180.11 [110/82] via 10.254.180.12, 00:24:40, GigabitEthernet1
O 10.254.180.12 [110/2] via 10.254.180.12, 00:24:45, GigabitEthernet1
O 10.254.181.1 [110/42] via 10.254.180.12, 00:24:40, GigabitEthernet1
O 10.254.181.11 [110/82] via 10.254.180.12, 00:24:40, GigabitEthernet1
O 10.254.181.12 [110/2] via 10.254.180.12, 00:21:45, GigabitEthernet1
WAN-RTR#ping 2.2.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/3/5 ms
WAN-RTR#show crypto ipsec sa
interface: Tunnel10
Crypto map tag: Tunnel10-head-0, local addr 1.1.1.1
protected vrf: (none)
local ident (addr/mask/prot/port): (1.1.1.1/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (1.1.1.2/255.255.255.255/47/0)
current_peer 1.1.1.2 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 32742, #pkts encrypt: 32742, #pkts digest: 32742
#pkts decaps: 32720, #pkts decrypt: 32720, #pkts verify: 32720
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0
local crypto endpt.: 1.1.1.1, remote crypto endpt.: 1.1.1.2
plaintext mtu 9150, path mtu 9216, ip mtu 9216, ip mtu idb GigabitEthernet2
current outbound spi: 0x4B316738(1261528888)
PFS (Y/N): Y, DH group: group2
inbound esp sas:
spi: 0xAFAF5054(2947502164)
transform: esp-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 2029, flow_id: CSR:29, sibling_flags FFFFFFFF80004048, crypto map: Tunnel10-head-0
sa timing: remaining key lifetime (k/sec): (4607992/2808)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)
inbound ah sas:
inbound pcp sas:
outbound esp sas:
spi: 0x4B316738(1261528888)
transform: esp-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 2030, flow_id: CSR:30, sibling_flags FFFFFFFF80004048, crypto map: Tunnel10-head-0
sa timing: remaining key lifetime (k/sec): (4607995/2808)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)
outbound ah sas:
outbound pcp sas:
Configuring IPV4 eBGP between WAN-RTR and AWS-GW
This is the WAN-RTR part
!
router bgp 65431
bgp router-id 10.254.180.13
bgp log-neighbor-changes
redistribute ospf 1
neighbor 2.2.2.2 remote-as 222
!
router ospf 1
redistribute bgp 65431
!
While this section is configured on AWS-GW
!
interface Loopback0
description RID
ip address 5.5.5.5 255.255.255.255
!
router bgp 222
bgp log-neighbor-changes
neighbor 2.2.2.1 remote-as 65431
!
address-family ipv4
network 5.5.5.5 mask 255.255.255.255
neighbor 2.2.2.1 activate
exit-address-family
!
Configuring a VXLAN-EVPN VTEP on the AWS-GW
10.254.180.12 is the loopback ip of the border leaf
!
interface Loopback1
description VTEP
ip address 6.6.6.6 255.255.255.255
!
interface nve1
no ip address
source-interface Loopback1
host-reachability protocol bgp
!
router bgp 222
neighbor 10.254.180.12 remote-as 65431
neighbor 10.254.180.12 ebgp-multihop 10
neighbor 10.254.180.12 update-source Loopback0
!
address-family ipv4
network 6.6.6.6 mask 255.255.255.255
no neighbor 10.254.180.12 activate
exit-address-family
!
address-family l2vpn evpn
rewrite-evpn-rt-asn
no neighbor 2.2.2.1 activate
neighbor 10.254.180.12 activate
neighbor 10.254.180.12 send-community both
exit-address-family
!
Configure EVPN Multi-Site and eBGP-EVPN on the Border leaf
!
interface loopback2
description MULTISITE-VIP
ip address 8.8.8.8/32
ip router ospf UNDERLAY area 0.0.0.10
ip pim sparse-mode
!
interface Ethernet1/1
evpn multisite fabric-tracking
!
interface Ethernet1/2
evpn multisite dci-tracking
!
evpn multisite border-gateway 100
!
interface nve1
multisite border-gateway interface loopback2
!
router bgp 65431
router-id 10.254.180.12
log-neighbor-changes
address-family l2vpn evpn
advertise-pip
neighbor 5.5.5.5
remote-as 222
update-source loopback0
ebgp-multihop 10
peer-type fabric-external
address-family l2vpn evpn
send-community
send-community extended
rewrite-evpn-rt-asn
!
Now it would be a good moment to test what we have done so far
BORDER-LEAF# show bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.254.180.12, local AS number 65431
BGP table version is 20, L2VPN EVPN config peers 2, capable peers 2
15 network entries and 15 paths using 2880 bytes of memory
BGP attribute entries [8/1312], BGP AS path entries [1/6]
BGP community entries [0/0], BGP clusterlist entries [1/4]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
5.5.5.5 4 222 52 46 20 0 0 00:42:22 0
10.254.180.1 4 65431 517 511 20 0 0 00:42:14 3
Neighbor T AS PfxRcd Type-2 Type-3 Type-4 Type-5
5.5.5.5 E 222 3 0 0 0 0
10.254.180.1 I 65431 3 1 0 0 2
AWS-GW#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
Gateway of last resort is not set
1.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 1.1.1.0/30 is directly connected, GigabitEthernet2
L 1.1.1.2/32 is directly connected, GigabitEthernet2
2.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 2.2.2.0/30 is directly connected, Tunnel10
L 2.2.2.2/32 is directly connected, Tunnel10
5.0.0.0/32 is subnetted, 1 subnets
C 5.5.5.5 is directly connected, Loopback0
6.0.0.0/32 is subnetted, 1 subnets
C 6.6.6.6 is directly connected, Loopback1
8.0.0.0/32 is subnetted, 1 subnets
B 8.8.8.8 [20/2] via 2.2.2.1, 00:39:58
10.0.0.0/32 is subnetted, 7 subnets
B 10.254.180.1 [20/42] via 2.2.2.1, 00:42:27
B 10.254.180.11 [20/82] via 2.2.2.1, 00:42:27
B 10.254.180.12 [20/2] via 2.2.2.1, 00:10:44
B 10.254.180.13 [20/0] via 2.2.2.1, 02:00:50
B 10.254.181.1 [20/42] via 2.2.2.1, 00:42:27
B 10.254.181.11 [20/82] via 2.2.2.1, 00:42:27
B 10.254.181.12 [20/2] via 2.2.2.1, 00:39:58
AWS-GW#show bgp l2vpn evpn summary
BGP router identifier 6.6.6.6, local AS number 222
BGP table version is 42, main routing table version 42
6 network entries using 2064 bytes of memory
6 path entries using 1248 bytes of memory
5/3 BGP path/bestpath attribute entries using 1440 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
6 BGP extended community entries using 224 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 5000 total bytes of memory
BGP activity 70/48 prefixes, 84/62 paths, scan interval 60 secs
7 networks peaked at 10:38:02 Jun 21 2020 UTC (09:05:30.876 ago)
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.254.180.12 4 65431 50 53 42 0 0 00:42:57 0
Everything looks in order, so let’s configure the VRF on the CSR 1000V now:
!
vrf definition PROD
rd 5.5.5.5:100
!
address-family ipv4
route-target export 222:1000000
route-target import 222:1000000
route-target export 222:1000000 stitching
route-target import 222:1000000 stitching
exit-address-family
!
interface Loopback100
description TENANT-LOOPBACK
vrf forwarding PROD
ip address 100.100.100.1 255.255.255.255
!
interface GigabitEthernet1
description INTERFACE FACING VMS INSIDE THE CLOUD
vrf forwarding PROD
ip address 10.10.10.1 255.255.255.0
!
interface BDI100
description L3VNI-SVI
vrf forwarding PROD
ip address 33.33.33.1 255.255.255.0
!
interface nve1
member vni 1000000 vrf PROD
no mop enabled
no mop sysid
!
router bgp 222
!
address-family ipv4 vrf PROD
advertise l2vpn evpn
redistribute connected
exit-address-family
!
So, now let’s verify if the control plane looks good before we test dataplane. On the Cloud side we see this, which is very promising:
AWS-GW#show bgp l2vpn evpn
BGP table version is 42, local router ID is 6.6.6.6
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 5.5.5.5:100 (default for vrf PROD)
*> [5][5.5.5.5:100][0][24][10.10.10.0]/17
0.0.0.0 0 32768 ?
*> [5][5.5.5.5:100][0][24][33.33.33.0]/17
0.0.0.0 0 32768 ?
*> [5][5.5.5.5:100][0][32][100.100.100.1]/17
0.0.0.0 0 32768 ?
Route Distinguisher: 10.254.180.11:3
*> [5][10.254.180.11:3][0][24][10.10.255.0]/17
8.8.8.8 1 0 65431 ?
*> [5][10.254.180.11:3][0][32][100.100.100.11]/17
8.8.8.8 1 0 65431 ?
Route Distinguisher: 10.254.180.12:3
Network Next Hop Metric LocPrf Weight Path
*> [5][10.254.180.12:3][0][32][100.100.100.12]/17
10.254.181.12 0 0 65431 ?
AWS-GW#show ip route vrf PROD
Routing Table: PROD
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C 10.10.10.0/24 is directly connected, GigabitEthernet1
L 10.10.10.1/32 is directly connected, GigabitEthernet1
B 10.10.255.0/24 [20/1] via 8.8.8.8, 01:07:09, BDI100
B 10.10.255.10/32 [20/2000] via 8.8.8.8, 00:15:16, BDI100
33.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 33.33.33.0/24 is directly connected, BDI100
L 33.33.33.1/32 is directly connected, BDI100
100.0.0.0/32 is subnetted, 3 subnets
C 100.100.100.1 is directly connected, Loopback100
B 100.100.100.11 [20/1] via 8.8.8.8, 00:48:38, BDI100
B 100.100.100.12 [20/0] via 10.254.181.12, 00:48:38, BDI100
AWS-GW#show nve peers
Interface VNI Type Peer-IP RMAC/Num_RTs eVNI state flags UP time
nve1 1000000 L3CP 8.8.8.8 0200.0808.0808 1000000 UP A/M/4 00:50:52
nve1 1000000 L3CP 10.254.181.12 0c99.26cd.6407 1000000 UP A/M/4 00:50:52
AWS-GW#show nve vni
Interface VNI Multicast-group VNI state Mode BD cfg vrf
nve1 1000000 N/A Up L3CP 100 CLI PROD
On the DC side (TOR leaf where the end host is connected) we see this:
LEAF# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 107, Local Router ID is 10.254.180.11
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 5.5.5.5:100
*>i[5]:[0]:[0]:[24]:[10.10.10.0]/224
8.8.8.8 100 0 222 ?
*>i[5]:[0]:[0]:[24]:[33.33.33.0]/224
8.8.8.8 100 0 222 ?
*>i[5]:[0]:[0]:[32]:[100.100.100.1]/224
8.8.8.8 100 0 222 ?
Route Distinguisher: 10.254.180.11:34001 (L2VNI 1001234)
*>l[2]:[0]:[0]:[48]:[0c99.2637.ab00]:[0]:[0.0.0.0]/216
10.254.181.11 100 32768 i
*>l[2]:[0]:[0]:[48]:[0c99.2637.ab00]:[32]:[10.10.255.10]/272
10.254.181.11 100 32768 i
Route Distinguisher: 10.254.180.12:3
*>i[2]:[0]:[0]:[48]:[0c99.26cd.6407]:[0]:[0.0.0.0]/216
10.254.181.12 100 0 i
*>i[5]:[0]:[0]:[32]:[100.100.100.12]/224
10.254.181.12 0 100 0 ?
Route Distinguisher: 10.254.180.11:3 (L3VNI 1000000)
*>l[2]:[0]:[0]:[48]:[0c99.2683.ff07]:[0]:[0.0.0.0]/216
10.254.181.11 100 32768 i
*>i[2]:[0]:[0]:[48]:[0c99.26cd.6407]:[0]:[0.0.0.0]/216
10.254.181.12 100 0 i
*>i[5]:[0]:[0]:[24]:[10.10.10.0]/224
8.8.8.8 100 0 222 ?
*>l[5]:[0]:[0]:[24]:[10.10.255.0]/224
10.254.181.11 0 100 32768 ?
*>i[5]:[0]:[0]:[24]:[33.33.33.0]/224
8.8.8.8 100 0 222 ?
*>i[5]:[0]:[0]:[32]:[100.100.100.1]/224
8.8.8.8 100 0 222 ?
*>l[5]:[0]:[0]:[32]:[100.100.100.11]/224
10.254.181.11 0 100 32768 ?
*>i[5]:[0]:[0]:[32]:[100.100.100.12]/224
10.254.181.12 0 100 0 ?
LEAF# show ip route vrf PROD
IP Route Table for VRF "PROD"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
10.10.10.0/24, ubest/mbest: 1/0
*via 8.8.8.8%default, [200/0], 01:09:48, bgp-65431, internal, tag 222 (evpn) segid: 1000000 tunnelid: 0x8080808 encap: VXLAN
10.10.255.0/24, ubest/mbest: 1/0, attached
*via 10.10.255.1, Vlan1234, [0/0], 09:39:35, direct, tag 12345
10.10.255.1/32, ubest/mbest: 1/0, attached
*via 10.10.255.1, Vlan1234, [0/0], 09:39:35, local, tag 12345
10.10.255.10/32, ubest/mbest: 1/0, attached
*via 10.10.255.10, Vlan1234, [190/0], 09:39:07, hmm
33.33.33.0/24, ubest/mbest: 1/0
*via 8.8.8.8%default, [200/0], 01:09:48, bgp-65431, internal, tag 222 (evpn) segid: 1000000 tunnelid: 0x8080808 encap: VXLAN
100.100.100.1/32, ubest/mbest: 1/0
*via 8.8.8.8%default, [200/0], 01:09:48, bgp-65431, internal, tag 222 (evpn) segid: 1000000 tunnelid: 0x8080808 encap: VXLAN
100.100.100.11/32, ubest/mbest: 2/0, attached
*via 100.100.100.11, Lo100, [0/0], 02:26:37, local, tag 12345
*via 100.100.100.11, Lo100, [0/0], 02:26:37, direct, tag 12345
100.100.100.12/32, ubest/mbest: 1/0
*via 10.254.181.12%default, [200/0], 01:09:50, bgp-65431, internal, tag 65431 (evpn) segid: 1000000 tunnelid: 0xafeb50c encap: VXLAN
LEAF# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- -------------------------------------- ----- --------- -------- -----------------
nve1 8.8.8.8 Up CP 01:09:56 0200.0808.0808
nve1 10.254.181.12 Up CP 01:09:57 0c99.26cd.6407
LEAF# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast
Xconn - Crossconnect
MS-IR - Multisite Ingress Replication
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 1000000 n/a Up CP L3 [PROD]
nve1 1001234 239.0.0.1 Up CP L2 [1234]
Since everything looks in order, let’s give it a try from our servers:
DC-SRV:~# ping 10.10.10.100
PING 10.10.10.100 (10.10.10.100): 56 data bytes
64 bytes from 10.10.10.100: seq=0 ttl=61 time=63.145 ms
64 bytes from 10.10.10.100: seq=1 ttl=61 time=33.286 ms
64 bytes from 10.10.10.100: seq=2 ttl=61 time=24.156 ms
64 bytes from 10.10.10.100: seq=3 ttl=61 time=152.668 ms
64 bytes from 10.10.10.100: seq=4 ttl=61 time=95.257 ms
^C
--- 10.10.10.100 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 24.156/73.702/152.668 ms
DC-SRV:~# traceroute 10.10.10.100
traceroute to 10.10.10.100 (10.10.10.100), 30 hops max, 46 byte packets
1 10.10.255.1 (10.10.255.1) 7.546 ms 10.992 ms 8.583 ms
2 100.100.100.12 (100.100.100.12) 58.480 ms 31.835 ms 35.748 ms
3 33.33.33.1 (33.33.33.1) 33.018 ms 34.620 ms 37.632 ms
4 10.10.10.100 (10.10.10.100) 42.153 ms 42.390 ms 37.878 ms
As you can see we have connectivity!!!
In summary
- Do not extend layer 2 into the cloud, it’s just dumb!
- VXLAN-EVPN gives you a very simple way to extend your network into the cloud
- The solution really works with every vendor, even if we just demoed Cisco
- A big advantage of EVPN is that you only need to define the VRFs at the edge and the control plane will take care of the rest, the transit will be totally transparent
- Do NOT forget Jumbo MTU, you really don’t want to fragment 🙂
Andrea Florio's Networking Blog 
Hi, Thanks for your post. What changes did you make in the VPC networking and EC2 instance networking to get this working end to end. I ask as there are certain considerations with ARP and routing when it comes to EC2.
LikeLike
Hello Andrea,
Great post. Trying to replicate what you have done with no success. Do you mind sharing the full configs of all devices?
Thanks.
LikeLike
Hello Andrea,
Great post. Is it possible if you could share the full configuration of all devices? Thank you.
LikeLike
how can I have jumbo frames if you use ipsec over internet?
LikeLike
Same question. Azure max MTU is 1400 bytes. Decrease MTU everywhere?
LikeLike
Sorry, I miss the point with this solution. If you don’t want to extend l2 from on-premises to cloud why all the vxlan stuff? Is not enough to have routing visibility between on-premises and cloud?
LikeLike
Ask yourself a few questions :
1 – why do you want layer 2? Layer 2 extension means Mac to Mac communication. You aren’t allowed to have it in most clouds, so you are likely looking for subnet extension. Vxlan evpn allows you to do so thanks to its host routing design.
2 – how many VRFs do I need to stretch? If you only have 1 or 2 VRFs, point taken, but if you have several ones (maybe because you are doing macro segmentation based on VRFs) then doing flat routing becomes cumbersome and un scalable very quickly.
Bring the 2 things together and you see why having evpn end to end will ensure that your connectivity becomes almost seamless
LikeLike