VXLAN-over-IPsec tunnel between two FortiGates

The aim:

Sometimes we need to take two LANs located at different places - maybe offices, maybe cloud regions - and turn them to a single LAN, to one broadcast domain.

With FortiGates, it's possible to achieve by building a VXLAN tunnel between FortiGate in one LAN to FortiGate in another. If encryption is not necessary, we can just use native VXLAN protocol. If it's required - let's say, there's Internet between the LANs - then we can use VXLAN-over-IPsec.

The architecture:

• Each side can be or not to be behind NAT.
• The private subnet between FortiGate and NAT router can be the same on both sides - in fact, even the specific addresses may be the same.
• The LAN-facing port on each FortiGates will not have IP addresses at all - we'll join it together with VXLAN tunnel interface into a software switch, and assign IP address to this switch. Of course, this IP should be different on each FortiGate, as they will belong to the same broadcast domain.
• IPsec authentication in this example is based on pre-shared keys. For certificate-based example, see this post.

Configuration of upper FortiGate:

Basics:
config system interface
    edit wan1
        set vdom root
        set type physical
        set ip 172.16.11.1 255.255.255.0
    next
    edit port2
        set vdom root
        set type physical
    next
end
config router static
    edit 1
        set gateway 172.16.11.254
        set device wan1
    next
end

• No IP address on LAN-facing port
• We will be reaching the peer FortiGate via the default route.

Now we add the IPsec tunnel with VXLAN encapsulation:
config vpn ipsec phase1-interface
    edit VXLAN-on-IPsec
        set interface wan1
        set peertype any
        set proposal aes256-sha256
        set dpd on-idle
        set local-gw 172.16.11.1
        set remote-gw 5.6.7.8
        set psksecret pre_shared_key
        set encapsulation vxlan
        set encapsulation-address ipv4
        set encap-local-gw4 172.16.11.1
        set encap-remote-gw4 172.16.22.1
    next
end
config vpn ipsec phase2-interface
    edit VXLAN-on-IPsec
        set phase1name VXLAN-on-IPsec
        set proposal aes256-sha256
    next
end

• set local-gw isn't actually required, it's here for clarity.
• note that the peer addresses for IPsec are different from peer addreses for VXLAN:
• for IPsec, we specify the remote public address, actually belonging to remote NAT router,
• for VXLAN, we specify the actual private addresses of both FortiGates. Interestingly enough, they can be equal, if the WAN subnets / addresses are the same, this doesn't lead to any collision:
• set encap-local-gw4  172.16.11.1
• set encap-remote-gw4 172.16.11.1

Now we join it to the software switch:
config system switch-interface
    edit LAN-Soft-Switch
        set vdom root
        set member port2 VXLAN-on-IPsec
    next
end

And configure an IP address on it. Now the interfaces configuration looks like this:
config system interface
    edit wan1
        set vdom root
        set type physical
        set ip 172.16.11.1 255.255.255.0
    next
    edit port2
        set vdom root
        set type physical
    next
    edit VXLAN-on-IPsec
        set vdom root
        set interface wan1
        set type tunnel
    next
    edit LAN-Soft-Switch
        set vdom root
        set ip 10.10.10.1 255.255.255.0
        set allowaccess ping
        set type switch
    next
end


Configuration of lower FortiGate:
It's symmetrical, so I'll just list all relevant sections:
config vpn ipsec phase1-interface
    edit VXLAN-on-IPsec
        set interface wan1
        set peertype any
        set proposal aes256-sha256
        set dpd on-idle
        set local-gw 172.16.22.1
        set remote-gw 1.2.3.4
        set psksecret ENC encrypted_pre_shared_key
        set encapsulation vxlan
        set encapsulation-address ipv4
        set encap-local-gw4 172.16.22.1
        set encap-remote-gw4 172.16.11.1
    next
end
config vpn ipsec phase2-interface
    edit VXLAN-on-IPsec
        set phase1name VXLAN-on-IPsec
        set proposal aes256-sha256
    next
end
config system switch-interface
    edit LAN-Soft-Switch
        set vdom root
        set member port2 VXLAN-on-IPsec
    next
end
config system interface
    edit wan1
        set vdom root
        set type physical
        set ip 172.16.22.1 255.255.255.0
    next
    edit port2
        set vdom root
        set type physical
    next
    edit VXLAN-on-IPsec
        set vdom root
        set interface wan1
        set type tunnel
    next
    edit LAN-Soft-Switch
        set vdom root
        set ip 10.10.10.2 255.255.255.0
        set allowaccess ping
        set type switch
    next
end
config router static
    edit 1
        set gateway 172.16.22.254
        set device wan1
    next
end


The packets flowing over Internet between the NAT routers will look, after all levels of encapsulation, more or less like this:

Of course, all these headers add significant overhead to our packets. So if FortiGates' physical interfaces have a standard MTU of 1500 bytes, then the MTU of VXLAN interface (and thus of the software switch) will be only 1374 bytes:

FortiGate # fnsysctl ifconfig VXLAN-on-IPsec
VXLAN-over-IPsec Link encap:Ethernet  HWaddr 12:35:08:F7:AC:52
        UP BROADCAST RUNNING MULTICAST  MTU:1374  Metric:1
        RX packets:2656 errors:0 dropped:0 overruns:0 frame:0
        TX packets:857 errors:2 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:467584 (456.6 KB)  TX bytes:55392 (54.1 KB)

FortiGate # fnsysctl ifconfig LAN-Soft-Switch​
LAN-Soft-Switch​ Link encap:Ethernet  HWaddr 00:50:56:9C:2E:8E
        inet addr:10.10.10.1  Bcast:10.10.10.255  Mask:255.255.255.0
        UP BROADCAST RUNNING MULTICAST  MTU:1374  Metric:1
        RX packets:85801 errors:0 dropped:0 overruns:0 frame:0
        TX packets:179544 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000

        RX bytes:7299089 (6.10 MB)  TX bytes:233154091 (222.4 MB)

Useful links:

• https://kb.fortinet.com/kb/documentLink.do?externalID=FD38614
• https://kb.fortinet.com/kb/documentLink.do?externalID=FD40170
• https://help.fortinet.com/fos50hlp/56/Content/FortiOS/fortigate-whats-new/Top-Network-vxlan.htm
• https://travelingpacket.com/2017/09/28/fortigate-vxlan-encapsulation

Possible alternative way of configuring VXLAN over IPsec:

1. add a loopback address on each FortiGate
2. set up regular IPsec tunnel between the FortiGates, allowed to usage between loopbacks only.
3. set up native VXLAN tunnel between these loopbacks.

But I didn't try it.