PF + gif + ipsec + racoon + routing problems

Discussion:

Daniel Duerr

2013-05-13 18:02:32 UTC

Hi everyone,

I wrote up a post on the FreeBSD forums about the issue I am having. It's rather long so I am providing a link to it here: http://forums.freebsd.org/showthread.php?t=39595

In summary, it seems that when the packets are routed in to the gateway from local network hosts, the src and dst addresses are changed to the public IPs of the tunnel -- at least from the perspective of the ipsec stack. This is breaking the ESP encryption in certain cases. I found a workaround, and while the workaround is fine for scenarios where I control both endpoints, I am trying to integrate a remote endpoint (i.e. Cisco ASA) where I cannot get them to implement an equivalent workaround on their end. Does anyone have any ideas that might help me get ipsec to properly match off of the private src and dst addresses?

(I apologize in advance if I'm breaking a mailing list rule by pointing you all to the forum URL -- I'm somewhat new to the list).

Thanks,
Daniel

ECEG / Daniel Duerr

2013-05-13 18:31:07 UTC

Permalink

Daniel Hartmeier

2013-05-17 11:34:29 UTC

Permalink

I rebuilt your setup but can't reproduce the problem.

I picked A.A.A.A=3.3.3.3 and B.B.B.B=4.4.4.4 and used FreeBSD 8.3-STABLE
i386 with GENERIC plus IPSEC, and installed ipsec-tools-0.8.0_3.

------------------------------ gatewayA ------------------------------

/etc/rc.conf
ifconfig_em0="inet 1.1.1.254 netmask 255.255.255.0"
ifconfig_em1="inet 3.3.3.3 netmask 255.255.255.0"
gif_interfaces="gif0"
gifconfig_gif0="3.3.3.3 4.4.4.4"
ifconfig_gif0="1.1.1.254 2.2.2.254 netmask 255.255.255.0"
defaultrouter="3.3.3.1"
static_routes="gif"
route_gif="-net 2.2.2.0/24 2.2.2.254"
gateway_enable="YES"
racoon_enable="YES"
pf_enable="YES"

# ifconfig gif0
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
tunnel inet 3.3.3.3 --> 4.4.4.4
inet 1.1.1.254 --> 2.2.2.254 netmask 0xffffff00
options=1<ACCEPT_REV_ETHIP_VER>

# netstat -anr
Destination Gateway Flags Refs Use Netif Expire
default 3.3.3.1 UGS 0 1948 em1
1.1.1.0/24 link#1 U 0 1270 em0
1.1.1.254 link#1 UHS 1 0 lo0
2.2.2.0/24 2.2.2.254 UGS 0 1873 gif0
2.2.2.254 link#5 UH 0 39 gif0
3.3.3.0/24 link#2 U 0 0 em1
3.3.3.3 link#2 UHS 0 0 lo0
127.0.0.1 link#4 UH 0 0 lo0

/etc/pf.conf is a simple (and identical on gatewayB)
set state-policy if-bound
set skip on { lo }
scrub in log all fragment reassemble
block log
pass

# pfctl -ss
em0 icmp 2.2.2.2:25352 <- 1.1.1.1 0:0
em1 esp 3.3.3.3 -> 4.4.4.4 MULTIPLE:MULTIPLE

/usr/local/etc/racoon/psk.txt
4.4.4.4 topsecret

/usr/local/etc/racoon/ipsec.conf
flush;
spdflush;
spdadd 1.1.1.0/24 2.2.2.0/24 any -P out ipsec esp/tunnel/3.3.3.3-4.4.4.4/use;
spdadd 2.2.2.0/24 1.1.1.0/24 any -P in ipsec esp/tunnel/4.4.4.4-3.3.3.3/use;

/usr/local/etc/racoon/racoon.conf
(exact copy of handbook example, only differences:)
listen
isakmp 3.3.3.3 [500];
isakmp_natt 3.3.3.3 [4500];
remote 4.4.4.4 [500]
my_identifier address 3.3.3.3;
peers_identifier address 4.4.4.4;
sainfo (address 1.1.1.0/24 any address 2.2.2.0/24 any)

------------------------------ gatewayB ------------------------------

ifconfig_em0="inet 2.2.2.254 netmask 255.255.255.0"
ifconfig_em1="inet 4.4.4.4 netmask 255.255.255.0"
gif_interfaces="gif0"
gifconfig_gif0="4.4.4.4 3.3.3.3"
ifconfig_gif0="2.2.2.254 1.1.1.254 netmask 255.255.255.0"
defaultrouter="4.4.4.1"
static_routes="gif"
route_gif="-net 1.1.1.0/24 1.1.1.254"
gateway_enable="YES"
racoon_enable="NO"
pf_enable="YES"

# ifconfig gif0
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
tunnel inet 4.4.4.4 --> 3.3.3.3
inet 2.2.2.254 --> 1.1.1.254 netmask 0xffffff00
options=1<ACCEPT_REV_ETHIP_VER>

# netstat -anr
Destination Gateway Flags Refs Use Netif Expire
default 4.4.4.1 UGS 0 2066 em1
1.1.1.0/24 1.1.1.254 UGS 0 2023 gif0
1.1.1.254 link#5 UH 0 0 gif0
2.2.2.0/24 link#1 U 0 1984 em0
2.2.2.254 link#1 UHS 1 0 lo0
4.4.4.0/24 link#2 U 0 0 em1
4.4.4.4 link#2 UHS 0 0 lo0
127.0.0.1 link#4 UH 0 0 lo0

# pfctl -ss
em1 esp 4.4.4.4 <- 3.3.3.3 MULTIPLE:MULTIPLE
em0 icmp 1.1.1.1:25352 -> 2.2.2.2 0:0

/usr/local/etc/racoon/psk.txt
3.3.3.3 topsecret

/usr/local/etc/racoon/ipsec.conf
flush;
spdflush;
spdadd 2.2.2.0/24 1.1.1.0/24 any -P out ipsec esp/tunnel/4.4.4.4-3.3.3.3/use;
spdadd 1.1.1.0/24 2.2.2.0/24 any -P in ipsec esp/tunnel/3.3.3.3-4.4.4.4/use;

/usr/local/etc/racoon/racoon.conf
listen
isakmp 4.4.4.4 [500];
isakmp_natt 4.4.4.4 [4500];
remote 3.3.3.3 [500]
my_identifier address 4.4.4.4;
peers_identifier address 3.3.3.3;
sainfo (address 2.2.2.0/24 any address 1.1.1.0/24 any)

------------------------------ router ------------------------------

When I ping from gatewayB to 1.1.1.1 (or from 1.1.1.1 to 2.2.2.2),
I see only encrypted packets:

13:23:52.800285 IP (tos 0x0, ttl 63, id 6391, offset 0, flags [none], proto ESP (50), length 136)
4.4.4.4 > 3.3.3.3: ESP(spi=0x016bdbe7,seq=0x5e), length 116
13:23:52.801401 IP (tos 0x0, ttl 64, id 5827, offset 0, flags [none], proto ESP (50), length 136)
3.3.3.3 > 4.4.4.4: ESP(spi=0x04049e8b,seq=0x5e), length 116
13:23:53.820296 IP (tos 0x0, ttl 63, id 6394, offset 0, flags [none], proto ESP (50), length 136)
4.4.4.4 > 3.3.3.3: ESP(spi=0x016bdbe7,seq=0x5f), length 116
13:23:53.821230 IP (tos 0x0, ttl 64, id 5829, offset 0, flags [none], proto ESP (50), length 136)
3.3.3.3 > 4.4.4.4: ESP(spi=0x04049e8b,seq=0x5f), length 116

There must be something in your setup that causes the difference.

If there's a non-trivial pf.conf, maybe try with a trivial one first.

Kind regards,
Daniel

Daniel Duerr

2013-05-17 15:46:14 UTC

Permalink

Hi Daniel,

Thank you so much for taking the time to recreate my (rather large) setup, and for posting it. I double checked my setup compared to your examples here to make sure all was equivalent. I dumbed down my pf.conf as you suggested. Still the same symptoms occur.

Then, as a last resort, I went and looked at my sysctl.conf file and started playing with some of my custom settings. Voila! net.inet.ip.fastforwarding is the culprit. Switching net.inet.ip.fastforwarding: 1 -> 0 fixes the issue and allows all packets to match and be encrypted. Switching net.inet.ip.fastforwarding: 0 -> 1 and the encryption breaks again.

My use of this sysctl dates back to some optimization I'd done when running FreeBSD v7.1. Does anyone here have a current understanding/opinion of its role and necessity on a FreeBSD v8.3 router/gateway?

Best,
Daniel

Post by Daniel Hartmeier
I rebuilt your setup but can't reproduce the problem.
I picked A.A.A.A=3.3.3.3 and B.B.B.B=4.4.4.4 and used FreeBSD 8.3-STABLE
i386 with GENERIC plus IPSEC, and installed ipsec-tools-0.8.0_3.
------------------------------ gatewayA ------------------------------
/etc/rc.conf
ifconfig_em0="inet 1.1.1.254 netmask 255.255.255.0"
ifconfig_em1="inet 3.3.3.3 netmask 255.255.255.0"
gif_interfaces="gif0"
gifconfig_gif0="3.3.3.3 4.4.4.4"
ifconfig_gif0="1.1.1.254 2.2.2.254 netmask 255.255.255.0"
defaultrouter="3.3.3.1"
static_routes="gif"
route_gif="-net 2.2.2.0/24 2.2.2.254"
gateway_enable="YES"
racoon_enable="YES"
pf_enable="YES"
# ifconfig gif0
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
tunnel inet 3.3.3.3 --> 4.4.4.4
inet 1.1.1.254 --> 2.2.2.254 netmask 0xffffff00
options=1<ACCEPT_REV_ETHIP_VER>
# netstat -anr
Destination Gateway Flags Refs Use Netif Expire
default 3.3.3.1 UGS 0 1948 em1
1.1.1.0/24 link#1 U 0 1270 em0
1.1.1.254 link#1 UHS 1 0 lo0
2.2.2.0/24 2.2.2.254 UGS 0 1873 gif0
2.2.2.254 link#5 UH 0 39 gif0
3.3.3.0/24 link#2 U 0 0 em1
3.3.3.3 link#2 UHS 0 0 lo0
127.0.0.1 link#4 UH 0 0 lo0
/etc/pf.conf is a simple (and identical on gatewayB)
set state-policy if-bound
set skip on { lo }
scrub in log all fragment reassemble
block log
pass
# pfctl -ss
em0 icmp 2.2.2.2:25352 <- 1.1.1.1 0:0
em1 esp 3.3.3.3 -> 4.4.4.4 MULTIPLE:MULTIPLE
/usr/local/etc/racoon/psk.txt
4.4.4.4 topsecret
/usr/local/etc/racoon/ipsec.conf
flush;
spdflush;
spdadd 1.1.1.0/24 2.2.2.0/24 any -P out ipsec esp/tunnel/3.3.3.3-4.4.4.4/use;
spdadd 2.2.2.0/24 1.1.1.0/24 any -P in ipsec esp/tunnel/4.4.4.4-3.3.3.3/use;
/usr/local/etc/racoon/racoon.conf
(exact copy of handbook example, only differences:)
listen
isakmp 3.3.3.3 [500];
isakmp_natt 3.3.3.3 [4500];
remote 4.4.4.4 [500]
my_identifier address 3.3.3.3;
peers_identifier address 4.4.4.4;
sainfo (address 1.1.1.0/24 any address 2.2.2.0/24 any)
------------------------------ gatewayB ------------------------------
ifconfig_em0="inet 2.2.2.254 netmask 255.255.255.0"
ifconfig_em1="inet 4.4.4.4 netmask 255.255.255.0"
gif_interfaces="gif0"
gifconfig_gif0="4.4.4.4 3.3.3.3"
ifconfig_gif0="2.2.2.254 1.1.1.254 netmask 255.255.255.0"
defaultrouter="4.4.4.1"
static_routes="gif"
route_gif="-net 1.1.1.0/24 1.1.1.254"
gateway_enable="YES"
racoon_enable="NO"
pf_enable="YES"
# ifconfig gif0
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
tunnel inet 4.4.4.4 --> 3.3.3.3
inet 2.2.2.254 --> 1.1.1.254 netmask 0xffffff00
options=1<ACCEPT_REV_ETHIP_VER>
# netstat -anr
Destination Gateway Flags Refs Use Netif Expire
default 4.4.4.1 UGS 0 2066 em1
1.1.1.0/24 1.1.1.254 UGS 0 2023 gif0
1.1.1.254 link#5 UH 0 0 gif0
2.2.2.0/24 link#1 U 0 1984 em0
2.2.2.254 link#1 UHS 1 0 lo0
4.4.4.0/24 link#2 U 0 0 em1
4.4.4.4 link#2 UHS 0 0 lo0
127.0.0.1 link#4 UH 0 0 lo0
# pfctl -ss
em1 esp 4.4.4.4 <- 3.3.3.3 MULTIPLE:MULTIPLE
em0 icmp 1.1.1.1:25352 -> 2.2.2.2 0:0
/usr/local/etc/racoon/psk.txt
3.3.3.3 topsecret
/usr/local/etc/racoon/ipsec.conf
flush;
spdflush;
spdadd 2.2.2.0/24 1.1.1.0/24 any -P out ipsec esp/tunnel/4.4.4.4-3.3.3.3/use;
spdadd 1.1.1.0/24 2.2.2.0/24 any -P in ipsec esp/tunnel/3.3.3.3-4.4.4.4/use;
/usr/local/etc/racoon/racoon.conf
listen
isakmp 4.4.4.4 [500];
isakmp_natt 4.4.4.4 [4500];
remote 3.3.3.3 [500]
my_identifier address 4.4.4.4;
peers_identifier address 3.3.3.3;
sainfo (address 2.2.2.0/24 any address 1.1.1.0/24 any)
------------------------------ router ------------------------------
When I ping from gatewayB to 1.1.1.1 (or from 1.1.1.1 to 2.2.2.2),
13:23:52.800285 IP (tos 0x0, ttl 63, id 6391, offset 0, flags [none], proto ESP (50), length 136)
4.4.4.4 > 3.3.3.3: ESP(spi=0x016bdbe7,seq=0x5e), length 116
13:23:52.801401 IP (tos 0x0, ttl 64, id 5827, offset 0, flags [none], proto ESP (50), length 136)
3.3.3.3 > 4.4.4.4: ESP(spi=0x04049e8b,seq=0x5e), length 116
13:23:53.820296 IP (tos 0x0, ttl 63, id 6394, offset 0, flags [none], proto ESP (50), length 136)
4.4.4.4 > 3.3.3.3: ESP(spi=0x016bdbe7,seq=0x5f), length 116
13:23:53.821230 IP (tos 0x0, ttl 64, id 5829, offset 0, flags [none], proto ESP (50), length 136)
3.3.3.3 > 4.4.4.4: ESP(spi=0x04049e8b,seq=0x5f), length 116
There must be something in your setup that causes the difference.
If there's a non-trivial pf.conf, maybe try with a trivial one first.
Kind regards,
Daniel

--
daniel duerr | president | ouido.net
***@ouido.net | +1 (831) 531-2272 x103
Managed hosting services for Business