forwarding loop

[Leslie Jensen]

For the last five years I've had a machine with two NIC's running as an
Internet gateway, firewall and proxy server. OS is FreeBSD 8.2-RELEASE.

With the introduction of Squid 3.2 I ran inte a problem that I need help
solving.

Squid has now begone to complain about a forwarding loop and I've gotten
some help at the FreeBSD mail lists.

The recommendation I've got is a change to one rule. Unfortunately this
change gives a syntax error when I try to apply it.

I would very much appreciate some help in understanding why my rules now
are creating this loop and how to solve this issue.

Any suggestions are welcome.

Thanks :-)



My pf.conf:

#
# macros
ext_if="xl0"
int_if="bge0"

tcp_services="{ 22, 993, 5910:5917 }"
tcp_priv_services="{ 389, 443 }"
proxy_services = "{ 21, 80 }"
icmp_types="{ echoreq unreach squench timex }"
internal_net = "172.18.0.0/16"
proxy = "172.18.0.1"
# proxy = "127.0.0.1"
# ftp proxy port
ftpproxyport="8021"
proxyport="8080"
vncports="{ 5900, 5901 }"
torrentport="{ 50505 }"

# tables
table <goodguys> persist
table <sshguard> persist

# options
set block-policy return # ports are closed but can be seen
set loginterface $ext_if

set skip on lo0

# scrub
scrub in

# For ftp proxy
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"

rdr pass proto tcp from any to any port ftp -> 127.0.0.1 port 8080
# rdr pass proto tcp from any to any port ftp -> 127.0.0.1 port 8021


Old rule, that has a forwarding loop
# redirect www trafic to proxy
rdr on $int_if inet proto tcp from $internal_net to any port
$proxy_services -> $proxy port 8080


New suggested rule that gives syntax error
# rdr in on $int_if inet proto tcp from ! $proxy to any port
$proxy_services -> $proxy $proxyport tag rdr_proxy


# ext_if IP address could be dynamic, hence ($ext_if)
nat on $ext_if from !($ext_if) to any -> ($ext_if)

# filter rules
block in log on $ext_if all

block drop in log quick inet6 all

block drop out log quick inet6 all

block in log quick on $ext_if from <sshguard> label "ssh bruteforce"

# Allow traffic through SQUID
pass in log on $int_if inet proto tcp from $internal_net to $proxy port
8080 keep state

pass out log on $ext_if inet proto tcp from $proxy to any port
$proxy_services keep state

# pass out
pass out log

# We need to have an anchor for ftp-proxy
anchor "ftp-proxy/*"

# Let the goodguys access the machine from the outside
pass in log on $ext_if inet proto tcp from <goodguys> to ($ext_if) port
$tcp_services flags S/SA keep state

# Torrent trafic on torrent_port
pass in log on $ext_if inet proto tcp from any to ($ext_if) port
$torrentport

# We need this for the rdr to VNC (change of portnumber)
pass in on $ext_if inet proto tcp from <goodguys> to $internal_net port
$vncports flags S/SA synproxy state

# ICMP answers (traffic) needs to be passed:
pass in inet proto icmp all icmp-type $icmp_types keep state

# traffic must be passed to and from the internal network
pass in log quick on $int_if
#

[Daniel Hartmeier]

Post by Leslie Jensen
New suggested rule that gives syntax error
# rdr in on $int_if inet proto tcp from ! $proxy to any port
$proxy_services -> $proxy $proxyport tag rdr_proxy

1) Remove "in" (rdr implies incoming direction)
2) Remove "tag rdr_proxy" (newer OpenBSD syntax, not needed here)
3) Add "port" after the "->" part (not optional)

So it reads:

rdr on $int_if inet proto tcp from ! $proxy to any port $proxy_services -> $proxy port $proxyport

Which loads fine for me on 8.3-RELEASE-p4.

HTH,
Daniel

[Leslie Jensen]

Post by Daniel Hartmeier

Post by Leslie Jensen
New suggested rule that gives syntax error
# rdr in on $int_if inet proto tcp from ! $proxy to any port
$proxy_services -> $proxy $proxyport tag rdr_proxy

1) Remove "in" (rdr implies incoming direction)
2) Remove "tag rdr_proxy" (newer OpenBSD syntax, not needed here)
3) Add "port" after the "->" part (not optional)
rdr on $int_if inet proto tcp from ! $proxy to any port $proxy_services -> $proxy port $proxyport
Which loads fine for me on 8.3-RELEASE-p4.
HTH,
Daniel

Yes it loads as it should. Now I do have a problem with squid getting a
load of tcp_miss errors.

If I disable the above rule and let the traffic go directly to the
Internet there's no problem.

I wonder if it might be any of the rules I have to allow traffic through
squid.

Will you take a look and see if there's anything obviously wrong?

# Allow traffic through SQUID
# New rule

Post by Daniel Hartmeier
I understand that this need to be changed now that we have removed
the tag from the redirecting rule

# pass in quick log on $int_if inet proto tcp flags S/SAFR tagged rdr_proxy

Post by Daniel Hartmeier
Changing to this does not help!

pass in quick log on $int_if inet proto tcp flags S/SAFR
#
# old rule
# pass in log on $int_if inet proto tcp from $internal_net to $proxy
port 8080 keep state

pass out log on $ext_if inet proto tcp from $proxy to any port
$proxy_services keep state


Thanks

/Leslie

[Daniel Hartmeier]

Wait, the squid server is on a separate host, on the $int_if side of the
firewall (the same side the clients are on)?

Then transparent proxying would require "reflection", and doesn't work, see
http://www.openbsd.org/faq/pf/rdr.html#reflect

If squid is seeing TCP_MISS errors, that probably means the clients are
using the proxy explicitely (not transparently), and the rdr is not
being used at all...

This worked for the past years how?

Daniel

[Daniel Hartmeier]

Post by Daniel Hartmeier
Wait, the squid server is on a separate host, on the $int_if side of the
firewall (the same side the clients are on)?

Yes! This machine has been in service since Freebsd 7.2.
It's one machine with two nic's functioning as a gateway running pf
and squid as proxy.

Well, is Squid running on the same host, or on a separate host?

Is proxy="172.18.0.1" an IP of int_if="bge0"?

Daniel

[Daniel Hartmeier]

You currently have the following rules

pass out log on $ext_if inet proto tcp from $proxy to any port $proxy_services keep state

# pass out
pass out log

What's the point of these? Whenever the first rule would match, the
second one would always override it, making the first one superfluous.

Further, the first rule never matches, because the proxy isn't using
$proxy=172.18.0.1 as source for outgoing connections (but the
IP of $ext_if).

The second rule matches, but doesn't have 'keep state', so the TCP
handshake fails (SYN passes out, but SYN+ACK reply is blocked).

Hence, delete the first rule and add "keep state" to the second rule.

HTH,
Daniel

[Leslie Jensen]

Post by Daniel Hartmeier
You currently have the following rules
pass out log on $ext_if inet proto tcp from $proxy to any port $proxy_services keep state
# pass out
pass out log
What's the point of these? Whenever the first rule would match, the
second one would always override it, making the first one superfluous.
Further, the first rule never matches, because the proxy isn't using
$proxy=172.18.0.1 as source for outgoing connections (but the
IP of $ext_if).
The second rule matches, but doesn't have 'keep state', so the TCP
handshake fails (SYN passes out, but SYN+ACK reply is blocked).
Hence, delete the first rule and add "keep state" to the second rule.
HTH,
Daniel

Thank you Daniel.

Unfortunately it did not help.

The squid access.log says tcp_miss which should mean that the website
has not replied.

The browser shows the squid access denied screen.

I cannot see any denied packets with tcpdump.

Commenting out the rdr rule gives direct access but I would like to get
the use of squid back again.

I would appreciate further suggestions because at the moment I'm stuck!

Thanks

/Leslie

[Daniel Hartmeier]

Post by Leslie Jensen
The squid access.log says tcp_miss which should mean that the
website has not replied.
The browser shows the squid access denied screen.
I cannot see any denied packets with tcpdump.
Commenting out the rdr rule gives direct access but I would like to
get the use of squid back again.
I would appreciate further suggestions because at the moment I'm stuck!

Standard debugging procedure, then:

On the firewall, run the following

fw# tcpdump -nvpi bge0 host 172.18.0.1
fw# tcpdump -nvpi xl0 tcp port 80
fw# tcpdump -nvvveeepi pflog0

Then initiate a single connection attempt from the proxy, like with

proxy# nc -v www.openbsd.org 80

and watch the output of the tcpdumps.

What do you expect to see?

1) TCP SYN from 172.18.0.1 to 129.128.5.194:80 on bge0
2) TCP SYN from $ext_if to 129.128.5.194:80 on xl0
3) TCP SYN+ACK from 129.128.5.194:80 to $ext_if on xl0
4) TCP SYN+ACK from 129.128.5.194:80 to 172.18.0.1 on bge0
5) nothing on pflog0 (except 'pass' messages)

What do you see instead?

If you see block messages on pflog0, re-post your current ruleset.

Daniel

[Leslie Jensen]

Post by Daniel Hartmeier

Post by Leslie Jensen
The squid access.log says tcp_miss which should mean that the
website has not replied.
The browser shows the squid access denied screen.
I cannot see any denied packets with tcpdump.
Commenting out the rdr rule gives direct access but I would like to
get the use of squid back again.
I would appreciate further suggestions because at the moment I'm stuck!

On the firewall, run the following
fw# tcpdump -nvpi bge0 host 172.18.0.1
fw# tcpdump -nvpi xl0 tcp port 80
fw# tcpdump -nvvveeepi pflog0
Then initiate a single connection attempt from the proxy, like with
proxy# nc -v www.openbsd.org 80
and watch the output of the tcpdumps.
What do you expect to see?
1) TCP SYN from 172.18.0.1 to 129.128.5.194:80 on bge0
2) TCP SYN from $ext_if to 129.128.5.194:80 on xl0
3) TCP SYN+ACK from 129.128.5.194:80 to $ext_if on xl0
4) TCP SYN+ACK from 129.128.5.194:80 to 172.18.0.1 on bge0
5) nothing on pflog0 (except 'pass' messages)
What do you see instead?
If you see block messages on pflog0, re-post your current ruleset.
Daniel

I did the above but there's to much traffic going through, it's a lot of
windows machines making to much noise.

If I remember correctly I used the example for transparent proxy from
http://www.benzedrine.cx/transquid.html when I installed the machine.

I went back and used the suggested rules on that page.

Question: Should I use brackets on the variable proxyport?

Now I got "Invalid URL" from squid.

So I'll post my pf.conf

Thanks



---------------------------------------------

# macros
ext_if="xl0"
int_if="bge0"

tcp_services="{ 22, 993, 5910:5917 }"
tcp_priv_services="{ 389, 443 }"
# proxy_services = "{ 21, 80, 443, 1025:65535 }"
proxy_services = "{ 21, 80 }"
icmp_types="{ echoreq unreach squench timex }"
internal_net = "172.18.0.0/16"
proxy = "172.18.0.1"
# proxy = "127.0.0.1"
# ftp proxy port
ftpproxyport="8021"
proxyport="8080"
vncports="{ 5900, 5901 }"
torrentport="{ 50505 }"

# tables
table <goodguys> persist
table <sshguard> persist

# options
set block-policy return # ports are closed but can be seen
set loginterface $ext_if

set skip on lo0

# scrub
scrub in

# For ftp proxy
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"

rdr pass proto tcp from any to any port ftp -> 127.0.0.1 port 8080
# rdr pass proto tcp from any to any port ftp -> 127.0.0.1 port 8021

# Testing for VNC!
# Translate incoming packets' destination addresses.
# As an example, redirect a TCP and UDP port to an internal machine.
# rdr on $ext_if inet proto tcp from <goodguys> to ($ext_if) port 5910 \
# -> 172.18.0.160 port 5900

# redirect www trafic to proxy
# Test
rdr on $int_if inet proto tcp from any to any port www -> 127.0.0.1 port
8080

# ext_if IP address could be dynamic, hence ($ext_if)
nat on $ext_if from !($ext_if) to any -> ($ext_if)

# filter rules
block in log on $ext_if all
# block in log all

block drop in log quick inet6 all

block drop out log quick inet6 all

block in log quick on $ext_if from <sshguard> label "ssh bruteforce"

# Allow traffic through SQUID
# Test
pass in on $int_if inet proto tcp from any to 127.0.0.1 port 8080 keep state

# pass out log on $ext_if inet proto tcp from $proxy to any port
$proxy_services keep state

# Test
#
pass out on $ext_if inet proto tcp from any to any port www keep state
#
pass out log keep state

# We need to have an anchor for ftp-proxy
anchor "ftp-proxy/*"

# Let the goodguys access the machine from the outside
pass in log on $ext_if inet proto tcp from <goodguys> to ($ext_if) port
$tcp_services flags S/SA keep state

# Torrent trafic on torrent_port
pass in log on $ext_if inet proto tcp from any to ($ext_if) port
$torrentport

# We need this for the rdr to VNC (change of portnumber)
pass in on $ext_if inet proto tcp from <goodguys> to $internal_net port
$vncports flags S/SA synproxy state

# ICMP answers (traffic) needs to be passed:
pass in inet proto icmp all icmp-type $icmp_types keep state

# traffic must be passed to and from the internal network
pass in log quick on $int_if
#

[Leslie Jensen]

Post by Daniel Hartmeier

Post by Leslie Jensen
The squid access.log says tcp_miss which should mean that the
website has not replied.
The browser shows the squid access denied screen.
I cannot see any denied packets with tcpdump.
Commenting out the rdr rule gives direct access but I would like to
get the use of squid back again.
I would appreciate further suggestions because at the moment I'm stuck!

On the firewall, run the following
fw# tcpdump -nvpi bge0 host 172.18.0.1
fw# tcpdump -nvpi xl0 tcp port 80
fw# tcpdump -nvvveeepi pflog0
Then initiate a single connection attempt from the proxy, like with
proxy# nc -v www.openbsd.org 80
and watch the output of the tcpdumps.
What do you expect to see?
1) TCP SYN from 172.18.0.1 to 129.128.5.194:80 on bge0
2) TCP SYN from $ext_if to 129.128.5.194:80 on xl0
3) TCP SYN+ACK from 129.128.5.194:80 to $ext_if on xl0
4) TCP SYN+ACK from 129.128.5.194:80 to 172.18.0.1 on bge0
5) nothing on pflog0 (except 'pass' messages)
What do you see instead?
If you see block messages on pflog0, re-post your current ruleset.
Daniel

Hello again.

I've now isolated the other machines from the network.

Please see results of the tests below.



with tcpdump -nvpi bge0 host 172.18.0.1 and nc -v www.openbsd.org 80 I get

Connection to www.openbsd.org 80 port [tcp/http] succeeded

Nothing shows with tcpdump



With tcpdump -nvpi xl0 tcp port 80
tcpdump: listening on xl0, link-type EN10MB (Ethernet), capture size 96
bytes
19:28:09.968833 IP (tos 0x0, ttl 64, id 31116, offset 0, flags [DF],
proto TCP (6), length 60)
194.237.176.105.43068 > 129.128.5.194.80: Flags [S], cksum 0xac3b
(correct), seq 3947358905, win 65535, options [mss 1460,nop,wscale
3,sackOK,TS val 45752295 ecr 0], length 0
19:28:10.118644 IP (tos 0x0, ttl 49, id 16050, offset 0, flags [DF],
proto TCP (6), length 64)
129.128.5.194.80 > 194.237.176.105.43068: Flags [S.], cksum 0x48b0
(correct), seq 3954007035, ack 3947358906, win 16384, options [mss
1460,nop,nop,sackOK,nop,wscale 3,nop,nop,TS val 2145602023 ecr
45752295], length 0
19:28:10.118688 IP (tos 0x0, ttl 64, id 31117, offset 0, flags [DF],
proto TCP (6), length 52)
194.237.176.105.43068 > 129.128.5.194.80: Flags [.], cksum 0xa862
(correct), ack 1, win 8326, options [nop,nop,TS val 45752445 ecr
2145602023], length 0
19:28:42.537896 IP (tos 0x0, ttl 49, id 24173, offset 0, flags [DF],
proto TCP (6), length 52)
129.128.5.194.80 > 194.237.176.105.43068: Flags [F.], cksum 0xc02a
(correct), seq 1, ack 1, win 2172, options [nop,nop,TS val 2145602088
ecr 45752445], length 0
19:28:42.537963 IP (tos 0x0, ttl 64, id 31125, offset 0, flags [DF],
proto TCP (6), length 52)
194.237.176.105.43068 > 129.128.5.194.80: Flags [.], cksum 0x297d
(correct), ack 2, win 8326, options [nop,nop,TS val 45784864 ecr
2145602088], length 0
19:28:42.537999 IP (tos 0x0, ttl 64, id 31126, offset 0, flags [DF],
proto TCP (6), length 52)
194.237.176.105.43068 > 129.128.5.194.80: Flags [F.], cksum 0x297c
(correct), seq 1, ack 2, win 8326, options [nop,nop,TS val 45784864 ecr
2145602088], length 0
19:28:42.689844 IP (tos 0x0, ttl 49, id 32623, offset 0, flags [DF],
proto TCP (6), length 52)
129.128.5.194.80 > 194.237.176.105.43068: Flags [.], cksum 0x4185
(correct), ack 2, win 2172, options [nop,nop,TS val 2145602089 ecr
45784864], length 0


And

Connection to www.openbsd.org 80 port [tcp/http] succeeded



With tcpdump -nvvveeepi pflog0
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file),
capture size 96 bytes
19:31:46.009901 rule 6/0(match): pass out on xl0: (tos 0x0, ttl 64, id
31360, offset 0, flags [none], proto UDP (17), length 72)
194.237.176.105.50724 > 195.67.199.39.53: [|domain]
19:31:46.018267 rule 6/0(match): pass out on xl0: (tos 0x0, ttl 64, id
31362, offset 0, flags [DF], proto TCP (6), length 60)
194.237.176.105.13765 > 129.128.5.194.80: [|tcp]
19:32:12.285752 rule 18/0(match): pass in on bge0: (tos 0x0, ttl 64, id
705, offset 0, flags [none], proto UDP (17), length 76)


And

Connection to www.openbsd.org 80 port [tcp/http] succeeded




I see that rules #6 and #18 matches, but where can I see if the packet
has been redirected to port 8080 and that it has been intercepted by squid?

Thanks

/Leslie

[Leslie Jensen]

Post by Daniel Hartmeier

Post by Daniel Hartmeier
Wait, the squid server is on a separate host, on the $int_if side of the
firewall (the same side the clients are on)?

Yes! This machine has been in service since Freebsd 7.2.
It's one machine with two nic's functioning as a gateway running pf
and squid as proxy.

Well, is Squid running on the same host, or on a separate host?

Same host

Post by Daniel Hartmeier
Is proxy="172.18.0.1" an IP of int_if="bge0"?

Yes it is

Post by Daniel Hartmeier
Daniel

/Leslie

[Leslie Jensen]

Post by Daniel Hartmeier
Wait, the squid server is on a separate host, on the $int_if side of the
firewall (the same side the clients are on)?
Then transparent proxying would require "reflection", and doesn't work, see
http://www.openbsd.org/faq/pf/rdr.html#reflect
If squid is seeing TCP_MISS errors, that probably means the clients are
using the proxy explicitely (not transparently), and the rdr is not
being used at all...
This worked for the past years how?
Daniel

Yes! This machine has been in service since Freebsd 7.2.

That's why I'm totally lost and ask for advise.

It's one machine with two nic's functioning as a gateway running pf and
squid as proxy.

The problems showed themselves when squid updated to version 3.2.

Any help to clarify the situation is appreciated.

Thanks

/Leslie

[Karl O. Pinc]

Post by Daniel Hartmeier
Wait, the squid server is on a separate host, on the $int_if side of the
firewall (the same side the clients are on)?
Then transparent proxying would require "reflection", and doesn't work, see
http://www.openbsd.org/faq/pf/rdr.html#reflect

Just read the FAQ and had a few thoughts.

Something that's not mentioned that
comes to mind is ICMP redirection. (Without thinking
about it a lot it seems like it should be a good candidate.)
However when I tried ICMP redirection on OpenBSD
years ago I couldn't get it to work. Never looked to see why,
or whether it's been fixed since. Or, I might have been doing
something wrong. If anyone can send a clue my way
I'd appreciate knowing more.

Another option for TCP proxying that's not mentioned
is socat. It's like nc, but (I think) would not require
inetd.

Regards,

Karl <***@meme.com>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein

[Daniel Hartmeier]

Post by Karl O. Pinc
Something that's not mentioned that
comes to mind is ICMP redirection. (Without thinking
about it a lot it seems like it should be a good candidate.)
However when I tried ICMP redirection on OpenBSD
years ago I couldn't get it to work. Never looked to see why,
or whether it's been fixed since. Or, I might have been doing
something wrong. If anyone can send a clue my way
I'd appreciate knowing more.

An ICMP redirect, if it is honored by a client, does not mean "if you
want to talk to external-server, connect to proxy-on-local-network
instead", but "if you want to reach external-server, route through
next-hop-on-local-network".

The difference is that the reaction will not change the destination
IP address, but the destination MAC address. Its really the same as if
you added a temporary host route on the client.

The proxy will simply ignore the IP packet for a foreign IP address sent
to its MAC address (like a default gateway that has IP forwarding
disabled).

OpenBSD ignores ICMP redirects by default, you can enable it with sysctl
net.inet.icmp.rediraccept, if you want to try. The security risk is that
someone on the local network could send you ICMP redirects for various
destinations (like DNS server), for a man-in-the-middle attack.

Daniel

[Karl O. Pinc]

Thanks very much for the reply.

Post by Daniel Hartmeier

Post by Karl O. Pinc
Something that's not mentioned that
comes to mind is ICMP redirection. (Without thinking
about it a lot it seems like it should be a good candidate.)
However when I tried ICMP redirection on OpenBSD
years ago I couldn't get it to work. Never looked to see why,
or whether it's been fixed since. Or, I might have been doing
something wrong. If anyone can send a clue my way
I'd appreciate knowing more.

An ICMP redirect, if it is honored by a client, does not mean "if you
want to talk to external-server, connect to proxy-on-local-network
instead", but "if you want to reach external-server, route through
next-hop-on-local-network".

Yes, when I tried it some time ago it was in the context of routing.

Post by Daniel Hartmeier
The difference is that the reaction will not change the destination
IP address, but the destination MAC address. Its really the same as if
you added a temporary host route on the client.
The proxy will simply ignore the IP packet for a foreign IP address sent
to its MAC address (like a default gateway that has IP forwarding
disabled).

I was thinking that the way to use icmp redirects in a "reflection"
setting is to assign a second IP address to the webserver, the server's
public IP address, and, on the router use the internal address
of the webserver as the gateway in a route statement designating
a route to the external address. The router would then issue icmp
redirects and packets would be delivered from the lan to the webserver.
The webserver, having a second IP address, would respond. However
I did not think the whole process through since you could
need additional foolery on the webserver to force it to send
all outbound packets from the LAN IP number -- or at least
that's something you'd always have to ensure.

The advantage
is that 2nd IP numbers on interfaces and adding a route
are easy configuration changes to make. The big disadvantage
is that you're messing around in the bottom part of the
network stack to solve a problem higher up in the stack,
and so it's a kludge that you don't want to hand over
to the random "next guy" who will maintain the network.

I always go for the split-horizon dns setup. It's
obvious what's happening and there's only one place where
the functionality is maintained.

Post by Daniel Hartmeier
OpenBSD ignores ICMP redirects by default, you can enable it with sysctl
net.inet.icmp.rediraccept, if you want to try. The security risk is that
someone on the local network could send you ICMP redirects for
various
destinations (like DNS server), for a man-in-the-middle attack.

FYI, FWIW.

I was not trying to get OpenBSD to respond to icmp redirects, but
to get it to issue icmp redirects. I don't recall messing with a
sysctl, but I also don't see a security disadvantage in
a router issuing icmp redirects since that's what they're supposed
to do. That does not mean there's not a sysctl I should have
frobbed, but I'd hope it'd be a different systcl from the one
involving responding to icmp redirects. Looking at the
list of sysctls in sysctl(8) nothing jumps out.

Thanks again for the reply.

Regards,

Karl <***@meme.com>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein