Linux transparent proxy internals
Learn how Linux's transparent proxying works by building a toy proxy server in C.
I've always thought the transparent proxying functionality in the Linux kernel is one of the coolest but hardest to understand.
This may have to do with the sheer lack of documentation on the subject or maybe the fact that configuring it requires quite a few moving pieces like IP rules, socket options, and routing.
Let's explore how transparent proxying works and build a toy transparent proxy server to drive the concepts home.
What is transparent proxying?
Most people will have colloquial familiarity with the term "proxy" as a networking infrastructure.
A proxy sits between a client, the source, and a server, the destination, and manages the connections and data between them.
If this is a TCP proxy, it will terminate the client's connection, holding a socket to it, and will connect to the server, holding a socket for this "leg" as well.
With both sockets in-hand the proxy will pipe the data between them, performing whatever actions on the data is required.
Proxies can do all sorts of things, but we just need to understand the fundamentals right now.
┌─────────────┐
│ 66.66.66.66 │
┌────────┐ conn A │ proxy │ conn B ┌────────┐
│10.0.5.1├────────────► ├────────────►10.0.6.1│
│ client ◄────────────┤ terminates ◄────────────┤ server │
└────────┘ │ and │ └────────┘
│ forwards │
└─────────────┘
The above is what, I think, most people imagine when they think 'proxy', though technically, this is a reverse proxy where the proxy sits in front of the server.
To understand what a transparent proxy is they must think about what IP addresses the client will use to connect to the server, and the client IPs seen by the server.
┌─────────────┐
│ 66.66.66.66 │
┌────────┐ conn A │ proxy │ conn B ┌────────┐
│10.0.5.1├────────────► ├────────────►10.0.6.1│
│ client ◄────────────┤ terminates ◄────────────┤ server │
└────────┘ │ and │ └────────┘
┌───────────────────────┐ │ forwards │ ┌───────────────────────┐
│ Conn A │ └─────────────┘ │ Conn B │
│10.0.5.1 -> 66.66.66.66│ │66.66.66.66 -> 10.0.6.1│
└───────────────────────┘ └───────────────────────┘Let's use the non-transparent proxy example above, both the client connection and the server's connections are well aware a proxy server sits between them.
A transparent proxy is fundamentally different.
┌─────────────┐
│ 66.66.66.66 │
┌────────┐ conn A │ proxy │ conn B ┌────────┐
│10.0.5.1├────────────► ├────────────►10.0.6.1│
│ client ◄────────────┤ terminates ◄────────────┤ server │
└────────┘ │ and │ └────────┘
┌───────────────────────┐ │ forwards │ ┌───────────────────────┐
│ Conn A │ └─────────────┘ │ Conn B │
│ 10.0.5.1 -> 10.0.6.1 │ │ 10.0.5.1 -> 10.0.6.1 │
└───────────────────────┘ └───────────────────────┘ Notice, in the transparent case the client perceives itself connected directly to the server and the server perceives itself connected directly to the client.
Both client and server maintain their IP addresses, across both "legs" of the proxy.
The example above is sometimes called a "fully transparent proxy" since the source IP of the client is maintained from the server's perspective. It's not uncommon for the server to see the proxy's original IP when, for example, routing the client's IP back to the proxy is difficult, among other reasons.
While the explanation is simple, like a lot of things with networking, the devil's in the details.
Let's start building a proxy server which we will modify into a transparent socket as we explain how we can accomplish the above.
The topology
We need a small testing environment which creates separate layer 3 networks for the client, proxy, and server.
We can do this with just Linux network namespaces, the ip tool, and veths.
┌───────────┐ ┌──────────────┐ ┌──────────────┐
│ client ns │ │ proxy ns │ │ server ns │
│ ┌────┐ ┌────┐ ┌────┐ ┌────┐ │
│ │ ┼────────┼ │ │ ┼────────┼ │ │
└──────┴────┘ └────┘────└────┘ └────┘─────────┘
10.0.5.1 10.0.5.10 10.0.6.10 10.0.6.1 The above diagram illustrates the topology, three network namespaces connected by veths.
Makefile
LINK = ip link
NETNS = ip netns
NETNS_EXEC = ip netns exec
NETNS_CLIENT = client
NETNS_PROXY = proxy
NETNS_SERVER = server
topology:
# create network namespaces
$(NETNS) add $(NETNS_CLIENT)
$(NETNS) add $(NETNS_PROXY)
$(NETNS) add $(NETNS_SERVER)
# wire them together with veths
$(LINK) add name client-a type veth peer name proxy-a
$(LINK) set dev client-a netns $(NETNS_CLIENT)
$(LINK) set dev proxy-a netns $(NETNS_PROXY)
$(LINK) add name proxy-b type veth peer name server-a
$(LINK) set dev proxy-b netns $(NETNS_PROXY)
$(LINK) set dev server-a netns $(NETNS_SERVER)
# configure IP networking
$(NETNS_EXEC) $(NETNS_CLIENT) ip addr add 10.0.5.1/24 dev client-a
$(NETNS_EXEC) $(NETNS_CLIENT) ip link set dev client-a up
$(NETNS_EXEC) $(NETNS_CLIENT) ip route add default via 10.0.5.10
$(NETNS_EXEC) $(NETNS_PROXY) ip addr add 10.0.5.10/24 dev proxy-a
$(NETNS_EXEC) $(NETNS_PROXY) ip link set dev proxy-a up
$(NETNS_EXEC) $(NETNS_PROXY) ip addr add 10.0.6.10/24 dev proxy-b
$(NETNS_EXEC) $(NETNS_PROXY) ip link set dev proxy-b up
$(NETNS_EXEC) $(NETNS_SERVER) ip addr add 10.0.6.1/24 dev server-a
$(NETNS_EXEC) $(NETNS_SERVER) ip link set dev server-a up
$(NETNS_EXEC) $(NETNS_SERVER) ip route add default via 10.0.6.10
# enable arp proxy in proxy network namespace
$(NETNS_EXEC) $(NETNS_PROXY) sysctl -w net.ipv4.ip_forward=1
$(NETNS_EXEC) $(NETNS_PROXY) sysctl -w net.ipv4.conf.all.proxy_arp=1
topology-destroy:
$(NETNS) del $(NETNS_CLIENT)
$(NETNS) del $(NETNS_PROXY)
$(NETNS) del $(NETNS_SERVER)We can now test end-to-end connectivity with the command:
$ make topology
$ sudo ip netns exec client ping 10.0.6.1This will ping from the client's IP network stack to the server's using the proxy network namespace as a router; it should succeed.
A non-transparent proxy server
Let's first build a non-transparent proxy server as a stepping stone.
This transparent proxy will be very dumb, simply to demonstrate the transparent proxying requirements.
It will only accept one connection at a time, wait for the client to write, return the response from the server back to the client, and then close both the client and server connections.
proxy.c
#define _GNU_SOURCE
#include <arpa/inet.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>
#include <unistd.h>
#define PROXY_PORT 6666
#define PROXY_TARGET_PORT 8080
#define PROXY_TARGET_IP 0x0a000601
#define MAX_MSG_SIZE 1024 * 1024
int proxy_do(int client_sock) {
struct sockaddr_in server_addr = {.sin_addr = htonl(PROXY_TARGET_IP),
.sin_port = htons(PROXY_TARGET_PORT),
.sin_family = AF_INET};
int server_sock = socket(AF_INET, SOCK_STREAM, 0);
int pipe[2];
pipe2(pipe, O_NONBLOCK);
if (server_sock < 0) {
perror("proxy_do: socket");
return 0;
}
if (connect(server_sock, (struct sockaddr*)&server_addr,
sizeof(server_addr)) < 0) {
perror("proxy_do: connect");
goto cleanup;
}
// client -> server via splice pipe
if (splice(client_sock, 0, pipe[1], 0, MAX_MSG_SIZE, SPLICE_F_MOVE) < 0) {
perror("proxy_do: client splice to pipe");
goto cleanup;
}
if (splice(pipe[0], 0, server_sock, 0, MAX_MSG_SIZE, SPLICE_F_MOVE) < 0) {
perror("proxy_do: pipe splice to server");
goto cleanup;
}
// client <- server via splice pipe
if (splice(server_sock, 0, pipe[1], 0, MAX_MSG_SIZE, SPLICE_F_MOVE) < 0) {
perror("proxy_do: server splice");
goto cleanup;
}
if (splice(pipe[0], 0, client_sock, 0, MAX_MSG_SIZE, SPLICE_F_MOVE) < 0) {
perror("proxy_do: pipe splice to client");
goto cleanup;
}
cleanup:
close(server_sock);
close(pipe[0]);
close(pipe[1]);
return 0;
}
int main(int argc, char* argv[]) {
int sock = socket(AF_INET, SOCK_STREAM, 0);
if (sock < 0) {
perror("socket");
exit(EXIT_FAILURE);
}
struct sockaddr_in addr = {
.sin_addr = 0, .sin_port = htons(PROXY_PORT), .sin_family = AF_INET};
if (bind(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0) {
perror("bind");
exit(EXIT_FAILURE);
}
if (listen(sock, 1) < 0) {
perror("listen");
exit(EXIT_FAILURE);
}
printf("proxy start *:%d \n", PROXY_PORT);
while (1) {
struct sockaddr_in client = {0};
socklen_t addr_n = sizeof(client);
char buf[INET_ADDRSTRLEN];
int client_sock = accept(sock, (struct sockaddr*)&client, &addr_n);
if (client_sock < 0) {
perror("accept");
continue;
}
if (inet_ntop(AF_INET, &client.sin_addr, buf, sizeof(buf)) == NULL) {
perror("inet_ntop");
continue;
}
printf("new client: %s\n", buf);
proxy_do(client_sock);
close(client_sock);
}
}Let's also update the Makefile to build the proxy server.
diff --git a/Makefile b/Makefile
index 217ff37..ef27ffb 100644
--- a/Makefile
+++ b/Makefile
@@ -5,6 +5,8 @@ NETNS_CLIENT = client
NETNS_PROXY = proxy
NETNS_SERVER = server
+proxy: proxy.o
+
topology:
# create network namespaces
$(NETNS) add $(NETNS_CLIENT)Run make to build the proxy binary.
At this point we can test the proxy server in a few terminals.
term1: deploy topology, start an echo server using nc
$ sudo make topology
$ sudo ip netns exec server ncat -v -l -k -p 8080 --exec /usr/bin/cat
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Listening on :::8080
Ncat: Listening on 0.0.0.0:8080term2: start the proxy
$ sudo ip netns exec proxy ./proxy
proxy start *:6666term3: issue a request to the proxy using nc
$ sudo ip netns exec client bash -c 'echo "all hail the topology!" | nc 10.0.6.10 6666'
all hail the topology!Once the action in term3 is completed we should see the following new lines in term1 and term2
term1:
Ncat: Connection from 10.0.6.10.
Ncat: Connection from 10.0.6.10:53918.term2:
new client: 10.0.5.1We just built a very dumb, yet functional, proxy server which terminates the client connection, opens a new connection to the server, and streams the bits between both client and server sockets.
Notice, from the server's perspective, the proxy is connecting to it.
Additionally, the client connects to the proxy, not to the server.
Recall, in the transparent proxying case, we want the client to connect to the proxy using the echo server's address and port, and we want the proxy to connect to the echo server using the original client's address and port.
The obstacles
For me, it was always easiest to understand how transparent proxying works by first outlining the obstacles it presents in the normal Linux network stack.
Let's start at the client who we want to send a request directly to the server 10.0.6.1:8080.
We can do this right now, since we have end-to-end connectivity, and it will work but we bypass the proxy altogether, not what we want.
Consider the current network path for this 10.0.6.1:8080 packet sent from a client which bypasses the proxy:
┌──────────────┐
│proxy-a (veth)│
└──────┬───────┘
│
┌────────────▼─────────────┐
│ layer 2 │
│ __netif_receive_skb_core │
└────────────┬─────────────┘
│
┌────────────▼─────────────┐
│ layer 3 │
│ ip_rcv │
│ ip_rcv_finish_core │
│ ip_route_input_noref │
│ ip_forward │
└────────────┬─────────────┘
│
┌──────▼───────┐
│proxy-b (veth)│
└──────────────┘ The packet is received on proxy-a veth inside the proxy network namespace.
The primary layer 2 function handles the packet, determines it's an IPv4 protocol packet, and calls into ip_rcv.
ip_rcv is the ingress IP packet handling function which does a bit of prep work and quickly calls into ip_rcv_finish_core.
In the core function a route lookup is done on the packet, this "host" doesn't have the 10.0.6.1 IP assigned to any of its interfaces, but it does have a route to it found by calling ip_route_input_noref.
Because the kernel has determined it can be routed the ip_forward function is called to eventually transmit the packet out the interface toward its next hop.
Herein lies the obstacles transparent proxying must overcome.
The routing layer must accept the packet for local delivery, not forward it.
Once accepted for local delivery, the proxy needs to deliver it to the proxy's TCP socket bound to a port that does not match the packet's destination.
Once the client side is connected the proxy must connect to the server with the original client's address and port.
The return traffic must pass through the proxy to ensure TCP connections are properly maintained.
Let's tackle these individually.
Obstacle 1: Tricking the routing layer into local delivery
It's not really a trick, more of a common practice used in a not-so-common way.
Firstly, we can force a packet, based on its destination, to be locally delivered with a specific routing table entry.
This is what the local keyword does when adding a route with ip route.
Let's take this route for example:
ip route add local 0.0.0.0/0 dev loThis route would take all traffic and accept it as local delivery.
Of course, it's not that simple, since this will also route traffic leaving the host, back into the network stack, which is not what we want.
We will need to use policy routing to select the traffic we want to force local delivery for.
This is done with a combination of packet marking, IP rules, and a dedicated routing table.
We can configure iptables to mark all packets destined for the echo server 10.0.6.1 with the value 0x1.
Next, we can create a policy routing rule with the ip rule tool, telling the kernel to use routing table 100 when performing route lookups for packets with the 0x1 mark.
Finally, we add the route we discussed earlier into table 100, instructing the kernel to deliver these packets locally, even if the IP address is not present on the host.
diff --git a/Makefile b/Makefile
index ef27ffb..1ce58e5 100644
--- a/Makefile
+++ b/Makefile
@@ -41,6 +41,12 @@ topology:
$(NETNS_EXEC) $(NETNS_PROXY) sysctl -w net.ipv4.ip_forward=1
$(NETNS_EXEC) $(NETNS_PROXY) sysctl -w net.ipv4.conf.all.proxy_arp=1
+ # configure policy routing for proxy namespace, marking traffic destined for echo server,
+ # and routing this traffic into the host for local delivery.
+ $(NETNS_EXEC) $(NETNS_PROXY) iptables -t mangle -A PREROUTING -d 10.0.6.1 -j MARK --set-mark 1
+ $(NETNS_EXEC) $(NETNS_PROXY) ip rule add fwmark 1 table 100
+ $(NETNS_EXEC) $(NETNS_PROXY) ip route add table 100 local 0.0.0.0/0 dev lo
+
topology-destroy:
$(NETNS) del $(NETNS_CLIENT)
$(NETNS) del $(NETNS_PROXY)The above diff in the Makefile does exactly this.
If we now rebuild the topology, start the echo and proxy servers, and issue the client request, you will see it just hang.
This is because, despite being accepted for local delivery by the routing subsystem, layer 4 cannot find a socket for the ingress packet.
Obstacle 2: Layer 4 socket delivery
To deliver the packet to the proxy, we must hijack layer 4 delivery.
The current socket state in the proxy network namespace results in a dropped packet.
┌──────────────────────────────┐
│ proxy ns │
┌──────────────────────────┐│ ┌───────────────────┐ │
│10.0.5.1 -> 10.0.6.10:8080┼┼────► layer 2 │ │
└──────────────────────────┘│ └─────────┬─────────┘ │
│ ┌─────────▼─────────┐ │
│ │ layer 3 │ │
│ └─────────┬─────────┘ │
│ ┌─────────▼─────────┐ │
│ │ layer 4 (DROP)│ │
└────└┌─────────────────┐┴─────┘
│listening *:6666 │
└─────────────────┘ We can accept the packet now for local delivery and get past layer 3.
However, layer 4 will attempt to lookup a listening socket bound to either 10.0.6.10:8080 or the wild card *:8080 which does not exist in this namespace and drop the packet.
We need a form of socket redirection which does not rely on modifying the packet's destination port, like a NAT function would.
This exists and is primarily implemented by the TPROXY iptables target, though, it requires a change in the application code as well.
For TPROXY redirection to work we must do two things:
Introduce a new
iptablesrule which redirects the marked traffic to the proxy port for deliveryEnsure the proxy's listening socket is set to
IP_TRANSPARENT
diff --git a/Makefile b/Makefile
index 1ce58e5..8840527 100644
--- a/Makefile
+++ b/Makefile
@@ -47,6 +47,10 @@ topology:
$(NETNS_EXEC) $(NETNS_PROXY) ip rule add fwmark 1 table 100
$(NETNS_EXEC) $(NETNS_PROXY) ip route add table 100 local 0.0.0.0/0 dev lo
+ # configure iptables to intercept marked packets toward the echo server and
+ # deliver to the proxy's port.
+ $(NETNS_EXEC) $(NETNS_PROXY) iptables -t mangle -A PREROUTING -p tcp -m mark --mark 1 -j TPROXY --on-port 6666
+
topology-destroy:
$(NETNS) del $(NETNS_CLIENT)
$(NETNS) del $(NETNS_PROXY)
diff --git a/proxy.c b/proxy.c
index 1f51994..a1be617 100644
--- a/proxy.c
+++ b/proxy.c
@@ -65,6 +65,12 @@ int main(int argc, char* argv[]) {
exit(EXIT_FAILURE);
}
+ int one = 1;
+ if (setsockopt(sock, SOL_IP, IP_TRANSPARENT, &one, sizeof(one)) < 0) {
+ perror("setsockopt");
+ exit(EXIT_FAILURE);
+ }
+
struct sockaddr_in addr = {
.sin_addr = 0, .sin_port = htons(PROXY_PORT), .sin_family = AF_INET};
if (bind(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0) {The newly introduced TPROXY rule runs in the mangle table, prior to any routing decision being made on the packet.
It will run for any packets with the 0x1 mark and will transparently redirect the packet to the proxy's listening port.
Next, we update the proxy's code to enable IP_TRANSPARENT on the listening socket.
This allows the proxy to accept connections for destinations it is not bound to, exactly what we need to accept a SYN packet destined to the echo server's 10.0.6.1 address.
If we re-deploy the topology and send a request, guess what?
Everything will work!
term2: proxy
$ sudo ip netns exec proxy ./proxy
proxy start *:6666
new client: 10.0.5.1term3: echo server
$ sudo ip netns exec server ncat -v -l -k -p 8080 --exec /usr/bin/cat
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Listening on :::8080
Ncat: Listening on 0.0.0.0:8080
Ncat: Connection from 10.0.6.10.
Ncat: Connection from 10.0.6.10:45318.term1, the client, is omitted since we simply run the same command as previously shown.
The client's packets were locally delivered and redirected into the proxy's listening socket.
Now, the kernel is smart here, and when it accepts the connection with the client, sending the SYN,ACK packet back, it uses the original destination as its source.
We can see this with tcpdump in the client's namespace.
$ sudo ip netns exec client tcpdump -i any -nnn
18:45:07.222064 client-a Out IP 10.0.5.1.54904 > 10.0.6.1.8080: Flags [S], seq 3819568740, win 64240, options [mss 1460,sackOK,TS val 479682827 ecr 0,nop,wscale 10], length 0
18:45:07.222247 client-a In IP 10.0.6.1.8080 > 10.0.5.1.54904: Flags [S.], seq 4110014804, ack 3819568741, win 65160, options [mss 1460,sackOK,TS val 3553422169 ecr 479682827,nop,wscale 10], length 0
18:45:07.222299 client-a Out IP 10.0.5.1.54904 > 10.0.6.1.8080: Flags [.], ack 1, win 63, options [nop,nop,TS val 479682828 ecr 3553422169], length 0In the above dump we see the client sending a SYN to the echo server.
Once the proxy accepts the incoming packet, because the connection is over a transparent socket, the kernel is smart and will send the SYN,ACK and all subsequent traffic sourced as the destination of the SYN.
What we've built so far is an example of a "non-fully transparent proxy", for lack of a better term.
We are transparent for the client, but the server still sees the traffic originating from the proxy, not the original client.
Let's make the proxy fully transparent next.
Obstacle 3 and 4: transparent server connect
In the previous sections we have the proxy performing client side transparency, but the connection to the server is still made from the proxy's source address.
Typically, Linux will not allow an application to bind to an address that's not assigned to an interface on the host.
This can be, yet again, overcome with the IP_TRANSPARENT socket option coupled with the bind function call.
However, this is not enough.
Consider, the reply traffic entering the proxy's network namespace now.
┌────────────────────────────┐
│ │
│ proxy ns │
│ │ echo server response
│ ┌─────────────────┐ │ ┌─────────────────────────────────┐
│ │ layer 2 ◄─────┼────┼ 10.0.6.10:8080 -> 10.0.5.1:1234 │
│ └────────┬────────┘ │ └─────────────────────────────────┘
│ ┌────────▼────────┐ │
◄─────┼────┼ layer 3 │ │
│ └─────────────────┘ │
│ ┌─────────────────┐ │
│ │ layer 4 │ │
│ └─────────────────┘ │
│ │
└────────────────────────────┘ The reply goes through layer 2 into layer 3 where it is simply routed out.
This is because there is no mechanism to deliver the return traffic locally and perform the subsequent socket redirection required.
The return traffic is unique however, since a socket which matches the return packets will exist.
We can exploit this fact and use a socket iptables match function.
The socket match function will inspect the incoming tuple and determine if a socket lookup for it would successfully return a socket, which in our reply case above, it would.
We can then pair this match with a MARK target to mark the packet for local delivery, using the same exact policy routing mechanisms we already configured.
Once delivered locally, layer 4 will attempt the same socket lookup the socket match function did, find the transparent socket, and deliver it to the transparent proxy.
diff --git a/Makefile b/Makefile
index 8840527..89e07f0 100644
--- a/Makefile
+++ b/Makefile
@@ -50,6 +50,7 @@ topology:
# configure iptables to intercept marked packets toward the echo server and
# deliver to the proxy's port.
$(NETNS_EXEC) $(NETNS_PROXY) iptables -t mangle -A PREROUTING -p tcp -m mark --mark 1 -j TPROXY --on-port 6666
+ $(NETNS_EXEC) $(NETNS_PROXY) iptables -t mangle -A PREROUTING -p tcp -m socket --transparent -j MARK --set-mark 1
topology-destroy:
$(NETNS) del $(NETNS_CLIENT)
$(NETNS) del $(NETNS_PROXY)
diff --git a/proxy.c b/proxy.c
index a1be617..3f06a26 100644
--- a/proxy.c
+++ b/proxy.c
@@ -11,7 +11,7 @@
#define PROXY_TARGET_IP 0x0a000601
#define MAX_MSG_SIZE 1024 * 1024
-int proxy_do(int client_sock) {
+int proxy_do(int client_sock, struct sockaddr_in *client) {
struct sockaddr_in server_addr = {.sin_addr = htonl(PROXY_TARGET_IP),
.sin_port = htons(PROXY_TARGET_PORT),
.sin_family = AF_INET};
@@ -25,6 +25,17 @@ int proxy_do(int client_sock) {
return 0;
}
+ int one = 1;
+ if (setsockopt(server_sock, SOL_IP, IP_TRANSPARENT, &one, sizeof(one)) < 0) {
+ perror("setsockopt");
+ goto cleanup;
+ }
+
+ if(bind(server_sock, (struct sockaddr *)client, sizeof(struct sockaddr_in)) < 0 ) {
+ perror("bind");
+ goto cleanup;
+ }
+
if (connect(server_sock, (struct sockaddr*)&server_addr,
sizeof(server_addr)) < 0) {
perror("proxy_do: connect");
@@ -102,7 +113,7 @@ int main(int argc, char* argv[]) {
}
printf("new client: %s\n", buf);
- proxy_do(client_sock);
+ proxy_do(client_sock, &client);
close(client_sock);
}
}Rebuild the proxy and redeploy the topology.
Issuing a new client request will now show a complete transparent proxy flow!
The interesting bit is the output of the echo server
term3: connection received from proxy, using client's address.
Ncat: Connection from 10.0.5.1.
Ncat: Connection from 10.0.5.1:44730.Congratulations you have a fully transparent proxy server!
TPROXY internals
We now understand the full end-to-end configuration of transparent proxying in the Linux kernel.
Let's turn our attention to how the crux of the operation works, the TPROXY target.
The TPROXY target is implemented within the kernel's source code located in the file xt_TPROXY.c.
The function which is invoked when the target is expressed follows.
xt_TPROXY.c
static unsigned int
tproxy_tg4(struct net *net, struct sk_buff *skb, __be32 laddr, __be16 lport,
u_int32_t mark_mask, u_int32_t mark_value)
{
const struct iphdr *iph = ip_hdr(skb);
struct udphdr _hdr, *hp;
struct sock *sk;
hp = skb_header_pointer(skb, ip_hdrlen(skb), sizeof(_hdr), &_hdr);
if (hp == NULL)
return NF_DROP;
/* check if there's an ongoing connection on the packet
* addresses, this happens if the redirect already happened
* and the current packet belongs to an already established
* connection */
sk = nf_tproxy_get_sock_v4(net, skb, iph->protocol,
iph->saddr, iph->daddr,
hp->source, hp->dest,
skb->dev, NF_TPROXY_LOOKUP_ESTABLISHED);
laddr = nf_tproxy_laddr4(skb, laddr, iph->daddr);
if (!lport)
lport = hp->dest;
/* UDP has no TCP_TIME_WAIT state, so we never enter here */
if (sk && sk->sk_state == TCP_TIME_WAIT)
/* reopening a TIME_WAIT connection needs special handling */
sk = nf_tproxy_handle_time_wait4(net, skb, laddr, lport, sk);
else if (!sk)
/* no, there's no established connection, check if
* there's a listener on the redirected addr/port */
sk = nf_tproxy_get_sock_v4(net, skb, iph->protocol,
iph->saddr, laddr,
hp->source, lport,
skb->dev, NF_TPROXY_LOOKUP_LISTENER);
/* NOTE: assign_sock consumes our sk reference */
if (sk && nf_tproxy_sk_is_transparent(sk)) {
/* This should be in a separate target, but we don't do multiple
targets on the same rule yet */
skb->mark = (skb->mark & ~mark_mask) ^ mark_value;
nf_tproxy_assign_sock(skb, sk);
return NF_ACCEPT;
}
return NF_DROP;
}
static unsigned int
tproxy_tg4_v0(struct sk_buff *skb, const struct xt_action_param *par)
{
const struct xt_tproxy_target_info *tgi = par->targinfo;
if (par->fragoff)
return NF_DROP;
return tproxy_tg4(xt_net(par), skb, tgi->laddr, tgi->lport,
tgi->mark_mask, tgi->mark_value);
}The tproxy_tg4_v0 function exists solely to extract the arguments of the TPROXY target and supply them to tproxy_tg4 proper.
Consider the target used in this scenario.
iptables -t mangle -A PREROUTING -p tcp -m mark --mark 1 -j TPROXY --on-port 6666We will be supplying the tgi->lport argument to tproxy_tg4 given the --on-port target argument translation.
We won't be supplying either of the tgi->mark_mask or tgi->laddr arguments, though it is nice to know they exist:
--on-ip address
This specifies a destination address to use. By default the address is the IP address of the incoming interface. This is only valid if
the rule also specifies -p tcp or -p udp.
--tproxy-mark value[/mask]
Marks packets with the given value/mask. The fwmark value set here can be used by advanced routing. (Required for transparent proxying
to work: otherwise these packets will get forwarded, which is probably not what you want.)Let's turn our attention to the tproxy_tg4 function now.
1.
const struct iphdr *iph = ip_hdr(skb);
struct udphdr _hdr, *hp;
struct sock *sk;
hp = skb_header_pointer(skb, ip_hdrlen(skb), sizeof(_hdr), &_hdr);
if (hp == NULL)
return NF_DROP;
/* check if there's an ongoing connection on the packet
* addresses, this happens if the redirect already happened
* and the current packet belongs to an already established
* connection */
sk = nf_tproxy_get_sock_v4(net, skb, iph->protocol,
iph->saddr, iph->daddr,
hp->source, hp->dest,
skb->dev, NF_TPROXY_LOOKUP_ESTABLISHED);The above gets a pointer to the packet's header and stores it into a struct udphdr, which is used regardless of whether layer 4 is UDP or TCP, since both protocols' wire formats start with the source and destination ports.
struct tcphdr {
__be16 source;
__be16 dest;
...
struct udphdr {
__be16 source;
__be16 dest;
...Next, a socket lookup is attempted for the current packet header details.
This socket lookup is done to find connected sockets that were already accepted by our transparent proxy and used to forward the request to the server.
2.
laddr = nf_tproxy_laddr4(skb, laddr, iph->daddr);
if (!lport)
lport = hp->dest;Next, the laddr and lport arguments are resolved, preferring the arguments to the TPROXY target or using the packet's wire data in lieu of the former.
3.
/* UDP has no TCP_TIME_WAIT state, so we never enter here */
if (sk && sk->sk_state == TCP_TIME_WAIT)
/* reopening a TIME_WAIT connection needs special handling */
sk = nf_tproxy_handle_time_wait4(net, skb, laddr, lport, sk);
else if (!sk)
/* no, there's no established connection, check if
* there's a listener on the redirected addr/port */
sk = nf_tproxy_get_sock_v4(net, skb, iph->protocol,
iph->saddr, laddr,
hp->source, lport,
skb->dev, NF_TPROXY_LOOKUP_LISTENER);Let's brush over the TIME_WAIT complexities for now.
If no socket was found, this is a new packet never seen by our transparent proxy before.
At this point, the kernel will perform a socket lookup using the arguments provided to the target.
This lookup will find the transparent proxy's listening socket, noting the NF_TPROXY_LOOKUP_LISTENER argument to nf_tproxy_get_sock_v4.
Because the proxy is listening on *:6666 and the lookup is being done with lport == 6666, the proxy's listening socket will be found.
4.
/* NOTE: assign_sock consumes our sk reference */
if (sk && nf_tproxy_sk_is_transparent(sk)) {
/* This should be in a separate target, but we don't do multiple
targets on the same rule yet */
skb->mark = (skb->mark & ~mark_mask) ^ mark_value;
nf_tproxy_assign_sock(skb, sk);
return NF_ACCEPT;
}Finally, if a socket was found previously, and the socket is marked transparent, an optional mark value is applied and, crucially, the skb is assigned the socket.
Once the skb is assigned the socket, it will now continue to layer 3, where our policy routing rules exist to force it into the local delivery path.
The skb then continues to layer 4, but now a socket is already assigned to the skb.
This socket will be used for subsequent delivery to an application, finalizing the usage of the TPROXY target.
Cleaning things up
For the sake of a clear explanation I took the longer route to achieve the full transparent proxy demo.
For one, its so common to mark packets for TPROXY that the TPROXY target can do this on its own.
We can remove the MARK rule and tell TPROXY to do the marking for us, collapsing the two rules for the client side leg to one.
diff --git a/Makefile b/Makefile
index 89e07f0..87f25be 100644
--- a/Makefile
+++ b/Makefile
@@ -43,13 +43,12 @@ topology:
# configure policy routing for proxy namespace, marking traffic destined for echo server,
# and routing this traffic into the host for local delivery.
- $(NETNS_EXEC) $(NETNS_PROXY) iptables -t mangle -A PREROUTING -d 10.0.6.1 -j MARK --set-mark 1
$(NETNS_EXEC) $(NETNS_PROXY) ip rule add fwmark 1 table 100
$(NETNS_EXEC) $(NETNS_PROXY) ip route add table 100 local 0.0.0.0/0 dev lo
# configure iptables to intercept marked packets toward the echo server and
# deliver to the proxy's port.
- $(NETNS_EXEC) $(NETNS_PROXY) iptables -t mangle -A PREROUTING -p tcp -m mark --mark 1 -j TPROXY --on-port 6666
+ $(NETNS_EXEC) $(NETNS_PROXY) iptables -t mangle -A PREROUTING -p tcp -d 10.0.6.1 -j TPROXY --on-port 6666 --tproxy-mark 1
$(NETNS_EXEC) $(NETNS_PROXY) iptables -t mangle -A PREROUTING -p tcp -m socket --transparent -j MARK --set-mark 1
topology-destroy:This is optional of course, but is preferred, especially on nodes with a lot of iptables rules.
Summing things up
I always enjoy going over the transparent proxy flow.
It's a clinic on how to think about Linux networking, or networking in general, in layers.
Once we start to understand that each layer of the network stack has a set of rules, we can start to understand how to bend them to get to a solution like transparent proxying.