Transparently proxying any tcp connection (including http and https without man-in-the-middle)

I recently ran into a situation where a proxy server was being installed, but some applications were not proxy aware. To illustrate, this is what the network looked like:

Here we have a web server running proxy unaware applications. The web server does not have routes to the internet; it must use a proxy server to communicate with web sites located on the public internet. The solution is to intercept all HTTP (port 80) and HTTPS (port 443) traffic at the firewall. For http, we'll use squid and for https, we'll use a custom program I wrote. First up, http. In vyatta, configure the webproxy with:

set service webproxy listen-address 10.18.11.5
commit
save

The squid service will start up after committing the changes, but some changes need to be made to the config file. Edit /etc/squid3/squid.conf to look like this:

#
# autogenerated by vyatta-update-webproxy.pl
#

acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl net src all
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT

http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access allow net
http_access deny all

cache_dir ufs /var/spool/squid3 100 16 256
cache_mem 20 MB
access_log /var/log/squid3/access.log squid

cache_store_log none

http_port 10.18.11.5:3128 transparent

forwarded_for off

# added by Ryan
visible_hostname fw01-1.web.lan
cache_mgr [email protected]
# Do not send the following domains to the proxy
acl DIRECTS dstdomain .lan
never_direct allow all
always_direct allow DIRECTS
nonhierarchical_direct off
# Stop sending requests to a peer if it does not respond within 5 seconds
peer_connect_timeout 5 seconds
# When all peers are available, balance requests between proxy1/2.  If those two go down, send requests to the lastresort
cache_peer 1.2.3.4 parent 80 0 proxy-only no-query name=proxy1 round-robin weight=999999
cache_peer 1.2.3.5 parent 80 0 proxy-only no-query name=proxy2 round-robin weight=999999
cache_peer 1.2.3.6 parent 80 0 proxy-only no-query name=proxy-lastresort round-robin weight=1

Restart squid3 with /etc/init.d/squid3 restart. Note that you'll want to edit vyatta-update-webproxy.pl to make sure the changes we placed in the squid config stay around after a re-commit (this was the best suggestion on the vyatta forums). A http request from the web server (10.18.11.40) will now be forwarded to the proxies transparently:

[root@web01 ~]# nc www.google.com 80
HEAD / HTTP/1.0

HTTP/1.0 200 OK
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Date: Sun, 07 Apr 2013 21:35:13 GMT
Server: gws
Expires: -1
Set-Cookie: NID=67=DUffxHjxxlJbUCbh58qhFLVJffrvGVX8lMm96QaFzfwjFyGNz4VlHPziZUozs0zh9cUwzxWX-WSftJ-JCPC652SYCVP5lXHIGCYXBabW3A8J1toyxBe1n7Lu8oqmiXEV; expires=Mon, 07-Oct-2013 21:35:13 GMT; path=/; domain=.; HttpOnly
Content-Type: text/html; charset=ISO-8859-1
Cache-Control: private, max-age=0
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Cache: MISS from fw01-1.web.lan
X-Cache-Lookup: MISS from fw01-1.web.lan:3128
Via: 1.0 fw01-1.web.lan (squid/3.1.6)
Connection: close

[root@web01 ~]#

Second, https. This one is a little trickier. The squid solutions I found on the internet suggested using a man-in-the-middle strategy, where squid decrypts the SSL/TLS data and then re-encrypts it. I wasn't fond of this method because not only it is technically difficult to pull off, it raises some ethical issues. Another way is to transparently do what proxy aware applications perform:

Application wants to connect to www.google.com on port 443. It first checks the environment for the variable https_proxy. Let's assume https_proxy=http://1.2.3.4:80
Application makes connection to proxy server at 1.2.3.4 on port 80
Application sends "CONNECT www.google.com:443 HTTP/1.1\r\nHost:www.google.com\r\n\r\n"
Proxy server responds "HTTP/1.0 200 Connection established"
Application speaks TLS as if it had directly connected to www.google.com:443 originally

I've written a program in Golang called any_proxy, available on Github that can transparently proxy https (there is also a newer version called any_proxy which supports proxying of any tcp protocol). Here is a network timing diagram to illustrate what's going on:

Compiling:

Make sure you have golang installed on your machine. Binaries are available from golang.org
Clone the go-any-proxy repo with git clone git://github.com/ryanchapman/go-any-proxy.git
Compile with ./make.bash

Installing:

Secure copy any_proxy to your vyatta:
```
scp any_proxy fw01-1.web.lan:/root
```
Ssh to your vyatta:
```
ssh [email protected]
```
Make sure you already set up the squid webproxy earlier. Add a firewall rule to have 443 traffic redirected to any_proxy:
```
iptables -t nat -A WEBPROXY -p tcp --dport 443 redir ports 3129
```

Start the proxy:

/root/any_proxy -l :3129 -p "1.2.3.4:80,1.2.3.5:80,1.2.3.6:80"

You should now be able to connect to a site with https now:

[root@web01 ~]# openssl s_client -host www.google.com -port 443
CONNECTED(00000003)
depth=2 C = US, O = Equifax, OU = Equifax Secure Certificate Authority
verify return:1
depth=1 C = US, O = Google Inc, CN = Google Internet Authority
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google Inc, CN = www.google.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
   i:/C=US/O=Google Inc/CN=Google Internet Authority
 1 s:/C=US/O=Google Inc/CN=Google Internet Authority
   i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDgDCCAumgAwIBAgIKKZxzswABAACCpzANBgkqhkiG9w0BAQUFADBGMQswCQYD
VQQGEwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzEiMCAGA1UEAxMZR29vZ2xlIElu
dGVybmV0IEF1dGhvcml0eTAeFw0xMzAzMjcxMzI4MjJaFw0xMzEyMzExNTU4NTBa
MGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1N
b3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgSW5jMRcwFQYDVQQDEw53d3cu
Z29vZ2xlLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA1zqnIRAV0QRL
49U0b7wi+rF0wdkY3BcWHdmYgBmF81QH6854F8NBFdECnPOpLpSIcB+r7NjK4Gj6
PQ0Kfg1BkqBC5fy1D5QeW5MOMuftbraURLu9LrELBG/DN2K5KVegQE60yitLOhU/
wSc/ac5sEdsfx6T1HaPm4eByQ5S8fhcCAwEAAaOCAVEwggFNMB0GA1UdJQQWMBQG
CCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUeTiJyDmB+bnMsRpBVwCCswsj
JLEwHwYDVR0jBBgwFoAUv8Aw6/VDET5nup6R+/xq2uNrEiQwWwYDVR0fBFQwUjBQ
oE6gTIZKaHR0cDovL3d3dy5nc3RhdGljLmNvbS9Hb29nbGVJbnRlcm5ldEF1dGhv
cml0eS9Hb29nbGVJbnRlcm5ldEF1dGhvcml0eS5jcmwwZgYIKwYBBQUHAQEEWjBY
MFYGCCsGAQUFBzAChkpodHRwOi8vd3d3LmdzdGF0aWMuY29tL0dvb2dsZUludGVy
bmV0QXV0aG9yaXR5L0dvb2dsZUludGVybmV0QXV0aG9yaXR5LmNydDAMBgNVHRMB
Af8EAjAAMBkGA1UdEQQSMBCCDnd3dy5nb29nbGUuY29tMA0GCSqGSIb3DQEBBQUA
A4GBAKAluLPebUSSuwF2wyablQT08GGyQQ0A+SW8gQXy7obaXpQJa/gw5BaZz64w
qMs4Bhmw7AECfq4nPDfY0k9eC4sIHOO95MwGXcH+14IFFDiF4o2f1q1SQQIbdS19
eepmrzYfjNn6KM0dahsnSf1zViIRYdu9gZ3HVwwfif1foidR
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
issuer=/C=US/O=Google Inc/CN=Google Internet Authority
---
No client certificate CA names sent
---
SSL handshake has read 1887 bytes and written 298 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: 20100F0A1526BDADC3E7E278D2925738CAC56CB49A209CB0F3E036FFD57EEFF5
    Session-ID-ctx:
    Master-Key: FE00D81636D02CF0259B4945A16B37A0ACF267918635D23D9034F46886993C0291A0E639573534413FC176D9A4E1253E
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 100800 (seconds)
    TLS session ticket:
    0000 - 1c f0 4d b2 70 a7 14 13-a4 e4 25 f4 dd 29 c8 52   ..M.p.....%..).R
    0010 - 96 22 0f a5 c7 2f e0 c5-ef 99 db 1c e8 8d 66 a8   .".../........f.
    0020 - c4 b1 c1 87 12 69 10 a3-c1 a8 7e 8f 32 3e 03 9e   .....i....~.2>..
    0030 - 97 8b 7e a5 b4 64 75 50-69 98 f7 da 11 d8 83 20   ..~..duPi.......
    0040 - 67 3a dc 66 11 8a b0 08-61 3b 94 53 7e c4 2e e5   g:.f....a;.S~...
    0050 - 9d dd 38 34 e6 de 42 69-88 a2 d4 8c be 94 aa 38   ..84..Bi.......8
    0060 - 2d be f3 b7 81 93 48 48-fc 53 af d3 98 87 c8 a8   -.....HH.S......
    0070 - d0 41 dc ab 1e fd 5f 05-34 ac 2f d3 b4 ec 43 c1   .A...._.4./...C.
    0080 - ac 84 0b ce 6f 98 6c 0c-3a d7 03 d7 40 f2 33 5d   ....o.l.:[email protected]]
    0090 - d3 cd 75 f5                                       ..u.

    Start Time: 1365373842
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
HEAD / HTTP/1.0

HTTP/1.0 200 OK
Date: Sun, 07 Apr 2013 22:30:47 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=e458ec6f20965cef:FF=0:TM=1365373847:LM=1365373847:S=IEPlOV-aQ9f_RF-u; expires=Tue, 07-Apr-2015 22:30:47 GMT; path=/; domain=.google.com
Set-Cookie: NID=67=BlzgMe73-RN5tlryHoZcQGdRxrJfJpwNkBjfyJ-Zl-0ucm4HwkzpjPIVyTFR_IFqvCoCPOuHgEa566XAGfmCYI1nQgydCjRoqM8-wu__n-H7nIC5nt1JukFP1YtLc0BL; expires=Mon, 07-Oct-2013 22:30:47 GMT; path=/; domain=.google.com; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

read:errno=0
[root@web01 ~]#

Of course, you'll want to make sure the services start on reboot. That is left as an exercise for the reader. Feel free to leave comments or send pull requests with any improvements to the code.