Error 500 when fetching behind squid proxy

[eddysureda]

I have this error with PulledPork version 0.7.3 and -W option:

Checking latest MD5 for snortrules-snapshot-2990.tar.gz....
Fetching md5sum for: snortrules-snapshot-2990.tar.gz.md5
** CONNECT https://www.snort.org/rules/snortrules-snapshot-2990.tar.gz.md5?oinkcode=MYOINKCODE==> 500 Can't connect to 172.16.1.5:3128
Error 500 when fetching https://www.snort.org/reg-rules/snortrules-snapshot-2990.tar.gz.md5 at /usr/local/bin/pulledpork.pl line 547.

My environment variables are fixed as:
http_proxy=http://172.16.1.5:3128
https_proxy=https://172.16.1.5:3128

Squid version: 3.1.19 with method CONNECT allowed

when I downloaded it with wget, It was succesfully.

Here are the tcdump captures of wget and pulledpork:
pulledpork.pcap.gz
wget.pcap.gz

ah here is the squid logs too:

access.txt

[vrtadmin]

Probably want to remove your oinkcode from the description.

[eddysureda]

Thanks for your advice vrtadmin. I removed my oinkcode from description, but I'm still having the problem.

[lravelo]

I'm having this same issue. Only my squid proxy generates this in the logs:

TCP_DENIED/400 1494 NONE NONE:// - NONE/- text/html ( squid 2.7.STABLE5 )
NONE/400 4178 NONE error:invalid-request - HIER_NONE/- text/html ( squid 3.3.8 )

[jescholl]

@eddysureda It looks like your proxy doesn't support inbound TLS. I would bet that changing your proxy environment variables to the following would fix it.

http_proxy=http://172.16.1.5:3128
https_proxy=http://172.16.1.5:3128

The proxy environment variables just split connections so that http requests goes to one proxy server, and https requests can go to another (they're usually the same though). The protocol that is used to communicate with the proxy itself is usually just plain HTTP (without TLS).

If you look at the pcap from wget, you can see the request that wget makes to the proxy in plain text.

CONNECT www.snort.org:443 HTTP/1.1
User-Agent: Wget/1.16 (linux-gnu)
Host: www.snort.org:443

This looks almost identical to a standard HTTP request, except that it uses the CONNECT verb which tells the proxy to open a tunnel to the destination server. The proxy opens the tunnel and responds with HTTP/1.1 200 Connection established, telling wget that it can now begin a normal TLS handshake with the destination server (not the proxy).

If you look at the pcap from pulledpork, you can see the standard TCP 3-way handshake (same as from wget), then the client sends a PSH, ACK (ACKnowledging the previous SYN,ACK and PUSHing some additional data). I can't tell for sure from the pcap, but I suspect this is the client attempting to begin the TLS handshake, to which the server responds with 400 Bad Request because it was expecting an HTTP request.

Setting your proxy environment variable to https://proxy:3128 requires the client to negotiate TLS with the proxy, before it can pass on the request to the destination server. Using this configuration, TLS will fail unless the proxy is covered by the cert used by the destination server. More information about this feature can be found in the squid docs, but in most cases you just want to use http://proxy:3128.

[fauzigo]

Hi,

Just to add my 2 cents here.

I was having a few issues with the script. It worth noting that I'm also behind squid proxy.

I was getting 400s errors when the script CONNECTS to the URLs where files were supposed to be downloaded. Looking at the code I noticed that there is a condition on where to use CONNECT or GET depending if you use a proxy or not. The script itself won't allow you to use either one of those methods.

I forced the script to use GET instead of CONNECT even though I'm behind a SQUID and the condition is true and CONNECT should be used (Commented out lines 400-404). After that files were downloaded seamlessly.

In my case GET was the solution, it might worth adding the option as argument or something that would let you elect which method to use (it already says May need to be addressed in the future right above though).

Regards,

[jescholl]

@fauzigo Are you saying that pulledpork was incorrectly sending a CONNECT to the proxy server for http:// URLs?

I'd be curious to see a pcap of the 400's you were getting and one after applying your fix.

[fauzigo]

I think I never said such thing as pulled pork did something incorrect, that would be a false allegation.

What I said instead, was that the way the pulledpork.pl script was trying to pull the .gz and .gz.md5 in wasn't working of my setup, which is basically behind Squid Cache-Proxy.

The script checks if you are behind a proxy by checking your ENV vars (iirc), and thereafter decides whether to use CONNECT or GET. If the script finds out you are behind a proxy it'll use CONNECT.

When I bypassed the check and forced the script to use GET, the files were downloaded just fine and the script executed without any error.

The 400 errors I was getting were mostly my proxy returning codes that the CONNECTs didn't like and the execution errored out. I think it has something to do with the fact that the URLs you set in pulledpork.conf end up redirecting to S3 buckets in AWS, the combination of all that breaks the connection and ultimately failing.

It worth noting that URLs were HTTPS, not HTTP.

Regards,

[jescholl]

Sorry, "incorrect" was my word because I thought you were saying that it was sending a CONNECT to an HTTP url (which would be incorrect).

The redirect to S3 shouldn't matter unless you're whitelisting specific URLs in Squid and the S3 endpoint isn't whitelisted.

Squid supports all standard HTTP methods (GET, POST, HEAD, etc.) for HTTP requests, but for HTTPS requests it only supports CONNECT. So, if you are requesting the .gz, and .gz.md5 files through Squid over HTTPS, it should be sending a CONNECT to Squid, and a GET to the end server once squid has set up tunnel.

The only reasons I can think of that pulledpork might be working as you describe are that somehow your proxy environment variables aren't being interpreted correctly, or for some reason it is sending a CONNECT both to Squid (as it should), and the end server.

In either case, it sounds like there may be a bug in there and it would be a huge help to see the proxy environment variables you're setting as well as a pcap of the connection working with your modification and failing without it.