Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support custom CONNECT headers (both request and response) during HTTPS proxy requests #330

Merged
merged 3 commits into from
Apr 7, 2016

Conversation

smudge
Copy link
Contributor

@smudge smudge commented Apr 6, 2016

Note: Just throwing this idea out there -- I implemented these changes for my own use case, but I'm happy to rework the implementation if it's not deemed suitable for merging. Let me know.

The Issue

Some proxies accept custom headers, for things like switching IPs, excluding certain IPs, or specifying custom behaviors (like retrying requests when certain codes are returned). Over normal HTTP, these can be sent along with the rest of your headers, and the proxy will filter the headers before passing the request along. Likewise, the proxy might inject a custom header into the response (such as one indicating the IP your request was passed through).

However, everything changes over a secured connection. When connecting to a proxy over HTTPS, this gem makes use of HTTP tunneling via the 'CONNECT' method. The underlying request and response are encrypted. As a result, the proxy is unable to read any headers you send with the underlying request or inject headers into the underlying response.

So adding custom proxy-specific headers to a HTTPS request looks something like this:

CONNECT example.com:1234 HTTP/1.1
Proxy-Authorization: Digest [your auth digest]
X-My-Custom-Header: Yes

<Your Underlying Request Goes Here>

And a response from the proxy (with a custom header) looks like this:

HTTP/1.1 200 Connection established
X-Custom-Proxy-Response: Success

<The Underlying Response Appears Here>

The problem is that, aside from Proxy-Authorization, this gem does not support adding custom headers (like X-My-Custom-Header and X-Custom-Proxy-Response above).

The Change(s)

This PR adds a parameter to the via method which allows you to specify custom headers intended for use by the proxy:

HTTP.via("proxy-hostname.local", 8080, 'x-retry-if' => 502)
    .get("https://example.com/resource")
HTTP.via("proxy-hostname.local", 8080, USERNAME, PASSWORD, 'x-use-ip' => '123.456.11.11')
    .get("https://example.com/resource")

Note that if the underlying request is 'http' instead of 'https', there is no CONNECT tunneling, so the proxy-specific headers will be merged into the requests's primary headers.

Similarly, if the proxy returns any custom headers (as shown above), they are made available as follows:

response = HTTP.via("proxy-hostname.local", 8080).get("https://example.com/resource")

response.proxy_headers
# => { 'x-ip-used' => '123.456.42.42' }

Note: I considered transparently merging these into the main request headers, but I worried that this would cause strange behavior. Either a proxy could overwrite a header you expect from the remote server, or the remote server could overwrite a header you expect from the proxy, and I wasn't sure how to handle these cases. If you have any suggestions I'd be happy to rework the behavior.

Questions?

Does all of this make sense? Let me know if you have any questions, ideas, suggestions, etc.

@zanker
Copy link
Contributor

zanker commented Apr 6, 2016

Seems reasonable to me, @tarcieri @ixti @sferik ?

@tarcieri
Copy link
Member

tarcieri commented Apr 7, 2016

👍

@tarcieri tarcieri merged commit 42733f4 into httprb:master Apr 7, 2016
@tarcieri
Copy link
Member

tarcieri commented Apr 7, 2016

This will go out in 2.0.0 when we eventually release it.

jsonn pushed a commit to jsonn/pkgsrc that referenced this pull request Oct 18, 2016
## 2.0.3 (2016-08-03)

* [#365](httprb/http#365)
  Add `HTTP::Response#content_length`
  ([@janko-m])

* [#335](httprb/http#335),
  [#360](httprb/http#360)
  Set `Content-Length: 0` header for `nil` bodies.
  ([@britishtea])


## 2.0.2 (2016-06-24)

* [#353](httprb/http#353)
  Avoid a dependency cycle between Client and Connection classes.
  ([@jhbabon])


## 2.0.1 (2016-05-12)

* [#341](httprb/http#341)
  Refactor some string manipulations so they are more performant
  (up to 3-4x faster) and more concise.
  ([@tonyta])

* [#339](httprb/http#341)
  Always use byte methods when writing/slicing the write buffer.
  ([@zanker])


## 2.0.0 (2016-04-23)

* [#333](httprb/http#333)
  Fix HTTPS request headline when sent via proxy.
  ([@Connorhd])

* [#331](httprb/http#331)
  Add `#informational?`, `#success?`, `#redirect?`, `#client_error?` and
  `#server_error?` helpers to `Response::Status`.
  ([@mwitek])

* [#330](httprb/http#330)
  Support custom CONNECT headers (request/response) during HTTPS proxy requests.
  ([@smudge])

* [#319](httprb/http#319)
  Drop Ruby 1.9.x support.
  ([@ixti])


## 1.0.4 (2016-03-19)

* [#320](httprb/http#320)
  Fix timeout regression.
  ([@tarcieri])


## 1.0.3 (2016-03-16)

* [#314](httprb/http#314)
  Validate charset before forcing encoding.
  ([@kylekyle])

* [#318](httprb/http#318)
  Remove redundant string allocations upon header names normalization.
  ([@ixti])
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants