-
-
Notifications
You must be signed in to change notification settings - Fork 839
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add httpx.SSLContext
configuration.
#3022
Changes from 42 commits
88f2ab7
58e9fdc
1afe2c9
6be802e
d124d8f
6f1162b
f613dee
55ec002
7346bc2
7f35134
8d5983a
f759dec
7227573
0567aab
2d79640
4d84953
df7bf31
93ca75f
2b7cf9e
fa065dd
f09a823
4eba150
f3b1dc7
998f445
a08ff77
827ae5b
b3673bd
d3a8aab
808a891
289cf9c
6a85f2c
d235e76
1a79bd9
3f9aaa8
46268b7
26de9d0
49545bb
01bb388
13cb957
51ec619
e7c607c
f894a06
cb71af3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,100 +1,224 @@ | ||
When making a request over HTTPS, HTTPX needs to verify the identity of the requested host. To do this, it uses a bundle of SSL certificates (a.k.a. CA bundle) delivered by a trusted certificate authority (CA). | ||
|
||
## Changing the verification defaults | ||
### Enabling and disabling verification | ||
|
||
By default, HTTPX uses the CA bundle provided by [Certifi](https://pypi.org/project/certifi/). This is what you want in most cases, even though some advanced situations may require you to use a different set of certificates. | ||
By default httpx will verify HTTPS connections, and raise an error for invalid SSL cases... | ||
|
||
If you'd like to use a custom CA bundle, you can use the `verify` parameter. | ||
```pycon | ||
>>> httpx.get("https://expired.badssl.com/") | ||
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997) | ||
``` | ||
|
||
```python | ||
import httpx | ||
You can configure the verification using `httpx.SSLContext()`. | ||
|
||
r = httpx.get("https://example.org", verify="path/to/client.pem") | ||
```pycon | ||
>>> ssl_context = httpx.SSLContext() | ||
>>> ssl_context | ||
SSLContext(verify=True) | ||
>>> httpx.get("https://www.example.com", ssl_context=ssl_context) | ||
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997) | ||
``` | ||
|
||
Alternatively, you can pass a standard library `ssl.SSLContext`. | ||
For example, you can use this to disable verification completely and allow insecure requests... | ||
|
||
```pycon | ||
>>> import ssl | ||
>>> import httpx | ||
>>> context = ssl.create_default_context() | ||
>>> context.load_verify_locations(cafile="/tmp/client.pem") | ||
>>> httpx.get('https://example.org', verify=context) | ||
>>> no_verify = httpx.SSLContext(verify=False) | ||
>>> no_verify | ||
SSLContext(verify=False) | ||
>>> httpx.get("https://expired.badssl.com/", ssl_context=no_verify) | ||
<Response [200 OK]> | ||
``` | ||
|
||
We also include a helper function for creating properly configured `SSLContext` instances. | ||
### Configuring client instances | ||
|
||
If you're using a `Client()` instance, then you should pass any SSL settings when instantiating the client. | ||
|
||
```pycon | ||
>>> context = httpx.create_ssl_context() | ||
>>> ssl_context = httpx.SSLContext() | ||
>>> client = httpx.Client(ssl_context=ssl_context) | ||
``` | ||
|
||
The `create_ssl_context` function accepts the same set of SSL configuration arguments | ||
(`trust_env`, `verify`, `cert` and `http2` arguments) | ||
as `httpx.Client` or `httpx.AsyncClient` | ||
The `client.get(...)` method and other request methods on a `Client` instance *do not* support changing the SSL settings on a per-request basis. | ||
|
||
If you need different SSL settings in different cases you should use more that one client instance, with different settings on each. Each client will then be using an isolated connection pool with a specific fixed SSL configuration on all connections within that pool. | ||
|
||
### Changing the verification defaults | ||
|
||
By default, HTTPX uses the CA bundle provided by [Certifi](https://pypi.org/project/certifi/). | ||
|
||
The following all have the same behaviour... | ||
|
||
Using the default SSL context. | ||
|
||
```pycon | ||
>>> import httpx | ||
>>> context = httpx.create_ssl_context(verify="/tmp/client.pem") | ||
>>> httpx.get('https://example.org', verify=context) | ||
>>> client = httpx.Client() | ||
>>> client.get("https://www.example.com") | ||
<Response [200 OK]> | ||
``` | ||
|
||
Or you can also disable the SSL verification entirely, which is _not_ recommended. | ||
Using the default SSL context, but specified explicitly. | ||
|
||
```python | ||
import httpx | ||
```pycon | ||
>>> default = httpx.SSLContext() | ||
>>> client = httpx.Client(ssl_context=default) | ||
>>> client.get("https://www.example.com") | ||
<Response [200 OK]> | ||
``` | ||
|
||
r = httpx.get("https://example.org", verify=False) | ||
Using the default SSL context, with `verify=True` specified explicitly. | ||
|
||
```pycon | ||
>>> default = httpx.SSLContext(verify=True) | ||
>>> client = httpx.Client(ssl_context=default) | ||
>>> client.get("https://www.example.com") | ||
<Response [200 OK]> | ||
``` | ||
|
||
## SSL configuration on client instances | ||
Using an SSL context, with `certifi.where()` explicitly specified. | ||
|
||
If you're using a `Client()` instance, then you should pass any SSL settings when instantiating the client. | ||
```pycon | ||
>>> default = httpx.SSLContext(verify=certifi.where()) | ||
>>> client = httpx.Client(ssl_context=default) | ||
>>> client.get("https://www.example.com") | ||
<Response [200 OK]> | ||
``` | ||
|
||
```python | ||
client = httpx.Client(verify=False) | ||
For some advanced situations may require you to use a different set of certificates, either by specifying a PEM file: | ||
|
||
```pycon | ||
>>> custom_cafile = httpx.SSLContext(verify="path/to/certs.pem") | ||
>>> client = httpx.Client(ssl_context=custom_cafile) | ||
>>> client.get("https://www.example.com") | ||
<Response [200 OK]> | ||
``` | ||
|
||
Or by providing an certificate directory: | ||
|
||
```pycon | ||
>>> custom_capath = httpx.SSLContext(verify="path/to/certs") | ||
>>> client = httpx.Client(ssl_context=custom_capath) | ||
>>> client.get("https://www.example.com") | ||
<Response [200 OK]> | ||
``` | ||
|
||
These usages are equivelent to using [`.load_verify_locations()`](https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_verify_locations) with either `cafile=...` or `capath=...`. | ||
|
||
### Client side certificates | ||
|
||
You can also specify a local cert to use as a client-side certificate, either a path to an SSL certificate file... | ||
|
||
```pycon | ||
>>> cert = "path/to/client.pem" | ||
>>> ssl_context = httpx.SSLContext(cert=cert) | ||
>>> httpx.get("https://example.org", ssl_context=ssl_context) | ||
<Response [200 OK]> | ||
``` | ||
|
||
The `client.get(...)` method and other request methods *do not* support changing the SSL settings on a per-request basis. If you need different SSL settings in different cases you should use more that one client instance, with different settings on each. Each client will then be using an isolated connection pool with a specific fixed SSL configuration on all connections within that pool. | ||
Or two-tuple of (certificate file, key file)... | ||
|
||
## Client Side Certificates | ||
```pycon | ||
>>> cert = ("path/to/client.pem", "path/to/client.key") | ||
>>> ssl_context = httpx.SSLContext(cert=cert) | ||
>>> httpx.get("https://example.org", ssl_context=ssl_context) | ||
<Response [200 OK]> | ||
``` | ||
|
||
You can also specify a local cert to use as a client-side certificate, either a path to an SSL certificate file, or two-tuple of (certificate file, key file), or a three-tuple of (certificate file, key file, password) | ||
Or a three-tuple of (certificate file, key file, password)... | ||
|
||
```pycon | ||
>>> cert = ("path/to/client.pem", "path/to/client.key", "password") | ||
>>> ssl_context = httpx.SSLContext(cert=cert) | ||
>>> httpx.get("https://example.org", ssl_context=ssl_context) | ||
<Response [200 OK]> | ||
``` | ||
|
||
These configurations are equivalent to using [`.load_cert_chain()`](https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_cert_chain). | ||
|
||
### Using alternate SSL contexts | ||
|
||
You can also use an alternate `ssl.SSLContext` instances. | ||
|
||
For example, [using the `truststore` package](https://truststore.readthedocs.io/)... | ||
|
||
```python | ||
cert = "path/to/client.pem" | ||
client = httpx.Client(cert=cert) | ||
response = client.get("https://example.org") | ||
import ssl | ||
import truststore | ||
import httpx | ||
|
||
ssl_context = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT) | ||
client = httpx.Client(ssl_context=ssl_context) | ||
``` | ||
|
||
Alternatively... | ||
Or working [directly with Python's standard library](https://docs.python.org/3/library/ssl.html)... | ||
|
||
```python | ||
cert = ("path/to/client.pem", "path/to/client.key") | ||
client = httpx.Client(cert=cert) | ||
response = client.get("https://example.org") | ||
import ssl | ||
import httpx | ||
|
||
ssl_context = ssl.create_default_context() | ||
client = httpx.Client(ssl_context=ssl_context) | ||
``` | ||
|
||
Or... | ||
### Working with `SSL_CERT_FILE` and `SSL_CERT_DIR` | ||
|
||
Unlike `requests`, the `httpx` package does not automatically pull in [the environment variables `SSL_CERT_FILE` or `SSL_CERT_DIR`](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_default_verify_paths.html). If you want to use these they need to be enabled explicitly. | ||
|
||
For example... | ||
|
||
```python | ||
# Use `SSL_CERT_FILE` or `SSL_CERT_DIR` if configured, otherwise use certifi. | ||
verify = os.environ.get("SSL_CERT_FILE", os.environ.get("SSL_CERT_DIR", True)) | ||
ssl_context = httpx.SSLContext(verify=verify) | ||
``` | ||
|
||
## `SSLKEYLOGFILE` | ||
|
||
Valid values: a filename | ||
|
||
If this environment variable is set, TLS keys will be appended to the specified file, creating it if it doesn't exist, whenever key material is generated or received. The keylog file is designed for debugging purposes only. | ||
|
||
Support for `SSLKEYLOGFILE` requires Python 3.8 and OpenSSL 1.1.1 or newer. | ||
|
||
Example: | ||
|
||
```python | ||
cert = ("path/to/client.pem", "path/to/client.key", "password") | ||
client = httpx.Client(cert=cert) | ||
response = client.get("https://example.org") | ||
# test_script.py | ||
import httpx | ||
|
||
with httpx.Client() as client: | ||
r = client.get("https://google.com") | ||
``` | ||
|
||
## Making HTTPS requests to a local server | ||
```console | ||
SSLKEYLOGFILE=test.log python test_script.py | ||
cat test.log | ||
# TLS secrets log file, generated by OpenSSL / Python | ||
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX | ||
EXPORTER_SECRET XXXX | ||
SERVER_TRAFFIC_SECRET_0 XXXX | ||
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX | ||
CLIENT_TRAFFIC_SECRET_0 XXXX | ||
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX | ||
EXPORTER_SECRET XXXX | ||
SERVER_TRAFFIC_SECRET_0 XXXX | ||
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX | ||
CLIENT_TRAFFIC_SECRET_0 XXXX | ||
``` | ||
|
||
### Making HTTPS requests to a local server | ||
|
||
When making requests to local servers, such as a development server running on `localhost`, you will typically be using unencrypted HTTP connections. | ||
|
||
If you do need to make HTTPS connections to a local server, for example to test an HTTPS-only service, you will need to create and use your own certificates. Here's one way to do it: | ||
|
||
1. Use [trustme](https://github.com/python-trio/trustme) to generate a pair of server key/cert files, and a client cert file. | ||
1. Pass the server key/cert files when starting your local server. (This depends on the particular web server you're using. For example, [Uvicorn](https://www.uvicorn.org) provides the `--ssl-keyfile` and `--ssl-certfile` options.) | ||
1. Tell HTTPX to use the certificates stored in `client.pem`: | ||
2. Pass the server key/cert files when starting your local server. (This depends on the particular web server you're using. For example, [Uvicorn](https://www.uvicorn.org) provides the `--ssl-keyfile` and `--ssl-certfile` options.) | ||
3. Tell HTTPX to use the certificates stored in `client.pem`: | ||
|
||
```python | ||
client = httpx.Client(verify="/tmp/client.pem") | ||
response = client.get("https://localhost:8000") | ||
```pycon | ||
>>> import httpx | ||
>>> ssl_context = httpx.SSLContext(verify="/tmp/client.pem") | ||
>>> r = httpx.get("https://localhost:8000", ssl_context=ssl_context) | ||
>>> r | ||
Response <200 OK> | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -171,12 +171,10 @@ Also note that `requests.Session.request(...)` allows a `proxies=...` parameter, | |
|
||
## SSL configuration | ||
|
||
When using a `Client` instance, the `trust_env`, `verify`, and `cert` arguments should always be passed on client instantiation, rather than passed to the request method. | ||
When using a `Client` instance, the ssl configurations should always be passed on client instantiation, rather than passed to the request method. | ||
|
||
If you need more than one different SSL configuration, you should use different client instances for each SSL configuration. | ||
|
||
Requests supports `REQUESTS_CA_BUNDLE` which points to either a file or a directory. HTTPX supports the `SSL_CERT_FILE` (for a file) and `SSL_CERT_DIR` (for a directory) OpenSSL variables instead. | ||
|
||
Comment on lines
-178
to
-179
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. perhaps it's better link to this section instead of removing? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We removed those support, do we need to mention them? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Yes, I think. Since it's removed now, it will be a thing that Requests has but we don't. |
||
## Request body on HTTP methods | ||
|
||
The HTTP `GET`, `DELETE`, `HEAD`, and `OPTIONS` methods are specified as not supporting a request body. To stay in line with this, the `.get`, `.delete`, `.head` and `.options` functions do not support `content`, `files`, `data`, or `json` arguments. | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -8,66 +8,6 @@ Environment variables are used by default. To ignore environment variables, `tru | |
|
||
Here is a list of environment variables that HTTPX recognizes and what function they serve: | ||
|
||
## `SSLKEYLOGFILE` | ||
|
||
Valid values: a filename | ||
Comment on lines
-11
to
-13
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we still need this part There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It was moved to the ssl documentation There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. does it make sense to have it in environment_variables.md with linking its description to ssl documentation? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it would be better to get rid of environment_variables.md. It's better to show env variables in each section, and only related ones. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I think that'll work out best yep. Let's work on that as a follow-up. |
||
|
||
If this environment variable is set, TLS keys will be appended to the specified file, creating it if it doesn't exist, whenever key material is generated or received. The keylog file is designed for debugging purposes only. | ||
|
||
Support for `SSLKEYLOGFILE` requires Python 3.8 and OpenSSL 1.1.1 or newer. | ||
|
||
Example: | ||
|
||
```python | ||
# test_script.py | ||
import httpx | ||
|
||
with httpx.AsyncClient() as client: | ||
r = client.get("https://google.com") | ||
``` | ||
|
||
```console | ||
SSLKEYLOGFILE=test.log python test_script.py | ||
cat test.log | ||
# TLS secrets log file, generated by OpenSSL / Python | ||
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX | ||
EXPORTER_SECRET XXXX | ||
SERVER_TRAFFIC_SECRET_0 XXXX | ||
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX | ||
CLIENT_TRAFFIC_SECRET_0 XXXX | ||
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX | ||
EXPORTER_SECRET XXXX | ||
SERVER_TRAFFIC_SECRET_0 XXXX | ||
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX | ||
CLIENT_TRAFFIC_SECRET_0 XXXX | ||
``` | ||
|
||
## `SSL_CERT_FILE` | ||
|
||
Valid values: a filename | ||
|
||
If this environment variable is set then HTTPX will load | ||
CA certificate from the specified file instead of the default | ||
location. | ||
|
||
Example: | ||
|
||
```console | ||
SSL_CERT_FILE=/path/to/ca-certs/ca-bundle.crt python -c "import httpx; httpx.get('https://example.com')" | ||
``` | ||
|
||
## `SSL_CERT_DIR` | ||
|
||
Valid values: a directory following an [OpenSSL specific layout](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_load_verify_locations.html). | ||
|
||
If this environment variable is set and the directory follows an [OpenSSL specific layout](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_load_verify_locations.html) (ie. you ran `c_rehash`) then HTTPX will load CA certificates from this directory instead of the default location. | ||
|
||
Example: | ||
|
||
```console | ||
SSL_CERT_DIR=/path/to/ca-certs/ python -c "import httpx; httpx.get('https://example.com')" | ||
``` | ||
|
||
## Proxies | ||
|
||
The environment variables documented below are used as a convention by various HTTP tooling, including: | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also needs remove
cert
andverify
here:httpx/docs/compatibility.md
Line 202 in 1bf1fc0
Do we need also mention Httpx doesn't support
cert
andverify
while Requests does?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed, thanks