Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add httpx.SSLContext configuration. #3022

Merged
merged 43 commits into from
Oct 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
88f2ab7
Add ssl_context argument
karpetrosyan Dec 25, 2023
58e9fdc
Use default ciphers and set minimum tls version to 1.2
karpetrosyan Dec 25, 2023
1afe2c9
Merge branch 'master' into add-ssl-context-argument
karpetrosyan Dec 28, 2023
6be802e
Fix init_proxy_transport arguments
karpetrosyan Dec 28, 2023
d124d8f
Merge branch 'master' into add-ssl-context-argument
karpetrosyan Dec 28, 2023
6f1162b
Use __init__ instead of __new__ in SSLContext
karpetrosyan Dec 29, 2023
f613dee
Merge branch 'master' into add-ssl-context-argument
karpetrosyan Jan 5, 2024
55ec002
Merge branch 'master' into add-ssl-context-argument
tomchristie Jan 15, 2024
7346bc2
Merge branch 'master' into add-ssl-context-argument
karpetrosyan Jan 16, 2024
7f35134
Docs
karpetrosyan Jan 16, 2024
8d5983a
Fix SSLContext repr
karpetrosyan Jan 16, 2024
f759dec
Merge branch 'master' into add-ssl-context-argument
karpetrosyan Jan 16, 2024
7227573
Merge branch 'master' into add-ssl-context-argument
tomchristie Mar 1, 2024
0567aab
Merge branch 'master' into add-ssl-context-argument
tomchristie Mar 1, 2024
2d79640
Fixes to merge master
tomchristie Mar 1, 2024
4d84953
comma
tomchristie Mar 1, 2024
df7bf31
Update _client.py
tomchristie Mar 1, 2024
93ca75f
Update _config.py
tomchristie Mar 1, 2024
2b7cf9e
Update _config.py
tomchristie Mar 1, 2024
fa065dd
Merge branch 'master' into add-ssl-context-argument
tomchristie May 8, 2024
f09a823
Merge branch 'master' into add-ssl-context-argument
karpetrosyan May 17, 2024
4eba150
Merge branch 'master' of github.com:encode/httpx into add-ssl-context…
karpetrosyan Sep 4, 2024
f3b1dc7
Merge branch 'master' into add-ssl-context-argument
karpetrosyan Sep 22, 2024
998f445
Merge branch 'master' into add-ssl-context-argument
tomchristie Sep 23, 2024
a08ff77
Update docs/advanced/ssl.md
karpetrosyan Sep 23, 2024
827ae5b
Merge branch 'master' into add-ssl-context-argument
tomchristie Sep 26, 2024
b3673bd
Tidy up ssl configuration process
karpetrosyan Sep 26, 2024
d3a8aab
Merge branch 'version-1.0' of github.com:encode/httpx into add-ssl-co…
karpetrosyan Sep 26, 2024
808a891
changelog
karpetrosyan Sep 26, 2024
289cf9c
Revert SSLKEYLOGFILE change
karpetrosyan Sep 28, 2024
6a85f2c
Add a test for SSLKEYLOGFILE
karpetrosyan Sep 28, 2024
d235e76
Fix the contributing docs
karpetrosyan Sep 28, 2024
1a79bd9
Update tests/test_config.py
karpetrosyan Oct 4, 2024
3f9aaa8
Fix ssl_context annotations
karpetrosyan Oct 4, 2024
46268b7
Update .github/CONTRIBUTING.md
karpetrosyan Oct 4, 2024
26de9d0
Update docs/advanced/ssl.md
karpetrosyan Oct 4, 2024
49545bb
Update docs/advanced/ssl.md
tomchristie Oct 4, 2024
01bb388
Update CHANGELOG.md
tomchristie Oct 6, 2024
13cb957
Apply suggestions from code review
tomchristie Oct 7, 2024
51ec619
Apply suggestions from code review
tomchristie Oct 7, 2024
e7c607c
Update docs/advanced/ssl.md
tomchristie Oct 7, 2024
f894a06
Apply suggestions from code review
tomchristie Oct 8, 2024
cb71af3
Update tests/test_config.py
tomchristie Oct 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,9 +211,9 @@ this is where our previously generated `client.pem` comes in:
```
import httpx

proxies = {"all": "http://127.0.0.1:8080/"}
ssl_context = httpx.SSLContext(verify="/path/to/client.pem"))

with httpx.Client(proxies=proxies, verify="/path/to/client.pem") as client:
with httpx.Client(proxy="http://127.0.0.1:8080/", ssl_context=ssl_context) as client:
response = client.get("https://example.org")
print(response.status_code) # should print 200
```
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

## Unreleased

* Added `httpx.SSLContext` class and `ssl_context` argument. (#3022)
* Removed `cert` and `verify` arguments, you should use the `ssl_context=...` instead. (#3022)
* The deprecated `proxies` argument has now been removed.
* The deprecated `app` argument has now been removed.
* The `URL.raw` property has now been removed.
Expand Down
220 changes: 172 additions & 48 deletions docs/advanced/ssl.md
Original file line number Diff line number Diff line change
@@ -1,100 +1,224 @@
When making a request over HTTPS, HTTPX needs to verify the identity of the requested host. To do this, it uses a bundle of SSL certificates (a.k.a. CA bundle) delivered by a trusted certificate authority (CA).

## Changing the verification defaults
### Enabling and disabling verification

By default, HTTPX uses the CA bundle provided by [Certifi](https://pypi.org/project/certifi/). This is what you want in most cases, even though some advanced situations may require you to use a different set of certificates.
By default httpx will verify HTTPS connections, and raise an error for invalid SSL cases...

If you'd like to use a custom CA bundle, you can use the `verify` parameter.
```pycon
>>> httpx.get("https://expired.badssl.com/")
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)
```

```python
import httpx
You can configure the verification using `httpx.SSLContext()`.

r = httpx.get("https://example.org", verify="path/to/client.pem")
```pycon
>>> ssl_context = httpx.SSLContext()
>>> ssl_context
SSLContext(verify=True)
>>> httpx.get("https://www.example.com", ssl_context=ssl_context)
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)
```

Alternatively, you can pass a standard library `ssl.SSLContext`.
For example, you can use this to disable verification completely and allow insecure requests...

```pycon
>>> import ssl
>>> import httpx
>>> context = ssl.create_default_context()
>>> context.load_verify_locations(cafile="/tmp/client.pem")
>>> httpx.get('https://example.org', verify=context)
>>> no_verify = httpx.SSLContext(verify=False)
>>> no_verify
SSLContext(verify=False)
>>> httpx.get("https://expired.badssl.com/", ssl_context=no_verify)
<Response [200 OK]>
```

We also include a helper function for creating properly configured `SSLContext` instances.
### Configuring client instances

If you're using a `Client()` instance, then you should pass any SSL settings when instantiating the client.

```pycon
>>> context = httpx.create_ssl_context()
>>> ssl_context = httpx.SSLContext()
>>> client = httpx.Client(ssl_context=ssl_context)
```

The `create_ssl_context` function accepts the same set of SSL configuration arguments
(`trust_env`, `verify`, `cert` and `http2` arguments)
as `httpx.Client` or `httpx.AsyncClient`
The `client.get(...)` method and other request methods on a `Client` instance *do not* support changing the SSL settings on a per-request basis.

If you need different SSL settings in different cases you should use more that one client instance, with different settings on each. Each client will then be using an isolated connection pool with a specific fixed SSL configuration on all connections within that pool.

### Changing the verification defaults

By default, HTTPX uses the CA bundle provided by [Certifi](https://pypi.org/project/certifi/).

The following all have the same behaviour...

Using the default SSL context.

```pycon
>>> import httpx
>>> context = httpx.create_ssl_context(verify="/tmp/client.pem")
>>> httpx.get('https://example.org', verify=context)
>>> client = httpx.Client()
>>> client.get("https://www.example.com")
<Response [200 OK]>
```

Or you can also disable the SSL verification entirely, which is _not_ recommended.
Using the default SSL context, but specified explicitly.

```python
import httpx
```pycon
>>> default = httpx.SSLContext()
>>> client = httpx.Client(ssl_context=default)
>>> client.get("https://www.example.com")
<Response [200 OK]>
```

r = httpx.get("https://example.org", verify=False)
Using the default SSL context, with `verify=True` specified explicitly.

```pycon
>>> default = httpx.SSLContext(verify=True)
>>> client = httpx.Client(ssl_context=default)
>>> client.get("https://www.example.com")
<Response [200 OK]>
```

## SSL configuration on client instances
Using an SSL context, with `certifi.where()` explicitly specified.

If you're using a `Client()` instance, then you should pass any SSL settings when instantiating the client.
```pycon
>>> default = httpx.SSLContext(verify=certifi.where())
>>> client = httpx.Client(ssl_context=default)
>>> client.get("https://www.example.com")
<Response [200 OK]>
```

```python
client = httpx.Client(verify=False)
For some advanced situations may require you to use a different set of certificates, either by specifying a PEM file:

```pycon
>>> custom_cafile = httpx.SSLContext(verify="path/to/certs.pem")
>>> client = httpx.Client(ssl_context=custom_cafile)
>>> client.get("https://www.example.com")
<Response [200 OK]>
```

Or by providing an certificate directory:

```pycon
>>> custom_capath = httpx.SSLContext(verify="path/to/certs")
>>> client = httpx.Client(ssl_context=custom_capath)
>>> client.get("https://www.example.com")
<Response [200 OK]>
```

These usages are equivelent to using [`.load_verify_locations()`](https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_verify_locations) with either `cafile=...` or `capath=...`.

### Client side certificates

You can also specify a local cert to use as a client-side certificate, either a path to an SSL certificate file...

```pycon
>>> cert = "path/to/client.pem"
>>> ssl_context = httpx.SSLContext(cert=cert)
>>> httpx.get("https://example.org", ssl_context=ssl_context)
<Response [200 OK]>
```

The `client.get(...)` method and other request methods *do not* support changing the SSL settings on a per-request basis. If you need different SSL settings in different cases you should use more that one client instance, with different settings on each. Each client will then be using an isolated connection pool with a specific fixed SSL configuration on all connections within that pool.
Or two-tuple of (certificate file, key file)...

## Client Side Certificates
```pycon
>>> cert = ("path/to/client.pem", "path/to/client.key")
>>> ssl_context = httpx.SSLContext(cert=cert)
>>> httpx.get("https://example.org", ssl_context=ssl_context)
<Response [200 OK]>
```

You can also specify a local cert to use as a client-side certificate, either a path to an SSL certificate file, or two-tuple of (certificate file, key file), or a three-tuple of (certificate file, key file, password)
Or a three-tuple of (certificate file, key file, password)...

```pycon
>>> cert = ("path/to/client.pem", "path/to/client.key", "password")
>>> ssl_context = httpx.SSLContext(cert=cert)
>>> httpx.get("https://example.org", ssl_context=ssl_context)
<Response [200 OK]>
```

These configurations are equivalent to using [`.load_cert_chain()`](https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_cert_chain).

### Using alternate SSL contexts

You can also use an alternate `ssl.SSLContext` instances.

For example, [using the `truststore` package](https://truststore.readthedocs.io/)...

```python
cert = "path/to/client.pem"
client = httpx.Client(cert=cert)
response = client.get("https://example.org")
import ssl
import truststore
import httpx

ssl_context = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
client = httpx.Client(ssl_context=ssl_context)
```

Alternatively...
Or working [directly with Python's standard library](https://docs.python.org/3/library/ssl.html)...

```python
cert = ("path/to/client.pem", "path/to/client.key")
client = httpx.Client(cert=cert)
response = client.get("https://example.org")
import ssl
import httpx

ssl_context = ssl.create_default_context()
client = httpx.Client(ssl_context=ssl_context)
```

Or...
### Working with `SSL_CERT_FILE` and `SSL_CERT_DIR`

Unlike `requests`, the `httpx` package does not automatically pull in [the environment variables `SSL_CERT_FILE` or `SSL_CERT_DIR`](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_default_verify_paths.html). If you want to use these they need to be enabled explicitly.

For example...

```python
# Use `SSL_CERT_FILE` or `SSL_CERT_DIR` if configured, otherwise use certifi.
verify = os.environ.get("SSL_CERT_FILE", os.environ.get("SSL_CERT_DIR", True))
ssl_context = httpx.SSLContext(verify=verify)
```

## `SSLKEYLOGFILE`

Valid values: a filename

If this environment variable is set, TLS keys will be appended to the specified file, creating it if it doesn't exist, whenever key material is generated or received. The keylog file is designed for debugging purposes only.

Support for `SSLKEYLOGFILE` requires Python 3.8 and OpenSSL 1.1.1 or newer.

Example:

```python
cert = ("path/to/client.pem", "path/to/client.key", "password")
client = httpx.Client(cert=cert)
response = client.get("https://example.org")
# test_script.py
import httpx

with httpx.Client() as client:
r = client.get("https://google.com")
```

## Making HTTPS requests to a local server
```console
SSLKEYLOGFILE=test.log python test_script.py
cat test.log
# TLS secrets log file, generated by OpenSSL / Python
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX
EXPORTER_SECRET XXXX
SERVER_TRAFFIC_SECRET_0 XXXX
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX
CLIENT_TRAFFIC_SECRET_0 XXXX
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX
EXPORTER_SECRET XXXX
SERVER_TRAFFIC_SECRET_0 XXXX
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX
CLIENT_TRAFFIC_SECRET_0 XXXX
```

### Making HTTPS requests to a local server

When making requests to local servers, such as a development server running on `localhost`, you will typically be using unencrypted HTTP connections.

If you do need to make HTTPS connections to a local server, for example to test an HTTPS-only service, you will need to create and use your own certificates. Here's one way to do it:

1. Use [trustme](https://github.com/python-trio/trustme) to generate a pair of server key/cert files, and a client cert file.
1. Pass the server key/cert files when starting your local server. (This depends on the particular web server you're using. For example, [Uvicorn](https://www.uvicorn.org) provides the `--ssl-keyfile` and `--ssl-certfile` options.)
1. Tell HTTPX to use the certificates stored in `client.pem`:
2. Pass the server key/cert files when starting your local server. (This depends on the particular web server you're using. For example, [Uvicorn](https://www.uvicorn.org) provides the `--ssl-keyfile` and `--ssl-certfile` options.)
3. Tell HTTPX to use the certificates stored in `client.pem`:

```python
client = httpx.Client(verify="/tmp/client.pem")
response = client.get("https://localhost:8000")
```pycon
>>> import httpx
>>> ssl_context = httpx.SSLContext(verify="/tmp/client.pem")
>>> r = httpx.get("https://localhost:8000", ssl_context=ssl_context)
>>> r
Response <200 OK>
```
4 changes: 1 addition & 3 deletions docs/compatibility.md
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also needs remove cert and verify here:

Besides, `httpx.Request()` does not support the `auth`, `timeout`, `follow_redirects`, `mounts`, `verify` and `cert` parameters. However these are available in `httpx.request`, `httpx.get`, `httpx.post` etc., as well as on [`Client` instances](advanced/clients.md#client-instances).

Do we need also mention Httpx doesn't support cert and verify while Requests does?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed, thanks

Original file line number Diff line number Diff line change
Expand Up @@ -171,12 +171,10 @@ Also note that `requests.Session.request(...)` allows a `proxies=...` parameter,

## SSL configuration

When using a `Client` instance, the `trust_env`, `verify`, and `cert` arguments should always be passed on client instantiation, rather than passed to the request method.
When using a `Client` instance, the ssl configurations should always be passed on client instantiation, rather than passed to the request method.

If you need more than one different SSL configuration, you should use different client instances for each SSL configuration.

Requests supports `REQUESTS_CA_BUNDLE` which points to either a file or a directory. HTTPX supports the `SSL_CERT_FILE` (for a file) and `SSL_CERT_DIR` (for a directory) OpenSSL variables instead.

Comment on lines -178 to -179
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps it's better link to this section instead of removing?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We removed those support, do we need to mention them?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We removed those support, do we need to mention them?

Yes, I think. Since it's removed now, it will be a thing that Requests has but we don't.

## Request body on HTTP methods

The HTTP `GET`, `DELETE`, `HEAD`, and `OPTIONS` methods are specified as not supporting a request body. To stay in line with this, the `.get`, `.delete`, `.head` and `.options` functions do not support `content`, `files`, `data`, or `json` arguments.
Expand Down
60 changes: 0 additions & 60 deletions docs/environment_variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,66 +8,6 @@ Environment variables are used by default. To ignore environment variables, `tru

Here is a list of environment variables that HTTPX recognizes and what function they serve:

## `SSLKEYLOGFILE`

Valid values: a filename
Comment on lines -11 to -13
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we still need this part

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was moved to the ssl documentation

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does it make sense to have it in environment_variables.md with linking its description to ssl documentation?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to get rid of environment_variables.md. It's better to show env variables in each section, and only related ones.

Copy link
Member

@tomchristie tomchristie Oct 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to get rid of environment_variables.md

I think that'll work out best yep. Let's work on that as a follow-up.


If this environment variable is set, TLS keys will be appended to the specified file, creating it if it doesn't exist, whenever key material is generated or received. The keylog file is designed for debugging purposes only.

Support for `SSLKEYLOGFILE` requires Python 3.8 and OpenSSL 1.1.1 or newer.

Example:

```python
# test_script.py
import httpx

with httpx.AsyncClient() as client:
r = client.get("https://google.com")
```

```console
SSLKEYLOGFILE=test.log python test_script.py
cat test.log
# TLS secrets log file, generated by OpenSSL / Python
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX
EXPORTER_SECRET XXXX
SERVER_TRAFFIC_SECRET_0 XXXX
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX
CLIENT_TRAFFIC_SECRET_0 XXXX
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX
EXPORTER_SECRET XXXX
SERVER_TRAFFIC_SECRET_0 XXXX
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX
CLIENT_TRAFFIC_SECRET_0 XXXX
```

## `SSL_CERT_FILE`

Valid values: a filename

If this environment variable is set then HTTPX will load
CA certificate from the specified file instead of the default
location.

Example:

```console
SSL_CERT_FILE=/path/to/ca-certs/ca-bundle.crt python -c "import httpx; httpx.get('https://example.com')"
```

## `SSL_CERT_DIR`

Valid values: a directory following an [OpenSSL specific layout](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_load_verify_locations.html).

If this environment variable is set and the directory follows an [OpenSSL specific layout](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_load_verify_locations.html) (ie. you ran `c_rehash`) then HTTPX will load CA certificates from this directory instead of the default location.

Example:

```console
SSL_CERT_DIR=/path/to/ca-certs/ python -c "import httpx; httpx.get('https://example.com')"
```

## Proxies

The environment variables documented below are used as a convention by various HTTP tooling, including:
Expand Down
Loading