Gzip decompression

[vbfox]

Motivation

When a tonic server we deployed receive grpc messages that are compressed (as gzip by a Go client) it currently fail as decompression isn't currently supported (#282)

I wanted to support this limited case as simply as possible.

Solution

I took the solution of only implementing this very specific case because:

• gzip is the most common compression for grpc and the one that is available by default in the go and c++ implementations
• It's an hard-failure, if a tonic server client doesn't send compressed requests the other party will understand it. But when a compressed message is received the only way to handle it is to reject it if compression isn't implemented.
• It doesn't require any additional API surface as a single hardcoded compressor is supported and only enabled via feature flags.

On the code decisions:

• Compressor contains compression and decompression method and aims to be/become the equivalent of https://godoc.org/google.golang.org/grpc/encoding#Compressor it could be the trait that users could implement to provide additional compression methods.
• Decompression hosts the decompression information provided to 'decode' (A single http header for now)
• Compression is the same on the other direction but is only used for responses in cases where the request was compressed in all other cases no compression take place. It also avoid a problem currently existing in envoy grpc-web support and mimics what grpc-go does
• I looked at enabling some of the interop tests on compression but go-grpc doesn't implement them. I'm not sure of how this can easily be integration-tested before compression is there and it can be tested via roundtrip. The C++ tests support compression but they seem to test advanced aspects too that aren't implemented yet in tonic with this PR.

Tests done:

• Added benches with gzip
• Greeter and routeguide were tested both way against go, enabling gzip from the go side clients
• Deploying an internal project using tonic and accessing it with Go requiring compression and via grpc-web translated by envoy

---

Fixes #282

[alce]

Thank you for this. It looks like a good approach at first glance.

I am going to go over the implementation in more depth in the next couple of days, primarily to see if I can spot details that may cause trouble (as in major breaking changes, for example) with further enhancements or other planned features. A bit later on, @LucioFranco can help us out sorting out the details.

[alce]

@vbfox I looked at this in a little more detail and I think it's neat. However, we can't proceed yet because we need to figure out how to best to integrate this with channels and clients. I understand this PR is not concerned with that (and it's fine) but wee need to have a clear path forward. We may want to make some changes to the transport before integrating this. It may take a little bit though.

Here are some thoughts I had while going through this. Nothing actionable yet, just things we may want to consider.

• I know Go and Java use interfaces but maybe we could use enums instead of traits. We could provide identity, gzip and deflate variants under a singlecompression feature, maybe even throw in brotli. We loose the ability to use arbitrary algorithms but I wonder how useful that is in practice. It could make the implementation much simpler and adding support for new algorithms would be easy.

• If we go with trait objects, maybe we could have an explicit registry instead of populating the compressors map based on feature flags.

• I think we should keep request and response compression asymmetric and not follow Go's lead on this one.

• Tiny tip: you can cast a boolean to an u8 compression_enabled as u8

We'll update with specific steps forward as soon as possible. Thanks again.

[davidpdrsn]

@vbfox Sorry for the slow progress on this. Is this still something you're interested in working on? If so I would like to help reviewing. If not I also wouldn't mind taking over the implementation.

I also asked a general question about how the spec requires you to implement compression here. Maybe you're able to help.

[vbfox]

@davidpdrsn I'm interested in gzip compression being available 😄

I can continue working on it but if someone wants to take over no problem, the remaining issues were more about how the project wants to approach integrating Compression and i'm not implicated enough in tonic design for that I admit :)

[davidpdrsn]

Awesome! I'm gonna help out with reviewing then.

I guess the first thing to do is get it up to date with the current master. It
seems your branch is still using tokio 0.2 whereas master has updated to tokio
1.

Here are my comments about what @alce said.

• I know Go and Java use interfaces but maybe we could use enums instead of
  traits.

I agree that going with traits to support custom algorithms probably isn't
necessary. I'm wondering if something like this might work:

/// Which encodings are enabled?
///
/// Used to construct the `grpc-accept-encoding` header and determine if the
/// `grpc-encoding` of an incoming messsage is supported.
pub(crate) struct MessageEncodingConfig {
    gzip: bool,
    // more encodings added here later
}

/// Which encoding should be used for a message
///
/// Corresponds to the `grpc-encoding` header.
pub(crate) MessageEncoding {
    Gzip,
    // more encodings added here later
}

Where Endpoint and Server somehow stores a MessageEncodingConfig and have
methods allowing the user to customize which encodings to support.

This is similar to how compression/decompression works in
tower-http.

We could also have methods to enable/disable the compression on a single
Response but stream items are problematic since they aren't wrapped in a type
that tonic controls. So we have nowhere to add new fields. I would be fine with
making compression global for a given Channel or Server and not support
customizing it for individual messages. I wonder if this violates the spec or
otherwise breaks some use-case 🤔

We could provide identity, gzip and deflate variants under a
singlecompression feature, maybe even throw in brotli.

I think only supporting gzip out of the box is fine but if supporting more
algorithms isn't too much additional work that would also be fine by me.

I think we should keep request and response compression asymmetric and not
follow Go's lead on this one.

I'm not quite sure what code is referenced here or what the implications would
be.

[davidpdrsn]

@vbfox Have you have time to look at what I wrote ☝️? Do you have questions for us?

[vbfox]

@davidpdrsn Yes I looked at it but didn't have time yet to change the code.

My main question is, could an adapted version of this PR lands by itself (no API change and add support for the other party sending compressed messages) with another PR later adding sending compressed messages and configuration or do you expect both at the same time ?

On the plugability/trait part the main interest is not only for new algorithms but also to allow users to provide their own dependencies for example to configure what gzip implementation to use (pure rust ? zlib-ng ? https://github.com/rust-lang/flate2-rs#backends ) as it can have performance/deployment complexity tradeofs that tonic might not want to impose on it's users.

[davidpdrsn]

My main question is, could an adapted version of this PR lands by itself (no API change and add support for the other party sending compressed messages) with another PR later adding sending compressed messages and configuration or do you expect both at the same time ?

We're currently thinking about doing a 0.5 release which will include a few breaking changes and grpc-web support. I would really like to have this feature included in that release as its very useful. However since this isn't a breaking change and therefore not something we have to include in 0.5 I would prefer if we aimed to make a single PR that added a compelling compression feature, meaning we would have support for both sending and receiving compressed messages. If we break things into multiple PRs we risk getting into a situation where we want to ship 0.5 but can't because compression is only half done.

I'm still up for taking over the implementation if you don't feel you have the bandwidth for it.

On the plugability/trait part the main interest is not only for new algorithms but also to allow users to provide their own dependencies for example to configure what gzip implementation to use (pure rust ? zlib-ng ? https://github.com/rust-lang/flate2-rs#backends ) as it can have performance/deployment complexity tradeofs that tonic might not want to impose on it's users.

Personally I don't think allowing users to plug in their own compression algorithms is an important feature. For example its not something supported by reqwest or warp. I think that is fine as it simplifies the implementation quite a bit.

[vbfox]

I'm still up for taking over the implementation if you don't feel you have the bandwidth for it.

I'll try to find time this weekend to advance on it but If I can't, I think i'll let you take over.

Personally I don't think allowing users to plug in their own compression algorithms is an important feature. For example its not something supported by reqwest or warp. I think that is fine as it simplifies the implementation quite a bit.

👍🏼

[davidpdrsn]

@vbfox thanks for your work on this! I've taken over in #692.