framed with framed

[nnovikov]

Version

tokio v0.1.22

Platform

Linux labor 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2 (2019-08-28) x86_64 GNU/Linux

Description

I'm writing my own TCP server. And I need stream encryption/decryption by AES. I wrote a codec, which can decrypt TcpStream and return decrypted buffer as Framed object. After that I need to apply another codec for parse resulting buffer to packet struct. But Framed object can't use another Framed as source of data (because doesn't implement AsyncRead or AsyncWrite traits). Is there a proper/idiomatic way to solve this problem?

[qm3ster]

It doesn't sound like you need another Framed.
What is the type of Item your Decoder produces?

[nnovikov]

First decoder can produce а BytesMut or [u8]. The problem is that the second codec needs to be changed some time ago to another.
I am sure that I can write code in such a way that this is not required. But for training/education purposes I want to try solve this problem in this way.

[qm3ster]

Do these shorter buffers each contain exactly one serialized struct? Exactly multiple? Or can these structs be fragmented across the buffers you are producing?
In the former cases, you can just map/flatmap the Stream of small buffers into a Stream of your structs. (However, that sounds like more allocation may be involved.)
In the latter case, if the input and output are unframed bytestreams, you probably need to make an AsyncRead adapter (that consumes and wraps an AsyncRead). For example see BufReader.
The design might also depend on how a hardware-accelerated AES library needs you to lay out the data.

[nnovikov]

First codec produce AES decodec chunks (1 chunk - 128 bytes). Second codec generate RPC query structure, which usually have length a several kbytes. The case with map/flatmap can not be applied in my case.
BufReader looks like as good point. Can you give a some links to real code/good example with writing adapters?

[tanriol]

I'd like to add that Nickolay is definitely not the only person with this kind of problem. My use case for this includes outer framing with error correction codes in 1032-byte frames (512 bytes of payload each) containing a byte stream of inner frames (not aligned to the outer ones and fragmented). IIRC, I've seen a few more people with this problem in the Tokio gitter channel.

Decoder already provides an API for splitting the incoming byte stream (as an impl AsyncRead) into packets. It would be good to avoid inventing a different API and/or solving this problem manually every time.

What's missing, IMHO, is either an adapter implementing AsyncRead over a Stream<Item=T> where T: AsRef<[u8]> (so that the user can have Framed<AsyncReadAdapter<ChunkStream>>), or a way of framing a Stream of chunks directly (into a FramedStream<ChunkStream>).

[qm3ster]

I only find examples for sync adaptors: AES, Cryptostream, gzip.
But it seems the differences would be very small, you just have to poll the underlying Read instead of blocking on it.

[nnovikov]

Big thanks!