Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatible with InfluxDB CLI - Support GZIP compression #3487

Closed
tisonkun opened this issue Mar 11, 2024 · 2 comments · Fixed by #3494
Closed

Compatible with InfluxDB CLI - Support GZIP compression #3487

tisonkun opened this issue Mar 11, 2024 · 2 comments · Fixed by #3494
Assignees
Labels
good first issue Good for newcomers

Comments

@tisonkun
Copy link
Collaborator

tisonkun commented Mar 11, 2024

What type of enhancement is this?

API improvement

What does the enhancement do?

Here is a related request:

2024/03/12 03:30:29 
POST /v1/influxdb/api/v2/write?bucket=xxx-public&org=my-org&precision=s HTTP/1.1
Host: yyy.us-west-2.aws.greptime.cloud
User-Agent: influx/dev (darwin) Sha/none Date/2024-03-11T19:17:02Z
Transfer-Encoding: chunked
Accept: application/json
Authorization: Token uuu:ppp
Content-Encoding: gzip
Content-Type: text/plain
Accept-Encoding: gzip

Now we return:

2024/03/12 03:38:50 
HTTP/2.0 400 Bad Request
Content-Length: 87
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization
Access-Control-Allow-Methods: OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 1728000
Content-Type: text/plain; charset=utf-8
Date: Mon, 11 Mar 2024 19:38:50 GMT
Strict-Transport-Security: max-age=15724800; includeSubDomains

Request body didn't contain valid UTF-8: invalid utf-8 sequence of 1 bytes from index 1

Implementation challenges

I suppose we should handle Content-Encoding header and use gzip if necessary.

cc @shuiyisong

@tisonkun
Copy link
Collaborator Author

Related InfluxDB CLI code:

// clients/write/write.go
		buf := bytes.Buffer{}
		gzw := gzip.NewWriter(&buf)
		_, err := gzw.Write(batch)
		gzw.Close()
		if err != nil {
			return err
		}

		req := c.PostWrite(ctx).Body(buf.Bytes()).ContentEncoding("gzip").Precision(params.Precision)

@tisonkun
Copy link
Collaborator Author

InfluxDB IOx related code:

// influxdbv3_server/src/http.rs
        // Unzip the gzip-encoded content
        use std::io::Read;
        let decoder = flate2::read::GzDecoder::new(&body[..]);

        // Read at most max_request_bytes bytes to prevent a decompression bomb
        // based DoS.
        //
        // In order to detect if the entire stream ahs been read, or truncated,
        // read an extra byte beyond the limit and check the resulting data
        // length - see the max_request_size_truncation test.
        let mut decoder = decoder.take(self.max_request_bytes as u64 + 1);
        let mut decoded_data = Vec::new();
        decoder
            .read_to_end(&mut decoded_data)
            .map_err(Error::InvalidGzip)?;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant