Implement capnproto replication #7659

fpetkovski · 2024-08-22T09:35:27Z

Our profiles from production show that a lot of CPU and memory in receivers is used for unmarshaling protobuf messages. Although it is not possible to change the remote-write format, we have the freedom to change the protocol used for replicating timeseries data.

This commit introduces a new feature in receivers where replication can be done using Cap'n Proto instead of gRPC + Protobuf. The advantage of the former protocol is that deserialization is far cheaper and fields can be accessed directly from the received message (byte slice) without allocating intermediate objects. There is an additional cost for serialization because we have to convert from Protobuf to the Cap'n proto format, but in our setup this still results in a net reduction in resource usage.

We have a split router-receiver setup and this is our resource usage in staging after enabling the new replication method:

Router usage did go up as well, but we still got an overall net reduction:

We can also experiment with other formats by having a more generic flag: receive.replication-format=...

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

Verification

GiedriusS

Doesn't this include a whole new RPC framework? I'm very interested in this. Glad to finally see #7071 tackled. How do you think this compared to dRPC? And is it worth eschewing gRPC altogether?

pkg/receive/capnp_server.go

saswatamcode · 2024-08-23T08:04:52Z

Really interested in this too! Our profiles show similar for unmarshaling

fpetkovski · 2024-08-26T10:11:56Z

Doesn't this include a whole new RPC framework? I'm very interested in this. Glad to finally see #7071 tackled. How do you think this compared to dRPC? And is it worth eschewing gRPC altogether?

Yes this comes with its own RPC framework which can be a double edge sword. It's one more thing to learn and troubleshoot if it goes wrong. I haven't looked into dRPC yet in practice, but maybe there is a way to abstract the RPC framework and not expose internals to end users, this way we can swap it out in future releases in a transparent manner. In this PR we expose the listen port as a parameter since we need separate connections for Cap'n Proto.

squat · 2024-08-26T10:21:48Z

DRPC allows serving both gRPC and DRPC on the same socket to allow for incrementally upgrading/migrating a fleet. We might consider doing something similar whether we go with capnproto or DRPC

GiedriusS · 2024-08-27T06:05:48Z

Mhm, we probably need to experiment more with the transparent approach before deciding what to do. For example, we could maybe deprecate --grpc* command line parameters and have a generic listener for RPCs. Perhaps let's just continue as is and mark this as experimental everywhere?

My only suggestion would be:

          {"address": "node-1:10901", "capnproto_address": "node-1:19391"},

Instead of having two addresses here, let's have "rpc_protocol": "grpc/capnproto" or something like that. I think it would be more future-proof with regard to transparent switching between RPC protocols on the receiving end.

fpetkovski · 2024-08-27T07:12:15Z

Perhaps we can use something like https://github.com/soheilhy/cmux to multiplex multiple protocols on the same port. This way we don't need to pollute arguments with flags for each protocol. Let me see if this is easily doable, and also add some tests for the new RPC.

pedro-stanaka

some comments and questions.

Apart from the stuff I pointed out, if I am getting this right, we won't have metrics for the ingestion process. Should we add some metrics for the most critical paths?

And to keep my MO, I leave that question: Can we add this to the changelog? 🤣

docs/components/receive.md

pedro-stanaka · 2024-10-16T09:57:12Z

pkg/receive/writecapnp/write_request.capnp.go

where is the generation target for this in makefile?

We don't have this yet because it's not easy to automate installation of the capnproto generator. Maybe we can add instructions for installations in the README, and the Makefile can assume the binary exists on the local machine?

Isn't it just yet another bingo tool? go install capnproto.org/go/capnp/v3/capnpc-go@latest?

We also need to install the tools from https://capnproto.org/install.html. It also requires the capnp-go repository to be cloned locally for the schemas.

Here are the docs: https://github.com/capnproto/go-capnp/blob/main/docs/Installation.md.

Added a make target which should do most of the heavy lifting.

pedro-stanaka · 2024-10-16T09:58:30Z

pkg/receive/writecapnp/write_request.capnp

I wonder if we can test somehow that if prometheus ever evolved their schemas, we still would be compatible here. maybe a simple e2e test with PRW?

I have this in my branch, about to open up a PR for this

pedro-stanaka · 2024-10-16T10:25:38Z

pkg/receive/capnproto_writer.go

+
+		var lset labels.Labels
+		// Check if the TSDB has cached reference for those labels.
+		ref, lset = getRef.GetRef(series.Labels, series.Labels.Hash())


The other writer has a bunch of validations of over labels, like if we have duplicate labels or labels are missing __name__, do we need that here?

I dont see those validations on the marshal part of the captnproto, that is why I ask.

We were sorting labels in the unmarshal method which I though removed the need for validation.
But since there's more than just to validate, I removed the sort and we do the validation in the writer now.

Long term we should do this in the router to keep CPU usage in the ingestor lower.

pedro-stanaka · 2024-10-16T10:31:07Z

pkg/receive/hashring.go

 	// Nodes returns a sorted slice of nodes that are in this hashring. Addresses could be duplicated
 	// if, for example, the same address is used for multiple tenants in the multi-hashring.
-	Nodes() []string
+	Nodes() []Endpoint


This will be a breaking change for people using this as library, right?

Yes, this is technically a breaking change, but I don't recall having any guarantees on using Thanos as a library.

GiedriusS · 2024-10-16T12:50:44Z

Adding e2e test: fpetkovski#7

pedro-stanaka

lg

GiedriusS · 2024-10-17T07:54:20Z

pkg/receive/writecapnp/marshal_bench_test.go

+	b.Run("unmarshal", func(b *testing.B) {
+		for i := 0; i < b.N; i++ {
+			msg, err := capnp.Unmarshal(bs)
+			require.NoError(b, err)


This gives me:

/home/giedriusstatkevicius/dev/thanos/pkg/receive/writecapnp/marshal_bench_test.go:101: Error Trace: /home/giedriusstatkevicius/dev/thanos/pkg/receive/writecapnp/marshal_bench_test.go:101 /usr/local/go/src/testing/benchmark.go:193 /usr/local/go/src/testing/benchmark.go:215 /usr/local/go/src/runtime/asm_amd64.s:1700 Error: Received unexpected error: unmarshal: short header section

Does this benchmark work for you?

Looks like we are unmarshaling proto bytes into a capnproto request. I've fixed the benchmark, all of them should be passing now.

GiedriusS

make docs failing 😬

Our profiles from production show that a lot of CPU and memory in receivers is used for unmarshaling protobuf messages. Although it is not possible to change the remote-write format, we have the freedom to change the protocol used for replicating timeseries data. This commit introduces a new feature in receivers where replication can be done using Cap'n Proto instead of gRPC + Protobuf. The advantage of the former protocol is that deserialization is far cheaper and fields can be accessed directly from the received message (byte slice) without allocating intermediate objects. There is an additional cost for serialization because we have to convert from Protobuf to the Cap'n proto format, but in our setup this still results in a net reduction in resource usage. Signed-off-by: Filip Petkovski <[email protected]>

Signed-off-by: Filip Petkovski <[email protected]>

Co-authored-by: Pedro Tanaka <[email protected]> Signed-off-by: Filip Petkovski <[email protected]>

Signed-off-by: Filip Petkovski <[email protected]>

Signed-off-by: Giedrius Statkevičius <[email protected]>

Signed-off-by: Filip Petkovski <[email protected]>

fpetkovski · 2024-10-17T09:05:55Z

Rebased on main and regenerated docs. We should be good to go!

We can keep iterating over time, I'd like to see how we can leverage interning in the future.

pull-request-size bot added the size/XXL label Aug 22, 2024

fpetkovski force-pushed the capnproto-replication branch 4 times, most recently from 253f417 to 7042bea Compare August 22, 2024 15:06

GiedriusS reviewed Aug 22, 2024

View reviewed changes

pkg/receive/capnp_server.go Show resolved Hide resolved

fpetkovski force-pushed the capnproto-replication branch 2 times, most recently from d22809b to 5cfae4c Compare August 28, 2024 09:01

fpetkovski marked this pull request as draft August 29, 2024 08:24

fpetkovski force-pushed the capnproto-replication branch 2 times, most recently from 3df360a to 2fdeba7 Compare September 3, 2024 16:05

fpetkovski mentioned this pull request Sep 17, 2024

Generalize the bucketed bytes pool #7756

Merged

2 tasks

fpetkovski force-pushed the capnproto-replication branch 4 times, most recently from b46ec5d to 79a7aa2 Compare September 27, 2024 11:30

fpetkovski force-pushed the capnproto-replication branch 8 times, most recently from 2bea7c9 to b340ea0 Compare October 15, 2024 14:31

fpetkovski marked this pull request as ready for review October 15, 2024 15:59

pedro-stanaka reviewed Oct 16, 2024

View reviewed changes

pedro-stanaka approved these changes Oct 17, 2024

View reviewed changes

GiedriusS reviewed Oct 17, 2024

View reviewed changes

GiedriusS previously approved these changes Oct 17, 2024

View reviewed changes

fpetkovski and others added 19 commits October 17, 2024 10:56

Pass logger

9f53a82

Signed-off-by: Filip Petkovski <[email protected]>

Update capnp

36af787

Signed-off-by: Filip Petkovski <[email protected]>

Modify flag

4b31096

Signed-off-by: Filip Petkovski <[email protected]>

Lint

503898d

Signed-off-by: Filip Petkovski <[email protected]>

Fix spellcheck

9ac8db4

Signed-off-by: Filip Petkovski <[email protected]>

Use previous version

ba5c325

Signed-off-by: Filip Petkovski <[email protected]>

Update docker base

cd8c0bc

Signed-off-by: Filip Petkovski <[email protected]>

Bump go

7721d56

Signed-off-by: Filip Petkovski <[email protected]>

Update docs/components/receive.md

2057bdf

Co-authored-by: Pedro Tanaka <[email protected]> Signed-off-by: Filip Petkovski <[email protected]>

Validate labels

a6fda55

Signed-off-by: Filip Petkovski <[email protected]>

e2e: add receive test with capnp replication

acfdae2

Signed-off-by: Giedrius Statkevičius <[email protected]>

receive: make copy only when necessary

2f9878c

Signed-off-by: Giedrius Statkevičius <[email protected]>

Fix failing test

1193690

Signed-off-by: Filip Petkovski <[email protected]>

Add CHANGELOG entry

bcc3f6a

Signed-off-by: Filip Petkovski <[email protected]>

Add capnproto Make target

186c089

Signed-off-by: Filip Petkovski <[email protected]>

Replace panics with errors

5f908a3

Signed-off-by: Filip Petkovski <[email protected]>

Fix benchmark

67ef38d

Signed-off-by: Filip Petkovski <[email protected]>

Fix CHANGELOG

d337e0a

Signed-off-by: Filip Petkovski <[email protected]>

fpetkovski dismissed GiedriusS’s stale review via d337e0a October 17, 2024 08:58

fpetkovski force-pushed the capnproto-replication branch from 4b34a07 to d337e0a Compare October 17, 2024 08:58

GiedriusS approved these changes Oct 17, 2024

View reviewed changes

GiedriusS enabled auto-merge (squash) October 17, 2024 09:39

GiedriusS merged commit 65b664c into thanos-io:main Oct 17, 2024
22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement capnproto replication #7659

Implement capnproto replication #7659

fpetkovski commented Aug 22, 2024 •

edited

Loading

GiedriusS left a comment

saswatamcode commented Aug 23, 2024

fpetkovski commented Aug 26, 2024 •

edited

Loading

squat commented Aug 26, 2024

GiedriusS commented Aug 27, 2024

fpetkovski commented Aug 27, 2024

pedro-stanaka left a comment •

edited

Loading

pedro-stanaka Oct 16, 2024

fpetkovski Oct 16, 2024

pedro-stanaka Oct 16, 2024

fpetkovski Oct 17, 2024 •

edited

Loading

fpetkovski Oct 17, 2024 •

edited

Loading

pedro-stanaka Oct 16, 2024

GiedriusS Oct 16, 2024

pedro-stanaka Oct 16, 2024

fpetkovski Oct 16, 2024

pedro-stanaka Oct 16, 2024

fpetkovski Oct 16, 2024

GiedriusS commented Oct 16, 2024

pedro-stanaka left a comment

GiedriusS Oct 17, 2024

fpetkovski Oct 17, 2024

GiedriusS left a comment

fpetkovski commented Oct 17, 2024

Implement capnproto replication #7659

Implement capnproto replication #7659

Conversation

fpetkovski commented Aug 22, 2024 • edited Loading

Changes

Verification

GiedriusS left a comment

Choose a reason for hiding this comment

saswatamcode commented Aug 23, 2024

fpetkovski commented Aug 26, 2024 • edited Loading

squat commented Aug 26, 2024

GiedriusS commented Aug 27, 2024

fpetkovski commented Aug 27, 2024

pedro-stanaka left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fpetkovski Oct 17, 2024 • edited Loading

Choose a reason for hiding this comment

fpetkovski Oct 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GiedriusS commented Oct 16, 2024

pedro-stanaka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GiedriusS left a comment

Choose a reason for hiding this comment

fpetkovski commented Oct 17, 2024

fpetkovski commented Aug 22, 2024 •

edited

Loading

fpetkovski commented Aug 26, 2024 •

edited

Loading

pedro-stanaka left a comment •

edited

Loading

fpetkovski Oct 17, 2024 •

edited

Loading

fpetkovski Oct 17, 2024 •

edited

Loading