Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement capnproto replication #7659

Merged
merged 19 commits into from
Oct 17, 2024
Merged

Implement capnproto replication #7659

merged 19 commits into from
Oct 17, 2024

Commits on Oct 17, 2024

  1. Implement capnproto replication 

    Our profiles from production show that a lot of CPU and memory in receivers
is used for unmarshaling protobuf messages. Although it is not possible to change
the remote-write format, we have the freedom to change the protocol used
for replicating timeseries data.

This commit introduces a new feature in receivers where replication can be done
using Cap'n Proto instead of gRPC + Protobuf. The advantage of the former protocol
is that deserialization is far cheaper and fields can be accessed directly from
the received message (byte slice) without allocating intermediate objects.
There is an additional cost for serialization because we have to convert from
Protobuf to the Cap'n proto format, but in our setup this still results in a net
reduction in resource usage.

Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    7a854cb View commit details
    Browse the repository at this point in the history

  2. Pass logger 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    9f53a82 View commit details
    Browse the repository at this point in the history

  3. Update capnp 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    36af787 View commit details
    Browse the repository at this point in the history

  4. Modify flag 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    4b31096 View commit details
    Browse the repository at this point in the history

  5. Lint 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    503898d View commit details
    Browse the repository at this point in the history

  6. Fix spellcheck 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    9ac8db4 View commit details
    Browse the repository at this point in the history

  7. Use previous version 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    ba5c325 View commit details
    Browse the repository at this point in the history

  8. Update docker base 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    cd8c0bc View commit details
    Browse the repository at this point in the history

  9. Bump go 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    7721d56 View commit details
    Browse the repository at this point in the history

  10. Update docs/components/receive.md 

    Co-authored-by: Pedro Tanaka <[email protected]>
Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski @pedro-stanaka
    fpetkovski and pedro-stanaka committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    2057bdf View commit details
    Browse the repository at this point in the history

  11. Validate labels 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    a6fda55 View commit details
    Browse the repository at this point in the history

  12. e2e: add receive test with capnp replication 

    Signed-off-by: Giedrius Statkevičius <[email protected]>
    @GiedriusS @fpetkovski
    GiedriusS authored and fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    acfdae2 View commit details
    Browse the repository at this point in the history

  13. receive: make copy only when necessary 

    Signed-off-by: Giedrius Statkevičius <[email protected]>
    @GiedriusS @fpetkovski
    GiedriusS authored and fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    2f9878c View commit details
    Browse the repository at this point in the history

  14. Fix failing test 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    1193690 View commit details
    Browse the repository at this point in the history

  15. Add CHANGELOG entry 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    bcc3f6a View commit details
    Browse the repository at this point in the history

  16. Add capnproto Make target 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    186c089 View commit details
    Browse the repository at this point in the history

  17. Replace panics with errors 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    5f908a3 View commit details
    Browse the repository at this point in the history

  18. Fix benchmark 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    67ef38d View commit details
    Browse the repository at this point in the history

  19. Fix CHANGELOG 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    d337e0a View commit details
    Browse the repository at this point in the history