Logs get lost under high load

[slinkydeveloper]

Might be related to this: https://github.com/docker-java/docker-java/blob/97579e5e330e16ed26fef25ada4de472aa998c56/docker-java-core/src/main/java/com/github/dockerjava/core/DefaultInvocationBuilder.java#L281

[slinkydeveloper]

One idea could be to collect logs just at the end of the tests, but this apparently doesn't always work as some logs might have been lost in the meantime: https://github.com/restatedev/e2e/tree/trying_to_change_logging_tools

[slinkydeveloper]

Another completely different solution for logs would be to just pick them up from journald (which keeps them even after the container is removed). Podman by default sends logs to journald, and docker supports that as well https://docs.docker.com/config/containers/logging/journald/ (although i don't know what is the default there). But we need to make sure this solution works on MacOS (and how complicated is to setup wrt privileges)

[tillrohrmann]

Can this be closed @slinkydeveloper? I remember faintly that you've solved it by switching to some other log daemon or so.

[slinkydeveloper]

On my machine yes, but i'm not sure this affects CI as well. Let me keep it open, and revisit once I enable CI in other repos as well.