reduce kafka mem allocation

[cartersocha]

Fixes: #780

Reduce kafkas memory allocation after some local testing. If we wanted to drop it further I think we'd need a different image.

[cartersocha]

towards: #780

@puckpuck could you take a look and test on your cluster?

[puckpuck]

I made a few more changes to how long we retain messages in Kafka and Java heap consumption. I pushed it to your branch, and I'm running an image with these changes implemented. I'll ask to wait at least 3 days before we take action, but so far, so good.

[cartersocha]

Nice ! Thanks for refinements

[puckpuck]

:( ... close, but it still restarts after about 28 hours

This one is baffling me. Java Max heap is set to 200M, and we have the container set for 450Mi, but it still hits OOM. If you look closely at the charts you will see it climbs very near to max memory, falls a bit, then right when it restarts there is a sharp sudden spike. Both times it happened after running for about 28 hours.

[puckpuck]

I'm trying a 500Mi resource limit. I will let this run for 3 days and report back.

Note I moved our long-term load testing to an ARM-based EKS cluster starting with this test.

[cartersocha]

I've tried to find a minimum memory requirement or more memory levers on Kafka but no good findings.

One stackoverflow post mentioned 512M is the minimum they would go but who knows how reliable of an answer that is.

[puckpuck]

3 days later, and Kafka is holding memory level without restarts using a 500MB resource limit. I pushed my changes in the branch and feel this is the best we can do using Kafka.

If we want to reduce this further, it will likely require us to use a different message queue platform instead.