Increase memory & disk size limit for pangeo prometheus #1906
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
While investigating why the pangeo hubs prometheus was dead, I discovered prometheus/prometheus#6934 - where there can be a big temporary *spike* in memory usage when prometheus is recovering from a restart. It tries to read the WAL (the write-ahead log), to make sure it hasn't lost any data during the restart process itself. The details of the WAL are unimportant (in this specific case), but just the fact that prometheus spikes memory usage on restarts! I manually hand edited the prometheus deployment with `k -n support edit deployment support-prometheus-server`, and gave it a higher limit (8G). Then I watched actual memory usage, with `watch kubectl -n support top pod`. I noticed that it momentarily spiked to almost 5G, before settling back to about 1.5G. The old memory limit was 4G, so during the spike the server gets killed! And then enters crashloopbackoff, as it can never survive. This commit raises the memory limit, so it won't keep crashing :) I also actually manually increased the size of the disk (with `kubectl -n support edit pvc`), but that wasn't the problem. However, we need to persist the change regardless, so here it is. Hopefully this will fix 2i2c-org#1843
5 tasks
|
@yuvipanda can we document the manual debug steps here please? https://infrastructure.2i2c.org/en/latest/sre-guide/common-problems-solutions.html In case this happens on another cluster in the future. |
sgibson91
approved these changes
Nov 14, 2022
|
will do, @sgibson91! |
16 tasks
|
🎉🎉🎉🎉 Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/runs/3465810690 |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While investigating why the pangeo hubs prometheus was dead, I discovered prometheus/prometheus#6934 - where there can be a big temporary spike in memory usage when prometheus is recovering from a restart. It tries to read the WAL (the write-ahead log), to make sure it hasn't lost any data during the restart process itself. The details of the WAL are unimportant (in this specific case), but just the fact that prometheus spikes memory usage on restarts!
I manually hand edited the prometheus deployment with
k -n support edit deployment support-prometheus-server, and gave it a higher limit (8G). Then I watched actual memory usage, withwatch kubectl -n support top pod. I noticed that it momentarily spiked to almost 5G, before settling back to about 1.5G. The old memory limit was 4G, so during the spike the server gets killed! And then enters crashloopbackoff, as it can never survive.This commit raises the memory limit, so it won't keep crashing :)
I also actually manually increased the size of the disk (with
kubectl -n support edit pvc), but that wasn't the problem. However, we need to persist the change regardless, so here it is.Hopefully this will fix #1843