receive: close DBReadOnly after flushing

[blockloop]

We are running thanos receive in production with 1w retention. When the receiver starts up it consumes about 20% memory on each of our 32 nodes. After the initial WAL flush the memory footprint progressively grows over time. I originally reported this in CNCF Slack. I initially tried to delete the WAL and restart the nodes but that didn’t solve the issue. The only solution was to clear out some TSDB blocks

Potentially fixes #1855

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Close the DBReadOnly connection after FlushableStorage.Flush()

Verification

I’ve not been able to verify this just yet. I’ll try to build the image with this patch and see if I see an improvement.

[bwplotka]

Nice! While this makes total sense, I doubt this is the potential source of the memory leak, unless you change hashring dynamically and very often.

cc @squat tomorrow (:

[blockloop]

Interesting you mention that. We have a watcher that monitors our nodes and updates the hashring.json file. I noticed some changes it picked up a few times. Is it bad for the hashring to update frequently? I have it do a list | sort on the available nodes and write to the hashring.json file.

[squat]

nice :)

No, it should absolutely not be a problem to update the file frequently. The thanos receive config watcher checks the hash of the file to check it the content has actually changed; if not, then no update is triggered and no cost is incurred.

On the other hand, if the file is actually changing very frequently, then you will be triggering lots of TSDB flushes, which could be costly.