fix: message cache unordered removal

[SionoiS]

Fixes a crash in message cache because the code assumed messages to be removed were ordered.

I also added a basic fuzzing test.

Closes #2680

[github-actions]

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2682-rln-v1

Built from 18916e6ea0c7283b0e9ec7dbc5d7353c72af3523

[github-actions]

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2682-rln-v2

Built from 18916e6ea0c7283b0e9ec7dbc5d7353c72af3523

[gabrielmer]

LGTM, thank you!

[vpavlin]

quay.io/wakuorg/nwaku-pr:2682-rln-v1

@erhant If you feel like it, you could use this image to check if it actually fixes your issues

[erhant]

quay.io/wakuorg/nwaku-pr:2682-rln-v1

@erhant If you feel like it, you could use this image to check if it actually fixes your issues

awesome, thanks!

[erhant]

update: the new image above worked like charm, great work!

For the record, we were able to reproduce the bug with @anilaltuner as follows:

1. he and I run a node at the same time (using the same wallet if that makes a difference). we have to content topics heartbeat and synthesis, both topics are published by him and subscribed by my node.
2. his node keeps sending heartbeat messages, to which my node responds with a unique content topic w.r.t the payload; think of this like simple ping-pong for health check
3. at some point, he sends my node a synthesis message which includes a prompt for LLM, I obtain the result from LLM and send my payload back; note that these are two threads, so heartbeat checks are going on in the background while I do my synthesis work
4. around 2-3 seconds after I see my log of "sending message" for synthesis, I see that heartbeat checks are failing because the server is down, and thats how I see nwaku is closed due to error
5. looking at the logs, its the same error (e.g. 2024-05-08 18:46:59 Error: unhandled exception: index 28 not in 0 .. 24 [IndexDefect])

we are working with https://github.com/firstbatchxyz/dkn-compute-node for the context*

anyways, im confident that this is fixed now

[NagyZoltanPeter]

Shall I say LGTM? Yes it is, but this code definetely lacks some explanatory comments, like why the descendant sorting needed.
Sometime we might think of implementing a multi index container, that can simplify such code a lot.

[SionoiS]

Shall I say LGTM? Yes it is, but this code definetely lacks some explanatory comments, like why the descendant sorting needed. Sometime we might think of implementing a multi index container, that can simplify such code a lot.

I went with the easy fix and I agree that the message cache impl. should be rewritten.