Relevant:
#757

When running the test from @infiniteregrets's branch on Mac, but changing the test to send all the requests 30 times in a loop, at some point the agent sends a ConnectTimedOut back to the layer and the application gets an ENETUNREACH. Is this the bug reported here, or is this another, unrelated issue? (it does not happen when running that test app without mirrord)

When running the test from @infiniteregrets's branch on Mac, but changing the test to send all the requests 30 times in a loop, at some point the agent sends a ConnectTimedOut back to the layer and the application gets an ENETUNREACH. Is this the bug reported here, or is this another, unrelated issue? (it does not happen when running that test app without mirrord)

I don't think we managed to reproduce the issue locally. It only happens in E2E (after the fix) - before the fix we had another issue (or same?) that occurred locally.

But what was the error? Was it ENETUNREACH?

But what was the error? Was it ENETUNREACH?

Before the fix, locally? It was connection reset.

Do we have the exact node error? Is it the same issue as #564?

Seems so, yes.

Anyone knows if this still happens or how to check if it still happens? I believe the internal proxy refactor should've solved it.

I have stress tested this on main locally and this does not happen. I will give a try on the CI with an e2e, if that passes will add an integration test. related to #1484

Looks like this passes with 30 loops of makeRequests() in CI now (https://github.com/metalbear-co/mirrord/actions/runs/10794249050/job/29938137749?pr=2748) but fails locally on more than about 10 loops - feels like it could be a workload problem where lots of requests are increasing the latency?
Weirdly sometimes the test_outgoing_traffic_many_requests_disabled (running the test without mirrord) also fails with ECONNRESET which makes me think this isn't a mirrord problem
EDIT After a few more runs im not convinced there is any pattern in the failures (other than more loops is more bad)

A short(ish) summary of investigation

Methods attempted:

• in CI, both with and without mirrord (tests test_outgoing_traffic_many_requests_enabled/_disabled respectively)
• locally on Mac M1, MacOS 14.6.1, via tests above and manually using mirrord as well as running node without mirrord
  All of the above was with the script in tests/node-e2e/outgoing/test_outgoing_traffic_many_requests.mjs - changing the number of loops to make more requests

Observations:

• the error that occurs (socket hang up) is a timeout issue - the client has waited too long for a response since sending the request and force closes the socket
• mirrord does not emit any interesting logs that suggest unexpected errors/ weird behaviour, even when using mirrord-console
• in node, all requests are sent before any responses are received, both with and without mirrord
• I believe this reveals that the issue is caused by all the requests (critical mass locally with mirrord was around 240 requests sent one after the other) competing for cpu time and choking eachother out - mirrord can only do so many things at once
• running locally without mirrord succeeds until around 2400 requests, at which point the same error occurs - any computer can only do so many things at once
• running a similar script in python did not encounter the same issue, probably because the script blocked on receiving the response from each request before making the next
• in node, the issue is completely fixed if you add a second of sleep() between each batch of 12 requests
• sending requests to localhost in a cluster does not cause the same issue (I assume because it's so much faster that it never reaches the point of serious cpu time contention)
• when running in CI, sending enough requests causes both test_outgoing_traffic_many_requests_enabled/_disabled to fail, just like running the script locally

Closing this as it seems it's not a bug on our end, just mirrord exacerbating timeouts with its (negligible) added latency, which is a known issue.