Steal + http filter passthrough doesn't work with large responses

[eyalb181]

Bug Description

When stealing and sending a request without the header with a large response with the following filter configuration:
"http_filter": { "header_filter": "x-mirrord-key: test", "ports": [50051, 8080] }

The agent crashes with the following logs:

Steps to Reproduce

Steal a request which returns a large response (~8mb).

Backtrace

No response

Relevant Logs

Your operating system and version

N/A

Local process

N/A

Local process version

No response

Additional Info

No response

[aviramha]

Archive.zip

Example that reproduces it is here.
Scenario:

1. Run greet server with http filter based on header.
2. Send a request using the client with no arguments that matches (compile with the headers commented out line) - works
3. Send a request using the client with no arguments that doesn't match (with the headers/metadata) - works
4. Send a request using the client with -name a that matches - works
5. Send a request using the client with -name a that doesn't match - same error.

[aviramha]

with RUST_BACKTRACE=full in agent:

2023-08-30T15:57:21.077511Z  WARN ThreadId(08) mirrord_agent::steal::ip_tables::flush_connections: `conntrack` output is Output {
    status: ExitStatus(
        unix_wait_status(
            256,
        ),
    ),
    stdout: "",
    stderr: "conntrack v1.4.7 (conntrack-tools): 0 flow entries have been deleted.\n",
}
thread '<unnamed>' panicked at 'cannot poll Select twice', /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/future/select.rs:112:42
stack backtrace:
   0:     0x563122d0eac2 - <unknown>
   1:     0x563122d3bb7f - <unknown>
   2:     0x563122d0a081 - <unknown>
   3:     0x563122d0e8c5 - <unknown>
   4:     0x563122d100a7 - <unknown>
   5:     0x563122d0fe95 - <unknown>
   6:     0x563122d105b1 - <unknown>
   7:     0x563122d104a9 - <unknown>
   8:     0x563122d0ef06 - <unknown>
   9:     0x563122d10202 - <unknown>
  10:     0x563122d391e3 - <unknown>
  11:     0x563122d38f83 - <unknown>
  12:     0x5631224d29db - <unknown>
  13:     0x5631225dfeb9 - <unknown>
  14:     0x563122836111 - <unknown>
  15:     0x56312272af61 - <unknown>
  16:     0x56312260b0fe - <unknown>
  17:     0x5631224c8a01 - <unknown>
  18:     0x56312260b358 - <unknown>
  19:     0x563122892417 - <unknown>
  20:     0x5631224c943c - <unknown>
  21:     0x5631226c023d - <unknown>
  22:     0x563122868a79 - <unknown>
  23:     0x563122d15ae5 - <unknown>
  24:     0x7f6c03801044 - <unknown>
  25:     0x7f6c038815fc - <unknown>

crap.

[aviramha]

seems related - hyperium/hyper#3289 will test with this.

[aviramha]

Fixed by #1876