[Core][State Observability][Log] gRPC max size limit when ray logs --tail 1000000

[rkooo567]

What happened + What you expected to happen

Run non_streaming_shuffle_1tb_1000_partition with a debug log on.

Run ray logs raylet.out --tail 1000000 causes gRPC max size limit. It should never happen in theory since we stream logs.

(base) ray:~/e2e-tests% ray logs raylet.out --tail 1000000
Traceback (most recent call last):
  File "/home/ray/anaconda3/bin/ray", line 8, in <module>
    sys.exit(main())
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/scripts/scripts.py", line 2539, in main
    return cli()
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/scripts/scripts.py", line 2120, in logs
    timeout=timeout,
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/experimental/state/api.py", line 805, in get_log
    raise RayStateApiException(error_msg)
ray.experimental.state.exception.RayStateApiException: 0Closing HTTP stream due to internal server error.
<AioRpcError of RPC that terminated with:
        status = StatusCode.RESOURCE_EXHAUSTED
        details = "Received message larger than max (195781839 vs. 4194304)"
        debug_error_string = "{"created":"@1658831854.038323713","description":"Received message larger than max (195781839 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":205,"grpc_status":8}"
>

Versions / Dependencies

Master

Reproduction script

Run non_streaming_shuffle_1tb_1000_partition with a debug log on.

Run ray logs raylet.out --tail 1000000

Issue Severity

No response

[rkooo567]

cc @rickyyx this is the issue we talked today. I think we should fix it after the branch cut. When specifying --tail -1, it worked

[rickyyx]

Yeah, I think this is the fact we only do chunking when tail == -1 in the log agent when serving the stream. I will see if I could get to it by branch-cut.