[Data] Refine estimate of object store memory usage from pending task outputs

[bveeramani]

Why are these changes needed?

Ray Core buffers blocks produced by running tasks. To estimate the amount of object store usage from buffered blocks, we use the following formula:

ray/python/ray/data/_internal/execution/interfaces/op_runtime_metrics.py

Lines 189 to 191 in de439b4

	self.num_tasks_running
	* estimated_bytes_per_output
	* context._max_num_blocks_in_streaming_gen_buffer

This expression is an overestimation because of the following reasons:

1. num_tasks_running represents the number of launched tasks, not the number of actually running tasks. This is especially an issue with ActorPoolMapOperator, because we launch multiple tasks per actor, but only one task runs at at time per actor.
2. context._max_num_blocks_in_streaming_gen_buffer the maximum number of buffered blocks, not the typical number of buffered blocks. Often, tasks only produce one output, so the number of buffered outputs is at most one.

This PR addresses the two issues described above.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[bveeramani]

I don't like how this couples OpRuntimeMetrics with ActorPoolMapOperator, but not sure how else to implement it. Open to alternative implementations.

[raulchen]

we can also add a method on the operator side to let it report the actual number. but that sounds a bit too over design. current way is fine as well.