[AIR] Execute GPU inference in a separate stage in BatchPredictor #26616
Conversation
Signed-off-by: Eric Liang <[email protected]>
| @@ -231,6 +253,9 @@ def predict_pipelined( | |||
| max_scoring_workers: If set, specify the maximum number of scoring actors. | |||
| num_cpus_per_worker: Number of CPUs to allocate per scoring worker. | |||
| num_gpus_per_worker: Number of GPUs to allocate per scoring worker. | |||
| separate_gpu_stage: If using GPUs, specifies whether to execute GPU | |||
There was a problem hiding this comment.
Is there a case when we would not want this enabled?
There was a problem hiding this comment.
If the preprocessor is very lightweight, then enabling this could hurt more than it helps, due to excess disk spilling (in non-pipelined mode).
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Matthew Deng <[email protected]>
Signed-off-by: Matthew Deng <[email protected]>
Signed-off-by: Matthew Deng <[email protected]>
Signed-off-by: Matthew Deng <[email protected]>
Signed-off-by: Matthew Deng <[email protected]>
Signed-off-by: Matthew Deng <[email protected]>
Signed-off-by: Matthew Deng <[email protected]>
Signed-off-by: Matthew Deng <[email protected]>
ericl
changed the title
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
| if num_gpus_per_worker is None: | ||
| num_gpus_per_worker = 0 | ||
| if num_cpus_per_worker is None: | ||
| if num_gpus_per_worker > 0: |
There was a problem hiding this comment.
should we be doing this across all of our library code 😅
There was a problem hiding this comment.
Yeah, it's probably a good idea.
| if separate_gpu_stage and num_gpus_per_worker > 0: | ||
| preprocessor = self.get_preprocessor() | ||
| if preprocessor: | ||
| override_prep = BatchMapper(lambda x: x) |
There was a problem hiding this comment.
doesnt seem like override_prep is being used anywhere?
There was a problem hiding this comment.
Oh, that's weird LINT didn't catch it.
There was a problem hiding this comment.
Oh it's actually a local variable from the previous PR.
There was a problem hiding this comment.
I think there might be a set_preprocessor line missing to disable the preprocessor for the predictor.
There was a problem hiding this comment.
It's actually correct--- I modified the unit test. If you remove this line, the test will fail. Also added a clarifying comment.
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
Co-authored-by: matthewdeng <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
…y-project#26616) Signed-off-by: Rohan138 <[email protected]>
…y-project#26616) Signed-off-by: Stefan van der Kleij <[email protected]>
Why are these changes needed?
This PR is stacked on #26600