Skip to content

Commit

Permalink
[python] Update vllm rolling batcher sampling params for 0.6.0 support (
Browse files Browse the repository at this point in the history
  • Loading branch information
tosterberg committed Sep 11, 2024
1 parent eeb435d commit 1d6d763
Showing 1 changed file with 4 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,10 @@
from djl_python.properties_manager.vllm_rb_properties import VllmRbProperties
from typing import List

VLLM_GENERATION_PARAMS = set(SamplingParams().__dict__.keys())
# FIXME: Once all vllm versions are past 0.6.0 we can move to just struct_fields
VLLM_GENERATION_PARAMS = set(SamplingParams().__struct_fields__) if hasattr(
SamplingParams(), "__struct_fields__") else set(
SamplingParams().__dict__.keys())


class VLLMRollingBatch(RollingBatch):
Expand Down

0 comments on commit 1d6d763

Please sign in to comment.