Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

In assisted decoding, pass model_kwargs to model's forward call (fix prepare_input_for_generation in all models) #25242

Merged
merged 7 commits into from
Oct 11, 2023
Merged

In assisted decoding, pass model_kwargs to model's forward call (fix prepare_input_for_generation in all models) #25242

merged 7 commits into from
Oct 11, 2023

Commits on Oct 11, 2023

  1. In assisted decoding, pass model_kwargs to model's forward call 

    Previously, assisted decoding would ignore any additional kwargs
that it doesn't explicitly handle. This was inconsistent with other
generation methods, which pass the model_kwargs through
prepare_inputs_for_generation and forward the returned dict to the
model's forward call.

The prepare_inputs_for_generation method needs to be amended in all
models, as previously it only kept the last input ID when a past_key_values
was passed.
    @sinking-point
    sinking-point committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    d8dd986 View commit details
    Browse the repository at this point in the history

  2. Improve variable names in _extend_attention_mask

    @sinking-point
    sinking-point committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    977dd72 View commit details
    Browse the repository at this point in the history

  3. Refactor extending token_type_ids into a function

    @sinking-point
    sinking-point committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    8ce58b4 View commit details
    Browse the repository at this point in the history

  4. Replace deepcopy with copy to optimize performance

    @sinking-point
    sinking-point committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    81a04c2 View commit details
    Browse the repository at this point in the history

  5. Update new persimmon model with llama changes for assisted generation

    @sinking-point
    sinking-point committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    2ee2e79 View commit details
    Browse the repository at this point in the history

  6. Update new mistral model for assisted generation with prepare_inputs_… 

    …for_generation
    @sinking-point
    sinking-point committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    008e7ca View commit details
    Browse the repository at this point in the history

  7. Update position_ids creation in falcon prepare_inputs_for_generation … 

    …to support assisted generation
    @sinking-point
    sinking-point committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    8ce040d View commit details
    Browse the repository at this point in the history