Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove tool usage for non current turns when looking up message history #2174

Merged
merged 2 commits into from
Oct 31, 2024

Conversation

sobolk
Copy link
Member

@sobolk sobolk commented Oct 31, 2024

Problem

Tool use and result blocks become irrelevant after turn is completed.
We persist "client side tools" blocks in history (as they require UI -> Appsync -> Lambda -> Bedrock roundtrip to work).

Including non-current turn's tools blocks in message history isn't incorrect but it costs tokens to process by Bedrock. Therefore, we remove them.

Changes

Add logic to message history retrieval that:

  1. Finds where current turn begins
  2. Removes any tool use/result blocks from non-current turns.
  3. Squashes messages that belong to the same "role" (in case tool removal leaves continuous sequences of them).

Validation

  1. Added unit test with most extreme case.
  2. Existing unit tests cover the rest.

Checklist

  • If this PR includes a functional change to the runtime behavior of the code, I have added or updated automated test coverage for this change.
  • If this PR requires a change to the Project Architecture README, I have included that update in this PR.
  • If this PR requires a docs update, I have linked to that docs PR above.
  • If this PR modifies E2E tests, makes changes to resource provisioning, or makes SDK calls, I have run the PR checks with the run-e2e label set.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@sobolk sobolk added the run-e2e Label that will include e2e tests in PR checks workflow label Oct 31, 2024
Copy link

changeset-bot bot commented Oct 31, 2024

🦋 Changeset detected

Latest commit: c321517

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@aws-amplify/ai-constructs Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@sobolk sobolk marked this pull request as ready for review October 31, 2024 14:53
@sobolk sobolk requested a review from a team as a code owner October 31, 2024 14:53
@sobolk sobolk merged commit 613bca9 into main Oct 31, 2024
70 checks passed
@sobolk sobolk deleted the tool-use-history branch October 31, 2024 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
run-e2e Label that will include e2e tests in PR checks workflow
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants