-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Anthropic claude-3-haiku model with text input #60
Comments
Hey @giordafrancis The changes from Claude v2 to Claude v3. The payload sent to Claude v3 differs from v2. Therefore the payload preparation done by this app needs to be updated/expanded to accommodate v3. I will edit the issue title to be a feature request for the model. This is also faced in other projects
In the mean time you can use the v2 models, with example here. From my initial look at this it appears to be because Claude v3 support different types of input in addition to text, whereas v2 was just text. So the payload needs to specify that it is indeed text. |
Initial support has been created upstream, but not in a tagged release yet, as it'll need some testing done. A dependency (CDK) also has an open issue that affects the custom resources for this project. This means we can't deploy the example models as references. It'd be nice to have this resolved for the next release. |
https://github.com/aws-samples/document-translation/releases/tag/v2.3.0 Please test and review. |
Hi @philipws thank you for this and apologies for the delay in testing. I've now upgraded to v2.3.5 and pushed to codecommit. I've added the below item to
|
Hey @giordafrancis tl:dr; Try changing Anthropic changed the payload structure from the v2 to v3 models. It looks like you've edited the provided example which is made for v2. It appears that max_tokens_to_sample does not exist for the messages API https://docs.anthropic.com/en/api/messages The v3 structure looks more like the below. Simply Readable takes care of building the text part, so we can ignore that, and focus on the other parameters. One that is different is {
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "hello world"
}
]
}
],
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 2000,
"temperature": 1,
"top_k": 250,
"top_p": 0.999,
"stop_sequences": ["\\n\\nHuman:"]
} For reference, within the AWS Bedrock console text playground you can select "View API Request". The payload is an escaped string value in the body field. I find this very helpful in taking what I've tested in Bedrock into the app. |
@philipws Thank you, that did it and worked like a charm. Feeling slightly embarrassed for not spotting that given that I've used v3 API in the past. Also thank you for the "View API request", a super helpful pointer. |
@philipws I've been unable to define a model definition structure on
DocTran-<branch>-app-basereadablereadableModelmodelTable-xxxxxxxx-xxxxxxxxxxxxx
for modelanthropic.claude-3-haiku-20240307-v1:0
. Below is my attempt to create a structure. The Text will not generate in the app.Any ideas about what is incorrect with the structure?
{ "id": { "S": "Text-01" }, "default": { "BOOL": true }, "name": { "S": "Text Simplifier" }, "text": { "M": { "modelId": { "S": "anthropic.claude-3-haiku-20240307-v1:0" }, "parameters": { "M": { "max_tokens": { "N": "300" }, "anthropic_version": { "S": "bedrock-2023-05-31" }, "stop_sequences": { "L": [ { "S": "Human:" } ] }, "temperature": { "N": "1" }, "top_k": { "N": "250" }, "top_p": { "N": "0.999" } } }, "prePrompt": { "S": "Simplify the following text so it can be easily understood by those with a low reading age, use short sentences, explain any abbreviations or words that can have two meanings and separate the sentences in to new lines." } } }, "type": { "S": "text" } }
The text was updated successfully, but these errors were encountered: