Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ai-plan): optimize KeyboardPress plan type reduce hallucinations #79

Merged
merged 4 commits into from
Aug 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions packages/midscene/src/ai-model/automation/planning.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,10 @@ export function systemPromptToTaskPlanning() {
* param: null
* type: 'Hover', hover the previous element found
* param: null
* type: 'Input', 'KeyboardPress', input something or press a key
* param: { value: string }, Based on the value of the existing input, give the required value of the final input (may add or subtract content from the original),Typing in the input box usually requires clicking input first. the value to input or the key to press. Use (Enter, Shift, Control, Alt, Meta, ShiftLeft, ControlOrMeta, ControlOrMeta) to represent the key.
* type: 'Input', input something
* param: { value: string }, Based on the value of the existing input, give the required value of the final input (may add or subtract content from the original),Typing in the input box usually requires clicking input first.
* type: 'KeyboardPress', press a key
* param: { value: string }, the value to input or the key to press. Use (Enter, Shift, Control, Alt, Meta, ShiftLeft, ControlOrMeta, ControlOrMeta) to represent the key.
* type: 'Scroll'
* param: { scrollType: 'ScrollUntilBottom', 'ScrollUntilTop', 'ScrollDown', 'ScrollUp' }
* type: 'Error'
Expand Down
4 changes: 1 addition & 3 deletions packages/web-integration/tests/ai/e2e/ai-auto-todo.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ test.beforeEach(async ({ page }) => {
const CACHE_TIME_OUT =
Boolean(process.env.MIDSCENE_CACHE) && Boolean(process.env.GITHUB_ACTIONS);

test('ai todo', async ({ ai, aiQuery, aiWaitFor }) => {
test('ai todo', async ({ ai, aiQuery }) => {
if (CACHE_TIME_OUT) {
test.setTimeout(1000 * 30);
}
Expand All @@ -18,8 +18,6 @@ test('ai todo', async ({ ai, aiQuery, aiWaitFor }) => {
'Add "JS today" to the existing content of the task box and press enter',
);

await aiWaitFor('the input box for task title is empty now');

await ai(
'Enter "Learn Rust tomorrow" in the task box, then press Enter to create',
);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ test('ai online order', async ({ ai, page, aiQuery }) => {
await ai('向下滚动一屏');
await ai('点击标准甜、点击绿妍(推荐)、点击标准口味');
await ai('滚动到最下面');
await ai('点击选好了按钮');
await ai('点击页面下边的“选好了”按钮');
await ai('点击右上角商品图标按钮');

const cardDetail = await aiQuery({
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,21 @@
]
},
{
"title": "waitFor: the input box for task title is empty now",
"title": "Enter \"Learn Rust tomorrow\" in the task box, then press Enter to create",
"actions": [
"Insight / Assert",
"Insight / Assert",
"Action / Sleep",
"Insight / Assert"
"Planning",
"Insight / Locate",
"Action / Input",
"Action / KeyboardPress"
]
},
{
"title": "Enter \"Learn Rust tomorrow\" in the task box, then press Enter to create",
"title": "Enter \"Learning AI the day after tomorrow\" in the task box, then press Enter to create",
"actions": [
"Planning",
"Insight / Locate"
"Insight / Locate",
"Action / Input",
"Action / KeyboardPress"
]
}
]
]
Loading