feat: add a playground page to debug (#131)

* fix: fix the Player style when the locator is failed * feat: add Playground UI --------- Co-authored-by: zhouxiao.shaw <[email protected]>
web-infra-dev · Oct 21, 2024 · 7d4f984 · 7d4f984
1 parent f312cec
commit 7d4f984
Show file tree

Hide file tree

Showing 83 changed files with 14,061 additions and 3,720 deletions.
diff --git a/README.ja.md b/README.ja.md
@@ -31,15 +31,14 @@ Midscene.jsは、自然言語を使用してページを制御し、アサーシ
 - **UIを理解し、JSONで回答 🔍**: データ形式に関するプロンプトを提供し、期待される応答をJSON形式で受け取ります。
 - **直感的なアサーション 🤔**: 自然言語でアサーションを行います。すべてAIの理解に基づいています。
 - **すぐに使えるLLM 🪓**: GPT-4oのような公開のマルチモーダルLLMを使用できます。カスタムトレーニングは不要です。
-- **可視化 🎞️**: 可視化ツールを使用して、プロセス全体を簡単に理解し、デバッグできます。
+- **可視化レポート 🎞️**：私たちのテストレポートとPlaygroundを通じて、全体のプロセスを簡単に理解し、デバッグすることができます。
 - **全く新しい体験! 🔥**: 自動化開発の全く新しい世界を体験してください。楽しんでください！
 
 ## リソース 📄
 
 * [ホームページ: https://midscenejs.com](https://midscenejs.com/)
 * [クイックスタート](https://midscenejs.com/docs/getting-started/quick-start.html)
 * [APIリファレンス](https://midscenejs.com/docs/usage/API.html)
-* [可視化ツール](https://midscenejs.com/visualization/index.html)
 
 ## ライセンス
 

diff --git a/README.md b/README.md
@@ -33,15 +33,14 @@ Midscene.js is an AI-powered automation SDK can control the page, perform assert
 - **Understand UI, Answer in JSON 🔍**: Provide prompts regarding the desired data format, and then receive the expected response in JSON format.
 - **Intuitive Assertion 🤔**: Make assertions in natural language; it’s all based on AI understanding.
 - **Out-of-box LLM 🪓**: It is fine to use public multimodal LLMs like GPT-4o. There is no need for any custom training.
-- **Visualization 🎞️**: With our visualization tool, you can easily understand and debug the whole process.
+- **Visualized Report 🎞️**: With our visualized report file, you can easily understand and debug the whole process.
 - **Brand New Experience! 🔥**: Experience a whole new world of automation development. Enjoy!
 
 ## Resources 📄
 
 * [Home Page: https://midscenejs.com](https://midscenejs.com/)
 * [Quick Start](https://midscenejs.com/docs/getting-started/quick-start.html)
 * [API Reference](https://midscenejs.com/docs/usage/API.html)
-* [Visualization Tool](https://midscenejs.com/visualization/index.html)
 
 ## License
 

diff --git a/README.zh.md b/README.zh.md
@@ -32,15 +32,14 @@ Midscene.js 是一个由 AI 驱动的自动化 SDK，能够使用自然语言对
 - **理解UI、JSON格式回答 🔍**：你可以提出关于数据格式的要求，然后得到 JSON 格式的预期回应。
 - **直观断言 🤔**：用自然语言表达你的断言，AI 会理解并处理。
 - **开箱即用的LLM 🪓**：使用公开的多模态大语言模型（ 如GPT-4o ），无需任何定制训练。
-- **可视化 🎞️**：通过我们的可视化工具，你可以轻松理解和调试整个过程。
+- **可视化报告 🎞️**：通过我们的测试报告和 Playground，你可以轻松理解和调试整个过程。
 - **全新体验 🔥**：体验全新的自动化开发世界，尽情享受吧！
 
 ## 资源 📄
 
 * [官网首页: https://midscenejs.com](https://midscenejs.com/)
 * [快速入门](https://midscenejs.com/docs/getting-started/quick-start.html)
 * [API 文档](https://midscenejs.com/docs/usage/API.html)
-* [可视化工具](https://midscenejs.com/visualization/index.html)
 
 ## 授权许可
 

diff --git a/apps/site/.gitignore b/apps/site/.gitignore
@@ -23,3 +23,7 @@ dist-ssr
 *.sln
 *.sw?
 doc_build
+
+# Midscene.js dump files
+midscene_run/report
+midscene_run/dump
diff --git a/apps/site/docs/en/_meta.json b/apps/site/docs/en/_meta.json
@@ -3,10 +3,5 @@
     "text": "Docs",
     "link": "/docs/getting-started/introduction",
     "activeMatch": "/docs"
-  },
-  {
-    "text": "Visualization Tool",
-    "link": "/visualization/",
-    "activeMatch": "/visualization/"
   }
 ]
diff --git a/apps/site/docs/en/docs/getting-started/introduction.mdx b/apps/site/docs/en/docs/getting-started/introduction.mdx
@@ -36,15 +36,17 @@ Quickly integrate GPT-4o and web automation tools like Playwright or Puppeteer i
 
 You can setup your automation now with all your familiar tools. No custom training is need.
 
-### Visualization Tool
+### Visualized Report
 
-With our visualization tool, you can easily debug the prompt and AI response. All intermediate data, such as queries, plans, and actions, can be visualized.
+Midscene will give a visualized report and playground after running. All intermediate data, such as queries, plans, and actions, can be visualized.
 
-You may open the [Online Visualization Tool](/visualization/index.html) to see the showcase.
+![](/report.gif)
 
-![](/visualizer.jpg)
+You can even debug the report in local Midscene playground.
 
-## Flow Chart
+![](/playground.png)
+
+## How it works
 
 Here is a flowchart that describes the core process of the interaction between Midscene and AI.
 

diff --git a/apps/site/docs/en/docs/getting-started/quick-start.mdx b/apps/site/docs/en/docs/getting-started/quick-start.mdx
@@ -37,9 +37,16 @@ npx @midscene/cli \
 
 If you want to dive deep into Midscene, we recommend using the SDK version and integrating it with Playwright or Puppeteer.
 
-### View test report after running
+### View test report
 
-After running, Midscene will generate a log dump, which is placed in `./midscene_run/report/latest.web-dump.json` by default. Then put this file into [Visualization Tool](/visualization/), and you will have a clearer understanding of the process.
+After running, Midscene will generate a visualized report file in HTML. Open it with browser and you will have a clearer understanding of the process.
+
+If you want to rerun a prompt together with UI context from the report file, just launch a Playground server and click the "Send to Playground" in the report.
+
+To launch the local Playground server:
+```
+npx --yes @midscene/web
+```
 
 ## Integrate with Playwright
 
@@ -202,14 +209,6 @@ npx ts-node demo.ts
 # ]
 ```
 
-### Step 4: View the Run Report
-
-After the above command executes successfully, the console will output: `Midscene - report file updated: ./current_cwd/midscene_run/report/some_id.html`. You can open this file in a browser to view the report.
-
-Alternatively, you can import the `./midscene_run/report/latest.web-dump.json` file into the [Visualization Tool](/visualization/) to view it.
-
-## View demo report
-
-Click the 'Load Demo' button in the [Visualization Tool](/visualization/), you will be able to see the results of the previous code as well as some other samples.
+### Step 4: view the report
 
-![](/view-demo-visualization.gif)
+After the above command executes successfully, the console will output: `Midscene - report file updated: /path/to/report/some_id.html`. You can open this file in a browser to view the report.
diff --git a/apps/site/docs/en/docs/more/prompting-tips.md b/apps/site/docs/en/docs/more/prompting-tips.md
@@ -30,9 +30,14 @@ Bad ❌: "string, hex value of text color"
 
 Bad ❌: "[number, number], the [x, y] coords of the main button"
 
-### Use visualization tool to debug
+### Use report file and playground tool to debug
 
-Use the visualization tool to debug and understand each step of Midscene. Just upload the log, and view the AI's parse results. You can find [the tool](/visualization/) on the navigation bar on this site. 
+Open the report file, you will see the detailed information about the steps. If you want to rerun a prompt together with UI context from the report file, just launch a Playground server and click "Send to Playground".
+
+To launch the local Playground server:
+```
+npx --yes @midscene/web
+```
 
 ### Infer or assert from the interface, not the DOM properties or browser status
 

diff --git a/apps/site/docs/en/docs/usage/API.md b/apps/site/docs/en/docs/usage/API.md
@@ -40,7 +40,7 @@ Steps should always be clearly and thoroughly described. A very brief prompt lik
 
 Under the hood, Midscene will plan the detailed steps by sending your page context and a screenshot to the AI. After that, Midscene will execute the steps one by one. If Midscene deems it impossible to execute, an error will be thrown. 
 
-The main capabilities of Midscene are as follows, and your task will be split into these types. You can see them in the visualization tools:
+The main capabilities of Midscene are as follows, and your task will be split into these types. You can see them in the visualized report:
 
 1. **Locator**: Identify the target element using a natural language description
 2. **Action**: Tap, scroll, keyboard input, hover

diff --git a/apps/site/docs/en/index.md b/apps/site/docs/en/index.md
@@ -30,8 +30,8 @@ features:
   - title: Immediate Integration
     details: Quickly integrate GPT-4o and web automation tools like Playwright or Puppeteer into your project with our SDK. Instant setup and seamless integration.
     icon: 🪓
-  - title: Visualization
-    details: With our visualization tool, you can easily understand and debug the whole process.
+  - title: Visualized Report
+    details: With our visualized report file, you can easily understand and debug the whole process.
     icon: 🎞️
   - title: Brand New Experience !
     details: Experience a whole new world of automation development. Enjoy !

diff --git a/apps/site/docs/public/playground.png b/apps/site/docs/public/playground.png
diff --git a/apps/site/docs/public/report.gif b/apps/site/docs/public/report.gif
diff --git a/apps/site/docs/public/view-demo-visualization.gif b/apps/site/docs/public/view-demo-visualization.gif
diff --git a/apps/site/docs/public/visualizer.jpg b/apps/site/docs/public/visualizer.jpg
diff --git a/apps/site/docs/zh/_meta.json b/apps/site/docs/zh/_meta.json
@@ -3,10 +3,5 @@
     "text": "文档",
     "link": "/docs/getting-started/introduction",
     "activeMatch": "/docs"
-  },
-  {
-    "text": "可视化工具",
-    "link": "/visualization/",
-    "activeMatch": "/visualization/"
   }
 ]
diff --git a/apps/site/docs/zh/docs/getting-started/introduction.mdx b/apps/site/docs/zh/docs/getting-started/introduction.mdx
@@ -1,6 +1,8 @@
 # 介绍
 
-UI 自动化太难写了。自动化脚本里到处都是选择器，比如 `#ids`、`data-test`、`.selectors`。在页面重构的时候，维护自动化脚本更将会是一场灾难。
+UI 自动化太难维护了。
+
+UI 自动化脚本里到处都是选择器，比如 `#ids`、`data-test`、`.selectors`。在页面重构的时候，维护自动化脚本更将会是一场灾难。
 
 我们在这里推出 Midscene.js，助你重拾编码的乐趣。
 
@@ -33,15 +35,17 @@ const items = await aiQuery(
 console.log("headphones in stock", items);
 ```
 
-### 可视化工具
+### 可视化报告
+
+运行结束后，Midscene 会提供可视化报告和调试 Playground，可以非常方便地调试提示和 AI 的响应。所有的中间数据，例如查询（Query）、计划（Planning）和动作（Actions），都可以被可视化。
 
-我们提供的可视化工具，可以非常方便地调试提示和 AI 的响应。所有的中间数据，例如查询（Query）、计划（Planning）和动作（Actions），都可以被可视化。
+![](/report.gif)
 
-你可以打开 [可视化工具](/visualization/index.html) 来查看示例。
+你还可以在本地 Midscene Playground 中变更 prompt，调试报告中的内容。
 
-![](/visualizer.jpg)
+![](/playground.png)
 
-## 流程图
+## Midscene 是如何运行的？
 
 下图展示了 Midscene 的核心流程。
 

diff --git a/apps/site/docs/zh/docs/getting-started/quick-start.mdx b/apps/site/docs/zh/docs/getting-started/quick-start.mdx
@@ -41,7 +41,14 @@ npx @midscene/cli \
 
 ### 查看运行报告
 
-运行 Midscene 之后，系统会生成一个日志文件，默认存放在 `./midscene_run/report/latest.web-dump.json`。然后，你可以把这个文件导入 [可视化工具](/visualization/)，这样你就能更清楚地了解整个过程。
+运行 Midscene 之后，系统会生成一个日志报告（HTML 文件）。使用浏览器打开它，你能更清楚地了解整个过程。
+
+如果你想结合报告里的 UI 状态重新调试 Prompt，你可以启动本地 Playground Server，然后点击“Send to Playground”.
+
+启动本地 Playground Server:
+```
+npx --yes @midscene/web
+```
 
 ## 集成到 Playwright
 
@@ -212,13 +219,4 @@ npx ts-node demo.ts
 
 ### 第四步：查看运行报告
 
-当上面的命令执行成功后，会在控制台输出：`Midscene - report file updated: ./current_cwd/midscene_run/report/some_id.html`， 通过浏览器打开该文件即可看到报告。
-
-你也可以将 `./midscene_run/report/latest.web-dump.json` 文件导入 [可视化工具](/visualization/) 查看。
-
-## 查看示例报告
-
-在[可视化工具](/visualization/)中点击"加载演示"按钮，你将能够看到之前代码的运行结果以及其他一些示例。
-
-![](/view-demo-visualization.gif)
-
+当上面的命令执行成功后，会在控制台输出：`Midscene - report file updated: /path/to/report/some_id.html`， 通过浏览器打开该文件即可看到报告。
diff --git a/apps/site/docs/zh/docs/more/prompting-tips.md b/apps/site/docs/zh/docs/more/prompting-tips.md
@@ -29,9 +29,14 @@
 
 错误示例 ❌：[number, number]，主按钮的 [x, y] 坐标
 
-### 使用可视化工具调试
+### 使用可视化报告和 Playground 进行调试
 
-使用可视化工具调试和理解 Midscene 的每个步骤。只需上传日志，就可以查看 AI 的解析结果。你可以在本站导航栏上找到 [可视化工具](/visualization/)。
+测试报告里有每个步骤的详细信息。如果你想结合报告里的 UI 状态重新运行 Prompt，你可以启动本地 Playground Server，然后点击“Send to Playground”.
+
+启动本地 Playground Server:
+```
+npx --yes @midscene/web
+```
 
 ### 从界面做推断，而不是 DOM 属性或者浏览器状态
 

diff --git a/apps/site/docs/zh/docs/usage/API.md b/apps/site/docs/zh/docs/usage/API.md
@@ -40,7 +40,7 @@ await mid.ai('点击任务列表下方的 "completed" 状态按钮');
 
 在底层，Midscene 会将页面上下文和截图发送给 LLM，以详细规划步骤。随后，Midscene 会逐步执行这些步骤。如果 Midscene 认为无法执行，将抛出一个错误。
 
-你的任务会被拆解成下述内置方法，你可以在可视化工具中看到它们：
+你的任务会被拆解成下述内置方法，你可以在可视化报告中看到它们：
 
 1. **定位（Locator）**：使用自然语言描述找到目标元素
 2. **操作（Action）**：点击、滚动、键盘输入、悬停（hover）

diff --git a/apps/site/docs/zh/index.md b/apps/site/docs/zh/index.md
@@ -29,8 +29,8 @@ features:
   - title: 立即集成
     details: 你可以使用 GPT-4o、Playwright 和 Puppeteer 等熟悉的工具，立即开始集成
     icon: 🪓
-  - title: 可视化
-    details: 通过我们的可视化工具，你可以轻松理解和调试整个过程
+  - title: 可视化报告
+    details: 通过我们的可视化报告，你可以轻松理解和调试整个过程
     icon: 🎞️
   - title: 全新体验
     details: 体验全新的自动化开发模式，尽情享受吧！

diff --git a/apps/site/package.json b/apps/site/package.json
@@ -16,7 +16,7 @@
     "react": "^18.2.0",
     "react-dom": "^18.2.0",
     "querystring": "0.2.1",
-    "rspress": "1.26.2",
+    "rspress": "1.34.1",
     "@midscene/visualizer": "workspace:*"
   },
   "devDependencies": {

diff --git a/apps/site/rspress.config.ts b/apps/site/rspress.config.ts
@@ -9,7 +9,7 @@ export default defineConfig({
   icon: '/midscene-icon.png',
   logo: {
     light: '/midscene_with_text_light.png',
-    dark: '/midscene_with_text_dark.png',
+    dark: '/midscene_with_text_light.png',
   },
   themeConfig: {
     darkMode: true,

diff --git a/apps/site/styles/index.css b/apps/site/styles/index.css
@@ -1,68 +1,51 @@
-
-.dark {
-  --rp-c-text-1: rgba(255, 255, 255, 1);
-}
-
-:root {
-  --rp-c-text-1: #000;
-  --rp-code-block-bg: #efefef;
-  --rp-c-text-code: #2d4956;
-  --rp-home-feature-bg: #F1F1f1;
-}
-
 /* Footer styles */
-.footer {
-  background-color: transparent;
-  color: #333;
-  text-align: center;
 
+.footer {
+    background-color: transparent;
+    text-align: center;
 }
 
 .footer::before,
 .footer::after {
-  content: none;
-  border: none;
+    content: none;
+    border: none;
 }
 
 .footer-content {
-  display: flex;
-  justify-content: center;
-  align-items: center;
-  flex-wrap: wrap;
-  gap: 15px;
-  border: none;
+    display: flex;
+    justify-content: center;
+    align-items: center;
+    flex-wrap: wrap;
+    gap: 15px;
+    border: none;
 }
 
 .footer-logo {
-  width: 40px;
-  border: none;
+    width: 40px;
+    border: none;
 }
 
 .footer-text {
-  margin: 0;
-  font-size: 14px;
-  color: #333;
-  border: none;
-  white-space: nowrap; 
+    margin: 0;
+    font-size: 14px;
+    border: none;
+    white-space: nowrap;
 }
 
 @media (max-width: 768px) {
-  .md\:text-6xl {
-    font-size: 3.3rem;
-    line-height: 1;
-  }
-
-  .footer-content {
-    flex-direction: column;
-    align-items: center;
-    text-align: center;
-  }
-
-  .footer-logo {
-    margin-bottom: 10px;
-  }
-
-  .footer-bottom {
-    text-align: center;
-  }
-}
+    .md\:text-6xl {
+        font-size: 3.3rem;
+        line-height: 1;
+    }
+    .footer-content {
+        flex-direction: column;
+        align-items: center;
+        text-align: center;
+    }
+    .footer-logo {
+        margin-bottom: 10px;
+    }
+    .footer-bottom {
+        text-align: center;
+    }
+}