Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Engine UI] partial pages of the engine #7602

Merged
merged 45 commits into from
Oct 25, 2024
Merged
Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
44a767b
[Feature][Dashboard] Init
QiaoJ-Chen Jul 7, 2024
42224fd
[feature][engine-ui] add a folder
QiaoJ-Chen Aug 4, 2024
789263d
[Feature][Engine UI] init UI project, add overview page, add job page…
QiaoJ-Chen Sep 7, 2024
659b73e
[Fix][Engine UI] delete dist files
QiaoJ-Chen Sep 16, 2024
55f071b
[Feature][Engine UI] git ignore dist files
QiaoJ-Chen Sep 16, 2024
a86aca8
[Feature][Engine UI] upgrade framework;upgrade vite to 5;add vitest d…
QiaoJ-Chen Sep 22, 2024
8683ea8
[Feature][Engine UI] add master/workers list page; add overview compo…
QiaoJ-Chen Sep 22, 2024
bdd745e
[Feature][Engine UI] add license header
QiaoJ-Chen Sep 22, 2024
93bd2ce
[Feature][Engine UI] add managers view component test unit
QiaoJ-Chen Sep 22, 2024
866bed2
[Fix][Engine UI] update the doc. How to use it
QiaoJ-Chen Sep 22, 2024
5eb19af
[Fix][Engine UI] update the doc. How to use it
QiaoJ-Chen Sep 22, 2024
db6b1e6
[Feature][Engine UI] change vue-router mode from history to hash
QiaoJ-Chen Sep 22, 2024
32e8279
[Feature][Engine UI] detail repair
QiaoJ-Chen Sep 22, 2024
90aa260
[Feature][Engine UI] storage
QiaoJ-Chen Oct 16, 2024
6222655
[Feature][Engine UI] storage
QiaoJ-Chen Oct 16, 2024
5dbee9e
[Feature][Engine UI] storage
QiaoJ-Chen Oct 16, 2024
e1177be
[Fix][Engine UI] update the doc. How to use it
QiaoJ-Chen Oct 16, 2024
b97c8f9
1
QiaoJ-Chen Oct 17, 2024
8182963
[Feature][Engine UI] ui
QiaoJ-Chen Oct 21, 2024
473c099
[Feature][Engine UI] ui
QiaoJ-Chen Oct 21, 2024
cffac84
[Feature][Engine UI] ui.yml
QiaoJ-Chen Oct 21, 2024
87c6808
feat: ui.yml
QiaoJ-Chen Oct 22, 2024
f6eac0e
feat: ui.yml
QiaoJ-Chen Oct 22, 2024
77dcf69
feat: ui.yml
QiaoJ-Chen Oct 22, 2024
9157378
Merge remote-tracking branch 'upstream/dev' into dev
QiaoJ-Chen Oct 22, 2024
f37f7e0
feat: ui.yml
QiaoJ-Chen Oct 22, 2024
d749746
feat: package-lock
QiaoJ-Chen Oct 22, 2024
e2d28a7
feat: ui.yml
QiaoJ-Chen Oct 22, 2024
c2ec996
feat: job log tab
QiaoJ-Chen Oct 22, 2024
28a2010
feat: ui.yml
QiaoJ-Chen Oct 22, 2024
9132cca
update test ci
Hisoka-X Oct 23, 2024
77109cf
update test ci
Hisoka-X Oct 23, 2024
d2f2d80
update test ci
Hisoka-X Oct 23, 2024
8c9e57a
feat: doc
QiaoJ-Chen Oct 23, 2024
028949f
feat: doc
QiaoJ-Chen Oct 23, 2024
8793082
Merge branch 'dev' of https://github.com/QiaoJ-Chen/seatunnel into dev
QiaoJ-Chen Oct 23, 2024
6f6a1f0
feat: countdown
QiaoJ-Chen Oct 23, 2024
0190cea
feat: countdown
QiaoJ-Chen Oct 23, 2024
921e83f
feat: md image
QiaoJ-Chen Oct 24, 2024
4420eda
feat: md image
QiaoJ-Chen Oct 24, 2024
6a0ad47
feat: doc sidebar
QiaoJ-Chen Oct 24, 2024
edaaf44
feat: doc sidebar
QiaoJ-Chen Oct 24, 2024
b5d16e9
feat: fix ip
QiaoJ-Chen Oct 24, 2024
d8d4501
[Feature][Engine UI] fix the bug of doc
QiaoJ-Chen Oct 25, 2024
cd936ee
[Feature][Engine UI] fix the bug of doc
QiaoJ-Chen Oct 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .github/workflows/backend.yml
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,37 @@ jobs:
npm set strict-ssl false
npm install
npm run build

seatunnel-ui:
if: needs.changes.outputs.api == 'true'
needs: [ changes, sanity-check ]
name: Build SeaTunnel UI
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout PR
uses: actions/checkout@v3

- uses: actions/setup-node@v2
with:
node-version: 20.x

- name: Install Dependencies and Check Code Style
run: |
cd seatunnel-engine/seatunnel-engine-ui/
npm install
npm run lint

- name: Run unit tests
run: |
cd seatunnel-engine/seatunnel-engine-ui/
npm run test:unit

- name: Build SeaTunnel UI
run: |
cd seatunnel-engine/seatunnel-engine-ui/
npm run build

unit-test:
needs: [ changes, sanity-check ]
if: needs.changes.outputs.api == 'true' || (needs.changes.outputs.api == 'false' && needs.changes.outputs.ut-modules != '')
Expand Down
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,8 @@ seatunnel-examples

/lib/*
version.properties
node/

dist/

ui/
33 changes: 33 additions & 0 deletions docs/en/seatunnel-engine/web-ui.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Apache SeaTunnel Web UI Documentation
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update

"seatunnel-engine/telemetry"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, it has been updated just now


## Overview

The Web UI of Apache SeaTunnel offers a user-friendly interface for monitoring and managing SeaTunnel jobs. Through the Web UI, users can view real-time information on currently running jobs, finished jobs, and the status of worker and master nodes within the cluster. The main functional modules include Jobs, Workers, and Master, each providing detailed status information and operational options to help users efficiently manage and optimize their data processing workflows.
![workflow.png](../../images/ui/overview.png)

## Jobs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick question: How can I access it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, images was ignored, I've just resolved it


### Running Jobs

The "Running Jobs" section lists all SeaTunnel jobs that are currently in execution. Users can view basic information for each job, including Job ID, submission time, status, execution time, and more. By clicking on a specific job, users can access detailed information such as task distribution, resource utilization, and log outputs, allowing for real-time monitoring of job progress and timely handling of potential issues.
![workflow.png](../../images/ui/running.png)
![workflow.png](../../images/ui/detail.png)

### Finished Jobs

The "Finished Jobs" section displays all SeaTunnel jobs that have either successfully completed or failed. This section provides execution results, completion times, durations, and failure reasons (if any) for each job. Users can review past job records through this module to analyze job performance, troubleshoot issues, or rerun specific jobs as needed.
![workflow.png](../../images/ui/finished.png)

## Workers

### Workers Information

The "Workers" section displays detailed information about all worker nodes in the cluster, including each worker's address, running status, CPU and memory usage, number of tasks being executed, and more. Through this module, users can monitor the health of each worker node, promptly identify and address resource bottlenecks or node failures, ensuring the stable operation of the SeaTunnel cluster.
![workflow.png](../../images/ui/workers.png)

## Master

### Master Information

The "Master" section provides the status and configuration information of the master node in the SeaTunnel cluster. Users can view the master's address, running status, job scheduling responsibilities, and overall resource allocation within the cluster. This module helps users gain a comprehensive understanding of the cluster's core management components, facilitating cluster configuration optimization and troubleshooting.
![workflow.png](../../images/ui/master.png)
33 changes: 33 additions & 0 deletions docs/zh/seatunnel-engine/web-ui.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Apache SeaTunnel Web UI 文档

## 概述

Apache SeaTunnel 的 Web UI 提供了一个友好的用户界面,用于监控和管理 SeaTunnel 作业。通过 Web UI,用户可以实时查看当前运行的作业、已完成的作业,以及集群中工作节点和管理节点的状态。主要功能模块包括 Jobs、Workers 和 Master,每个模块都提供了详细的状态信息和操作选项,帮助用户高效地管理和优化其数据处理流程。
![workflow.png](../../images/ui/overview.png)

## 作业

### 运行中的作业

“运行中的作业”模块列出了当前正在执行的所有 SeaTunnel 作业。用户可以查看每个作业的基本信息,包括作业 ID、提交时间、状态、执行时间等。点击具体作业可以查看更多详细信息,如任务分布、资源使用情况和日志输出,便于用户实时监控作业进度并及时处理潜在问题。
![workflow.png](../../images/ui/running.png)
![workflow.png](../../images/ui/detail.png)

### 已完成的作业

“已完成的作业”模块展示了所有已成功完成或失败的 SeaTunnel 作业。此部分提供了每个作业的执行结果、完成时间、耗时以及失败原因(如果有)。用户可以通过此模块回顾过去的作业记录,分析作业性能,进行故障排查或重复执行某些特定作业。
![workflow.png](../../images/ui/finished.png)

## 工作节点

### 工作节点信息

“工作节点”模块展示了集群中所有工作节点的详细信息,包括每个工作节点的地址、运行状态、CPU 和内存使用情况、正在执行的任务数量等。通过该模块,用户可以监控各个工作节点的健康状况,及时发现和处理资源瓶颈或节点故障,确保 SeaTunnel 集群的稳定运行。
![workflow.png](../../images/ui/workers.png)

## 管理节点

### 管理节点信息

“管理节点”模块提供了 SeaTunnel 集群中主节点的状态和配置信息。用户可以查看 Master 节点的地址、运行状态、负责的作业调度情况以及整体集群的资源分配情况。该模块帮助用户全面了解集群的核心管理部分,便于进行集群配置优化和故障排查。
![workflow.png](../../images/ui/master.png)
1 change: 1 addition & 0 deletions seatunnel-engine/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
<module>seatunnel-engine-core</module>
<module>seatunnel-engine-storage</module>
<module>seatunnel-engine-serializer</module>
<module>seatunnel-engine-ui</module>
</modules>

<properties>
Expand Down
5 changes: 5 additions & 0 deletions seatunnel-engine/seatunnel-engine-server/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,11 @@
<artifactId>seatunnel-engine-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-engine-ui</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>checkpoint-storage-hdfs</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@

import javax.servlet.DispatcherType;

import java.net.URL;
import java.util.EnumSet;

import static org.apache.seatunnel.engine.server.rest.RestConstant.ENCRYPT_CONFIG;
Expand Down Expand Up @@ -88,7 +89,15 @@ public void createJettyServer() {
FilterHolder filterHolder = new FilterHolder(new ExceptionHandlingFilter());
context.addFilter(filterHolder, "/*", EnumSet.of(DispatcherType.REQUEST));

context.addServlet(new ServletHolder("default", new DefaultServlet()), "/");
ServletHolder defaultServlet = new ServletHolder("default", DefaultServlet.class);
URL uiResource = JettyService.class.getClassLoader().getResource("ui");
if (uiResource != null) {
defaultServlet.setInitParameter("resourceBase", uiResource.toExternalForm());
} else {
log.warn("UI resources not found in classpath");
}

context.addServlet(defaultServlet, "/");

ServletHolder overviewHolder = new ServletHolder(new OverviewServlet(nodeEngine));
ServletHolder runningJobsHolder = new ServletHolder(new RunningJobsServlet(nodeEngine));
Expand Down
22 changes: 22 additions & 0 deletions seatunnel-engine/seatunnel-engine-ui/.env.development
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

NODE_ENV=development

# if you want to use the online server, you need to set this to `http://124.221.211.72:8081`
# VITE_APP_API_SERVICE='http://localhost:5801'
VITE_APP_API_SERVICE='http://124.221.211.72:8081'
# the context path of api service
VITE_APP_API_BASE='/api'
19 changes: 19 additions & 0 deletions seatunnel-engine/seatunnel-engine-ui/.env.production
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

NODE_ENV=production

# the context path of api service
VITE_APP_API_BASE=''
46 changes: 46 additions & 0 deletions seatunnel-engine/seatunnel-engine-ui/.eslintrc.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

/* eslint-env node */
require('@rushstack/eslint-patch/modern-module-resolution')

module.exports = {
root: true,
'extends': [
'plugin:vue/vue3-essential',
'eslint:recommended',
'@vue/eslint-config-typescript',
'@vue/eslint-config-prettier/skip-formatting'
],
overrides: [
{
files: [
'cypress/e2e/**/*.{cy,spec}.{js,ts,jsx,tsx}',
'cypress/support/**/*.{js,ts,jsx,tsx}'
],
'extends': [
'plugin:cypress/recommended'
]
}
],
parserOptions: {
ecmaVersion: 'latest'
},
rules: {
"vue/multi-word-component-names": "off"
}
}
30 changes: 30 additions & 0 deletions seatunnel-engine/seatunnel-engine-ui/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*

node_modules
.DS_Store
dist
dist-ssr
coverage
*.local

/cypress/videos/
/cypress/screenshots/

# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

*.tsbuildinfo
8 changes: 8 additions & 0 deletions seatunnel-engine/seatunnel-engine-ui/.prettierrc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"$schema": "https://json.schemastore.org/prettierrc",
"semi": false,
"tabWidth": 2,
"singleQuote": true,
"printWidth": 100,
"trailingComma": "none"
}
60 changes: 60 additions & 0 deletions seatunnel-engine/seatunnel-engine-ui/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# seatunnel-engine-ui

## Development Environment Dependencies

- Node 18+/20+ required
- npm 7+

- modify `VITE_APP_API_SERVICE` and `VITE_APP_API_BASE` in `.env.development`
- quick start

```sh
npm install
npm run dev
```

## Project Setup

```sh
npm install
```

### Compile and Hot-Reload for Development

```sh
npm run dev
```

### Type-Check, Compile and Minify for Production

```sh
npm run build
```

### Run Unit Tests with [Vitest]

```sh
npm run test:unit
```

### Run End-to-End Tests with [Cypress]

```sh
npm run test:e2e:dev
```

This runs the end-to-end tests against the Vite development server.
It is much faster than the production build.

But it's still recommended to test the production build with `test:e2e` before deploying (e.g. in CI environments):

```sh
npm run build
npm run test:e2e
```

### Lint with [ESLint]

```sh
npm run lint
```
25 changes: 25 additions & 0 deletions seatunnel-engine/seatunnel-engine-ui/cypress.config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import { defineConfig } from 'cypress'

export default defineConfig({
e2e: {
specPattern: 'cypress/e2e/**/*.{cy,spec}.{js,jsx,ts,tsx}',
baseUrl: 'http://localhost:4173'
}
})
Loading
Loading