Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question]: Why after running Spark task, there is no change of the database? (both on postgresql and mysql) #216

Open
huanghuihui0904 opened this issue Apr 8, 2024 · 7 comments
Labels
question Further information is requested

Comments

@huanghuihui0904
Copy link

Contact Details

[email protected]

What would you like to ask or discuss?

After I ran the Spark task by the instructions you provide, there is no change of the database. Can you tell me how to solve that? Also I wonder how the program write the data to the database, in which way or in which java files?
As I have checked the error logs of my program, it just has the following two errors which I think not the key point.
In /opt/compass-v1.1.2/task-canal/logs
image
In /opt/compass-v1.1.2/task-portal/logs
image
Look forward to your reply!

@huanghuihui0904 huanghuihui0904 added the question Further information is requested label Apr 8, 2024
@huanghuihui0904
Copy link
Author

huanghuihui0904 commented Apr 9, 2024

I want to add that now I fill all the table in mysql database(except blocklist), but the webpage still has no data.
I add the data by the following sqls:

INSERT INTO task_application VALUES (1, 'application_1712475671665_0002', 'Spark Pi', 'Spark Pi', 'Spark Pi', 'Spark', '2024-04-07 16:22:47', 1, 'http://192.168.106.192:18018/api/v1/applications/application_1712475671665_0001/1/logs', '2024-04-07 16:22:47', '2024-04-07 16:23:01');
INSERT INTO project VALUES (1, 'Spark Pi', 'description', 1, 1, '2024-04-07 16:22:47','2024-04-07 16:23:01');
INSERT INTO task VALUES (1,'Spark Pi',1,'Spark Pi',1,'Spark Pi','description',1,'SPARK',2,'2024-04-07 16:22:47', '2024-04-07 16:23:01');
INSERT INTO flow VALUES (1,'Spark Pi','description',1,1,1,'Spark Pi','2024-04-07 16:22:47', '2024-04-07 16:23:01');
INSERT INTO template VALUES(1,1,'cluster','advice','log','2024-04-07 16:23:01','2024-04-07 16:22:47')
INSERT INTO task_instance VALUES(1,'Spark Pi','Spark Pi','Spark Pi','2024-04-07 16:23:01','2024-04-07 16:23:01','2024-04-07 16:23:01.000000','FINISHED','SPARK',4,5,'worker group','schedule','2024-04-07 16:23:01','2024-04-07 16:23:01')
INSERT INTO task_datum VALUES(1,'Spark Pi','Spark Pi','Spark Pi','2024-04-07 16:23:01','baseline','2024-04-07 16:22:47')
INSERT INTO flink_task VALUES(1,'Spark Pi',1,'Spark Pi',1,'Spark Pi',1,0,'2024-04-07 16:22:47','2024-04-07 16:23:01')
INSERT INTO flink_task_app VALUES(1,'compass',1,'Spark Pi',1,'Spark Pi',1,'Spark Pi',1,'finish',1,'2024-04-07 16:23:01',1,'flink url',20,30,40,'Hadoop',100,'2024-04-07 16:22:47','2024-04-07 16:24:01',60,40,'queue','cluster',9,'compass','diagnosis',2,3,4,6,7,'Spark Pi','2024-04-07 16:22:47','2024-04-07 16:23:01')

Addtionally, I try the url http://192.****:7075/compass/api/v1/report/projects and the right response, which means there is no error in the mysql connection. But the table of database dolphinscheduler is empty, does this lead to the empty webpage?

@nilnon
Copy link
Collaborator

nilnon commented Apr 9, 2024

Contact Details

[email protected]

What would you like to ask or discuss?

After I ran the Spark task by the instructions you provide, there is no change of the database. Can you tell me how to solve that? Also I wonder how the program write the data to the database, in which way or in which java files? As I have checked the error logs of my program, it just has the following two errors which I think not the key point. In /opt/compass-v1.1.2/task-canal/logs image In /opt/compass-v1.1.2/task-portal/logs image Look forward to your reply!

@huanghuihui0904 Check whether the canal dependency package has been downloaded?

@nilnon
Copy link
Collaborator

nilnon commented Apr 9, 2024

I want to add that now I fill all the table in mysql database(except blocklist), but the webpage still has no data. I add the data by the following sqls:

INSERT INTO task_application VALUES (1, 'application_1712475671665_0002', 'Spark Pi', 'Spark Pi', 'Spark Pi', 'Spark', '2024-04-07 16:22:47', 1, 'http://192.168.106.192:18018/api/v1/applications/application_1712475671665_0001/1/logs', '2024-04-07 16:22:47', '2024-04-07 16:23:01');
INSERT INTO project VALUES (1, 'Spark Pi', 'description', 1, 1, '2024-04-07 16:22:47','2024-04-07 16:23:01');
INSERT INTO task VALUES (1,'Spark Pi',1,'Spark Pi',1,'Spark Pi','description',1,'SPARK',2,'2024-04-07 16:22:47', '2024-04-07 16:23:01');
INSERT INTO flow VALUES (1,'Spark Pi','description',1,1,1,'Spark Pi','2024-04-07 16:22:47', '2024-04-07 16:23:01');
INSERT INTO template VALUES(1,1,'cluster','advice','log','2024-04-07 16:23:01','2024-04-07 16:22:47')
INSERT INTO task_instance VALUES(1,'Spark Pi','Spark Pi','Spark Pi','2024-04-07 16:23:01','2024-04-07 16:23:01','2024-04-07 16:23:01.000000','FINISHED','SPARK',4,5,'worker group','schedule','2024-04-07 16:23:01','2024-04-07 16:23:01')
INSERT INTO task_datum VALUES(1,'Spark Pi','Spark Pi','Spark Pi','2024-04-07 16:23:01','baseline','2024-04-07 16:22:47')
INSERT INTO flink_task VALUES(1,'Spark Pi',1,'Spark Pi',1,'Spark Pi',1,0,'2024-04-07 16:22:47','2024-04-07 16:23:01')
INSERT INTO flink_task_app VALUES(1,'compass',1,'Spark Pi',1,'Spark Pi',1,'Spark Pi',1,'finish',1,'2024-04-07 16:23:01',1,'flink url',20,30,40,'Hadoop',100,'2024-04-07 16:22:47','2024-04-07 16:24:01',60,40,'queue','cluster',9,'compass','diagnosis',2,3,4,6,7,'Spark Pi','2024-04-07 16:22:47','2024-04-07 16:23:01')

Addtionally, I try the url http://192.****:7075/compass/api/v1/report/projects and the right response, which means there is no error in the mysql connection. But the table of database dolphinscheduler is empty, does this lead to the empty webpage?

@huanghuihui0904 Part of the data of webpage comes from mysql, and part of it comes from elasticsearch/opensearch.

@nilnon
Copy link
Collaborator

nilnon commented Apr 9, 2024

@huanghuihui0904 You need to run all modules successfully and run spark tasks in dolphinscheduler so that task-portal can display data.

@huanghuihui0904
Copy link
Author

But my opensearch can work well as I can open the web and see the data.
So do you mean the error with canal can lead to the problem: (1) after running spark task, no data written to mysql. (2)the webpage(actually the request http://192.*****:7075/compass/api/v1/report/statistics?projectName=) cannot get data.
Add some info to the request http://192.*****:7075/compass/api/v1/report/statistics?projectName=:
The responce as the following.

{
"code": 200,
"msg": "Succeed",
"data": null
}
And I find that this is the main request to get the data and then let the webpage show.

@huanghuihui0904
Copy link
Author

@huanghuihui0904 You need to run all modules successfully and run spark tasks in dolphinscheduler so that task-portal can display data.

Do you mean the dolphinscheduler database? As I just fill the data in the compass database.

@nilnon
Copy link
Collaborator

nilnon commented Apr 9, 2024

@huanghuihui0904 You need to run all modules successfully and run spark tasks in dolphinscheduler so that task-portal can display data.

Do you mean the dolphinscheduler database? As I just fill the data in the compass database.

But my opensearch can work well as I can open the web and see the data. So do you mean the error with canal can lead to the problem: (1) after running spark task, no data written to mysql. (2)the webpage(actually the request http://192.*****:7075/compass/api/v1/report/statistics?projectName=) cannot get data. Add some info to the request http://192.*****:7075/compass/api/v1/report/statistics?projectName=: The responce as the following.

{
"code": 200,
"msg": "Succeed",
"data": null
}
And I find that this is the main request to get the data and then let the webpage show.

the /api/v1/report/statistics data mainly come from opensearch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants