[Feature] Import/Export Orchestration Transaction Optimization

[wxyn]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Problem Description

When users publish a workflow with a large number of nodes, the publishing time will be too long, which leads to the long transaction timeout of the database and the workflow publishing failure.

Description

Currently, the importOrchestrator and exportOrchestrator methods have @transactional annotations. Because of the time-consuming rpc operations in both methods, a large number of workflow nodes can cause database connections to time out. Therefore, the execution logic in these two methods needs to be optimized to remove transaction annotations

Use case

No response

solutions

Previously, large transactions were added when the importOrchestrator and exportOrchestrator methods were executed on the OrchestratorServer service to ensure data consistency on the orchestrator database.
Take the import phase as an example, the main process sequence is as follows:

1. Parse the DSSOrchestratorInfo from the workflow zip package, determine whether the orchestration has been imported before according to the uuid, if not, call insertOrchestrator to insert new orchestration information; Otherwise, call updateOrchestrator to update orchestrator information.
2. Create an orchestration version object DSSOrchestratorVersion, and the version number is the latest version of the original orchestration plus 1.
3. Generate a new context ContextId based on the changed ranking and new version, and save it to the DSSOrchestratorVersion object.
4. Initiate a RequestImportWorkflow import request to the workflow service through rpc.
5. The workflow service sends an importRef import request to the third-party appconn, and the third-party service executes the internal node import operation.
6. After the importRef operation is complete, the orchestrator service receives the new appId returned by the workflow service, sets it to DSSOrchestratorVersion, and calls addOrchestratorVersion to insert it into the database.

You can see that the transaction is added to the importOrchestrator to ensure the consistency of dss_orchestrator_version in the dss_orchestrator_info table, and the time-consuming operation of importing to the three-party service invocation nodes is included in the middle of the two database operations. The transaction timed out.
Optimization: According to the logic in the method, the update and add operations on the database tables dss_orchestrator_info and dss_orchestrator_version are centralized and executed together after the workflow service returns, so that transaction annotations can be removed to ensure data consistency.

Comparison of the old and new versions:
Pre-change code: The orchestrator_info table operates before the workflow service is invoked, while the orchestrator_version table operates after workflow returns. This results in transaction timeouts when rpc calls take time (for example, copying operations within a three-party service takes a long time).

Modified code: The dss_orchestrator_info table and the dss_orchestrator_version table are operated in a unified manner after rpc calls the worklfow service, so that even if the entire method is not transacted, because orchestrator's database operations are put together, It will not lead to data inconsistency problems.

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[zqburde]

Add it in DSS1.1.2