Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add system test and docs for DataprocSubmitJobOperator (Trino job) #32796

Merged

Conversation

moiseenkov
Copy link
Contributor

Add system test and docs for DataprocSubmitJobOperator (Trino job)

Related issues: #32407, #22430, #22447

@moiseenkov moiseenkov force-pushed the system_tests/dataproc_submit_job_trino branch 4 times, most recently from bd777b1 to 2be9684 Compare July 25, 2023 07:04
@moiseenkov
Copy link
Contributor Author

@potiuk , hi!
Can we merge it please?

@moiseenkov moiseenkov force-pushed the system_tests/dataproc_submit_job_trino branch 3 times, most recently from 0ecbe14 to 2e58443 Compare July 25, 2023 14:47
Copy link
Member

@pankajastro pankajastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just wondering if makes sense to combine presto #32798 and trino example?

@moiseenkov
Copy link
Contributor Author

LGTM, just wondering if makes sense to combine presto #32798 and trino example?

Although these two examples are very similar, they are not identical:

  • job configurations have different fields: presto_job vs trino_job,
  • optional components Trino and Presto have incompatible requirements for the image version (trino >=2.1, presto >=1.3,<2.1), which forces us to create separate clusters for each type of job.

That's why I think these two examples look simpler and more straightforward when they are divided. If we combine them together, then we will just increase the complexity of the system test (create two clusters, run two jobs, destroy two clusters) and the documentation.

@moiseenkov moiseenkov force-pushed the system_tests/dataproc_submit_job_trino branch from 2e58443 to 616f4e6 Compare July 26, 2023 06:57
@pankajastro pankajastro merged commit 43f7da7 into apache:main Jul 26, 2023
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants