Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Remove gcloud CLI dependency for Dataproc platform #1223

Merged
merged 3 commits into from
Jul 25, 2024
Merged

[FEA] Remove gcloud CLI dependency for Dataproc platform #1223

merged 3 commits into from
Jul 25, 2024

Conversation

cindyyuanjiang
Copy link
Collaborator

Contributes to #1191

Changes

  • Dataproc Qual tool will use generated instance catalog file to pull instance type info.
  • In Initialization phase, Dataproc Qual tool used to call gcloud CLI cmds to set configuration properties, this PR updates the tool to use the local config file.

Testing

  • With cluster properties file
spark_rapids qualification --eventlogs <my-event-logs> --platform dataproc --cluster <my-cluster-props-file>
  • With cluster name
spark_rapids qualification --eventlogs <my-event-logs> --platform dataproc --cluster <my-cluster-name>

@cindyyuanjiang
Copy link
Collaborator Author

We plan to add support for Dataproc-GKE in a separate PR, since we encountered an issue when testing it. The issue needs to be resolved first.

@cindyyuanjiang cindyyuanjiang self-assigned this Jul 23, 2024
@cindyyuanjiang cindyyuanjiang mentioned this pull request Jul 23, 2024
Closed
@cindyyuanjiang cindyyuanjiang added feature request New feature or request user_tools Scope the wrapper module running CSP, QualX, and reports (python) usability track issues related to the Tools's user experience labels Jul 23, 2024
@cindyyuanjiang
fixed tox failure
33b7bc7 
Signed-off-by: cindyyuanjiang <[email protected]>
parthosa
parthosa reviewed Jul 24, 2024
View reviewed changes
Copy link
Collaborator

@parthosa parthosa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cindyyuanjiang. Tested the functionality on a clean ubuntu machine. Made some minor refactoring comments.

user_tools/src/spark_rapids_pytools/cloud_api/dataproc.py Outdated Show resolved Hide resolved
user_tools/src/spark_rapids_pytools/cloud_api/dataproc.py Outdated Show resolved Hide resolved
user_tools/src/spark_rapids_pytools/cloud_api/dataproc.py Show resolved Hide resolved
user_tools/src/spark_rapids_pytools/cloud_api/dataproc.py Show resolved Hide resolved
@cindyyuanjiang
improve comment
d5ff89f 
Signed-off-by: cindyyuanjiang <[email protected]>
parthosa
parthosa approved these changes Jul 24, 2024
View reviewed changes
Copy link
Collaborator

@parthosa parthosa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cindyyuanjiang for this feature.

user_tools/src/spark_rapids_pytools/cloud_api/dataproc.py Outdated Show resolved Hide resolved
@amahussein amahussein merged commit 7ea8a10 into NVIDIA:dev Jul 25, 2024
14 checks passed
@cindyyuanjiang cindyyuanjiang deleted the spark-rapids-tools-1191-dataproc branch July 26, 2024 00:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@parthosa parthosa parthosa approved these changes

@amahussein amahussein Awaiting requested review from amahussein

@nartal1 nartal1 Awaiting requested review from nartal1

Assignees

@cindyyuanjiang cindyyuanjiang

Labels
feature request New feature or request usability track issues related to the Tools's user experience user_tools Scope the wrapper module running CSP, QualX, and reports (python)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@cindyyuanjiang @parthosa @amahussein