Unified Dockerfile Image for Spark Jobs #174
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tushartathgur
force-pushed
the
unified_dockerfile
branch
from
2a222be
to
8cda64f
Compare
tushartathgur
requested review from
yanjunz97,
dreamtalen,
elton-furtado and
salv-orlando
|
Can you update the documentation to reflect the changes? |
tushartathgur
force-pushed
the
unified_dockerfile
branch
3 times, most recently
from
470ad43
to
bd70023
Compare
dreamtalen
reviewed
Mar 6, 2023
| COPY plugins/policy-recommendation/policy_recommendation_job.py /opt/spark/work-dir/policy_recommendation_job.py | ||
| COPY plugins/policy-recommendation/policy_recommendation_utils.py /opt/spark/work-dir/policy_recommendation_utils.py | ||
| COPY plugins/policy-recommendation/antrea_crd.py /opt/spark/work-dir/antrea_crd.py | ||
| COPY plugins/anomaly-detection/AnomalyDetection.py /opt/spark/work-dir/AnomalyDetection.py | ||
| COPY plugins/anomaly-detection/requirements.txt /opt/spark/work-dir/requirements.txt |
There was a problem hiding this comment.
nit: let's have a dedicated filename for anomaly-detection/requirements.txt, since there might be other requirements.txt for future spark jobs.
| RUN pip3 install --upgrade pip && \ | ||
| pip3 install pyyaml && \ | ||
| pip3 install kubernetes && \ | ||
| pip3 install -r /opt/spark/work-dir/requirements.txt |
| @@ -1,7 +1,7 @@ | |||
| FROM gcr.io/spark-operator/spark-py:v3.1.1 | |||
|
|
|||
| LABEL maintainer="Antrea <[email protected]>" | |||
| LABEL description="A docker image to deploy policy recommendation Spark job." | |||
| LABEL description="A docker image to deploy policy recommendation and throughput anomaly detector Spark job." | |||
yanjunz97
reviewed
Mar 6, 2023
.github/workflows/build.yml
Outdated
|
|
||
| check-policy-recommendation-changes: | ||
| name: Check whether policy-recommendation image needs to be built based on diff | ||
|
|
There was a problem hiding this comment.
Suggested change
Remove the extra spaces?
| RUN pip3 install --upgrade pip && \ | ||
| pip3 install pyyaml && \ | ||
| pip3 install kubernetes && \ | ||
| pip3 install -r /opt/spark/work-dir/requirements.txt |
There was a problem hiding this comment.
Suggested change
| pip3 install -r /opt/spark/work-dir/requirements.txt | |
| pip3 install -r /opt/spark/work-dir/requirements.txt |
Extra space
tushartathgur
force-pushed
the
unified_dockerfile
branch
from
23d2aaa
to
165cf8b
Compare
|
/theia-test-e2e |
3 similar comments
|
/theia-test-e2e |
|
/theia-test-e2e |
|
/theia-test-e2e |
yanjunz97
approved these changes
Mar 7, 2023
Currently Theia supports separate dockerfiles for anomalydetector
and policyrecommendation, this PR helps unify these dockerfiles
resulting into lesser storage space.
Unified Dockerimage now uses 1.4 GB of image space than earlier
2.4GB combined
Spark Jobs constants have been moved under utils.go in order to
avoid duplicacy of constants
Solve #161
Signed-off-by: Tushar Tathgur <[email protected]>
tushartathgur
force-pushed
the
unified_dockerfile
branch
from
165cf8b
to
1d48515
Compare
|
/theia-test-e2e |
Codecov Report
@@ Coverage Diff @@
## main #174 +/- ##
==========================================
- Coverage 52.97% 52.80% -0.17%
==========================================
Files 74 78 +4
Lines 5720 6692 +972
Branches 41 0 -41
==========================================
+ Hits 3030 3534 +504
- Misses 2379 2852 +473
+ Partials 311 306 -5
Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
dreamtalen
reviewed
Mar 7, 2023
| COPY plugins/policy-recommendation/policy_recommendation_job.py /opt/spark/work-dir/policy_recommendation_job.py | ||
| COPY plugins/policy-recommendation/policy_recommendation_utils.py /opt/spark/work-dir/policy_recommendation_utils.py | ||
| COPY plugins/policy-recommendation/antrea_crd.py /opt/spark/work-dir/antrea_crd.py | ||
| COPY plugins/anomaly-detection/AnomalyDetection.py /opt/spark/work-dir/AnomalyDetection.py | ||
| COPY plugins/anomaly-detection/requirements.txt /opt/spark/work-dir/anomaly_detector_requirements.txt |
There was a problem hiding this comment.
Suggestion: follow our naming convention of python files, AnomalyDetection.py -> anomaly_detection.py, anomaly_detector_requirements.txt -> anomaly_detection_requirements.txt.
tushartathgur
force-pushed
the
unified_dockerfile
branch
from
1d48515
to
b55d2eb
Compare
|
/theia-test-e2e |
Signed-off-by: Tushar Tathgur <[email protected]>
tushartathgur
force-pushed
the
unified_dockerfile
branch
from
b55d2eb
to
557cd9a
Compare
|
/theia-test-e2e |
|
/theia-test-e2e |
2 similar comments
|
/theia-test-e2e |
|
/theia-test-e2e |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently Theia supports separate docker files for anomalydetector and policyrecommendation, this PR helps unify these docker files resulting into lesser storage space.
Solve #161