Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: artifact subdir error when using volumeMount #12638

Merged
merged 4 commits into from
Feb 8, 2024

Conversation

tczhao
Copy link
Member

@tczhao tczhao commented Feb 7, 2024

Fixes #12174

I tried contacting original author from #12200 but met no success

We want to expedite this fix and have this ready for the next patch release

Motivation

To make sure if an artifact contains a subdir and is mounted using volume claim, the workflow runs without failing.
VolumeMount is necessary if the user wants to use ephemeral storage.

Modifications

Make sure the executor creates a subdirectory before downloading files

Verification

Added new e2e test -> e2e failed (verified original issue)
Added code change -> e2e succeed (verified fixed the issue)

@tczhao tczhao changed the title fix: artifact dir in ephemeral mount fix: artifact subdir error when using volumeMount Feb 8, 2024
@tczhao tczhao marked this pull request as ready for review February 8, 2024 01:42
Copy link
Member

@terrytangyuan terrytangyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Welcome back!

@terrytangyuan terrytangyuan merged commit 6c8a715 into argoproj:main Feb 8, 2024
30 checks passed
@juliev0
Copy link
Contributor

juliev0 commented Feb 8, 2024

Hey @tczhao - I'm not sure how much you plan on contributing going forward but in case you're interested, I wanted to link you to this just in case you haven't seen it :)

@tczhao
Copy link
Member Author

tczhao commented Feb 8, 2024

Hey @tczhao - I'm not sure how much you plan on contributing going forward but in case you're interested, I wanted to link you to this just in case you haven't seen it :)

Hi @juliev0, thanks for letting me know.
I should sign up for that.

@agilgur5 agilgur5 added the area/artifacts S3/GCP/OSS/Git/HDFS etc label Feb 8, 2024
isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this pull request Feb 20, 2024
isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this pull request Feb 27, 2024
isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this pull request Feb 28, 2024
isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this pull request Mar 12, 2024
@@ -204,6 +204,11 @@ func (we *WorkflowExecutor) LoadArtifacts(ctx context.Context) error {
// the file is a tarball or not. If it is, it is first extracted then renamed to
// the desired location. If not, it is simply renamed to the location.
tempArtPath := artPath + ".tmp"
// Ensure parent directory exist, create if missing
tempArtDir := filepath.Dir(tempArtPath)
if err := os.MkdirAll(tempArtDir, 0o700); err != nil {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: why do we set this 700?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code here creates a temporary path, then artifact driver will load(download) the object to this path.
The minimum dir permission across the artifact drivers is 700.
Are you experiencing an issue with this?

tczhao added a commit to tczhao/argo that referenced this pull request Apr 19, 2024
isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this pull request May 6, 2024
isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this pull request May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/artifacts S3/GCP/OSS/Git/HDFS etc
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Single file input artifact not getting loaded if artifact path is in ephemeral volume mount
5 participants