feat(sync-files): allow multiple destinations and speed up clone #295

xmfcx · 2024-06-07T13:35:53Z

Description

Speed up how?

Added --depth 1 to the git clone operation, performing a shallow clone, avoiding downloading entire history and branches of a repository.

It doesn't use anything more, so it should be safe.

Why iterate over multiple destinations? What was wrong with the existing approach?

Assume the following sync-files.yaml configuration:

- repository: autowarefoundation/autoware_common
  files:
    - source: .github/workflows/build-and-test.yaml
      dest: .github/workflows/build-and-test-daily.yaml
    - source: .github/workflows/build-and-test.yaml
      dest: .github/workflows/build-and-test-daily-arm64.yaml

It wants to copy from a source file into multiple destinations.

But when we run it with the existing configuration, it only generates the dest: .github/workflows/build-and-test-daily-arm64.yaml file. And completely ignores the first destination.

Proof

🖱️Click here to expand🔛

Input:

https://github.com/autowarefoundation/autoware.universe/actions/runs/9415404629/job/25936750262#step:3:208

It only creates arm64 version:

https://github.com/autowarefoundation/autoware.universe/actions/runs/9415404629/job/25936750262#step:3:626

Why does this occur?

To tackle this, I've ran https://gist.github.com/xmfcx/054a81b38ea583d39f792c6679c7bcde?permalink_comment_id=5081803#gistcomment-5081803 this script that I copied from the action itself. Put this in my autoware.universe directory.

Then put this https://gist.github.com/xmfcx/054a81b38ea583d39f792c6679c7bcde?permalink_comment_id=5081799#gistcomment-5081799 file in /tmp/sync-files.yaml

Also installed:

When I ran it, I saw:

source_path: .github/workflows/build-and-test.yaml
dest_path: .github/workflows/build-and-test-daily-arm64.yaml
/tmp/repository/.github/workflows/build-and-test.yaml and .github/workflows/build-and-test-daily-arm64.yaml are the same.
source_path: .github/workflows/build-and-test.yaml
dest_path: .github/workflows/build-and-test-daily-arm64.yaml
/tmp/repository/.github/workflows/build-and-test.yaml and .github/workflows/build-and-test-daily-arm64.yaml are the same.

Duplicate entries.

But then I ran it with the updated version from https://gist.github.com/xmfcx/054a81b38ea583d39f792c6679c7bcde?permalink_comment_id=5081803#file-sync-files-bash

And it was fixed:

source_path: .github/workflows/build-and-test.yaml
dest_path: .github/workflows/build-and-test-daily.yaml
/tmp/repository/.github/workflows/build-and-test.yaml and .github/workflows/build-and-test-daily.yaml are the same.
source_path: .github/workflows/build-and-test.yaml
dest_path: .github/workflows/build-and-test-daily-arm64.yaml
/tmp/repository/.github/workflows/build-and-test.yaml and .github/workflows/build-and-test-daily-arm64.yaml are the same.

We have both files generated.

Effects on system behavior

I can't see this breaking any existing behavior.

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

I've confirmed the contribution guidelines.
The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

The PR follows the pull request guidelines.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

Signed-off-by: M. Fatih Cırıt <[email protected]>

xmfcx · 2024-06-07T14:14:35Z

To make better sense of things, I will share outputs of some commands so they are more concrete for everyone:

Files

🖱️Click here to expand🔛

The input file

mfc@mfc-leo:~$ cat /tmp/sync-files.yaml 
- files:
  - delete-orphaned: true
    dest: .github/workflows/build-and-test-daily.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .github/workflows/build-and-test.yaml
  - delete-orphaned: true
    dest: .github/workflows/build-and-test-daily-arm64.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .github/workflows/build-and-test.yaml
  - delete-orphaned: true
    dest: .github/workflows/check-build-depends.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .github/workflows/check-build-depends.yaml
  - delete-orphaned: true
    dest: .github/workflows/clang-tidy-pr-comments.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .github/workflows/clang-tidy-pr-comments.yaml
  - delete-orphaned: true
    dest: .github/workflows/clang-tidy-pr-comments-manually.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .github/workflows/clang-tidy-pr-comments-manually.yaml
  - delete-orphaned: true
    dest: .github/workflows/update-codeowners-from-packages.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .github/workflows/update-codeowners-from-packages.yaml
  - delete-orphaned: true
    dest: .pre-commit-config.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .pre-commit-config.yaml
  - delete-orphaned: true
    dest: codecov.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: codecov.yaml
  ref: ''
  repository: https://github.com/autowarefoundation/autoware_common.git

Iterated last repository

mfc@mfc-leo:~$ cat /tmp/repo-config.yaml
files:
  - delete-orphaned: true
    dest: .github/workflows/build-and-test-daily.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .github/workflows/build-and-test.yaml
  - delete-orphaned: true
    dest: .github/workflows/build-and-test-daily-arm64.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .github/workflows/build-and-test.yaml
  - delete-orphaned: true
    dest: .github/workflows/check-build-depends.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .github/workflows/check-build-depends.yaml
  - delete-orphaned: true
    dest: .github/workflows/clang-tidy-pr-comments.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .github/workflows/clang-tidy-pr-comments.yaml
  - delete-orphaned: true
    dest: .github/workflows/clang-tidy-pr-comments-manually.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .github/workflows/clang-tidy-pr-comments-manually.yaml
  - delete-orphaned: true
    dest: .github/workflows/update-codeowners-from-packages.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .github/workflows/update-codeowners-from-packages.yaml
  - delete-orphaned: true
    dest: .pre-commit-config.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: .pre-commit-config.yaml
  - delete-orphaned: true
    dest: codecov.yaml
    post-commands: ''
    pre-commands: ''
    replace: true
    source: codecov.yaml
ref: ''
repository: https://github.com/autowarefoundation/autoware_common.git

Iterated last file item; (`codecov.yaml`) from above:

mfc@mfc-leo:~$ cat /tmp/file-config.yaml
delete-orphaned: true
dest: codecov.yaml
post-commands: ''
pre-commands: ''
replace: true
source: codecov.yaml

Some `yq` commands executed:

🖱️Click here to expand🔛

mfc@mfc-leo:~$ yq ".files[].dest" /tmp/repo-config.yaml
.github/workflows/build-and-test-daily.yaml
.github/workflows/build-and-test-daily-arm64.yaml
.github/workflows/check-build-depends.yaml
.github/workflows/clang-tidy-pr-comments.yaml
.github/workflows/clang-tidy-pr-comments-manually.yaml
.github/workflows/update-codeowners-from-packages.yaml
.pre-commit-config.yaml
codecov.yaml

mfc@mfc-leo:~$ export dest_file=.github/workflows/build-and-test-daily.yaml

mfc@mfc-leo:~$ yq ".files[] | select(.dest == \"$dest_file\")" /tmp/repo-config.yaml
delete-orphaned: true
dest: .github/workflows/build-and-test-daily.yaml
post-commands: ''
pre-commands: ''
replace: true
source: .github/workflows/build-and-test.yaml

Signed-off-by: M. Fatih Cırıt <[email protected]>

mitsudome-r

LGTM

xmfcx · 2024-06-07T15:25:41Z

https://github.com/autowarefoundation/autoware.universe/pull/7368/files

@mitsudome-r works now, thanks for reviewing!

xmfcx added 3 commits June 7, 2024 15:55

feat(sync-files): speed up and allow multiple copies

e48fe04

Signed-off-by: M. Fatih Cırıt <[email protected]>

iterate over .dest fields instead of .source

81a47c9

Signed-off-by: M. Fatih Cırıt <[email protected]>

fix dest

6c998d5

Signed-off-by: M. Fatih Cırıt <[email protected]>

xmfcx requested review from mitsudome-r and isamu-takagi June 7, 2024 14:23

xmfcx self-assigned this Jun 7, 2024

xmfcx marked this pull request as ready for review June 7, 2024 14:25

xmfcx requested a review from HansRobo June 7, 2024 14:27

name the var to file_config for clarity

6cc80ac

Signed-off-by: M. Fatih Cırıt <[email protected]>

mitsudome-r approved these changes Jun 7, 2024

View reviewed changes

xmfcx merged commit dcd51f2 into main Jun 7, 2024
13 checks passed

xmfcx deleted the feat/improve-sync-files branch June 7, 2024 15:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(sync-files): allow multiple destinations and speed up clone #295

feat(sync-files): allow multiple destinations and speed up clone #295

xmfcx commented Jun 7, 2024 •

edited

Loading

xmfcx commented Jun 7, 2024 •

edited

Loading

The input file

Iterated last repository

Iterated last file item; (`codecov.yaml`) from above:

mitsudome-r left a comment

xmfcx commented Jun 7, 2024

feat(sync-files): allow multiple destinations and speed up clone #295

feat(sync-files): allow multiple destinations and speed up clone #295

Conversation

xmfcx commented Jun 7, 2024 • edited Loading

Description

Speed up how?

Why iterate over multiple destinations? What was wrong with the existing approach?

Proof

Why does this occur?

Effects on system behavior

Pre-review checklist for the PR author

In-review checklist for the PR reviewers

Post-review checklist for the PR author

xmfcx commented Jun 7, 2024 • edited Loading

Files

The input file

Iterated last repository

Iterated last file item; (codecov.yaml) from above:

Some yq commands executed:

mitsudome-r left a comment

Choose a reason for hiding this comment

xmfcx commented Jun 7, 2024

xmfcx commented Jun 7, 2024 •

edited

Loading

xmfcx commented Jun 7, 2024 •

edited

Loading

Iterated last file item; (`codecov.yaml`) from above:

Some `yq` commands executed: