Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report background download for BwoB #17604

Closed
wants to merge 2 commits into from
Closed

Conversation

coeuvre
Copy link
Member

@coeuvre coeuvre commented Feb 27, 2023

When building without the bytes, downloads can happen when Bazel need to fetch inputs for local actions, download outputs of toplevel targets, and etc. The downloads happen silently at the background.

This PR makes output downloads of toplevel targets visible to the UI.

For downloads during the build, the UI looks like:

[2 / 5] 2 actions, 1 running
    [Prepa] Executing genrule //:foo
    Executing genrule //:bar; 2s
    Fetching bazel-out/darwin-fastbuild/bin/baz; 50.8 MiB / 1.0 GiB; 4s

For downloads after build is completed, the UI looks like:

INFO: 2 processes: 1 remote cache hit, 1 internal.
INFO: Build completed successfully, 2 total actions
    Fetching bazel-out/darwin-fastbuild/bin/baz; 48.8 MiB / 1.0 GiB

Fixes #17417.

@coeuvre coeuvre requested a review from a team as a code owner February 27, 2023 16:08
@sgowroji sgowroji added the team-Remote-Exec Issues and PRs for the Execution (Remote) team label Feb 27, 2023
@brentleyjones
Copy link
Contributor

@coeuvre is this cherry-pickable? Nice change!

@coeuvre
Copy link
Member Author

coeuvre commented Feb 28, 2023

Thanks! I think it's possible to include it in 6.1.

@brentleyjones
Copy link
Contributor

@bazel-io flag

@bazel-io bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Feb 28, 2023
coeuvre added a commit to coeuvre/bazel that referenced this pull request Feb 28, 2023
When building without the bytes, downloads can happen when Bazel need to fetch inputs for local actions, download outputs of toplevel targets, and etc. The downloads happen silently at the background.

This PR makes output downloads of toplevel targets visible to the UI.

For downloads during the build, the UI looks like:

```
[2 / 5] 2 actions, 1 running
    [Prepa] Executing genrule //:foo
    Executing genrule //:bar; 2s
    Fetching bazel-out/darwin-fastbuild/bin/baz; 50.8 MiB / 1.0 GiB; 4s
```

For downloads after build is completed, the UI looks like:

```
INFO: 2 processes: 1 remote cache hit, 1 internal.
INFO: Build completed successfully, 2 total actions
    Fetching bazel-out/darwin-fastbuild/bin/baz; 48.8 MiB / 1.0 GiB
```

Fixes bazelbuild#17417.

Closes bazelbuild#17604.

PiperOrigin-RevId: 512934756
Change-Id: I502489420ab9b84efb74ceabbe3dd4cb4a7a62e2
ShreeM01 pushed a commit that referenced this pull request Feb 28, 2023
When building without the bytes, downloads can happen when Bazel need to fetch inputs for local actions, download outputs of toplevel targets, and etc. The downloads happen silently at the background.

This PR makes output downloads of toplevel targets visible to the UI.

For downloads during the build, the UI looks like:

```
[2 / 5] 2 actions, 1 running
    [Prepa] Executing genrule //:foo
    Executing genrule //:bar; 2s
    Fetching bazel-out/darwin-fastbuild/bin/baz; 50.8 MiB / 1.0 GiB; 4s
```

For downloads after build is completed, the UI looks like:

```
INFO: 2 processes: 1 remote cache hit, 1 internal.
INFO: Build completed successfully, 2 total actions
    Fetching bazel-out/darwin-fastbuild/bin/baz; 48.8 MiB / 1.0 GiB
```

Fixes #17417.

Closes #17604.

PiperOrigin-RevId: 512934756
Change-Id: I502489420ab9b84efb74ceabbe3dd4cb4a7a62e2
@coeuvre coeuvre deleted the fix-17417 branch March 1, 2023 10:20
@ShreeM01 ShreeM01 removed the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Mar 2, 2023
fweikert pushed a commit to fweikert/bazel that referenced this pull request May 25, 2023
When building without the bytes, downloads can happen when Bazel need to fetch inputs for local actions, download outputs of toplevel targets, and etc. The downloads happen silently at the background.

This PR makes output downloads of toplevel targets visible to the UI.

For downloads during the build, the UI looks like:

```
[2 / 5] 2 actions, 1 running
    [Prepa] Executing genrule //:foo
    Executing genrule //:bar; 2s
    Fetching bazel-out/darwin-fastbuild/bin/baz; 50.8 MiB / 1.0 GiB; 4s
```

For downloads after build is completed, the UI looks like:

```
INFO: 2 processes: 1 remote cache hit, 1 internal.
INFO: Build completed successfully, 2 total actions
    Fetching bazel-out/darwin-fastbuild/bin/baz; 48.8 MiB / 1.0 GiB
```

Fixes bazelbuild#17417.

Closes bazelbuild#17604.

PiperOrigin-RevId: 512934756
Change-Id: I502489420ab9b84efb74ceabbe3dd4cb4a7a62e2
copybara-service bot pushed a commit that referenced this pull request Jun 12, 2023
…r#prefetchFiles`.

Due to recent overhual of `prefetchFiles`, the call sites now always have the reference to `ActionExecutionMetadata`, which makes it possible to pass it to `prefetchFiles`.

The reason why we want to access `ActionExecutionMetadata` inside `prefetchFiles` is to associate downloads with actions. For example:

1. When downloading from remote cache, we want to provide action metadata to the RPCs so that remote server can associate them with actions. `actionId` was removed from remote file metadata by f62a8b9 because it is not the right action to associate with for downloads inside this context. We left the action metadata empty for RPCs until this change. #17724 (comment)
2. When building with the bytes, we are able to report download progress to UI. However, `prefetchFiles` is the major way to download when building without the bytes and we can't report download progress on action basis. #17604 added the download report for background downloads, but now, we no longer download toplevel artifacts in background (a5dde12). With this change, we are able to report download progress for each action, either when they download outputs, or prefetch inputs for building without the bytes.

PiperOrigin-RevId: 539635790
Change-Id: Ic9cc1024f5d3560ac46bc462bd549dd81712d92f
coeuvre added a commit to coeuvre/bazel that referenced this pull request Jun 13, 2023
…r#prefetchFiles`.

Due to recent overhual of `prefetchFiles`, the call sites now always have the reference to `ActionExecutionMetadata`, which makes it possible to pass it to `prefetchFiles`.

The reason why we want to access `ActionExecutionMetadata` inside `prefetchFiles` is to associate downloads with actions. For example:

1. When downloading from remote cache, we want to provide action metadata to the RPCs so that remote server can associate them with actions. `actionId` was removed from remote file metadata by f62a8b9 because it is not the right action to associate with for downloads inside this context. We left the action metadata empty for RPCs until this change. bazelbuild#17724 (comment)
2. When building with the bytes, we are able to report download progress to UI. However, `prefetchFiles` is the major way to download when building without the bytes and we can't report download progress on action basis. bazelbuild#17604 added the download report for background downloads, but now, we no longer download toplevel artifacts in background (a5dde12). With this change, we are able to report download progress for each action, either when they download outputs, or prefetch inputs for building without the bytes.

PiperOrigin-RevId: 539635790
Change-Id: Ic9cc1024f5d3560ac46bc462bd549dd81712d92f
coeuvre added a commit to coeuvre/bazel that referenced this pull request Jun 13, 2023
…r#prefetchFiles`.

Due to recent overhual of `prefetchFiles`, the call sites now always have the reference to `ActionExecutionMetadata`, which makes it possible to pass it to `prefetchFiles`.

The reason why we want to access `ActionExecutionMetadata` inside `prefetchFiles` is to associate downloads with actions. For example:

1. When downloading from remote cache, we want to provide action metadata to the RPCs so that remote server can associate them with actions. `actionId` was removed from remote file metadata by f62a8b9 because it is not the right action to associate with for downloads inside this context. We left the action metadata empty for RPCs until this change. bazelbuild#17724 (comment)
2. When building with the bytes, we are able to report download progress to UI. However, `prefetchFiles` is the major way to download when building without the bytes and we can't report download progress on action basis. bazelbuild#17604 added the download report for background downloads, but now, we no longer download toplevel artifacts in background (a5dde12). With this change, we are able to report download progress for each action, either when they download outputs, or prefetch inputs for building without the bytes.

PiperOrigin-RevId: 539635790
Change-Id: Ic9cc1024f5d3560ac46bc462bd549dd81712d92f
iancha1992 added a commit that referenced this pull request Jun 13, 2023
…r#prefetchFiles`. (#18656)

Due to recent overhual of `prefetchFiles`, the call sites now always have the reference to `ActionExecutionMetadata`, which makes it possible to pass it to `prefetchFiles`.

The reason why we want to access `ActionExecutionMetadata` inside `prefetchFiles` is to associate downloads with actions. For example:

1. When downloading from remote cache, we want to provide action metadata to the RPCs so that remote server can associate them with actions. `actionId` was removed from remote file metadata by f62a8b9 because it is not the right action to associate with for downloads inside this context. We left the action metadata empty for RPCs until this change. #17724 (comment)
2. When building with the bytes, we are able to report download progress to UI. However, `prefetchFiles` is the major way to download when building without the bytes and we can't report download progress on action basis. #17604 added the download report for background downloads, but now, we no longer download toplevel artifacts in background (a5dde12). With this change, we are able to report download progress for each action, either when they download outputs, or prefetch inputs for building without the bytes.

PiperOrigin-RevId: 539635790
Change-Id: Ic9cc1024f5d3560ac46bc462bd549dd81712d92f

Co-authored-by: Ian (Hee) Cha <[email protected]>
traversaro pushed a commit to traversaro/bazel that referenced this pull request Jun 24, 2023
…r#prefetchFiles`.

Due to recent overhual of `prefetchFiles`, the call sites now always have the reference to `ActionExecutionMetadata`, which makes it possible to pass it to `prefetchFiles`.

The reason why we want to access `ActionExecutionMetadata` inside `prefetchFiles` is to associate downloads with actions. For example:

1. When downloading from remote cache, we want to provide action metadata to the RPCs so that remote server can associate them with actions. `actionId` was removed from remote file metadata by f62a8b9 because it is not the right action to associate with for downloads inside this context. We left the action metadata empty for RPCs until this change. bazelbuild#17724 (comment)
2. When building with the bytes, we are able to report download progress to UI. However, `prefetchFiles` is the major way to download when building without the bytes and we can't report download progress on action basis. bazelbuild#17604 added the download report for background downloads, but now, we no longer download toplevel artifacts in background (a5dde12). With this change, we are able to report download progress for each action, either when they download outputs, or prefetch inputs for building without the bytes.

PiperOrigin-RevId: 539635790
Change-Id: Ic9cc1024f5d3560ac46bc462bd549dd81712d92f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team-Remote-Exec Issues and PRs for the Execution (Remote) team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

--remote_download_toplevel downloads stuff silently
5 participants