Clarify asset caching output #1409

JavierMatosD · 2024-05-20T14:38:17Z

https://devdiv.visualstudio.com/DevDiv/_workitems/edit/2049386

The output is changed such that, without --debug enabled:

Customers can observe a promise that Asset Caching is enabled or disabled.
Customers can observe a promise that x-block-origin is enabled or disabled.
Customers do not observe output that could be interpreted as downloading an external URL. (Unless it is downloading an external URL)

Scenario: asset cache hit!
Before:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2) Downloading portable cmake 3.29.2...
Downloading cmake...
https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz->/Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
Extracting cmake...

After:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
Asset cache hit for cmake-3.29.2-macos-universal.tar.gz; downloaded from: file:///Users/javiermatos/dev/vcpkg-tool/work/testing/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533
Extracting cmake...

Scenario: asset cache miss (configured), x-block-origin disabled
Before:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2) Downloading portable cmake 3.29.2...
Downloading cmake...
https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz->/Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
Extracting cmake...

After:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
Asset cache miss; downloading from https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz
Successfully stored cmake-3.29.2-macos-universal.tar.gz to  file:///Users/javiermatos/dev/vcpkg-tool/work/testing/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533. 
Extracting cmake...

Scenario: asset cache miss (configured), x-block-origin enabled

Before:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2) Downloading portable cmake 3.29.2...
Downloading cmake...
https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz->/Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
error: Failed to download from mirror set

After:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
error: Asset cache miss for cmake-3.29.2-macos-universal.tar.gz and downloads are blocked by x-block-origin.

Scenario: asset cache miss (not configured), x-block-origin enabled

Before:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2) Downloading portable cmake 3.29.2...
Downloading cmake...
https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz->/Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
error: Failed to download from mirror set

After:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
error: Missing cmake-3.29.2-macos-universal.tar.gz and downloads are blocked by x-block-origin.

Scenario: asset cache miss (not configured), x-block-origin disabled

Before:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2) Downloading portable cmake 3.29.2...
Downloading cmake...
https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz->/Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
Extracting cmake...

After:

Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
Downloading https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz
Extracting cmake...

** When --debug the following promise is printed:**

Asset caching not enabled, x-block-origin disabled

...
[DEBUG] External asset downloads are allowed (x-block-origin is disabled)...
[DEBUG] Asset cache is not configured.
...

Asset caching enabled, x-block-origin enabled

...
[DEBUG] External asset downloads are blocked (x-block-origin is enabled)...
[DEBUG] Asset cache is enabled.
...

autoantwort · 2024-05-22T09:32:32Z

External asset downloads are allowed (x-block-origin is disabled).
Asset caching is enabled.

Seems to be very spammy to print this every time something is downloaded

Asset cache miss!
Attempting to download from: "https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz".

Asset cache hit!

I would prefer Asset cache hit/miss for <url>. Downloading...

include/vcpkg/base/message-data.inc.h

src/vcpkg/base/downloads.cpp

BillyONeal · 2024-05-28T19:55:15Z

Could we make the output look something like this?

Asset Caching Enabled (that means x-azurl)
Scenario: x-azurl Asset Cache Hit!

Downloading file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533 (asset cache URI for https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz ) -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz

Scenario: x-block-origin disabled, x-azurl Asset Cache Miss!

Downloading file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533 (asset cache URI for https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz ) -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
Asset cache miss, downloading original

Scenario: x-block-origin enabled, x-azurl Asset Cache Miss!

Downloading file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533 (asset cache URI for https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz ) -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
Asset cache miss and fetching the original is prohibited by x-block-origin

Scenario: x-block-origin enabled, x-azurl Asset Caching Disabled

Can't download https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz because no asset caches are configured and fetching the original is prohibited by x-block-origin

Scenario: x-block-origin disabled, no asset cache configured

Downloading https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz

Scenario: x-script Asset Cache Hit!

Using script to downloads https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz

Scenario: x-block-origin disabled, x-script Asset Cache Miss!

???????? Do we try to download https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz anyway??? I don't know what we do.

Scenario: x-block-origin enabled, x-script Asset Cache Miss!

Using script to downloads https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz
Asset cache miss and fetching the original is prohibited by x-block-origin

JavierMatosD · 2024-05-28T22:29:46Z

Could we make the output look something like this?

Asset Caching Enabled (that means x-azurl) Scenario: x-azurl Asset Cache Hit!

Downloading file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533 (asset cache URI for https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz ) -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz

Scenario: x-block-origin disabled, x-azurl Asset Cache Miss!

Downloading file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533 (asset cache URI for https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz ) -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz Asset cache miss, downloading original

Scenario: x-block-origin enabled, x-azurl Asset Cache Miss!

Downloading file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533 (asset cache URI for https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz ) -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz Asset cache miss and fetching the original is prohibited by x-block-origin

Scenario: x-block-origin enabled, x-azurl Asset Caching Disabled

Can't download https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz because no asset caches are configured and fetching the original is prohibited by x-block-origin

Scenario: x-block-origin disabled, no asset cache configured

Downloading https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz

Scenario: x-script Asset Cache Hit!

Using script to downloads https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz

Scenario: x-block-origin disabled, x-script Asset Cache Miss!

???????? Do we try to download https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz anyway??? I don't know what we do.

Scenario: x-block-origin enabled, x-script Asset Cache Miss!

Using script to downloads https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz -> /Users/javiermatos/dev/vcpkg/downloads/cmake-3.29.2-macos-universal.tar.gz Asset cache miss and fetching the original is prohibited by x-block-origin

I've updated the output to include some of the information you've suggested here. It's not one for one but I think it gets the point across.

BillyONeal

I don't agree with the design change here to hide more things behind --debug

azure-pipelines/end-to-end-tests-dir/asset-caching.ps1

include/vcpkg/base/message-data.inc.h

src/vcpkg/base/downloads.cpp

…utput

JavierMatosD · 2024-06-12T20:39:46Z

I don't agree with the design change here to hide more things behind --debug

No longer hidden behind --debug and urls are sanitized :)

stale

Javier Matos and others added 5 commits May 20, 2024 10:28

clarify asset caching output

ec2158f

Ignore .vscode/settings.json

019f3be

undo

0e70a8a

remove unused message

1f5a61f

Format and regenerate messages

44287ef

Javier Matos added 3 commits May 24, 2024 12:39

propogate filename

70313b6

print asset cache and x-block-origin info when --debug

9a1be75

consolidate messaging

94eae19

vicroms reviewed May 27, 2024

View reviewed changes

include/vcpkg/base/message-data.inc.h Outdated Show resolved Hide resolved

src/vcpkg/base/downloads.cpp Outdated Show resolved Hide resolved

add e2e tests

9676b53

Javier Matos added 5 commits May 28, 2024 16:31

generalize expected output

4441c2b

generalize even more

8d149a8

try again

35795b2

maybe

6b20afa

try setting VCPKG_FORCE_DOWNLOADED_BINARIES in script

fde4271

Javier Matos and others added 7 commits May 28, 2024 18:51

normalize line endings

36411d1

remove extra downloading message on windows

ab0f726

remove unused messages

ebc59fd

Format and regenerate messages

beed0fe

solidify asset caching output;add more tests; mirror url hidden

8e3bd45

Merge remote-tracking branch 'origin/main' into clarify_asset_caching

01e3a65

clang-format

1932451

BillyONeal previously requested changes Jun 5, 2024

View reviewed changes

Javier Matos added 4 commits June 12, 2024 11:09

sanitize mirror url, print mirror whenever applicable, restore curl o…

4f553d7

…utput

update messaging

ff453cd

merge

3fd99ba

fix message signature

86d622d

Format and regenerate messages

1de2f3f

JavierMatosD requested a review from BillyONeal June 19, 2024 14:56

Merge remote-tracking branch 'origin/main' into clarify_asset_caching

afdbe0a

BillyONeal approved these changes Jul 6, 2024

View reviewed changes

Fix missing trailing newline in test.

1c7397e

BillyONeal merged commit 9eb99f7 into microsoft:main Jul 6, 2024
5 checks passed

BillyONeal mentioned this pull request Jul 11, 2024

Update vcpkg-tool to 2024-07-10 microsoft/vcpkg#39836

Merged

JavierMatosD deleted the clarify_asset_caching branch July 20, 2024 19:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clarify asset caching output #1409

Clarify asset caching output #1409

JavierMatosD commented May 20, 2024 •

edited

Loading

autoantwort commented May 22, 2024

BillyONeal commented May 28, 2024

JavierMatosD commented May 28, 2024

BillyONeal left a comment

JavierMatosD commented Jun 12, 2024

Clarify asset caching output #1409

Clarify asset caching output #1409

Conversation

JavierMatosD commented May 20, 2024 • edited Loading

autoantwort commented May 22, 2024

BillyONeal commented May 28, 2024

JavierMatosD commented May 28, 2024

BillyONeal left a comment

Choose a reason for hiding this comment

JavierMatosD commented Jun 12, 2024

JavierMatosD commented May 20, 2024 •

edited

Loading