[vcpkg_download_distfile] Fix the parameter mismatch issues of some internal functions and fix the recursive call (step 3)

[JackBoosY]

• Progress:

• step1: [vcpkg] Add aria2 as vcpkg required tool (step 1) #20971

• step2: Add aria2 to tools list (step 2) vcpkg-tool#229

• step3: [vcpkg_download_distfile] Fix the parameter mismatch issues of some internal functions and fix the recursive call (step 3) #20585

• Issue:

1. In the PR [rollup:2021-08-09] Rollup PR #19469, the modification of some parameters of z_vcpkg_download_distfile_test_hash has not been completed, and caused an error when using aira2 to verify the hash:

Starting package 1/1001: abseil:x64-linux
Building package abseil[core]:x64-linux...
-- Downloading abseil-abseil-cpp-278e0a071885a22dcd2fd1b5576cc44757299343.tar.gz...
CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:139 (z_vcpkg_download_distfile_test_hash):
  z_vcpkg_download_distfile_test_hash Function invoked with incorrect
  arguments for function named: z_vcpkg_download_distfile_test_hash
Call Stack (most recent call first):
  scripts/cmake/vcpkg_download_distfile.cmake:227 (z_vcpkg_download_distfile_via_aria)
  scripts/cmake/vcpkg_from_github.cmake:173 (vcpkg_download_distfile)
  ports/abseil/portfile.cmake:5 (vcpkg_from_github)
  scripts/ports.cmake:141 (include)


Downloaded sources for package abseil[core]:x64-linux

2. Fix recursive calls when using extra option --x-use-aria2:

tarting package 3/164: boost-config:x86-windows
Building package boost-config[core]:x86-windows...
CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:110 (vcpkg_find_acquire_program):
  Maximum recursion depth of 1000 exceeded
Call Stack (most recent call first):
  scripts/cmake/vcpkg_download_distfile.cmake:231 (z_vcpkg_download_distfile_via_aria)
  scripts/cmake/vcpkg_find_acquire_program.cmake:552 (vcpkg_download_distfile)
  scripts/cmake/vcpkg_download_distfile.cmake:110 (vcpkg_find_acquire_program)
  scripts/cmake/vcpkg_download_distfile.cmake:231 (z_vcpkg_download_distfile_via_aria)
  scripts/cmake/vcpkg_find_acquire_program.cmake:552 (vcpkg_download_distfile)
  scripts/cmake/vcpkg_download_distfile.cmake:110 (vcpkg_find_acquire_program)
  scripts/cmake/vcpkg_download_distfile.cmake:231 (z_vcpkg_download_distfile_via_aria)
  scripts/cmake/vcpkg_find_acquire_program.cmake:552 (vcpkg_download_distfile)
  scripts/cmake/vcpkg_download_distfile.cmake:110 (vcpkg_find_acquire_program)
  scripts/cmake/vcpkg_download_distfile.cmake:231 (z_vcpkg_download_distfile_via_aria)
  scripts/cmake/vcpkg_find_acquire_program.cmake:552 (vcpkg_download_distfile)
  scripts/cmake/vcpkg_download_distfile.cmake:110 (vcpkg_find_acquire_program)

• Related:
  Fixes [z_vcpkg_download_distfile_test_hash] Some parameters are incorrect #20574 vcpkg fails to download files using --x-use-aria2 when destination folder/directory does not previously exist #10199

[JackBoosY]

@strega-nil-ms
It looks like we have a problem with recursive calls (use aria2 in download process):

tarting package 3/164: boost-config:x86-windows
Building package boost-config[core]:x86-windows...
CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:110 (vcpkg_find_acquire_program):
  Maximum recursion depth of 1000 exceeded
Call Stack (most recent call first):
  scripts/cmake/vcpkg_download_distfile.cmake:231 (z_vcpkg_download_distfile_via_aria)
  scripts/cmake/vcpkg_find_acquire_program.cmake:552 (vcpkg_download_distfile)
  scripts/cmake/vcpkg_download_distfile.cmake:110 (vcpkg_find_acquire_program)
  scripts/cmake/vcpkg_download_distfile.cmake:231 (z_vcpkg_download_distfile_via_aria)
  scripts/cmake/vcpkg_find_acquire_program.cmake:552 (vcpkg_download_distfile)
  scripts/cmake/vcpkg_download_distfile.cmake:110 (vcpkg_find_acquire_program)
  scripts/cmake/vcpkg_download_distfile.cmake:231 (z_vcpkg_download_distfile_via_aria)
  scripts/cmake/vcpkg_find_acquire_program.cmake:552 (vcpkg_download_distfile)
  scripts/cmake/vcpkg_download_distfile.cmake:110 (vcpkg_find_acquire_program)
  scripts/cmake/vcpkg_download_distfile.cmake:231 (z_vcpkg_download_distfile_via_aria)
  scripts/cmake/vcpkg_find_acquire_program.cmake:552 (vcpkg_download_distfile)
  scripts/cmake/vcpkg_download_distfile.cmake:110 (vcpkg_find_acquire_program)
...

z_vcpkg_download_distfile_via_aria requires ARIA2 using vcpkg_find_acquire_program(ARIA2)
vcpkg_find_acquire_program requires vcpkg_download_distfile
vcpkg_download_distfile calls z_vcpkg_download_distfile_via_aria

[julianxhokaxhiu]

One small note: it's ARIA2 not AIRA2.

Other than that, thanks, looking forward to it :)

[JackBoosY]

Ping @strega-nil-ms for review this PR.

[Akarinnnnn]

Thank you, this works for #20648

[BillyONeal]

Tagging requires:discussion to decide if we want to route fetching aria2 through vcpkg fetch since it behaves "logically" as a builtin like cmake or ninja here.

[JackBoosY]

Fine, I will change aria2 as a vcpkg requires tools.

[JackBoosY]

I will test this changes in my private mirror tonight.

[JackBoosY]

This PR also needs to update the vcpkg min version.

[JackBoosY]

Checked with my private mirror: success.

[julianxhokaxhiu]

Is the PR stuck?

[BillyONeal]

Is the PR stuck?

No, but since it effectively triggers a world rebuild it's slow going because a build failure pretty much anywhere forces a lot of compute time so cycle time is long

[JackBoosY]

Depends on #21344.

[JackBoosY]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[JackBoosY]

For tensorflow and tensorflow-cc regression:

ERROR: Failed cleaning build dir for numpy

@BillyONeal Any ideas to solve this?

[BillyONeal]

Thanks for the fix for aria2. :)

[julianxhokaxhiu]

Thank you! One more question: how are we supposed to use Aria2 now? I noticed the --x-use-aria2 flag is gone, is it now used by default?

[BillyONeal]

Thank you! One more question: how are we supposed to use Aria2 now? I noticed the --x-use-aria2 flag is gone, is it now used by default?

It shouldn't be gone, but we did remove some x- switches from help text recently in order to reduce console spew.

[julianxhokaxhiu]

It shouldn't be gone, but we did remove some x- switches from help text recently in order to reduce console spew.

Oh I see thanks, where can I find those flags if not printed on the console? Just to know for the future :) Thanks!

//EDIT: Nvm, found it! https://github.com/microsoft/vcpkg-tool/blob/main/src/vcpkg/install.cpp#L795

[BillyONeal]

Just be aware that -x'd things can go away :)

[julianxhokaxhiu]

Just be aware that -x'd things can go away :)

Is there a long term flag I can rely upon then? Maybe via CMake?

[BillyONeal]

Not for this aria2 setting. My limited understanding is that it was x-'d because we don't really have a concrete case for it to be a strongly supported option. Someone contributed it, and as long as we have no reason to break it, we aren't going to. But we also aren't testing it etc. like non x-'d scenarios, resulting in bugs like the one that spawned this PR in the first place.

That aria2's licensing isn't compatible with ours so we can't distribute together, and we have this catch22 problem of needing to use something else to download it in the first place, remains kind of an issue here.

[julianxhokaxhiu]

That aria2's licensing isn't compatible with ours so we can't distribute together, and we have this catch22 problem of needing to use something else to download it in the first place, remains kind of an issue here.

Do you think it's possible to open a discussion with ARIA2 author somehow to dual license the project to fit the needs of vcpkg?

[BillyONeal]

No idea. Either way it would depend on a better understanding of why someone is attempting to replace it with aria2 in the first place.