Support testing without AWS secrets

[maleadt]

It looks like CI here has AWS secrets provided:

AWSS3.jl/.github/workflows/CI.yml

Lines 51 to 53 in 5eb7a78

	- env:
	AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
	AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

This of course isn't the case on PkgEval, resulting in this package not being tested (or rather it it always failing its tests):

AWSS3.jl: Error During Test at /home/pkgeval/.julia/packages/AWSS3/eu6sJ/test/runtests.jl:25
  Got exception outside of a @test
  Can't find AWS credentials!
  
  Stacktrace:
    [1] AWSCredentials(; profile::Nothing, throw_cred_error::Bool)
      @ AWS ~/.julia/packages/AWS/Rj4FY/src/AWSCredentials.jl:133
    [2] AWSConfig()
      @ AWS ~/.julia/packages/AWS/Rj4FY/src/AWSConfig.jl:11
    [3] macro expansion
      @ ~/.julia/packages/AWSS3/eu6sJ/test/runtests.jl:56 [inlined]
    [4] macro expansion
      @ /opt/julia/share/julia/stdlib/v1.9/Test/src/Test.jl:1496 [inlined]
    [5] top-level scope
      @ ~/.julia/packages/AWSS3/eu6sJ/test/runtests.jl:26
    [6] include(fname::String)
      @ Base.MainInclude ./client.jl:478
    [7] top-level scope
      @ none:6
    [8] eval
      @ ./boot.jl:370 [inlined]
    [9] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:280
   [10] _start()
      @ Base ./client.jl:522
Test Summary: | Pass  Error  Broken  Total     Time
AWSS3.jl      |  274      1       4    279  2m40.7s
  Minio       |  274              4    278  2m08.2s
ERROR: LoadError: Some tests did not pass: 274 passed, 0 failed, 1 errored, 4 broken.
in expression starting at /home/pkgeval/.julia/packages/AWSS3/eu6sJ/test/runtests.jl:25

FAILED: Package AWSS3 errored during testing
Stacktrace:
  [1] pkgerror(msg::String)
    @ Pkg.Types /opt/julia/share/julia/stdlib/v1.9/Pkg/src/Types.jl:68
  [2] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool)
    @ Pkg.Operations /opt/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1878
  [3] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::Base.Pairs{Symbol, Base.PipeEndpoint, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.PipeEndpoint}}})
    @ Pkg.API /opt/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:435
  [4] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.PipeEndpoint, kwargs::Base.Pairs{Symbol, Cmd, Tuple{Symbol}, NamedTuple{(:julia_args,), Tuple{Cmd}}})
    @ Pkg.API /opt/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
  [5] #test#75
    @ /opt/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:144 [inlined]
  [6] #test#74
    @ /opt/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:143 [inlined]
  [7] top-level scope
    @ none:38
  [8] eval
    @ ./boot.jl:370 [inlined]
  [9] eval(x::Expr)
    @ Base.MainInclude ./client.jl:480
 [10] top-level scope
    @ none:1
 [11] eval
    @ ./boot.jl:370 [inlined]
 [12] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:280
 [13] _start()
    @ Base ./client.jl:522

It would be great if this package could be tested by PkgEval. I guess the tests could just be made dependent on the availability of secrets? Alternatively, you can also check the PKGEVAL/JULIA_PKGEVAL env vars.

[mattBrzezinski]

The CI runs on a public AWS account provided by Invenia so we can do some integration testing for uploading, deleting, etc. objects.

If these secrets were added into PkgEval would that resolve the issue?

[maleadt]

If these secrets were added into PkgEval would that resolve the issue?

No, adding secrets (or other environmental information) to PkgEval on a per-package basis is not a scalable solution.

[mattBrzezinski]

If these secrets were added into PkgEval would that resolve the issue?

No, adding secrets (or other environmental information) to PkgEval on a per-package basis is not a scalable solution.

I'm not quite sure what the path forward is here in that case. We need to limit access to the AWS account, but still be able to test the package concretely.

I would not want to add in special code checking for PkgEval env vars and restructuring tests to have it work as this is not a dependency of this package.

We could create a public bucket in the account and place some objects in there, restructure tests to separate ones which do and do not require credentials. But that limits things to read only operations and does not seem value.

[maleadt]

I would not want to add in special code checking for PkgEval env vars and restructuring tests to have it work as this is not a dependency of this package.

You only need to check an environment variable. Why is that problematic? It's generally a good thing that registered, public packages can be tested without special set-up.

Alternatively, CloubBase.jl's minio wrapper could be used to create a public, S3-compatible environment, e.g., https://github.com/JuliaLang/BugReporting.jl/blob/00d09eede70ec9e3784e4f83ac26d338460fff6b/test/rr.jl#L104-L116

Ultimately though, I don't really care about AWSS3.jl's PkgEval-compatibility, so feel free to close. The only disadvantage is that this package won't be considered when testing for upstream Julia changes.

[mattBrzezinski]

You only need to check an environment variable. Why is that problematic? It's generally a good thing that registered, public packages can be tested without special set-up.

IMO, this package as well as AWS.jl and others are a bit more unique where if you're using them there's somewhat of an expectation that you have AWS credentials present. Although they do support credential-less requests, as part of testing you would want to test that your changes and nightly runs check everything.

Alternatively, CloubBase.jl's minio wrapper could be used to create a public, S3-compatible environment, e.g., https://github.com/JuliaLang/BugReporting.jl/blob/00d09eede70ec9e3784e4f83ac26d338460fff6b/test/rr.jl#L104-L116
Ultimately though, I don't really care about AWSS3.jl's PkgEval-compatibility, so feel free to close. The only disadvantage is that this package won't be considered when testing for upstream Julia changes.

Using MinIO would work, could refactor this to:

1. Run tests against MinIO
2. If credentials are present, run them on AWS

Personally I do not have the time to make these currently, if you make the PR though I can review though. Honestly this package really needs to be re-written anyways so I'm not too concerned about its current state that much.

[maleadt]

Yeah, I don't have the time for that either. I'll just blacklist the package.