AWS Profile credentials no longer working in object_store 0.6.1

[watfordkcf]

Describe the bug
I am bringing in object_store via delta-rs. I bumped from delta-rs v0.12.0 (object_store v0.5.6) to delta-rs v0.13.0 (object_store v0.6.1) and now profile credentials no longer work.

To Reproduce
Given:

$ export AWS_PROFILE=my_profile_here
$ export AWS_REGION=us-west-2
$ RUST_LOG=debug RUST_BACKTRACE=full cargo run -- s3://my-bucket-here

With the following code unchanged between delta-rs / object_store versions:

let table = deltalake::open_table(format!("{}/{}", self.table_root, table_name))
     .await
     .unwrap();

The following is then received:

    Finished release [optimized] target(s) in 5.58s
     Running `target/release/my_rust_project 's3://my-bucket-here'`
 DEBUG deltalake::action > loading checkpoint from _delta_log/_last_checkpoint
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG deltalake::action > loading checkpoint from _delta_log/_last_checkpoint
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG deltalake::action > loading checkpoint from _delta_log/_last_checkpoint
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG deltalake::action > loading checkpoint from _delta_log/_last_checkpoint
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
 DEBUG reqwest::connect  > starting new connection: http://169.254.169.254/
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ObjectStore { source: Generic { store: "S3", source: Error { retries: 10, message: "request error", source: Some(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(169.254.169.254)), port: None, path: "/latest/api/token", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 64, kind: Uncategorized, message: "Host is down" })) }), status: None } } }', src/data/database.rs:23:14
stack backtrace:
   0:        0x104628008 - std::backtrace_rs::backtrace::libunwind::trace::h0a647ce7e8dc2fab
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:        0x104628008 - std::backtrace_rs::backtrace::trace_unsynchronized::hea920694a2a8ac80
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x104628008 - std::sys_common::backtrace::_print_fmt::h7b4e20c1da2ebb61
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:65:5
   3:        0x104628008 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h819e9cbdf1a9e730
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:44:22
   4:        0x1046485bc - core::fmt::write::ha5e9bf3131ecb7c0
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/fmt/mod.rs:1254:17
   5:        0x1046249e0 - std::io::Write::write_fmt::h414ce9994bf17404
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/io/mod.rs:1698:15
   6:        0x104627e1c - std::sys_common::backtrace::_print::h28d98f2094da6d1d
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:47:5
   7:        0x104627e1c - std::sys_common::backtrace::print::h8072db0bbd5bcc3d
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:34:9
   8:        0x10462944c - std::panicking::default_hook::{{closure}}::h2c85c5b0c2ede151
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:269:22
   9:        0x10462920c - std::panicking::default_hook::hcf2f70992d02f6fe
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:288:9
  10:        0x104629924 - std::panicking::rust_panic_with_hook::h023af7f90b47eb8b
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:691:13
  11:        0x104629858 - std::panicking::begin_panic_handler::{{closure}}::h14283519edc1d634
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:582:13
  12:        0x104628428 - std::sys_common::backtrace::__rust_end_short_backtrace::hc366c0b0cef5b747
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:150:18
  13:        0x1046295ec - rust_begin_unwind
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
  14:        0x1046fd404 - core::panicking::panic_fmt::h324f50b29db90195
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
  15:        0x1046fd73c - core::result::unwrap_failed::hf783e6a14bbaf60b
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/result.rs:1687:5
  16:        0x10253ab10 - <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll::h22f19ab28f25268b
  17:        0x102559034 - <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll::h8d0aff057d3a9ac9
  18:        0x102512bdc - tokio::runtime::park::CachedParkThread::block_on::hb93ac530558beb5e
  19:        0x10251331c - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::h1bb79c8698d80794
  20:        0x10260f06c - tokio::runtime::runtime::Runtime::block_on::h28a6fea255a05c54
  21:        0x10255be10 - my_rust_project::main::he33b3ec66cdcca6b
  22:        0x1024e1ff0 - std::sys_common::backtrace::__rust_begin_short_backtrace::h726a84314bc45b78
  23:        0x1024e2008 - std::rt::lang_start::{{closure}}::h1cef783781c93022
  24:        0x10461f560 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h6f7eb9f266759f90
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
  25:        0x10461f560 - std::panicking::try::do_call::h54b2febb9ea02379
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
  26:        0x10461f560 - std::panicking::try::h95a2f9f45aeb75ea
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
  27:        0x10461f560 - std::panic::catch_unwind::h9686256fa0fc97a1
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
  28:        0x10461f560 - std::rt::lang_start_internal::{{closure}}::h227e8b10bc4e486b
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
  29:        0x10461f560 - std::panicking::try::do_call::h414d500a3ee5fa44
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
  30:        0x10461f560 - std::panicking::try::h4f025820961f1c3f
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
  31:        0x10461f560 - std::panic::catch_unwind::h0b71dfe3538d125d
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
  32:        0x10461f560 - std::rt::lang_start_internal::h8ee16b8f6c950a26
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  33:        0x10255bf10 - _main

Expected behavior
The table is opened and read to be used.

Additional context
Here is the abbreviated output from the previous version of delta-rs/object_store:

    Finished release [optimized] target(s) in 5.58s
     Running `target/release/health_of_health 's3://my-bucket-here'`
 INFO  object_store::aws > Using profile "my_profile_here" credential provider
 DEBUG tracing::span     > build_profile_provider;
 DEBUG hyper_rustls::config > with_native_roots processed 164 valid and 0 invalid certs
 DEBUG deltalake::action    > loading checkpoint from _delta_log/_last_checkpoint
 DEBUG aws_config::fs_util  > loaded home directory src="HOME"
 DEBUG aws_config::profile::parser::source > load_config_file; file=Default(Config)
 DEBUG aws_config::profile::parser::source > performing home directory substitution home="/Users/watford" path="~/.aws/config"
 DEBUG aws_config::profile::parser::source > home directory expanded before="~/.aws/config" after="/Users/watford/.aws/config"
 DEBUG aws_config::profile::parser::source > config file loaded path=Some("/Users/watford/.aws/config") size=2526
 DEBUG aws_config::profile::parser::source > load_config_file; file=Default(Credentials)
 DEBUG aws_config::profile::parser::source > performing home directory substitution home="/Users/watford" path="~/.aws/credentials"
 DEBUG aws_config::profile::parser::source > home directory expanded before="~/.aws/credentials" after="/Users/watford/.aws/credentials"
 DEBUG aws_config::profile::parser::source > config file loaded path=Some("/Users/watford/.aws/credentials") size=6
 INFO  aws_config::profile::credentials    > constructed abstract provider from config file chain=ProfileChain { base: Sso { sso_account_id: "...

If I eval my SSO session credentials into environment variables then this works just fine on 0.6.1, but that's a pretty big regression.

[tustvold]

First-party AWS profile support was removed as part of #4238 and released as part of 0.6.0.

Downstreams wishing to provide AWS profile support will need to hook up their chosen AWS SDK as a credential provider, providing full support for all the various authentication options AWS CLIs provide is out of scope for this crate, and was causing friction as we never replicated the CLI behaviour consistently.

An example of doing this can be found in datafusion-cli - https://github.com/apache/arrow-datafusion/blob/main/datafusion-cli/src/object_storage.rs.

Further reasoning behind this decision can be found on #4137 and #2178

[watfordkcf]

Ouch, okay. Going to pin object_store to 0.5.6 for now and determine the right path forward.

[tustvold]

Feel free to let me know if you get stuck and I can try to help out, the datafusion-cli may be a good starting point.

Alternatively you may consider using something like aws-vault, as an added bonus this will avoid caching temporary credentials on disk in plain text

[watfordkcf]

aws-vault was a nice suggestion, certainly makes this easier. I'm unblocked for now. We're consuming this via delta-rs so it wasn't immediately apparent how I adapt the datafusion-cli logic there.

[stevenmanton]

This issue took me a very long time to track down. If the AWS_PROFILE environment variable is being ignored, would it be better to throw a warning (or even an exception) so that it's clear that the profile is not being used? Otherwise, it's unclear because some standard environment variables are used, whereas others aren't. I understand that implementing a fully functional AWS credential provider is out of scope for this package.