No AWS client logs are surfaced at TF_LOG=trace level

[t0yv0]

What happened?

When diagnosing performance degradation or apparent hangs for clients we often have logs and traces but no repro; in these situations it can be extremely helpful to have visibility into what the provider is doing, in particular to see what calls are being made against the AWS API. Currently the upstream provider uses a mixture of Go v1 and v2 SDKs for Go, but none of these logs are made available when troubleshooting the Pulumi provider.

Example

Create a simple program using pulumi new aws-typescript.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

// Create an AWS resource (S3 Bucket)
const bucket = new aws.s3.Bucket("my-bucket");

// Export the name of the bucket
export const bucketName = bucket.id;

Run the following command taken from https://www.pulumi.com/docs/support/troubleshooting/

TF_LOG=TRACE pulumi up --logtostderr --logflow -v=10 2> out.txt

Observe the logs in out.txt.

pulumi-aws has performed some operations against the AWS clients, in particular:

• it checked credentials by issuing an awsbase.GetAwsConfig call https://github.com/pulumi/pulumi-aws/blob/master/provider/resources.go#L611 call
• it provisioned an S3 bucket

Expected: the associated AWS client operations have an entry in the out.txt log
Actual: no logs from the AWS clients

Output of pulumi about

CLI
Version 3.92.0
Go Version go1.21.3
Go Compiler gc

Plugins
NAME VERSION
aws 6.10.0
awsx 2.3.0
docker 4.5.0
docker 3.6.1
nodejs unknown

Host
OS darwin
Version 14.1.1
Arch x86_64

This project is written in nodejs: executable='/Users/t0yv0/bin/node' version='v18.18.2'

Current Stack: t0yv0/check-aws-calls/dev

TYPE URN
pulumi:pulumi:Stack urn:pulumi:dev::check-aws-calls::pulumi:pulumi:Stack::check-aws-calls-dev
pulumi:providers:aws urn:pulumi:dev::check-aws-calls::pulumi:providers:aws::default_6_10_0
aws:s3/bucket:Bucket urn:pulumi:dev::check-aws-calls::aws:s3/bucket:Bucket::my-bucket

Found no pending operations associated with dev

Backend
Name pulumi.com
URL https://app.pulumi.com/t0yv0
User t0yv0
Organizations t0yv0, pulumi
Token type personal

Dependencies:
NAME VERSION
@pulumi/awsx 2.3.0
@pulumi/pulumi 3.94.2
@types/node 18.18.13
@pulumi/aws 6.10.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

Related: #3047 upgrades the bridge that fixes a few issues with log propagation, however I've not confirmed yet if those changes completely fix the issue with AWS client logs specifically described here.

[t0yv0]

Reclassifying as engineering as most of the impact is on the engineering's team ability to diagnose issues.