pulumi refresh shows diff in the code property of a Lambda function even though nothing has changed

[praneetloke]

Expected Behavior

No diff in the code property if the Lambda function body really hasn't changed.

Current Behavior

When I ran a pulumi refresh for a stack that has Lambda functions deployed, I saw a diff similar to this.

  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::aws-lambda::pulumi:pulumi:Stack::aws-lambda-dev]
            ~ aws:lambda/function:Function: (update)
                [id=zipTpsReports-616a7ed]
                [urn=urn:pulumi:dev::aws-lambda::aws:s3/bucket:Bucket$aws:s3:BucketEventSubscription$aws:lambda/function:Function::zipTpsReports]
                [provider=urn:pulumi:dev::aws-lambda::pulumi:providers:aws::default_3_26_1::8340567c-260d-44f2-bb91-2adb05011074]
                --outputs:--
              - code                        : archive(assets:7cb0b9e) {
              -     "__index.js": asset(text:48326e2) {
                        function __f1(__0, __1, __2, __3) {
                          return (function() {
                            with (__closure) {
                        
                        return function (thisArg, _arguments, P, generator) {
                            function adopt(value) { return value instanceof P ? value : new P(function (resolve) { resolve(value); }); }
                            return new (P || (P = Promise))(function (resolve, reject) {
                                function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
                                function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
                                function step(result) { result.done ? resolve(result.value) : adopt(result.value).then(fulfilled, rejected); }
                                step((generator = generator.apply(thisArg, _arguments || [])).next());
                            });
                        }
                        
                        () => require("aws-sdk")
                        
                        function __f3(__0) {
                          return (function() {
                            with (__closure) {
                        
                        return function /*constructor*/(value) {
                                this.value = value;
                            }
                        
                        function __f4(__0) {
                          return (function() {
                            with (__closure) {
                        
                        return function /*apply*/(func) {
                                throw new Error("'apply' is not allowed from inside a cloud-callback. Use 'get' to retrieve the value of this Output directly.");
                            }
                        
                        function /*get*/() {
                                return this.value;
                            }
                        
                        function __f0(__0) {
                          return (function() {
                            with (__closure) {
                        
                        return (e) => __awaiter(void 0, void 0, void 0, function* () {
                            const admZip = require("adm-zip");
                            const s3 = new aws.sdk.S3();
                            for (const rec of e.Records || []) {
                                const zip = new admZip();
                                const [buck, key] = [rec.s3.bucket.name, rec.s3.object.key];
                                console.log(`Zipping ${buck}/${key} into ${tpsZips.bucket.get()}/${key}.zip`);
                                const data = yield s3.getObject({ Bucket: buck, Key: key }).promise();
                                zip.addFile(key, data.Body);
                                yield s3.putObject({
                                    Bucket: tpsZips.bucket.get(),
                                    Key: `${key}.zip`,
                                    Body: zip.toBuffer(),
                                }).promise();
                            }
                        })
                        
                    }
                }
              + code                        : "/var/folders/kp/8lxqldss24d739m_js3xtxyr0000gn/T/pulumi-asset-7cb0b9e63b2a3c997fb763caf0aa04ae661f67341d3f4878fd47f494cd5a7dbd"
              ~ lastModified                : "2021-02-01T21:16:45.171+0000" => "2021-02-01T21:16:51.265+0000"

Steps to Reproduce

1. Create a stack with any Lambda function. I used the one from our examples repo.
2. Run pulumi up
3. Run pulumi refresh and see the details when the "confirmation to proceed" pops up

Context (Environment)

Note that I observed this with Node.js functions and have not tried the other Lambda runtimes.

Pat and Luke said this sounds like it is an "artifact of the rendering", if that's helpful in triaging this.

Node version: v14.15.3
Pulumi CLI version: v2.19.0

package.json

{
    "name": "aws-lambda",
    "devDependencies": {
        "@types/node": "^10.0.0"
    },
    "dependencies": {
        "@pulumi/aws": "^3.26.1",
        "@pulumi/awsx": "^0.22.0",
        "@pulumi/pulumi": "^2.0.0"
    }
}

[v-the-cmd]

I believe this is imminent given how Pulumi implements passing things to AWS provider.

Ref:

pulumi-aws/provider/resources.go

Lines 2165 to 2192 in c9a0980

	"aws_lambda_function": {
	Tok: awsResource(lambdaMod, "Function"),
	IDFields: []string{"function_name"},
	Fields: map[string]*tfbridge.SchemaInfo{
	"function_name": tfbridge.AutoName("name", 64, "-"),
	"role": {Type: awsTypeDefaultFile(awsMod, "ARN")},
	// Terraform accepts two sources for lambdas: a local filename or a S3 bucket/object. To bridge
	// with Pulumi's asset model, we will hijack the filename property. A Pulumi archive is passed in
	// its stead and we will turn around and emit the archive as a temp file that Terraform can read.
	// We also automatically populate the asset hash property as this is used in diffs/updates/etc.
	"filename": {
	Name: "code",
	Asset: &tfbridge.AssetTranslation{
	Kind: tfbridge.FileArchive,
	Format: resource.ZIPArchive,
	HashField: "source_code_hash",
	},
	},
	"runtime": {
	Type: "string",
	AltTypes: []tokens.Type{awsType(lambdaMod, "Runtime", "Runtime")},
	},
	"architectures": {
	MaxItemsOne: tfbridge.False(),
	Name: "architectures",
	},
	},
	},

That chunk would always transform whatever we feed into code attribute to a local ZIP archive upon refresh.

A potential workaround (applicable for lambda functions only I think) is to create the function from S3 object and not convert it to a zip file upon refresh. I would assume the first part already works (as one can specify s3_bucket and s3_key parameters), but then S3 object itself is prone to that refresh bug if managed by Pulumi.

[lukehoban]

This issue isn't unique to Lambda, it occurs for any Asset/Archive usage.

As noted above, #1521 is an example of the same issue with S3 BucketObject.

[marns93]

@lukehoban Any update for this? Any plans when this will be fixed?

[kjirapinyo-calibrate]

Any update on this pls? Still running into similar issues.

[t0yv0]

Just bumping into this myself, and indeed I believe the impact is not just cosmetic but this results in invalid Pulumi state especially if multiple boxes are cooperating on the same Pulumi statefile. I believe this can be addressed by fixing the bridge import logic to reconcile refreshes for assets and archives that are not physically changing.

pulumi/pulumi-terraform-bridge#1595

[t0yv0]

Checking up on this using aws-6.24.0:

$ pulumi refresh
No resources will be modified as part of this refresh; just your stack's state will be.
 details
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:pulumi-6235::slss-tps::pulumi:pulumi:Stack::slss-tps-pulumi-6235]
            ~ aws:lambda/function:Function: (update)
                [id=zipTpsReports-31ac5a7]
                [urn=urn:pulumi:pulumi-6235::slss-tps::aws:s3/bucket:Bucket$aws:s3:BucketEventSubscription$aws:lambda/function:Function::zipTpsReports]
                [provider=urn:pulumi:pulumi-6235::slss-tps::pulumi:providers:aws::default_6_24_0::491d748f-ccf1-4011-a54c-5e238dd84acc]
                --outputs:--
              ~ lastModified                : "2024-03-01T18:02:42.906+0000" => "2024-03-01T18:02:49.187+0000"
              + replacementSecurityGroupIds : []
            ~ aws:iam/role:Role: (update)
                [id=zipTpsReports-d8156e6]
                [urn=urn:pulumi:pulumi-6235::slss-tps::aws:s3/bucket:Bucket$aws:s3:BucketEventSubscription$aws:iam/role:Role::zipTpsReports]
                [provider=urn:pulumi:pulumi-6235::slss-tps::pulumi:providers:aws::default_6_24_0::491d748f-ccf1-4011-a54c-5e238dd84acc]
                --outputs:--
              ~ managedPolicyArns  : [
                  + [0]: "arn:aws:iam::aws:policy/AWSLambda_FullAccess"
                  + [1]: "arn:aws:iam::aws:policy/AmazonS3FullAccess"
                  + [2]: "arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess"
                  + [3]: "arn:aws:iam::aws:policy/AmazonKinesisFullAccess"
                  + [4]: "arn:aws:iam::aws:policy/AmazonCognitoPowerUser"
                  + [5]: "arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess"
                  + [6]: "arn:aws:iam::aws:policy/AmazonSQSFullAccess"
                  + [7]: "arn:aws:iam::aws:policy/CloudWatchEventsFullAccess"
                  + [8]: "arn:aws:iam::aws:policy/CloudWatchFullAccessV2"
                ]

$ pulumi up --yes  
Previewing update (pulumi-6235)

View in Browser (Ctrl+O): https://app.pulumi.com/anton-pulumi-corp/slss-tps/pulumi-6235/previews/3faab414-d27c-448a-acfe-93922ccfd2a3

     Type                 Name                  Plan     
     pulumi:pulumi:Stack  slss-tps-pulumi-6235           

Resources:
    17 unchanged

[t0yv0]

The refresh is still not empty but it is much more benign and the original issue is fixed I believe.

#2246 has a good explanation and discussion on why we still get a non-empty refresh here.

[t0yv0]

I think we should close the issue as stated should be fixed now, however there's a few possibilities for future work here. Specifically this fixes refresh of unchanged lambda. If the code of the lambda changes in the cloud it's another matter. @v-the-cmd comments hold I think - Assets and Archives are a "one way" mechanism and Pulumi import never creates them on import. We have #3376 to delve into this further someday and cross-compare with TF behavior.