Deploying new version of lambda function

[duarten]

❓ General Issue

The Question

I'm attempting to setup a CodeDeploy deployment group for a Lambda function. The CDK documentation for the Version class states:

If you want to deploy through CloudFormation and use aliases, you need to
add a new version (with a new name) to your Lambda every time you want
to deploy an update. An alias can then refer to the newly created Version.

This suggests that if I want to make a new CodeDeploy deployment, I should change the version name. For this end, I'm naming the versions using the sha1 of the latest Git commit that affects the Lambda's code or configuration. However, if I commit code that makes cosmetic changes to the configuration (i.e., to CDK code pertaining to the lambda), that will produce no differences in the CloudFormation template, and I will get the error A version for this Lambda function exists ( 1 ). Modify the function to create a new version.

This suggests that for a new version to be deployed I need to change the code or make semantic changes to the configuration. If this is the case, why require the version name to be unique between versions?

Code:

const lambdaName = basename(__dirname)

async function getRevision(dir: string): Promise<string> {
    const root = basename((await run("git rev-parse --show-toplevel")).line())
    const p = dir.substring(dir.lastIndexOf(root) + root.length + 1, dir.length)
    return (await run(`git rev-list --abbrev-commit -1 HEAD -- ${p}`)).line()
}

export async function myLambda(stack: Stack): Promise<Alias> {
    const f = new Function(stack, lambdaName, {
        runtime: Runtime.GO_1_X,
        handler: "main",
        code: Code.asset(join(__dirname, "lambda.zip")),
    })

    const alias = new Alias(stack, lambdaName + "-alias", {
        aliasName: "live",
        version: f.addVersion(await getRevision(__dirname)),
    })

    new LambdaDeploymentGroup(stack, lambdaName + "-deployment", {
        alias: alias,
        deploymentConfig: LambdaDeploymentConfig.LINEAR_10PERCENT_EVERY_10MINUTES,
    })

    return alias
}

Environment

• **CDK CLI Version: 1.18.0
• **Module Version: 1.18.0
• **OS: macOS Catalina
• **Language: Typescript

Other information

[skinny85]

Hello @duarten ,

the error you're getting:

"A version for this Lambda function exists ( 1 ). Modify the function to create a new version.".

I assume is coming up during deployment, in CloudFormation?

Thanks,
Adam

[duarten]

Hi @skinny85,

Yes, that's correct.

[skinny85]

Right. So the way we usually deal with that in the CDK is to have a new version for every synthesis - this way, the new version will always be created. Take a look at this example in our docs.

[duarten]

Correct me if I'm wrong, but in that case won't a synthesis always change from the previous one, even though the Lambda's code and configuration didn't? It also doesn't seem to solve my original problem, since I may deploy a stack which contains a lambda function whose code and configuration didn't change, but the CDK version did, thus leading to the "A version for this Lambda function exists ( 1 ). Modify the function to create a new version." error.

[skinny85]

Correct me if I'm wrong, but in that case won't a synthesis always change from the previous one, even though the Lambda's code and configuration didn't?

Yes. It's a tradeoff: either you remember to change it, or you accept the fact that you'll get a new version every deployment. I don't think it's too big of an issue to be honest.

It also doesn't seem to solve my original problem, since I may deploy a stack which contains a lambda function whose code and configuration didn't change, but the CDK version did, thus leading to the "A version for this Lambda function exists ( 1 ). Modify the function to create a new version." error.

No. It doesn't matter that the code/configuration did not change, there will be a new version, so you won't get that error.

[duarten]

No. It doesn't matter that the code/configuration did not change, there will be a new version, so you won't get that error.

Sorry, I don't follow. The issue is that I specified a new version, and I got that error. If I don't change the version, then the CloudFormation template is the same and CDK doesn't attempt a deployment. If I change the version, CDK will attempt a deployment, but will fail with that error. (I guess that's some other layer complaining that the Lambda's code and configuration have not changed.)

Example:

 15/24 | 12:23:34 AM | CREATE_IN_PROGRESS   | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version86f7530 (cognitopostconfirmationnewuserVersion86f753088FD94FA)
 15/24 | 12:23:35 AM | CREATE_IN_PROGRESS   | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version86f7530 (cognitopostconfirmationnewuserVersion86f753088FD94FA) Resource creation Initiated
 16/24 | 12:23:36 AM | CREATE_COMPLETE      | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version86f7530 (cognitopostconfirmationnewuserVersion86f753088FD94FA)

Note that version is 86f7530. Changing that to 7dad47a:

 0/4 | 12:29:33 AM | CREATE_IN_PROGRESS   | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version7dad47a (cognitopostconfirmationnewuserVersion7dad47aE2212F63)
 1/4 | 12:29:33 AM | CREATE_FAILED        | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version7dad47a (cognitopostconfirmationnewuserVersion7dad47aE2212F63) A version for this Lambda function exists ( 1 ). Modify the function to create a new version.
	new Version (/Users/duarten/code/umani/bazel-bin/deploy/app.sh.runfiles/npm/node_modules/@aws-cdk/aws-lambda/lib/lambda-version.js:28:25)
	\_ Function.addVersion (/Users/duarten/code/umani/bazel-bin/deploy/app.sh.runfiles/npm/node_modules/@aws-cdk/aws-lambda/lib/function.js:259:16)
	\_ UmaniLambda.<anonymous> (/Users/duarten/code/umani/bazel-bin/deploy/app.sh.runfiles/umani/constructs/umani-lambda.js:51:37)
	\_ Generator.next (<anonymous>)
	\_ fulfilled (/Users/duarten/code/umani/bazel-bin/deploy/app.sh.runfiles/umani/constructs/umani-lambda.js:4:58)
	\_ processTicksAndRejections (internal/process/task_queues.js:93:5)

[skinny85]

Hmm, you are correct. Apologies. I wonder whether Lambda added this validation recently...? I swear this used to work.

[duarten]

No worries :)

[skinny85]

So here's my research on the topic.

Currently, we recommend customers to do the following:

    const version = func.addVersion(new Date().toISOString()); // <==
    const alias = new lambda.Alias(this, 'LambdaAlias', {
      aliasName: 'Prod',
      version,
    });
      
    new codedeploy.LambdaDeploymentGroup(this, 'DeploymentGroup', {
      alias,
      deploymentConfig: codedeploy.LambdaDeploymentConfig.LINEAR_10PERCENT_EVERY_1MINUTE,
    });

However, it seems that this no longer works - if the Lambda itself is unchanged from the previous version, the new Version will fail creation.

This needs some pretty serious changes in our API:

• We should create a new API that substitutes Function.addVersion(), like Function.addHashedVersion() (name TBD of course). It should base the name of the version on the hash of the code property of the Function (either the asset, or the inline string) - if the code does not change, a new Version will not be created.
• We should probably deprecate the old addVersion() method, because it seems like it's very easy to shoot yourself in the foot using it.

[skinny85]

Actually, I figured out a workaround :) changing the description of the Function is enough to make the Version creation succeed, so this works:

        const func = new lambda.Function(this, 'lambdaName', {
            // whatever properties you need...
            description: `Generated on: ${new Date().toISOString()}`,
        });

        const version = func.addVersion(new Date().toISOString());

        const alias = new lambda.Alias(this, 'lambdaName-alias', {
            aliasName: 'live',
            version: version,
        });

        new codedeploy.LambdaDeploymentGroup(this, 'lambdaName-deployment', {
            alias: alias,
            deploymentConfig: codedeploy.LambdaDeploymentConfig.LINEAR_10PERCENT_EVERY_1MINUTE,
        });

[duarten]

Thanks for the workaround :)

[cbertozzi]

thanks for the working workaround ;) I'm facing the same issue and I tried many actions before. I think that a better way to solve this problem is to wait until the SAM CDK module is finally stable and released and then use the autoPublishAlias property that takes care for all the dynamic of recognize a new codebase and create (or not) a new version

[skinny85]

I'm glad it worked @cbertozzi :) But I think the addHashedVersion() method I talked about above is the way to go here (determine the name of the function's version based on the hash of its code property), not the SAM package.

[eladb]

+1 on addHashedVersion. I would just make name an optional argument for addVersion and default to the asset source hash. @nija-at definitely worth getting into our planning.

[hoegertn]

Additionally, I would love a feature to set the version to be retained on update. Sometimes I want the new version to be deployed but old versions to be retained for callers outside of my Stack. (e.g. Alexa Skills)

[kjpgit]

I'm just trying to get AWS Lambda provisioned concurrency and autoscaling working, and ran into this issue. I think this is ridiculous - I don't care about versions or aliases, but I have to use them to get provisioned concurrency, but versions and aliases are broken in the CDK. Great. I suggest you guys think of an easier way to "deploy a lambda with provisioned autoscaling" with CDK.

The workaround of "random description for lambda fn" seems to work for me, but man are things slow. A single function alias update takes 2 1/2 minutes with provisioned concurrency:

2/5 | 10:54:05 PM | UPDATE_IN_PROGRESS | AWS::Lambda::Alias | webhook-alias (webhookalias09ADCD64)
2/5 Currently in progress: webhookalias09ADCD64
3/5 | 10:56:37 PM | UPDATE_COMPLETE | AWS::Lambda::Alias | webhook-alias (webhookalias09ADCD64)

[aripalo]

There's also the case of Lambda@Edge, which I think relates to this issue:

CloudFront requires that a specific version of Lambda function is associated with the distribution.

Currently my options are:

1. Manually control the Lambda versions and only create a new version when I know the Lambda code has changes. This is somewhat annoying as one tends to forget to do that.

… or

2. Always create a new version of the Lambda function (even if there are no changes to the actual code). This in turn will always trigger a CloudFront update which takes 20 mimutes and this is even more annoying 😅

I'd like to see exactly that kind of code hash based solution @skinny85 suggested in #5334 (comment) as it should resolve this challenge with Lambda@Edge.

[iph]

+1 on addHashedVersion. I would just make name an optional argument for addVersion and default to the asset source hash. @nija-at definitely worth getting into our planning.

What about other information that belongs to specified lambda's version? The function version includes also information like lambda runtime and all of the function settings, including the environment variables. Changing the settings without modifying code is possible so I think using just asset source hash as a version name is not enough. Instead we should somehow use hash of whole "AWS::Lambda::Function" resource.

[From Lambda, just throwing opinion here]

Ran a few experiments, and this is partially dangerous and can lead to weird edge cases, though I do agree with the opinion that we should do this.

As an example, let's say you have the following environment variables in your function:

      Environment:
        Variables:
          hi: there
          hi2: there

and you "update" to:

      Environment:
        Variables:
          hi2: there
          hi: there

Lambda does not see this as an update to your function. It's an idempotent "update", because it's a map which has no explicit ordering. If you were to try to publish-version, and you were at version 1, Lambda would idempotently "publish" version 1 again.

Cloudformation does not interact kindly when you go from 1 -> 1. It will fail updates with Modify the function to create a new version. (this isn't Lambda). My guess is they are deeply tied in with the function version arn. Why? ¯\_(ツ)_/¯. I'll pop an issue in their queue, since that's a bit strange. Unfortunately, that could take some time to fix, so let's focus within the constraints of the problem presented.

If we were to naively hash the text within the function, Cloudformation will fail this update if you move from version{OLDHASH} to version{NEWHASH} in cloudformation logical id. So we will need to take extra steps to figure out what is idempotent in Lambda's eyes and produce a consistent hash with Lambda's expectations (e.g. sort a map and then produce a consistent hash off that sorted map, sort all function resource names before hashing). Tags and VPC configs are commutative, so would probably need to be sorted too. I think layer order is not commutative, but worth testing.

Sorry that it's frustratingly hard to get this right. We (Lambda+Cfn) need to do a bit better with these interactions and I'll start opening up communications to see what we can do --but due to backwards compatibility constraints of the past 5 years since this was originally shipped, the ship of "changing this" may have sailed.

[iph]

Examples of what I'm talking about above here: https://github.com/iph/lambda-experiments/tree/master/versioning-updates-cfn

Specifically experiment 3 in the README.

[michaelfecher]

Facing the same issue in combination with CDK pipelines.
Are there any updates about a non-workaround solution to deploy each and every time?
More critically, the .addVersion method is deprecated.
In general, do you recommend using Aliases and this versioning mechanism?

[dudutwizer]

Facing the same issue with SAM... any ideas for workarounds?

[adrian-baker]

A single function alias update takes 2 1/2 minutes with provisioned concurrency:

5-6 minutes for mine . Slower than a rolling deploy of a small ECS Fargate cluster : (

[iancullinane]

I am using a lambda.DockerImageFunction and it seems like existing workarounds do not help.

[mikoz93]

Is there any fix to this? Creating and deploying new versions on each cdk deploy is a massive waste of resource.

[skinny85]

@mikoz93 this should be fixed with hotswap deployments.

[ajhool]

When using hotswap deployments along with provisioned concurrency and aliases, I'm still seeing the same delay. The Alias continues to weight the old version at 100% and the new version at 0% for several minutes. I've also tried adding an AllAtOnce deployment group but that didn't work either. The deployment group appeared to have no effect on the Alias deployment and version cutover

    new LambdaDeploymentGroup(this, "DeploymentGroup", {
      alias,
      deploymentConfig: LambdaDeploymentConfig.ALL_AT_ONCE,
    });

When using hotswap without aliases, the hotswap switch is instantaneous.

[skinny85]

@ajhool I think that’s because you’re using a CodeDeploy deployment group. Try removing it, and the Alias should be updated immediately.

[jtaub]

I know this is closed, but it's hard to tell if this issue is really resolved or not. Is it still necessary to append a timestamp to the description? As far as I can tell, it is. But if not, in what version of the CDK is this issue fixed?

[skinny85]

@jtaub it should be resolved, unless you put the Alias inside a CodeDeploy DeploymentGroup.

[juber-mulani]

@skinny85 i am still facing same issue. until few days back its working fine, when we added lambda auto scaling to alias. it has started failing with same error. now my cdk stack few lambdas, how can we handle this? even there are no changes to other lambdas it shows A version for this Lambda function exists ( 20 ). Modify the function to create a new version.

i cant keep changing description for other lambdas or anything that try to deploy lambda every time. what is the solution to this? wondering its working till Nov, 15th. Is there any configuration that we need to set?

[skinny85]

i cant keep changing description for other lambdas or anything that try to deploy lambda every time.

Why?

[juber-mulani]

@skinny85 What I am trying to say is that previously, if there were no changes in a Lambda function with Alias, AWS CDK would skip deploying that Lambda. I have 20 stacks, each with dozens of Lambdas, and my pipeline runs on a daily schedule. This setup used to work fine—if there were no changes to a Lambda, CDK would skip it.

However, now CDK seems to expect every Lambda with an alias to have changes and deploy a new version, even if there’s no actual change.

I have read few suggestions in comments to this thread like updating the description on every deployment to force a change, but that’s not practical in my case. Doing so would significantly increase compute utilisation in my pipeline, pushing it to its limits.

What changed recently to cause this behaviour, and how can I make my pipeline work as it did before? Is there a reliable solution to prevent unnecessary Lambda deployments while keeping the pipeline efficient?

[skinny85]

Doing so would significantly increase compute utilisation in my pipeline, pushing it to its limits.

Can you explain this? What would increase here? This is just a CloudFormation deployment, no?

[juber-mulani]

@skinny85 Exactly! This is a CloudFormation deployment executed via AWS CodeCatalyst workflows (CI/CD pipeline). For example, if I have 10 Lambdas with no changes, CDK skipping those would make my CodeCatalyst workflow run faster, saving time. This is because the workflow waits for the current CloudFormation deployment to complete, and that waiting time counts toward the total execution time.

However, if I force updates to every Lambda, it will unnecessarily increase the total execution time of the CodeCatalyst workflow.

[skinny85]

In that case, I think you're only option is top update the description of the changes Lambdas manually, unfortunately 😕.

[juber-mulani]

@skinny85 😢

I am wondering—this was working as expected just a few days ago. Why isn’t it working now? Did something change?

[skinny85]

I have no idea. Did you update your CDK version?

According to the comments on this issue, it never worked, so I have no idea how it worked for you without knowing details about your setup, and its history.

[juber-mulani]

@skinny85 looks the bug is introduced with new version of cdk.