(lambda): EFS and Lambda create cyclic reference when deployed in separate stacks

[israelg99]

What is the problem?

EFS (particularly AccessPoint) and Lambda create cyclic reference when deployed in separate stacks.

Very related: #5760, #10942, #11245.
All of which are P1. Check the reproducible code in those issues as well.

Reproduction Steps

Here is a working example:

export class TheEfsLambdaStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const vpc = new ec2.Vpc(this, 'Vpc', {
      maxAzs: 2, // Default is all AZs in the region
    });

    // Create a file system in EFS to store information
    const fs = new efs.FileSystem(this, 'FileSystem', {
      vpc,
      removalPolicy: cdk.RemovalPolicy.DESTROY
    });

    const accessPoint = fs.addAccessPoint('AccessPoint',{
      createAcl: {
        ownerGid: '1001',
        ownerUid: '1001',
        permissions: '750'
      },
      path:'/export/lambda',
      posixUser: {
        gid: '1001',
        uid: '1001'
      }
    });

    // This lambda function is given access to our EFS File System
    const efsLambda = new lambda.Function(this, 'efsLambdaFunction', {
      runtime: lambda.Runtime.PYTHON_3_8,
      code: lambda.Code.fromAsset('lambda-fns'), 
      handler: 'message_wall.lambda_handler',
      vpc: vpc,
      filesystem: lambda.FileSystem.fromEfsAccessPoint(accessPoint, '/mnt/msg')
    });

    // defines an API Gateway Http API resource backed by our "efsLambda" function.
    let api = new apigw.HttpApi(this, 'Endpoint', {
      defaultIntegration: new integrations.LambdaProxyIntegration({
        handler: efsLambda
      })
    });

   new cdk.CfnOutput(this, 'HTTP API Url', {
     value: api.url ?? 'Something went wrong with the deploy'
   });
  }
}

The code breaks if Function is deployed in separate stack and the AccessPoint is passed in props, with the following error:

/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/stack.ts:395
      throw new Error(`'${target.node.path}' depends on '${this.node.path}' (${cycle.join(', ')}). Adding this dependency (${reason}) would create a cyclic reference.`);
            ^
Error: 'Stack2' depends on 'Stack' ("Stack2/lambda/lambda/ServiceRole/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget1", "Stack2/lambda/lambda/ServiceRole/DefaultPolicy/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget1", "Stack2/lambda/lambda/SecurityGroup/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget1", "Stack2/lambda/lambda/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget1", "Stack2/lambda/lambda/ServiceRole/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget2", "Stack2/lambda/lambda/ServiceRole/DefaultPolicy/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget2", "Stack2/lambda/lambda/SecurityGroup/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget2", "Stack2/lambda/lambda/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget2", "Stack2/lambda/lambda/ServiceRole/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget3", "Stack2/lambda/lambda/ServiceRole/DefaultPolicy/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget3", "Stack2/lambda/lambda/SecurityGroup/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget3", "Stack2/lambda/lambda/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget3", "Stack2/lambda/lambda/Resource" depends on "Stack/efs/FileSystem/EfsSecurityGroup/from StackefsFileSystemEfsSecurityGroup2F9415D6:ALL PORTS", "Stack2/lambda/lambda/Resource" depends on "Stack/efs/FileSystem/EfsSecurityGroup/from StackfunctionSecurityGroupD8C55079:2049", "Stack2/lambda/lambda/Resource" depends on "Stack/efs/FileSystem/EfsSecurityGroup/from Stack2clipmodeldownloadSecurityGroupB2C2BEB1:2049"). Adding this dependency (Stack -> Stack2/lambda/lambda/SecurityGroup/Resource.GroupId) would create a cyclic reference.
    at Stack._addAssemblyDependency (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/stack.ts:395:13)
    at Object.addDependency (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/deps.ts:52:20)
    at Stack.addDependency (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/stack.ts:266:5)
    at resolveValue (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/private/refs.ts:100:12)
    at Object.resolveReferences (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/private/refs.ts:30:24)
    at Object.prepareApp (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/private/prepare-app.ts:30:3)
    at Object.synthesize (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/private/synthesis.ts:32:3)
    at App.synth (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/stage.ts:90:23)
    at Object.<anonymous> (/Users/julian/work/project/bin/infra.ts:134:5)
    at Module._compile (internal/modules/cjs/loader.js:1072:14)

For more code insight check similar issues: #5760, #10942, #11245.

I can add simple self-contained reproducible code but it should be simple enough to infer.

What did you expect to happen?

Should work. We can pass Vpc in props across stacks, why EFS and others fail? The error is confusing as well.

What actually happened?

Fails as described above.

CDK CLI Version

2.10

Framework Version

No response

Node.js Version

14 LTS

OS

Mac

Language

Typescript

Language Version

TS 4.x

Other information

No response

[israelg99]

Potential workaround is to not pass EFS as props, but get the EFS resource using FileSystem.fromFileSystemAttributes inside the lambda stack. Interestingly, passing VPC as props works well as usual. So this looks like an inconsistency in the interface.

[israelg99]

@LukvonStrom Sounds good, I will write a unit test for it

[corymhall]

It looks like the issue is with this code. We already have a dependency Lambda -> AccessPoint and this is trying to create the security group rules in the AccessPoint stack (AccessPoint -> Lambda)

aws-cdk/packages/@aws-cdk/aws-lambda/lib/function.ts

Lines 1007 to 1011 in bb8d6f6

	if (props.filesystem) {
	if (props.filesystem.config.connections) {
	props.filesystem.config.connections.allowDefaultPortFrom(this);
	}
	}

This should be updated to be something like this so that the security group rules are added in the lambda stack.

if (props.filesystem) {
  if (props.filesystem.config.connections) {
    this.connections.allowTo(props.filesystem.config.connections, props.filesystem.config.connections.defaultPort);
  }
}

[Annegies]

Hey, also running into this issue. Is there an update?

[ShivamJoker]

not sure even if this will be fixed in next year, it starts throwing error just passing env to the lambda

[fivepapertigers]

For those that are struggling for a workaround here, I hacked around this for my needs by using escape hatches (shoutout to @icj217 for the suggestion). Here's what my Python implementation looked like:

def repair_function_to_efs_security_groups(func: lambda_.Function, efs: efs.IFileSystem):
    """
    When an EFS filesystem is added to a Lambda Function (via the file_system= param)
    it automatically sets up networking access between the two by adding
    an ingress rule on the EFS security group. However, the ingress rule resource
    gets attached to whichever CDK Stack the EFS security group is defined on.
    If the Lambda Function is defined on a different stack, it then creates
    a circular dependency issue, where the EFS stack is dependent on the Lambda
    security group's ID and the Lambda stack is dependent on the EFS stack's file
    system object.

    To resolve this, we manually remove the ingress rule that gets automatically created
    and recreate it on the Lambda's stack instead.
    """

    # Collect IDs of all security groups attached to the function
    func_sec_groups = {
        func_sec_group.security_group_id
        for func_sec_group in func.connections.security_groups
    }
    # Iterate over the security groups attached to EFS
    for sec_group in efs.connections.security_groups:
        # Iterate over the security group's child nodes
        for child in sec_group.node.find_all():
            # If this is an ingress rule with a "source" equal to one of
            # the function's security groups
            if (
                isinstance(child, ec2.CfnSecurityGroupIngress) and
                child.source_security_group_id in func_sec_groups
            ):
                # Try to remove the node (raise an error if removal fails)
                node_id = child.node.id
                if not sec_group.node.try_remove_child(node_id):
                    raise RuntimeError(f'Could not remove child node: {node_id}')

    # Finally, configure the connection between the Function and the EFS drive
    # which will define the new ingress rule on the Lambda's stack instead.
    func.connections.allow_to_default_port(efs)

Adapt for your needs.

[moltar]

Thanks @fivepapertigers for this escape hatch!

Here's the same in TS:

    fileSystem.connections.securityGroups.forEach((fssg) => {
      fssg.node.findAll().forEach((child)=> {
        if (child instanceof CfnSecurityGroupIngress &&
          lambda.connections.securityGroups.some(({ securityGroupId }) => securityGroupId === child.sourceSecurityGroupId)) {
          fssg.node.tryRemoveChild(child.node.id);
        }
      });
    });

    lambda.connections.allowToDefaultPort(fileSystem);

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.