(aws-appsync): Custom Resource to Perform Manual Merge of a Source API And Signal Cloudformation Based On Merge Operation Result

[ndejaco2]

Describe the feature

AppSync recently launched Merged APIs: https://aws.amazon.com/blogs/mobile/introducing-merged-apis-on-aws-appsync/.
CDK support has been added to support Merged APIs with the following PRs:

1. feat(appsync): merged APIs #26895
2. feat(appsync): Standalone L2 construct for SourceApiAssociation #27121

The main resource for Merged APIs is the CfnSourceApiAssociation which links a source API with a merged API. When creating this resource, either through the GraphqlApi construct or the standalone SourceApiAssociation construct, you need to specify a merge type. The default type in CDK is AUTO_MERGE. AUTO_MERGE ensures that any change that is made on a source API is propagated to the Merged API endpoint in the background by the AppSync service itself. However, since this happens outside of Cloudformation, if there is a failure due to a merge conflict, the Cloudformation stack update is not aware of it and will not rollback automatically. Developers may want to ensure that a merge succeeds prior to the Cloudformation stack update succeeding.

Use Case

This feature allows the source API owner to ensure that when merge conflicts are introduced, the Cloudformation stack update would fail and cause a rollback. Otherwise, if the merge is successful, the team knows that the updates were successfully applied to the merged API.

Proposed Solution

I believe we can support this use case with a custom resource in the appsync CDK library. The custom resource would have a lambda function that submits the merge of the source API manually and a Lambda function that handles polling to check when the merge operation has finished. When the operation has finished, it will signal Cloudformation to succeed or fail based on the status of the merge operation.

Some key considerations:

• The custom resource needs to dependOn the source API and all children of the source API such as datasources, resolvers and functions. Otherwise, it might merge the API metadata prior to it being updated again.
• Ideally, we would only invoke the custom resource when children of the source API have been updated / added / deleted. Otherwise, if there is no good way to determine whether something has changed and trigger merge of the source api, we might have to settle for allowing no-ops. If the community has some recommendations on the best way to implement this that would be great!
• The Provider framework is ideal for this use case because it allows us to submit the merge and poll separately in a different lambda function.
• Ideally, we would want to have a single Lambda function per stack rather than creating a separate instance of the Lambda per source api association. While there would be a single Lambda function per stack, there would be a CustomResource per source api association.
• I believe this could be functionality that is added within the ISourceApiAssociation interface and available through a function like addCustomSourceApiMergeOperation() so that it is an opt in feature for the user.

Below is a snippet of code for the POC I have built for this feature:

/**
   * Creates a custom resource which will submit a schema merge of the source api each time the stack is deployed.
   * The custom resource will wait for the merge to finish and fail the update if the merge operation has any failure.
   */
  public addCustomSourceApiMergeOperation() {

    // Use NODEJS_LATEST so that it bundles the most recent aws sdk which has the Merged APIs operations.
    const schemaMergeLambda = new NodejsFunction(this, 'MergeSourceApiSchemaLambda', {
      runtime: Runtime.NODEJS_LATEST,
      entry: path.join(__dirname, 'mergeSourceApiSchemaHandler', 'index.ts'),
      handler: 'onEvent',
      timeout: Duration.minutes(2),
    });

    schemaMergeLambda.addToRolePolicy(new PolicyStatement({
      effect: Effect.ALLOW,
      resources: [this.associationArn],
      actions: ['appsync:StartSchemaMerge'],
    }));

    const sourceApiStabilizationLambda = new NodejsFunction(this, 'PollSourceApiMergeLambda', {
      runtime: Runtime.NODEJS_LATEST,
      entry: path.join(__dirname, 'mergeSourceApiSchemaHandler', 'index.ts'),
      handler: 'isComplete',
      timeout: Duration.minutes(2),
    });

    sourceApiStabilizationLambda.addToRolePolicy(new PolicyStatement({
      effect: Effect.ALLOW,
      resources: [this.associationArn],
      actions: ['appsync:GetSourceApiAssociation'],
    }));

    const provider = new Provider(this, 'SourceApiSchemaMergeOperationProvider', {
      onEventHandler: schemaMergeLambda,
      isCompleteHandler: sourceApiStabilizationLambda,
    });

    // The goal is for this custom resource to run once per Cloudformation stack update.
    // We force this by generating a property that is a random uuid each time cdk is synthesized.
    // This will force creation of a new CustomResource rather than recognizing it as existing, which will trigger the merge.
    // Ideally, we could figure out a better way to avoid the potential no-ops. 
    const customResource = new CustomResource(this, 'SourceApiSchemaMergeOperationCustomResource', {
      serviceToken: provider.serviceToken,
      resourceType: 'Custom::AppSyncSourceApiMergeOperation',
      properties: {
        associationId: this.associationId,
        mergedApiIdentifier: this.mergedApi.arn,
        sourceApiIdentifier: this.sourceApi.arn,
        alwaysUpdate: randomUUID(),
      },
    });

    // If a reference to the source API exists,
    // add a dependency on all children of the source api in order to ensure that this resource is created at the end.
    if (this.sourceApi) {
      this.sourceApi.node.children.forEach((child) => {
        if (CfnResource.isCfnResource(child)) {
          customResource.node.addDependency(child);
        }

        if (Construct.isConstruct(child) && child.node.defaultChild && CfnResource.isCfnResource(child.node.defaultChild)) {
          customResource.node.addDependency(child.node.defaultChild);
        }
      });
    }
}

Implementation for the Lambda handlers:

// eslint-disable-next-line import/no-extraneous-dependencies
import {
  StartSchemaMergeCommand,
  AppSyncClient,
  SourceApiAssociationStatus,
  GetSourceApiAssociationCommand,
  GetSourceApiAssociationCommandInput,
} from '@aws-sdk/client-appsync';

const appSyncClient = new AppSyncClient();

type SchemaMergeResult = {
  associationId?: string,
  mergedApiIdentifier?: string,
  PhysicalResourceId: string,
  sourceApiAssociationStatus?: SourceApiAssociationStatus,
};

type IsCompleteResult = {
  IsComplete: boolean,
  Data?: Object
}

export async function onEvent(event: AWSLambda.CloudFormationCustomResourceEvent): Promise<SchemaMergeResult> {
  const params = {
    associationId: event.ResourceProperties.associationId,
    mergedApiIdentifier: event.ResourceProperties.mergedApiIdentifier,
  };

  switch (event.RequestType) {
    case 'Update':
    case 'Create':
      return performSchemaMerge(params);
    case 'Delete':
    default:
      return {
        ...params,
        PhysicalResourceId: params.associationId,
      };
  }
}

export async function isComplete(event: AWSLambda.CloudFormationCustomResourceEvent): Promise<IsCompleteResult> {
  const params = {
    associationId: event.ResourceProperties.associationId,
    mergedApiIdentifier: event.ResourceProperties.mergedApiIdentifier,
  };

  return getSchemaMergeStatus(params);
}

async function performSchemaMerge(params: any): Promise<SchemaMergeResult> {
  const command = new StartSchemaMergeCommand(params);

  try {
    const response = await appSyncClient.send(command);
    switch (response.sourceApiAssociationStatus) {
      case SourceApiAssociationStatus.MERGE_SCHEDULED:
      case SourceApiAssociationStatus.MERGE_IN_PROGRESS:
      case SourceApiAssociationStatus.MERGE_SUCCESS:
        break;
      default:
        throw new Error('Unexpected status after starting schema merge:' + response.sourceApiAssociationStatus);
    }

    return {
      ...params,
      PhysicalResourceId: params.associationId,
      sourceApiAssociationStatus: response.sourceApiAssociationStatus,
    };
  } catch (error: any) {

    // eslint-disable-next-line no-console
    console.error('An error occurred submitting the schema merge', error);
    throw error;
  }
}

async function getSchemaMergeStatus(params: GetSourceApiAssociationCommandInput): Promise<IsCompleteResult> {
  const command = new GetSourceApiAssociationCommand(params);
  var response;

  try {
    response = await appSyncClient.send(command);
  } catch (error: any) {
    // eslint-disable-next-line no-console
    console.error('Error starting the schema merge operation', error);
    throw error;
  }

  if (!response.sourceApiAssociation) {
    throw new Error(`SourceApiAssociation ${params.associationId} not found.`);
  }

  switch (response.sourceApiAssociation.sourceApiAssociationStatus) {
    case SourceApiAssociationStatus.MERGE_SCHEDULED:
    case SourceApiAssociationStatus.MERGE_IN_PROGRESS:
    case SourceApiAssociationStatus.DELETION_SCHEDULED:
    case SourceApiAssociationStatus.DELETION_IN_PROGRESS:
      return {
        IsComplete: false,
      };

    case SourceApiAssociationStatus.MERGE_SUCCESS:
      return {
        IsComplete: true,
        Data: {
          ...params,
          sourceApiAssociationStatus: response.sourceApiAssociation.sourceApiAssociationStatus,
          sourceApiAssociationStatusDetail: response.sourceApiAssociation.sourceApiAssociationStatusDetail,
          lastSuccessfulMergeDate: response.sourceApiAssociation.lastSuccessfulMergeDate?.toString(),
        },
      };

    case SourceApiAssociationStatus.MERGE_FAILED:
    case SourceApiAssociationStatus.DELETION_FAILED:
    case SourceApiAssociationStatus.AUTO_MERGE_SCHEDULE_FAILED:
      throw new Error(`Source API Association: ${response.sourceApiAssociation.associationArn} failed to merge with status: `
        + `${response.sourceApiAssociation.sourceApiAssociationStatus} and message: ${response.sourceApiAssociation.sourceApiAssociationStatusDetail}`);

    default:
      throw new Error(`Unexpected source api association status: ${response.sourceApiAssociation.sourceApiAssociationStatus}`);
  }
}

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

v2.96.2

Environment details (OS name and version, etc.)

macOS 13.4

[peterwoodworth]

Thanks for the request and PR

[ashishdhingra]

PR #27121 merged. Closing this issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.