Path resolution for APIGW with custom domain and base mapping using proxy Lambda integration

[dnutels]

I am pretty positive that the bug is on APIGW side of things, but I feel like this needs to be documented somehow.

Setup

• APIGW proxy-integrated with Lambda via ANY method
• Custom domain api.some-domain.com is defined on top of APIGW(using APIGW console and not manually via Route53)
• Lambda exposes a GET /healthz endpoint that returns a 'OK' string
• APIGW is deployed to v1 stage, so that the URL to invoke the /healthz endpoint above is https://randomstring-execute-api-<region>.amazonaws.com/v1/healthz

Explanation

When custom domain api.some-domain.com is defined using the default base path mapping (which is /) to v1 stage, then both the random and custom domain URLs work:

• https://api.some-domain.com/healthz
• https://randomstring-execute-api-<region>.amazonaws.com/v1/healthz

which is the expected result.

However, when a non-default base path mapping is used (for example /custom), then:

https://randomstring-execute-api-<region>.amazonaws.com/v1/healthz

continues to work (as it should), while neither of the following works (the first one should have worked):

• https://api.some-domain.com/healthz
• https://api.some-domain.com/online/healthz
• https://api.some-domain.com/online/v1/healthz

Any other combination of the above doesn't work either.

Reason

After contacting AWS support and doing some testing, the reason is a difference in path resolution in two cases.

1. When a default / base path mapping is used, the Lambda handler is passed the following event structure (abridged to contain only relevant parts for clarity):

{ 
    resource: '/{proxy+}',
    path: '/healthz',
    httpMethod: 'GET',
    ...
    pathParameters: { proxy: 'healthz' },
    stageVariables: { alias: 'v1' },
    ...
}

2. When a non-default /custom base path mapping is used, the following event structure is passed:

{ 
    resource: '/{proxy+}',
    path: '/custom/healthz',
    httpMethod: 'GET',
    ...
    pathParameters: { proxy: 'healthz' },
    stageVariables: { alias: 'v1' },
    ...
}

In aws-serverless-express code there is event handling:

function getPathWithQueryStringParams (event) {
  return url.format({ pathname: event.path, query: event.queryStringParameters })
}

that refers to event.path, as it should, of course.

Of course, in the above case the following code would result in Lambda timeout, as there is no /custom/healthz endpoint defined (if only accidentally).

(Non)Resolution

Based on a suggestion from AWS support, we should use event.pathParameters.proxy instead of event.path, which is consistent in all cases.

This is, of course, absolutely incorrect and constitutes a bug in AWS APIGW resolution mechanism that breaks the Lambda encapsulation. The /custom path should have never been propagated beyond APIGW.

Suggestions and such

On a practical level, this issue means that we can't use custom base path mapping, as aws-serverless-express doesn't provide any means (since the function in question is in closure and as such can't be overridden) to "hack" around this, without altering the Lambda's code.

It is also clear that AWS APIGE won't fix this any time soon or ever.

What can you guys suggest?

[brettstack]

Please see #86 for workarounds. We will fix this in the library in the future.

[kwhitejr]

@dnutels Thank you so much for this write up. I've been pulling my hair out over this for a week and finally found this post.

🙏 thank you

Are you able to share the workaround you decided upon?

[dnutels]

@kwhitejr

We eventually abandoned the idea (which was to have a single consolidating APIGW for a set of similar deployment/scale-wise services, implemented as lambdas).

Right now we have an APIGW per Lambda, basically, with aliases (v1, v2, v3) pointing to different lambda aliases (alias-v1, alias-v2.2. etc.).

It wasn't an operational overhead for us, since it's in Terraform anyway, but you know... less moving parts, less money -- would have been nice.

Alas.