Empty json object is returned randomly with a content-type: application/json instead of image

[gcarrion-gfrmedia]

We're using this AWS Solution for Serverless Image Handler that uses AWS Lambda for resizing images (https://aws.amazon.com/solutions/serverless-image-handler/)

We've encountered some random issue where the images are not generated and instead an empty json object is returned with a content-type: application/json and status code 200.

The problem is that the CloudFront distribution then caches this response (since it was 200) and this image is never showed again if that POP location serves the file. However, if a different POP location is used (ie: by connecting via a VPN to a different country), the image might be generated correctly this other time when it runs on the Lambda origin again.

I have some sample requests here that I want to share.

Sample URL: https://d1tv77les5f1m1.cloudfront.net/eyJidWNrZXQiOiJzaG9wcHItYXNzZXRzIiwia2V5IjoidGVuYW50L3Nob3AtcHIvc2hvcHBlci9wYWdlcy8wMy0yNS0yMDIwL2MxZWM5YWU0NTY3NzUzZmNkN2Q5NDA1NjkyZGI0MjRmLVNpbiB0aV90dWxvIDUtMS5wbmciLCJlZGl0cyI6eyJleHRyYWN0Ijp7IndpZHRoIjoxOTc0LCJoZWlnaHQiOjEwODgsImxlZnQiOjU0OTksInRvcCI6MTE4M30sInJlc2l6ZSI6eyJ3aWR0aCI6MTAwLCJoZWlnaHQiOjEwMCwiZml0IjoiY292ZXIifSwianBlZyI6eyJxdWFsaXR5Ijo5MH0sInRvRm9ybWF0IjoianBlZyJ9fQ==

Response headers:
Name Value
Status HTTP/1.1 200
status 200
date Tue, 31 Mar 2020 14:59:46 GMT
content-type application/json
content-length 2
x-amzn-requestid d64a71c9-afef-4701-bc69-d278a15152ee
access-control-allow-origin *
access-control-allow-headers Content-Type, Authorization
x-amz-apigw-id KQit9FYvIAMFYKQ=
access-control-allow-methods GET
x-amzn-trace-id Root=1-5e834b26-b1c8ea3cb9ae45d8355ed648;Sampled=0
access-control-allow-credentials true
x-cache Hit from cloudfront
via 1.1 b03cdde00266d1a718dac82499df6b8f.cloudfront.net (CloudFront)
x-amz-cf-pop MIA3-C3
x-amz-cf-id Svg846NYHSYxPcNc629IEY2sEGdbQT15Oop2aqBHu0g5q-8ZvNHz_A==
age 4027

Whenever I requested that URL it always returned the empty json object and it continue serving that, because it was cached on POP.

Then I connected via NordVPN to a server in Italy and tried reloading the image, this is the response headers for that same URL:

Name Value
Status HTTP/1.1 200
status 304
date Tue, 31 Mar 2020 15:02:35 GMT
x-amzn-requestid d2dafe96-a992-4f6c-871a-38d74276499c
access-control-allow-origin *
access-control-allow-headers Content-Type, Authorization
x-amz-apigw-id KQs5_HgLoAMFyfw=
cache-control max-age=31536000,public
access-control-allow-methods GET
x-amzn-trace-id Root=1-5e835b73-238504bc2f9cf534299f9b58;Sampled=0
access-control-allow-credentials true
x-cache Hit from cloudfront
via 1.1 95e275e2550c87aeaa644f1f37b346e0.cloudfront.net (CloudFront)
x-amz-cf-pop LHR50-C1
x-amz-cf-id dp2LM-PvctWsk0Bcs0W3p3iohtBHb5dMENIC581URfSGir9E8C1rlg==
age 22
content-type image/jpeg
content-length 3278
last-modified Wed, 25 Mar 2020 21:57:12 GMT

This time the image was returned fine while reloading connecting to my closest POP that image is not returned. In order to work again I need to invalidate the object on CloudFront and try again to see if fetching again from the Lambda origin generates the image.

After doing that, the image was generated, here are the headers:

Name Value
Status HTTP/1.1 200
status 304
date Tue, 31 Mar 2020 19:30:49 GMT
x-amzn-requestid b7fe3128-7f4f-4db0-a714-25199e53ee56
access-control-allow-origin *
access-control-allow-headers Content-Type, Authorization
x-amz-apigw-id KRTwcGnAoAMFh3w=
cache-control max-age=31536000,public
access-control-allow-methods GET
x-amzn-trace-id Root=1-5e83999c-b3c9aafd014255c292879c02;Sampled=0
access-control-allow-credentials true
x-cache Hit from cloudfront
via 1.1 b3879c23ec3b402566708cfe9d0ddc19.cloudfront.net (CloudFront)
x-amz-cf-pop MIA3-C3
x-amz-cf-id MjH1wTuvGWgqpH2mKg9cgDtPHMM7qYKcN4J7IiPH-zbb-KHqLMPLEg==
age 203
cf-ray 57cc7cb3ccabe968-MIA
content-type image/jpeg
content-length 3278
last-modified Wed, 25 Mar 2020 21:57:12 GMT

I think a possible fix would be that the function doesn't return a status code of 2XX if the response is something different than an image type response. For example, if an error or something occurs while processing an image and it returns an empty json and content-type: application/json, have a response code of 5XX or 4XX, this way the response is not cached on the CDN.

Any suggestions?

[djavier]

👍

[beomseoklee]

@gcarrion-gfrmedia Can you check the CloudWatch logs?

[gcarrion-gfrmedia]

Hi @beomseoklee! Thanks for the reply.

I've looked around CloudWatch and this is what I've found:

I can't find a specific detailed log for those sample requests above. Is there a way to filter these logs by x-amzn-requestid or any other header from those requests?

Thanks

[beomseoklee]

Thanks for your information @gcarrion-gfrmedia
However, it's impossible to say what would go wrong without CloudWatch log, but I'll try to reproduce the issue.

[ppkinev]

Is there any update on this issue? it still happens randomly from time to time.

[beomseoklee]

@ppkinev We haven't fixed this one because we don't have detail information and how to reproduce this one. If you can provide how I can reproduce this one consistently (e.g. certain image always fails and so on), it would be helpful to fix the issue.

[ppkinev]

@beomseoklee Found one of the reasons that happen, not sure if it's the only one.
In my case, I was taking image width and height from data-attributes so they were strings.

So if using:

{ 
  bucket: "kinonation-imagebucket-1l7ud25lkgnrz",
  key: "image_id_4534_4e974f42f1b6c188443d505a91fb66af.jpg",
  edits: {
    resize: {
      width: "150",
      height: "200",
      fit: "cover"
    },
    toFormat: "jpeg",
    jpeg: { quality: 60 }
  }
}

https://dblir06cn1w4c.cloudfront.net/eyJidWNrZXQiOiJraW5vbmF0aW9uLWltYWdlYnVja2V0LTFsN3VkMjVsa2ducnoiLCJrZXkiOiJpbWFnZV9pZF80NTM0XzRlOTc0ZjQyZjFiNmMxODg0NDNkNTA1YTkxZmI2NmFmLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6IjE1MCIsImhlaWdodCI6IjIwMCIsImZpdCI6ImNvdmVyIn0sInRvRm9ybWF0IjoianBlZyIsImpwZWciOnsicXVhbGl0eSI6NjB9fX0=

Returns an empty {} as a response with 200 code.

If numbers are used everything works as expected. Would be nice to have 400 or 422 in that case.

[beomseoklee]

@ppkinev Thanks for data. We will consider your opinion in the next release.

[beomseoklee]

Thanks for waiting. We've released v5.1.0.

@ppkinev We fixed the issue you mentioned. @gcarrion-gfrmedia You will not going to see the empty json anymore. The solution will always throw an error if any error happens.