0.76.0: SNS -> publish to subscribed SQS queue, "Records" assumption leads to lost message, empty array.

[bitsofinfo]

using 0.76.0

My code is sending a message to SNS like this:

    sns = boto3.client(service_name='sns',
                                endpoint_url=http://localhost:4002)

    message = {
        "my_event": {
            "x": "something"
        }
    }

    # publish it
    response = sns.publish(
        TopicArn=event_topic_id,
        Subject="test-my-event",
        Message=json.dumps(message),
        MessageAttributes={
            "data_ingress_type": {
                "DataType": "String",
                "StringValue": "test"
            }
        })  

I have an SQS queue (in serverless-offline-sqs) that is subscribed to the event topic the code is publishing to (which is properly setup and I see the queue existing in elasticmq)

  serverless-offline-sns:
    port: 9324
    debug: true
    subscriptions:
      - topic: ${self:provider.environment.INGRESS_EVENT_TOPIC_NAME}
        queue: http://localhost:9324/queue/my-events

When I startup the app, I see the queue being successfully subscribed to my serverless-offline-sns created topic as expected.

When the code runs to send the SNS message, it is never received.

Tracking things down in the code I see that sns-server.publish() gets called and for each subscription it creates a new SNS message that ends up looking like this which wraps the message above that I published via the boto client:

serverless-offline-sns/src/sns-server.ts

Line 351 in 6f6cde4

event = JSON.stringify(

{
  "SignatureVersion": "1",
  "Timestamp": "2021-08-01T23:07:52.737Z",
  "Signature": "EXAMPLE",
  "SigningCertUrl": "EXAMPLE",
  "MessageId": "c0f69f87-ce52-483d-ac74-c61f579fee28",
  "Message": "{\"my_event\": { \"x\": \"something\"}}",
  "MessageAttributes": {
    "data_ingress_type": { "Type": "String", "Value": "test" }
  },
  "Type": "Notification",
  "UnsubscribeUrl": "EXAMPLE",
  "TopicArn": "arn:aws:sns:us-west-2:123456789012:my-events",
  "Subject": "test-my-event"
}   

The sns-server.publish() method then calls sns-server.publishSqs(). In particular this line erroneously assumes that that event contains a Records property array... which it doesn't... then yields an empty array [] ultimately resulting in my message being lost and never sent:

serverless-offline-sns/src/sns-server.ts

Line 310 in 6f6cde4

const records = JSON.parse(event).Records ?? [];

Seems like the sns-server.publishSqs() event is expecting the passed event to be in the format of the helpers.createSnsLambdaEvent() structure instead?

serverless-offline-sns/src/helpers.ts

Line 75 in 6f6cde4

export function createSnsLambdaEvent(

Original issue comment:
#88 (comment)

[erik-am]

For me it works if I specify it like this. (Using your example)

  serverless-offline-sns:
    port: 9324
    debug: true
    subscriptions:
      - topic:
          topicName: ${self:provider.environment.INGRESS_EVENT_TOPIC_NAME}
          rawMessageDelivery: 'true'
        queue: http://localhost:9324/queue/my-events    

Note the quotes in rawMessageDelivery: 'true'! It doesn't work as a boolean, it should be a string.

Then the line of code that you changed in your PR is not even touched, because it will take this branch:

serverless-offline-sns/src/sns-server.ts

Lines 348 to 349 in 5a29200

	if (isRaw) {
	event = message;

[rcambrj]

I made a similar fix to yours @bitsofinfo, but preserving the event shape for publishHttp. I also based from 0.74.0 due to #141 because later versions seem to be break on inclusion into typescript serverless templates.

This alone causes #143 for me. But instead of using the fix proposed by that same PR #143, I just disabled the AWS::SNS::Subscription resources based on an environment variable present only when using serverless-offline-*. This way, serverless-offline-sns does not create any subscriptions

With the serverless-offline-sns subscriptions disabled between SNS & SQS, I installed and configured serverless-offline-sqs+elasticmq and pointed serverless-offline-sns to elasticmq as described in #88.

That was quite a journey. Hope this helps some others.

[ormu5]

I also forked from a previous tag to get this working and am maintaining the code at https://github.com/ormu5/serverless-offline-sns.