Spaces in SQS message cause signature failure

[josheschulz]

Python 3.8
aiosqs==1.0.3

from aiosqs import SQSClient
import asyncio

async def send_message():
    client = SQSClient(
        aws_access_key_id="****",
        aws_secret_access_key="****",
        region_name="us-east-1",
        host="sqs.us-east-1.amazonaws.com",
    )
    response = await client.send_message(
        queue_url="<QUEUE_URL>",
        message_body="message    with a space",
        delay_seconds=0,
    )
    print(response)
    await client.close()

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(send_message())
    loop.close()

Results in:

SQS API error: status_code=403, body=<?xml version="1.0"?><ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Type>Sender</Type><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'GET
/
Action=SendMessage&amp;DelaySeconds=0&amp;MessageBody=message%20%20with%20a%20space&amp;QueueUrl=https%3A%2F%2Fsqs.us-east-1.amazonaws.com%2F763215857860%2Fcontent-repository-processing-queue-prod&amp;Version=2012-11-05
host:sqs.us-east-1.amazonaws.com
x-amz-date:20231205T232537Z

host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20231205T232537Z
20231205/us-east-1/sqs/aws4_request
4ea765b1b277df9a9b929ee3383f75060631e8f75fb38b9c8b8b05a84c009474'
</Message><Detail/></Error><RequestId>f679cf26-effe-5e59-81ca-92cf5c4c713b</RequestId></ErrorResponse>
Traceback (most recent call last):
  File "test.py", line 21, in <module>
    loop.run_until_complete(send_message())
  File "/Users/jschulz/.pyenv/versions/3.8.0/lib/python3.8/asyncio/base_events.py", line 608, in run_until_complete
    return future.result()
  File "test.py", line 11, in send_message
    response = await client.send_message(
  File "/Users/jschulz/.pyenv/versions/buyerbase/lib/python3.8/site-packages/aiosqs/client.py", line 157, in send_message
    return await self.request(params=params)
  File "/Users/jschulz/.pyenv/versions/buyerbase/lib/python3.8/site-packages/aiosqs/client.py", line 139, in request
    raise SQSClientBaseError
aiosqs.exceptions.SQSClientBaseError
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10ee36fd0>

If you change the message to "message__with_a_space" and it posts correctly.

This is primarily a problem if you want to do something like:
message_body=json.dumps(some_python_object)

But you can work around that (assuming no spaces in your json) by doing the following:
json.dumps(some_python_object,separators=(',', ':'))

[xZanon]

Yep,

I can confirm the problem with spaces. Currently I am using base64 encode/decode for body of the messages, and using workaround with json.dumps(some_python_object,separators=(',', ':')) but will look into finding the problem. maybe a urllib.parse.quote(msg) in a proper place before signing would solve the issue .

On the other side this library is fantastic ! I was able to process and send 2 mil msg with average speed 5964 m/s
Great job @d3QUone

Regards,
xZanon

[d3QUone]

@josheschulz @xZanon thank you for your feedback! I'll test it myself and try to find a better solution.

In my internal project I use a serialized JSON object json.dumps(...) as a message body, and actually it worked without problems.

[d3QUone]

@josheschulz I think it's a requirement from Amazon:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html#API_SendMessage_RequestSyntax

A message can include only XML, JSON, and unformatted text. The following Unicode characters are allowed:

So they actually expect string to be a serialized JSON or XML inside.

In my implementation I made it a string because I don't know whether it a JSON or XML. So I think in your code - just do the following:

response = await client.send_message(
    queue_url=queue_url,
    message_body=json.dumps({"demo": 1, "key": "value"}),
    delay_seconds=0,
)

---

[d3QUone]

I tested it locally using another SQS provider (https://cloud.vk.com/en/) and I cannot reproduce it there.
Unfortunately I don't have any project inside Amazon to test it.

I tested both with Python 3.8 and Python 3.11.

message_body="a     b    c     d",

In both cases signature is worked well:

{'MessageId': '2d6c0bd6-0b42-4f5a-adff-75a9e0da58ea', 'MD5OfMessageBody': '862f1b21a1366234d6c189166dcb5011'}

[{'MD5OfBody': '862f1b21a1366234d6c189166dcb5011', 'Body': 'a     b    c     d', 'ReceiptHandle': '1703167200-2d6c0bd6-0b42-4f5a-adff-75a9e0da58ea', 'MessageId': '2d6c0bd6-0b42-4f5a-adff-75a9e0da58ea'}]

[xZanon]

Hi @d3QUone ,
On this machine : CentOS Linux release 7.9.2009 (Core) + Python 3.8.13 , using code like:

    msg = json.dumps({"test_num": 12, "test_exec": "some text with spaces"})
    response = await client.send_message(queue_url=queue_url,
                    message_body=msg,
                    delay_seconds=0,)

return error :

SQS API error: status_code=403
<Error><Type>Sender</Type><Code>SignatureDoesNotMatch</Code>
The Canonical String for this request should have been
'GET
/
Action=SendMessage&amp;DelaySeconds=0&amp;MessageBody=%7B%22test_num%22%3A12%2C%22test_exec%22%3A%22some%20text%20no%20spaces%22%7D&amp;QueueUrl=https%3A%2F%2Fsqs.eu-west-1.amazonaws.com%2F400193009206%2Fzzzzzzzz&amp;Version=2012-11-05
host:sqs.eu-west-1.amazonaws.com

same with :

msg = json.dumps({"test_num": 12, "test_exec": "some text no spaces"}, separators=(',', ':'))

The only working combination in my case is :

msg = json.dumps({"test_num": 12, "test_exec": "some_text_no_spaces"},separators=(',', ':'))

I think @josheschulz has the same problem.

Let me know if there any additional debug info, or var dumps I could provide .

Regards,

[xZanon]

Hi again,

maybe I am wrong, but I think, that issue could be that you are calculating headers by using default quote_via=urllib.parse.quote_plus) , line 72 in client.py, and then you are sending both headers and params via self.session.get , and real aiohttp.ClientSession is using different url encode - "IDNA" , equal to quote_via=urllib.parse.quote after you have generated the hash.

https://docs.aiohttp.org/en/stable/client_quickstart.html

aiohttp internally performs URL canonicalization before sending request.
Canonicalization encodes host part by IDNA codec and applies requoting to path and query parts.

Just for testing purposes, changing line 72 like :

        canonical_querystring = urllib.parse.urlencode(list(sorted(params.items())), quote_via=urllib.parse.quote)

Works for me.

I think the best solution would be to keep the same quote_via for both def get_headers (72) and self.session = aiohttp.ClientSession(timeout=self.timeout) (47)

I hope this could help to solve this issue in our case.

Regards,

[xZanon]

Hello @d3QUone and Happy new Year !
I hope this year will bring peace and prosperity to all people around the globe.

Now back to the issue. After some investigation looks like aiohttp.ClientSession does not support quote_via and support only quote(). So we have 2 options:
a) use quote_via=urllib.parse.quote) on line 72, or
b) prepare the url string in advance , and use it in both places, but add encoded=True to line 135
In case B we have to handle params . "Passing params overrides encoded=True, never use both options."

I do not like to push on this, but could we expect fix in the near feature, or should we adopt the package internally and modify it to work for us?

Kind Regards, ...

[d3QUone]

Hi @xZanon! Happy New Year!
Sorry for delay in responses - I was a bit busy with my main project.
I think I can prepare a fix in a separate branch soon. Do you mind help me with testing this branch?

---

Update:

• the branch is here Allow different quote_via functions #19

[xZanon]

Hi, @d3QUone ,

Happy to report that the patch is working in our case with AWS.
Just send 2'532'019 messages with speed of : 6033 m/s

Great job !!!

[d3QUone]

Good news! I'll release and publish the new version of the lib soon!
Thank you for your help @xZanon!