SDK request are not compressed, add header Accept-Encoding: gzip, deflate in all http requests

[danielesegato]

The SDK does not set Accept-Encoding: gzip, deflate when performing request, the result is that when there is lot of text the SDK:

• is much slower (a request can take 30 seconds instead of 3 seconds)
• consume more battery
• consume more data plan

[fosterzhang]

Understood. Gzip compression had been turned on since v2.2.0 release. However we had to turned it off in v2.2.1 because it caused crc32 checksum exception in DynamoDB. lscheinkman proposed a promising solution #40 and we will test it before taking it. Please keep an eye on the next release.

[danielesegato]

Sorry I opened another bug I didn't see that one.

Yeah I was just digging it out and find out your comment about disabling for the CRC issue.

I went forward and edited the SDK code by re-enabling http compression and disabling the CRC check (actually just commented out the exception if missmatch)

I tested my app out and it was the result in speed was amazing compared to what I had before.

Of course disabling CRC check is not a good thing and yes, I think that the right solution should be to manually handle the compression and perform the CRC check after decompressing the response.

This is a show stopper for us tough, do you have any ETA for when the release with gzip compression will be enabled?

[fosterzhang]

We can't make you a promise, but we are aiming at a release in a week or so.

[fosterzhang]

The latest release v2.2.2 enables HTTP compression. It also correctly handles crc32 checksum with DynamoDB. See http://aws.amazon.com/releasenotes/4067314458888112 for details.

[paulpv]

I just started noticing that I am getting a lot of crc32 errors starting in 2.3.1.
My last known good working version was 2.2.20.
Shall I open a new issue on this?

com.amazonaws.AmazonClientException: Unable to execute HTTP request: Client calculated crc32 checksum didn't match that calculated by server side
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:424)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:199)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:4181)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchGetItem(AmazonDynamoDBClient.java:594)
...
Caused by: com.amazonaws.internal.CRC32MismatchException: Client calculated crc32 checksum didn't match that calculated by server side
at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:119)
at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:47)
at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:632)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:368)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:199) 
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:4181) 
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchGetItem(AmazonDynamoDBClient.java:594) 
...

[navybk]

Caused by: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Client calculated crc32 checksum didn't match that calculated by server side at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:430) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:200) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:4181) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.query(AmazonDynamoDBClient.java:1450) at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.PaginatedQueryList.fetchNextPage(PaginatedQueryList.java:84) at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.PaginatedList.loadNextResults(PaginatedList.java:162) at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.PaginatedList.nextResultsAvailable(PaginatedList.java:149) at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.PaginatedList.loadAllResults(PaginatedList.java:133) at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.PaginatedList.size(PaginatedList.java:417) […] Caused by: com.amazonaws.internal.CRC32MismatchException: Client calculated crc32 checksum didn't match that calculated by server side at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:119) at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:47) at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:639) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:372) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:200)  at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:4181)  at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.query(AmazonDynamoDBClient.java:1450)  at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.PaginatedQueryList.fetchNextPage(PaginatedQueryList.java:84)  at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.PaginatedList.loadNextResults(PaginatedList.java:162)  at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.PaginatedList.nextResultsAvailable(PaginatedList.java:149)  at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.PaginatedList.loadAllResults(PaginatedList.java:133)  at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.PaginatedList.size(PaginatedList.java:417) 

gradle
compile 'com.amazonaws:aws-android-sdk-ddb:2.3.9'
compile 'com.amazonaws:aws-android-sdk-ddb-mapper:2.3.9'

Tested on different devices with different Android-Versions.

[ghost]

catch an exceptioncom.amazonaws.AmazonClientException: Unable to execute HTTP request: Client calculated crc32 checksum didn't match that calculated by server side

I got a same exception. Is there something i can do about it.Going back to an older sdk version?