AWS Access Key ID not exist for CN users

[KeeeeiZ]

Expected Behavior

I am a Chinese user using AWS. I want to use ice to check my bills on AWS. And I expect the AWS SDK can successfully connect to AWS in CN region.

Current Behavior

I follow the instructions on the manual of ice but the program kept telling me the following

Error com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 51215BB2D9A12365), S3 Extended Request ID: /qEmWFa2q5U2nFg8WBb0KlWUBiNmfc2ZzEC3cbWwGuBNlWC20Uuho1Tk5BkfG5PPCyjdrmiBvDE=
| Error 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
| Error 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
| Error 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
| Error 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
| Error 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
| Error 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
| Error 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
| Error 	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
| Error 	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
| Error 	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221)
| Error 	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168)
| Error 	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4162)
| Error 	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:821)
| Error 	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:798)
| Error 	at com.netflix.ice.basic.BasicManagers.doWork(BasicManagers.java:95)
| Error 	at com.netflix.ice.basic.BasicManagers.init(BasicManagers.java:60)
| Error 	at com.netflix.ice.reader.ReaderConfig.<init>(ReaderConfig.java:102)
| Error 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
| Error 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
| Error 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
| Error 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
| Error 	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
| Error 	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
| Error 	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57)
| Error 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
| Error 	at BootStrap$_closure1.doCall(BootStrap.groovy:237)
| Error 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| Error 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
| Error 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
| Error 	at java.lang.reflect.Method.invoke(Method.java:498)
| Error 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
| Error 	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
| Error 	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)
| Error 	at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
| Error 	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)
| Error 	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1081)
| Error 	at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
| Error 	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)
| Error 	at groovy.lang.Closure.call(Closure.java:423)
| Error 	at groovy.lang.Closure.call(Closure.java:417)
| Error 	at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:327)
| Error 	at grails.util.Environment.executeForEnvironment(Environment.java:320)
| Error 	at grails.util.Environment.executeForCurrentEnvironment(Environment.java:296)
| Error 	at org.codehaus.groovy.grails.commons.DefaultGrailsBootstrapClass.callInit(DefaultGrailsBootstrapClass.java:60)
| Error 	at org.codehaus.groovy.grails.web.context.GrailsConfigUtils.executeGrailsBootstraps(GrailsConfigUtils.java:78)
| Error 	at org.codehaus.groovy.grails.web.context.GrailsContextLoaderListener.initWebApplicationContext(GrailsContextLoaderListener.java:110)
| Error 	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)
| Error 	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4728)
| Error 	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5162)
| Error 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
| Error 	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1409)
| Error 	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1399)
| Error 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
| Error 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
| Error 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
| Error 	at java.lang.Thread.run(Thread.java:748)
| Error 2019-10-17 09:40:28,455 [localhost-startStop-1] ERROR BootStrap  - Startup failed
Message: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 51215BB2D9A12365)
    Line | Method
->> 1588 | handleErrorResponse              in com.amazonaws.http.AmazonHttpClient$RequestExecutor
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|   1258 | executeOneRequest                in     ''
|   1030 | executeHelper . . . . . . . . .  in     ''
|    742 | doExecute                        in     ''
|    716 | executeWithTimer . . . . . . . . in     ''
|    699 | execute                          in     ''
|    667 | access$500 . . . . . . . . . . . in     ''
|    649 | execute                          in com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl
|    513 | execute . . . . . . . . . . . .  in com.amazonaws.http.AmazonHttpClient
|   4221 | invoke                           in com.amazonaws.services.s3.AmazonS3Client
|   4168 | invoke . . . . . . . . . . . . . in     ''
|   4162 | invoke                           in     ''
|    821 | listObjects . . . . . . . . . .  in     ''
|    798 | listObjects                      in     ''
|     95 | doWork . . . . . . . . . . . . . in com.netflix.ice.basic.BasicManagers
|     60 | init                             in     ''
|    102 | <init> . . . . . . . . . . . . . in com.netflix.ice.reader.ReaderConfig
|    237 | doCall                           in BootStrap$_closure1
|    327 | evaluateEnvironmentSpecificBlock in grails.util.Environment
|    320 | executeForEnvironment            in     ''
|    296 | executeForCurrentEnvironment . . in     ''
|    266 | run                              in java.util.concurrent.FutureTask
|   1149 | runWorker . . . . . . . . . . .  in java.util.concurrent.ThreadPoolExecutor
|    624 | run                              in java.util.concurrent.ThreadPoolExecutor$Worker
^    748 | run . . . . . . . . . . . . . .  in java.lang.Thread

As far as I know, the CN region uses a different series of domains for DNS/endpoints/etc... I am wondering whether this problem is caused by the config in the SDK which could have not taken CN region into consideration. (I am just a Java beginner so I come here to ask for help)

Possible Solution

A possible guess is that the AWS SDK has not taken CN region into consideration. If I know where to modify the corresponding settings, an adjustment to the region settings might help solve the problem above.

Steps to Reproduce (for bugs)

I use docker-ice. After modification of endpoints in the Dockerfile for ice(in the last part of my issue) as below:

FROM openjdk:alpine
MAINTAINER Jon Brouse @jonbrouse

ENV INSTALL_DIR /opt/ice
ENV HOME_DIR /root
ENV GRAILS_VERSION 2.4.4
ENV GRAILS_HOME ${HOME_DIR}/.grails/wrapper/${GRAILS_VERSION}/grails-${GRAILS_VERSION}
ENV PATH $PATH:${HOME_DIR}/.grails/wrapper/${GRAILS_VERSION}/grails-${GRAILS_VERSION}/bin/
ENV ICE_VERSION 1.1.2

ARG JAVA_OPTS

WORKDIR ${HOME_DIR}/.grails/wrapper/${GRAILS_VERSION}

# Install required software
RUN apk add --no-cache bash curl unzip && ¥
    curl -O http://dist.springframework.org.s3.amazonaws.com/release/GRAILS/grails-${GRAILS_VERSION}.zip && ¥
    unzip grails-${GRAILS_VERSION}.zip && ¥
    rm -rf grails-${GRAILS_VERSION}.zip

WORKDIR ${INSTALL_DIR} 

# Ice setup
RUN mkdir /mnt/ice_processor /mnt/ice_reader && ¥
    curl -fsSL https://github.com/Teevity/ice/archive/v${ICE_VERSION}.tar.gz | tar zx --strip-components=1 && ¥
    sed -i 's#amazonaws.com#amazonaws.com.cn#g' src/java/com/netflix/ice/basic/BasicReservationService.java && ¥
    sed -i 's#amazonaws.com#amazonaws.com.cn#g' src/java/com/netflix/ice/common/AwsUtils.java && ¥
    sed -i 's#amazonaws.com#amazonaws.com.cn#g' src/java/com/netflix/ice/processor/ReservationCapacityPoller.java && ¥
    grails ${JAVA_OPTS} wrapper && ¥
    rm grails-app/i18n/messages.properties && ¥
    sed -i -e '1i#!/bin/bash¥' grailsw

EXPOSE 8080

ENTRYPOINT ["/opt/ice/grailsw"]


CMD []

The endpoints have been modified in the Dockerfile above.

Then docker-compose up -d and the docker-compose.yml is as below:

version: '3'
services:
  ice:
    build: ice
    hostname: ice
    image: jonbrouse/ice:latest
    tty: true
    command: |
      -Djava.net.preferIPv4Stack=true
      -Djava.net.preferIPv4Addresses
      -Duser.timezone=<Your Timezone ie America/New_York>
      -Dice.s3AccessKeyId=<s3AccessKeyId>
      -Dice.s3SecretKey=<s3SecretKeyId>
      run-app
    volumes:
      - ./ice.properties:/opt/ice/src/java/ice.properties
  nginx:
    hostname: nginx
    image: nginx:1.13.3-alpine
    ports:
      - "80:80"
    links:
      - ice:ice
    volumes:
      - ./default.conf:/etc/nginx/conf.d/default.conf

The log of the docker-compose would show me the bug in the Current Behavior part.

Context

If the problem is really coused by what I thought it is, lots of CN users would benefit from it.

Your Environment

• AWS Java SDK version used:
• JDK version used:
• Operating System and version:
  I run the application in docker and the docker requires openjdk:alpine as the environment inside the docker.

[debora-ito]

@KeeeeiZ to access the AWS services in regions located within China you need to use specific endpoints, and it looks like the Teevity/ice project does not support these endpoints.

They have open issues on their repository (1, 2) asking for China endpoints support, I suggest you +1 them.

Closing this, feel free to reach out if you have further questions.