AWS S3 Frankfurt region not working #28

FPurchess · 2015-02-15T16:15:06Z

COPY OF BITBUCKET ISSUE #214 - https://bitbucket.org/david/django-storages/issue/214/aws-s3-frankfurt-region-not-working

"Andreas Schilling created an issue 2015-01-04

Using Frankfurt region (Germany) with django-storages produces HTTP 400 error. S3 in the the new region supports only Signature Version 4. In all other regions, Amazon S3 supports both Signature Version 4 and Signature Version 2.

I assume django-storages only supports Signature Version 2. Is there any chance to support Version 4?"

Thanks @jschneier for the fork! Is there a chance for django-storages-redux to support the eu-central-1 region?

FPurchess · 2015-02-15T17:47:28Z

s3 boto user can find a temporary workaround here: boto/boto#2741

EDIT: here is a sample workaround

import os
from storages.backends.s3boto import S3BotoStorage

os.environ['S3_USE_SIGV4'] = 'True'

class S3Storage(S3BotoStorage):
    @property
    def connection(self):
        if self._connection is None:
            self._connection = self.connection_class(
                self.access_key, self.secret_key,
                calling_format=self.calling_format, host='s3.eu-central-1.amazonaws.com')
        return self._connection

frisellcpl · 2015-02-18T12:11:58Z

I can confirm that the above workaround works for Frankfurt.

Big thanks @FPurchess !

rotherfuchs · 2015-02-18T15:30:45Z

I can confirm this, too! Thank you!!

BodhiSukha · 2015-03-18T09:20:35Z

You can just add this to the settings:

os.environ['S3_USE_SIGV4'] = 'True'
AWS_S3_HOST = 's3.eu-central-1.amazonaws.com'

(Note: I'm using the bitbucket repo)

jschneier · 2015-03-18T19:39:14Z

@BodhiSukha thanks for that, going to close now. If you notice any regression in moving from the BitBucket repo to this one please open an issue.

satyrius · 2015-03-19T22:43:33Z

@BodhiSukha AWS_S3_HOST = 's3.eu-central-1.amazonaws.com' does not work for me. Any idea why? Hacking class as @FPurchess works fine.

boto==2.36.0
django-storages==1.1.8

BodhiSukha · 2015-03-30T08:55:23Z

@satyrius sorry just saw you comment.

I'm using it in 2 different setups

boto==2.34.0 
django-storages==1.1.8

and

boto==2.36.0 
django-storages==1.1.8

works fine on both of them. Unfortunately I'm not sure about your problem. Only other thing different in my settings is the calling format. In python 2.7.9 environment I use this extra setting

AWS_S3_CALLING_FORMAT = boto.s3.connection.OrdinaryCallingFormat()

miracle2k · 2015-04-16T14:04:30Z

When I try using this workaround, I get HostRequiredError: BotoClientError: When using SigV4, you must specify a 'host' parameter..

django-storages==1.1.8
boto==2.38.0

frisellcpl · 2015-04-17T13:27:15Z

Funny this has started to throw 403:Forbidden all of a sudden. Only when requesting from localhost though. Might have something to do with http / https?

#EDIT#
This had to do with my local timesettings. Synced to ntp and everything worked as expected. I blame friday afternoon. And apologies for the stupid comment.

Leistungsabfall · 2015-05-28T13:37:02Z

@miracle2k Make sure you're providing the modified Storage Class in STATICFILES_STORAGE and not the boto default one.

This works for me:

import os
from storages.backends.s3boto import S3BotoStorage

os.environ['S3_USE_SIGV4'] = 'True'

class S3Storage(S3BotoStorage):
    @property
    def connection(self):
        if self._connection is None:
            self._connection = self.connection_class(
                self.access_key, self.secret_key,
                calling_format=self.calling_format, host='s3.eu-central-1.amazonaws.com')
        return self._connection

STATICFILES_STORAGE = 'MyProject.settings.S3Storage'

somacci · 2016-02-18T02:20:42Z

I have checked the issues about Signature 4 with boto for Frankfurt region. I have had same problem with Seoul Frankfurt. I know that both Seoul and Frankfurt regions only support Signture 4 only.
So, is this closed or not. If it is closed, how to solve it. Please let me know.

galuszkak · 2016-06-12T10:21:20Z

@somacci solution by @FPurchess works. Try it.

lusentis · 2016-09-30T16:52:23Z

Thanks @Leistungsabfall!
The 400 Bad Request exception persisted until I updated to boto==2.38.0 (I was running 2.22).

hugodes · 2016-11-27T16:32:22Z

Could we re-open this issue please ? It's not solved, we just have a work-around.
Suport for s3v4 auth is really important for all who are deploying outside the U.S.

sdeleeuw · 2016-12-08T22:36:20Z

I had the same issue and fixed it by using S3Boto3Storage instead of S3BotoStorage and two not documented parameters in my settings.py.

requirements.txt:

boto3==1.4.1
django-storages==1.5.1

settings.py:

STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
AWS_ACCESS_KEY_ID = ''
AWS_SECRET_ACCESS_KEY = ''
AWS_STORAGE_BUCKET_NAME = ''
AWS_S3_REGION_NAME = 'eu-central-1' (not documented)
AWS_S3_SIGNATURE_VERSION = 's3v4' (not documented)
AWS_QUERYSTRING_AUTH = False
AWS_S3_FILE_OVERWRITE = True

To be honest, I didn't set the STATICFILES_STORAGE setting, because my application uses the S3Boto3Storage class for specific tasks, not for all static files. But I think it should work this way.

I stumbled upon the not documented options by going through the source code of django-storages.

Hope this will be a solution for others too.

nehaljwani · 2017-01-05T19:55:47Z

@sdeleeuw 's work around works beautifully.

jjanczyszyn · 2017-03-24T17:02:35Z

In my case for Frankfurt region it was enough to only add these two lines to settings (no need to override S3BotoStorage)
S3_USE_SIGV4 = True
AWS_S3_HOST = 's3.eu-central-1.amazonaws.com'

Tested with
django-storages==1.5.2
boto==2.46.1

BigglesZX · 2017-04-03T17:14:28Z

I experienced an issue with using the London region (eu-west-2) and @tramwaj29's solution fixed it for me, substituting the region name in the value of AWS_S3_HOST.

jschneier · 2017-04-03T18:55:52Z

I'm going to reopen this until I can get it added to the docs.

oesah · 2017-06-18T15:39:56Z

I used the method by @sdeleeuw, and at least it does not throw any errors anymore. However, when I upload a File, it does not create the file in S3. I create an IAM user and use the credentials from the user. Also, I added the user as a principal

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "PublicReadForGetBucketObjects",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::bucket/media/*"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789:user/myuser"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::bucket/media",
                "arn:aws:s3:::bucket/media/*"
            ]
        }
    ]
}

And CORS:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>HEAD</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>Authorization</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

Any ideas?

oesah · 2017-06-18T19:10:19Z

So I solved my problem my switching to:

boto3==1.4.4
django-storages==1.5.2

Not sure what the problem was, but I am using eu-central-1. Maybe that is the issue, but anyway, seems like its fixed.

jschneier · 2017-06-21T05:12:19Z

@oesah I think that s3boto3 handles this automagically but the same doesn't play out for s3boto. I have opened #335 which includes documentation for the fixes discovered in this thread and changes the default signature version so no one will be bit by this in the future. Thanks for your patience and your discoveries everyone. If you see anything wrong in #335 please say something. I'm going to merge it in the morning and immediately cut 1.6 afterwards.

Tllew · 2018-05-24T13:04:47Z

I'm getting :

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>
        InvalidRequest
    </Code>
    <Message>
        The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
    </Message>
    <RequestId>548992053424FAF0</RequestId> . 
    <HostId>
        bE/0RI6Wn5J41Qkf/kKRnA8A+uLuy9N1C6wyGsptJeqqOqmyaV+JCbXbj8hzHI7Bp1PtFOo897k= . 
    </HostId>
</Error>

with this code:

AWS_S3_HOST = 's3.eu-west-2.amazonaws.com'
s3 = boto.connect_s3(settings.S3_CALLS_KEY, settings.S3_CALLS_SECRET, host=AWS_S3_HOST)
bucket = s3.get_bucket(settings.S3_CALLS_BUCKET)

Can anyone see the issue here?

I'm trying for london, I got the s3 host from here

EDIT : I updated boto but it went to a different python environment, updated the correct boto and now my code above works.

coler-j · 2018-09-14T13:44:24Z

This also appears to happen in the 'Canada (Central)' region. Was getting a Cross-Origin Read Blocking (CORB) blocked cross-origin (error was caused because the response was just error XML and content type xml instead of the expected image content type) in the browser until I changed region. Going to implement workaround by @sdeleeuw

Edit: Workaround worked great for Canada region.

vnikolayev1 · 2019-04-26T19:28:25Z

Adding
AWS_S3_REGION_NAME = 'eu-central-1'
fixed it for me. I guess you just need to set your region, and you good to go.
Full code:

AWS_SECRET_ACCESS_KEY = ""
AWS_STORAGE_BUCKET_NAME = ""
AWS_S3_REGION_NAME = 'eu-central-1'

AWS_S3_FILE_OVERWRITE = False
AWS_DEFAULT_ACL = None

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

filipeximenes · 2019-09-13T11:59:53Z

Just my 2 cents here, I might be missing something but it seems like boto reads S3_USE_SIGV4 straight from the env vars and thus setting it in settings.py has no effect whatsoever. See here:
https://github.com/boto/boto/blob/03b2268348ea81d80e3e5ddea0970f4968561010/boto/auth.py#L1067

Can someone confirm this? If this is true docs need to be updated.

filipeximenes · 2019-09-13T12:11:19Z

Setting the region name to one of the following also does the job because of this

SIGV4_DETECT = [
    '.cn-',
    # In eu-central and ap-northeast-2 we support both host styles for S3
    '.eu-central',
    '-eu-central',
    '.ap-northeast-2',
    '-ap-northeast-2',
    '.ap-south-1',
    '-ap-south-1',
    '.us-east-2',
    '-us-east-2',
    '-ca-central',
    '.ca-central',
    '.eu-west-2',
    '-eu-west-2',
]```

While trying to crawl dandiarchive bucket with authentication, to fetch also files which are not publicly available, I have ran into <Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.< for which discussion was ongoing in 2017: jschneier/django-storages#28 . A workaround which worked for me was to specify host option to boto.connect_s3 to point to the specific region. So with this fix now it would be possible to use it in the provider configuration, e.g. [provider:dandi-s3] url_re = s3://dandiarchive($|/.*) credential = dandi-s3-backup authentication_type = aws-s3 aws-s3_host = s3.us-east-2.amazonaws.com There might be other options we might want to add later on, so I did not store host in the attribute, but right within the dictionary of optional kwargs for connect_s3.

ikahitin · 2021-03-16T19:12:05Z

So I solved my problem my switching to:
boto3==1.4.4
django-storages==1.5.2
Not sure what the problem was, but I am using eu-central-1. Maybe that is the issue, but anyway, seems like its fixed.

Saved my day. Also using eu-central-1

jschneier closed this as completed Mar 18, 2015

chrisseto mentioned this issue Jul 31, 2015

[feature] Add s3 bucket server side encryption CenterForOpenScience/osf.io#3812

Merged

mlurie mentioned this issue Feb 1, 2016

Add ability to specify AWS s3 host through AWS_S3_HOST environment variable pandas-dev/pandas#12198

Closed

yannh mentioned this issue Apr 26, 2016

'thread.lock' object has no attribute 'lock' lucidsoftware/apt-boto-s3#1

Closed

dtaylor-iidnet mentioned this issue Nov 16, 2016

S3 module doesn't use AWS Signature Version 4 ansible/ansible-modules-core#3140

Closed

sdeleeuw mentioned this issue Dec 9, 2016

Backend boto3 and signature v4 #203

Closed

ad-m mentioned this issue Dec 30, 2016

Add 'AWS4-HMAC-SHA256 signatures' section for Amazon S3 #241

Closed

nehaljwani mentioned this issue Jan 5, 2017

S3 bucket lookup / get_bucket broken for eu-central-1 (Frankfurt) boto/boto#2741

Open

jamstooks mentioned this issue Jan 26, 2017

The authorization mechanism you have provided is not supported jamstooks/django-s3-folder-storage#15

Closed

sleibrock mentioned this issue Jan 31, 2017

SignatureDoesNotMatch when calling PutObject #262

Closed

bhanuvrat mentioned this issue Feb 8, 2017

enable support for newer aws regions Fueled/django-init#214

Merged

jschneier reopened this Apr 3, 2017

patrys mentioned this issue Apr 13, 2017

exposes aws s3 region saleor/saleor#1010

Closed

jleclanche added s3boto and removed s3boto labels Jun 5, 2017

jschneier closed this as completed Jun 21, 2017

ansibot mentioned this issue Sep 12, 2017

S3 module doesn't use AWS Signature Version 4 ansible/ansible#29872

Closed

This was referenced Jun 17, 2020

Fix s3 course downloads un UK appsembler/edx-platform#623

Merged

Add new var for S3 london bucket if necesary appsembler/configuration#315

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS S3 Frankfurt region not working #28

AWS S3 Frankfurt region not working #28

FPurchess commented Feb 15, 2015

FPurchess commented Feb 15, 2015

frisellcpl commented Feb 18, 2015

rotherfuchs commented Feb 18, 2015

BodhiSukha commented Mar 18, 2015

jschneier commented Mar 18, 2015

satyrius commented Mar 19, 2015

BodhiSukha commented Mar 30, 2015

miracle2k commented Apr 16, 2015

frisellcpl commented Apr 17, 2015

Leistungsabfall commented May 28, 2015

somacci commented Feb 18, 2016

galuszkak commented Jun 12, 2016

lusentis commented Sep 30, 2016

hugodes commented Nov 27, 2016

sdeleeuw commented Dec 8, 2016

nehaljwani commented Jan 5, 2017

jjanczyszyn commented Mar 24, 2017 •

edited

Loading

BigglesZX commented Apr 3, 2017

jschneier commented Apr 3, 2017

oesah commented Jun 18, 2017

oesah commented Jun 18, 2017

jschneier commented Jun 21, 2017 •

edited

Loading

Tllew commented May 24, 2018 •

edited

Loading

coler-j commented Sep 14, 2018 •

edited

Loading

vnikolayev1 commented Apr 26, 2019 •

edited

Loading

filipeximenes commented Sep 13, 2019

filipeximenes commented Sep 13, 2019

ikahitin commented Mar 16, 2021

AWS S3 Frankfurt region not working #28

AWS S3 Frankfurt region not working #28

Comments

FPurchess commented Feb 15, 2015

FPurchess commented Feb 15, 2015

frisellcpl commented Feb 18, 2015

rotherfuchs commented Feb 18, 2015

BodhiSukha commented Mar 18, 2015

jschneier commented Mar 18, 2015

satyrius commented Mar 19, 2015

BodhiSukha commented Mar 30, 2015

miracle2k commented Apr 16, 2015

frisellcpl commented Apr 17, 2015

Leistungsabfall commented May 28, 2015

somacci commented Feb 18, 2016

galuszkak commented Jun 12, 2016

lusentis commented Sep 30, 2016

hugodes commented Nov 27, 2016

sdeleeuw commented Dec 8, 2016

nehaljwani commented Jan 5, 2017

jjanczyszyn commented Mar 24, 2017 • edited Loading

BigglesZX commented Apr 3, 2017

jschneier commented Apr 3, 2017

oesah commented Jun 18, 2017

oesah commented Jun 18, 2017

jschneier commented Jun 21, 2017 • edited Loading

Tllew commented May 24, 2018 • edited Loading

coler-j commented Sep 14, 2018 • edited Loading

vnikolayev1 commented Apr 26, 2019 • edited Loading

filipeximenes commented Sep 13, 2019

filipeximenes commented Sep 13, 2019

ikahitin commented Mar 16, 2021

jjanczyszyn commented Mar 24, 2017 •

edited

Loading

jschneier commented Jun 21, 2017 •

edited

Loading

Tllew commented May 24, 2018 •

edited

Loading

coler-j commented Sep 14, 2018 •

edited

Loading

vnikolayev1 commented Apr 26, 2019 •

edited

Loading