Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fog-aws not working with Hitachi #122

Closed
palonsoro opened this issue May 22, 2015 · 11 comments
Closed

Fog-aws not working with Hitachi #122

palonsoro opened this issue May 22, 2015 · 11 comments

Comments

@palonsoro
Copy link

Good morning.
I have been trying to use fog to connect to the S3-compatible endpoint of an Hitachi instance and I get the following error (this comes from an irb session):

2.1.5 :014 > connection.directories
Excon::Errors::Found: Expected(200) <=> Actual(302 Found)
excon.error.response
  :body          => ""
  :headers       => {
    "Content-Length" => "0"
    "Content-Type"   => "text/xml"
    "Date"           => "Fri, 22 May 2015 09:19:10 GMT"
    "Location"       => "https://mybucket.hitachiinstance.net/rest"
    "Server"         => "HCP V7.1.0.10"
  }
  :local_address => "10.0.0.2"
  :local_port    => 47938
  :reason_phrase => "Found"
  :remote_ip     => "10.5.0.30"
  :status        => 302
  :status_line   => "HTTP/1.1 302 Found\r\n"

        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/expects.rb:6:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/response_parser.rb:8:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:372:in `response'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:236:in `request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/idempotent.rb:26:in `error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:73:[41/1875]
_call'                                                                                          [0/1876]
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:256:in `rescue in 
request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:204:in `request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/idempotent.rb:26:in 
`error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error
_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error
_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:256:in `rescue in 
request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:204:in `request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/idempotent.rb:26:in 
`error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error
_call'
... 7 levels...
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-aws-0.2.2/lib/fog/aws/requests/storage/get_service.
rb:21:in `get_service'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-aws-0.2.2/lib/fog/aws/models/storage/directories.rb
:10:in `all'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:113:in `lazy
_load'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:17:in `map'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:85:in `inspect_ob
ject'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:72:in `block in n
ested_objects_string'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/formatador-0.2.5/lib/formatador.rb:92:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:41:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:72:in `nested_obj
ects_string'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:54:in `object_str
ing'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:14:in `block in f
ormat'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/formatador-0.2.5/lib/formatador.rb:92:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:41:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:14:in `format'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:73:in `inspe
ct'
        from /usr/local/rvm/rubies/ruby-2.1.5/bin/irb:11:in `<main>'2.1.5

It seemed that Excon was not following a 302 redirection. Thus, I did this:

Excon.defaults[:middlewares].insert(0,Excon::Middleware::RedirectFollower)

So that Excon::Middleware::RedirectFollower is placed before Excon::Middleware::Expects.

Then, the 302 response became a 403 forbidden, although I was providing good credentials (I triple-checked this). The full stack trace is:

2.1.5 :090 > connection.directories
Excon::Errors::Forbidden: Expected(200) <=> Actual(403 Forbidden)
excon.error.response
  :body          => ""
  :headers       => {
    "Content-Length" => "0"
    "Date"           => "Fri, 22 May 2015 09:53:34 GMT"
    "Expires"        => "Thu, 01 Jan 1970 00:00:00 GMT"
    "Server"         => "Jetty(7.6.0.v20120127)"
    "Set-Cookie"     => "JSESSIONID=1n23cfhzrnb074sniv7jnfkxm;Path=/"
  }
  :local_address => "10.0.0.2"
  :local_port    => 48045
  :reason_phrase => "Forbidden"
  :remote_ip     => "10.5.0.30"
  :status        => 403
  :status_line   => "HTTP/1.1 403 Forbidden\r\n"

        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/expects.rb:6:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/redirect_follower.rb:48:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/response_parser.rb:8:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/redirect_follower.rb:51:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:372:in `response'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:236:in `request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/redirect_follower.rb:45:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/response_parser.rb:8:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/redirect_follower.rb:51:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:372:in `response'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:236:in `request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/idempotent.rb:26:in `error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error_call'
... 20 levels...
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-aws-0.2.2/lib/fog/aws/requests/storage/get_service.rb:21:in `get_service'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-aws-0.2.2/lib/fog/aws/models/storage/directories.rb:10:in `all'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:113:in `lazy_load'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:17:in `map'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:85:in `inspect_object'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:72:in `block in nested_objects_string'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/formatador-0.2.5/lib/formatador.rb:92:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:41:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:72:in `nested_objects_string'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:54:in `object_string'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:14:in `block in format'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/formatador-0.2.5/lib/formatador.rb:92:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:41:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:14:in `format'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:73:in `inspect'
        from /usr/local/rvm/rubies/ruby-2.1.5/bin/irb:11:in `<main>'2.1.5 :091 >

Can you help me with this, please?

Thinking more in the future, is there any plan to natively support 302 redirections at fog out-of-the-box? I say this because, in my irb session, I can change the Excon defaults with no problem. But I need to do this kind of connection from a more complex program, which uses fog and only accepts a fog connection hash (and does not allow me setting anything else).

@lanej
Copy link
Member

lanej commented May 22, 2015

302s won't work because the path is of the canonical string used to calculate the HMAC address. we'd need a much more complex middleware to support this scenario.

why are you getting redirects at all? is it not possible to use the correct url at initialization ?

@palonsoro
Copy link
Author

I tried specifying the URL given at 'Location:' response header (with and without the '/rest' ending) and I get the same redirection again and again (although I should not, as far as I know). However, the redirection is not likely to be "endless", because using the Excon::Middleware::RedirectFollower or other heavy clients like CrossFTP or CloudBerry, the connection with Hitachi works perfectly. Exconn or fog might be doing something I do not fully understand with the URL (I am very new to fog, I must say).

@geemus
Copy link
Member

geemus commented May 28, 2015

@palonsoro you might also try with omit_default_port => true for excon, sometimes including the port will loop redirects like this. Fog itself should accept connection_options to be passed to excon if I remember right, so I think you should be able to pass stuff in that way in more complex programs.

@palonsoro
Copy link
Author

Ok. I did try setting omit_default_port => true in Excon.defaults. I still get redirections, but it makes much sense to set this anyway because the 'Location' header specifies no ports.

On the other hand, I did not know about the conection_options parameter, so thank you very much for that!!!

@geemus
Copy link
Member

geemus commented Jun 1, 2015

Hmm, I'm not sure beyond the port stuff. In particular I definitely wouldn't expect a redirect if you go directly to the location (which would be ideal since it would work around the signature issue).

Could you run it with excon_debug => true and share the output? I think that might help pin down where things are going off the rail, or at least help eliminate some more possibilities. Thanks!

@palonsoro
Copy link
Author

I am sorry for the delay.

First: I have been able to avoid redirections by doing two things:

  • Using omit_default_port => true, as you suggested.
  • Using path_style => false at the connection hash.

However, I still get 403 errors with valid credentials (at least, with credentials correctly working with CloudBerry client, for example).

Here is the debug output produced when invoking connection.directories:

excon.request
  :chunk_size          => 1048576
  :ciphers             => "HIGH:!SSLv2:!aNULL:!eNULL:!3DES"
  :connect_timeout     => 60
  :debug_request       => true
  :debug_response      => true
  :expects             => 200
  :headers             => {
    "Authorization" => "REDACTED"
    "Date"          => "Fri, 05 Jun 2015 07:32:53 +0000"
    "Host"          => "myhitachi.cloud.local"
    "User-Agent"    => "fog/1.30.0 fog-core/1.30.0"
  }
  :host                => "myhitachi.cloud.local"
  :hostname            => "myhitachi.cloud.local"
  :idempotent          => true
  :instrumentor        => Excon::StandardInstrumentor
  :instrumentor_name   => "excon"
  :method              => "GET"
  :middlewares         => [
    Excon::Middleware::ResponseParser
    Excon::Middleware::Expects
    Excon::Middleware::Idempotent
    Excon::Middleware::Instrumentor
    Excon::Middleware::Mock
  ]
  :mock                => false
  :nonblock            => true
  :omit_default_port   => true
  :path                => "/"
  :persistent          => false
  :port                => 80
  :query               => nil
  :read_timeout        => 60
  :response_block      => #<Proc:0x00000005052c38@/usr/local/rvm/gems/ruby-2.1.6/gems/fog-xml-0.1.2/lib/
fog/xml/sax_parser_connection.rb:29 (lambda)>
  :retries_remaining   => 4
  :retry_limit         => 4
  :scheme              => "http"
  :ssl_verify_peer     => false
  :tcp_nodelay         => false
  :thread_safe_sockets => true
  :uri_parser          => URI
  :versions            => "excon/0.45.3 (x86_64-linux) ruby/2.1.6"
  :write_timeout       => 60
excon.response
  :body          => ""
  :headers       => {
    "Content-Type" => "application/xml;charset=UTF-8"
    "Date"         => "Fri, 05 Jun 2015 07:32:48 GMT"
    "Server"       => "HCP V7.1.0.10"
  }
  :local_address => "10.0.5.2"
  :local_port    => 42041
  :reason_phrase => "OK"
  :remote_ip     => "192.168.7.33"
  :status        => 200
  :status_line   => "HTTP/1.1 200 OK\r\n"
excon.request
  :chunk_size          => 1048576
  :ciphers             => "HIGH:!SSLv2:!aNULL:!eNULL:!3DES"
  :connect_timeout     => 60
  :debug_request       => true
  :debug_response      => true
  :expects             => 200
  :headers             => {
    "Authorization" => "REDACTED"
    "Date"          => "Fri, 05 Jun 2015 07:32:53 +0000"
    "Host"          => "myhitachi.cloud.local"
    "User-Agent"    => "fog/1.30.0 fog-core/1.30.0"
  }
  :host                => "myhitachi.cloud.local"
  :hostname            => "myhitachi.cloud.local"
  :idempotent          => true
  :instrumentor        => Excon::StandardInstrumentor
  :instrumentor_name   => "excon"
  :method              => "GET"
  :middlewares         => [
    Excon::Middleware::ResponseParser
    Excon::Middleware::Expects
    Excon::Middleware::Idempotent
    Excon::Middleware::Instrumentor
    Excon::Middleware::Mock
  ]
  :mock                => false
  :nonblock            => true
  :omit_default_port   => true
  :path                => "/my-bucket/"
  :persistent          => false
  :port                => 80
  :query               => {
    "location" => nil
  }
  :read_timeout        => 60
  :response_block      => #<Proc:0x00000005004470@/usr/local/rvm/gems/ruby-2.1.6/gems/fog-xml-0.1.2/lib/
  fog/xml/sax_parser_connection.rb:29 (lambda)>
  :retries_remaining   => 4
  :retry_limit         => 4
  :scheme              => "http"
  :ssl_verify_peer     => false
  :tcp_nodelay         => false
  :thread_safe_sockets => true
  :uri_parser          => URI
  :versions            => "excon/0.45.3 (x86_64-linux) ruby/2.1.6"
  :write_timeout       => 60
  excon.retry
  :chunk_size          => 1048576
  :ciphers             => "HIGH:!SSLv2:!aNULL:!eNULL:!3DES"
  :connect_timeout     => 60
  :debug_request       => true
  :debug_response      => true
  :expects             => 200
  :headers             => {
    "Authorization" => "REDACTED"
    "Date"          => "Fri, 05 Jun 2015 07:32:53 +0000"
    "Host"          => "myhitachi.cloud.local"
    "User-Agent"    => "fog/1.30.0 fog-core/1.30.0"
  }
  :host                => "myhitachi.cloud.local"
  :hostname            => "myhitachi.cloud.local"
  :idempotent          => true
  :instrumentor        => Excon::StandardInstrumentor
  :instrumentor_name   => "excon"
  :method              => "GET"
  :middlewares         => [
    Excon::Middleware::ResponseParser
    Excon::Middleware::Expects
    Excon::Middleware::Idempotent
    Excon::Middleware::Instrumentor
    Excon::Middleware::Mock
  ]
  :mock                => false
  :nonblock            => true
  :omit_default_port   => true
  :path                => "/my-bucket/"
  :persistent          => false
  :port                => 80
  :query               => {
    "location" => nil
  }
  :read_timeout        => 60
  :response_block      => #<Proc:0x00000005004470@/usr/local/rvm/gems/ruby-2.1.6/gems/fog-xml-0.1.2/lib/
  fog/xml/sax_parser_connection.rb:29 (lambda)>
  :retries_remaining   => 3
  :retry_limit         => 4
  :scheme              => "http"
  :ssl_verify_peer     => false
  :tcp_nodelay         => false
  :thread_safe_sockets => true
  :uri_parser          => URI
  :versions            => "excon/0.45.3 (x86_64-linux) ruby/2.1.6"
  :write_timeout       => 60
  excon.retry
  :chunk_size          => 1048576
  :ciphers             => "HIGH:!SSLv2:!aNULL:!eNULL:!3DES"
  :connect_timeout     => 60
  :debug_request       => true
  :debug_response      => true
  :expects             => 200
  :headers             => {
    "Authorization" => "REDACTED"
    "Date"          => "Fri, 05 Jun 2015 07:32:53 +0000"
    "Host"          => "myhitachi.cloud.local"
    "User-Agent"    => "fog/1.30.0 fog-core/1.30.0"
  }
  :host                => "myhitachi.cloud.local"
  :hostname            => "myhitachi.cloud.local"
  :idempotent          => true
  :instrumentor        => Excon::StandardInstrumentor
  :instrumentor_name   => "excon"
  :method              => "GET"
  :middlewares         => [
    Excon::Middleware::ResponseParser
    Excon::Middleware::Expects
    Excon::Middleware::Idempotent
    Excon::Middleware::Instrumentor
    Excon::Middleware::Mock
  ]
  :mock                => false
  :nonblock            => true
  :omit_default_port   => true
  :path                => "/my-bucket/"
  :persistent          => false
  :port                => 80
  :query               => {
    "location" => nil
    }
  :read_timeout        => 60
  :response_block      => #<Proc:0x00000005004470@/usr/local/rvm/gems/ruby-2.1.6/gems/fog-xml-0.1.2/lib/
fog/xml/sax_parser_connection.rb:29 (lambda)>
  :retries_remaining   => 2
  :retry_limit         => 4
  :scheme              => "http"
  :ssl_verify_peer     => false
  :tcp_nodelay         => false
  :thread_safe_sockets => true
  :uri_parser          => URI
  :versions            => "excon/0.45.3 (x86_64-linux) ruby/2.1.6"
  :write_timeout       => 60
excon.retry
  :chunk_size          => 1048576
  :ciphers             => "HIGH:!SSLv2:!aNULL:!eNULL:!3DES"
  :connect_timeout     => 60
  :debug_request       => true
  :debug_response      => true
  :expects             => 200
  :headers             => {
    "Authorization" => "REDACTED"
    "Date"          => "Fri, 05 Jun 2015 07:32:53 +0000"
    "Host"          => "myhitachi.cloud.local"
    "User-Agent"    => "fog/1.30.0 fog-core/1.30.0"
  }
  :host                => "myhitachi.cloud.local"
  :hostname            => "myhitachi.cloud.local"
  :idempotent          => true
  :instrumentor        => Excon::StandardInstrumentor
  :instrumentor_name   => "excon"
  :method              => "GET"
  :middlewares         => [
    Excon::Middleware::ResponseParser
    Excon::Middleware::Expects
    Excon::Middleware::Idempotent
    Excon::Middleware::Instrumentor
    Excon::Middleware::Mock
  ]
  :mock                => false
  :nonblock            => true
  :omit_default_port   => true
  :path                => "/my-bucket/"
  :persistent          => false
  :port                => 80
  :query               => {
    "location" => nil
  }
  :read_timeout        => 60
  :response_block      => #<Proc:0x00000005004470@/usr/local/rvm/gems/ruby-2.1.6/gems/fog-xml-0.1.2/lib/
fog/xml/sax_parser_connection.rb:29 (lambda)>
  :retries_remaining   => 1
  :retry_limit         => 4
  :scheme              => "http"
  :ssl_verify_peer     => false
  :tcp_nodelay         => false
  :thread_safe_sockets => true
  :uri_parser          => URI
  :versions            => "excon/0.45.3 (x86_64-linux) ruby/2.1.6"
  :write_timeout       => 60
excon.error
  :error => #<Excon::Errors::Forbidden: Expected(200) <=> Actual(403 Forbidden)
excon.error.request
  :chunk_size          => 1048576
  :ciphers             => "HIGH:!SSLv2:!aNULL:!eNULL:!3DES"
  :connect_timeout     => 60
  :debug_request       => true
  :debug_response      => true
  :expects             => 200
  :headers             => {
    "Authorization" => "REDACTED"
    "Date"          => "Fri, 05 Jun 2015 07:32:53 +0000"
    "Host"          => "myhitachi.cloud.local"
    "User-Agent"    => "fog/1.30.0 fog-core/1.30.0"
  }
  :host                => "myhitachi.cloud.local"
  :hostname            => "myhitachi.cloud.local"
  :idempotent          => true
  :instrumentor        => Excon::StandardInstrumentor
  :instrumentor_name   => "excon"
  :method              => "GET"
  :middlewares         => [
    Excon::Middleware::ResponseParser
    Excon::Middleware::Expects
    Excon::Middleware::Idempotent
  Excon::Middleware::Instrumentor
  Excon::Middleware::Mock
  ]
  :mock                => false
  :nonblock            => true
  :omit_default_port   => true
  :path                => "/my-bucket/"
  :persistent          => false
  :port                => 80
  :query               => {
  "location" => nil
  }
  :read_timeout        => 60
  :response_block      => #<Proc:0x00000005004470@/usr/local/rvm/gems/ruby-2.1.6/gems/fog-xml-0.1.2/lib/
  fog/xml/sax_parser_connection.rb:29 (lambda)>
  :retries_remaining   => 1
  :retry_limit         => 4
  :scheme              => "http"
  :ssl_verify_peer     => false
  :tcp_nodelay         => false
  :thread_safe_sockets => true
  :uri_parser          => URI
  :versions            => "excon/0.45.3 (x86_64-linux) ruby/2.1.6"
  :write_timeout       => 60
  excon.error.response
  :body          => "<?xml version='1.0' encoding='UTF-8'?>\n<Error>\n  <Code>AccessDenied</Code>\n  <Me
  ssage>Access Denied</Message>\n  <RequestId>1433489568948</RequestId>\n  <HostId>aGRpLmJvc3RoY3AuY2xvdWQ
  uY29ycDoyMQ==</HostId>\n</Error>\n\n"
  :headers       => {
  "Content-Type" => "application/xml;charset=UTF-8"
  "Date"         => "Fri, 05 Jun 2015 07:32:48 GMT"
  "Server"       => "HCP V7.1.0.10"
  }
  :local_address => "10.0.5.2"
  :local_port    => 53562
  :reason_phrase => "Forbidden"
  :remote_ip     => "192.168.7.33"
  :status        => 403
  :status_line   => "HTTP/1.1 403 Forbidden\r\n"
>
Excon::Errors::Forbidden: Expected(200) <=> Actual(403 Forbidden)
excon.error.request
  :chunk_size          => 1048576
  :ciphers             => "HIGH:!SSLv2:!aNULL:!eNULL:!3DES"
  :connect_timeout     => 60
  :debug_request       => true
  :debug_response      => true
  :expects             => 200
  :headers             => {
    "Authorization" => "REDACTED"
    "Date"          => "Fri, 05 Jun 2015 07:32:53 +0000"
    "Host"          => "myhitachi.cloud.local"
    "User-Agent"    => "fog/1.30.0 fog-core/1.30.0"
  }
  :host                => "myhitachi.cloud.local"
  :hostname            => "myhitachi.cloud.local"
  :idempotent          => true
  :instrumentor        => Excon::StandardInstrumentor
  :instrumentor_name   => "excon"
  :method              => "GET"
  :middlewares         => [
    Excon::Middleware::ResponseParser
    Excon::Middleware::Expects
    Excon::Middleware::Idempotent
    Excon::Middleware::Instrumentor
    Excon::Middleware::Mock
  ]
  :mock                => false
  :nonblock            => true
  :omit_default_port   => true
  :path                => "/my-bucket/"
  :persistent          => false
  :port                => 80
  :query               => {
    "location" => nil
  }
  :read_timeout        => 60
  :response_block      => #<Proc:0x00000005004470@/usr/local/rvm/gems/ruby-2.1.6/gems/fog-xml-0.1.2/lib/
fog/xml/sax_parser_connection.rb:29 (lambda)>
  :retries_remaining   => 1
  :retry_limit         => 4
  :scheme              => "http"
  :ssl_verify_peer     => false
  :tcp_nodelay         => false
  :thread_safe_sockets => true
  :uri_parser          => URI
  :versions            => "excon/0.45.3 (x86_64-linux) ruby/2.1.6"
  :write_timeout       => 60
  excon.error.response
  :body          => "<?xml version='1.0' encoding='UTF-8'?>\n<Error>\n  <Code>AccessDenied</Code>\n  <Me
  ssage>Access Denied</Message>\n  <RequestId>1433489568948</RequestId>\n  <HostId>aGRpLmJvc3RoY3AuY2xvdWQ
  uY29ycDoyMQ==</HostId>\n</Error>\n\n"
  :headers       => {
    "Content-Type" => "application/xml;charset=UTF-8"
    "Date"         => "Fri, 05 Jun 2015 07:32:48 GMT"
    "Server"       => "HCP V7.1.0.10"
  }
  :local_address => "10.0.5.2"
  :local_port    => 53562
  :reason_phrase => "Forbidden"
  :remote_ip     => "192.168.7.33"
  :status        => 403
  :status_line   => "HTTP/1.1 403 Forbidden\r\n"

If I have understood it well, it does list the containers but gets the error while trying to get information about particular containers, am I right?

Thank you very much.

@palonsoro
Copy link
Author

I finally was able to get rid of 403. It was a permissions issue. I was given a minimum set of permissions that seemed not to be enough.

So, as it has been a bit complicated to get this working, I am going to summarize everything we found (as well as some other things I found on my own previously) for future reference:

  • To avoid redirections
    • Use omit_default_port => true at connection options.
    • Use path_style => false at the connection hash.
  • To avoid a bad request error
    • Use aws_signature_version => '2' at the connection hash.
  • To avoid getting a 'forbidden' error
    • Give the following permissions at the bucket in Hitachi: Browse, read, write, delete, search, read ACL, write ACL. We have not tried to reduce this list, but it might be reduced a bit.

Thank you very much for your help.

@lanej
Copy link
Member

lanej commented Jun 10, 2015

@palonsoro thanks for the summary doc. good luck!

@geemus
Copy link
Member

geemus commented Jun 16, 2015

Thanks for the detailed report on what ended up working, glad to hear you found your way through.

@shrenuj-dm
Copy link

shrenuj-dm commented Sep 26, 2021

It was nice to see that Hitachi now allows you to register for a HCP trial - https://trycontent.hitachivantara.com/

While this thread has been helpful, I am adding some notes which hopefully will save someone's time.

In order to make our application compatible with Hitachi Vantara, we had to make the following changes

  • update keys and add host in the configuration, so the correct connection gets established
"provider"=>"AWS",
"aws_access_key_id"=>"wEIbbsH-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-4i6Q3MlrXiKu4F0ptWbO2g-OtCqY",
"aws_secret_access_key"=>"f2eb98ee-xxxx-xxxx-xxxx-xxxxxxx9f0c99",
"region"=>"us-east-1",
"host"=>"tryhcpforcloudscale.hitachivantara.com"
> MimeMagic.by_magic(<file-data>)
  • 403 Forbidden errors can also appear when updating an existing object. Not sure why but stripping the "Cache-Control" headers from the call does the trick.

@geemus
Copy link
Member

geemus commented Sep 27, 2021

@shrenuj-dm Glad you got this working and thanks for taking the time to share what you've learned with us!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants