Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The GW responses with CreateBucket operation: Access Denied when a user creates a bucket with enabled object-lock. #772

Closed
masterSplinter01 opened this issue Mar 27, 2023 · 2 comments · Fixed by #795
Assignees
Milestone

Comments

@masterSplinter01
Copy link
Contributor

masterSplinter01 commented Mar 27, 2023

$ aws s3 ls --endpoint https://s3.neofs.devenv:8080 --no-verify-ssl                                                                                                        
urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.neofs.devenv'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
2023-03-27 15:29:58 b2bf2a65-d6b2-4601-9b77-b2182ce2cb2a
2023-03-27 15:28:02 7MPEzn6hdtQvDrwtS98qtBByhEPzhribmisD3p9uiFN4
2023-03-27 15:31:47 test
2023-03-27 15:33:13 hmmm

$ aws --no-verify-ssl --no-paginate s3api create-bucket --bucket heh  --object-lock-enabled-for-bucket --endpoint https://s3.neofs.devenv:8080 --acl public-read-write
urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.neofs.devenv'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied.

 $ aws s3 ls --endpoint https://s3.neofs.devenv:8080 --no-verify-ssl                                                                                                   
urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.neofs.devenv'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

# Access denied or not??? 
2023-03-27 15:33:44 heh       

2023-03-27 15:29:58 b2bf2a65-d6b2-4601-9b77-b2182ce2cb2a
2023-03-27 15:28:02 7MPEzn6hdtQvDrwtS98qtBByhEPzhribmisD3p9uiFN4
2023-03-27 15:31:47 test
2023-03-27 15:33:13 hmmm

Oh, I get it. Parameter --object-lock-enabled-for-bucket caused receiving of AccessDenied.

$ aws --no-verify-ssl --no-paginate s3api create-bucket --bucket test  --endpoint https://s3.neofs.devenv:8080 --acl public-read-write
urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.neofs.devenv'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

$ aws s3 ls --endpoint https://s3.neofs.devenv:8080 --no-verify-ssl                                                                   
urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.neofs.devenv'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
2023-03-27 17:37:24 test
2023-03-27 17:36:22 C6cWHmbsBwVR927beD8KsxuGtwa7rZLLL1ngZN6gTvNq

$ aws --no-verify-ssl --no-paginate s3api create-bucket --bucket lock  --object-lock-enabled-for-bucket --endpoint https://s3.neofs.devenv:8080 --acl public-read-write
urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.neofs.devenv'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied.

$ aws s3 ls --endpoint https://s3.neofs.devenv:8080 --no-verify-ssl                                                                                                    
urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.neofs.devenv'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
2023-03-27 17:38:24 lock
2023-03-27 17:37:24 test
2023-03-27 17:36:22 C6cWHmbsBwVR927beD8KsxuGtwa7rZLLL1ngZN6gTvNq

But this response still seems confusing.

Originally posted by @masterSplinter01 in nspcc-dev/neofs-testcases#521 (comment)

@masterSplinter01 masterSplinter01 changed the title The GW responses with CreateBucket operation: Access Denied when a user creates bucket with enabled object-lock. The GW responses with CreateBucket operation: Access Denied when a user creates a bucket with enabled object-lock. Mar 27, 2023
@roman-khimov roman-khimov modified the milestones: v0.27.1, v0.27.2 Jun 13, 2023
@smallhive smallhive self-assigned this Jul 11, 2023
@smallhive
Copy link
Contributor

smallhive commented Jul 12, 2023

The problem is connected to --object-lock-enabled-for-bucket. When this flag is set, the gate makes an extra request to the tree service to put some information.
This requests returns deny error (logs from gate):

2023-07-12T15:56:45.603+0400    info    api/router.go:162       call method     {"status": 200, "host": "localhost:9080", "request_id": "b4a32776-61ff-461d-859e-6f1b5c6b1ed7", "method": "CreateBucket", "bucket": "heh1689163002", "object": "", "description": "OK"}

2023-07-12T15:57:26.969+0400    debug   layer/object.go:252     put object      {"reqId": "69c5350e-9dba-4a45-95c5-270baa87807b", "bucket": "heh1689163002", "cid": "4RQiEUofcYZq5wJpskaLNo81X9b8xLtRJb41E9mMoRsJ", "object": "acc.wallet.json", "oid": "GZpQfJLQAXogKrJ9RwLQCS13xasN6tyskEkdXAyBypTN"}

2023-07-12T15:57:26.972+0400    error   handler/util.go:29      call method     {"status": 500, "request_id": "69c5350e-9dba-4a45-95c5-270baa87807b", "method": "PutObject", "bucket": "heh1689163002", "object": "acc.wallet.json", "description": "could not upload object", "error": "couldn't add new verion to tree service: not found: rpc error: code = Unknown desc = access to operation PUT is denied by extended ACL check: not found allowing rules for the request", "body close errors": []}

We see the container was created successfully. But the extra request failed.
Meanwhile, the container has good eacl. It allows to write to container for everyone:

{
    "version": {
        "major": 2,
        "minor": 13
    },
    "containerID": null,
    "records": [
        {"operation": "GET", "action": "ALLOW", "filters": [], "targets": [{"role": "ROLE_UNSPECIFIED", "keys": ["Axx/h0oGcnwlvogMVqQJMTu9JydWIRUxp/ZtIUJnshom"]}]},
        {"operation": "HEAD", "action": "ALLOW", "filters": [], "targets": [{"role": "ROLE_UNSPECIFIED", "keys": ["Axx/h0oGcnwlvogMVqQJMTu9JydWIRUxp/ZtIUJnshom"]}]},
        {"operation": "PUT", "action": "ALLOW", "filters": [], "targets": [{"role": "ROLE_UNSPECIFIED", "keys": ["Axx/h0oGcnwlvogMVqQJMTu9JydWIRUxp/ZtIUJnshom"]}]},
        {"operation": "DELETE", "action": "ALLOW", "filters": [], "targets": [{"role": "ROLE_UNSPECIFIED", "keys": ["Axx/h0oGcnwlvogMVqQJMTu9JydWIRUxp/ZtIUJnshom"]}]},
        {"operation": "SEARCH", "action": "ALLOW", "filters": [], "targets": [{"role": "ROLE_UNSPECIFIED", "keys": ["Axx/h0oGcnwlvogMVqQJMTu9JydWIRUxp/ZtIUJnshom"]}]},
        {"operation": "GETRANGE", "action": "ALLOW", "filters": [], "targets": [{"role": "ROLE_UNSPECIFIED", "keys": ["Axx/h0oGcnwlvogMVqQJMTu9JydWIRUxp/ZtIUJnshom"]}]},
        {"operation": "GETRANGEHASH", "action": "ALLOW", "filters": [], "targets": [{"role": "ROLE_UNSPECIFIED", "keys": ["Axx/h0oGcnwlvogMVqQJMTu9JydWIRUxp/ZtIUJnshom"]}]},
        {"operation": "GET", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
        {"operation": "HEAD", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
        {"operation": "PUT", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
        {"operation": "DELETE", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
        {"operation": "SEARCH", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
        {"operation": "GETRANGE", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
        {"operation": "GETRANGEHASH", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]}
    ]
}

Also even without --object-lock-enabled-for-bucket we can't upload objects, we have a similar error:

2023-07-12T15:57:28.277+0400    debug   layer/object.go:252     put object      {"reqId": "08183346-c189-462f-85da-b35c5f1e5dbb", "bucket": "heh1689163002", "cid": "4RQiEUofcYZq5wJpskaLNo81X9b8xLtRJb41E9mMoRsJ", "object": "acc.wallet.json", "oid": "HpYd3qtKHdfUqEtdGagrcHapJvj4oLxuu5uGxrYLWV8U"}

2023-07-12T15:57:28.280+0400    error   handler/util.go:29      call method     {"status": 500, "request_id": "08183346-c189-462f-85da-b35c5f1e5dbb", "method": "PutObject", "bucket": "heh1689163002", "object": "acc.wallet.json", "description": "could not upload object", "error": "couldn't add new version to tree service: not found: rpc error: code = Unknown desc = access to operation PUT is denied by extended ACL check: not found allowing rules for the request", "body close errors": []}

Bearer token, used to checking:

{
    "records": [
        {"operation": "PUT", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
        {"operation": "GET", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
        {"operation": "HEAD", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
        {"operation": "DELETE", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
        {"operation": "SEARCH", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
        {"operation": "GETRANGE", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]},
        {"operation": "GETRANGEHASH", "action": "ALLOW", "filters": [], "targets": [{"role": "OTHERS", "keys": []}]}
    ]
}

@cthulhu-rider
Copy link
Contributor

i noticed one mistake in the code

func handleError(msg string, err error) error {
if strings.Contains(err.Error(), "not found") {
return fmt.Errorf("%w: %s", layer.ErrNodeNotFound, err.Error())
} else if strings.Contains(err.Error(), "is denied by") {
return fmt.Errorf("%w: %s", layer.ErrNodeAccessDenied, err.Error())
}
return fmt.Errorf("%s: %w", msg, err)
}

and it will work incorrectly for the error mentioned by @smallhive

rpc error: code = Unknown desc = access to operation PUT is denied by extended ACL check: not found allowing rules for the request contains both not found and is denied by substrings, so handleError will return wrapped layer.ErrNodeNotFound which is incorrect

smallhive added a commit that referenced this issue Jul 13, 2023
Clarify which waller should be used in some situations.

close #772

Signed-off-by: Evgenii Baidakov <[email protected]>
roman-khimov added a commit that referenced this issue Jul 18, 2023
After another iteration and correct wallet/secret generation, looks like
the problem is not reproducing. A bucket is created successfully with
`--object-lock-enabled-for-bucket` flag and without it. Also,
upload/download files work properly without errors.

According to this, updated documentation to clarify which wallet should
be used in each situation

close #772
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants