Object is available only throught the gate it was uploaded.

[MaxGelbakhiani]

Environment

Test stand config as follows:
4 nodes running neofs-ir service
22 nodes running neofs-node service.
Each of the storage nodes runs neofs-s3-gw service.

root@max0-0402:~# neofs-node --version
NeoFS Storage node
Version: 0.38.1-175-g42cf34f4-dirty
GoVersion: go1.21.4

root@max0-0402:~# neofs-s3-gw --version
NeoFS S3 Gateway

Version: v0.29.0
GoVersion: go1.21.1

Configs from one of the hosts for neofs-node and neofs-s3-gw are attached:
neofs-node_config.txt
neofs-s3-gw_config.txt

Current Behavior

Test load consisted of a large number of PUT requests through 6 S3 gateways and the following phase with separate GETs.
On trying to get a stored object, it appeared that objects are available only from the gate it were uploaded.

The client-side (aws cli) returned 404 - object not found. In GW logs:

Nov 24 16:41:29 max0-0391.mb neofs-s3-gw[1578285]: 2023-11-24T16:41:29.278Z        info        layer/layer.go:651        resolve bucket        {"reqId": "3b7a674e-2238-481e-b34c-e326e06649ba", "bucket": "mds-23388-11-24", "cid": "DM5pdU9jiP8V2dghMF4cgrqWu9zvXUxNn9aRtJeBWac6"}
Nov 24 16:41:29 max0-0391.mb neofs-s3-gw[1578285]: 2023-11-24T16:41:29.282Z        error        handler/util.go:29        call method        {"status": 404, "request_id": "3b7a674e-2238-481e-b34c-e326e06649ba", "method": "GetObject", "bucket": "mds-23388-11-24", "object": "OkS3GkuN/3.yzphh4u)HOSdxaPh.rnd", "description": "could not find object", "error": "NoSuchKey: 404 => The specified key does not exist."}

But it's accessible via another GW used to put the object.

root@max0-0402:~# journalctl -f -u neofs-s3.service
-- Logs begin at Sun 2023-11-26 12:36:42 UTC. --


Nov 27 07:56:11 max0-0402.mb neofs-s3-gw[1517671]: 2023-11-27T07:56:11.007Z        debug        layer/layer.go:480        get object        {"reqId": "0d82660c-3949-46c6-9743-ef7cde14c153", "bucket": "mds-23388-11-24", "cid": "DM5pdU9jiP8V2dghMF4cgrqWu9zvXUxNn9aRtJeBWac6", "object": "OkS3GkuN/3.yzphh4u)HOSdxaPh.rnd", "oid": "H9o8PDeuDvgrHaL9s1NYedrxfNDiWVtfRhbDSew9KmSN"}
Nov 27 07:56:11 max0-0402.mb neofs-s3-gw[1517671]: 2023-11-27T07:56:11.133Z        info        api/router.go:162        call method        {"status": 200, "host": "max0-0402.mb:8084", "request_id": "0d82660c-3949-46c6-9743-ef7cde14c153", "method": "GetObject", "bucket": "mds-23388-11-24", "object": "OkS3GkuN/3.yzphh4u)HOSdxaPh.rnd", "description": "OK"}

Expected Behavior

To have an object be accessible from every S3 gate.

[MaxGelbakhiani]

Noticed, that

1. The expected behavior is restored in a while without external interference, and the objects are available via every S3-gw.
2. While the issue persisted, it only affected objects already stored on NeoFs. The new objects uploaded to the storage were available via every GW.