Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

settings & storage-system: runtime error: invalid memory address or nil pointer dereference #4961

Closed
NexZhu opened this issue Nov 3, 2022 · 5 comments
Closed

settings & storage-system: runtime error: invalid memory address or nil pointer dereference #4961

NexZhu opened this issue Nov 3, 2022 · 5 comments
Assignees
@butonic
Labels
Type:Bug

Comments

@NexZhu
Copy link

NexZhu commented Nov 3, 2022

Describe the bug

I deployed the latest version owncloud/ocis:2.0.0-beta.8 with the official Helm chart and configured it to authenticate users with an OIDC provider Authelia instance, here's what happens:

  1. Open owncloud URL, redirected to Authelia.
  2. Login and clickAccept in the OAuth2 consent page.
  3. Redirected to https://owncloud.example.com/oidc-callback.html?code=a0j4L6_ZuH2QbH5mPFJJbCVLH-OO9jIJg5mscuxPSiw.oqHE9PYJP1SnVboJth46PyRES17YKo9FT2WyONQhSTg&scope=openid+profile+email&state=9d6a21e638474d40bdaab11ebfb27088
  4. Browser sent XHR request to https://owncloud.example.com/ocs/v1.php/cloud/user with header: authorization: Bearer p9UAAo2wIMfb8RLlN_f7dW9u5mpxUl76IfUH3jAMEos.4b8WcEj1mLZOen34w719wsjoJa5tpq_ArXvub4M6NrY, resulted in 500 status code.
  5. I'm redirected back to OAuth provider's consent page.

Error logs

settings pod:

2022-10-28T08:31:37Z DBG setup cors middleware allow_credentials=true allowed_headers="Authorization, Origin, Content-Type, Accept, X-Requested-With" allowed_methods="GET, POST, PUT, PATCH, DELETE, OPTIONS" allowed_origins=* service=settings
2022-10-28T08:33:38Z DBG bytes=219330 duration=3.629855 method=GET path=/settings.js proto=HTTP/1.1 request= service=settings status=200
2022-10-28T08:33:40Z ERR error initializing metadata client error="rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference" service=ocis
2022-10-28T08:33:40Z ERR error initializing metadata client error="rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference" service=ocis
2022-10-28T08:34:00Z DBG bytes=219330 duration=0.146869 method=GET path=/settings.js proto=HTTP/1.1 request= service=settings status=200
2022-10-28T08:34:11Z DBG bytes=219330 duration=0.188705 method=GET path=/settings.js proto=HTTP/1.1 request= service=settings status=200
2022-10-28T08:34:11Z ERR error initializing metadata client error="rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference" service=ocis
2022-10-28T08:34:11Z ERR error initializing metadata client error="rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference" service=ocis

storage-system pod:

2022-10-28T08:50:33Z INF user idp:"internal" opaque_id:"4735a501-2b40-4ddc-a238-58727bc72ad2" type:USER_TYPE_PRIMARY  authenticated pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:53452 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=681650 traceid=00000000000000000000000000000000 uri=/cs3.auth.provider.v1beta1.ProviderAPI/Authenticate user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://172.16.11.148:57038 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=1301450 traceid=00000000000000000000000000000000 uri=/cs3.gateway.v1beta1.GatewayAPI/Authenticate user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:52036 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=72744 traceid=00000000000000000000000000000000 uri=/cs3.permissions.v1beta1.PermissionsAPI/CheckPermission user-agent=grpc-go/1.49.0
goroutine 1609 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	runtime/debug/stack.go:16 +0x19
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x3cb11e0, 0xc000e6bc80}, {0x38b5f40, 0x4fecfc0})
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/recovery/recovery.go:48 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x3cb11e0?, 0xc000e6bc80?}, {0x38b5f40?, 0x4fecfc0?}, 0xc001316558?)
	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:29 +0x7b
panic({0x38b5f40, 0x4fecfc0})
	runtime/panic.go:838 +0x207
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).InternalPath(...)
	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/node/node.go:465
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).CreateStorageSpace(0xc0000c2050, {0x3cb11e0, 0xc0010e4150}, 0xc012d160c0)
	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/spaces.go:95 +0x4b0
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).CreateStorageSpace(0xc000112000, {0x3cb11e0, 0xc0010e4150}, 0xc012d160c0)
	github.com/cs3org/reva/[email protected]/internal/grpc/services/storageprovider/storageprovider.go:469 +0x49
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler.func1({0x3cb11e0, 0xc0010e4150}, {0x3b971c0?, 0xc012d160c0})
	github.com/cs3org/[email protected]/cs3/storage/provider/v1beta1/provider_api.pb.go:6441 +0x78
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x3cb11e0, 0xc0010e4060}, {0x3b971c0, 0xc012d160c0}, 0xc0014071c0, 0xc000c13fe0)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:325 +0x664
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0010e4060?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func1({0x3cb11e0, 0xc000e6bc80}, {0x3b971c0, 0xc012d160c0}, 0xc0014071c0, 0xc0014071e0)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/auth/auth.go:145 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?}, 0xc000edf2b8?, 0x4905b7?)
	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x3cb11e0, 0xc000e6bc80}, {0x3b971c0, 0xc012d160c0}, 0xc0014071c0, 0xc001407220)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x3cb11e0, 0xc000e6bc80}, {0x3b971c0, 0xc012d160c0}, 0x1?, 0xc001407240)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/useragent/useragent.go:38 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x3cb11e0, 0xc000e6b620}, {0x3b971c0, 0xc012d160c0}, 0x5191050?, 0xc001407260)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/token/token.go:44 +0x158
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6b620?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x3cb11e0, 0xc000e6b110}, {0x3b971c0, 0xc012d160c0}, 0x7f1171c7c3a8?, 0xc001407280)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6b110?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x3cb11e0, 0xc000e6b110}, {0x3b971c0, 0xc012d160c0}, 0xc001325ac8?, 0x38b4c20?)
	github.com/grpc-ecosystem/[email protected]/chain.go:34 +0xbf
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler({0x3c434e0?, 0xc000112000}, {0x3cb11e0, 0xc000e6b110}, 0xc00029eb60, 0xc0010f2120)
	github.com/cs3org/[email protected]/cs3/storage/provider/v1beta1/provider_api.pb.go:6443 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc000044480, 0xc0010f2570, 0x500e150, 0x0)
	google.golang.org/[email protected]/server.go:1301 +0xb0b
google.golang.org/grpc.(*Server).handleStream(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc000044480, 0x0)
	google.golang.org/[email protected]/server.go:1642 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/[email protected]/server.go:938 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/[email protected]/server.go:936 +0x28a
2022-10-28T08:50:33Z ERR runtime error: invalid memory address or nil pointer dereference; stack: goroutine 1609 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x3cb11e0, 0xc000e6bc80}, {0x38b5f40, 0x4fecfc0})
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/recovery/recovery.go:50 +0x65
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x3cb11e0?, 0xc000e6bc80?}, {0x38b5f40?, 0x4fecfc0?}, 0xc001316558?)
	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:29 +0x7b
panic({0x38b5f40, 0x4fecfc0})
	runtime/panic.go:838 +0x207
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).InternalPath(...)
	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/node/node.go:465
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).CreateStorageSpace(0xc0000c2050, {0x3cb11e0, 0xc0010e4150}, 0xc012d160c0)
	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/spaces.go:95 +0x4b0
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).CreateStorageSpace(0xc000112000, {0x3cb11e0, 0xc0010e4150}, 0xc012d160c0)
	github.com/cs3org/reva/[email protected]/internal/grpc/services/storageprovider/storageprovider.go:469 +0x49
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler.func1({0x3cb11e0, 0xc0010e4150}, {0x3b971c0?, 0xc012d160c0})
	github.com/cs3org/[email protected]/cs3/storage/provider/v1beta1/provider_api.pb.go:6441 +0x78
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x3cb11e0, 0xc0010e4060}, {0x3b971c0, 0xc012d160c0}, 0xc0014071c0, 0xc000c13fe0)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:325 +0x664
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0010e4060?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func1({0x3cb11e0, 0xc000e6bc80}, {0x3b971c0, 0xc012d160c0}, 0xc0014071c0, 0xc0014071e0)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/auth/auth.go:145 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?}, 0xc000edf2b8?, 0x4905b7?)
	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x3cb11e0, 0xc000e6bc80}, {0x3b971c0, 0xc012d160c0}, 0xc0014071c0, 0xc001407220)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x3cb11e0, 0xc000e6bc80}, {0x3b971c0, 0xc012d160c0}, 0x1?, 0xc001407240)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/useragent/useragent.go:38 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6bc80?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x3cb11e0, 0xc000e6b620}, {0x3b971c0, 0xc012d160c0}, 0x5191050?, 0xc001407260)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/token/token.go:44 +0x158
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6b620?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x3cb11e0, 0xc000e6b110}, {0x3b971c0, 0xc012d160c0}, 0x7f1171c7c3a8?, 0xc001407280)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc000e6b110?}, {0x3b971c0?, 0xc012d160c0?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x3cb11e0, 0xc000e6b110}, {0x3b971c0, 0xc012d160c0}, 0xc001325ac8?, 0x38b4c20?)
	github.com/grpc-ecosystem/[email protected]/chain.go:34 +0xbf
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler({0x3c434e0?, 0xc000112000}, {0x3cb11e0, 0xc000e6b110}, 0xc00029eb60, 0xc0010f2120)
	github.com/cs3org/[email protected]/cs3/storage/provider/v1beta1/provider_api.pb.go:6443 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc000044480, 0xc0010f2570, 0x500e150, 0x0)
	google.golang.org/[email protected]/server.go:1301 +0xb0b
google.golang.org/grpc.(*Server).handleStream(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc000044480, 0x0)
	google.golang.org/[email protected]/server.go:1642 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/[email protected]/server.go:938 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/[email protected]/server.go:936 +0x28a
 pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z ERR unary code=Internal end="28/Oct/2022:08:50:33 +0000" from=tcp://172.16.11.148:57040 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=1297155 traceid=00000000000000000000000000000000 uri=/cs3.storage.provider.v1beta1.ProviderAPI/CreateStorageSpace user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/Authenticate pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG skipping auth method=/cs3.auth.registry.v1beta1.RegistryAPI/GetAuthProviders pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:53450 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=53140 traceid=00000000000000000000000000000000 uri=/cs3.auth.registry.v1beta1.RegistryAPI/GetAuthProviders user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG skipping auth method=/cs3.auth.provider.v1beta1.ProviderAPI/Authenticate pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/GetUserByClaim pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG skipping auth method=/cs3.identity.user.v1beta1.UserAPI/GetUserByClaim pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:53456 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=70877 traceid=00000000000000000000000000000000 uri=/cs3.identity.user.v1beta1.UserAPI/GetUserByClaim user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:53454 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=478103 traceid=00000000000000000000000000000000 uri=/cs3.gateway.v1beta1.GatewayAPI/GetUserByClaim user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z INF user idp:"internal" opaque_id:"4735a501-2b40-4ddc-a238-58727bc72ad2" type:USER_TYPE_PRIMARY  authenticated pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:53452 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=907968 traceid=00000000000000000000000000000000 uri=/cs3.auth.provider.v1beta1.ProviderAPI/Authenticate user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://172.16.11.148:57038 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=1628545 traceid=00000000000000000000000000000000 uri=/cs3.gateway.v1beta1.GatewayAPI/Authenticate user-agent=grpc-go/1.49.0
2022-10-28T08:50:33Z DBG unary code=OK end="28/Oct/2022:08:50:33 +0000" from=tcp://127.0.0.1:52036 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=68173 traceid=00000000000000000000000000000000 uri=/cs3.permissions.v1beta1.PermissionsAPI/CheckPermission user-agent=grpc-go/1.49.0
goroutine 54 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	runtime/debug/stack.go:16 +0x19
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x3cb11e0, 0xc0013bd6b0}, {0x38b5f40, 0x4fecfc0})
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/recovery/recovery.go:48 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x3cb11e0?, 0xc0013bd6b0?}, {0x38b5f40?, 0x4fecfc0?}, 0xc001316558?)
	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:29 +0x7b
panic({0x38b5f40, 0x4fecfc0})
	runtime/panic.go:838 +0x207
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).InternalPath(...)
	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/node/node.go:465
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).CreateStorageSpace(0xc0000c2050, {0x3cb11e0, 0xc0013bd9e0}, 0xc000fe6e40)
	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/spaces.go:95 +0x4b0
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).CreateStorageSpace(0xc000112000, {0x3cb11e0, 0xc0013bd9e0}, 0xc000fe6e40)
	github.com/cs3org/reva/[email protected]/internal/grpc/services/storageprovider/storageprovider.go:469 +0x49
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler.func1({0x3cb11e0, 0xc0013bd9e0}, {0x3b971c0?, 0xc000fe6e40})
	github.com/cs3org/[email protected]/cs3/storage/provider/v1beta1/provider_api.pb.go:6441 +0x78
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x3cb11e0, 0xc0013bd8f0}, {0x3b971c0, 0xc000fe6e40}, 0xc0003a0be0, 0xc000cc2d38)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:325 +0x664
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd8f0?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func1({0x3cb11e0, 0xc0013bd6b0}, {0x3b971c0, 0xc000fe6e40}, 0xc0003a0be0, 0xc0003a0c00)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/auth/auth.go:145 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?}, 0xc000f772b8?, 0x4905b7?)
	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x3cb11e0, 0xc0013bd6b0}, {0x3b971c0, 0xc000fe6e40}, 0xc0003a0be0, 0xc0003a0c40)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x3cb11e0, 0xc0013bd6b0}, {0x3b971c0, 0xc000fe6e40}, 0x1?, 0xc0003a0c60)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/useragent/useragent.go:38 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x3cb11e0, 0xc0013bd620}, {0x3b971c0, 0xc000fe6e40}, 0x5191050?, 0xc0003a0c80)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/token/token.go:44 +0x158
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd620?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x3cb11e0, 0xc0013bd530}, {0x3b971c0, 0xc000fe6e40}, 0x7f11718a25e0?, 0xc0003a0ca0)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd530?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x3cb11e0, 0xc0013bd530}, {0x3b971c0, 0xc000fe6e40}, 0xc012d1aac8?, 0x38b4c20?)
	github.com/grpc-ecosystem/[email protected]/chain.go:34 +0xbf
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler({0x3c434e0?, 0xc000112000}, {0x3cb11e0, 0xc0013bd530}, 0xc0003a44d0, 0xc0010f2120)
	github.com/cs3org/[email protected]/cs3/storage/provider/v1beta1/provider_api.pb.go:6443 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc00029b200, 0xc0010f2570, 0x500e150, 0x0)
	google.golang.org/[email protected]/server.go:1301 +0xb0b
google.golang.org/grpc.(*Server).handleStream(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc00029b200, 0x0)
	google.golang.org/[email protected]/server.go:1642 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/[email protected]/server.go:938 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/[email protected]/server.go:936 +0x28a
2022-10-28T08:50:33Z ERR runtime error: invalid memory address or nil pointer dereference; stack: goroutine 54 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x3cb11e0, 0xc0013bd6b0}, {0x38b5f40, 0x4fecfc0})
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/recovery/recovery.go:50 +0x65
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x3cb11e0?, 0xc0013bd6b0?}, {0x38b5f40?, 0x4fecfc0?}, 0xc001316558?)
	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:29 +0x7b
panic({0x38b5f40, 0x4fecfc0})
	runtime/panic.go:838 +0x207
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).InternalPath(...)
	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/node/node.go:465
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).CreateStorageSpace(0xc0000c2050, {0x3cb11e0, 0xc0013bd9e0}, 0xc000fe6e40)
	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/spaces.go:95 +0x4b0
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).CreateStorageSpace(0xc000112000, {0x3cb11e0, 0xc0013bd9e0}, 0xc000fe6e40)
	github.com/cs3org/reva/[email protected]/internal/grpc/services/storageprovider/storageprovider.go:469 +0x49
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler.func1({0x3cb11e0, 0xc0013bd9e0}, {0x3b971c0?, 0xc000fe6e40})
	github.com/cs3org/[email protected]/cs3/storage/provider/v1beta1/provider_api.pb.go:6441 +0x78
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x3cb11e0, 0xc0013bd8f0}, {0x3b971c0, 0xc000fe6e40}, 0xc0003a0be0, 0xc000cc2d38)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:325 +0x664
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd8f0?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func1({0x3cb11e0, 0xc0013bd6b0}, {0x3b971c0, 0xc000fe6e40}, 0xc0003a0be0, 0xc0003a0c00)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/auth/auth.go:145 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?}, 0xc000f772b8?, 0x4905b7?)
	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x3cb11e0, 0xc0013bd6b0}, {0x3b971c0, 0xc000fe6e40}, 0xc0003a0be0, 0xc0003a0c40)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x3cb11e0, 0xc0013bd6b0}, {0x3b971c0, 0xc000fe6e40}, 0x1?, 0xc0003a0c60)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/useragent/useragent.go:38 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd6b0?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x3cb11e0, 0xc0013bd620}, {0x3b971c0, 0xc000fe6e40}, 0x5191050?, 0xc0003a0c80)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/token/token.go:44 +0x158
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd620?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x3cb11e0, 0xc0013bd530}, {0x3b971c0, 0xc000fe6e40}, 0x7f11718a25e0?, 0xc0003a0ca0)
	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3cb11e0?, 0xc0013bd530?}, {0x3b971c0?, 0xc000fe6e40?})
	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x3cb11e0, 0xc0013bd530}, {0x3b971c0, 0xc000fe6e40}, 0xc012d1aac8?, 0x38b4c20?)
	github.com/grpc-ecosystem/[email protected]/chain.go:34 +0xbf
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler({0x3c434e0?, 0xc000112000}, {0x3cb11e0, 0xc0013bd530}, 0xc0003a44d0, 0xc0010f2120)
	github.com/cs3org/[email protected]/cs3/storage/provider/v1beta1/provider_api.pb.go:6443 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc00029b200, 0xc0010f2570, 0x500e150, 0x0)
	google.golang.org/[email protected]/server.go:1301 +0xb0b
google.golang.org/grpc.(*Server).handleStream(0xc000000b40, {0x3cbfd38, 0xc000cf1380}, 0xc00029b200, 0x0)
	google.golang.org/[email protected]/server.go:1642 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/[email protected]/server.go:938 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/[email protected]/server.go:936 +0x28a
 pkg=rgrpc service=storage-system traceid=00000000000000000000000000000000
2022-10-28T08:50:33Z ERR unary code=Internal end="28/Oct/2022:08:50:33 +0000" from=tcp://172.16.11.148:57040 pkg=rgrpc service=storage-system start="28/Oct/2022:08:50:33 +0000" time_ns=1112793 traceid=00000000000000000000000000000000 uri=/cs3.storage.provider.v1beta1.ProviderAPI/CreateStorageSpace user-agent=grpc-go/1.49.0
@NexZhu NexZhu added the Type:Bug label Nov 3, 2022
@rhafer rhafer mentioned this issue Nov 3, 2022
Closed
@butonic butonic mentioned this issue Nov 3, 2022
Merged
@butonic
Copy link
Member

butonic commented Nov 3, 2022

@NexZhu I added an error check to prevent the panic in cs3org/reva#3430. It seems we get an error from the os when trying to read the node. It might be a different ERRNO from the internal stat or getxatr call. Maybe the filesystem does not support xattrs? we will know more when trying with that PR.

@butonic butonic self-assigned this Nov 3, 2022
@NexZhu
Copy link
Author

NexZhu commented Nov 3, 2022 •
edited
Loading

@NexZhu I added an error check to prevent the panic in cs3org/reva#3430. It seems we get an error from the os when trying to read the node. It might be a different ERRNO from the internal stat or getxatr call. Maybe the filesystem does not support xattrs? we will know more when trying with that PR.

Yeah, I think you guessed right. I'm using Alibaba Cloud NAS and I think it doesn't support xattrs.

Is there a way to workaround this limitation in the software layer?

@micbar
Copy link
Contributor

micbar commented Nov 3, 2022

Is there a way to workaround this limitation in the software layer?

Unfortunately with the current ocis storage interface we need xattrs to store file metadata and access permissions.

@NexZhu
Copy link
Author

NexZhu commented Nov 3, 2022

I tested with volumes witch support xattrs, now storage-system and store pods are pending forever and I'm getting these errors, could you please take a look?

storage-system pod failed to start:

PodScheduled	False	2022-11-03 23:51:27	SchedulerError	running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

The only log is:

{"code":"SERVER_ERROR_CODE","message":"Cannot invoke method getContent() on null object","requestId":"b6f922f7-5bb0-46ae-b771-ad0ed239b49a","successResponse":false}

store pod, the same:

{"code":"SERVER_ERROR_CODE","message":"Cannot invoke method getContent() on null object","requestId":"f50b4b50-def7-4587-bee3-2c08193b3468","successResponse":false}

@rhafer
Copy link
Contributor

rhafer commented Nov 4, 2022

I tested with volumes witch support xattrs, now storage-system and store pods are pending forever and I'm getting these errors, could you please take a look?

storage-system pod failed to start:

PodScheduled	False	2022-11-03 23:51:27	SchedulerError	running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

To me this look like an error of your underlying infrastructure. Kubernetes fails to schedule the pod because of some issue with the volumes. I don't think there is much we can do about that. You might be able to find some more detail about what failed by taking a look at the volumes and pvc (using kubectl), or by looking at the lower level kubernetes logs (if you have access to that).

@NexZhu NexZhu closed this as completed Nov 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@butonic butonic

Labels
Type:Bug
Projects
Infinite Scale Team Board
Archived in project
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rhafer @butonic @NexZhu @micbar