You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What is your environment(Kubernetes version, Fluid version, etc.)
k8s:v1.20.2, fluid: 0.4.0 Describe the bug
alluxio-fuse :alluxio-fuse容器creating失败,好像挂载和attach volume失败导致
容器状态如下:
[root@VM-0-7-centos kubernetes]# kubectl get pods
NAME READY STATUS RESTARTS AGE
hbase-fuse-xc75n 0/1 ContainerCreating 0 8h
hbase-fuse-xl7rw 0/1 ContainerCreating 0 8h
hbase-master-0 2/2 Running 0 8h
hbase-worker-8hrxl 2/2 Running 0 8h
hbase-worker-lm2qb 2/2 Running 0 8h
kubelete组件日志如下:
Feb 28 13:58:24 VM-0-7-centos kubelet[28984]: E0228 13:58:24.187806 28984 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/host-path/d7fbe499-3e98-43d1-91be-2d3f6a914e31-alluxio-fuse-device podName:d7fbe499-3e98-43d1-91be-2d3f6a914e31 nodeName:}" failed. No retries permitted until 2021-02-28 14:00:26.187775008 +0800 CST m=+1205.008960846 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume "alluxio-fuse-device" (UniqueName: "kubernetes.io/host-path/d7fbe499-3e98-43d1-91be-2d3f6a914e31-alluxio-fuse-device") pod "hbase-fuse-xc75n" (UID: "d7fbe499-3e98-43d1-91be-2d3f6a914e31") : hostPath type check failed: /dev/fuse is not a file"
Feb 28 13:59:37 VM-0-7-centos kubelet[28984]: E0228 13:59:37.772414 28984 kubelet.go:1656] Unable to attach or mount volumes for pod "hbase-fuse-xc75n_default(d7fbe499-3e98-43d1-91be-2d3f6a914e31)": unmounted volumes=[alluxio-fuse-device], unattached volumes=[alluxio-fuse-device alluxio-fuse-mount mem default-token-5r284]: timed out waiting for the condition; skipping pod
Feb 28 13:59:37 VM-0-7-centos kubelet[28984]: E0228 13:59:37.772451 28984 pod_workers.go:191] Error syncing pod d7fbe499-3e98-43d1-91be-2d3f6a914e31 ("hbase-fuse-xc75n_default(d7fbe499-3e98-43d1-91be-2d3f6a914e31)"), skipping: unmounted volumes=[alluxio-fuse-device], unattached volumes=[alluxio-fuse-device alluxio-fuse-mount mem default-token-5r284]: timed out waiting for the condition
Feb 28 14:00:26 VM-0-7-centos kubelet[28984]: I0228 14:00:26.256973 28984 reconciler.go:269] operationExecutor.MountVolume started for volume "alluxio-fuse-device" (UniqueName: "kubernetes.io/host-path/d7fbe499-3e98-43d1-91be-2d3f6a914e31-alluxio-fuse-device") pod "hbase-fuse-xc75n" (UID: "d7fbe499-3e98-43d1-91be-2d3f6a914e31")
Feb 28 14:00:26 VM-0-7-centos kubelet[28984]: E0228 14:00:26.257111 28984 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/host-path/d7fbe499-3e98-43d1-91be-2d3f6a914e31-alluxio-fuse-device podName:d7fbe499-3e98-43d1-91be-2d3f6a914e31 nodeName:}" failed. No retries permitted until 2021-02-28 14:02:28.257080139 +0800 CST m=+1327.078265886 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume "alluxio-fuse-device" (UniqueName: "kubernetes.io/host-path/d7fbe499-3e98-43d1-91be-2d3f6a914e31-alluxio-fuse-device") pod "hbase-fuse-xc75n" (UID: "d7fbe499-3e98-43d1-91be-2d3f6a914e31") : hostPath type check failed: /dev/fuse is not a file"
Feb 28 14:00:27 VM-0-7-centos kubelet[28984]: I0228 14:00:27.910602 28984 container_manager_linux.go:490] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Feb 28 14:01:53 VM-0-7-centos kubelet[28984]: E0228 14:01:53.772048 28984 kubelet.go:1656] Unable to attach or mount volumes for pod "hbase-fuse-xc75n_default(d7fbe499-3e98-43d1-91be-2d3f6a914e31)": unmounted volumes=[alluxio-fuse-device], unattached volumes=[alluxio-fuse-device alluxio-fuse-mount mem default-token-5r284]: timed out waiting for the condition; skipping pod
Feb 28 14:01:53 VM-0-7-centos kubelet[28984]: E0228 14:01:53.772103 28984 pod_workers.go:191] Error syncing pod d7fbe499-3e98-43d1-91be-2d3f6a914e31 ("hbase-fuse-xc75n_default(d7fbe499-3e98-43d1-91be-2d3f6a914e31)"), skipping: unmounted volumes=[alluxio-fuse-device], unattached volumes=[alluxio-fuse-device alluxio-fuse-mount mem default-token-5r284]: timed out waiting for the condition
alluxio-worker:
[root@VM-0-7-centos ~]# kubectl logs hbase-worker-8hrxl alluxio-worker
2021-02-28 05:44:05,542 INFO NettyUtils - EPOLL_MODE is available
2021-02-28 05:44:05,993 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=172.31.0.16, rack=null)
2021-02-28 05:44:06,225 INFO BlockWorkerFactory - Creating alluxio.worker.block.BlockWorker
2021-02-28 05:44:06,308 INFO StorageDir - Folder /dev/shm/default/hbase/alluxioworker was created!
2021-02-28 05:44:06,323 WARN StorageTier - Failed to verify memory capacity
2021-02-28 05:44:06,391 INFO log - Logging initialized @1598ms
2021-02-28 05:44:06,642 INFO GrpcDataServer - Alluxio worker gRPC server started, listening on /0.0.0.0:20002
2021-02-28 05:44:06,643 INFO ProcessUtils - Starting Alluxio worker @172.31.0.16:20002.
2021-02-28 05:44:06,644 INFO MetricsSystem - Starting sinks with config: {}.
2021-02-28 05:44:06,822 INFO AlluxioWorkerProcess - Started Alluxio worker @172.31.0.16:20002 with id 5599774766131283346
2021-02-28 05:44:06,823 INFO WebServer - Alluxio worker web service starting @ /0.0.0.0:20003
2021-02-28 05:44:06,826 INFO Server - jetty-9.2.z-SNAPSHOT
2021-02-28 05:44:06,851 INFO ContextHandler - Started o.e.j.s.ServletContextHandler@24111ef1{/metrics/prometheus,null,AVAILABLE}
2021-02-28 05:44:06,852 INFO ContextHandler - Started o.e.j.s.ServletContextHandler@1fde5d22{/metrics/json,null,AVAILABLE}
2021-02-28 05:44:06,853 WARN SecurityHandler - [email protected]@7ca20101{/,null,STARTING} has uncovered http methods for path: /
2021-02-28 05:44:12,357 INFO ContextHandler - Started o.e.j.s.ServletContextHandler@7ca20101{/,null,AVAILABLE}
2021-02-28 05:44:12,364 INFO ServerConnector - Started ServerConnector@6999cd39{HTTP/1.1}{0.0.0.0:20003}
2021-02-28 05:44:12,364 INFO Server - Started @7572ms
2021-02-28 05:44:12,364 INFO WebServer - Alluxio worker web service started @ /0.0.0.0:20003
2021-02-28 05:44:12,364 INFO AlluxioWorkerProcess - Alluxio worker version 2.3.1-SNAPSHOT started. bindHost=0.0.0.0, connectHost=172.31.0.16, rpcPort=20002, webPort=20003
alluxio-job-worker:
[root@VM-0-7-centos ~]# kubectl logs hbase-worker-8hrxl alluxio-job-worker
2021-02-28 05:44:05,702 INFO NettyUtils - EPOLL_MODE is available
2021-02-28 05:44:06,458 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=172.31.0.16, rack=null)
2021-02-28 05:44:06,479 INFO log - Logging initialized @1466ms
2021-02-28 05:44:06,593 INFO ProcessUtils - Starting Alluxio job worker.
2021-02-28 05:44:06,594 INFO WebServer - Alluxio Job Manager Worker Web service starting @ /0.0.0.0:20007
2021-02-28 05:44:06,596 INFO Server - jetty-9.2.z-SNAPSHOT
2021-02-28 05:44:06,621 INFO ContextHandler - Started o.e.j.s.ServletContextHandler@72ade7e3{/metrics/json,null,AVAILABLE}
2021-02-28 05:44:06,622 WARN SecurityHandler - [email protected]@560348e6{/,null,STARTING} has uncovered http methods for path: /
2021-02-28 05:44:09,691 INFO ContextHandler - Started o.e.j.s.ServletContextHandler@560348e6{/,null,AVAILABLE}
2021-02-28 05:44:09,698 INFO ServerConnector - Started ServerConnector@36dce7ed{HTTP/1.1}{0.0.0.0:20007}
2021-02-28 05:44:09,699 INFO Server - Started @4686ms
2021-02-28 05:44:09,699 INFO WebServer - Alluxio Job Manager Worker Web service started @ /0.0.0.0:20007
2021-02-28 05:44:09,701 INFO MetricsSystem - Starting sinks with config: {}.
2021-02-28 05:44:09,892 INFO AlluxioJobWorkerProcess - Started Alluxio job worker with id 1614491014664
2021-02-28 05:44:09,893 INFO AlluxioJobWorkerProcess - Alluxio job worker version 2.3.1-SNAPSHOT started. bindHost=/0.0.0.0:20006, connectHost=172.31.0.16:20006, rpcPort=20006, webPort=20007
2021-02-28 05:44:09,894 INFO AlluxioJobWorkerProcess - Starting gRPC server on address 172.31.0.16:20006
2021-02-28 05:44:09,928 INFO AlluxioJobWorkerProcess - Started gRPC server on address 172.31.0.16:20006
What you expect to happen:
How to reproduce it
do as accelerate_data_accessing.md Additional Information
The text was updated successfully, but these errors were encountered:
cheyang
changed the title
[BUG](accelerate_data_accessing.md5, alluxio-fuse在k8s 1.20.2版本 volume挂载失败)
[BUG]Compatibility issue with K8s 1.19+
Mar 10, 2021
What is your environment(Kubernetes version, Fluid version, etc.)
k8s:v1.20.2, fluid: 0.4.0
Describe the bug
alluxio-fuse :alluxio-fuse容器creating失败,好像挂载和attach volume失败导致
容器状态如下:
[root@VM-0-7-centos kubernetes]# kubectl get pods
NAME READY STATUS RESTARTS AGE
hbase-fuse-xc75n 0/1 ContainerCreating 0 8h
hbase-fuse-xl7rw 0/1 ContainerCreating 0 8h
hbase-master-0 2/2 Running 0 8h
hbase-worker-8hrxl 2/2 Running 0 8h
hbase-worker-lm2qb 2/2 Running 0 8h
kubelete组件日志如下:
Feb 28 13:58:24 VM-0-7-centos kubelet[28984]: E0228 13:58:24.187806 28984 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/host-path/d7fbe499-3e98-43d1-91be-2d3f6a914e31-alluxio-fuse-device podName:d7fbe499-3e98-43d1-91be-2d3f6a914e31 nodeName:}" failed. No retries permitted until 2021-02-28 14:00:26.187775008 +0800 CST m=+1205.008960846 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume "alluxio-fuse-device" (UniqueName: "kubernetes.io/host-path/d7fbe499-3e98-43d1-91be-2d3f6a914e31-alluxio-fuse-device") pod "hbase-fuse-xc75n" (UID: "d7fbe499-3e98-43d1-91be-2d3f6a914e31") : hostPath type check failed: /dev/fuse is not a file"
Feb 28 13:59:37 VM-0-7-centos kubelet[28984]: E0228 13:59:37.772414 28984 kubelet.go:1656] Unable to attach or mount volumes for pod "hbase-fuse-xc75n_default(d7fbe499-3e98-43d1-91be-2d3f6a914e31)": unmounted volumes=[alluxio-fuse-device], unattached volumes=[alluxio-fuse-device alluxio-fuse-mount mem default-token-5r284]: timed out waiting for the condition; skipping pod
Feb 28 13:59:37 VM-0-7-centos kubelet[28984]: E0228 13:59:37.772451 28984 pod_workers.go:191] Error syncing pod d7fbe499-3e98-43d1-91be-2d3f6a914e31 ("hbase-fuse-xc75n_default(d7fbe499-3e98-43d1-91be-2d3f6a914e31)"), skipping: unmounted volumes=[alluxio-fuse-device], unattached volumes=[alluxio-fuse-device alluxio-fuse-mount mem default-token-5r284]: timed out waiting for the condition
Feb 28 14:00:26 VM-0-7-centos kubelet[28984]: I0228 14:00:26.256973 28984 reconciler.go:269] operationExecutor.MountVolume started for volume "alluxio-fuse-device" (UniqueName: "kubernetes.io/host-path/d7fbe499-3e98-43d1-91be-2d3f6a914e31-alluxio-fuse-device") pod "hbase-fuse-xc75n" (UID: "d7fbe499-3e98-43d1-91be-2d3f6a914e31")
Feb 28 14:00:26 VM-0-7-centos kubelet[28984]: E0228 14:00:26.257111 28984 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/host-path/d7fbe499-3e98-43d1-91be-2d3f6a914e31-alluxio-fuse-device podName:d7fbe499-3e98-43d1-91be-2d3f6a914e31 nodeName:}" failed. No retries permitted until 2021-02-28 14:02:28.257080139 +0800 CST m=+1327.078265886 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume "alluxio-fuse-device" (UniqueName: "kubernetes.io/host-path/d7fbe499-3e98-43d1-91be-2d3f6a914e31-alluxio-fuse-device") pod "hbase-fuse-xc75n" (UID: "d7fbe499-3e98-43d1-91be-2d3f6a914e31") : hostPath type check failed: /dev/fuse is not a file"
Feb 28 14:00:27 VM-0-7-centos kubelet[28984]: I0228 14:00:27.910602 28984 container_manager_linux.go:490] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
Feb 28 14:01:53 VM-0-7-centos kubelet[28984]: E0228 14:01:53.772048 28984 kubelet.go:1656] Unable to attach or mount volumes for pod "hbase-fuse-xc75n_default(d7fbe499-3e98-43d1-91be-2d3f6a914e31)": unmounted volumes=[alluxio-fuse-device], unattached volumes=[alluxio-fuse-device alluxio-fuse-mount mem default-token-5r284]: timed out waiting for the condition; skipping pod
Feb 28 14:01:53 VM-0-7-centos kubelet[28984]: E0228 14:01:53.772103 28984 pod_workers.go:191] Error syncing pod d7fbe499-3e98-43d1-91be-2d3f6a914e31 ("hbase-fuse-xc75n_default(d7fbe499-3e98-43d1-91be-2d3f6a914e31)"), skipping: unmounted volumes=[alluxio-fuse-device], unattached volumes=[alluxio-fuse-device alluxio-fuse-mount mem default-token-5r284]: timed out waiting for the condition
alluxio-worker:
[root@VM-0-7-centos ~]# kubectl logs hbase-worker-8hrxl alluxio-worker
2021-02-28 05:44:05,542 INFO NettyUtils - EPOLL_MODE is available
2021-02-28 05:44:05,993 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=172.31.0.16, rack=null)
2021-02-28 05:44:06,225 INFO BlockWorkerFactory - Creating alluxio.worker.block.BlockWorker
2021-02-28 05:44:06,308 INFO StorageDir - Folder /dev/shm/default/hbase/alluxioworker was created!
2021-02-28 05:44:06,323 WARN StorageTier - Failed to verify memory capacity
2021-02-28 05:44:06,391 INFO log - Logging initialized @1598ms
2021-02-28 05:44:06,642 INFO GrpcDataServer - Alluxio worker gRPC server started, listening on /0.0.0.0:20002
2021-02-28 05:44:06,643 INFO ProcessUtils - Starting Alluxio worker @172.31.0.16:20002.
2021-02-28 05:44:06,644 INFO MetricsSystem - Starting sinks with config: {}.
2021-02-28 05:44:06,822 INFO AlluxioWorkerProcess - Started Alluxio worker @172.31.0.16:20002 with id 5599774766131283346
2021-02-28 05:44:06,823 INFO WebServer - Alluxio worker web service starting @ /0.0.0.0:20003
2021-02-28 05:44:06,826 INFO Server - jetty-9.2.z-SNAPSHOT
2021-02-28 05:44:06,851 INFO ContextHandler - Started o.e.j.s.ServletContextHandler@24111ef1{/metrics/prometheus,null,AVAILABLE}
2021-02-28 05:44:06,852 INFO ContextHandler - Started o.e.j.s.ServletContextHandler@1fde5d22{/metrics/json,null,AVAILABLE}
2021-02-28 05:44:06,853 WARN SecurityHandler - [email protected]@7ca20101{/,null,STARTING} has uncovered http methods for path: /
2021-02-28 05:44:12,357 INFO ContextHandler - Started o.e.j.s.ServletContextHandler@7ca20101{/,null,AVAILABLE}
2021-02-28 05:44:12,364 INFO ServerConnector - Started ServerConnector@6999cd39{HTTP/1.1}{0.0.0.0:20003}
2021-02-28 05:44:12,364 INFO Server - Started @7572ms
2021-02-28 05:44:12,364 INFO WebServer - Alluxio worker web service started @ /0.0.0.0:20003
2021-02-28 05:44:12,364 INFO AlluxioWorkerProcess - Alluxio worker version 2.3.1-SNAPSHOT started. bindHost=0.0.0.0, connectHost=172.31.0.16, rpcPort=20002, webPort=20003
alluxio-job-worker:
[root@VM-0-7-centos ~]# kubectl logs hbase-worker-8hrxl alluxio-job-worker
2021-02-28 05:44:05,702 INFO NettyUtils - EPOLL_MODE is available
2021-02-28 05:44:06,458 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=172.31.0.16, rack=null)
2021-02-28 05:44:06,479 INFO log - Logging initialized @1466ms
2021-02-28 05:44:06,593 INFO ProcessUtils - Starting Alluxio job worker.
2021-02-28 05:44:06,594 INFO WebServer - Alluxio Job Manager Worker Web service starting @ /0.0.0.0:20007
2021-02-28 05:44:06,596 INFO Server - jetty-9.2.z-SNAPSHOT
2021-02-28 05:44:06,621 INFO ContextHandler - Started o.e.j.s.ServletContextHandler@72ade7e3{/metrics/json,null,AVAILABLE}
2021-02-28 05:44:06,622 WARN SecurityHandler - [email protected]@560348e6{/,null,STARTING} has uncovered http methods for path: /
2021-02-28 05:44:09,691 INFO ContextHandler - Started o.e.j.s.ServletContextHandler@560348e6{/,null,AVAILABLE}
2021-02-28 05:44:09,698 INFO ServerConnector - Started ServerConnector@36dce7ed{HTTP/1.1}{0.0.0.0:20007}
2021-02-28 05:44:09,699 INFO Server - Started @4686ms
2021-02-28 05:44:09,699 INFO WebServer - Alluxio Job Manager Worker Web service started @ /0.0.0.0:20007
2021-02-28 05:44:09,701 INFO MetricsSystem - Starting sinks with config: {}.
2021-02-28 05:44:09,892 INFO AlluxioJobWorkerProcess - Started Alluxio job worker with id 1614491014664
2021-02-28 05:44:09,893 INFO AlluxioJobWorkerProcess - Alluxio job worker version 2.3.1-SNAPSHOT started. bindHost=/0.0.0.0:20006, connectHost=172.31.0.16:20006, rpcPort=20006, webPort=20007
2021-02-28 05:44:09,894 INFO AlluxioJobWorkerProcess - Starting gRPC server on address 172.31.0.16:20006
2021-02-28 05:44:09,928 INFO AlluxioJobWorkerProcess - Started gRPC server on address 172.31.0.16:20006
What you expect to happen:
How to reproduce it
do as accelerate_data_accessing.md
Additional Information
The text was updated successfully, but these errors were encountered: