transport endpoint is not connected when juicefs rmr

[zhoucheng361]

What happened:
https://github.com/juicedata/juicefs/runs/6913391698?check_suite_focus=true

  meta_url=badger://badger-data
  mp=/tmp/myjfs
  ./juicefs mdtest $meta_url test2 --dirs 10 --depth 3 --files 30 --threads 10 --no-usage-report
  ./juicefs mount -d $meta_url $mp
  ./juicefs rmr $mp/test2/

2022/06/16 06:35:19.909896 juicefs[7506] : OK, myjfs is ready at /tmp/myjfs [mount_unix.go:45]
2022/06/16 06:35:35.666628 juicefs[7541] : Read message: 0 read /tmp/myjfs/.control: transport endpoint is not connected [warmup.go:87]

tail -100 /var/log/juicefs.log
2022/06/17 10:10:43.048916 juicefs[264590] <ERROR>: error: cannot create new mem table error: newMemTable error: While opening memtable: badger-data/00002.mem error: while opening file: badger-data/00002.mem error: open badger-data/00002.mem: no such file or directory
unable to open: badger-data/00002.mem
github.com/dgraph-io/ristretto/z.OpenMmapFile
        /root/go/pkg/mod/github.com/dgraph-io/[email protected]/z/file.go:80
github.com/dgraph-io/badger/v3.(*logFile).open
        /root/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/memtable.go:556
github.com/dgraph-io/badger/v3.(*DB).openMemTable
        /root/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/memtable.go:130
github.com/dgraph-io/badger/v3.(*DB).newMemTable
        /root/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/memtable.go:153
github.com/dgraph-io/badger/v3.(*DB).ensureRoomForWrite
        /root/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:996
github.com/dgraph-io/badger/v3.(*DB).writeRequests
        /root/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:832
github.com/dgraph-io/badger/v3.(*DB).doWrites.func1
        /root/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:887
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1571
goroutine 1865686 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/juicedata/juicefs/pkg/meta.errno({0x3070760, 0xc015f62170})
        /root/juicefs2/pkg/meta/utils.go:93 +0xc5
github.com/juicedata/juicefs/pkg/meta.(*kvMeta).doUnlink(0xc000172280, {0x309e170?, 0xc006f8b080}, 0xa54fe, {0xc0163881e0, 0x10})
        /root/juicefs2/pkg/meta/tkv.go:1190 +0x365
github.com/juicedata/juicefs/pkg/meta.(*baseMeta).Unlink(0xc000089400, {0x309e170, 0xc006f8b080}, 0xa54fe, {0xc0163881e0, 0x10})
        /root/juicefs2/pkg/meta/base.go:720 +0x1b8
github.com/juicedata/juicefs/pkg/meta.(*baseMeta).emptyDir(0xc000089400, {0x309e170, 0xc006f8b080}, 0xa54fe, 0xc000028410, 0xc00075a400)
        /root/juicefs2/pkg/meta/utils.go:269 +0x2b0
github.com/juicedata/juicefs/pkg/meta.(*baseMeta).emptyEntry(0x0?, {0x309e170, 0xc006f8b080}, 0xc012f27720?, {0xc0128b7050, 0xd}, 0x0?, 0xc000028410, 0xc017b0d7a8?)
        /root/juicefs2/pkg/meta/utils.go:286 +0x65
github.com/juicedata/juicefs/pkg/meta.(*baseMeta).emptyDir.func1(0xc012f27720?, {0xc0128b7050?, 0xc012f27720?})
        /root/juicefs2/pkg/meta/utils.go:254 +0x9e
created by github.com/juicedata/juicefs/pkg/meta.(*baseMeta).emptyDir
        /root/juicefs2/pkg/meta/utils.go:252 +0x4b0 [utils.go:93]

it works if we use absolute path:

root@bench-tikv-1:~/juicefs2# ./juicefs format badger:///root/badger-test  myjfs8
2022/06/17 10:27:20.530744 juicefs[264934] <INFO>: Meta address: badger:///root/badger-test [interface.go:397]
2022/06/17 10:27:20.539535 badger[264934] <INFO>: All 0 tables opened in 0s [logger.go:46]
2022/06/17 10:27:20.541759 badger[264934] <INFO>: Discard stats nextEmptySlot: 0 [logger.go:46]
2022/06/17 10:27:20.541813 badger[264934] <INFO>: Set nextTxnTs to 0 [logger.go:46]
2022/06/17 10:27:20.547605 juicefs[264934] <INFO>: Data use file:///var/jfs/myjfs8/ [format.go:420]
2022/06/17 10:27:20.548424 juicefs[264934] <INFO>: Volume is formatted as {Name:myjfs8 UUID:0b9f3a10-67c7-49ab-8cb9-1dfa794f5678 Storage:file Bucket:/var/jfs/ AccessKey: SecretKey: BlockSize:4096 Compression:none Shards:0 HashPrefix:false Capacity:0 Inodes:0 EncryptKey: KeyEncrypted:false TrashDays:1 MetaVersion:1 MinClientVersion: MaxClientVersion:} [format.go:458]
root@bench-tikv-1:~/juicefs2# ./juicefs  mdtest badger:///root/badger-test  test2 --dirs 10 --depth 3 --files 30 --threads 10 --no-usage-report
2022/06/17 10:27:52.170138 juicefs[264959] <INFO>: Meta address: badger:///root/badger-test [interface.go:397]
2022/06/17 10:27:52.180460 badger[264959] <INFO>: All 0 tables opened in 0s [logger.go:46]
2022/06/17 10:27:52.183271 badger[264959] <INFO>: Discard stats nextEmptySlot: 0 [logger.go:46]
2022/06/17 10:27:52.183346 badger[264959] <INFO>: Set nextTxnTs to 1 [logger.go:46]
2022/06/17 10:27:52.183552 badger[264959] <INFO>: Deleting empty file: /root/badger-test/000001.vlog [logger.go:46]
2022/06/17 10:27:52.188057 juicefs[264959] <INFO>: Data use file:///var/jfs/myjfs8/ [mdtest.go:202]
2022/06/17 10:27:52.188755 juicefs[264959] <INFO>: Disk cache (/var/jfsCache/0b9f3a10-67c7-49ab-8cb9-1dfa794f5678/): capacity (102400 MB), free ratio (10%), max pending pages (15) [disk_cache.go:90]
2022/06/17 10:27:52.189371 juicefs[264959] <INFO>: Create session 1 OK with version: 1.0.0-dev+2022-06-14.c9121425 [base.go:209]
2022/06/17 10:27:52.189912 juicefs[264959] <INFO>: Prometheus metrics listening on 127.0.0.1:9567 [mount.go:160]
2022/06/17 10:27:52.190107 juicefs[264959] <INFO>: Create 333000 files in 1110 dirs [mdtest.go:96]
2022/06/17 10:27:52.232818 juicefs[264959] <INFO>: Created 1110 dirs in 42.667023ms (26015 dirs/s) [mdtest.go:112]
create file count: 333300 / 333300 [==============================================================]  done      
2022/06/17 10:27:58.501743 juicefs[264959] <INFO>: Created 333000 files in 6.268920456s (53119 files/s) [mdtest.go:128]
2022/06/17 10:27:58.501930 juicefs[264959] <INFO>: close session 1: %!s(<nil>) [base.go:271]
root@bench-tikv-1:~/juicefs2# ./juicefs mount -d  badger:///root/badger-test  /tmp/badger-test
2022/06/17 10:28:22.001569 juicefs[264984] <INFO>: Meta address: badger:///root/badger-test [interface.go:397]

2022/06/17 10:28:24.040174 badger[264984] <INFO>: All 1 tables opened in 0s [logger.go:46]
2022/06/17 10:28:24.043245 badger[264984] <INFO>: Discard stats nextEmptySlot: 0 [logger.go:46]
2022/06/17 10:28:24.043313 badger[264984] <INFO>: Set nextTxnTs to 337775 [logger.go:46]
2022/06/17 10:28:24.043418 badger[264984] <INFO>: Deleting empty file: /root/badger-test/000002.vlog [logger.go:46]
2022/06/17 10:28:24.047827 juicefs[264984] <INFO>: Data use file:///var/jfs/myjfs8/ [mount.go:420]
2022/06/17 10:28:24.047953 juicefs[264984] <INFO>: Disk cache (/var/jfsCache/0b9f3a10-67c7-49ab-8cb9-1dfa794f5678/): capacity (102400 MB), free ratio (10%), max pending pages (15) [disk_cache.go:90]
2022/06/17 10:28:24.048397 badger[264984] <INFO>: Lifetime L0 stalled for: 0s [logger.go:46]
2022/06/17 10:28:24.516125 badger[264984] <INFO>: 
Level 0 [ ]: NumTables: 03. Size: 18 MiB of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB
Level 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 3 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 4 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 5 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 6 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level Done [logger.go:46]
2022/06/17 10:28:25.022235 juicefs[264984] <INFO>: OK, myjfs8 is ready at /tmp/badger-test [mount_unix.go:45]
root@bench-tikv-1:~/juicefs2# 
root@bench-tikv-1:~/juicefs2# 
root@bench-tikv-1:~/juicefs2# ./juicefs rmr /tmp/badger-test/
.accesslog  .config     .stats      test2/      .trash/     
root@bench-tikv-1:~/juicefs2# ./juicefs rmr /tmp/badger-test/test2/
Removing entries count: 334413   44459.84/s   

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?

Environment:

• JuiceFS version (use juicefs --version) or Hadoop Java SDK version:
• Cloud provider or hardware configuration running JuiceFS:
• OS (e.g cat /etc/os-release):
• Kernel (e.g. uname -a):
• Object storage (cloud provider and region, or self maintained):
• Metadata engine info (version, cloud provider managed or self maintained):
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
• Others:

[davies]

This means the mount process crashed.

[codeboytj]

Is there a solution to avoid the "transport endpoint is not connected" error when crashed?