nydusd can't mount with dummycache backend

[vishvananda]

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.

Version of nydus being used (nydusd --version)

Version: v2.2.3
Git Commit: e8c3246
Build Time: 2023-10-27T17:42:24.203378048Z
Profile: release
Rustc: rustc 1.66.1 (90743e729 2023-01-10)

Version of nydus-snapshotter being used (containerd-nydus-grpc --version)

not using it

Kernel information (uname -r)

6.2.0-1012-aws

GNU/Linux Distribution, if applicable (cat /etc/os-release)

PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Details about issue

Using the following config, nydusd is perfectly serving images

{
  "device": {
    "backend": {
      "type": "localfs",
      "config": {
        "dir": "/home/ubuntu/ebs/blobs"
      }
    },
    "cache": {
      "type": "blobcache",
      "config": {
        "work_dir": "/home/ubuntu/ebs/cache"
      }
    }
  },
  "mode": "direct",
  "digest_validate": false,
  "iostats_files": false,
  "enable_xattr": true
}

command line:

sudo nydusd --config nydusd-config.localfs.json --bootstrap blobs/6788acafb778c0e94f81512e133b0e2367ce13d6e63501b924008dbbe73c63ba --log-level trace --mountpoint mount

When I attempt to run with the dummycache backend, the bootstrap fails:

{
  "device": {
    "backend": {
      "type": "localfs",
      "config": {
        "dir": "/home/ubuntu/ebs/blobs"
      }
    },
    "cache": {
      "type": "dummycache"
    }
  },
  "mode": "direct",
  "digest_validate": false,
  "iostats_files": false,
  "enable_xattr": true
}

output:

[2023-10-27 17:40:01.177295 +00:00] INFO Program Version: v2.2.3, Git Commit: "e8c324687a472243d4dedadafd7fda63044aa8ac", Build Time: "2023-10-27T17:30:34.284507956Z", Profile: "release", Rustc Version: "rustc 1.66.1 (90743e729 2023-01-10)"
[2023-10-27 17:40:01.177337 +00:00] INFO Set rlimit-nofile to 1000000, maximum 1000000
[2023-10-27 17:40:01.177600 +00:00] DEBUG [/fuse-backend-rs-0.10.5/src/api/pseudo_fs.rs:161] pseudo fs iterate "/"
[2023-10-27 17:40:01.177770 +00:00] ERROR [/src/error.rs:22] Error:
        "unsupported cache type 'dummycache'"
        at api/src/config.rs:1082
        note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2023-10-27 17:40:01.177787 +00:00] ERROR [/src/error.rs:22] Error:
        "failed to parse configuration information"
        at api/src/config.rs:243
        note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2023-10-27 17:40:01.177801 +00:00] ERROR [/src/error.rs:22] Error:
        Rafs(LoadConfig(Os { code: 22, kind: InvalidInput, message: "Invalid argument" }))
        at service/src/lib.rs:121
        note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2023-10-27 17:40:01.178073 +00:00] ERROR [src/bin/nydusd/main.rs:539] Failed in starting daemon: Invalid argument (os error 22)

I'm attempting to performance test reading directly from the backend without local disk caching. Am I doing something wrong?

[imeoer]

It seems the rafs v6 (nydus-image create --fs-version 6) doesn't support the dummy cache? @jiangliu

[adamqqqplay]

It seems the rafs v6 (nydus-image create --fs-version 6) doesn't support the dummy cache? @jiangliu

@imeoer In our previous tests, rafs v6 does not support dummy cache.

@vishvananda You can use rafs v5 to work around. (Create image with nydus-image create --fs-version 5)

[vishvananda]

@adamqqqplay I've tried with nydus-image create -t dir-rafs -v5 -D ../blobs rootfs And I still get the above error with v2.2.3

[vishvananda]

I get a similar but different error using master. It seems to be failing during the bootstrap process:

s$ sudo env "RUST_BACKTRACE=1" "PATH=$PATH" nydusd --config dummy.json --bootstrap blobs/8a05eb1bf0a76619b42afaf52f23ae96efb369b5d39d75ff2c57e74ea4326768 --log-level trace --mountpoint mount
[2023-10-30 15:51:21.433117 +00:00] INFO Program Version: v2.3.0-alpha.0-13-gf55d2c94, Git Commit: "f55d2c948f85e22041bda18d0f3c8de11ca4b8b7", Build Time: "2023-10-30T15:46:57.843474431Z", Profile: "release", Rustc Version: "rustc 1.68.2 (9eb3afe9e 2023-03-27)"
[2023-10-30 15:51:21.433156 +00:00] INFO Set rlimit-nofile to 1000000, maximum 1000000
[2023-10-30 15:51:21.433318 +00:00] DEBUG [/fuse-backend-rs-0.10.5/src/api/pseudo_fs.rs:161] pseudo fs iterate "/"
[2023-10-30 15:51:21.433417 +00:00] ERROR [/src/fusedev.rs:595] service mount error: RAFS failed to handle request, Failed to load config: failed to parse configuration information`
[2023-10-30 15:51:21.439430 +00:00] ERROR [/src/error.rs:18] Stack:
   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new

[2023-10-30 15:51:21.439453 +00:00] ERROR [/src/error.rs:19] Error:
        Rafs(LoadConfig(Custom { kind: InvalidInput, error: "failed to parse configuration information" }))
        at service/src/fusedev.rs:596
[2023-10-30 15:51:21.439502 +00:00] ERROR [src/bin/nydusd/main.rs:525] Failed in starting daemon:
Error: Custom { kind: Other, error: "" }

[imeoer]

@vishvananda Sorry for the unverified answer :(, the bug should be fixed in #1458.

The usage steps:

cargo build --release

sudo ./target/release/nydus-image create \
  --fs-version 5 \
  --bootstrap /path/to/bootstrap \
  --blob-dir /path/to/blobs \
  /path/to/SOURCE

sudo ./target/release/nydusd \
  --config /path/to/nydusd-config.json \
  --bootstrap /path/to/bootstrap \
  --mountpoint /path/to/mountpoint