Some problems about Nydus unpack func supporting blob dir

[zhangshuiyong]

Some day ago ,I see the branch imeoer:fix-unpack-blob-dir

about nydus unpack func supporting blob dir, then i try to use it ,and i find some problem below:

1. The branch can build successfully, but when i run any cmd about nydus, it will print a same err(see the pic):

then i annotate the code in 698~699 & 5 lines(see the pic)

then i can run nydus create and unpack normally.

2. nydus unpack data seem like inconsistent

I run a set of cmd from nydus create raw file dir to nydus unpack(see the below cmds):

./nydus-image create -v 5 -t dir-rafs -D /Users/terrill/Documents/urchin/image-service-blob-dir/image-service-fix-unpack-blob-dir/tests/echarts-image /Users/terrill/Documents/urchin/image-service-blob-dir/image-service-fix-unpack-blob-dir/tests/echarts
[2023-06-27 18:39:43.730654 +08:00] INFO successfully built RAFS filesystem:
meta blob path: /Users/terrill/Documents/urchin/image-service-blob-dir/image-service-fix-unpack-blob-dir/tests/echarts-image/d78b4b1bfb82844c408ae5f64659f78d221c5df3abe3a4ba670e3b9822d1655b
data blob size: 0xc0e1
data blobs: ["fa964c10cd36503b3bbd0d8218ec03d132214a51b0e4b03d2684770290306bce"]

./nydus-image create -v 5 -t dir-rafs --chunk-dict /Users/terrill/Documents/urchin/image-service-blob-dir/image-service-fix-unpack-blob-dir/tests/echarts-image/meta -D /Users/terrill/Documents/urchin/image-service-blob-dir/image-service-fix-unpack-blob-dir/tests/echarts-image /Users/terrill/Documents/urchin/zhangshuiyong/urfs/tests/echarts-add
[2023-06-27 18:42:13.087824 +08:00] INFO RAFS v5 super block features: HASH_BLAKE3 | EXPLICIT_UID_GID | COMPRESSION_ZSTD
[2023-06-27 18:42:13.111437 +08:00] INFO successfully built RAFS filesystem:
meta blob path: /Users/terrill/Documents/urchin/image-service-blob-dir/image-service-fix-unpack-blob-dir/tests/echarts-image/2eec9becfcfebfb3a58918c3920b990a05a2af95d89037be29501453e0433b15
data blob size: 0xc0e1
data blobs: ["5ee38535110c9470daeb434efbace48b15dfb7c08878cc9effa2bc0b29638a0d", "fa964c10cd36503b3bbd0d8218ec03d132214a51b0e4b03d2684770290306bce"]

./nydus-image unpack --blob-dir /Users/terrill/Documents/urchin/image-service-blob-dir/image-service-fix-unpack-blob-dir/tests/echarts-image --output ./echarts-add.tar /Users/terrill/Documents/urchin/image-service-blob-dir/image-service-fix-unpack-blob-dir/tests/echarts-image/meta2
[2023-06-27 18:44:09.327079 +08:00] INFO RAFS v5 super block features: HASH_BLAKE3 | EXPLICIT_UID_GID | COMPRESSION_ZSTD

then I untar the echarts-add.tar and get the unpack file md5 hash , i found the hash is diff from raw file

raw file md5 hash

file md5 hash after unpack

for reappear this problem conveniently, I upload the source file dir of nydus create(1 image file in dir named 'echarts') & nydus create --chunk-dict(2 image file in dir named 'echarts-add')

1 image file in dir named 'echarts'

2 image file in dir named 'echarts-add'

[imeoer]

cc @ccx1024cc

[imeoer]

@ccx1024cc is working on this.

[An-DJ]

Try to fix the inconsistent issue in #1426, @zhangshuiyong @imeoer plz take a look : )

[zhangshuiyong]

Try to fix the inconsistent issue in #1426, @zhangshuiyong @imeoer plz take a look : )

Thx for your work, i will test this PR today or tomorrow.

[An-DJ]

BTW, the panic caused by unsupported export subcmd on mac is tried to be fixed in #1429 @zhangshuiyong @imeoer PTAL

[zhangshuiyong]

Try to fix the inconsistent issue in #1426, @zhangshuiyong @imeoer plz take a look : )

Thx for your work, i will test this PR today or tomorrow.

I complie this fix-blobs-unpack-bug branch and test, the above test case with bug is pass.
Good job

[zhangshuiyong]

BTW, the panic caused by unsupported export subcmd on mac is tried to be fixed in #1429 @zhangshuiyong @imeoer PTAL

review the fix-export-panic-on-mac branch commit of fixing this export panic，the resolution is remove the export subcmd on mac?
Can retain the export subcmd on mac and fix the this panic of export subcmd?

[An-DJ]

BTW, the panic caused by unsupported export subcmd on mac is tried to be fixed in #1429 @zhangshuiyong @imeoer PTAL

review the fix-export-panic-on-mac branch commit of fixing this export panic，the resolution is remove the export subcmd on mac? Can retain the export subcmd on mac and fix the this panic of export subcmd?

Actually export has been disabled already on mac: https://github.com/dragonflyoss/image-service/blob/master/src/bin/nydus-image/main.rs#L504

The work in pr #1429 just tries to fix panic caused by unexisted export cmd.

BTW, when I remove all the target_os(linux) on export, it compiled failed, so I guess export is not supported by design.

[zhangshuiyong]

BTW, the panic caused by unsupported export subcmd on mac is tried to be fixed in #1429 @zhangshuiyong @imeoer PTAL

review the fix-export-panic-on-mac branch commit of fixing this export panic，the resolution is remove the export subcmd on mac? Can retain the export subcmd on mac and fix the this panic of export subcmd?

ok, I realize this resolution after reading more codes, because the export subcmd function is unimplemented of not(target_os = "linux"))

I will test fix-export-panic-on-mac branch later.
Thx for your work.

[An-DJ]

Dig it deeply, the export subcmd only support "exporting RAFS as raw block disk images" now:

(https://github.com/dragonflyoss/image-service/blob/master/src/bin/nydus-image/main.rs#L1747)

which relies on tokio-uring lib and linux os.

So disable export on mac now is reasonable : ) Maybe we can restrict it to only "exporting as block" in future work.