Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nydus Supports Encyrpted Images #1334

Closed
5 tasks done
taoohong opened this issue Jun 20, 2023 · 2 comments
Closed
5 tasks done

Nydus Supports Encyrpted Images #1334

taoohong opened this issue Jun 20, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@taoohong
Copy link
Contributor

taoohong commented Jun 20, 2023 •
edited
Loading

1. Changes

  • Encrypt Nydus bootstrap layer with Ocicrypt. Merged ✅
  • Encrypt Nydus data blobs with AES-128-XTS. Support to fetch encrypted chunks' data on demand. Merged ✅
  • Expand Nydusify to support encrypted nydus images. PR
  • Expand acceleration-service to support converting encrypted nydus images. Merged ✅
  • Expand nydus-snapshotter to support encrypted nydus images. Merged ✅

2. Build encrypted nydus image

2.1. convert to encrypted nydus image

Convert OCI image to encrypted nydus image by nydusify convert. An example is blow, using jwe to encrypt the nydus bootstrap layer.

nydusify convert \
--source registry.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2 \
--target localhost:5000/busybox:nydus-encrypted \
--encrypt-recipients jwe:/tmp/mypubkey.pem

INFO[2023-06-20T11:28:19+08:00] pulling image registry.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2  module=converter
INFO[2023-06-20T11:28:20+08:00] pulled image registry.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2, elapse 1.154472526s  module=converter
INFO[2023-06-20T11:28:20+08:00] converting image registry.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2  module=converter
INFO[2023-06-20T11:28:21+08:00] converted image localhost:5000/busybox:nydus-encrypted, elapse 192.597968ms  module=converter
INFO[2023-06-20T11:28:21+08:00] pushing image localhost:5000/busybox:nydus-encrypted  module=converter
INFO[2023-06-20T11:28:21+08:00] try to push with plain HTTP for localhost:5000/busybox:nydus-encrypted  module=converter
INFO[2023-06-20T11:28:21+08:00] pushed image localhost:5000/busybox:nydus-encrypted, elapse 34.957722ms  module=converter

2.2. build from directory

Build encrypted nydus image from directory by nydusify build. An example is blow, using jwe to encrypt the nydus bootstrap layer.

nydusify build \
--source-dir /usr/bin \
--name buildtest \
--output-dir build \
--encrypt-recipients jwe:/tmp/mypubkey.pem

INFO[2023-06-20T12:54:02+08:00] found 'nydus-image' binary at /root/Programs/image-service/target/debug/nydus-image
INFO[2023-06-20T12:54:02+08:00] start to build image from source directory "/usr/bin"
INFO[2023-06-20T12:54:13+08:00] successfully built Nydus image (bootstrap:'build/buildtest.meta', blob:'build/buildtest.blob')

3. Check the encrypted nydus image

Check the encrypted nydus image. An example is blow.

nydusify check \
--source registry.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2 \
--target localhost:5000/busybox:nydus-encrypted \
--decrypt-keys /tmp/mykey.pem

INFO[2023-06-20T12:33:30+08:00] Parsing image localhost:5000/busybox:nydus-encrypted
INFO[2023-06-20T12:33:30+08:00] trying next host                              error="failed to do request: Head \"https://localhost:5000/v2/busybox/manifests/nydus-encrypted\": http: server gave HTTP response to HTTPS client" host="localhost:5000"
INFO[2023-06-20T12:33:30+08:00] Parsing image localhost:5000/busybox:nydus-encrypted
INFO[2023-06-20T12:33:30+08:00] Parsing image registry.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2
INFO[2023-06-20T12:33:31+08:00] Dumping OCI and Nydus manifests to ./output
INFO[2023-06-20T12:33:31+08:00] Pulling Nydus bootstrap to output/nydus_bootstrap
INFO[2023-06-20T12:33:31+08:00] Decrypting Nydus bootstrap layer
INFO[2023-06-20T12:33:31+08:00] Checking Nydus manifest
INFO[2023-06-20T12:33:31+08:00] Checking Nydus bootstrap
RAFS filesystem metadata is valid, referenced data blobs:
	 0: 01280c48c2e2c24afd92405611f23685db0c4612a1b29d1f099316857707687d, compressed data size 0xb536e, compressed file size 0xb6ac2, uncompressed file size 0x121000, chunks: 0x7, features: aligned cap_toc chunk-digest chunk-v2 tar-header toc
INFO[2023-06-20T12:33:31+08:00] Mounting source image to output/fs/source_mounted
INFO[2023-06-20T12:33:32+08:00] Mounting Nydus image to output/fs/nydus_mounted
INFO[2023-06-20T12:33:32+08:00] Verifying filesystem for source and Nydus image
INFO[2023-06-20T12:33:32+08:00] Verified Nydus image localhost:5000/busybox:nydus-encrypted

check output manifest

cat output/nydus_manifest.json

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "digest": "sha256:efe9b0bf7aedf95f5dca884827d0f2593d04e205bfcb7b142bc74f8577d0642d",
    "size": 735
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:01280c48c2e2c24afd92405611f23685db0c4612a1b29d1f099316857707687d",
      "size": 753363,
      "annotations": {
        "containerd.io/snapshot/encrypted-blob": "true",
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
      "digest": "sha256:467c3fc99d7043c177d7f55708a87bf6c9eaa35689418a55218cd3dc439f5b94",
      "size": 3906,
      "annotations": {
        "containerd.io/snapshot/nydus-bootstrap": "true",
        "containerd.io/snapshot/nydus-fs-version": "6",
        "org.opencontainers.image.enc.keys.jwe": "eyJwcm90ZWN0ZWQiOiJleUpoYkdjaU9pSlNVMEV0VDBGRlVDSXNJbVZ1WXlJNklrRXlOVFpIUTAwaWZRIiwiZW5jcnlwdGVkX2tleSI6Ik14eGc1cVl5VlAzdV9LemdMcFJ1Q3lZaWY1V2o2aVhmQnFnNC0ydVNhNzg5LXN4bTQ2WTh6eVRoWmYzV2RrR043T3dMdmtYOHZ6TXNXZGp4dzl0LUx4T09ZTnJmMGRhcEFsV3JkTFczcGwzNmZlRWYzMFd5U2t2VFpQMEJ4azRFRUo3dFlKWWI2ZlM3UVhOdUhXR3ROcUgyUHFKWmd3bEVjOTNjRjFTYVlBcERpa0pNRTZ2bmhIZmNGLUNJVW44MjVoT0lMV052QWZhMFlJX0dfeUNYN0s4RFBtclZNaVhPOGdLcHU3ODlXZzA1bDlzUnhEdGZQbTFBRWYxQnBPQjhhQ2s4S0s0MWhZM0NycmdrUE5fNWVRb1pTenlQSTJrM1c1elRFNjNJT2s0Y2pDZ1dXdkZ3OTVKNU1YYVV6NVFJaHRwY0dSZ2w5SW1FRUczY3VzLUNiQSIsIml2IjoiVGpzbTAzR0hzTG1XUkFjTSIsImNpcGhlcnRleHQiOiJkcnJGZmlQOVlDZlFQVEtSVFZOZnowblNYNmF3cHRXMmJaUExUeVZralo0UEFIeFQ1VHMwdks1TnN2TlZpTFdaTllaZDFMZElUUkV0RmlwSDRkMUlDU1AwUzUzN2FaUE1HdFg3b3RtTTVwOElrQy1jR2NCUUFaMldEWlgxd1F5eVUxM0ZkNlV3UXczR0t4Nk9PUDQ0LVcyUmo4cWhtMURnUFFIR0x2VFhyS1Z3eGg5c2Q4TXIweHAtR204OEQzZ3prWUxtNldzM09XOVQ4X0l3ZzNweEc2bExaS05zcFB4MnU3SktIQVZQR1FxdkNNMU93bDVzWnhKZEdibnFwbUYya0EiLCJ0YWciOiJMTEE1bEl4UF9XNl9wT1FDOXN2U2ZnIn0=",
        "org.opencontainers.image.enc.pubopts": "eyJjaXBoZXIiOiJBRVNfMjU2X0NUUl9ITUFDX1NIQTI1NiIsImhtYWMiOiJTZENHK1VNV3JoV3RYRUNBbngzN0NRcEQ4V2UyektGMEtKeEFhUE9zMFNNPSIsImNpcGhlcm9wdGlvbnMiOnt9fQ=="
      }
    }
  ],
  "subject": {
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "digest": "sha256:5e8e0509e829bb8f990249135a36e81a3ecbe94294e7a185cc14616e5fad96bd",
    "size": 527,
    "platform": {
      "architecture": "amd64",
      "os": "linux"
    }
  },
  "annotations": {
    "containerd.io/snapshot/nydus-builder-version": "v2.2.0-250-g66761f2d",
    "containerd.io/snapshot/nydus-fs-version": "6",
    "containerd.io/snapshot/nydus-source-digest": "sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd",
    "containerd.io/snapshot/nydus-source-reference": "registry.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2"
  }
}

Data blobs add an annotation "containerd.io/snapshot/encrypted-blob" to imply encrypted.
Bootstrap layer using mediatype "application/vnd.oci.image.layer.v1.tar+gzip+encrypted".

check output config

cat output/fs/nydus_config.json

{
	"device": {
		"backend": {
			"type": "registry",
			"config": {"scheme":"http","host":"localhost:5000","repo":"busybox"}
		},
		"cache": {
			"type": "blobcache",
			"config": {
				"work_dir": "output/fs/nydus_blobs"
			}
		}
	},
	"mode": "direct",
	"iostats_files": false,
	"fs_prefetch": {
		"enable": true,
		"threads_count": 10,
		"merging_size": 131072
	},
	"digest_validate": false,
	"enable_xattr": true
}

4. Mount encrypted nydus image

nydusify mount \
--target localhost:5000/busybox:nydus-encrypted \
--target-insecure \
--decrypt-keys /tmp/mykey.pem

INFO[2023-06-20T13:13:29+08:00] Parsing image localhost:5000/busybox:nydus-encrypted
INFO[2023-06-20T13:13:29+08:00] trying next host                              error="failed to do request: Head \"https://localhost:5000/v2/busybox/manifests/nydus-encrypted\": http: server gave HTTP response to HTTPS client" host="localhost:5000"
INFO[2023-06-20T13:13:29+08:00] Parsing image localhost:5000/busybox:nydus-encrypted
INFO[2023-06-20T13:13:29+08:00] Pulling Nydus bootstrap to tmp/nydus_bootstrap
INFO[2023-06-20T13:13:29+08:00] Decrypting Nydus bootstrap layer
decryptKeys {[/tmp/mykey.pem]}
INFO[2023-06-20T13:13:29+08:00] Mounting Nydus image to ./image-fs
INFO[2023-06-20T13:13:29+08:00] Please send signal SIGINT/SIGTERM to umount the file system

# ls image-fs/
bin  dev  etc  home  root  tmp  usr  var
# ls image-fs/bin/
'['           crond           fdformat      ifenslave   lsof         nsenter         rmdir               strings       ubirsvol
'[['          crontab         fdisk         ifplugd     lspci        nslookup        rmmod               stty          ubiupdatevol
 acpid        cryptpw         fgconsole     ifup        lsscsi       ntpd            route               su            udhcpc
 ...

5. Inspect encrypted nydus image

Inspect encrypted nydus image with nydus-image. Encrypted nydus images can be recognized be the RAFS features and cipher tag of the blob info.

nydus-image inspect -B output/nydus_bootstrap

[2023-06-16 18:00:26.293492 +08:00] INFO RAFS features: HASH_BLAKE3 | EXPLICIT_UID_GID | HAS_XATTR | COMPRESSION_ZSTD | INLINED_CHUNK_DIGEST | ENCRYPTION_ASE_128_XTS
Inspecting RAFS :> stats

Version:                6
Inodes Count:           416
Chunk Size:             1024KB
Root Inode:             128
Flags:                  HASH_BLAKE3 | EXPLICIT_UID_GID | HAS_XATTR | COMPRESSION_ZSTD | INLINED_CHUNK_DIGEST | ENCRYPTION_ASE_128_XTS
Blob table offset:      0x1000
Blob table size:        0x100
Prefetch table offset:  0x1100
Prefetch table entries: 0x1
Chunk table offset:     0x6000
Chunk table size:       0x230

Inspecting RAFS :> blobs

Blob Index:             0
Blob ID:                e489ea831910d8b27e3beae9504e955debb0aa34978e76436cd6ddebd7f12c8b
Raw Blob ID:            e489ea831910d8b27e3beae9504e955debb0aa34978e76436cd6ddebd7f12c8b
Blob Size:              748226
Compressed Data Size:   742254
Uncompressed Data Size: 1183744
Mapped Block Address:   128
Features:               ALIGNED | CHUNK_INFO_V2 | INLINED_CHUNK_DIGEST | HAS_TAR_HEADER | HAS_TOC | CAP_TAR_TOC
Compressor:             Zstd
Digester:               Blake3
Cipher:                 Aes128Xts
Chunk Size:             0x100000
Chunk Count:            7
Prefetch Table Offset:  0
Prefetch Table Size:    0
Meta Compressor:        Zstd
Meta Offset:            742766
Meta Compressed Size:   116
Meta Uncompressed Size: 168
ToC Digest:             0000000000000000000000000000000000000000000000000000000000000000
ToC Size:               0
RAFS Blob Digest:       04b974bcfaf58cff9f1a8e9579bfe690400fc9f8c8250332e102799dc6abd80c
RAFS Blob Size:         13108545664970892636

Inspecting RAFS :>chunk 0

...

File: yes                              Parent Path: /bin/                           
Compressed Offset: 0, Compressed Size: 692632
Decompressed Offset: 0, Decompressed Size: 1048576
Chunk ID: 15707ff0f5bb541ab2cf7f1886ef815edc64aec99fb4b363561569d6008f3d1f, 
Blob ID: e489ea831910d8b27e3beae9504e955debb0aa34978e76436cd6ddebd7f12c8b


File: zcat                             Parent Path: /bin/                           
Compressed Offset: 0, Compressed Size: 692632
Decompressed Offset: 0, Decompressed Size: 1048576
Chunk ID: 15707ff0f5bb541ab2cf7f1886ef815edc64aec99fb4b363561569d6008f3d1f, 
Blob ID: e489ea831910d8b27e3beae9504e955debb0aa34978e76436cd6ddebd7f12c8b


File: zcip                             Parent Path: /bin/                           
Compressed Offset: 0, Compressed Size: 692632
Decompressed Offset: 0, Decompressed Size: 1048576
Chunk ID: 15707ff0f5bb541ab2cf7f1886ef815edc64aec99fb4b363561569d6008f3d1f, 
Blob ID: e489ea831910d8b27e3beae9504e955debb0aa34978e76436cd6ddebd7f12c8b

6. Export encrypted nydus image

Export encrypted nydus RAFS filesystem as raw block disk images or tar files with nydus-image. An example is blow.

nydus-image export -B test/bootstrap -D test --block

[2023-06-20 12:58:56.287040 +08:00] INFO RAFS features: HASH_BLAKE3 | EXPLICIT_UID_GID | HAS_XATTR | COMPRESSION_ZSTD | ENCRYPTION_ASE_128_XTS

ls test

96d85666fdf311b89214821c411e63534c3493a8cb5e453f49271e4ac47d58a6
96d85666fdf311b89214821c411e63534c3493a8cb5e453f49271e4ac47d58a6.blob.data
96d85666fdf311b89214821c411e63534c3493a8cb5e453f49271e4ac47d58a6.blob.meta
96d85666fdf311b89214821c411e63534c3493a8cb5e453f49271e4ac47d58a6.chunk_map
bootstrap
bootstrap.disk
This was referenced Jun 20, 2023
Merged
Merged
@adamqqqplay
Copy link
Member

After PR merged, the rest PRs will be pushed.

Thanks for your wonderful work.

@adamqqqplay adamqqqplay added the enhancement New feature or request label Jul 3, 2023
@taoohong taoohong mentioned this issue Jul 17, 2023
Merged
@taoohong taoohong mentioned this issue Aug 1, 2023
Open
6 tasks
@imeoer
Copy link
Collaborator

imeoer commented Jun 18, 2024

Maybe close the issue if no more usage, we can still reopen it if necessary. cc @taoohong

@imeoer imeoer closed this as completed Jun 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@imeoer @adamqqqplay @taoohong