Volume Cloning creates Blob Container in wrong Storage Account

[acortelyou]

I am trying to use the CSI Volume Cloning feature to copy a blob volume from a source storage account with Blob Data Reader into a destination storage account with Blob Data Contributor.

This is desirable in order to dynamically provision a copy of a blob container from a read-only global source storage account into a read-write cluster adjacent regional destination storage account, which can then be automatically reclaimed when no longer needed.

It appears that when cloning a blob volume, the controller server mistakenly attempts to provision the new blob container in the source storage account instead of the destination storage account.

The system identity does not have the permissions to write to the source storage and fails.

I am including a lightly anonymized copy of the logs and kubernetes resources for repro.

What happened:

controller.go:1366] provision "datacopy" class "datacopy": started

event.go:298] Event(External provisioner is provisioning volume for claim "datacopy"

utils.go:104] GRPC call: /csi.v1.Controller/CreateVolume

utils.go:105] GRPC request: {"capacity_range":{"required_bytes":274877906944},"name":"pvc-f0b58822-7342-4933-85b1-17ecf2ea94a6","parameters":{"csi.storage.k8s.io/pv/name":"pvc-f0b58822-7342-4933-85b1-17ecf2ea94a6","csi.storage.k8s.io/pvc/name":"data","protocol":"fuse2","resourceGroup":"resourceGroupDatacopy","storageAccount":"storageAccountDatacopy","storeAccountKey":"false"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["--cancel-list-on-mount-seconds=0","-o allow_other","-o attr_timeout=120","-o entry_timeout=120","-o negative_timeout=120"]}},"access_mode":{"mode":3}}],"volume_content_source":{"Type":{"Volume":{"volume_id":"resourceGroupDatasource#storageAccountDatasource#containerNameDatasource"}}}}

controllerserver.go:802] use system-assigned managed identity to authorize azcopy

controllerserver.go:750] begin to copy blob container datasource to pvc-f0b58822-7342-4933-85b1-17ecf2ea94a6

controllerserver.go:760] copy blob container datasource to pvc-f0b58822-7342-4933-85b1-17ecf2ea94a6

controllerserver.go:767] CopyBlobContainer(resourceGroupDatasource, storageAccountDatasource, https://storageAccountDatasource.blob.core.windows.net/pvc-f0b58822-7342-4933-85b1-17ecf2ea94a6) failed with error(exit status 1): 

INFO: Scanning...
INFO: Login with identity succeeded.
INFO: Authenticating to destination using Azure AD
INFO: Authenticating to source using Azure AD
INFO: Failed to create one or more destination container(s). Your transfers may still succeed if the container already exists.
INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission PUT https://storageAccountDatasource.blob.core.windows.net/pvc-f0b58822-7342-4933-85b1-17ecf2ea94a6/file4.bin

What you expected to happen:

A dynamic blob container should have been provisioned in the storage account specified in the datacopy storage class.

How to reproduce it:

See example kubernetes specs below.

Anything else we need to know?:

Functioning volume cloning is a highly desirable feature.

Environment:

• CSI Driver version:
  mcr.microsoft.com/oss/kubernetes-csi/blob-csi:v1.23.4
• Kubernetes version (use kubectl version):
  Client Version: v1.29.2
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.28.10

# datasource volume for existing blob container in existing storage account on which the aks system identity has Blob Data Reader
apiVersion: v1
kind: PersistentVolume
metadata:
  name: datasource
spec:
  capacity:
    storage: 1
  accessModes:
  - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ''
  claimRef:
    name: datasource
    namespace: blob-csi-test
  csi:
    driver: blob.csi.azure.com
    volumeHandle: resourceGroupDatasource#storageAccountDatasource#containerNameDatasource
    volumeAttributes:
      protocol: fuse2
      storeAccountKey: 'false'
      resourceGroup: resourceGroupDatasource
      storageAccount: storageAccountDatasource
      containerName: containerNameDatasource

---
# datasource volume claim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datasource
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 1
  storageClassName: ''
  volumeName: datasource

---
# storage class for dynamic blob container in existing storage account on which the aks system identity has Blob Data Contributor
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: datacopy
provisioner: blob.csi.azure.com
parameters:
  protocol: fuse2
  storeAccountKey: 'false'
  resourceGroup: resourceGroupDatacopy
  storageAccount: storageAccountDatacopy
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
  - --cancel-list-on-mount-seconds=0
  - -o allow_other
  - -o attr_timeout=120
  - -o entry_timeout=120
  - -o negative_timeout=120

---
# datacopy volume claim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datacopy
spec:
  accessModes:
  - ReadOnlyMany
  storageClassName: datacopy
  resources:
    requests:
      storage: 256Gi
  dataSource:
    kind: PersistentVolumeClaim
    name: datasource

---
# datareader to consume the datacopy
apiVersion: apps/v1
kind: Deployment
metadata:
  name: datareader
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: datareader
  template:
    metadata:
      labels:
        app.kubernetes.io/name: datareader
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: datacopy
      containers:
      - name: datareader
        image: mcr.microsoft.com/cbl-mariner/base/core:2.0
        volumeMounts:
        - name: data
          mountPath: /vol/data
        command:
          - /bin/bash
          - -c
          - |-
            ls -alR /vol
            sleep 3600s
            exit 0

[acortelyou]

I believe the copyVolume method should receive dstResourceGroup, dstAccountName parameters from CreateVolume to be used inside copyBlobContainer which currently builds the dstPath using the source accountName which is incorrect.

[andyzhangx]

/kind bug

[andyzhangx]

fixed in v1.23.5 and v1.24.2