[BUG] Nested directories creation throws an exception

[pditommaso]

Describe the bug

Creating nested directories via Files.createDirectories results in an exception.

Exception or Stack Trace

java.io.IOException: com.azure.storage.blob.models.BlobStorageException: Status code 404, (empty body)

	at com.azure.storage.blob.nio.AzureBasicFileAttributes.<init>(AzureBasicFileAttributes.java:59)
	at com.azure.storage.blob.nio.AzureBasicFileAttributeView.readAttributes(AzureBasicFileAttributeView.java:50)
	at com.azure.storage.blob.nio.AzureBasicFileAttributeView.readAttributes(AzureBasicFileAttributeView.java:20)
	at com.azure.storage.blob.nio.AzureFileSystemProvider.readAttributes(AzureFileSystemProvider.java:868)
	at com.azure.storage.blob.nio.AzureFileSystemProvider.checkAccess(AzureFileSystemProvider.java:801)
	at java.nio.file.Files.createDirectories(Files.java:746)
	at nextflow.cloud.azure.file.AzureFileSystemTest.should create directories(AzureFileSystemTest.groovy:92)
Caused by: com.azure.storage.blob.models.BlobStorageException: Status code 404, (empty body)
	at com.azure.core.http.rest.RestProxy.instantiateUnexpectedException(RestProxy.java:320)
	at com.azure.core.http.rest.RestProxy.lambda$ensureExpectedStatus$6(RestProxy.java:383)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onComplete(MonoFlatMap.java:174)
	at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:138)
	at reactor.core.publisher.FluxHandle$HandleSubscriber.onComplete(FluxHandle.java:206)
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:262)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:138)
	at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:195)
	at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:96)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
	at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:121)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252)
	at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
	at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:378)
	at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:373)
	at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:429)
	at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:625)
	at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:96)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1526)
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1287)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1324)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.kqueue.AbstractKQueueStreamChannel$KQueueStreamUnsafe.readReady(AbstractKQueueStreamChannel.java:544)
	at io.netty.channel.kqueue.AbstractKQueueChannel$AbstractKQueueUnsafe.readReady(AbstractKQueueChannel.java:381)
	at io.netty.channel.kqueue.KQueueEventLoop.processReady(KQueueEventLoop.java:211)
	at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:289)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
		at reactor.core.publisher.Mono.block(Mono.java:1680)
		at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:99)
		at com.azure.storage.blob.specialized.BlobClientBase.getPropertiesWithResponse(BlobClientBase.java:708)
		at com.azure.storage.blob.specialized.BlobClientBase.getProperties(BlobClientBase.java:686)
		at com.azure.storage.blob.nio.AzureBasicFileAttributes.<init>(AzureBasicFileAttributes.java:57)
		at com.azure.storage.blob.nio.AzureBasicFileAttributeView.readAttributes(AzureBasicFileAttributeView.java:50)
		at com.azure.storage.blob.nio.AzureBasicFileAttributeView.readAttributes(AzureBasicFileAttributeView.java:20)
		at com.azure.storage.blob.nio.AzureFileSystemProvider.readAttributes(AzureFileSystemProvider.java:868)
		at com.azure.storage.blob.nio.AzureFileSystemProvider.checkAccess(AzureFileSystemProvider.java:801)

To Reproduce

Use the following snippet below (Spock test) or equivalent code.

Code Snippet

        when:
        def dirs = fileSystem.getPath( 'mydir1/mydir2/mydir3')
        Files.createDirectories(dirs)
        then:
        Files.isDirectory(fileSystem.getPath('mydir1'))
        Files.isDirectory(fileSystem.getPath('mydir1/mydir2'))
        Files.isDirectory(fileSystem.getPath('mydir1/mydir2/mydir3'))

Expected behavior

The nested structure directory should be created.

Screenshots

--

Setup (please complete the following information):

• OS: MacOS
• IDE : IntelliJ
• Version of the Library used: com.azure:azure-storage-blob-nio:12.0.0-beta.2

Additional context
Add any other context about the problem here.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[rickle-msft]

@pditommaso Thanks for reporting this! My hypothesis is that if you create each directory individually this should work and that there's an extra check or operation happening inside the Paths method that I haven't supported yet. It's definitely important to make sure this method works, though, so I'll make that happen

[pditommaso]

Thanks a lot. This a blocker for us and not even sure how to patch it.

[rickle-msft]

Sounds good! I'll work to get it out soon!

[rickle-msft]

The underlying problem here is that check access was not supported. I just put out a PR to fix that. That unveiled another issue here, though, which is that containers(root directories) and blobs (directories and files) are handled a little bit differently. The way that shows up here is trying to call this method when the top level directory is held in a root still throws an exception. I need to consider the best option to work around that.

[pditommaso]

Indeed, I've tried your patch and I'm getting the error you are mentioning:

java.lang.IllegalArgumentException: Root directory not supported. Path: my-data:

	at com.azure.storage.blob.nio.AzureResource.validateNotRoot(AzureResource.java:209)
	at com.azure.storage.blob.nio.AzureResource.<init>(AzureResource.java:53)
	at com.azure.storage.blob.nio.AzureBasicFileAttributes.<init>(AzureBasicFileAttributes.java:57)
	at com.azure.storage.blob.nio.AzureBasicFileAttributeView.readAttributes(AzureBasicFileAttributeView.java:50)
	at com.azure.storage.blob.nio.AzureBasicFileAttributeView.readAttributes(AzureBasicFileAttributeView.java:20)
	at com.azure.storage.blob.nio.AzureFileSystemProvider.readAttributes(AzureFileSystemProvider.java:888)
	at com.azure.storage.blob.nio.AzureFileSystemProvider.checkAccess(AzureFileSystemProvider.java:811)
	at java.nio.file.Files.createDirectories(Files.java:746)