Use Directory.CreateDirectory instead of DirectoryUtility to create the obj folder

[nkolev92]

Bug

Fixes: NuGet/Home#7908
Regression: Yes

• Last working version:
• How are we preventing it in future:

Fix

Details:

System.IO.DirectoryNotFoundException: Could not find a part of the path '/Users/vsts/agent/2.148.1/work/1/s/artifacts/obj/70d88d93-1df8-47b8-8c56-447722545fd1'.
   at System.IO.FileSystem.CreateDirectory(String fullPath)
   at System.IO.Directory.CreateDirectory(String path)
   at NuGet.Common.DirectoryUtility.CreateSingleSharedDirectory(String path)
   at NuGet.Common.DirectoryUtility.CreateSharedDirectory(String path)
   at NuGet.Commands.NoOpRestoreUtilities.PersistDGSpecFile(DependencyGraphSpec spec, RestoreRequest request, ILogger log)
   at NuGet.Commands.NoOpRestoreUtilities.GetHash(RestoreRequest request)
   at NuGet.Commands.RestoreCommand.EvaluateCacheFile()
   at NuGet.Commands.RestoreCommand.ExecuteAsync(CancellationToken token)
   at NuGet.Commands.RestoreRunner.ExecuteAsync(RestoreSummaryRequest summaryRequest, CancellationToken token)
   at NuGet.Commands.RestoreRunner.ExecuteAndCommitAsync(RestoreSummaryRequest summaryRequest, CancellationToken token)
   at NuGet.Commands.RestoreRunner.CompleteTaskAsync(List`1 restoreTasks)
   at NuGet.Commands.RestoreRunner.RunAsync(IEnumerable`1 restoreRequests, RestoreArgs restoreContext, CancellationToken token)
   at NuGet.Commands.RestoreRunner.RunAsync(RestoreArgs restoreContext, CancellationToken token)
   at NuGet.Build.Tasks.RestoreTask.ExecuteAsync(ILogger log)

The scenario is restore on mac/linux & it repros equivalently on 2.1, 2.2 and 3.0 preview runtimes.

NuGet writes to various directories during restore. NuGet uses the /tmp directory and as it’s a machine-wide directory NuGet has to create a directory that’s shareable across users as multiple users.
We have an implementation, that basically creates a folder as the same level as the wanted one, changes the permissions and moves to the intended directory path.

NuGet restore is frequently the first tool to write to the obj folder during build. As such we need to create the obj folder. An oversight during an implementation of a new feature caused this CreateSharedDirectory method to be called for the obj folder. Now changing back to Directory.CreateDirectory fixes this particular problem, but understanding what exactly happened is important for our future investigation of issues in a similar codepath.

The scenario is the following. All the projects in a solution are redirected from root directory of the project obj to /obj/projectName/.
NuGet runs 8 concurrent project restore operations at the same time.
All 8 make a call to create the path/to/solution/root/obj/projectName.
Due to reasons I don’t understand Directory.CreateDirectory method throws with an error such as:
error : Could not find a part of the path '/home/nikolev/nuget.client/artifacts/39b567ca-81be-4b14-a271-a9be767a6888'. [/home/nikolev/nuget.client/src/NuGet.Core/NuGet.DependencyResolver.Core/NuGet.DependencyResolver.Core.csproj]

This only repros when there are at least 6-7 projects being restored. I was never able to repro it on with fewer projects.

I have added logs to the DirectoryUtility, and generated the logs.
https://github.com/NuGet/NuGet.Client/blob/1f9ad07fbbc5e80ab11eee4b4ea125d99dc6b291/src/NuGet.Core/NuGet.Common/PathUtil/DirectoryUtility.cs

The branch with the logs is dev-nkolev92-pathIssues

PathIssueLogs.txt

On the side I'll be working to truly understand the root cause.

Note:
After making the change, 1100 runs did not repro the problem.

Testing/Validation

Tests Added: No
Reason for not adding tests: Difficult to add tests for this scenario. We need cross platform dotnet tests which we currently do not have infrastructure for.
Validation: Manual