Use subdirectories in publish for native

[brthor]

First pass on including subdirectories for use in Native.

PTAL @schellap @gkhanna79

[schellap]

This sounds good. Does it run fine on the E2E test? We should co-ordinate checking in since CoreRT publish needs to have --native-subdirectory flag for --ilcpath to not break CoreRT CI.

For your future PR, corerun/corehost has to be copied as well and the appdep build hack to drop inside this folder.

[brthor]

@schellap Waiting on results here for E2E.

[brthor]

@schellap I am seeing that on windows, the subdirectories are included and things are just fine.

On Ubuntu, the files are getting flat packed.

Can you quickly confirm that the Ubuntu NuGet Packages are structured to take advantage of the subdirectories? This will help diagnose what's going wrong here.

[schellap]

@brthor, can you give permissions to your fork so I can buddy test for you? Or if you find it easy, you can clone dotnet/corert and switch to native-subdir branch and drop a CLI (your stage2) inside /bin/tools/cli/.

dotnet/corert#580

[schellap]

[schellap]

Are your path separators good?

[brthor]

@schellap added as a collaborator

[schellap]

@brthor, I'll test this on our repo once you are ready to go with CLI CI.

[brthor]

Error: Unable to load DLL 'jitinterface': The specified module could not be found.

@schellap Do you know how this dlll is being loaded?

[brthor]

This dll is moved to the path bin/lib/dotnet/jitinterface.dylib

It looks like this dll is picked up by ILCompiler, @schellap does it need accompanying changes?

[schellap]

This needs to be PInvoked by ILC. Is there a way it can land next to ilc like ryujit? Eventually it might come from ryujit like package.

[brthor]

The dll would need to not be in a subdirectory in the nuget package, so it doesn't get published to a subdirectory when publishingdotnet-compile-native

Think this:
https://github.com/dotnet/corert/blob/master/src/packaging/packages.targets#L53

<Text><![CDATA[        <file src="../%(Identity)" target="runtimes/$(NuPkgRid)/native/lib/dotnet/%(Identity)" /> ]]></Text>

Would need the path changed:

<Text><![CDATA[        <file src="../%(Identity)" target="runtimes/$(NuPkgRid)/native/%(Identity)" /> ]]></Text>

[schellap]

Ah, I see. I thought the lib/dotnet/.dll was required by NuGet and publish for it to be picked up. I was just going to ask if we can omit special folders like lib, sdk that are "known names" and split on "native/content". I will get a PR and new packages out for you!

[schellap]

@brthor, having it at root of native folder won't break when NuGet content support comes in and native-subdirectory goes away, right?

[brthor]

At that point we'd need to take it out of the native folder anyways wouldn't we?

[schellap]

Yeah, I realized that later. Anyway, we would have to make too many changes, so it is a non-issue.

[schellap]

How does this look for you? 1.0.4-prerelease-00005

[schellap]

I don't think we need to remove from the native folder as it is a native library. We would however change the layout of the SDK package to use the appropriate content folders. Note: jitinterface.dylib is not content but a PInvoke binary which should just do what other PInvoke binaries do. How is always using --native-subdirectory not an issue for the CLI, you guys don't have any PInvoke libraries?

The real issue is that --native-subdirectory hack overrides known folder (sdk/lib) treatment. I am fine with your current plan as anyway changes are necessary in the future and the support for content will come sooner.

[gkhanna79]

The changes, at a highlevel, look fine to me. However, have you looked into the OSX failures?

Also, please do squash the commits.

[brthor]

@gkhanna79 OSX failures are related to the discussion above. This would happen on Ubuntu as well but the test is skipped for the moment. Will definitely squash the commits, thanks.

@schellap I see what you are saying with this dll that it will not be part of the contents and will remain in native. Looking into similar packages though, the lib/dotnet portion is typically used instead of native not with it.

For example in runtime.ubuntu.14.04-x64.Microsoft.NETCore.ConsoleHost version 1.0.0-rc2-23616 I see the following path for native components: runtimes\ubuntu.14.04-x64\native

Whereas in runtime.win7.System.IO.Pipes I see the following path (for a managed dll): runtimes\win7\lib\dotnet5.4

I think removing the lib/dotnet portion will not affect the usability of the package even after content support comes online.

[brthor]

@dotnet-bot test this please

should incorporate that latest pushed package

[brthor]

@dotnet-bot test Ubuntu Debug Build please

[brthor]

@Sridhar-MS This centos error seems to be caused by an outdated stage0 package. Can I safely ignore this for now, or should I wait on it for the merge?

[Sridhar-MS]

@brthor Let me quickly create an updated stage0 for centos. Merging this at this point will fail all the PRs.

[Sridhar-MS]

@dotnet-bot test CentOS7.1 Debug Build please

[Sridhar-MS]

@dotnet-bot test CentOS7.1 Release Build please

[Sridhar-MS]

@brthor The CentOS builds are passing, go ahead with the merge. Meanwhile I am also setting up the internal CI to push latest CentOS tarballs to azure blob storage.

[brthor]

Excellent 👍