Don't block specific generic types in field marshalling scenarios.

[jkoritzinsky]

Bing is using Marshal.SizeOf to calculate the size of structs that contain a Nullable<T> field. In the old system, this would unexpectedly work since we apparently allowed using generic structs as fields in interop types (but not generic classes). See the && ELEMENT_TYPE_CLASS at the end of the condition on line 208: https://github.com/dotnet/coreclr/blob/release/3.1/src/vm/fieldmarshaler.cpp#L208-L211

Semantically, the correct function for users to call here would be Unsafe.SizeOf<T>, but there's a large number of users who use Marshal.SizeOf instead of Unsafe.SizeOf for various reasons.

Fixes #31640

[jkotas]

Test?

[jkoritzinsky]

Added

[jkotas]

Thanks!

[jkotas]

Vector64/128/256 need to be blocked even in the field scenario.

.NET Core 3.x shipped without this block. E.g. the following works fine in .NET Core 3.x:

using System;
using System.Runtime.InteropServices;
using System.Runtime.Intrinsics;

struct Foo
{
    Vector64<int> f;
}

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine(Marshal.SizeOf<Foo>());
    }
}

Is adding the block for these a breaking change?

[tannergooding]

.NET Core 3.x shipped without this block.

Was it allowed to cross the P/Invoke boundary or just allowed in functions like Marshal.SizeOf<T>()?

[jkotas]

It was allowed to cross P/Invoke boundary too.

[tannergooding]

That's unfortunate. The intent was that it shouldn't be allowed as struct Mat4x4 { Vector128<float> x, y, z, w; } has special ABI requirements on some platforms (it can be passed in 4 registers; rather than by reference).

[jkoritzinsky]

I take it based on this feedback that we want to switch this back to only explicitly blocking ByReference<T> in field scenarios?

[jkotas]

For this PR, we should make it do what it did in .NET Core 3.0.

If you would like to propose breaking changes against the .NET Core 3.0 behavior, it should be separate issue / PR.

[jkoritzinsky]

Sounds good to me.

[jkotas]

Could you please add tests for the different cases we have discussed above?

[jkoritzinsky]

Will do!

[MichalStrehovsky]

Could you also match this with MarshalHelpers.cs in Crossgen2?

runtime/src/coreclr/src/tools/Common/TypeSystem/Interop/IL/MarshalHelpers.cs

Lines 337 to 355 in a716fa2

	// Blittable generics are allowed to be marshalled with the following exceptions:
	// * ByReference<T>: This represents an interior pointer and is not actually blittable
	// * Nullable<T>: We don't want to be locked into the default behavior as we may want special handling later
	// * Vector64<T>: Represents the __m64 ABI primitive which requires currently unimplemented handling
	// * Vector128<T>: Represents the __m128 ABI primitive which requires currently unimplemented handling
	// * Vector256<T>: Represents the __m256 ABI primitive which requires currently unimplemented handling
	// * Vector<T>: Has a variable size (either __m128 or __m256) and isn't readily usable for inteorp scenarios
	if (type.HasInstantiation && (!isBlittable
	|| InteropTypes.IsSystemByReference(context, type)
	|| InteropTypes.IsSystemNullable(context, type)
	|| InteropTypes.IsSystemRuntimeIntrinsicsVector64T(context, type)
	|| InteropTypes.IsSystemRuntimeIntrinsicsVector128T(context, type)
	|| InteropTypes.IsSystemRuntimeIntrinsicsVector256T(context, type)
	|| InteropTypes.IsSystemNumericsVectorT(context, type)))
	{
	// Generic types cannot be marshaled.
	return MarshallerKind.Invalid;
	}

If you need to narrow down to field marshalling only, the marshallerType parameter has it.

[jkoritzinsky]

Linux job failure is an intermittent issue tracked with AzDO.
MacOS failure is #2176

I'm going to merge this in so we can unblock Bing.