Marshal.SizeOf<T>() not working on linux

[ondrejtomcik]

I have a problem with Marshal.SizeOf<T>() in linux. When I run .net core 1.1 application on windows, it's working properly. When I run it on linux, I always receive exception:

Unhandled Exception: System.ArgumentException: Type 'ListenContext' cannot be marshaled as an unmanaged structure; no meaningful size or offset can be computed.

Here is the ListenContext

[StructLayout(LayoutKind.Sequential)]
public struct ListenContext
{
    public FindCallback Callback;
    public IClientWrapper ClientWrapper;

    public ListenContext(FindCallback cb, IClientWrapper cw)
    {
        Callback = cb;
        ClientWrapper = cw;
    }
}

FindCallback

[UnmanagedFunctionPointer(CallingConvention.StdCall)]
public delegate void FindCallback(OCResource res);

I can post also IClientWrapper, but not sure if that's a problem because when I run Marshal.SizeOf<FindCallback>() I receive same error.

What is the difference between executing the command on linux and on windows and where could be the issue? Thanks.

[jkotas]

There is no COM interop on Unix, and so the IClientWrapper interface cannot be marshalled on Unix. It means that the size of the marshalled structure cannot be computed.

What are you using the value returned from Marshal.SizeOf<FindCallback>() for?

[ondrejtomcik]

Hello @jkotas, thanks for an answer.
I am wrapping IoTivity c libraries. This is exact use-case:

public OCStackResult ListenForResource(string serviceUrl, string resourceType, OCConnectivityType connectivityType, FindCallback callback, QualityOfService qoS)
        {
            if (callback == null)
                return OCStackResult.OC_STACK_INVALID_PARAM;
            
            OCStackResult result;
            var resourceUri = new StringBuilder();
            resourceUri.Append(serviceUrl).Append(resourceType);

            
            var context = new ListenContext(callback, this);
            var pnt = Marshal.AllocHGlobal(Marshal.SizeOf<ListenContext>());
            Marshal.StructureToPtr(context, pnt, false);

            var cbdata = new OCCallbackData
            {
                context = pnt,
                cb = Marshal.GetFunctionPointerForDelegate(new OCClientResponseHandler(ListenCallback)),
                cd = Marshal.GetFunctionPointerForDelegate(new OCClientContextDeleter((ctx) => { }))
            };

            OCHeaderOption[] opt = {};
            lock (mLock)
            {
                result = OcStack.OCDoResource(IntPtr.Zero, OCMethod.OC_REST_DISCOVER, resourceUri.ToString(),
                    IntPtr.Zero, IntPtr.Zero, connectivityType, (OCQualityOfService) qoS, ref cbdata, opt, 0);
            }
            
            return result;
        }

[yizhang82]

@ondrejtomcik It appears that OCCallbackData is only asking for a opaque void * for the context - and you want to put arbitrary managed data structure on it. The best way is perhaps to allocate a GCHandle and point to your class instead. This way you don't need to marshal anything and can put any data you want in the data structure.

[yizhang82]

@ondrejtomcik Given that this is a known limitation (COM interop not supported in Linux) and your code doesn't really need it, I'm closing this issue for now. Let us know if you still have issues and feel free to re-open it. Thanks!

[ondrejtomcik]

Hello @yizhang82, thanks for the reply. I was on holiday. You're right, it was not necessary to put whole data structure there. I am now using GCHandle and callback is called properly. But I have two more question please. 1) Should I call GCHandle.Alloc with pinned option? 2) Callback is called but I cannot figure out why some of the expected data are not there. Code:

            var ctx = new SetContext(callback);
            OCCallbackData cbdata;
            var gch = GCHandle.Alloc(ctx);
            cbdata.context = GCHandle.ToIntPtr(gch);
            cbdata.cb = Marshal.GetFunctionPointerForDelegate(new OCClientResponseHandler(SetResourceCallback));
            cbdata.cd = Marshal.GetFunctionPointerForDelegate(new OCClientContextDeleter(ptrw => { }));

            OcStack.OCDoResource(IntPtr.Zero, OCMethod.OC_REST_POST,  ..., ref cbdata, ...);

SetResourceCallback is called

OCStackApplicationResult SetResourceCallback(IntPtr ctx, IntPtr handle, ref OCClientResponse clientResponse) {
            var gch = GCHandle.FromIntPtr(ctx);
            var context = (SetContext)gch.Target;
            var result = clientResponse.result; // this value exist
            var connType = clientResponse.connType; //also fine

But clientResponse.payload is IntPtr.Zero and I don't know why. There should be data.

OCClientResponse:

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]
    public struct OCClientResponse {
        public OCDevAddr devAddr;
        public IntPtr addr;
        public OCConnectivityType connType;
        public OCIdentity identity;
        public OCStackResult result;
        public uint sequenceNumber;

        [MarshalAs(UnmanagedType.LPStr)]
        public string resourceUri;

        public IntPtr payload;
        public byte numRcvdVendorSpecificHeaderOptions;

        [MarshalAs(UnmanagedType.ByValArray, SizeConst = 50, ArraySubType = UnmanagedType.Struct)]
        public OCHeaderOption[] rcvdVendorSpecificHeaderOptions;
    }

in C:

typedef struct
{
    OCDevAddr devAddr;
    OCDevAddr *addr;
    OCConnectivityType connType;
    OCIdentity identity;
    OCStackResult result;
    uint32_t sequenceNumber;
    const char * resourceUri;
    OCPayload *payload;
    uint8_t numRcvdVendorSpecificHeaderOptions;
    OCHeaderOption rcvdVendorSpecificHeaderOptions[MAX_HEADER_OPTIONS];
} OCClientResponse;

Where could be an issue? Thank you for your offtopic support! :)

[yizhang82]

@ondrejtomcik

1. Should I call GCHandle.Alloc with pinned option?

Pinning is unnecessary since you are only accessing the target from managed code and that is always safe to do. It is only needed if you need to pass the address of the target itself into native and therefore need to prevent it from moving. GCHandle value itself would never move since it is a unmanaged pointer to internal GC handle data structure that GC knows about, and is not really a managed object per-se and not subject to garbage collection and compaction.

2. Callback is called but I cannot figure out why some of the expected data are not there. Code:

Unfortunately without seeing your type definition for OCDevAddr/OCConnectivityType/OcIdentity/OCStackResult/etc, it's hard for me to tell where exactly it went wrong. There are a few things you can do to help you locate the issue:

• Call Marshal.OffsetOf for each field and compare it with native struct offset and see which one is off.
• Print out value of each field and see if all the field makes sense, and which ones are off
• Double check the size / alignment for each type and compare with native

[ondrejtomcik]

Hi @yizhang82 , thank you for the explanation and the idea how to test it. Marshaling is a new area for me. Therefore thanks!
But, if I may have one more question :) One pointer which should not be null is in c# always null. So I compared offsets and yes, that's the problem, but I cannot find a solution how to properly define those objects in c#.

typedef struct OCRepPayloadValue
{
    char* name;
    OCRepPayloadPropType type;
    union
    {
        int64_t i;
        double d;
        bool b;
        char* str;

        /** ByteString object.*/
        OCByteString ocByteStr;

        struct OCRepPayload* obj;
        OCRepPayloadValueArray arr;
    };
    struct OCRepPayloadValue* next;

} OCRepPayloadValue;

This I defined as

[StructLayout(LayoutKind.Sequential)]
    public class OCRepPayloadValue {
        [MarshalAs(UnmanagedType.LPStr)]
        public string name;
        public OCRepPayloadPropType type;
        public RepPayloadEnum rpEnum;
        public IntPtr next;
    }
[StructLayout(LayoutKind.Explicit)]
    public struct PayloadArrayUnion {
        [FieldOffset(0)]
        public IntPtr iArray;

        [FieldOffset(0)]
        public IntPtr dArray;

        [FieldOffset(0)]
        public IntPtr bArray;

        [FieldOffset(0)]
        public IntPtr strArray;

        [FieldOffset(0)]
        public IntPtr ocByteStrArray;

        [FieldOffset(0)]
        public IntPtr objArray;
    }

    [StructLayout(LayoutKind.Explicit)]
    public struct RepPayloadEnum {
        [FieldOffset(0)]
        public Int64 i;

        [FieldOffset(0)]
        public double d;

        [FieldOffset(0)]
        public bool b;

        [FieldOffset(0)]
        public IntPtr str;

        [FieldOffset(0)]
        public OCByteString ocByteStr;

        [FieldOffset(0)]
        public IntPtr obj;

        [FieldOffset(0)]
        public IntPtr arr;
    }

But problem is that offset of next in C is 56, in .NET 32. As you can see, I specified arr field as IntPtr and not as OCRepPayloadValueArray. When I change it to OCRepPayloadValueArray I always get exception: Unhandled Exception: System.TypeLoadException: Could not load type 'IoTivityCore.Stack.RepPayloadEnum' from assembly 'IoTivityNet, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' because it contains an object field at offset 0 that is incorrectly aligned or overlapped by a non-object field.

OCRepPayloadValueArray in C:

#define MAX_REP_ARRAY_DEPTH 3
typedef struct OCRepPayloadValueArray
{
    OCRepPayloadPropType type;
    size_t dimensions[MAX_REP_ARRAY_DEPTH];

    union
    {
        int64_t* iArray;
        double* dArray;
        bool* bArray;
        char** strArray;

        /** pointer to ByteString array.*/
        OCByteString* ocByteStrArray;

        struct OCRepPayload** objArray;
    };
} OCRepPayloadValueArray;

in C#:

[StructLayout(LayoutKind.Sequential)]
    public struct OCRepPayloadValueArray {
        public OCRepPayloadPropType type;

        //[MarshalAs(UnmanagedType.ByValArray, SizeConst = 3, ArraySubType = UnmanagedType.U4)]
        //[MarshalAs(UnmanagedType.ByValArray, SizeConst = 3)]
        public UIntPtr[] dimensions;

        public PayloadArrayUnion union;
    }
[StructLayout(LayoutKind.Explicit)]
    public struct PayloadArrayUnion {
        [FieldOffset(0)]
        public IntPtr iArray;

        [FieldOffset(0)]
        public IntPtr dArray;

        [FieldOffset(0)]
        public IntPtr bArray;

        [FieldOffset(0)]
        public IntPtr strArray;

        [FieldOffset(0)]
        public IntPtr ocByteStrArray;

        [FieldOffset(0)]
        public IntPtr objArray;
    }

Could you please point me to right direction? Thank you :)
I will no more spam this thread.

[yizhang82]

@ondrejtomcik Happy to help :)

Unhandled Exception: System.TypeLoadException: Could not load type 'IoTivityCore.Stack.RepPayloadEnum' from assembly 'IoTivityNet, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' because it contains an object field at offset 0 that is incorrectly aligned or overlapped by a non-object field.

This is most likely because your OCByteString type is a reference type. You can't have overlapping fields that GC reference (any reference types) or a non-GC reference (value types) - GC wouldn't know what to do with them. For example, 0x123 could either be an integer or an object reference (or even worse, half of a object reference), but GC wouldn't know which one since it doesn't have knowledge of your program.

The easiest way to fix this is to probably change OCByteString into a struct. But this also means OCByteString itself can't have any managed reference fields either, and recursively so for any struct fields it has. This can be rather inconvenient, but that's the price you have to pay to make it part of the union. There are other ways as well - such as using GCHandles to work around the limitation of can't have any GC fields, or declare a struct version of OCByteString strictly for holding the value but not the operations (those you would put the managed class).

BTW, you need to ByValArray for dimensions array since you need to 'embed' the elements into the structs (rather than a pointer):

[StructLayout(LayoutKind.Sequential)]
    public struct OCRepPayloadValueArray {
        public OCRepPayloadPropType type;

        [MarshalAs(UnmanagedType.ByValArray, SizeConst = 3)]
        public UIntPtr[] dimensions;

        public PayloadArrayUnion union;
    }

Your PayloadArrayUnion definition seems to be correct. Note that you can't define multiple arrays here because they would conflict with each other (C# / .NET wouldn't know which one to marshal so it marshals each one-by-one). Using IntPtr is the right thing to do.

[ondrejtomcik]

Hi @yizhang82 ,
sorry I forgot to include OCByteString. It is and was the structure. I had this exception because I was using string field type instead of IntPtr.

Thanks for explanation but I am not sure if problem is solved. As I mentioned, offset of "next" field is different as in C. It's because whole OcRepPayloadValue had size 16, but another 24 was missing to have correct offset of next.

What I did is that I specified size manually and I am not sure if it's correct approach. Can it be different in x86 and in x64? How to handle it?

[StructLayout(LayoutKind.Explicit, Size=40)]
    public struct RepPayloadEnum {

This marshalling and invoking is nice challenge, I am moving from problem to different problem :) Or let's say continuous improvement :) Now I have problem that if I call same method twice, callback is not called for the second time... But it will be something different, hope so. :)

[yizhang82]

It's because whole OcRepPayloadValue had size 16, but another 24 was missing to have correct offset of next.

It's difficult for me to interpret the numbers because I don't know whether 16 is 32-bit or 64-bit. Let's assume it is 32-bit (because it is too small for 64-bit). In order to get to 40, it means OCByteString has a pretty big size (probably inlines the strings as byte[]). My guess is that your OCByteString is defined incorrectly. One way to confirm that is to call Marshal.SizeOf and see if the size is different from native.

Usually I'd avoid using explicit sizes exactly because it doesn't work in different architectures and you need to build/deploy your assembly specific to each architecture. The best way is to make sure you define all the fields correctly.

Now I have problem that if I call same method twice, callback is not called for the second time...

Usually this is because the managed delegate is not kept alive - it needs to be alive when you pass to native function as a function pointer. Just a blind guess :)

[ondrejtomcik]

Hi again @yizhang82 :)
Problem with callback is not called for the second time was still not solved. I spent at least week on it. I used valgrind, gdb, custom trace logs and cannot figure out the problem so maybe you will be able to point me to the right way.

What is happening? There are two native functions I am calling.
OCDoResource(data,data,..., callback,... ); //posting data to cloud service
OCProcess(); // runs on background thread, processing incomming messages, taking care of heartbeat, etc

When I call OCDoResource twice, + on background is running the OCProcess(), second time the callback is not called and segfault occurs on malloc. Funny thing is then when I make it sequential, with 1500ms+ delay, it's working.
OCDoResource
timeout 1500ms
OCProcess ... callback called
OCDoResource
OCProcess
callbackcalled

If there is no timeout, second callback is never called. I really don't know how where could be the problem. I found out, that when malloc is called before second callback, it has the same address of different field called payload. Always the same. In case I put timeout, this address is different. Do you have any idea where could be the problem? :)

[yizhang82]

@ondrejtomcik Sorry, I'm not familiar with the SDK you are using so I can't provide more concrete advice. The only thing I would suggest is to make sure the callback is properly "rooted" by a local/static so that GC don't collect it. But otherwise you need to work with the IoTivity C++ SDK folks to understand why callback is not called. Personally I would set a breakpoint right before the callback is called from IoTivity code and see if that's hit. If the code did execute the callback, but it didn't made it into the managed code, it's likely related to .NET Core, otherwise it might be something else unrelated.