-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix ABI problem when marshalling X509VerifyStatusCode #52789
Conversation
The X509VerifyStatusCode type is defined as an enum in C code, but as a struct (with a single member) in C# code. This means that marshalling this type only works correctly on platforms where the ABI treats these two types as equivalent. This fails e.g. on Linux on s390x. Fixed by changing all native functions that take X509VerifyStatusCode as argument or return type to use a plain "int" instead.
Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq, @GrabYourPitchforks Issue DetailsThe X509VerifyStatusCode type is defined as an enum in C code, but as Fixed by changing all native functions that take X509VerifyStatusCode
|
I'm not fundamentally opposed here, but I am confused... what ends up happening on s390x? |
For return values, an integer is returned in general purpose register %r2. However, all structs, including a struct that contains only a single integer, are returned on the stack via implicit reference. So if one side of the marshalling assumes we need an implicit reference, and the other side doesn't, the arguments end up all confused. (This is different on x64 and arm, because there is an ABI special case where structs that contain only a single integer are not returned via implicit reference, but rather in the same register where a plain integer would be returned.) |
Ah, so the native library (which sees the type as an enum/integer) still does single register, but since the P/Invoke Marshaller sees the type as a struct it tries unpacking it per struct rules, resulting in a mismatch? |
Exactly. The marshaller prepares a buffer on the stack, and passes a pointer to that buffer to the native routine, expecting the native routine to fill (what it thinks to be) the struct to be returned in. But native code thinks it returns an integer, and so doesn't expect a buffer pointer but simply returns the integer in register. (Plus the presence of the return pointer shifts the other arguments by one slot, which native code also doesn't expect, so the incoming arguments are already wrong.) |
The X509VerifyStatusCode type is defined as an enum in C code, but as
a struct (with a single member) in C# code. This means that marshalling
this type only works correctly on platforms where the ABI treats these
two types as equivalent. This fails e.g. on Linux on s390x.
Fixed by changing all native functions that take X509VerifyStatusCode
as argument or return type to use a plain "int" instead.