WHIP from OBS v30.1.2 on Windows/Apple platforms fails to establishe DTLS session with "Bad certificate" reply from OBS

[ianzag]

Dear developers,

I have an issue with recently released OBS v30.1.2 and WHIP Streaming. I'm trying to broadcast media stream to WHIP compatible server. Problem is DTLS handshake fails with Windows/Apple OBS official builds but can be successfully established with Linux official build. Not much in OBS logs in both cases, but I got SDP exchange and DTLS network traffic traces. Looks like depending on OBS's platform (and used TLS engine?) it behaves slightly different. Please see attached traces. Will be glad to supply any additional info. Thanks!
Bad_DTLS.txt
Bad_SDP.txt
Good_DTLS.txt
Good_SDP.txt

[paullouisageneau]

It looks indeed like a different TLS library behaving differently.

@Sean-Der Is the TLS library in OBS builds for Linux different from the one for Windows and OSX? I guess Windows and OSX rely on Mbed TLS so it would be an issue specific to MbedTLS.

[Sean-Der]

@paullouisageneau exactly!

Sorry I missed this @ianzag is this still a problem? Could I test against a server with the issue?

[ianzag]

@Sean-Der pardon for long response had very busy days. I'll check if it's still applicable tomorrow. On server side there's an ordinary OpenSSL's DTLS state machine so I believe it can be wrapped into dedicated regression test. How FooSSL performs with BarSSL.

[Sean-Der]

@ianzag Is this still a problem? Maybe OBS upgrade fixed?