SampleBuilder can't handle H264

[Sean-Der]

We just got a report in Slack from a user that wasn't able to use rtp-to-webrtc with H264.

I am not sure what the issues are, investigating now.

[Sean-Der]

It seems like the SampleBuilder is condensing the SPS/PPS/IDR into one RTP packet and that is breaking things.

GStreamer sends them as three distinict packets, with one timestamp. We then emit a Sample with them all in one Payload. That one Payload gets turned into one RTP packet.

[WhalenChan]

Definitely, there‘s an issue with H264. My test case(examples/rtp-to-webrtc) is under Pion webrtc version 3.0.4.

Here‘s one piece of my code.

videoTrack, err := webrtc.NewTrackLocalStaticRTP(webrtc.RTPCodecCapability{ MimeType: webrtc.MimeTypeH264, SDPFmtpLine: "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f", }, "video", "pion")

Environment: Windowns 10, Internal network with IP camera using UDP to push media streams(standard RTP packet).

1. IP camera:HIK VISION（海康威视）. Push RTP packet to the the specific port, work well in both web page and VLC player.

2. Another IP camera: I have no idea about it's manufacturer. Work well only in VLC player, it seems there's something wrong in web page.

3. Use ffmpeg command line ffmpeg -f gdigrab -i desktop -vcodec libx264 -preset:v ultrafast -tune:v zerolatency -f rtp udp://192.168.1.88:3232 to push screen record. The result just like the test case 2 above.

It's weird when I use rtsp(H264) with this repo(https://github.com/deepch/RTSPtoWebRTC), all my IP camera work well in web page.

Could you please give a clue about my problems?

Thanks a lot!

[Sean-Der]

Hey @WhalenChan. Your issue isn't related to the SampleBuilder, but happy to help!

My guess is that you aren't attaching a SPS/PPS to every IDR. You can see here the work that is involved.

Can you try rtsp-bench it is a much more naieve version and might be all you need.

Would you mind moving this conversation to Slack or the GitHub Discussions? Just so we can keep this issue related to original issue.

thanks!

[WhalenChan]

@Sean-Der I got it, thanks for your help!

[maddanio]

did you get it to work? do you have an ffmpeg command line or patch to pion to make it work? we need to stream both from ffmpeg and hik cameras.

[maddanio]

what I gather from the sdp that ffmpeg spits out is it always uses packetization mode 1 (non-interleaved single nalu, with large nalus being fragmented into fu-a)

[maddanio]

btw: does pion have to actually do anything to the packets? I would have thought it can just forward them untouched. thats what we do in our own rtsp proxy implementation, though not sure if that works for webrtc?

[maddanio]

ok, I managed to solve it by adding some package processing stolen from here

[liceim]

Definitely, there‘s an issue with H264. My test case(examples/rtp-to-webrtc) is under Pion webrtc version 3.0.4.

Here‘s one piece of my code.

videoTrack, err := webrtc.NewTrackLocalStaticRTP(webrtc.RTPCodecCapability{ MimeType: webrtc.MimeTypeH264, SDPFmtpLine: "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f", }, "video", "pion")

Environment: Windowns 10, Internal network with IP camera using UDP to push media streams(standard RTP packet).

1. IP camera:HIK VISION（海康威视）. Push RTP packet to the the specific port, work well in both web page and VLC player.
2. Another IP camera: I have no idea about it's manufacturer. Work well only in VLC player, it seems there's something wrong in web page.
3. Use ffmpeg command line ffmpeg -f gdigrab -i desktop -vcodec libx264 -preset:v ultrafast -tune:v zerolatency -f rtp udp://192.168.1.88:3232 to push screen record. The result just like the test case 2 above.

It's weird when I use rtsp(H264) with this repo(https://github.com/deepch/RTSPtoWebRTC), all my IP camera work well in web page.

Could you please give a clue about my problems?

Thanks a lot!

Definitely, there‘s an issue with H264. My test case(examples/rtp-to-webrtc) is under Pion webrtc version 3.0.4.

Here‘s one piece of my code.

videoTrack, err := webrtc.NewTrackLocalStaticRTP(webrtc.RTPCodecCapability{ MimeType: webrtc.MimeTypeH264, SDPFmtpLine: "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f", }, "video", "pion")

Environment: Windowns 10, Internal network with IP camera using UDP to push media streams(standard RTP packet).

1. IP camera:HIK VISION（海康威视）. Push RTP packet to the the specific port, work well in both web page and VLC player.
2. Another IP camera: I have no idea about it's manufacturer. Work well only in VLC player, it seems there's something wrong in web page.
3. Use ffmpeg command line ffmpeg -f gdigrab -i desktop -vcodec libx264 -preset:v ultrafast -tune:v zerolatency -f rtp udp://192.168.1.88:3232 to push screen record. The result just like the test case 2 above.

It's weird when I use rtsp(H264) with this repo(https://github.com/deepch/RTSPtoWebRTC), all my IP camera work well in web page.

Could you please give a clue about my problems?

Thanks a lot!

ffmpeg -re -f lavfi -i testsrc=size=640x480:rate=30 -pix_fmt yuv420p -c:v libx264 -bsf:v h264_mp4toannexb -cpu-used 5 -deadline 1 -g 10 -error-resilient 1 -auto-alt-ref 1 -f rtp rtp://127.0.0.1:5004?pkt_size=1200

Use this ffmpeg command , work well in both web page and VLC player.

[maddanio]

That is exactly the example i based my version on. I added signaling via websocket and had to modify the h264 nalus by adding start codes.

[xiaodeedaijia]

Hello,I made many tests.And I find that web can play normal if we remove -preset ultrafast.
ffmpeg -re -f lavfi -i testsrc=size=640x480:rate=30 -pix_fmt yuv420p -c:v libx264 -g 10 -preset ultrafast -tune zerolatency -f rtp rtp://127.0.0.1:5004?pkt_size=1200
In this way,just VLC can play,not the web.
ffmpeg -re -f lavfi -i testsrc=size=640x480:rate=30 -pix_fmt yuv420p -c:v libx264 -g 10 -tune zerolatency -f rtp rtp://127.0.0.1:5004?pkt_size=1200
And in this way,both VLC and web can play.
WHY IS THAT? I can't understand,but we can think about it.