Trying to add h265 support (for modern clients with wired connection) (was *** ERROR: unsupported h265 video detected)

[DYY-Studio]

iOS 16.5.1
iPad Pro 4 (11-inch)

s 1920x1342@60
nohold
FPSdata
fps 60

When I use Ethernet to AirPlay（Both iPad and my Windows Computer）, it shows the error message, but Wi-Fi would not.
Do we have any plan to support the h265 video？Thanks.

By the way, Wi-Fi here is too bad to AirPlay a fluent video, that's why I want to use Ethernet.

[thiccaxe]

Take a look at the discussion in #231

[DYY-Studio]

Thanks, I'll wait for good news.

Should I close the issue?

[fduncanh]

The h265 seems to be decoded properly, and now its recognized, it should be easy to handle =.

But it needs a gstreamer pipeline to be written. As of now I don't know if it will just be modifying "h264" to "h265", or more complicated. I will need to get a macO or iPadOS client to deliver h265. It seems ethernet connections instead of wifi are what triggers it?

@DYY-Studio, how to you get the iPad to use ethernet? Is there a dongle for that? I have one for a mac M2 with usb c
and an iPad m2 with usb c as well.

[DYY-Studio]

@fduncanh
Yes. I use an UGREEN USB-C Hub (Type-C to 2*USB-A + USB-C + HDMI + 3.5mm)
and then plug an UGREEN 1Gbps USB-A to RJ45 (AX88179A) into one of the USB-As.

The H265 just happened when I use Ethernet on iPad. I have tried using Ethernet on my Windows laptop but Wi-Fi on my iPad, it didn't show the error.

I haven't tried whether the connection between my laptop and iPad can reach full 1Gbps, but the Speed Test on both devices show that the connections to Internet are 1Gbps.

[thiccaxe]

I tried with a usb-c to ethernet adapter. I was achieving full gigabit speeds on LAN, but the ipad decided to use h264. I think I have an older model so I probably won't be able to help out here

[fduncanh]

@DYY-Studio

I have a
" USB C to Ethernet Adapter, uni Driver Free RJ45 to USB C [Thunderbolt 3/4 Compatible], 1Gbps Type-C Gigabit Ethernet LAN Network Adapter for MacBook Pro"

What else should I need? (M2 macbook pro and M2 ipad pro are available as test clients)

?just plug into client usb port and connect to 1GB hub of local network?

[DYY-Studio]

@fduncanh
I think that's all.
I just do like this.

The cable go directly to the router's 1Gbps port, and my laptop was using the router's Wi-Fi.

It seems that it's just about iPad's Ethernet connection.

• Both Wi-Fi

H264

• Both Ethernet

H265

• iPad Ethernet, Laptop Wi-Fi

H265

• Laptop Ethernet, iPad Wi-Fi

H264

All the Ethernet connection I used is 1Gbps.

[fduncanh]

@ionull @DYY-Studio
We have confirmed and duplicated the issue. We will see what can be done.

[fduncanh]

The difficulty in getting h265 supported is that the first video packet (which is unencrypted, "type 0x01") carries no payload:

• in h264 video, it delivers the SPS (sequence parameter set NAL unit) and PPS (picture parameter set NAL unit) , needed for the decoding process)

• in h265 video, there should be 3 such NAL units, SPS, PPS and VPS (Video parameter set), but the unencrypted initial packet does not seem to contain them. The first encrypted packet just contains the expected initial IDR video slice, and the subsequent packets contain regular video slices. But without the VPS+SPS+PPS, feeding them to the gstreamer h265parse element produces a "malformed NAL error".

??? why is the type 0x01 packet (see lib/raop_rtp_mirror.c) missing the VPS+SPS+PPS data? Where to find it?

Client identified as User-Agent: AirPlay/710.79.1
Accepted IPv4 client on socket 38
Local: 10.5.5.1
Remote: 10.5.5.2
raop_rtp_mirror starting mirroring
received video packet, header                   (header of initial unencrypted packet)
00 00 00 00 01 00 16 01 b8 c6 67 f5 83 0e 00 00    <===  the first 4 bytes 00 00 00 00 are the (zero)
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00          payload length, next byte is type 0x01 (unencrypted)
00 00 00 00 00 00 00 00 00 00 20 45 00 00 b4 44   <===  this data is picture width and height
00 00 00 00 00 00 00 00 00 00 20 45 00 00 b4 44   <===  this is width_source and height_source
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 


Received unencrypted codec packet from client: payload_size 0 header 01 00 16 01  ts_client = 3715.958615
raop_rtp_mirror, discard type 0x01 packet with no payload


received video packet, header            this is header of first encrypted packet
cb fa 03 00 00 10 00 00 b8 c6 67 f5 83 0e 00 00      <== first 4 bytes are payload  length (littleendian)    
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00     "type = 0x00"  (next byte)
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

decrypted payload,   size 260811    (size adds the initial 4 bytes 00 00 00 01 that code for a new NAL)
00 03 fa c7 28 01 af 56 0b 47 0d fa 07 f8 32 a1     <=== first 4 bytes repeat length  (bigendian), next
11 03 ac 16 b9 05 43 ef 8c 84 6d aa a2 6d 61 3d       two bytes 28 01 are the h265 header, indicating
23 03 dc 0a 10 d7 46 39 05 de 82 f8 ab df 3b a8       that it is a type 20 IDR video slice NAL unit.
d6 a7 78 2f ce b2 d9 3a d3 eb 90 a6 58 9a 9a a5 
c9 f6 aa d8 a0 fa 50 a9 45 a3 b8 88 aa 02 c7 3e 
1d 2a 81 9a 1b 48 cf c0 26 a7 a6 14 a3 41 cb a7 
6d 52 97 f3 7f a3 e6 a1 96 47 67 fa d9 fb eb f3 
<snip>

https://www.codeproject.com/tips/896030/the-structure-of-hevc-video

EDIT: maybe the VPS+SPS+PPS data is in the IDR slice but not as separated NALS?? If so, how to tell gstreamer h265parse ?

[fduncanh]

*Thinking some more about this, I'm wondering whether there is in fact a payload with the unencrypted packet containing the VPS, SPS and PPS NAL units, but Apple unhelpfully doesn't give the payload size when it is unencrypted. The test would be to see if more data can be read from the socket after getting the header. Since the first four bytes of the NAL units contain their length (which we then replace with the NAL start code 00 00 00 01) this would be a viable way to send the data.

Encrypted packets need to have a known size before decryption, and since their payload is encrypted, the size can't be read from the first 4 payload bytes, so Apple has to specify them in the packet header, but could get away with not specifying them when the payload is unencrypted (just to be mean!)

I will test this idea when I have the time.

• This didn't work. When I read from the socket after the initial type 0x01 packet with no payload, I just get the first type 0x00 packet with an encrypted payload.

• so how does Apple deliver the three NAL units VPS, SPS, PPS needed to set up decoding the h265 data???

• without solving this puzzle, adding support for h265/hevc video "hits a brick wall"

EDIT: It might be that the new h265 protocol won't send any unencrypted data to an AirPlay Server that does not use the newer AirPlay 2 protocol that used fully-encrypted server-client communications.??? Some wireshark studies on how h265 works on an AppleTV 4K might help, but its not clear if any other project besides UxPlay is interested in AirPlay video as opposed to Audio. Suggestions welcomed!

[fduncanh]

@DYY-Studio

You write

When I use Ethernet to AirPlay（Both iPad and my Windows Computer）

Just out of interest:

Are you using AirMyPC on the Windows computer (or something else)?

[DYY-Studio]

@fduncanh
No, I use UxPlay only now.

Other softwares I have tried once but in that time I didn't have the idea that to use Ethernet to enhance stability, and also I didn't have iPad.

Some of the software I tried limit the bitrate of AirPlay, the using time or the display size. That makes me try to find a better one, which leads me to UxPlay.
I remember I have tried AirMyPC once, but it seems like I didn't understand how to run it on a Windows PC that time?

I can't remember what softwares I have tried, sorry.

[fduncanh]

@DYY-Studio

I see. Am I correct that you are running UxPlay as an AirPlay server on a windows PC, using it to mirror an iPad client?

AirMyPC emulates an AirPlay client on windows

[DYY-Studio]

@fduncanh
Yes.
After all I don't have Apple TV and Mac.
And generally I prefer to use Microsoft Remote Desktop.

I want to replace iOS system screen-recorder but a good HDMI video capture card is too expensive. AirPlay seems like a good choice for its high bitrates and full resolution display.
(System record only provide 1920x1342, with long saving time to scale.)

[fduncanh]

This is an issue that can't be solved without understanding the Apple AirPlay2 protocol better.
A useful "fix" would be to systematically see if any of the undocumented bits in the two settings bytes
(se lib/dnssdint.h) perhaps tell the client NOT to use h265/hevc coding when connected using wired ethernet

closing for now.

[shuax]

I tried to copy the file header vps+sps+pps from other h265 streams. Obviously I failed. I continued to try more possibilities.

[fduncanh]

The mystery is WHERE does Apple send the vps+sps+pps?, or what causes the client to NOT send them to UxPlay with the unencrypted packet

[thiccaxe]

If nothing else I can suggest that you plug in your phone into your laptop and use idevice tools' logger to see if there are any clues

[connorh315]

I've got a 4K Apple TV and have captured the Wireshark packets (they were captured over awdl0 since it prefers p2p) - I'm willing to send them over to you, only issue is that I do believe the data is encrypted as after the pair-verify stage there is no readable transfer of data. I really would like to get H265 going as I want to make a portable equivalent of an Apple TV, I've got Bluetooth IP Address advertisement working now so it has more reliable discovery and I just want to get some crisper quality to go with it.

[fduncanh]

The Apple TV 4k will probably tell the client it supports coreutils pairing and encryption (features bit 48) some aspects of this are understood (but not by me). Uxplay is using legacy pairing. you could post what you found up to pair verify, just to see what the initial phase before encryption is. Probably wont help solve the h265 riddle though.

[connorh315]

The Apple TV 4k will probably tell the client it supports coreutils pairing and encryption (features bit 48) some aspects of this are understood (but not by me). Uxplay is using legacy pairing. you could post what you found up to pair verify, just to see what the initial phase before encryption is. Probably wont help solve the h265 riddle though.

Okay, I have a different idea, I'll see if I can set up a mitm proxy and the only thing I change is zeroing all pairing bits except for legacy pairing so that then the packets we're looking for should be readable to then determine if the h265 header contains anything different in that scenario.

[fduncanh]

If you could post the initial unencrypted communications you captured one can look at the features code to see what the apple TV is telling the client.

[connorh315]

Okay so I did some testing and after setting up a proxy for the Apple TV 4k (being spoofed so that it supports only the legacy pairing and the UxPlay features bits) the device will connect, get the full way through the handshake (and initiate the stream), but once the data port is sent from the server, the client will immediately initiate a TEARDOWN, almost as though the Apple TV isn't sending through enough info for it to start sending the stream?? I've literally got right up to the finish line here and it's refusing to go and send the next packet (which is the most important one as it will be the h265 header!!)

[fduncanh]

Reopened:some new info may be coming soon (waiting for more details)
See next comment:

[connorh315]

Okay nevermind, there is hardware accel decoding for Raspberry Pi 4 (and I assume 5) as when I playback a raw hevc video from online on VLC that says in the console "avcodec decoder: Using DRM Video Accel for Hardware Decoding" but the same message does not display when I play back a video dump. I'll see if I can investigate why this is happening...

I wonder if the hardware decoder just doesn't support the format it's being fed? Maybe bt709 isn't supported? Or maybe it's the chroma format?

[fduncanh]

the GST_DEBUG=4 messages will show if that is the case. (or if anything else like that is happening)

It was for the v4l2 ,h264, which is why the -bt709 option is there (that was a fix I used, before the fix I got into gstreamer
was accepted).

[connorh315]

the GST_DEBUG=4 messages will show if that is the case. (or if anything else like that is happening)

It was for the v4l2 ,h264, which is why the -bt709 option is there (that was a fix I used, before the fix I got into gstreamer was accepted).

I don't think I can get to that stage atm, since GStreamer doesn't have a h265 hardware decoder sink at all for the RPi 4 from what I can tell. Is that the same with you @fduncanh ? I do have most of the gst plugins installed (good, bad and base)

[fduncanh]

what modules does lsmod show are working (or at least loaded) when vlc is using acceleration.?

take a look at https://github.com/FDH2/UxPlay/wiki/Gstreamer-Video4Linux2-plugin-patches

[connorh315]

what modules does lsmod show are working (or at least loaded) when vlc is using acceleration.?

Could you explain what you need me to do sorry - Is it literally just run lsmod | grep bcm in the console with/without anything running or is there something else I'm supposed to do?

[fduncanh]

yes lsmod | grep bcm, probably,
but just post the full output of lsmod here, I'll look to see if anything else catches my eye.
This is pi 4b or pi5?

[connorh315]

yes lsmod | grep bcm, probably, but just post the full output of lsmod here, I'll look to see if anything else catches my eye. This is pi 4b or pi5?

Pi 4b

Full output (lsmod):
Module Size Used by
rfcomm 53248 4
snd_seq_dummy 12288 0
snd_hrtimer 12288 1
snd_seq 81920 7 snd_seq_dummy
snd_seq_device 16384 1 snd_seq
cmac 12288 3
algif_hash 12288 1
aes_arm64 12288 3
aes_generic 32768 1 aes_arm64
algif_skcipher 12288 1
af_alg 24576 6 algif_hash,algif_skcipher
bnep 24576 2
vc4 376832 5
brcmfmac_wcc 12288 0
snd_soc_hdmi_codec 20480 0
drm_display_helper 16384 1 vc4
cec 53248 1 vc4
drm_dma_helper 24576 3 vc4
brcmfmac 360448 1 brcmfmac_wcc
drm_kms_helper 212992 2 drm_dma_helper,vc4
hci_uart 49152 0
binfmt_misc 16384 1
btbcm 24576 1 hci_uart
snd_soc_core 286720 2 vc4,snd_soc_hdmi_codec
brcmutil 24576 1 brcmfmac
bluetooth 606208 33 hci_uart,btbcm,bnep,rfcomm
bcm2835_v4l2 40960 0
cfg80211 983040 1 brcmfmac
rpivid_hevc 45056 0
videobuf2_vmalloc 12288 1 bcm2835_v4l2
bcm2835_codec 45056 0
v4l2_mem2mem 45056 2 bcm2835_codec,rpivid_hevc
bcm2835_isp 28672 0
snd_compress 16384 1 snd_soc_core
bcm2835_mmal_vchiq 36864 3 bcm2835_codec,bcm2835_v4l2,bcm2835_isp
videobuf2_dma_contig 16384 3 bcm2835_codec,rpivid_hevc,bcm2835_isp
videobuf2_memops 12288 2 videobuf2_vmalloc,videobuf2_dma_contig
videobuf2_v4l2 32768 5 bcm2835_codec,bcm2835_v4l2,rpivid_hevc,v4l2_mem2mem,bcm2835_isp
videodev 303104 6 bcm2835_codec,videobuf2_v4l2,bcm2835_v4l2,rpivid_hevc,v4l2_mem2mem,bcm2835_isp
snd_bcm2835 24576 2
snd_pcm_dmaengine 16384 1 snd_soc_core
snd_pcm 139264 5 snd_bcm2835,snd_soc_hdmi_codec,snd_compress,snd_soc_core,snd_pcm_dmaengine
videobuf2_common 69632 9 bcm2835_codec,videobuf2_vmalloc,videobuf2_dma_contig,videobuf2_v4l2,bcm2835_v4l2,rpivid_hevc,v4l2_mem2mem,videobuf2_memops,bcm2835_isp
ecdh_generic 16384 2 bluetooth
v3d 90112 5
raspberrypi_hwmon 12288 0
ecc 36864 1 ecdh_generic
snd_timer 36864 3 snd_seq,snd_hrtimer,snd_pcm
rfkill 32768 6 bluetooth,cfg80211
snd 110592 14 snd_seq,snd_seq_device,snd_bcm2835,snd_soc_hdmi_codec,snd_timer,snd_compress,snd_soc_core,snd_pcm
libaes 12288 3 aes_arm64,bluetooth,aes_generic
gpu_sched 49152 1 v3d
mc 61440 7 videodev,bcm2835_codec,videobuf2_v4l2,videobuf2_common,rpivid_hevc,v4l2_mem2mem,bcm2835_isp
vc_sm_cma 28672 2 bcm2835_mmal_vchiq,bcm2835_isp
drm_shmem_helper 24576 1 v3d
raspberrypi_gpiomem 12288 0
nvmem_rmem 12288 0
uio_pdrv_genirq 12288 0
uio 20480 1 uio_pdrv_genirq
drm 643072 17 gpu_sched,drm_kms_helper,drm_dma_helper,v3d,vc4,drm_shmem_helper,drm_display_helper
i2c_dev 16384 0
fuse 139264 5
dm_mod 143360 0
drm_panel_orientation_quirks 28672 1 drm
backlight 24576 2 drm_kms_helper,drm
ip_tables 32768 0
x_tables 49152 1 ip_tables
ipv6 569344 50

lsmod | grep bcm:
btbcm 24576 1 hci_uart
bluetooth 606208 33 hci_uart,btbcm,bnep,rfcomm
bcm2835_v4l2 40960 0
videobuf2_vmalloc 12288 1 bcm2835_v4l2
bcm2835_codec 45056 0
v4l2_mem2mem 45056 2 bcm2835_codec,rpivid_hevc
bcm2835_isp 28672 0
bcm2835_mmal_vchiq 36864 3 bcm2835_codec,bcm2835_v4l2,bcm2835_isp
videobuf2_dma_contig 16384 3 bcm2835_codec,rpivid_hevc,bcm2835_isp
videobuf2_v4l2 32768 5 bcm2835_codec,bcm2835_v4l2,rpivid_hevc,v4l2_mem2mem,bcm2835_isp
videodev 303104 6 bcm2835_codec,videobuf2_v4l2,bcm2835_v4l2,rpivid_hevc,v4l2_mem2mem,bcm2835_isp
snd_bcm2835 24576 2
snd_pcm 139264 5 snd_bcm2835,snd_soc_hdmi_codec,snd_compress,snd_soc_core,snd_pcm_dmaengine
videobuf2_common 69632 9 bcm2835_codec,videobuf2_vmalloc,videobuf2_dma_contig,videobuf2_v4l2,bcm2835_v4l2,rpivid_hevc,v4l2_mem2mem,videobuf2_memops,bcm2835_isp
snd 110592 14 snd_seq,snd_seq_device,snd_bcm2835,snd_soc_hdmi_codec,snd_timer,snd_compress,snd_soc_core,snd_pcm
mc 61440 7 videodev,bcm2835_codec,videobuf2_v4l2,videobuf2_common,rpivid_hevc,v4l2_mem2mem,bcm2835_isp
vc_sm_cma 28672 2 bcm2835_mmal_vchiq,bcm2835_isp

[fduncanh]

Well you can see it all there...
you can compare to what you see when uxplay decodes h264 with v4l2.

The more obvious new module is rpivid_hevc.

[connorh315]

From what I've been reading I think the HEVC decoder block is very specific and only supports certain compression codecs. I don't think we have much control over what gets sent by the client but from memory I think there is a receiverSupports444 entry in one of the PLISTS that I can only assume is in ref. to HEVC 4:4:4. I was going to buy a compute module 4 to use as a plug-n-play UxPlay device but now I think I might buy an alternative that has a SOC with a more compliant HEVC decoder.

[fduncanh]

https://forums.raspberrypi.com/viewtopic.php?t=280180

https://forums.raspberrypi.com/viewtopic.php?t=322979

[fduncanh]

https://www.jeffgeerling.com/blog/2024/can-raspberry-pi-5-handle-4k

[fduncanh]

https://raspberrypi.stackexchange.com/questions/99925/can-raspberry-pi-4-hardware-decode-4k-60-hz-h-265-10-bit-video

[connorh315]

Hmm, perhaps it's worth investing some time into getting 4:4:4 implemented then? I'll need to have a look some time this week at the flat that I referenced earlier (receiverSupports444) as I can't remember which plist it was in and where. Even then I'd assume it's more of an advisory and the client doesn't have to adhere to it (also considering the fact it's not compressed I am slightly concerned as to whether it will be an ethernet/awdl exclusive?)

[fduncanh]

I posted a question on GStreamer forum to see if anyone gives advice

https://discourse.gstreamer.org/t/v4l2-stateless-api-support/3339

[connorh315]

I posted a question on GStreamer forum to see if anyone gives advice

https://discourse.gstreamer.org/t/v4l2-stateless-api-support/3339

Perfect, should anyone respond just double check what yuv standards are supported since it seems like only a select few (not the one that's used by airplay) are supported

[fduncanh]

This is the code in question. (ffmpeg patch for rpi HEVC)

See the issues for comments by its author ("jc-kynesim/rpi-ffmpeg#80
#80 opened Feb 14, 2024 by Abdull

https://github.com/jc-kynesim/rpi-ffmpeg

EDIT: its a whole version of ffmpeg that is patched. hopefully the history can help spot where the rpi work is found

[fduncanh]

some info from the author of the ffmpeg patch:
jc-kynesim/rpi-ffmpeg#94 (comment)

[fduncanh]

@connorh315
https://discourse.gstreamer.org/t/v4l2-stateless-api-support/3339

got a (terse) response:

The v4l2 elements already support that. No need to go through ffmpeg

• if the ffmpeg code by jc-kinesim shows how the R Pi h265 decoder is interfaced, it might be possible to give gstreamer a handle to it....

[connorh315]

I've been doing quite a lot of research into alternatives for this. Since all official Raspberry Pis run off the Broadcom chip there is no chance since interfacing with the VPU is kept proprietary from my understanding. So instead I looked into alternative SBCs. I went ahead and bought an Orange Pi Zero 2W about 2 weeks ago that arrived yesterday. The OPi runs with an AllWinner SOC (rather than a Broadcom)... I flashed an image and installed UxPlay and h.264 was alright (except for a certain resolution where eventually I started to get severe artifacting) and then h.265 is non-existent too as the driver only supports h.264. Supposedly Android TV can decode h.265 but I can't even get an image bootable to be able to test that fact. However I found a wiki online that quite clearly outlines that the VPU is not supported in mainline linux for the AllWinner SOCs (and also countless hours of research backs this) so that rules out the Allwinner chips. This then leaves one final SOC that would offer H.265 which is the Rockchip processors. There's about 6/7 SOCs that have been used in the last 5 years that all supposedly support h.265 decoding. From research though, the VPU driver (rkvdec) only ever supported h.264 decoding and h.265 was planned but has never made it to fruition. The only chip that has any support is the RK3399 but that's because of a patch that was rolled in. The rest of them have zero support for h.265. Even the newer processors only seem to have had their VPU driver rolled out back in June (rkvdec2) and that again only supports h.264. It's a bit of a minefield atm with these boards. I don't really understand how manufacturers can justify advertising SBCs as having hardware capabilities but they don't actually provide any way of accessing it - That's off topic though. I'm thinking that I need to look some more into the RPi's h265 decoder that seems to have some partial support? Last time I checked though I couldn't actually get it to accurately decode the uxplay stream from a file as it was an unsupported compression format??

Kind of hoping someone else has made more progress as unfortunately I'm all out of options now and I really still want to make a small plug-n-play uxplay receiver that can run off a USB-C connection.

[fduncanh]

T here is another response on the gstreamer discourse:

slomo
To add to add: as part of the v4l2codecs plugin from gst-plugins-bad, not the v4l2 plugin from gst-plugins-good.

If the ffmpeg work by js-kinesim shows how to interface to broadcom, the stateless gstreamer v4l2 interface (which we now know is in gstreamer1.0-plugins-bad, can perhaps be got to work with it.

active development is going on:

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-bad/sys/v4l2codecs?ref_type=heads

[connorh315]

Good to see that something is going on but I have a feeling it'll be another year until we get h265 working. On another note though, I was surprised to find that the raspberry pi 4 can decode the h265 sent from apple devices correctly using hello_drmprime. I'll have a go at building rpi-ffmpeg (which I believe should do the same) later on today but at the very least the hardware is there to support it. All that's needed now is for the necessary changes to be ported over to gstreamer.

EDIT: rpi-ffmpeg can also decode the airplay h265 streams. I'm going to have a go at hijacking ffmpeg in place of gstreamer in UxPlay and output the decoded result directly to the framebuffer over the weekend.

[fduncanh]

some more info here:
jc-kynesim/rpi-ffmpeg#94 (comment)

[fduncanh]

because the jc-kinesim github repository is difficult to view directly (because of a github 1000 file per directory viewing limit) I created a clone with all but the v4l2 files removed from the libavcodec directory of branch "test/5.1.6/main.
(that way their history of changes can be examined)

https://github.com/fduncanh/rpi-ffmpeg/tree/test/5.1.6/main/libavcodec

[connorh315]

@fduncanh you said you've done something similar to this before right? How difficult of a process is this?

[fduncanh]

I just did a few fixes in the gstreamer 44l2 code that accesses the STATEFUL h264 decoded on rpi 4b.

The stateless decoder seems to be simpler, leaving userspace to do all the difficult organising needed. The process is only viable if that's already there in the gstreamer code (this might be the case).

There is an OUTPUT queue into which the encoded frames are put. The R Pi driver decodes them and puts the decoded frame in the CAPTURE queue for gstreamer to use them. I am not really sure how the connection to the driver (proprietary) gets made, maybe see this in the jc-kinesim code or ask again.

see

https://www.linuxtv.org/downloads/v4l-dvb-apis/userspace-api/index.html - Part I Section 4.5.3 is probably the most help.

[fduncanh]

https://www.linuxtv.org/downloads/v4l-dvb-apis/userspace-api/v4l/dev-mem2mem.html

• OUTPUT

• The source buffer queue; for decoders, the queue of buffers containing an encoded bytestream; for encoders, the queue of buffers containing raw frames; V4L2_BUF_TYPE_VIDEO_OUTPUT or V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; the hardware is fed with data from OUTPUT buffers.

*CAPTURE

• the destination buffer queue; for decoders, the queue of buffers containing decoded frames; for encoders, the queue of buffers containing an encoded bytestream; V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE; data is captured from the hardware into CAPTURE buffers.

https://www.linuxtv.org/downloads/v4l-dvb-apis/userspace-api/v4l/dev-stateless-decoder.html

[fduncanh]

@connorh315 can we move further discussion of this to the new issue #358 I opened:

https://github.com/FDH2/UxPlay/issues/358

• LibreELEC seems to have interesting info on R Pi HEVC.

[fduncanh]

this issue moved to #358
closing