How to enable i.MX vpuenc_h264 ?

[l4es]

Dear All,

I'm integrating gstreamer-imx for the iMX8M Plus in Buildroot borrowing package implemented here : https://github.com/boundarydevices/buildroot-external-boundary/tree/master/package/gst1-imx-v2

When verifying the available hardware encoder/decoder plugins, I cannot find vpuenc_h264 :

root@imx-camera:~# dmesg | grep hantro
[ 2.063741] hantrodec 0 : module inserted. Major = 236
[ 2.064220] hantrodec 1 : module inserted. Major = 236
[ 2.065434] hantroenc: HW at base <0000000038320000> with ID <0x80006200>

root@imx-camera:~# gst-inspect-1.0 | grep imx
libav: avdec_simbiosis_imx: libav Simbiosis Interactive IMX Video decoder
imxvpu: imxvpudec_sspark: i.MX VPU Sorenson Spark video decoder
imxvpu: imxvpudec_divx5: i.MX VPU DivX 5 & 6 video decoder
imxvpu: imxvpudec_divx4: i.MX VPU DivX 4 video decoder
imxvpu: imxvpudec_divx3: i.MX VPU DivX 3 video decoder
imxvpu: imxvpudec_cavs: i.MX VPU AVS (Audio and Video Coding Standard) video decoder
imxvpu: imxvpudec_vp6: i.MX VPU VP6 video decoder
imxvpu: imxvpudec_vc1: i.MX VPU VC-1 advanced profile video decoder
imxvpu: imxvpudec_wmv3: i.MX VPU WMV3 / Window Media Video 9 / VC-1 simple profile video decoder
imxvpu: imxvpudec_h263: i.MX VPU h.263 video decoder
imxvpu: imxvpudec_mpeg4: i.MX VPU MPEG-4 video decoder
imxvpu: imxvpudec_mpeg2: i.MX VPU MPEG-1 & 2 video decoder
imxvpu: imxvpudec_webp: i.MX VPU WebP video decoder
imxvpu: imxvpudec_jpeg: i.MX VPU JPEG video decoder
imxvpu: imxvpudec_vp9: i.MX VPU VP9 video decoder
imxvpu: imxvpudec_vp8: i.MX VPU VP8 video decoder
imxvpu: imxvpudec_h265: i.MX VPU h.265 / HEVC video decoder
imxvpu: imxvpudec_h264: i.MX VPU h.264 / AVC video decoder
imxv4l2video: imxv4l2videosink: NXP i.MX V4L2 video sink
imxv4l2video: imxv4l2videosrc: NXP i.MX V4L2 video source
imxaudio: imxuniaudiodec: NXP i.MX uniaudio decoder
imx2d: imxpxpvideotransform: i.MX PxP video transform
imx2d: imxpxpvideosink: i.MX PxP video sink
imx2d: imxipuvideotransform: i.MX IPU video transform
imx2d: imxg2dvideotransform: i.MX G2D video transform
imx2d: imxg2dvideosink: i.MX G2D video sink
imx2d: imxg2dcompositor: i.MX G2D video transform

Could someone help to point out what possible missing dependency or configuration that prevents the vpuenc_h264, please ?

Thank very much in advance and best regards,
Khang

[dv1]

This currently is not possible, since the i.MX8m plus has a different encoder (VC8000E) that is not supported by libimxvpuapi. I'm working on this.

[l4es]

Dear @dv1,

This currently is not possible, since the i.MX8m plus has a different encoder (VC8000E) that is not supported by libimxvpuapi. I'm working on this.

Could you share any progress on this, please ?
Thanks and Best Regards,
Pascal

[dv1]

Progress is slow, because the VC8000E encoder is not trivial to use, and because many other tasks keep me busy. I am trying to get it in for Yocto Kirkstone, but I cannot guarantee it.

[d-lareg]

Hi @dv1,

since I am in the same situation as @l4es (developing with buildroot), I would be happy to get an update. Is the encoding already implemented for GStreamer in Yocto. I am using the linux kernel 5.10.9, do you see any problems if I integrate the Yocto package in my Buildroot pipeline, or is the kernel to outdated?

[l4es]

Hi @d-lareg,

Clarification of VPU plugins :
Prior BSP 5.10.35 : vpuenc_h264 & vpudec_h264
Since BSP 5.10.35 : v4l2h264enc & v4l2h264dec

More detail here : https://community.nxp.com/t5/i-MX-Processors/iMX8MPlus-How-to-mix-two-video-sources-into-one-with-Gstreamer/m-p/1532117

Personally, kernel 5.10.9 seems a little bit outdated even-though it has been stayed quite long time with Buildroot.

[dv1]

@d-lareg @l4es I am currently facing the problem that parts of the encoder actually collide with libimvxpuapi's current design. I am trying to implement basic encoding support, but it would only expose a subset of the VC8000E's full functionality. I estimate this will take me 1-2 months to finish at this point.

[dv1]

@d-lareg @l4es I finally have a working version in git master. It turned out that a combination of hardware problems and driver bugs were causing a ton of confusion, but I can now encode with the VC8000E. h.264 only for now, but everything is prepared for h.265 encoding.

[l4es]

Hi @dv1,

Great news! i am eager to test it, if possible! Could you advice which commit that I could use for the H.264 encoding, please? Or should I wait for some time for having H265 encoding available as well ?
Many thanks.

[l4es]

Hi @dv1,
I found that the implementation was done in https://github.com/Freescale/libimxvpuapi.

Thank you

[l4es]

Hi @dv1,

Just FYI from Gary Bisson of Boundary Devices :

Actually this isn't needed any longer as NXP now provides a userspace daemon that makes the VPU appear as a standard V4L2 device:
https://github.com/buildroot/buildroot/tree/master/package/freescale-imx/imx-vpu-hantro-daemon
With this, you can use the standard v4l2enc plugin of GStreamer without the need for any additional plugin.

But I think we still need your implementation in older BSPs.

[dv1]

The last time I tried the daemon, it was prone to multiple crashes, and did need the NXP fork of GStreamer, because some patches on top of v4l2enc were needed. Perhaps this improved, I'll check, but yeah, this works with older BSPs, and also works with upstream GStreamer.

[l4es]

Dear @dv1,
For using the H264 encoding with the VC8000E of iMX8M Plus, could you advice the commits of gstreamer-imx and libimxdmabuffer that are associated with Freescale/libimxvpuapi@b79a50a, please ?

[dv1]

@l4es I recommend using b69bef1ac976d71d8e9f527d61446d2a7e8c77eb for libimxdmabuffer and a2c0128 for gstreamer-imx .

[l4es]

Hi @dv1,
Thanks very much. Sorry to bother you again, could you share some quick gstreamer command for testing the encoder on iMX8M Plus please?

[dv1]

@l4es Sorry for the late reply. A simple example would be:

gst-launch-1.0 videotestsrc num-buffers=300 ! imxvpuenc_h264 ! h264parse ! matroskamux ! filesink location=test.mkv

Another one would be:

gst-launch-1.0 v4l2src ! queue ! imxvpuenc_h264 ! h264parse ! matroskamux ! filesink location=test.mkv

[d-lareg]

Hi @dv1

is OpenMax required to use the encoder? I tested the following pipe....
gst-launch-1.0 videotestsrc num-buffers=300 ! imxvpuenc_h264 ! fakesink
and get the error...
OMX H264 ! HantroHwEncOmx_encoder_create_h264 H264EncInit failed! (-1) openmax_il/source/encoder/encoder_h264.c:773

My buildroot configuration is:
imx-codec = 4.3.5
imx-vpuwrap = 4.3.5
imxdmabuffer = 1.1.3
imx-vpu-hantro = 1.27.0
imx-vpu-hantro-vc = 1.9.0
imxvpuapi = b79a50a (newest)
gstreamer-imx = 4368bd0 (2 weeks old)

The log for GST_DEBUG=4 is attached.

gst_h264.log

[dv1]

@d-lareg This is an imx8m mini? Can you re-run this with GST_DEBUG set to 2,*imx*:9 ?

[d-lareg]

@dv1 no its a self developed board equipped with an imx8m+ soc.
gst_h264_detailed.log

[dv1]

@d-lareg There are three variants of imx8m: imx8m quad, imx8m mini, imx8m plus. Which one is yours?

[d-lareg]

imx8m+ = imx8m plus

[dv1]

Oh, right, I missed the "+" :) The logs indicate a misconfigured libimxvpuapi then:

imxvpuapi2_imx8m_hantro_h1_encoder.c:812:imx_vpu_api_enc_open:�[00m could not create h.264 encoder

The imx8m plus does not have an H1 encoder, it has the VC8000E encoder. Use the latest git master version of libimxvpuapi, and when using its --imx-platform switch, be sure to pass imx8mp to it, not imx8mm. If you are using Yocto, just modify libimxvpuapi's recipe accordingly. I will update the recipes soon when I make new releases, but until then, you have to do that.

[d-lareg]

Argh, my bad! You are right! But now I am facing a build error when building the image with buildroot.

In the mk file of imx-vpu-hantro-vc the header will copied to the main include path:

define IMX_VPU_HANTRO_VC_INSTALL_STAGING_CMDS
	mkdir -p $(STAGING_DIR)/usr/include/
	cp -dpfr $(@D)/usr/include/hantro_VC8000E_enc/*.h $(STAGING_DIR)/usr/include/
	$(INSTALL) -D -m 0755 $(@D)/usr/lib/libhantro_vc8000e.so $(STAGING_DIR)/usr/lib/libhantro_vc8000e.so
	$(INSTALL) -D -m 0755 $(@D)/usr/lib/libhantro_vc8000e.so.1 $(STAGING_DIR)/usr/lib/libhantro_vc8000e.so.1
endef

But libimxvpuapi is searching for a subfolder called hantro_VC8000E_enc that does not exists.

elif self.soc_type == 'MX8MP':
	# i.MX8m plus has the Hantro VC8000E encoder
	conf.define('IMXVPUAPI2_VPU_HAS_VC8000E_ENCODER', 1)
	conf.check_cc(uselib_store = 'HANTRO', uselib = 'HANTRO', define_name = '', mandatory = 1, lib = ['hantro_vc8000e', 'm'])
	conf.check_cc(uselib_store = 'HANTRO_ENC', uselib = 'HANTRO', define_name = '', mandatory = 1, includes = [os.path.join(sysroot_path, 'usr/include')], header_name = 'hantro_VC8000E_enc/hevcencapi.h')

[dv1]

@d-lareg Please issue a ticket for fixing that mk file. Copying the headers into /usr/include/ like that is wrong. The hantro_VC8000E_enc subdirectory exists for a reason. Some of the headers have very generic names like base_type.h.

[dv1]

@d-lareg Also, can you try to patch that mk file yourself? Just change:

cp -dpfr $(@D)/usr/include/hantro_VC8000E_enc/*.h $(STAGING_DIR)/usr/include/

to:

cp -dpfr $(@D)/usr/include/hantro_VC8000E_enc/ $(STAGING_DIR)/usr/include/

If everything works with this patch, then I can make a 2.3.0 release, and buildroot can be notified about this problem in that mk file.

[d-lareg]

@dv1 I fixed the copy issue but still having problems with the encoder. I am now getting the following error:

imxvpuapi imxvpuapi2_imx8m_hantro_vc8000_encoder.c:860:imx_vpu_api_enc_open: could not initialize encoder: EWL error (-5)

What does EWL mean?

gst.log

[dv1]

@d-lareg EWL errors can happen when certain device nodes are not available.

Check for the presence of /dev/dma_heap/linux,cma-uncached (or /dev/dma_heap/linux,cma) and /dev/mxc_hantro_vc8000e on your machine.

If one of these are missing, then this might be why you get that error.

Another idea is to re-run your program with strace, like: strace -f -e trace=openat ./my-program You might have to install strace on your machine.

[dv1]

@d-lareg Did you have any success in fixing this? Was it a problem with the kernel config or the devicetree?

[Wooker]

Hello! I'm also struggling with this issue while building gstreamer-imx and its dependencies with buildroot for imx8mp-evk.

The problem arises due to the use of kernel version 5.4.70, where /dev/dma_heap/ is not available.

According to the libimxvpuapi documentation, ION or dma heap CMA should be enabled in the kernel for proper functionality of vpu elements. Despite configuring libimxdmabuffer to use only ION and enabling ION CMA in the kernel, imxvpuenc_h264 still attempts to access /dev/dma_heap/linux,cma-uncached. This behavior was observed through strace, as you suggested previously:

[pid   288] openat(AT_FDCWD, "/dev/mxc_hantro_vc8000e", O_RDWR) = 20
[pid   288] openat(AT_FDCWD, "/dev/dma_heap/linux,cma-uncached", O_RDWR) = -1 ENOENT (No such file or directory)
VCEncInit: EWL Initialization failed

Versions of libraries:

• libimxdmabuffer 1.1.3
• libimxvpuapi 2.3.0
• gstreamer-imx 2.2.0

Any insights or suggestions on resolving this issue would be greatly appreciated!

Please also see the full strace log: trace.log

[Wooker]

Hello again! As it turned out, the issue described above appeared due to inconsistent builds of gstreamer and some of its dependencies. While libimxvpuapi was rebuilt, libraries that depend on it were not. Hence, usage of dma_heap was not replaced by usage of ion.

So, simple buildroot rebuild solved the issue.

[ebanez]

Hi @dv1!

I'm having a very similar issue with the imx8mm. Using Yocto Kirkstone 5.15.60.

My message is slightly different from the original post:

OMX H264 ! HantroHwEncOmx_encoder_create_h264 H264EncInit failed! (-5) openmax_il/source/encoder/encoder_h264.c:773

where the error code is -5 instead of -3.

The device mxc_hantro_h1 and mxc_hantro is present and cma is enabled

Installed dependency versions:

• libimxdmabuffer 1.1.2
• gstreamer-imx 2.1.0
• libimxvpuapi 2.2.1

Does anybody know what the difference in error code could mean and any help in general? The sample gst test won't even work.