ffmpeg 2.0 support

[totaam]

Issue migrated from trac ticket # 415

component: client | priority: critical | resolution: fixed

2013-08-10 06:25:32: totaam created the issue

---

Split from #398

There are API changes in ffmpeg which cause compatibility problems with how we use the buffer API. (see #398 for details)

See also the problems with libav in #421

[totaam]

2013-10-17 08:38:22: totaam changed status from new to assigned

[totaam]

2013-10-17 08:38:22: totaam changed owner from ahuillet to totaam

[totaam]

2013-10-17 08:38:22: totaam edited the issue description

[totaam]

2013-11-06 11:09:18: totaam uploaded file ffmpeg2.patch (5.0 KiB)

untested patch

[totaam]

2013-11-12 08:37:08: totaam commented

---

The solution is quite simple it seems: Re: get_buffer vs get_buffer2, how do I manage/release buffers?

We can just drop all the context lookup code for v2!

[totaam]

2013-11-14 13:58:07: totaam uploaded file ffmpeg2-v3.patch (23.3 KiB)

patch using refcounted_frames - not working, crashes in av_frame_unref dereferencing a NULL pointer for side_data...

[totaam]

2013-11-15 14:00:20: totaam changed priority from major to critical

[totaam]

2013-11-15 14:00:20: totaam commented

---

Fedora 20 is going to be out soon, this needs to be dealt with.

[totaam]

2013-11-30 10:10:08: totaam uploaded file ffmpeg2-v4.patch (12.0 KiB)

working ffmpeg v2 patch to apply on top of r4816

[totaam]

2013-11-30 10:45:38: totaam changed status from assigned to new

[totaam]

2013-11-30 10:45:38: totaam changed owner from totaam to afarr

[totaam]

2013-11-30 10:45:38: totaam commented

---

Patch above merged as a new codec (too many changes to manage with IFDEFs alone) in r4821 + r4835.
The RPM spec file adds this for Fedora 20 and 21:

--without-dec_avcodec --with-dec_avcodec2

Notes:

• this will need to be backported to v0.10.x
• we need to know if the cost of doing client-side buffer copying is too high (in which case we need to use the other approach and duplicate the code that allocates a frame, and hope we don't make mistakes or that the code doesn't change from underneath us again with version upgrades..)
• need to check Debian sid - probably uses a newer version of libav/ffmpeg, which will need the new flag

afarr: can you please compare clients ffmpeg v1 builds (like Fedora 19) and ffmpeg v2 builds (like Fedora 20) and see if there are any regressions or any major differences in client CPU usage when under load (high FPS with big windows) - if so, we'll need to implement the complicated code before we can move win32 and OSX to ffmpeg v2.

[totaam]

2013-12-01 12:49:47: totaam commented

---

Backported to v0.10.x in 4827 + 4829 + 4837 + 4838 + r4844

[totaam]

2014-01-04 06:28:23: totaam commented

---

Works fine on win32, as of r5111 simply:

• build ffmpeg as per [build minimal x264 / libav on win32 rather than fat binaries #270#comment:13]
• flick the avcodec2 switch in the BAT file

Notes:

• this may help with client memory leak #457, which seems to be avcodec related, in which case we must switch to ffmpeg2 asap. If not, as per comment:4 above, it would be very useful to know the CPU/memory cost of avcodec2 vs old avcodec code.
• on Fedora 19 with ffmpeg-1.2.1, I had to add:

--disable-symver

to avoid this error: xpra/codecs/dec_avcodec2/decoder.so: version node not found for symbol av_fast_malloc@LIBAVCODEC_55

[totaam]

2014-01-10 04:06:56: totaam commented

---

Some clarifications for testing:

• we want to know if it works well (no crashes, etc), so resizing windows, showing high FPS video, etc..
• we want to know if the CPU/memory cost has gone up. For this one, the only way to make sure that what we are measuring is the ffmpeg library change is to use the same distribution, same version, same state, etc.. Ideally just the same system with a different build on top. This can be done on *nix, but it is probably easier to test on win32 since the installer is fully self-contained.

The application used for generating video frames should produce exactly the same video stream each time. I usually use glxgears, but something like mplayer playing the same local video will also do. A browser is not a good test since advertising may vary, network speed may vary, etc..

As usual, it is important to bear in mind that different runs may give different results in terms of number of frames, quality and speed settings, etc. To truly compare the two libraries, let them run freely and record the settings they settle on (xpra info), and also do test runs with fixed settings. Bonus points for using pretty wiki tables to present the results.

[totaam]

2014-01-10 21:37:08: maxmylyn commented

---

I tested on version 0.11.0 r5158 with and without ffmpeg2. I sat and watched the same twitch.tv stream for a good half hour. I had to make a second attempt for the version without ffmpeg2 because my network connection fussed up and the connection dropped. Anyways, I logged my CPU and memory usage watching the stream every 5 minutes and here they are:

without ffmpeg2:

|| Time || Memory (in Megabytes) || CPU % usage ||
|| Start || 107.9mb || 6-8% ||
|| 5 min. || 112.3mb || same ||
|| 10 min. || 113.8mb || same ||
|| 15 min. || 116.1mb || same ||
|| 20 min. || 113.1mb || higher 9-10% ||
|| 25 min. || 116.1mb || slightly higher 9-11% ||
|| 30 min. || 117.0mb || down to 8-10% ||

with ffmpeg2:

|| Time || Memory (in Megabytes) || CPU % usage ||
|| Start || 120.0mb || 6-8% ||
|| 5 min. || 121.0mb || same ||
|| 10 min. || 124.1mb || same ||
|| 15 min. || 125.7mb || same ||
|| 20 min. || 126.5mb || same ||
|| 25 min. || 125.9mb || same ||
|| 30 min. || 126.1mb || same ||

After 30 minutes on both versions, I closed the stream and tried out a couple of websites, like reddit, gmail, along with youtube. The performance was about the same, if not better on ffmpeg2. I'm running Win7 64-bit here on my laptop, and it was running pretty stable the entire time. From my experience the performance and CPU cost is the same with and without ffmpeg2. All in all, it only broke 10% a couple of times while watching the stream, and when not doing anything graphics intensive it never went above 2% usage. With the exception of the network, it ran stable the whole time.

Laptop specs for the curious:

• Dell Studio 16 something
• Win7 64-bit
• 4gb DDR3 Dual-Channel @ 532mhz
• Intel Core i5 M 430 @ 2.27GHz
• Radeon HD 4670m

[totaam]

2014-01-11 00:30:27: totaam changed owner from afarr to maxmylyn

[totaam]

2014-01-11 00:30:27: totaam commented

---

Please see comment:7 where I specifically said that a browser was not a good way of testing. Also, since the number of frames is not reported, it is impossible to know if the decoding effort was equivalent (one cannot visually discern changes in dropped frames at ~25fps)

Both result tables are named without ffmpeg2 - I assume this is a mistake? (it would be easier to read if each version had a distinct column(s) in the same table)

[totaam]

2014-01-11 00:59:17: maxmylyn commented

---

Oops that was a typo. I updated it. Also, I must have missed your comment when I started testing.

[totaam]

2014-01-14 00:29:27: maxmylyn commented

---

I re-ran the test using a local video file in mplayer. I used clive http://clive.sourceforge.net/ to download the file from Youtube. The video I used was:

http://www.youtube.com/watch?v=KaOC9danxNo

It's 5:30 long and has some good high frame rate footage. I watched it and took down the CPU and memory usage every 2 minutes, with the last part taken down at ~5 minutes into the video for both tables.

Here is the relevant data:

|| With ffmpeg1 ||
|| Time || Memory (in Megabytes) || CPU % usage ||
|| Start || 113.4mb || 9-15% ||
|| 2 min. || 118.4mb || same ||
|| 4 min. || 119.4mb || same ||
|| 5 min. || 119.58mb || same ||

|| With ffmpeg2 ||
|| Time || Memory (in Megabytes) || CPU % usage ||
|| Start || 109.9mb || 9-15% ||
|| 2 min. || 112.6mb || same ||
|| 4 min. || 114.8mb || same ||
|| 5 min. || 115.7mb || same ||

[totaam]

2014-01-14 14:22:22: totaam changed status from new to assigned

[totaam]

2014-01-14 14:22:22: totaam changed owner from maxmylyn to totaam

[totaam]

2014-01-14 14:22:22: totaam commented

---

Here's how I tested:

• I run a 1080p video in a loop with:

while true; do
  mplayer Videos/somevideo.mkv
  sleep 1
done

Started just after connecting with --no-speaker (to prevent the sound from interfering).

• I also start this script on the server to keep an eye on some of the data which we may be interested in, found through trial and error (or one could also capture the output of xpra info and parse it afterwards):

STARTTIME=$(date +%s)
while true; do
  ENDTIME=$(date +%s)
  xpra info |& grep -v pygobject | egrep \
    "total_frames|speed.avg=|quality.avg=|batch.actual_delay.avg=|encoder.preset=|client.latency.max=|client.latency.avg=|client-decode-speed.avg=|.csc.dst_format="
  echo "$((($ENDTIME - $STARTTIME)/60)) minutes $((($ENDTIME - $STARTTIME)%60)) seconds"
  sleep 10
done

(the window we are interested in is the mplayer window - the "second" one)

• The memory usage is manually inspected using the win32 task manager (yuk).
• The host this is running on is kept idle as much as possible to prevent other processes from interfering

---

• ffmpeg1 Encoding_Info.exe:

avcodec           : (54, 92, 100)

• ffmpeg2 Encoding_Info.exe:

avcodec           : (54, 39, 101)

||=elapsed time=||||# memory usage (MB)||||# client latency avg/max||||# actual batch delay||||=client decode speed=||||=quality=||||=speed=||=total frames=||
|| ||ffmpeg1||ffmpeg2||ffmpeg1||ffmpeg2||ffmpeg1||ffmpeg2||ffmpeg1||ffmpeg2||ffmpeg1||ffmpeg2||ffmpeg1||ffmpeg2||ffmpeg1||ffmpeg2||
||0 ()||55||55||41 / 276||27 / 199||45||47||50||67||72||62||77||59||n/a||n/a||
||10m||74||71||24 / 95||24 / 92||51||49||70||45||60||61||61||61||8215 h264 + 8214 rgb24||8133 h264 + 8132 rgb24||
||20m||88||88||25 / 104||22 / 186||51||58||65||54||61||61||60||60||16389 h264 + 16384 rgb24||16504 h264 + 16503 rgb24||
() the first line of data is not particularly relevant for most values as things haven't had time to settle down

---

This raises more questions than it answers:

• why do we get as many rgb24 updates as h264? (auto-refresh bug?)
• the values fluctuate too much between each capture, we need an average of the many captures instead
• the speed is far too low, this is on a VLAN, we should push a lot more frames
• is there still a memory leak? (client memory leak #457)
• it seems that ffmpeg2 decodes faster, but this causes the x264 encoder to switch to fast instead of faster preset, which in turn leads to fewer frames..
• why do we converge on speed=quality=60?

[totaam]

2014-01-16 11:13:02: totaam commented

---

The rgb24 frames are due to the xterm which has a size of:

window[1].size=(499, 316)

And it is encoded with x264, which does not support odd dimensions, so we send the 1 pixel edge as rgb24. (not a bug)

There was a small bug in the rgb processing though, fixed in r5204, which prevented proper accounting of rgb frame decoding (as they were acked as failed).

[totaam]

2014-01-17 09:42:57: totaam changed status from assigned to closed

[totaam]

2014-01-17 09:42:57: totaam changed resolution from ** to fixed

[totaam]

2014-01-17 09:42:57: totaam commented

---

Works well enough, now used with Fedora 20+ builds, static builds (r5194), win32 (r5188) and osx (r5193)

Memleak will be tracked in #457