Deadlock in MpdBuilder causing packetization to stop after several hours #45
Assignees
Labels
status: archived
Archived and locked; will not be updated
type: bug
Something isn't working correctly
Milestone
Comments
|
I suspect that somehow packager program was no longer able to receive from udp. I created a debug code at https://github.com/kqyang/edash-packager, which includes two changes:
You can try with that code and see what happens. You could increase verbosity level with flag '--vmodule', e.g. --vmodule='*=5', the larger the number, the more loggings. You could also use vlc to confirm whether the stream is still good: vlc udp://@239.101.2.1:1234 (remember to close packager program first). |
|
Hello, So, With vmodule=5, the last lines before "hang" are: Please also note, that I'm able to see the incoming traffic with tcpdump on the interface and on parallel computer edash packager release 1.1 is still working. BRS/ |
|
Em... It is good to know that it works with v1.1. I don't see anything interesting from the log. What happened after "the last debug lines"? Did packager program quit or still running but without any more logging? Notice that the time difference between the two logs is only 3 minutes. So, you can reproduce the problem in just minutes? You mentioned 'ffmpeg'. Are you using 'ffmpeg' to reproduce it? Can you explain your duplication steps in more detail? Thanks. |
|
Hi, FFmpeg config is quite simple. That's the setup. BRS/ |
|
I tried to duplicate the failure, but still not able to after running for days. Can you use gdb to attach to packager process to find out what it is doing, e.g. extract stack traces? |
|
Hi, I'm seeing this on my setup with a very recent build using docker. In my debugging efforts (and before finding this report) I've managed to generate a full strace (-f) of the packager while failing like this, the whole file is huge (~500M) but here I attach the last lines including the last open()+write()+close() in hope to help pin this down. packager_strace_after_last_seg.txt (recvfrom data was edited out for size-sake) The execve was: Let me know if I can help any further. |
|
Thanks for the report. Can you try adding a new option "--io_cache_size=0" to see if it resolves the problem? Also, use debug version which may reveal more information for debugging. |
|
Ok, so I've tried your fork, with and without "--io_cache_size=0", both with "--vmodule='*=5'", both Release version. Both hanged after a while with the following log: I'll try the debug version of your fork now, and maybe attach a gdb and try to get a trace. Let me know if I can help any further. |
|
Hi again, I've tried the debug version, and it fails a lot quickier, but looks like the same error as it just hangs there, attached a gdb while it was hanging there: also bt of a RemuxJob: and bt of the KeyProductionThread: Again, let me know if there's anything else I can do to help nail this one. |
|
Thanks for the traces. Can you get the back trace for all threads? |
|
Sure (kept it running just in case): |
|
Cool, thanks for the detailed traces. The traces show that there is a dead lock in mpd generation. Thread 2: waiting for SimpleMpdNotifier::Lock in NotifySampleDuration @rkuroiwa , can you take a look? Thanks. |
|
Hello, |
|
It's been running fine for 3 hours now (longer than the average before), I'll leave it running for 24hs and confirm again tomorrow. |
|
I can confirm now, that after 24hs it keeps running ok. |
kqyang
changed the title
|
Cool. Thanks! |
|
Hi, Thank you. |
|
Cool. Thanks for the update. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Hello,
I'm using v1.2.0.
The command line is:
packager --udp_interface_address=172.18.0.10 'input=udp://239.101.2.1:1234,stream=audio,init_segment=/storage/web/dash/bnt1/bnt1_edash-audio.m4a,segment_template=/storage/web/dash/bnt1/bnt1_edash-audio-$Number$.m4a' 'input=udp://239.101.2.1:1234,stream=video,init_segment=/storage/web/dash/bnt1/bnt1_edash-video-1080.m4v,segment_template=/storage/web/dash/bnt1/bnt1_edash-video-1080-$Number$.m4v' 'input=udp://239.102.2.1:1234,stream=video,init_segment=/storage/web/dash/bnt1/bnt1_edash-video-720.m4v,segment_template=/storage/web/dash/bnt1/bnt1_edash-video-720-$Number$.m4v' 'input=udp://239.103.2.1:1234,stream=video,init_segment=/storage/web/dash/bnt1/bnt1_edash-video-576.m4v,segment_template=/storage/web/dash/bnt1/bnt1_edash-video-576-$Number$.m4v' --profile live --mpd_output /storage/web/dash/bnt1/bnt1.mpd -time_shift_buffer_depth 1800 -dump_stream_info
After several hours of operation, edash packager stops to fill the segments directory and .mpd file updates.
The three groups are still joined and tcpdump shows correct traffic to the system.
The last console output is:
[1026/105614:INFO:demuxer.cc(135)] Demuxer::Run() on file 'udp://239.101.2.1:1234'.
[1026/105614:INFO:demuxer.cc(135)] Demuxer::Run() on file 'udp://239.102.2.1:1234'.
[1026/105614:INFO:demuxer.cc(135)] Demuxer::Run() on file 'udp://239.103.2.1:1234'.
[1026/105614:WARNING:mpd_builder.cc(211)] Video info does not contain frame duration required for calculating framerate. @framerate is required for DASH IOP.
[1026/105614:WARNING:mpd_builder.cc(211)] Video info does not contain frame duration required for calculating framerate. @framerate is required for DASH IOP.
[1026/105615:WARNING:mpd_builder.cc(211)] Video info does not contain frame duration required for calculating framerate. @framerate is required for DASH IOP.
Any help how I can debug what's wrong? How I can increase the verbosity?
BRS/
Peyo
The text was updated successfully, but these errors were encountered: