Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SRS crash when forward and http hooks #752

Closed
snavruzov opened this issue Jan 18, 2017 · 18 comments
Closed

SRS crash when forward and http hooks #752

snavruzov opened this issue Jan 18, 2017 · 18 comments
Assignees
@winlinvip
Labels
Bug It might be a bug. EnglishNative This issue is conveyed exclusively in English.
Milestone
2.0

Comments

@snavruzov
Copy link

Hi, the SRS is crashed for some reason that I couldn't research further but getting crash log(attached below).

It's happened yesterday, though it was working more than 2 months perfectly.
SRS version:XCORE-SRS/2.0.221(ZhouGuowen)
OS type:Ubuntu 14.04. deployed on AWS EC2 t2.medium instance\

SRS config file:

# main config for srs.
# @see full.conf for detail config.
listen              1935;
max_connections     100;
srs_log_tank        file;
srs_log_file        ./objs/srs.log;
http_api {
    enabled         on;
    listen          1985;
}
http_server {
    enabled         on;
    listen          8080;
    dir             ./objs/nginx/html;
}
stats {
    network         0;
    disk            sda sdb xvda xvdb;
}
vhost __defaultVhost__ {

	transcode liveff {
        enabled     on;
        ffmpeg      ./objs/ffmpeg/bin/ffmpeg;
       engine ff {
            # whether the engine is enabled
            # default: off.
            enabled         on;
            iformat         flv;
            vcodec          libx264;
            vfps            20;
            vthreads        2;
            vprofile        baseline;
            vpreset         ultrafast;
            
	    # other x264 or ffmpeg video params
            vparams {
            #    # ffmpeg options, @see: http://ffmpeg.org/ffmpeg.html
            #    #t               100;
	    	level		30;
                crf		28;
	    #	# 264 params, @see: http://ffmpeg.org/ffmpeg-codecs.html#libx264
                coder           1;
                b_strategy      2;
                bf              0;
                refs            6;
		g		40;
            }
            acodec          copy;
            oformat         flv;
            output          rtmp://127.0.0.1:[port]/livelow/[stream];
        }
    }
    gop_cache       on;
    queue_length    5;
}
@snavruzov
Copy link
Author

opt_srs_trunk_objs_srs.0.crash.txt

@snavruzov
Copy link
Author

Last two connection's log:

[2017-01-17 10:56:46.058][trace][7651][17691] RTMP client ip=x.x.x.x
[2017-01-17 10:56:46.109][trace][7651][17691] complex handshake success
[2017-01-17 10:56:46.153][trace][7651][17691] connect app, tcUrl=rtmp://x.x.x.x:1935/liveff, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=1935, app=liveff, args=null
[2017-01-17 10:56:46.153][trace][7651][17691] out chunk size to 60000
[2017-01-17 10:56:46.358][trace][7651][17691] input chunk size to 60000
[2017-01-17 10:56:46.358][trace][7651][17691] client identified, type=fmle-publish, stream_name=rtmpLibs, duration=-1.00
[2017-01-17 10:56:46.358][trace][7651][17691] source url=/liveff/rtmpLibs, ip=x.x.x.x, cache=1, is_edge=0, source_id=-1[-1]
[2017-01-17 10:56:46.568][trace][7651][17691] encoder thread cid=17692, current_cid=17691
[2017-01-17 10:56:46.568][trace][7651][17692] start ffmpeg, log: ./objs/ffmpeg-encoder-__defaultVhost__-liveff-rtmpLibs.log, params: ./objs/ffmpeg/bin/ffmpeg -f flv -i rtmp://127.0.0.1:1935/liveff?vhost=__defaultVhost__/rtmpLibs -vcodec libx264 -r 20.00 -threads 2 -profile:v baseline -preset ultrafast -level 30 -crf 28 -coder 1 -b_strategy 2 -bf 0 -refs 6 -g 40 -acodec copy -f flv -y rtmp://127.0.0.1:1935/livelow/rtmpLibs
[2017-01-17 10:56:46.569][trace][7651][17692] vfored ffmpeg encoder engine, pid=1488
[2017-01-17 10:56:46.569][trace][7651][17692] -> ENC time=54000, encoders=1, input=__defaultVhost__/liveff/rtmpLibs
[2017-01-17 10:56:46.578][trace][7651][17691] start publish mr=0/350, p1stpt=20000, pnt=20000, tcp_nodelay=0, rtcid=17693
[2017-01-17 10:56:46.596][trace][7651][17695] RTMP client ip=127.0.0.1
[2017-01-17 10:56:46.597][trace][7651][17695] complex handshake success
[2017-01-17 10:56:46.637][trace][7651][17695] connect app, tcUrl=rtmp://127.0.0.1:1935/liveff?vhost=__defaultVhost__, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=1935, app=liveff, args=null
[2017-01-17 10:56:46.637][trace][7651][17695] out chunk size to 60000
[2017-01-17 10:56:46.717][trace][7651][17695] ignore AMF0/AMF3 command message.
[2017-01-17 10:56:46.757][trace][7651][17695] client identified, type=Play, stream_name=rtmpLibs, duration=-1.00
[2017-01-17 10:56:46.757][trace][7651][17695] source url=/liveff/rtmpLibs, ip=127.0.0.1, cache=1, is_edge=0, source_id=17691[17691]
[2017-01-17 10:56:46.757][trace][7651][17695] dispatch cached gop success. count=0, duration=-1
[2017-01-17 10:56:46.757][trace][7651][17695] create consumer, queue_size=5.00, jitter=1
[2017-01-17 10:56:46.768][trace][7651][17695] mw changed sleep 350=>350, max_msgs=128, esbuf=218750, sbuf 2626560=>218750, realtime=0
[2017-01-17 10:56:46.768][trace][7651][17695] start play smi=0.00, mw_sleep=350, mw_enabled=1, realtime=0, tcp_nodelay=0
[2017-01-17 10:56:46.768][trace][7651][17696] ignored. set buffer length to 3000
[2017-01-17 10:56:47.234][trace][7651][17692] send SIGTERM to pid=1488
[2017-01-17 10:56:48.246][trace][7651][17692] SIGKILL stop process pid=1488 ok.
[2017-01-17 10:56:48.246][trace][7651][17691] cleanup when unpublish
[2017-01-17 10:56:48.247][warn][7651][17691][4] client disconnect peer. ret=1004
[2017-01-17 10:56:48.247][warn][7651][17695][104] client disconnect peer. ret=1004
[2017-01-17 10:57:18.480][trace][7651][17698] RTMP client ip=**x.x.x.x**
[2017-01-17 10:57:18.533][trace][7651][17698] complex handshake success
[2017-01-17 10:57:18.580][trace][7651][17698] connect app, tcUrl=rtmp://x.x.x.x:1935/liveff, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=1935, app=liveff, args=null
[2017-01-17 10:57:18.580][trace][7651][17698] out chunk size to 60000
[2017-01-17 10:57:18.782][trace][7651][17698] input chunk size to 60000
[2017-01-17 10:57:18.783][trace][7651][17698] client identified, type=fmle-publish, stream_name=rtmpLibs, duration=-1.00
[2017-01-17 10:57:18.783][trace][7651][17698] source url=/liveff/rtmpLibs, ip=x.x.x.x, cache=1, is_edge=0, source_id=-1[-1]
[2017-01-17 10:57:18.977][trace][7651][100] cleanup die source, total=2

@winlinvip
Copy link
Member

I can't get the coredump and call stack information. Maybe it's caused by source cleanup. So please show me the callstack or upgrade to latest 2.0.

https://github.com/ossrs/srs#v2-changes

@winlinvip winlinvip added this to the srs 2.0 release milestone Jan 19, 2017
@winlinvip winlinvip added the Bug It might be a bug. label Jan 19, 2017
@snavruzov
Copy link
Author

Another crash log passing a couple of hours after upgrading to XCORE-SRS/2.0.231(ZhouGuowen)\

@snavruzov
Copy link
Author

_opt_srs_trunk_objs_srs.0.crash.txt

@snavruzov
Copy link
Author

Last connection logs from srs.log:

[2017-01-19 14:03:31.405][trace][18219][106] RTMP client ip=x.x.x.x
[2017-01-19 14:03:31.405][trace][18219][106] srand initialized the random.
[2017-01-19 14:03:31.704][trace][18219][106] complex handshake success
[2017-01-19 14:03:32.009][trace][18219][106] connect app, tcUrl=rtmp://x.x.x.x:1935/stream, pageUrl=, swfUrl=, schema=rtmp, vhost=x.x.x.x, port=1935, app=stream, args=null
[2017-01-19 14:03:32.828][trace][18219][106] client identified, type=fmle-publish, stream_name=testLibs, duration=-1.00
[2017-01-19 14:03:32.849][trace][18219][106] source url=x.x.x.x/stream/testLibs, ip=x.x.x.x, cache=0, is_edge=0, source_id=-1[-1]
[2017-01-19 14:03:33.760][trace][18219][106] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0, rtcid=109
[2017-01-19 14:03:34.389][trace][18219][106] got metadata, width=1280, height=960, vcodec=7, acodec=10
[2017-01-19 14:03:34.389][trace][18219][106] protocol in.buffer=0, in.ack=0, out.ack=2500000, in.chunk=60000, out.chunk=60000
[2017-01-19 14:03:34.389][trace][18219][106] 33B video sh,  codec(7, profile=Baseline, level=3.2, 1280x960, 0kbps, 0fps, 0s)
[2017-01-19 14:03:34.389][trace][18219][106] 4B audio sh, codec(10, profile=LC, 1channels, 0kbps, 44100HZ), flv(16bits, 2channels, 44100HZ)
[2017-01-19 14:03:38.878][trace][18219][106] cleanup when unpublish
[2017-01-19 14:03:38.878][warn][18219][106][104] client disconnect peer. ret=1004
[2017-01-19 14:03:40.111][trace][18219][111] RTMP client ip=x.x.x.x
[2017-01-19 14:03:40.414][trace][18219][111] complex handshake success
[2017-01-19 14:03:40.717][trace][18219][111] connect app, tcUrl=rtmp://x.x.x.x:1935/dgtz, pageUrl=, swfUrl=, schema=rtmp, vhost=x.x.x.x, port=1935, app=dgtz, args=null
[2017-01-19 14:03:41.532][trace][18219][111] client identified, type=fmle-publish, stream_name=live_id117_55279, duration=-1.00
[2017-01-19 14:03:41.552][trace][18219][111] source url=x.x.x.x/dgtz/live_id117_55279, ip=x.x.x.x, cache=0, is_edge=0, source_id=-1[-1]
[2017-01-19 14:03:42.477][trace][18219][115] api get peer ip success. ip=127.0.0.1
[2017-01-19 14:03:42.477][trace][18219][115] HTTP API GET http://localhost:1985/api/v1/clients/111, content-length=-1, chunked=0/0
[2017-01-19 14:03:42.479][trace][18219][115] HTTP API GET http://localhost:1985/api/v1/streams/18221, content-length=-1, chunked=0/0
[2017-01-19 14:03:42.557][trace][18219][111] ignore private address: x.x.x.x
[2017-01-19 14:03:42.557][warn][18219][111][11] use private address as ip: x.x.x.x
[2017-01-19 14:03:42.557][trace][18219][111] http hook on_publish success. client_id=111, url=http://127.0.0.1:8443/live/start, request={"action":"on_publish","client_id":111,"ip":"x.x.x.x","vhost":"x.x.x.x","app":"dgtz","tcUrl":"rtmp://x.x.x.x:1935/dgtz","stream":"live_id117_55279"}, response=0, ret=0
[2017-01-19 14:03:42.557][trace][18219][111] start forward rtmp://x.x.x.x:1935?vhost=x.x.x.x to rtmp://127.0.0.1:1940?vhost=x.x.x.x, tcUrl=rtmp://x.x.x.x:1940/dgtz, stream=live_id117_55279
[2017-01-19 14:03:42.567][trace][18219][111] forward thread cid=116, current_cid=111
[2017-01-19 14:03:42.568][trace][18219][116] forward connected, stream=live_id117_55279, tcUrl=rtmp://x.x.x.x:1935/dgtz to server=127.0.0.1, port=1940
[2017-01-19 14:03:42.568][trace][18219][116] complex handshake success.
[2017-01-19 14:03:42.577][trace][18219][111] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0, rtcid=117
[2017-01-19 14:03:42.647][trace][18219][116] connected, version=, ip=, pid=0, id=0, dsu=1
[2017-01-19 14:03:43.527][trace][18219][111] got metadata, width=640, height=360, vcodec=7, acodec=10
[2017-01-19 14:03:43.527][trace][18219][111] protocol in.buffer=0, in.ack=0, out.ack=2500000, in.chunk=60000, out.chunk=60000
[2017-01-19 14:03:43.527][trace][18219][111] 34B video sh,  codec(7, profile=Baseline, level=3, 640x368, 0kbps, 0fps, 0s)
[2017-01-19 14:03:43.527][trace][18219][111] 4B audio sh, codec(10, profile=LC, 1channels, 0kbps, 44100HZ), flv(16bits, 2channels, 44100HZ)
[2017-01-19 14:03:43.687][trace][18219][116] protocol in.buffer=0, in.ack=5000000, out.ack=2500000, in.chunk=4096, out.chunk=60000
[2017-01-19 14:03:53.188][trace][18219][116] -> FWR time=10000, msgs=32, okbps=397,0,0, ikbps=2,0,0
[2017-01-19 14:04:03.188][trace][18219][116] -> FWR time=20000, msgs=35, okbps=476,0,0, ikbps=1,0,0
[2017-01-19 14:04:07.577][trace][18219][111] <- CPB time=21000, okbps=1,0,0, ikbps=458,0,0, mr=0/350, p1stpt=20000, pnt=5000
[2017-01-19 14:04:13.179][warn][18219][115][104] client disconnect peer. ret=1004
[2017-01-19 14:04:13.188][trace][18219][116] -> FWR time=30000, msgs=26, okbps=521,521,0, ikbps=0,0,0
[2017-01-19 14:04:17.577][trace][18219][111] <- CPB time=31000, okbps=0,1,0, ikbps=520,455,0, mr=0/350, p1stpt=20000, pnt=5000
[2017-01-19 14:04:23.189][trace][18219][116] -> FWR time=40000, msgs=34, okbps=577,521,0, ikbps=0,0,0
[2017-01-19 14:04:27.578][trace][18219][111] <- CPB time=41000, okbps=0,1,0, ikbps=564,455,0, mr=0/350, p1stpt=20000, pnt=5000
[2017-01-19 14:04:28.445][trace][18219][119] RTMP client ip=x.x.x.x
[2017-01-19 14:04:28.661][trace][18219][119] complex handshake success
[2017-01-19 14:04:28.863][trace][18219][119] connect app, tcUrl=rtmp://x.x.x.x:1935/stream, pageUrl=, swfUrl=, schema=rtmp, vhost=x.x.x.x, port=1935, app=stream, args=null
[2017-01-19 14:04:29.549][trace][18219][119] client identified, type=fmle-publish, stream_name=testLibs, duration=-1.00
[2017-01-19 14:04:29.549][trace][18219][119] source url=x.x.x.x/stream/testLibs, ip=x.x.x.x, cache=0, is_edge=0, source_id=-1[-1]
[2017-01-19 14:04:30.290][trace][18219][119] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0, rtcid=120
[2017-01-19 14:04:30.794][trace][18219][119] got metadata, width=1280, height=960, vcodec=7, acodec=10
[2017-01-19 14:04:30.794][trace][18219][119] protocol in.buffer=0, in.ack=0, out.ack=2500000, in.chunk=60000, out.chunk=60000
[2017-01-19 14:04:30.794][trace][18219][119] 33B video sh,  codec(7, profile=Baseline, level=3.2, 1280x960, 0kbps, 0fps, 0s)
[2017-01-19 14:04:30.794][trace][18219][119] 4B audio sh, codec(10, profile=LC, 1channels, 0kbps, 44100HZ), flv(16bits, 2channels, 44100HZ)
[2017-01-19 14:04:32.937][trace][18219][119] cleanup when unpublish
[2017-01-19 14:04:32.937][warn][18219][119][104] client disconnect peer. ret=1004
[2017-01-19 14:04:33.189][trace][18219][116] -> FWR time=50000, msgs=32, okbps=613,521,0, ikbps=0,0,0
[2017-01-19 14:04:33.625][trace][18219][122] RTMP client ip=x.x.x.x
[2017-01-19 14:04:33.778][trace][18219][124] RTMP client ip=x.x.x.x
[2017-01-19 14:04:33.837][trace][18219][122] complex handshake success
[2017-01-19 14:04:33.983][trace][18219][124] complex handshake success
[2017-01-19 14:04:34.041][trace][18219][122] connect app, tcUrl=rtmp://x.x.x.x:1935/dgtz, pageUrl=, swfUrl=, schema=rtmp, vhost=x.x.x.x, port=1935, app=dgtz, args=null
[2017-01-19 14:04:34.273][trace][18219][124] connect app, tcUrl=rtmp://x.x.x.x:1935/debate, pageUrl=, swfUrl=, schema=rtmp, vhost=x.x.x.x, port=1935, app=debate, args=null
[2017-01-19 14:04:34.782][trace][18219][122] ignore AMF0/AMF3 command message.
[2017-01-19 14:04:35.021][trace][18219][124] client identified, type=fmle-publish, stream_name=live_id153_55280, duration=-1.00
[2017-01-19 14:04:35.041][trace][18219][124] source url=x.x.x.x/debate/live_id153_55280, ip=x.x.x.x, cache=0, is_edge=0, source_id=-1[-1]
[2017-01-19 14:04:35.300][trace][18219][122] ignore AMF0/AMF3 command message.
[2017-01-19 14:04:35.300][trace][18219][122] client identified, type=Play, stream_name=live_id117_55279, duration=-1.00
[2017-01-19 14:04:35.300][trace][18219][122] source url=x.x.x.x/dgtz/live_id117_55279, ip=x.x.x.x, cache=0, is_edge=0, source_id=111[111]
[2017-01-19 14:04:35.300][trace][18219][122] dispatch cached gop success. count=0, duration=0
[2017-01-19 14:04:35.300][trace][18219][122] create consumer, queue_size=30.00, jitter=1
[2017-01-19 14:04:35.310][trace][18219][122] mw changed sleep 350=>350, max_msgs=128, esbuf=218750, sbuf 46080=>218750, realtime=0
[2017-01-19 14:04:35.310][trace][18219][122] start play smi=0.00, mw_sleep=350, mw_enabled=1, realtime=0, tcp_nodelay=0
[2017-01-19 14:04:35.310][trace][18219][122] protocol in.buffer=3000, in.ack=2500000, out.ack=2500000, in.chunk=128, out.chunk=60000
[2017-01-19 14:04:35.877][trace][18219][124] http hook on_publish success. client_id=124, url=http://127.0.0.1:8443/live/debate/start, request={"action":"on_publish","client_id":124,"ip":"x.x.x.x","vhost":"x.x.x.x","app":"debate","tcUrl":"rtmp://x.x.x.x:1935/debate","stream":"live_id153_55280"}, response=0, ret=0
[2017-01-19 14:04:35.877][trace][18219][124] start forward rtmp://x.x.x.x:1935?vhost=x.x.x.x to rtmp://127.0.0.1:1940?vhost=x.x.x.x, tcUrl=rtmp://x.x.x.x:1940/debate, stream=live_id153_55280
[2017-01-19 14:04:35.888][trace][18219][124] forward thread cid=128, current_cid=124
[2017-01-19 14:04:35.888][trace][18219][128] forward connected, stream=live_id153_55280, tcUrl=rtmp://x.x.x.x:1935/debate to server=127.0.0.1, port=1940
[2017-01-19 14:04:35.888][trace][18219][128] complex handshake success.
[2017-01-19 14:04:35.898][trace][18219][124] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0, rtcid=129
[2017-01-19 14:04:35.967][trace][18219][128] connected, version=, ip=, pid=0, id=0, dsu=1
[2017-01-19 14:04:36.708][trace][18219][124] got metadata, width=640, height=360, vcodec=7, acodec=10
[2017-01-19 14:04:36.708][trace][18219][124] protocol in.buffer=0, in.ack=0, out.ack=2500000, in.chunk=60000, out.chunk=60000
[2017-01-19 14:04:36.708][trace][18219][124] 34B video sh,  codec(7, profile=Baseline, level=3, 640x368, 0kbps, 0fps, 0s)
[2017-01-19 14:04:36.708][trace][18219][124] 4B audio sh, codec(10, profile=LC, 1channels, 0kbps, 44100HZ), flv(16bits, 2channels, 44100HZ)
[2017-01-19 14:04:37.007][trace][18219][128] protocol in.buffer=0, in.ack=5000000, out.ack=2500000, in.chunk=4096, out.chunk=60000
[2017-01-19 14:04:37.199][trace][18219][131] RTMP client ip=x.x.x.x
[2017-01-19 14:04:37.439][trace][18219][131] complex handshake success
[2017-01-19 14:04:37.642][trace][18219][131] connect app, tcUrl=rtmp://x.x.x.x:1935/debate, pageUrl=, swfUrl=, schema=rtmp, vhost=x.x.x.x, port=1935, app=debate, args=null
[2017-01-19 14:04:38.324][trace][18219][131] ignore AMF0/AMF3 command message.
[2017-01-19 14:04:38.772][trace][18219][131] ignore AMF0/AMF3 command message.
[2017-01-19 14:04:38.772][trace][18219][131] client identified, type=Play, stream_name=live_id153_55280, duration=-1.00
[2017-01-19 14:04:38.772][trace][18219][131] source url=x.x.x.x/debate/live_id153_55280, ip=x.x.x.x, cache=0, is_edge=0, source_id=124[124]
[2017-01-19 14:04:38.772][trace][18219][131] dispatch cached gop success. count=0, duration=0
[2017-01-19 14:04:38.772][trace][18219][131] create consumer, queue_size=30.00, jitter=1
[2017-01-19 14:04:38.782][trace][18219][131] mw changed sleep 350=>350, max_msgs=128, esbuf=218750, sbuf 46080=>218750, realtime=0
[2017-01-19 14:04:38.782][trace][18219][131] start play smi=0.00, mw_sleep=350, mw_enabled=1, realtime=0, tcp_nodelay=0
[2017-01-19 14:04:38.782][trace][18219][131] protocol in.buffer=3000, in.ack=2500000, out.ack=2500000, in.chunk=128, out.chunk=60000
[2017-01-19 14:04:45.008][trace][18219][128] -> FWR time=8000, msgs=28, okbps=1223,0,0, ikbps=3,0,0
[2017-01-19 14:04:46.994][trace][18219][131] -> PLA time=8000, msgs=22, okbps=1113,0,0, ikbps=2,0,0, mw=350
[2017-01-19 14:04:47.578][trace][18219][111] <- CPB time=61000, okbps=0,0,0, ikbps=646,792,0, mr=0/350, p1stpt=20000, pnt=5000
[2017-01-19 14:04:55.008][trace][18219][128] -> FWR time=18000, msgs=28, okbps=1312,0,0, ikbps=1,0,0
[2017-01-19 14:04:57.578][trace][18219][111] <- CPB time=71000, okbps=0,0,0, ikbps=684,792,0, mr=0/350, p1stpt=20000, pnt=5000
[2017-01-19 14:04:57.719][trace][18219][122] -> PLA time=23000, msgs=63, okbps=802,0,0, ikbps=1,0,0, mw=350
[2017-01-19 14:05:04.690][trace][18219][116] -> FWR time=82000, msgs=0, okbps=734,860,0, ikbps=0,0,0
[2017-01-19 14:05:07.578][trace][18219][111] <- CPB time=81000, okbps=0,0,0, ikbps=727,792,0, mr=0/350, p1stpt=20000, pnt=5000
[2017-01-19 14:05:07.921][trace][18219][131] -> PLA time=29000, msgs=0, okbps=1316,1300,0, ikbps=0,0,0, mw=350
[2017-01-19 14:05:10.826][error][18219][128][4] recv server control message failed. ret=1007(Interrupted system call)

@snavruzov
Copy link
Author

I'm not sure whether I run correctly to trace the crash file, I did as shown below and got some results:

apport-retrace -g _opt_srs_trunk_objs_srs.0.crash 
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading symbols from /opt/srs/trunk/objs/srs...done.
[New LWP 18219]
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
    from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
Core was generated by `./objs/srs -c conf/srs.conf'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f42ee07acc9 in __GI___libc_sigaction (sig=18219, act=0x472b, oact=0x6) at ../sysdeps/unix/sysv/linux/x86_64/sigaction.c:53
53	../sysdeps/unix/sysv/linux/x86_64/sigaction.c: No such file or directory.

@snavruzov
Copy link
Author

I think I caught a consequence of the crash.
When I use http callback options: on_publish, the server is getting crashed.
In fact it's crashing, when I stop RTMP broadcasting, with messages:

pure virtual method called
terminate called without an active exception

Without http callback function the server is working well

@snavruzov
Copy link
Author

The same crashes with the exact output's been noticed in the releases started from v2.0-b3.
In the v2.0-b2(XCORE-SRS/2.0.221(ZhouGuowen)) there is no crashes so far.

@winlinvip
Copy link
Member

The crash signal SIGABRT is weird. I had never got this signal before.
Why there is no call stack of apport-retrace -g _opt_srs_trunk_objs_srs.0.crash? Like the command bt of gdb?
And can you show me the replay step? Only config the http hooks on_publish then will crash? I will try it.

@winlinvip
Copy link
Member

我试了2.0.231加上了on_publish，没有crash。你能否给出配置，重现步骤，或者崩溃时的call stack？

@snavruzov
Copy link
Author

snavruzov commented Jan 20, 2017 •
edited
Loading

There is call stack version srs-2.0-b4 I got:

root@ip-172-31-52-206:/var/crash# apport-retrace -g _opt_srs-2.0-b4_trunk_objs_srs.0.crash 
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading symbols from /opt/srs-2.0-b4/trunk/objs/srs...done.
[New LWP 22270]
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
    from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
Core was generated by `./objs/srs -c conf/srs.conf'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f0f1f57cc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007f0f1f57cc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f0f1f580028 in __GI_abort () at abort.c:89
#2  0x00007f0f1fe87535 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007f0f1fe856d6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f0f1fe85703 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f0f1fe861bf in __cxa_pure_virtual () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x0000000000469f2c in SrsKbps::set_io (this=0x139fcc0, in=0x0, out=0x0) at src/protocol/srs_protocol_kbps.cpp:123
#7  0x000000000049c531 in SrsForwarder::on_unpublish (this=0x13a27b0) at src/app/srs_app_forward.cpp:155
#8  0x000000000048cfa8 in SrsSource::destroy_forwarders (this=0x139c360) at src/app/srs_app_source.cpp:2379
#9  0x000000000048c0ee in SrsSource::on_unpublish (this=0x139c360) at src/app/srs_app_source.cpp:2193
#10 0x000000000047f6ff in SrsRtmpConn::release_publish (this=0x139b6c0, source=0x139c360, is_edge=false) at src/app/srs_app_rtmp_conn.cpp:974
#11 0x000000000047e959 in SrsRtmpConn::publishing (this=0x139b6c0, source=0x139c360) at src/app/srs_app_rtmp_conn.cpp:846
#12 0x000000000047cdc3 in SrsRtmpConn::stream_service_cycle (this=0x139b6c0) at src/app/srs_app_rtmp_conn.cpp:537
#13 0x000000000047c1d1 in SrsRtmpConn::service_cycle (this=0x139b6c0) at src/app/srs_app_rtmp_conn.cpp:415
#14 0x000000000047b145 in SrsRtmpConn::do_cycle (this=0x139b6c0) at src/app/srs_app_rtmp_conn.cpp:210
#15 0x0000000000479467 in SrsConnection::cycle (this=0x139b748) at src/app/srs_app_conn.cpp:89
#16 0x00000000004aa51f in SrsOneCycleThread::cycle (this=0x139b370) at src/app/srs_app_thread.cpp:372
#17 0x00000000004a9cbb in internal::SrsThread::thread_cycle (this=0x1396be0) at src/app/srs_app_thread.cpp:207
#18 0x00000000004a9ec5 in internal::SrsThread::thread_fun (arg=0x1396be0) at src/app/srs_app_thread.cpp:245
#19 0x0000000000530f7b in _st_thread_main () at sched.c:327
#20 0x00000000005316eb in st_thread_create (start=0x535e4f <_st_epoll_pollset_add+746>, arg=0x2710, joinable=0, stk_size=5441783) at sched.c:591
#21 0x00007f0f2047ae30 in ?? ()
#22 0x00007f0f2052bbf0 in ?? ()
#23 0x0000000000530779 in st_poll (pds=<error reading variable: Cannot access memory at address 0x8ffffffa9>, 
    npds=<error reading variable: Cannot access memory at address 0x8ffffffa5>, timeout=<error reading variable: Cannot access memory at address 0x8ffffff99>) at sched.c:86
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

@snavruzov
Copy link
Author

srs.conf config file where I call my http server:

vhost stream.*****.com {

   http_hooks {
        enabled         on;
        on_publish      http://127.0.0.1:8443/live/start;
        on_unpublish    http://127.0.0.1:8443/live/stop;
   }

   forward      127.0.0.1:1940;

   gop_cache       on;
   queue_length    10;
}

@snavruzov
Copy link
Author

snavruzov commented Jan 20, 2017 •
edited
Loading

Yes, without http_hooks section the server acts perfectly without any crashes.

@snavruzov
Copy link
Author

OK, following by a recent tests, I tried the same steps on Ubuntu 16.04, no crashes.

@winlinvip winlinvip changed the title SRS crash log SRS crash when forward and http hooks Jan 22, 2017
@winlinvip
Copy link
Member

It seems the forwarder crash when unpublish, maybe caused by http hooks. I will finger it out.

@winlinvip
Copy link
Member

winlinvip commented Jan 22, 2017 •
edited
Loading 

void SrsForwarder::on_unpublish()
{
    pthread->stop();
    
    close_underlayer_socket();
    
    srs_freep(client);
    srs_freep(io); // The io is freed here.
    kbps->set_io(NULL, NULL); // In this function, the previous io, that is the freed io up there, will be used and it sometimes cause the crash.
}

The io must be released by set_io(NULL) before free it:

kbps->set_io(NULL, NULL);
srs_freep(io);

This bug also exists in:

SrsEdgeIngester::stop()
SrsEdgeForwarder::stop()
SrsEdgeIngester::connect_server()
SrsForwarder::connect_server()
SrsForwarder::on_unpublish()

winlinvip added a commit that referenced this issue Jan 22, 2017
@winlinvip
for #752, release the io then free it for kbps. 2.0.232
6706bda
@winlinvip
Copy link
Member

Fixed in 2.0.232

@winlinvip winlinvip mentioned this issue Feb 25, 2017
Closed
@winlinvip winlinvip self-assigned this Sep 18, 2021
@winlinvip winlinvip added TransByAI Translated by AI/GPT. EnglishNative This issue is conveyed exclusively in English. and removed TransByAI Translated by AI/GPT. labels Jul 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@winlinvip winlinvip

Labels
Bug It might be a bug. EnglishNative This issue is conveyed exclusively in English.
Projects
None yet
Milestone
2.0
Development

No branches or pull requests
2 participants
@winlinvip @snavruzov