WebRTC causes crash on close

[RogerDavenport]

Which version are you using?

v0.21.4

Which operating system are you using?

• Linux amd64 standard
• Linux amd64 Docker
• Linux arm64 standard
• Linux arm64 Docker
• Linux arm7 standard
• Linux arm7 Docker
• Linux arm6 standard
• Linux arm6 Docker
• Windows amd64 standard
• Windows amd64 Docker (WSL backend)
• macOS amd64 standard
• macOS amd64 Docker
• Other (please describe)

Describe the issue

panic: close of closed channel: Hard crash when reading RTSP streams through WebRTC.

Description

Describe how to replicate the issue

Have several RTSP streams pulling from security cameras and after several hours of WebRTC stream open/closes you will get a crash. Happens once a day or so.

Paths:

"paths": {
"100-100-101": {
"source": "rtsp://camera:554/Profile2/media.smp",
"sourceProtocol": "tcp",
"sourceAnyPortEnable": false,
"sourceFingerprint": "",
"sourceOnDemand": true,
"sourceOnDemandStartTimeout": "10s",
"sourceOnDemandCloseAfter": "10s",
"sourceRedirect": "",
"disablePublisherOverride": false,
"fallback": ""

WebRTC config:
"webrtcDisable": false,
"webrtcAddress": ":8889",
"webrtcEncryption": true,
"webrtcServerKey": "/certs/server.key",
"webrtcServerCert": "/certs/server.cer",
"webrtcAllowOrigin": "*",
"webrtcTrustedProxies": [],
"webrtcICEServers": [
"stun:stun.l.google.com:19302"
],
"webrtcICEHostNAT1To1IPs": [
"2.3.4.5"
],
"webrtcICEUDPMuxAddress": ":8189",
"webrtcICETCPMuxAddress": "",

1. Start the docker image
2. publish with a few live RTSP streams
3. read multiple streams with WebRTC, close and open the streams
4. Crash eventually

Did you attach the server logs?

rtsp-simple-server_1 | 2023/02/17 11:57:30 INF [WebRTC] [conn 147.206.80.144:50349] is reading from path '1204-838310701-B316', 2 tracks (H264, G711)
rtsp-simple-server_1 | 2023/02/17 11:57:33 INF [WebRTC] [conn 10.209.20.163:58968] closed (websocket error: websocket: close 1001 (going away))
rtsp-simple-server_1 | 2023/02/17 11:57:33 INF [WebRTC] [conn 10.209.20.163:58991] closed (websocket error: websocket: close 1001 (going away))
rtsp-simple-server_1 | panic: close of closed channel
rtsp-simple-server_1 |
rtsp-simple-server_1 | goroutine 75992 [running]:
rtsp-simple-server_1 | github.com/aler9/rtsp-simple-server/internal/core.(*webRTCConn).runInner.func3(0x6)
rtsp-simple-server_1 | /s/internal/core/webrtc_conn.go:366 +0xc9
rtsp-simple-server_1 | created by github.com/pion/webrtc/v3.(*PeerConnection).onConnectionStateChange
rtsp-simple-server_1 | /go/pkg/mod/github.com/pion/webrtc/[email protected]/peerconnection.go:492 +0x173

yes

Did you attach a network dump?

NO

[camfort48]

similar problem, after some time panic:

Feb 20 08:32:39 portalNew rtsp-simple-server[760]: panic: close of closed channel
Feb 20 08:32:39 portalNew rtsp-simple-server[760]: goroutine 51533 [running]:
Feb 20 08:32:39 portalNew rtsp-simple-server[760]: github.com/aler9/rtsp-simple-server/internal/core.(*webRTCConn).runInner.func3(0x6)
Feb 20 08:32:39 portalNew rtsp-simple-server[760]: #11/s/internal/core/webrtc_conn.go:366 +0xc9
Feb 20 08:32:39 portalNew rtsp-simple-server[760]: created by github.com/pion/webrtc/v3.(*PeerConnection).onConnectionStateChange
Feb 20 08:32:39 portalNew rtsp-simple-server[760]: #11/go/pkg/mod/github.com/pion/webrtc/[email protected]/peerconnection.go:492 +0x173
Feb 20 08:32:39 portalNew systemd[1]: rtsp-simple-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 20 08:32:39 portalNew systemd[1]: rtsp-simple-server.service: Failed with result 'exit-code'.

[arkanoid87]

same here with v0.21.4

panic: close of closed channel

goroutine 20281 [running]:
github.com/aler9/rtsp-simple-server/internal/core.(*webRTCConn).runInner.func3(0x6)
        /s/internal/core/webrtc_conn.go:366 +0xb4
created by github.com/pion/webrtc/v3.(*PeerConnection).onConnectionStateChange
        /go/pkg/mod/github.com/pion/webrtc/[email protected]/peerconnection.go:492 +0x17c

[aler9]

Thanks all for reporting the issue, i assumed that callbacks of the WebRTC library (pion/webrtc) were thread-safe but in fact they were not, and the same line was sometimes called twice in parallel, leading to the crash. This is fixed in v0.21.5.

[arkanoid87]

thanks!

[RogerDavenport]

Thank you, excellent!

…

________________________________ From: arkanoid87 ***@***.***> Sent: Tuesday, February 21, 2023 5:54:03 PM To: aler9/rtsp-simple-server ***@***.***> Cc: Roger Davenport ***@***.***>; Author ***@***.***> Subject: [EXTERNAL] Re: [aler9/rtsp-simple-server] WebRTC causes crash on close (Issue #1482) CAUTION EXTERNAL EMAIL! DO YOU RECOGNIZE THE SENDER? ARE YOU EXPECTING THIS EMAIL? If this email looks suspicious, click the Report Suspicious Email button in Outlook to report it. DO NOT click any links and NEVER input your username and password. thanks! — Reply to this email directly, view it on GitHub<#1482 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AV6L3TRDYLKWNXC6Y6A4Q53WYVIRXANCNFSM6AAAAAAU7QLFOQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[camfort48]

Thank you!

[ajianjalil]

Hi @aler9

I see the same issue in normal rtsp as well, I am running this on the latest docker container and 8-12 rtsp videos are connected and disconnected at random times, eventually crashing with this message. Kindly help.

2023/09/01 09:08:37 INF [RTSP] [conn 10.5.1.95:63449] closed (no one is publishing to path 'video6')
2023/09/01 09:08:37 INF [RTSP] [conn 127.0.0.1:56528] opened
2023/09/01 09:08:37 INF [RTSP] [conn 127.0.0.1:56540] opened
2023/09/01 09:08:38 INF [RTSP] [conn 127.0.0.1:53856] closed (EOF)
panic: close of closed channel
goroutine 83 [running]:
github.com/bluenviron/gortsplib/v4/pkg/rtcpreceiver.(*RTCPReceiver).Close(...)
	/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/pkg/rtcpreceiver/rtcpreceiver.go:99
github.com/bluenviron/gortsplib/v4.(*serverSessionFormat).stop(...)
	/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server_session_format.go:60
github.com/bluenviron/gortsplib/v4.(*serverSessionMedia).stop(0xc00016c080?)
	/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server_session_media.go:108 +0x11c
github.com/bluenviron/gortsplib/v4.(*ServerSession).run(0xc000360000)
	/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server_session.go:345 +0x1c6
created by github.com/bluenviron/gortsplib/v4.newServerSession
	/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/server_session.go:224 +0x41c

[aler9]

@ajianjalil this is fixed in v1.0.2

[ajianjalil]

@aler9 Thank you very much.

[github-actions]

This issue is being locked automatically because it has been closed for more than 6 months.
Please open a new issue in case you encounter a similar problem.