"bad cseq xxxx expected=xxxx" in logs after agent docker container restart

[TadasSu]

Agent v3.1.3.0 (DOCKER)

These errors goes without stopping after agent container restart and stops only if camera disabled:
9:49:17 AM [rtsp @ 0x7f0c6015bd00] RTP: PT=61: bad cseq 4ad7 expected=8733
9:49:17 AM [rtsp @ 0x7f0c6015bd00] RTP: PT=61: bad cseq 8733 expected=4ad9
....

If I enable camera, errors starts again.

If I disable camera before container restart - all goes well after starting. Then I can enable the camera without errors.

How to reproduce:
Restart docker container when ONVIF camera enabled and in use (showing image).
docker restart _agent_container_id_

Log attached:
log2.txt

[ispysoftware]

These errors are from ffmpeg. Generally it's just informational unless your camera isn't working?

[TadasSu]

Camera works with media distortion when these occurs. (and logs grows pretty fast about 100 000 lines per hour).
Can I take some action to prevent from this?

[ispysoftware]

logs are capped at 100kb so they won't cause issues.
unfortunately this seems to be an unresolved issue with ffmpeg. You could try changing the transport in video source settings, ffmpeg options, RTSP mode

[maxirus]

logs are capped at 100kb so they won't cause issues.

@ispysoftware Although they are capped, it does cause issues with disk thrashing, which subsequently causes IOWAIT. This is exacerbated when using NFS or iSCSI volumes. IMO this issue should be re-opened.

Was a workaround ever discovered?

Based on my shallow Google search, it appears to be an issue in the way ffmpeg process is closed. Is this issue related? Also, it appears it's related to Reolink cameras (which is what I'm using).

[ispysoftware]

It doesn't write the logs to disk, it stores them in memory.

Also Agent doesn't call the ffmpeg executable, it uses the ffmpeg libraries via https://github.com/Ruslan-B/FFmpeg.AutoGen

Have you checked for updated firmware for your camera? If this was a bug in ffmpeg it'd be happening with all rtsp streams - as it just seems to be specific cameras i'm guessing the camera's themselves are producing malformed rtsp streams.

you could try installing VLC and set VLC as the decoder in Agent .

[ispysoftware]

Seems the issue is that if the stream hangs (connection lost) then ffmpeg doesn't send the teardown or goodbye rtsp message to the camera - so the camera keeps the stream going for another 30 seconds or so and when Agent tries to reconnect the camera just sends the existing stream which doesn't have headers and you get all these errors. VLC has better code than ffmpeg for this so that should resolve it. There are a few things i can try but my cameras don't have this issue so it's hard to debug it. Try the next update and let me know if it resolves anything.

[ispysoftware]

@maxirus did the latest update help?

[maxirus]

It doesn't write the logs to disk, it stores them in memory.

When you're running in a Docker container, it sends to STDOUT which subsequently gets written to disk (docker logs).

Have you checked for updated firmware for your camera? If this was a bug in ffmpeg it'd be happening with all rtsp streams - as it just seems to be specific cameras i'm guessing the camera's themselves are producing malformed rtsp streams.

I'm running the latest Firmware for my Reolink RLC-410 cameras. It very well could be the cameras. Reolink cams are fairly popular though so I imagine I'm not the only one facing this issue; or maybe it localized to this specific model.

did the latest update help?

Will try to test the new version this weekend. Thanks

[ispysoftware]

@maxirus ah i didnt know that can you not turn that off in docker?

[alfredopironti]

I just started playing with agent DVR, faced the same "bad cseq" issue, and found this post. Interestingly, I also own a Reolink RLC-410W camera. Unfortunately, I don't own other cameras for testing.

Are there any specific tests I could help with? e.g. a script to connect/disconnect the device or agentdvr, that may help with pinpointing which piece is at fault?

EDIT: I also found this post where apparently Reolink cameras are particularly sensitive to how a stream is terminated? Not sure if it can be of any help.

[alfredopironti]

After further digging, it seems this post has a fair explanation of the issue. Reolink cameras use a very old Live555 RTSP server version (from 2013), which, among other bugs, don't correctly free TCP sessions when they drop. So when re-connecting, the camera may be streaming two (or more) streams over the same TCP connection, which confuses the client (hence the bad cseq; usually in the ffmpeg logs you'll see two or three sequences of seq numbers interleaved, representing the conflicting sessions).

It seems the fix is for Reolink to update their firmware to use newer versions of Live555. I've contacted their support.

[doitandbedone]

Converting into a conversation to archive for others.