Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support live555 servers older than 2017.06.04 (eg some Reolink models), which have buggy RTP/TCP #17

Closed
scottlamb opened this issue Aug 14, 2021 · 21 comments
Labels
interop interoperability problem with another (often broken) RTSP implementation

Comments

@scottlamb
Copy link
Owner

scottlamb commented Aug 14, 2021

Latest understanding: these cameras are using LIVE555 Streaming Media v2013.04.08 (according to the session-level tool attribute in their DESCRIBE response) which has at least two major bugs described in live555's changelog:

2015.05.03:
- Updated the "RTSPServer" implementation to fix a bug in RTP/RTCP-over-TCP streaming.
  Before, if the "RTSPClientConnection" object closed before the "RTSPClientSession" object,
  and the TCP connection was also being used for RTP/RTCP-over-TCP streaming, then the
  streaming state (in the "RTSPClientSession") would stay alive, even though the TCP socket
  had closed (and the socket number possibly reused for a subsequent connection).
  This could cause a problem when the "RTSPClientSession" was later reclaimed (due to inactivity).
  Now, whenever a "RTSPClientConnection" object is closed (due to the RTSP TCP connection closing),
  we make sure that we also close any stream that had been using the same TCP connection
  for RTP/RTCP-over-TCP streaming.  (Thanks to Kirill Zhegulev for noting this issue.)
2013.12.04:
- Updated the "sendDataOverTCP()" function (in "RTPInterface.cpp") to allow for the possibility of
  one of the "send()" calls partially succeeding - i.e., writing some, but not all, of its data.

also, there was a follow-up to the 2015.05.03 entry two years later:

2017.06.04:
- Fixed a bug in "RTPInterface::removeStreamSocket()" that could cause not all 'TCP stream'
  records for a given socket number to be removed if a TCP socket I/O error occurred
  (during RTP/RTCP-over-TCP streaming).  (Thanks to Gerald Hansink et al for reporting this.)

Currently your best bet to successfully talk with these cameras is to ask the manufacturer for a firmware upgrade (see note below for the Reolink 420-5MP / hardware version IPC_51516M5M) or connect with UDP via retina::client::Transport::Udp (#30). We have some other ideas below.


Original comment:

As mentioned in this Apr 29th comment on a moonfire-nvr issue, Reolink cameras have an odd behavior. They sometimes set up a stream with one ssrc, eg:

RTP-Info: url=trackID=1;seq=40477;rtptime=2077624148;ssrc=c517011f,url=trackID=2;seq=0;rtptime=0;ssrc=00000000

then send RTP packets for both this one and another ssrc. I think the latter is for an earlier failed session or something. Sometimes they'll even send the RTP packets for the other ssrc before the connection is even authenticated, which seems like a security hole (if you expose the cameras to an untrusted network, which is probably a bad idea anyway).

Currently when this happens, Retina will fail with an error such as the following (from one of @jlpoolen 's logs today in this moonfire-nvr issue):

Expected ssrc=Some(21cd90d7) seq=Some(8f4a) got ssrc=6d87df1a seq=88b9 ts=1866533263 (mod-2^32: 1866533263), npt 0.000 at [192.168.1.88:33514(me)->192.168.1.48:554@2021-08-14T11:00:57 pos=1510@2021-08-14T11:00:57]

I think we should add an option to so that when we know what ssrc to expect, we just (log and) ignore packets with any other ssrc. This will likely make these cameras behave acceptably, especially if (as I suspect) the other ssrc goes away after a minute or so when the stale session times out.

@scottlamb scottlamb added the interop interoperability problem with another (often broken) RTSP implementation label Aug 14, 2021
@jlpoolen
Copy link

A possible tool: a program that uses the Rust rtsp client that simply opens a connection and tallies the packets received and logs anomalies. This, along with other items you may have learned about with other cameras, could give a quality report on the camera's ability to keep a stream going (and the libraries ability to handle error/resets). It could be run outside of moonfire-nvr and left running for a period of time, say up to 24-72 hours. Almost like a nurse taking vitals before a doctor sees a patient: getting a profile of the camera's ability to send packets conforming to the specifications.

@jlpoolen
Copy link

I was starting to do something like what you looking into in this issue with ffmpeg and the real server when I got interrupted or distracted. (Having these hardware issues really throws me off track.) Generating logs has taken some time for me to perfect and I'll go the extra distance (i.e. handling & preserving ANSI codes) where a lot of prospective users may not. If there were a simple program that you feed the IP, logon credentials to that would then generate a log on the camera's ability to manage an rtsp feed and its resilience (and that of the network) could help crystallize issues that need consideration or handling and provide a stream-lined path of zeroing in on issues.

@scottlamb
Copy link
Owner Author

This specific issue is in theory fixed in Retina. I'm going to fix another Reolink problem I found on my own test camera then prepare a Moonfire NVR change to pick up both.

Re: tool: did you see this comment? It's a little bit like what you're talking about. But I'm not super motivated right now to polish it into something that gets detailed spec compliance info automatically. I have more improvements I want to make than time already, so I'm not going spend a lot of time trying to to get detailed bug reports from people who don't care enough to gather them. I'm prioritizing getting Retina and Moonfire NVR working really well on cameras I have access to (including polished on-NVR and on-camera analytics, live view, and scrub bar UI) over getting it working on every camera out there.

@scottlamb
Copy link
Owner Author

I found a live555 changelog entry about this:

2015.05.03:
- Updated the "RTSPServer" implementation to fix a bug in RTP/RTCP-over-TCP streaming.
  Before, if the "RTSPClientConnection" object closed before the "RTSPClientSession" object,
  and the TCP connection was also being used for RTP/RTCP-over-TCP streaming, then the
  streaming state (in the "RTSPClientSession") would stay alive, even though the TCP socket
  had closed (and the socket number possibly reused for a subsequent connection).
  This could cause a problem when the "RTSPClientSession" was later reclaimed (due to inactivity).
  Now, whenever a "RTSPClientConnection" object is closed (due to the RTSP TCP connection closing),
  we make sure that we also close any stream that had been using the same TCP connection
  for RTP/RTCP-over-TCP streaming.  (Thanks to Kirill Zhegulev for noting this issue.)

The problematic Reolink cameras say a=tool:LIVE555 Streaming Media v2013.04.08 so no surprise that they're affected by this bug.

My workaround isn't working well: at the timeout of the original session, the camera seems to be sending bytes which aren't not properly framed as RTSP interleaved data, we're dropping the connection, and the cycles repeats.

Another option is to just ensure we send a TEARDOWN anywhere possible, and then wait out the session timeout any time it's not possible.

@scottlamb scottlamb reopened this Aug 24, 2021
@jlpoolen
Copy link

I'm updating the ip 48 GarageWest camera which is:

 model: RLC-420-5MP
 Hw No.: IPC_51516M5M

and if successfully accomplished, I will test with the Retina client example.

@jlpoolen
Copy link

You have to be careful in selecting which upgrade package. Firmware number is not sufficient, you also must align to the model number of your camera. I mistakenly downloaded the first firmware match only read the PDF within the zip archive that the model that upgrade applied to did not match my model. I then returned to Reolink's site and found several matches to my firmware, and thus discerned which one applied to my camera model. I applied the patch and rebooted. Will test this evening with Retina.

@scottlamb
Copy link
Owner Author

Here's the cause of the framing error in the old live555 code:

Boolean RTPInterface::sendDataOverTCP(int socketNum, u_int8_t const* data, unsigned dataSize, Boolean forceSendToSucceed) {
  if (send(socketNum, (char const*)data, dataSize, 0/*flags*/) != (int)dataSize) {
    // The TCP send() failed.

    if (forceSendToSucceed && envir().getErrno() == EAGAIN) {
      // The OS's TCP send buffer has filled up (because the stream's bitrate has exceeded the capacity of the TCP connection!).
      // Force this data write to succeed, by blocking if necessary until it does:
#ifdef DEBUG_SEND
      fprintf(stderr, "sendDataOverTCP: resending %d-byte send (blocking)\n", dataSize); fflush(stderr);
#endif
      makeSocketBlocking(socketNum);
      Boolean sendSuccess = send(socketNum, (char const*)data, dataSize, 0/*flags*/) == (int)dataSize;
      makeSocketNonBlocking(socketNum);
      return sendSuccess;
    }
    return False;
  }

  return True;
}

send can partially send the data, which happens when the TCP window fills. This code sends the whole thing from the beginning when that happens, which is wrong. This bug has also since been fixed:

2013.12.04:
- Updated the "sendDataOverTCP()" function (in "RTPInterface.cpp") to allow for the possibility of
  one of the "send()" calls partially succeeding - i.e., writing some, but not all, of its data.

(btw, live555's latest code is still bad. If one client's TCP window fills, the entire server will stall waiting for it to come back.)

@scottlamb
Copy link
Owner Author

scottlamb commented Aug 24, 2021

I flashed a RLC-420-5MP, hardware version IPC_51516M5M with the 12/11/2020 firmware, aka v3.0.0.136_20121101. It still says a=tool:LIVE555 Streaming Media v2013.04.08. So this was ancient history for live555 but is a current problem for Reolink cameras.

Here are some options I see:

  1. complain to Reolink support that the latest available firmware for these models has badly flawed support for RTSP/TCP and ask them to update the live555 version. (Unless they have negotiated a special license with live555, they are legally required to comply in countries that care about licenses. But Reolink is Chinese, and I don't think China is such a country...) [edit: Reolink came through and prepared new firmware for the RLC-420-5MP. See below.]
  2. switch to RTP/UDP. Normally I'd expect RTP/UDP to be flakier and fussier than RTP/TCP but with these bugs it's likely the reverse. Retina doesn't support RTP/UDP today but it's something I'd like to add eventually. [edit: done! UDP #30]
  3. don't use Reolink's RTSP support at all. Instead use their proprietary protocol (see Neolink) or RTMP. Via an RTSP bridge, or in Moonfire's case I could eventually add direct support for these. [edit: Neolink says "Neolink does not support other cameras such as the RLC-420, since they already provide native RTSP." But maybe they'd be open to adding this support if we point out that Reolink's native RTSP really sucks.]
  4. avoid the additional ssrc via teardown (when possible) or waiting 65 seconds (when not). Hope that without the redundant data, the framing error doesn't come up enough to be a big problem. (Another idea I tried, discarding the extra interleaved data, didn't work out well. With Reolink cameras the stale session apparently never times out if it's sending data on a still-active session. So it really needs teardown or a cooling-off period.) [edit: Retina 0.3.1 implements this idea. It still has some rough edges; see retry TEARDOWN and avoid having to wait for session expiration #34.)
  5. do 4 and also make an ugly, old-live555-specific hack to avoid the framing error also: before interpreting an interleaved data packet, wait for the following message. If the second message doesn't parse, figure out how many redundant bytes were added mid-packet by the sendDataOverTCP() bug and strip them out before parsing the first message.

@scottlamb scottlamb changed the title for Reolink cameras: ignore unexpected additional ssrc support old live555 servers with buggy RTP/TCP (some Reolink models) Aug 24, 2021
@scottlamb
Copy link
Owner Author

I just wrote to Reolink.

@jlpoolen
Copy link

jlpoolen commented Aug 24, 2021

or
6. Declare Reolink cameras unsupported at this time.

I'd be interested to learn what other manufacturers use as servers and their conformance to standards. It seems having Scott spend time on one brand which has been documented to be using a buggy server has diminishing returns.

I've been using Reolink because they're cheap and if they get stolen or vandalized, it's not going to be expensive to replace. If other brands provide a lot less problems conforming to standards, then it would be appropriate to weigh their cost. If Reolinks cost $40, and Brand X costs $70 and works well within the RTSP protocol, then it may make sense to switch brands. Now, if the next least expensive brand that plays nicely costs $150, then it might merit further consideration.

I think further time Scott spends on Reolink's issues is time away from other enhancements that affect all users of moonfire-nvr.

@scottlamb
Copy link
Owner Author

I'm currently not sure what manufacturers to recommend, though. :-(

  • Dahua, Hikvision, and Uniview are supporting genocide, so they're out regardless of technical merit or value.
  • I have a Geovision camera being shipped to me that will arrive Thursday; I'll try it out and see if it's any better.
  • Hanwha seems a bit pricy; Axis more so.

@jlpoolen
Copy link

I flashed my older one as described above with the correct firmware version for the model I had. I, too, have the session (using the LIVE555 openRTSPclient) showing:

v=0
o=- 1629795645777963 1 IN IP4 192.168.1.48
s=Session streamed by "preview"
i=h264Preview_01_main
t=0 0
a=tool:LIVE555 Streaming Media v2013.04.08
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session streamed by "preview"
a=x-qt-text-inf:h264Preview_01_main
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:500
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=640033;sprop-parameter-sets=Z2QAM6zoAoALWQ==,aO48MA==
a=control:trackID=1
m=audio 0 RTP/AVP 97
c=IN IP4 0.0.0.0
b=AS:256
a=rtpmap:97 MPEG4-GENERIC/16000
a=fmtp:97 streamtype=5;profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1408; profile=1;
a=control:trackID=2

Opened URL "rtsp://192.168.1.48:554/h264Preview_01_main", returning a SDP description:
v=0
o=- 1629795645777963 1 IN IP4 192.168.1.48
s=Session streamed by "preview"
i=h264Preview_01_main
t=0 0
a=tool:LIVE555 Streaming Media v2013.04.08
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session streamed by "preview"
a=x-qt-text-inf:h264Preview_01_main
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:500
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=640033;sprop-parameter-sets=Z2QAM6zoAoALWQ==,aO48MA==
a=control:trackID=1
m=audio 0 RTP/AVP 97
c=IN IP4 0.0.0.0
b=AS:256
a=rtpmap:97 MPEG4-GENERIC/16000
a=fmtp:97 streamtype=5;profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1408; profile=1;
a=control:trackID=2

I tried several sessions using the Retina client and continued to have problems. The best session would be about 16 seconds and then I'd get an error message. Examples:

282699224 (mod-2^32: 282699224), npt 11.968: 249-byte audio frame
652994919 (mod-2^32: 652994919), npt 11.900: 29782-byte video frame
282700248 (mod-2^32: 282700248), npt 12.032: 249-byte audio frame
652997889 (mod-2^32: 652997889), npt 11.933: 30061-byte video frame
653000859 (mod-2^32: 653000859), npt 11.966: 30276-byte video frame
E20210824 17:30:42.477 main client] Fatal: [192.168.1.28:48462(me)->192.168.1.48:554@2021-08-24T17:30:29, 12688689@2021-08-24T17:30:42] RTSP framing error: Invalid RTSP message; next 128 of 790 buffered bytes are:
0000:   0b 32 1a 41  68 15 23 ea  79 50 6c e3  d1 f7 6f a2   .2.Ah.#.yPl...o.
0010:   6e 61 82 e0  2a db df 8c  d4 7a 3d 11  fa 38 3e c5   na..*....z=..8>.
0020:   9d 02 84 cb  49 5e c4 d4  bb b0 25 16  57 50 24 b6   ....I^....%.WP$.
0030:   76 98 91 26  60 90 ff a2  d5 01 da c2  ce d2 56 5f   v..&`.........V_
0040:   79 39 a5 02  b5 ae 9c 92  44 3e 42 2d  72 28 6d 84   y9......D>B-r(m.
0050:   2b a1 8e 5d  cc 8d 89 23  29 05 f7 7a  4e d8 48 5b   +..]...#)..zN.H[
0060:   ed d3 07 43  a1 7f 25 9b  b4 6f e1 90  4d 96 37 61   ...C..%..o..M.7a
0070:   45 f7 1d 6a  ad d4 4c cc  3b c2 45 35  21 c8 27 f3   E..j..L.;.E5!.'.
I20210824 17:30:42.478 main client] Done
Tue Aug 24 05:30:42 PM PDT 2021
jlpoole@taurus /usr/local/src/retina $ 


4240106907 (mod-2^32: 4240106907), npt 15.932: 32993-byte video frame
4240109877 (mod-2^32: 4240109877), npt 15.965: 33618-byte video frame
4061414557 (mod-2^32: 4061414557), npt 16.000: 249-byte audio frame
E20210824 17:46:18.304 main client] Fatal: [192.168.1.28:48544(me)->192.168.1.48:554@2021-08-24T17:46:02, 16728945@2021-08-24T17:46:18] RTerror: Invalid RTSP message; next 128 of 1447 buffered bytes are:
0000:   60 50 14 fc  bb 08 c3 9e  00 10 96 5c  81 9a 24 23   `P.........\..$#
0010:   bf 15 b9 c5  d4 34 85 69  ca 20 c9 ff  14 4b b9 d9   .....4.i. ...K..
0020:   fd 7d 87 af  c8 26 67 63  71 c4 88 01  09 27 09 7d   .}...&gcq....'.}
0030:   37 67 9c 86  1c c3 94 a2  1b ac 34 dd  69 4f 70 d5   7g........4.iOp.
0040:   ac d6 e3 2a  33 f2 2a 25  0f 03 54 d0  e5 9c 90 10   ...*3.*%..T.....
0050:   cc 3b 98 c1  99 01 7a ed  ee d9 fb ca  93 f9 ad 40   .;....z........@
0060:   2e 3c 5f 29  46 bf ed 78  eb ad b4 a5  b2 27 ee cf   .<_)F..x.....'..
0070:   58 82 7a 24  1b 35 c0 b7  1f 94 de 74  dc 85 2f 08   X.z$.5.....t../.
I20210824 17:46:18.304 main client] Done
Tue Aug 24 05:46:18 PM PDT 2021
jlpoole@taurus /usr/local/src/retina $ date; cargo run --example client mp4 --url rtsp://192.168.1.48:554/h264Preview_01_main --username moon --password fire123 /tmp/retina_out.mp4;date;

I had to wait several minutes or I would get receiving data packets type errors which is line with Scott's analysis.

Now, I did successfully save a 10 minute video (with audio) using the Live555 openRTSP client. There may be blips, but it continued on. Here's what was printed to my console during the 10 session:

Created new TCP socket 3 for connection
Connecting to 192.168.1.48, port 554 on socket 3...
...remote connection opened
Sending request: OPTIONS rtsp://192.168.1.48:554/h264Preview_01_main RTSP/1.0
CSeq: 2
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)


Received 152 new bytes of response data.
Received a complete OPTIONS response:
RTSP/1.0 200 OK
CSeq: 2
Date: Wed, Aug 25 2021 02:21:10 GMT
Public: OPTIONS, DESCRIBE, SETUP, TEARDOWN, PLAY, PAUSE, GET_PARAMETER, SET_PARAMETER


Sending request: DESCRIBE rtsp://192.168.1.48:554/h264Preview_01_main RTSP/1.0
CSeq: 3
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)
Accept: application/sdp


Received 175 new bytes of response data.
Received a complete DESCRIBE response:
RTSP/1.0 401 Unauthorized
CSeq: 3
Date: Wed, Aug 25 2021 02:21:10 GMT
WWW-Authenticate: Digest realm="LIVE555 Streaming Media", nonce="435a79a4baaee7fe09baf6278cfe05ce"


Resending...
Sending request: DESCRIBE rtsp://192.168.1.48:554/h264Preview_01_main RTSP/1.0
CSeq: 4
Authorization: Digest username="moon", realm="LIVE555 Streaming Media", nonce="435a79a4baaee7fe09baf6278cfe05ce", uri="rtsp://192.168.1.48:554/h264Preview_01_main", response="892c0c789b008ce2d51853d8c85a81df"
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)
Accept: application/sdp


Received 890 new bytes of response data.
Received a complete DESCRIBE response:
RTSP/1.0 200 OK
CSeq: 4
Date: Wed, Aug 25 2021 02:21:10 GMT
Content-Base: rtsp://192.168.1.48/h264Preview_01_main/
Content-Type: application/sdp
Content-Length: 717

v=0
o=- 1629795645777963 1 IN IP4 192.168.1.48
s=Session streamed by "preview"
i=h264Preview_01_main
t=0 0
a=tool:LIVE555 Streaming Media v2013.04.08
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session streamed by "preview"
a=x-qt-text-inf:h264Preview_01_main
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:500
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=640033;sprop-parameter-sets=Z2QAM6zoAoALWQ==,aO48MA==
a=control:trackID=1
m=audio 0 RTP/AVP 97
c=IN IP4 0.0.0.0
b=AS:256
a=rtpmap:97 MPEG4-GENERIC/16000
a=fmtp:97 streamtype=5;profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1408; profile=1;
a=control:trackID=2

Opened URL "rtsp://192.168.1.48:554/h264Preview_01_main", returning a SDP description:
v=0
o=- 1629795645777963 1 IN IP4 192.168.1.48
s=Session streamed by "preview"
i=h264Preview_01_main
t=0 0
a=tool:LIVE555 Streaming Media v2013.04.08
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session streamed by "preview"
a=x-qt-text-inf:h264Preview_01_main
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:500
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=640033;sprop-parameter-sets=Z2QAM6zoAoALWQ==,aO48MA==
a=control:trackID=1
m=audio 0 RTP/AVP 97
c=IN IP4 0.0.0.0
b=AS:256
a=rtpmap:97 MPEG4-GENERIC/16000
a=fmtp:97 streamtype=5;profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1408; profile=1;
a=control:trackID=2

Created receiver for "video/H264" subsession (client ports 35696-35697)
Created receiver for "audio/MPEG4-GENERIC" subsession (client ports 56590-56591)
Sending request: SETUP rtsp://192.168.1.48/h264Preview_01_main/trackID=1 RTSP/1.0
CSeq: 5
Authorization: Digest username="moon", realm="LIVE555 Streaming Media", nonce="435a79a4baaee7fe09baf6278cfe05ce", uri="rtsp://192.168.1.48/h264Preview_01_main/", response="d463e3d308152052aa01ed9305469a97"
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)
Transport: RTP/AVP;unicast;client_port=35696-35697


Received 203 new bytes of response data.
Received a complete SETUP response:
RTSP/1.0 200 OK
CSeq: 5
Date: Wed, Aug 25 2021 02:21:10 GMT
Transport: RTP/AVP;unicast;destination=192.168.1.28;source=192.168.1.48;client_port=35696-35697;server_port=6970-6971
Session: 54C4E3AC


Setup "video/H264" subsession (client ports 35696-35697)
Sending request: SETUP rtsp://192.168.1.48/h264Preview_01_main/trackID=2 RTSP/1.0
CSeq: 6
Authorization: Digest username="moon", realm="LIVE555 Streaming Media", nonce="435a79a4baaee7fe09baf6278cfe05ce", uri="rtsp://192.168.1.48/h264Preview_01_main/", response="d463e3d308152052aa01ed9305469a97"
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)
Transport: RTP/AVP;unicast;client_port=56590-56591
Session: 54C4E3AC


Received 203 new bytes of response data.
Received a complete SETUP response:
RTSP/1.0 200 OK
CSeq: 6
Date: Wed, Aug 25 2021 02:21:10 GMT
Transport: RTP/AVP;unicast;destination=192.168.1.28;source=192.168.1.48;client_port=56590-56591;server_port=6972-6973
Session: 54C4E3AC


Setup "audio/MPEG4-GENERIC" subsession (client ports 56590-56591)
Outputting to the file: "stdout"
Sending request: PLAY rtsp://192.168.1.48/h264Preview_01_main/ RTSP/1.0
CSeq: 7
Authorization: Digest username="moon", realm="LIVE555 Streaming Media", nonce="435a79a4baaee7fe09baf6278cfe05ce", uri="rtsp://192.168.1.48/h264Preview_01_main/", response="7dce274e8dc69e172c45fc9125d88472"
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)
Session: 54C4E3AC
Range: npt=0.000-600.000


Received 251 new bytes of response data.
Received a complete PLAY response:
RTSP/1.0 200 OK
Server: Rtsp Server/2.0
CSeq: 7
Date: Wed, Aug 25 2021 02:21:10 GMT
Range: npt=0.000-
Session: 54C4E3AC
RTP-Info: url=trackID=1;seq=8202;rtptime=718423480;ssrc=d89a16ec,url=trackID=2;seq=51872;rtptime=3942782428;ssrc=6aca52b4


Started playing session
Receiving streamed data (for up to 600.000000 seconds)...
Sending request: TEARDOWN rtsp://192.168.1.48/h264Preview_01_main/ RTSP/1.0
CSeq: 8
Authorization: Digest username="moon", realm="LIVE555 Streaming Media", nonce="435a79a4baaee7fe09baf6278cfe05ce", uri="rtsp://192.168.1.48/h264Preview_01_main/", response="00429a724edcc215d201530ed4392596"
User-Agent: ./openRTSP (LIVE555 Streaming Media v2021.08.24)
Session: 54C4E3AC


Received 65 new bytes of response data.
Received a complete TEARDOWN response:
RTSP/1.0 200 OK
CSeq: 8
Date: Wed, Aug 25 2021 02:31:10 GMT


jlpoole@taurus /usr/local/src/Live/live/testProgs $

@scottlamb
Copy link
Owner Author

It looks like live555/openRTSP is just skipping all the bytes it doesn't understand. The frame before will be corrupt (because it has some duplicated data in place of some fresh data), and it might do weird things as it interprets video/audio data as headers. I'd prefer to avoid that if possible. Not only does it feel gross, but it could mask other problems (including bugs in Retina itself), and it'd be a security problem in some contexts (the equivalent of an HTTP desync attack). I think if we add a workaround for the bad framing, I'd prefer to do one more narrowly focused on this bug. Or just avoid the issue by adding UDP support and recommending UDP with these cameras.

@jlpoolen
Copy link

what about having three modes in Retina? 1) as-is - croaks on certain anomolies as it does now, 2) tries to continue, but logs problem for retrospective analysis, 3) dangerous mode --- imitates ffmpeg & live and does what it can to continue saving MP4 files without error reporting? I like the 2nd mode if it could have a log file that might be reviewed every 24-48 hours that could be processed with a grep command to give statistics such as 453 bad packets, 800 incomplete data, 1300 time aberrations. These, in turn, could provide some profile of a particular camera helping to triage the panoply is issues. Maybe even have a facility for pushing the logs files to a server which digests the entries?

@scottlamb
Copy link
Owner Author

scottlamb commented Aug 25, 2021

I've moved a little in the direction of log-but-continue with eg dadbd44 (which logs about something that used to be a fatal error, respecting enabled log levels), and I'll likely continue to as Retina gets more mature. But it's a fair bit of work to generate excellent logs, especially after things start going downhill. Eg once we're no longer sure we're at a correct packet boundary, every assumption we make from then on is suspect.

A central server is even more work, and I imagine many privacy-conscious users wouldn't be interested in using it, and/or we'd have to be very careful about what information it collects. The most useful thing to me is pcaps, followed by full TCP-level stream contents according to Retina. Either of those would be out of the question. Realistically, this would stay on my list for a very long time before I run out of tasks with lower cost and more certainty of pay-off.

@scottlamb scottlamb mentioned this issue Aug 26, 2021
Closed
scottlamb added a commit that referenced this issue Aug 31, 2021
This was an attempt at #17 that didn't work out. Besides additionally
hitting the framing error, my Reolink test camera never timed out
the old session while the new one was in progress. This behavior is
different than the vanilla 2013.04.06 live555 server, so apparently
Reolink sprinkled in their own brokenness on top. In any case,
the other solutions described in that bug are more likely to work.

There's one narrow case I want to keep working: ignoring RTCP messages
immediately prior to the PLAY response. More recent Reolink cameras
do this. I just have this behavior on all the time now.
@scottlamb scottlamb changed the title support old live555 servers with buggy RTP/TCP (some Reolink models) support live555 servers older than 2017.06.04 (eg some Reolink models), which have buggy RTP/TCP Aug 31, 2021
@scottlamb
Copy link
Owner Author

scottlamb commented Sep 16, 2021

For the Reolink RLC-420-5MP / hardware version IPC_51516M5M: at my request, Reolink built firmware v3.0.0.589_21091583 which advertises a=tool:LIVE555 Streaming Media v2020.08.12. It no longer has the stale TCP session bug, so there's no need to wait 65 seconds after an unclean shutdown to connect again.

The behavior when the TCP window fills is still bad. (I verified the current version of live555 is buggy; Reolink neither added nor fixed this bug.) But this is likely to come up less often now that there's no extra data from stale TCP sessions.

Reolink told me that the RLC-410 / hardware version IPC_3816M is too old and has been discontinued, so they won't be updating its firmware.

@jlpoolen
Copy link

For posterity, I purchased directly from Reolink 2 RLC-410 5mp cameras on 7/14/2021; $76.98 inc. shipping for 2 cameras. They arrived 7/24/21.

Here is the firmware they currently have
Reolink_Client_2021-09-16_12-38-05
:

@jlpoolen
Copy link

I ran ffmpeg to see what the rtsp server version is. Note, you need to include "-loglevel debug " to get line 45 "a=tool:LIVE555 Streaming Media v2013.04.08"

jlpoole@taurus /usr/local/src $ ./ffmpeg1.sh 
Thu Sep 16 08:59:25 PM PDT 2021
Here is the log: /tmp/stderr_Sep_16_205901.log
 1  Commenced Thu Sep 16 08:59:01 PM PDT 2021
 2  Command:
 3     ffmpeg -loglevel debug -fflags nobuffer -i rtsp://moon:[email protected]:554/h264Preview_01_main -t 15 -y /tmp/Sep_16_205901.mp4 
 4     
 5  ffmpeg version 4.4 Copyright (c) 2000-2021 the FFmpeg developers
 6    built with gcc 10.3.0 (Gentoo 10.3.0-r2 p3)
 ...
39  [rtsp @ 0x55692ec55700] SDP:
40  v=0
41  o=- 1631548228552005 1 IN IP4 192.168.1.56
42  s=Session streamed by "preview"
43  i=h264Preview_01_main
44  t=0 0
45  a=tool:LIVE555 Streaming Media v2013.04.08
46  a=type:broadcast
47  a=control:*
48  a=range:npt=0-
49  a=x-qt-text-nam:Session streamed by "preview"
50  a=x-qt-text-inf:h264Preview_01_main
51  m=video 0 RTP/AVP 96
52  c=IN IP4 0.0.0.0
53  b=AS:500
54  a=rtpmap:96 H264/90000
55  a=fmtp:96 packetization-mode=1;profile-level-id=640033;sprop-parameter-sets=Z2QAM6zoAoAPGQ==,aO48sA==
56  a=control:trackID=1
57  m=audio 0 RTP/AVP 97
58  c=IN IP4 0.0.0.0
59  b=AS:256
60  a=rtpmap:97 MPEG4-GENERIC/16000
61  a=fmtp:97 streamtype=5;profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1408; profile=1;
62  a=control:trackID=2
63
64  Failed to parse interval end specification ''
65  [rtsp @ 0x55692ec55700] video codec set to: h264
66  [rtsp @ 0x55692ec55700] RTP Packetization Mode: 1

Conclusion: The change from 2013 to 2020 of the LIVE555 server in the Reolink camera occurred after July 2021.

@scottlamb
Copy link
Owner Author

I think I'm going to call this done. The major improvements here we've made to date are:

  • supporting UDP, which is less buggy on these models.
  • logging a warning when connecting to these models with TCP
  • trying to send a TEARDOWN when ending even a TCP session to these cameras, to avoid triggering the stale session bug
  • recognizing stale sessions (either by failed TEARDOWN or detected by the erroneous packets) and tracking their expiration. Moonfire NVR explicitly waits them out; other callers can do the same.

I haven't added any kind of "resync" to recover from these cameras' buggy behavior after the TCP window fills. Maybe I'll add that yet, but I'm not seeing complaints right now anyway.

@hn
Copy link

hn commented Mar 10, 2022

I would like to kindly point out my findings on Reolink cameras if this is not already known: Reolink RLC-410-5MP IP camera firmware unpacker and reverse engineered technical details. Maybe someone is able to patch the camera Linux system so that the Live555 server is updated (... or even replaced with a completely different streaming-server).

@jlpoolen
Copy link

jlpoolen commented Mar 10, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
interop interoperability problem with another (often broken) RTSP implementation
Projects
None yet
Development

No branches or pull requests

3 participants