Convert rtmp to HLS and only keep the audio in m3u8 format.

[min372344453]

Transcode RTMP to HLS and only keep the audio in the m3u8 file.

1. SRS version: srs-3.0release
2. The log of SRS is as follows:

[2020-03-16 17:49:38.384][Trace][9547][37161] http match file=./objs/nginx/html/live/a2535790-11ca-4672-ba82-2e7f4a212fc7-21895.ts, pattern=/, upath=/live/a2535790-11ca-4672-ba82-2e7f4a212fc7-21895.ts
[2020-03-16 17:49:38.386][Trace][9547][37161] HTTP client ip=127.0.0.1, request=9, to=15000ms
[2020-03-16 17:49:40.684][Trace][9547][37166] HTTP client ip=127.0.0.1, request=0, to=15000ms
[2020-03-16 17:49:40.684][Trace][9547][37166] HTTP GET http://127.0.0.1:81/live/a2535790-11ca-4672-ba82-2e7f4a212fc7.m3u8, content-length=-1
[2020-03-16 17:49:40.685][Trace][9547][37166] http match file=./objs/nginx/html/live/a2535790-11ca-4672-ba82-2e7f4a212fc7.m3u8, pattern=/, upath=/live/a2535790-11ca-4672-ba82-2e7f4a212fc7.m3u8
[2020-03-16 17:49:40.685][Trace][9547][37166] HTTP client ip=127.0.0.1, request=1, to=15000ms
[2020-03-16 17:49:40.750][Warn][9547][37166][104] client disconnect peer. ret=1007
[2020-03-16 17:49:40.756][Trace][9547][37161] HTTP GET http://127.0.0.1:81/live/a2535790-11ca-4672-ba82-2e7f4a212fc7-21896.ts, content-length=-1
[2020-03-16 17:49:40.756][Trace][9547][37161] http match file=./objs/nginx/html/live/a2535790-11ca-4672-ba82-2e7f4a212fc7-21896.ts, pattern=/, upath=/live/a2535790-11ca-4672-ba82-2e7f4a212fc7-21896.ts
[2020-03-16 17:49:40.758][Trace][9547][37161] HTTP client ip=127.0.0.1, request=10, to=15000ms

1. The configuration of SRS is as follows:

listen              1935;
max_connections     1000;
daemon              no;
srs_log_tank        file;
srs_log_level       trace;
http_server {
    enabled         on;
    listen          81;
    dir             ./objs/nginx/html;
}
vhost __defaultVhost__ {
    tcp_nodelay     on
    min_latency     on;
   http_hooks {
    enabled         on;
   on_publish      http://127.0.0.1:8085/api/v1/streams;
}
  hls {
        enabled         on;
        hls_fragment    2;
        hls_window      10;
        hls_path        ./objs/nginx/html;
        hls_m3u8_file   [app]/[stream].m3u8;
        hls_ts_file     [app]/[stream]-[seq].ts;
        hls_dispose     6;
        hls_cleanup     on;
    }
}

Replay

How to replay bug?

Steps to reproduce the bug

When switching video streaming, an issue occurs.

TRANS_BY_GPT3

[jyy1082]

Encounter the same problem.

TRANS_BY_GPT3

[limjoe]

Encounter the same problem.

TRANS_BY_GPT3

[lihp1603]

The HLS slicing of srs does not guarantee that every slice contains a keyframe, so you need to check whether the locally sliced ts slices can be played after being copied locally. Secondly, you need to check your streaming to see if the source stream file has sps, pps for every IDR of the video, and whether they are all complete. If any of them are missing, it will also not work.

TRANS_BY_GPT3

[winlinvip]

Pushing two different streams will cause the player to freeze and refreshing will not restore it. It is likely a problem with the server's HLS switching.

Reproduction path:

1. Push the avatar stream.
2. Wait for two slices to be output.
3. Switch to a different stream and push.
4. The player displays distorted images, even after refreshing.

TRANS_BY_GPT3

[winlinvip]

Confirm that it is a problem with the encoding header, because even if you manually process the m3u8, it will still take time, only keeping the ts of the new stream.

Below is the original objs/nginx/html/live/livestream.m3u8:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TARGETDURATION:15
#EXT-X-DISCONTINUITY
#EXTINF:14.755, no desc
livestream-0.ts
#EXTINF:11.675, no desc
livestream-1.ts
#EXTINF:7.520, no desc
livestream-2.ts
#EXT-X-DISCONTINUITY
#EXTINF:12.266, no desc
livestream-3.ts
#EXTINF:12.386, no desc
livestream-4.ts
#EXTINF:7.480, no desc
livestream-5.ts

Remove the slices before DISCONTINUITY, but there is still lag when playing with VLC.

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TARGETDURATION:15
#EXTINF:12.266, no desc
livestream-3.ts
#EXTINF:12.386, no desc
livestream-4.ts
#EXTINF:7.480, no desc
livestream-5.ts

It indicates that the issue is not with the m3u8 file, but with the ts file. Even directly playing livestream-3.ts results in lag.

Please refer to the attachment: live.tar.gz

Use EasyICE to check livestream-0.ts, it has SEI packets, let's ignore them for now.

Check livestream-1.ts, it doesn't have SEI, only SPS/PPS+IDR.

Video
ID                               : 256 (0x100)
Menu ID                          : 1 (0x1)
Format                           : AVC
Format/Info                      : Advanced Video Codec
Format profile                   : [email protected]
Format settings, CABAC           : Yes
Format settings, ReFrames        : 4 frames
Format settings, GOP             : M=1, N=16
Duration                         : 11s 640ms
Width                            : 768 pixels
Height                           : 320 pixels
Display aspect ratio             : 2.40:1
Frame rate mode                  : Variable
Color space                      : YUV
Chroma subsampling               : 4:2:0
Bit depth                        : 8 bits
Scan type                        : Progressive

Check livestream-3.ts, it is the first slice of the new stream, and the encoding/decoding information is as follows:

Video
ID                               : 256 (0x100)
Menu ID                          : 1 (0x1)
Format                           : AVC
Format/Info                      : Advanced Video Codec
Format profile                   : [email protected]
Format settings, CABAC           : Yes
Format settings, ReFrames        : 4 frames
Duration                         : 12s 240ms
Width                            : 768 pixels
Height                           : 320 pixels
Display aspect ratio             : 2.40:1
Frame rate mode                  : Variable
Color space                      : YUV
Chroma subsampling               : 4:2:0
Bit depth                        : 8 bits
Scan type                        : Progressive

Clearly, this encoding/decoding header is incorrect. The encoding/decoding information for the new stream is definitely different. In fact, it is still the information from the previous video. The encoding/decoding information for the new stream should be:

Stream #0:0: Video: h264 (High), yuv420p(progressive), 640x360 [SAR 1:1 DAR 16:9], 
250 kb/s, 25 fps, 25 tbr, 1k tbn, 50 tbc

Comparing livestream-1.ts and livestream-3.ts, it can be seen that their SPS (Sequence Parameter Set) is the same, indicating that there was no update:

TRANS_BY_GPT3

[winlinvip]

After investigation, it was found that there was a bug introduced. Previously, in order to solve the issue of multiple SPS, only the first SPS was saved, and the subsequent SPS were discarded.

However, this aspect did not take into account the situation where SPS could change. There are two formats that will parse SPS/PPS:

source:SrsLiveSource -> meta:SrsMetaCache -> vformat:SrsRtmpFormat
source:SrsLiveSource -> hub:SrsOriginHub -> format:SrsRtmpFormat

Both of these formats store the parsed codec information, one in the cache and the other in each video packet.

To see where the parsed SPS/PPS is used, one is in RTMP to RTC conversion, and the other is in HLS.

srs_error_t SrsRtcFromRtmpBridger::package_stap_a(SrsRtcSource* source, SrsSharedPtrMessage* msg, SrsRtpPacket* pkt) {
    const vector<char>& sps = format->vcodec->sequenceParameterSetNALUnit;
    const vector<char>& pps = format->vcodec->pictureParameterSetNALUnit;


srs_error_t SrsTsMessageCache::do_cache_avc(SrsVideoFrame* frame) {
                video->payload->append(&codec->pictureParameterSetNALUnit[0], (int)codec->pictureParameterSetNALUnit.size());

Therefore, after re-pushing RTMP, there are issues with HLS/RTMP2RTC because it uses the SPS/PPS information. RTC/RTMP/FLV have no issues because they do not use this encoding/decoding information, and directly forward the updated SPS/PPS packets.

RTMP also experiences screen flickering because it does not support updating SPS/PPS, which is different from the screen flickering issue in HLS/RTMP2RTC. RTMP2RTC refers to the conversion of RTMP to RTC.

TRANS_BY_GPT3