DVR: Segments are not stored according to dvr_duration. #2717

sudolee · 2021-11-05T02:22:36Z

Description'

Please ensure that you maintain the markdown structure.

The dvr segment was not stored in segments according to the dvr_duration.

SRS version (Version): 4.0.177
The log of SRS is as follows (Log): When dvr_wait_keyframe is enabled, there is no segmentation and no abnormal log. When dvr_wait_keyframe is disabled, there is segmentation, but the segmentation is incorrect. The log is as follows:

[2021-11-05 01:58:13.180][Trace][595][18b245c7] new source, stream_url=/live/test
[2021-11-05 01:58:13.180][Trace][595][18b245c7] source url=/live/test, ip=172.17.0.1, cache=0, is_edge=0, source_id=/
[2021-11-05 01:58:13.182][Trace][595][18b245c7] new source, stream_url=/live/test
[2021-11-05 01:58:13.185][Trace][595][18b245c7] RTC bridge from RTMP, rtmp2rtc=1, keep_bframe=0, merge_nalus=0
[2021-11-05 01:58:13.185][Trace][595][18b245c7] dvr stream test to file ./objs/nginx/html/data/__defaultVhost__/live/test/2021/11/05/2021.11.05.01.58.13.flv
[2021-11-05 01:58:13.185][Trace][595][18b245c7] ignore disabled exec for vhost=__defaultVhost__
[2021-11-05 01:58:13.185][Trace][595][18b245c7] http: mount flv stream for sid=/live/test, mount=/live/test.flv
[2021-11-05 01:58:13.185][Trace][595][18b245c7] set fd=12 TCP_NODELAY 0=>1
[2021-11-05 01:58:13.185][Trace][595][18b245c7] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=1
[2021-11-05 01:58:13.841][Trace][595][18b245c7] got metadata, width=1024, height=556, vcodec=2, acodec=2
[2021-11-05 01:58:14.129][Trace][595][91s7ox64] Hybrid cpu=1.00%,13MB
[2021-11-05 01:58:19.133][Trace][595][91s7ox64] Hybrid cpu=2.00%,14MB, cid=2,1, timer=58,0,0, clock=0,13,27,3,0,0,0,0,0, objs=(pkt:0,raw:0,fua:0,msg:35,oth:0,buf:0)
[2021-11-05 01:58:23.804][Trace][595][18b245c7] dvr stream test to file ./objs/nginx/html/data/__defaultVhost__/live/test/2021/11/05/2021.11.05.01.58.23.flv
[2021-11-05 01:58:23.804][Warn][595][18b245c7][2] dvr: ignore audio error code=3082 : update duration : request sh : dvr audio : update duration : segment open : DVR can't append to exists path=./objs/nginx/html/data/__defaultVhost__/live/test/2021/11/05/2021.11.05.01.58.23.flv
thread [595][18b245c7]: on_audio() [src/app/srs_app_dvr.cpp:816][errno=2]
thread [595][18b245c7]: update_duration() [src/app/srs_app_dvr.cpp:880][errno=2]
thread [595][18b245c7]: on_dvr_request_sh() [src/app/srs_app_source.cpp:1224][errno=2]
thread [595][18b245c7]: on_audio() [src/app/srs_app_dvr.cpp:816][errno=2]
thread [595][18b245c7]: update_duration() [src/app/srs_app_dvr.cpp:875][errno=2]
thread [595][18b245c7]: open() [src/app/srs_app_dvr.cpp:79][errno=2]
[2021-11-05 01:58:23.806][Warn][595][18b245c7][11] dvr: ignore audio error code=3082 : update duration : segment open : DVR can't append to exists path=./objs/nginx/html/data/__defaultVhost__/live/test/2021/11/05/2021.11.05.01.58.23.flv
thread [595][18b245c7]: on_audio() [src/app/srs_app_dvr.cpp:816][errno=11]
thread [595][18b245c7]: update_duration() [src/app/srs_app_dvr.cpp:875][errno=11]
thread [595][18b245c7]: open() [src/app/srs_app_dvr.cpp:79][errno=11]
[2021-11-05 01:58:23.808][Warn][595][18b245c7][11] dvr: ignore audio error code=3082 : update duration : segment open : DVR can't append to exists path=./objs/nginx/html/data/__defaultVhost__/live/test/2021/11/05/2021.11.05.01.58.23.flv
thread [595][18b245c7]: on_audio() [src/app/srs_app_dvr.cpp:816][errno=11]
thread [595][18b245c7]: update_duration() [src/app/srs_app_dvr.cpp:875][errno=11]
thread [595][18b245c7]: open() [src/app/srs_app_dvr.cpp:79][errno=11]
[2021-11-05 01:58:23.809][Warn][595][18b245c7][11] dvr: ignore audio error code=3082 : update duration : segment open : DVR can't append to exists path=./objs/nginx/html/data/__defaultVhost__/live/test/2021/11/05/2021.11.05.01.58.23.flv
thread [595][18b245c7]: on_audio() [src/app/srs_app_dvr.cpp:816][errno=11]
thread [595][18b245c7]: update_duration() [src/app/srs_app_dvr.cpp:875][errno=11]
thread [595][18b245c7]: open() [src/app/srs_app_dvr.cpp:79][errno=11]
[2021-11-05 01:58:23.944][Warn][595][18b245c7][11] dvr: ignore audio error code=3082 : update duration : segment open : DVR can't append to exists path=./objs/nginx/html/data/__defaultVhost__/live/test/2021/11/05/2021.11.05.01.58.23.flv

The configuration of SRS is as follows (Config):

listen              1935;
max_connections     1000;
daemon              on;
srs_log_tank        console;

http_server {
    enabled         on;
    listen          8080;
    dir             ./objs/nginx/html;
}

http_api {
    enabled         on;
    listen          1985;
}
stats {
    network         0;
}

rtc_server {
    enabled on;
    listen 8000;
    # @see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#config-candidate
    #candidate $CANDIDATE;
    candidate 127.0.0.1;
}

vhost __defaultVhost__ {
    rtc {
        enabled     on;
        # @see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#rtmp-to-rtc
        rtmp_to_rtc on;
        # @see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#rtc-to-rtmp
        rtc_to_rtmp off;
    }
    http_remux {
        enabled     on;
        mount       [vhost]/[app]/[stream].flv;
    }
    dvr {
        enabled         on;
        dvr_apply       all;
        dvr_plan        segment;
        dvr_path ./objs/nginx/html/[app]/[stream].[timestamp].flv;
        dvr_duration    10;
        dvr_wait_keyframe       on;
        time_jitter             full;
    }
}

Replay

How to replay bug?

Steps to reproduce the bug

Steps to reproduce the bug:

Run ./objs/srs -c myconf.conf in docker.
Push the stream from the docker host using ffmpeg -re -i ./1024x556.rmvb -y -f flv rtmp://127.0.0.1/live/test.
Check that the corresponding directory does not store segmented files according to dvr_duration (10 seconds).
Additionally, when dvr_wait_keyframe is disabled, there will be issues starting from the second segment.

Expected behavior:

Make sure to maintain the markdown structure.

> Store recorded videos in segments according to dvr_duration (10 seconds).

TRANS_BY_GPT3

The text was updated successfully, but these errors were encountered:

winlinvip · 2021-11-08T23:39:25Z

What codec is this?

got metadata, width=1024, height=556, vcodec=2, acodec=2

Is it related to your codec? Can you reproduce this issue by changing it to doc/source.flv?

TRANS_BY_GPT3

chundonglinlin · 2021-12-31T14:21:53Z

There is a problem with the streaming command: ffmpeg -re -i ./1024x556.rmvb -y -f flv rtmp://127.0.0.1/live/test
If ffmpeg does not specify a codec or use the copy option, it will default to streaming with flv1 video codec and mp3 audio codec.

TRANS_BY_GPT3

chundonglinlin · 2021-12-31T14:43:39Z

Configuration 1: dvr_wait_keyframe on;
Reason for not segmenting: When vcodec=2, which means the codec is H263, if dvr_wait_keyframe is enabled, it will keep waiting for a keyframe of H264. Otherwise, it will not segment and keep writing to the same file continuously. You can refer to this link for more information:

srs/trunk/src/app/srs_app_dvr.cpp

Line 866 in 8bc2759

    
           bool is_key_frame = SrsFlvVideo::h264(payload, size) && SrsFlvVideo::keyframe(payload, size) && !SrsFlvVideo::sh(payload, size);

Configuration 2: dvr_wait_keyframe off;
Reason for incorrect segmentation: This is unrelated to the codec, whether it is H263 or H264. When the first FLV file is closed, and the second FLV file is opened, the duration is not reset to zero. It remains the same as the duration of the previous FLV file. As a result, the newly opened file always meets the segmentation condition.

srs/trunk/src/app/srs_app_dvr.cpp

Line 853 in 8bc2759

if (cduration <= 0 || fragment->duration() < cduration) {

At this point, it will enter into an infinite loop logic, resulting in a Segmentation fault error when writing the DVR file.

TRANS_BY_GPT3

chundonglinlin · 2021-12-31T17:37:24Z

If streaming is done according to the following configuration, it will enter a DVR infinite loop. The main reason is that when reopening a fragment, the duration of the fragment is not reset to zero.

vhost __defaultVhost__ {
    dvr {
        enabled         on;
        dvr_apply       all;
        dvr_plan        segment;
        dvr_path ./objs/nginx/html/[app]/[stream].[timestamp].flv;
        dvr_duration    10;
        dvr_wait_keyframe       off;
        time_jitter             full;
    }
}

Streaming commands:
H264+AAC: ffmpeg -re -i source.200kbps.768x320.flv -c copy -y -f flv rtmp://127.0.0.1/live/stream
H263+MP3: ffmpeg -re -i source.200kbps.768x320.flv -y -f flv rtmp://127.0.0.1/live/stream

PS: Configuration for causing an infinite loop is guaranteed to occur.
dvr_plan segment; dvr_wait_keyframe off;

TRANS_BY_GPT3

chundonglinlin · 2021-12-31T18:21:21Z

#2836

winlinvip · 2022-01-01T04:11:45Z

Minimum reproduction path, setting dvr_wait_keyframe off; will cause problems in the second DVR file.

vhost __defaultVhost__ {
    dvr {
        enabled         on;
        dvr_plan        segment;
        dvr_duration    10;
        dvr_wait_keyframe       off;
    }
    ingest livestream {
        enabled      on;
        input {
            url     ./doc/source.flv;
        }
        ffmpeg      ./objs/ffmpeg/bin/ffmpeg;
        engine {
            output          rtmp://127.0.0.1:[port]/live?vhost=[vhost]/livestream;
        }
    }
}

Around 10 seconds, the second slice starts to refresh the screen, continuously generating new DVR files.

[2022-01-01 12:11:09.130][Trace][45946][wh3t8u2q] dvr stream livestream to file ./objs/nginx/html/live/livestream.1641010269130.flv
[2022-01-01 12:11:09.130][Warn][45946][wh3t8u2q][2] dvr: ignore video error code=3082 : update duration : request sh : dvr video : update duration : segment open : DVR can't append to exists path=./objs/nginx/html/live/livestream.1641010269130.flv
thread [45946][wh3t8u2q]: on_video() [src/app/srs_app_dvr.cpp:835][errno=2]
thread [45946][wh3t8u2q]: update_duration() [src/app/srs_app_dvr.cpp:884][errno=2]
thread [45946][wh3t8u2q]: on_dvr_request_sh() [src/app/srs_app_source.cpp:1222][errno=2]
thread [45946][wh3t8u2q]: on_video() [src/app/srs_app_dvr.cpp:835][errno=2]
thread [45946][wh3t8u2q]: update_duration() [src/app/srs_app_dvr.cpp:879][errno=2]
thread [45946][wh3t8u2q]: open() [src/app/srs_app_dvr.cpp:79][errno=2]

[2022-01-01 12:11:09.167][Trace][45946][wh3t8u2q] dvr stream livestream to file ./objs/nginx/html/live/livestream.1641010269167.flv
[2022-01-01 12:11:09.167][Warn][45946][wh3t8u2q][2] dvr: ignore audio error code=3082 : update duration : request sh : dvr video : update duration : segment open : DVR can't append to exists path=./objs/nginx/html/live/livestream.1641010269167.flv
thread [45946][wh3t8u2q]: on_audio() [src/app/srs_app_dvr.cpp:820][errno=2]
thread [45946][wh3t8u2q]: update_duration() [src/app/srs_app_dvr.cpp:884][errno=2]
thread [45946][wh3t8u2q]: on_dvr_request_sh() [src/app/srs_app_source.cpp:1222][errno=2]
thread [45946][wh3t8u2q]: on_video() [src/app/srs_app_dvr.cpp:835][errno=2]
thread [45946][wh3t8u2q]: update_duration() [src/app/srs_app_dvr.cpp:879][errno=2]
thread [45946][wh3t8u2q]: open() [src/app/srs_app_dvr.cpp:79][errno=2]

[2022-01-01 12:11:09.167][Warn][45946][wh3t8u2q][2] dvr: ignore audio error code=3082 : update duration : segment open : DVR can't append to exists path=./objs/nginx/html/live/livestream.1641010269167.flv
thread [45946][wh3t8u2q]: on_audio() [src/app/srs_app_dvr.cpp:820][errno=2]
thread [45946][wh3t8u2q]: update_duration() [src/app/srs_app_dvr.cpp:879][errno=2]
thread [45946][wh3t8u2q]: open() [src/app/srs_app_dvr.cpp:79][errno=2]

Root Cause

Debugging revealed that update_duration called itself, causing infinite recursion and exhausting the stack, resulting in a crash. Please refer to the following diagram for the specific call stack and function references:

update_duration is responsible for updating the duration of a slice. It has been observed that if the duration exceeds the configured limit, for example, after 10 seconds, the DVR file will be closed and a new DVR file will be created.

Of course, new DVR files will be continuously generated. Except for the first DVR file, the rest will only contain sequence header and metadata if there is no content.

There is an assumption here that is problematic, which is the recursion of update_duration -> close -> open -> on_request_sh -> ... -> update_duration. It is incorrect to trigger update_duration again when on_request_sh requests the sequence header because although audio and video packets are received, they only consist of the sequence header and metadata, and the entire segment does not need to update the duration.

Why Works for Wait Keyframe

Why does it work fine when waiting for a keyframe? The debugging is shown in the following image:

It can be seen that although update_duration will call on_request_sh and recursively call itself update_duration, the sequence header is considered as a non-sequencer header, so it is considered that the DVR file cannot be closed yet.

The subsequent duration calculation is correct, which is 0, so there is no problem. Therefore, there is an issue here, which is that the duration calculation is incorrect, especially when on_request_sh is recursively called, the duration of the slice is incorrect. It needs to be examined in detail.

`on_request_sh` will call the following function.

SrsDvrSegmentPlan::on_video(SrsSharedPtrMessage* shared_video, SrsFormat* format)
{

// When waitkeyframe=on, if the condition is not met, it returns. If waitkeyframe=off, it will recursively call indefinitely.

    if ((err = update_duration(shared_video)) != srs_success) {

// Here, the duration of the slice will be recalculated and reset to 0, which is why there is no problem when waitkeyframe=on.

    if ((err = SrsDvrPlan::on_video(shared_video, format)) != srs_success) {

From here, we can see that there are two issues:

1. The calculation of duration is ambiguous, it is not clear which function is responsible for the calculation.

2. The process of resetting duration to 0 is very obscure.

Solution

The root cause of the problem is:

1. on_request_sh should not cause the invocation of update_duration, nor should it close DVR slices.

2. During on_request_sh, the duration calculation becomes abnormal. The calculation of duration is quite obscure, even if it is calculated correctly.

TRANS_BY_GPT3

chundonglinlin · 2022-01-01T16:59:30Z

Test the 3.0 release version, and this problem also exists. The time when this problem was introduced is approximately Thu Feb 9 15:42:42 2017. Compare: 455d9b2...00abaf4

The implementation logic before 3.0 release is as follows:

int SrsDvrSegmentPlan::update_duration(SrsSharedPtrMessage* msg)
{
    // update sequence header
    if (metadata && (ret = SrsDvrPlan::on_meta_data(metadata)) != ERROR_SUCCESS) {
        return ret;
    }
    if (sh_video && (ret = SrsDvrPlan::on_video(sh_video)) != ERROR_SUCCESS) {
        return ret;
    }
    if (sh_audio && (ret = SrsDvrPlan::on_audio(sh_audio)) != ERROR_SUCCESS) {
        return ret;
    }

There will be no recursive call to update_duration when on_request_sh is called with update_duration. Instead, SrsDvrPlan::on_meta_data, SrsDvrPlan::on_audio, and SrsDvrPlan::on_video will be directly called, followed by on_update_duration to update the segment duration dur. This will skip SrsDvrSegmentPlan::update_duration and directly write the segment.

TRANS_BY_GPT3

chundonglinlin · 2022-01-01T17:03:50Z

Thank you, Boss Yang, for your guidance. The analysis is very clear. Currently, I have thought of several repair methods:

Method 1: Clean up fragment duration when open/close;
Method 2: Exclude sh from duration calculation when updating duration;
Method 3: Add a variable flag in on_request_sh to block the call to update_duration;
Method 4: Refer to the handling logic in the 2.0 release.

TRANS_BY_GPT3

winlinvip · 2022-01-05T00:30:28Z

Solving the infinite recursion problem: 4d09b8c

After trying it out, the problem of continuous slicing no longer occurs, and it has been resolved. However, the duration issue has not been completely resolved and still poses a potential risk.

TRANS_BY_GPT3

zhangzhenglava · 2022-03-19T07:07:41Z

Using Weibo Vision 5G terminal to push stream to SRS 4.0.249, if RTC is configured, the same server error code=3082 error code will appear. The problem exhibits the same symptoms; commenting out the RTC configuration restores normal operation.

Terminal settings:

SRS 4.0 configuration:
listen 1949;
max_connections 1000;
srs_log_tank file;
srs_log_file ./objs/srs.log;
daemon on;
http_api {
enabled on;
listen 1985;
raw_api {
enabled on;
allow_reload on;
allow_query on;
allow_update on;
}
}
stats {
network 0;
disk sda sdb xvda xvdb;
}
#rtc_server {

enabled on;

listen 8000; # UDP port

candidate $CANDIDATE;

#}
http_server {
enabled on;
listen 8817;
dir ./objs/nginx/html;
}
vhost defaultVhost {

rtc {

#    enabled     on;
 #   rtmp_to_rtc on;
 #   rtc_to_rtmp on;
#}
tcp_nodelay on min_latency on;
play {
    gop_cache off;
    queue_length 10;
    mw_latency 100;
}
publish {
    mr off;
    mr_latency 350;
}
hls {
    enabled on;
    hls_fragment 0.2;
    hls_window 2;
    hls_wait_keyframe off;
}
http_remux {
    enabled on;
    mount [vhost]/[app]/[stream].flv;
}
dvr {
    enabled on;
    dvr_path /usr/sdb/srs-video/[app]/[stream]/[2006][01][02]/[2006][01][02][15][04].mp4;
    dvr_plan session;
    dvr_wait_keyframe on;
    time_jitter full;
    dvr_apply all;
}
http_hooks {
    enabled on;
    on_dvr http://localhost:8087/srs/receive;
    on_connect http://localhost:8087/srs/receive;
    on_close http://localhost:8087/srs/receive;
    on_publish http://localhost:8087/srs/receive;
    on_unpublish http://localhost:8087/srs/receive;
}

}
Console error message:
[2022-03-17 16:39:50.797][Error][22038][w9232kw0][0] serve error code=3082 : service cycle : rtmp: stream service : hub publish : dvr publish : publish : open segment : DVR can't append to exists path=/usr/sdb/srs-video/ry_live/M300002/20220317/202203171639.mp4
thread [22038][w9232kw0]: do_cycle() [src/app/srs_app_rtmp_conn.cpp:217][errno=0]
thread [22038][w9232kw0]: service_cycle() [src/app/srs_app_rtmp_conn.cpp:414][errno=0]
thread [22038][w9232kw0]: on_publish() [src/app/srs_app_source.cpp:2507][errno=0]
thread [22038][w9232kw0]: on_publish() [src/app/srs_app_source.cpp:1143][errno=0]
thread [22038][w9232kw0]: on_publish() [src/app/srs_app_dvr.cpp:973][errno=0]
thread [22038][w9232kw0]: on_publish() [src/app/srs_app_dvr.cpp:716][errno=0]
thread [22038][w9232kw0]: open() [src/app/srs_app_dvr.cpp:79][errno=0]
DVR recording:

TRANS_BY_GPT3

merged code according to issue 2717 link: ossrs/srs#2717

winlinvip · 2023-01-02T13:00:44Z

Should be fixed in SRS 4

winlinvip changed the title ~~dvr segment没有按照dvr_duration分段存储~~ DVR: Segment没有按照dvr_duration分段存储 Nov 8, 2021

winlinvip self-assigned this Nov 8, 2021

chundonglinlin added a commit to chundonglinlin/srs that referenced this issue Dec 31, 2021

DVR: fix loop fault when configure dvr_plan segment.(ossrs#2717)

8ba4e8c

winlinvip added the Bug It might be a bug. label Jan 1, 2022

winlinvip added this to the 4.0 milestone Jan 1, 2022

winlinvip linked a pull request Jan 1, 2022 that will close this issue

DVR: fix infinite recursion when configure dvr_plan segment.(#2717) #2836

Closed

chundonglinlin added a commit to chundonglinlin/srs that referenced this issue Jan 4, 2022

DVR: fix loop fault when configure dvr_plan segment.(ossrs#2717)

4562986

chundonglinlin added a commit to chundonglinlin/srs that referenced this issue Jan 4, 2022

DVR: fix infinite recursion when configure dvr_plan segment.(ossrs#2717)

3f35307

winlinvip added a commit that referenced this issue Jan 5, 2022

When reopening segment, never update the duration. (#2717). v4.0.219

4d09b8c

tyisme614 added a commit to tyisme614/srs-gb28181 that referenced this issue Jun 6, 2022

merged code

ea42aa3

merged code according to issue 2717 link: ossrs/srs#2717

winlinvip modified the milestones: 4.0, 5.0 Dec 9, 2022

winlinvip closed this as completed Jan 2, 2023

winlinvip modified the milestones: 5.0, 4.0 Jan 2, 2023

winlinvip changed the title ~~DVR: Segment没有按照dvr_duration分段存储~~ DVR: Segments are not stored according to dvr_duration. Jul 27, 2023

winlinvip added the TransByAI Translated by AI/GPT. label Jul 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DVR: Segments are not stored according to dvr_duration. #2717

DVR: Segments are not stored according to dvr_duration. #2717

sudolee commented Nov 5, 2021 •

edited by winlinvip

Loading

winlinvip commented Nov 8, 2021 •

edited

Loading

chundonglinlin commented Dec 31, 2021 •

edited by winlinvip

Loading

chundonglinlin commented Dec 31, 2021 •

edited by winlinvip

Loading

chundonglinlin commented Dec 31, 2021 •

edited by winlinvip

Loading

chundonglinlin commented Dec 31, 2021

winlinvip commented Jan 1, 2022 •

edited

Loading

chundonglinlin commented Jan 1, 2022 •

edited by winlinvip

Loading

chundonglinlin commented Jan 1, 2022 •

edited by winlinvip

Loading

winlinvip commented Jan 5, 2022 •

edited

Loading

zhangzhenglava commented Mar 19, 2022 •

edited by winlinvip

Loading

winlinvip commented Jan 2, 2023

DVR: Segments are not stored according to dvr_duration. #2717

DVR: Segments are not stored according to dvr_duration. #2717

Comments

sudolee commented Nov 5, 2021 • edited by winlinvip Loading

winlinvip commented Nov 8, 2021 • edited Loading

chundonglinlin commented Dec 31, 2021 • edited by winlinvip Loading

chundonglinlin commented Dec 31, 2021 • edited by winlinvip Loading

chundonglinlin commented Dec 31, 2021 • edited by winlinvip Loading

chundonglinlin commented Dec 31, 2021

winlinvip commented Jan 1, 2022 • edited Loading

Root Cause

Why Works for Wait Keyframe

Solution

chundonglinlin commented Jan 1, 2022 • edited by winlinvip Loading

chundonglinlin commented Jan 1, 2022 • edited by winlinvip Loading

winlinvip commented Jan 5, 2022 • edited Loading

zhangzhenglava commented Mar 19, 2022 • edited by winlinvip Loading

enabled on;

listen 8000; # UDP port

candidate $CANDIDATE;

rtc {

winlinvip commented Jan 2, 2023

sudolee commented Nov 5, 2021 •

edited by winlinvip

Loading

winlinvip commented Nov 8, 2021 •

edited

Loading

chundonglinlin commented Dec 31, 2021 •

edited by winlinvip

Loading

chundonglinlin commented Dec 31, 2021 •

edited by winlinvip

Loading

chundonglinlin commented Dec 31, 2021 •

edited by winlinvip

Loading

winlinvip commented Jan 1, 2022 •

edited

Loading

chundonglinlin commented Jan 1, 2022 •

edited by winlinvip

Loading

chundonglinlin commented Jan 1, 2022 •

edited by winlinvip

Loading

winlinvip commented Jan 5, 2022 •

edited

Loading

zhangzhenglava commented Mar 19, 2022 •

edited by winlinvip

Loading