Live: Source cleanup to free memory for multiple streams.

[szhuangzan]

SrsSource* source = SrsSource::fetch(req);
if (!source) {
    if ((ret = SrsSource::create(req, server, server, &source)) != ERROR_SUCCESS) {
        return ret;
    }
}

If a server is continuously requested with rtmp://xxx/live/(1-100000), the number of objects keeps increasing, and no release logic is found.

Related issues:

• Source Cleanup: When there is a large amount of streaming, Source leakage causes OOM (Out of Memory). #1509
• Live: Source cleanup to free memory for multiple streams. #413
• Concurrent access, memory leak #857
• After stopping the stress test, the CPU and memory have remained consistently high. #1507
• srs occasional crash caused by cleaning up unused SrsSource mechanism #713
• 3.0 release has a memory leak, it has been identified, please merge. #2606

[winlinvip]

That's right, this overhead can be calculated.

Note: There is a workaround, Gracefully Quit, which can be considered as a more mature solution.

[willkk]

Same question, if SRS keeps running, the number of sources will keep increasing, when will they be released?

[winlinvip]

9900 streams only occupy 65MB of memory.

Note: There is a workaround, Gracefully Quit, which can be considered as a more mature solution.

[winlinvip]

The situation of Source cleanup is as follows:

• In SRS2, an attempt was made to clean up the Source (reference: c7b97aa1c3), but there were too many problems, so this feature was later disabled: srs occasional crash caused by cleaning up unused SrsSource mechanism #713, SOURCE: The SRS server crashed after stopping the stream. #671
• When supporting K8S, SRS supports Gracefully Quit, which can safely exit when there is no traffic and then be pulled up by the watchdog (there are related Linux solutions). This solution can temporarily avoid this problem.

Therefore, I still need to restart this Issue to support a more comprehensive Source cleanup capability.

[winlinvip]

Source cleanup will be resolved in SRS 5.0, mainly in monitoring and RTC scenarios, where there will be a large number of streams (Sources).

Since the Source pointer is widely referenced, after a long period of consideration, the best solution is to refer to C++11's smart pointers to avoid crashes after cleanup.

Note: There is a workaround, Gracefully Quit, which can be considered as a more mature solution.

[winlinvip]

Pushing a stream every 5 seconds, the memory will increase to 4.2GB in about three days. In scenarios with a large number of push-pull streams, such as stress testing, monitoring, and calling, this problem is quite serious.

Some optimizations are planned for 5.0, without deleting Source, to do some cleanup and slow down the memory growth. The plan is to delete Source in 6.0 and completely solve this problem.

[winlinvip]

Synchronize the progress: It is confirmed that WebRTC's own implementation of ShraredPtr and WeakPtr smart pointers will be referenced, but a simpler version will be implemented and used only in Source objects.

It will be done in two steps to avoid affecting stability: it will be implemented and partially cleaned up in 5.0, and it is expected to be completely resolved in 6.0.

[jinleileiking]

Another approach is to refactor with Rust, but the implementation might be challenging.

[cxw620]

It's impossible unless all core maintainers make up their mind to do so, since SRS is too complicated and used by may many companies now, which means stabilization is of most importance.
Of course you can just implement a simple SRS server with Rust by yourself if you were familiar with C++, video or audio codecs, rtmp, webrtc, and more, which is really a good chance learning them, especially SRS itself.

[winlinvip]

Consider creating a streaming server in Rust instead of having someone else rewrite 200,000 lines of C++ code. When Go was popular, there were suggestions to rewrite the SRS codebase in Go. Now that Rust has become trendy, it's natural that there are suggestions to rewrite it in Rust. This seems to be a recurring event that happens every so often.

TRANS_BY_GPT4

[winlinvip]

Before any work addressing this issue, we need refine the shared ptr of SRS, which is used in RTMP shared message, GB sessions, and in future source objects.

Refer to Using C++11’s Smart Pointers, there are three varieties of smart pointers:

• Shared ptr: Akin to SrsSharedPtrMessage used for RTMP message, and SrsLazyObjectWrapper for GB28181.
• Weak ptr: No such ptr in SRS before 5.0 version. We can avoid this by removing any circle reference of pointer.
• Unique ptr: The corresponding ptr is SrsAutoFree that free the ptr in the local scope.

If we carefully design and limit the usage of smart ptr, such as no cycle link that means we do not need weak ptr, we can archive a simple and easy to maintain smart ptr for SRS. We also consider bellow features:

• Inheritance: Do not use inheritance with smart pointer, although we can write correct code, but it's complex to discuss.
• Comparing: Do not support comparing smart pointers, because we only need to manage the memory automatically.
• make_shared: Do not support this helper function, because we do not consider performance issue by new operator.
• shared_from_this: Do not support this helper template, because we do not need this feature.
• make_unique: Do not support this, because we allow naked new operator, and carefully code review is required.

Again, we don't want to switch to C++11 because I think the modern C++ 11/14/17/20 is really a horrible thing that does not fix anything, but introduces a lot of bad grammar sugar. We do not want any grammar sugar, which is very harmful for communication. We have experienced a lot of disaster of smart pointers and sugars, so we want to keep the usage very simple:

1. No grammar sugar: No weak ptr, no make shared, no make unique, no inheritance, no comparing, no circle reference, no C++11/14/17/20/22 etc.
2. Follow the C++ standard API if any API is really required, do not invent new wheels, no new functions and names.
3. Carefully code review, programmer should take responsibility for code and memory issues.

In the other hand, we should not reinvent wheels especially for smart ptr, so we must use almost the same API of C++ standard library. That is why we should learn from C++11 and refactor the similar smart ptrs in SRS.

Smart pointers might seem simple when they’re just about memory management, but when you delve into inheritance, copying, comparison, performance, circular references, pointer conversion, creation, and the purism of "no naked new", their capabilities become significantly richer. In reality, the main issues with C++ programs stem from programmers messing around, and this purism along with various syntactic sugar makes it hard to notice when they are.

Most problems can be addressed with something as straightforward as an AK47, yet there's an obsession with acquiring more advanced weaponry, which leads to a situation where, on the battlefield, you can’t even fire a shot due to jams and blowbacks. This is because it’s very hard to spot when programmers are messing about in companies, with large amounts of code being submitted, the usage of various new features, and tight deadlines.

The true gates of hell in C++ are not the limited old features of C, but its plethora of new features.

Change log:

• Use shared ptr and used in GB28181
• Use shared ptr in RTC TCP connection
• SmartPtr: Support shared ptr for SRT source.
• SmartPtr: Support shared ptr for RTC source.
• SmartPtr: Support shared ptr for live source.
• SmartPtr: Support load test for source by srs-bench
• SmartPtr: Fix SRT source memory leaking.
• UniquePtr: Support SrsUniquePtr to replace SrsAutoFree.

To verify this issue, start 1k clients over and over, note that we sleep for a while for SRS to cleanup sources:

cd srs/trunk/3rdparty/srs-bench
make
for ((i=0;;i++)); do 
  ./objs/srs_bench -sfu=live -pr=rtmp://localhost/live${i}/stream -sn=1000 -cap=true; 
  sleep 10; 
done

Note: We support benchmark RTMP, SRT, WHIP protocols.

Before v5.0.213, the RES of memory continue to growing:

docker run --name srs --rm -it -p 1935:1935 ossrs/srs:v5.0.205

Check the RES by top, which is the actually used memory:

docker exec -it srs top
#  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                          
#    1 root      20   0 2696388 273144  10624 S  24.7   3.4   0:35.64 srs 

The memory should never continue growing for v5.0.213 or later version.

TRANS_BY_GPT4

[winlinvip]

Fixed by SmartPtr: Support shared ptr for live source. v6.0.129