majestic segfaults after some time on hi3518ev100 + ov9712 #171
Labels
low priority
Not considered without funding
Comments
|
I am seeing the same thing. The watchdog seems to be broken on other Chipsets aswell, looks like it gets disabled on crash, |
|
The watchdog has been effectively disabled on all HiSilicon platforms. If you want to enable it again, re-add the |
|
I got access to a device for testing (HI3518CV100+IMX138), but unfortunately could not reproduce the rtsp crash. |
|
Try leaving it running for longer. I am seeing this on and the camera has an uptime of about 20 hours now. So it does happen eventually, but it is not really plannable. |
|
anyone from majestic dev team get a chance to analyze the coredump? with closed source and no symbols, it would be exceedingly hard for us to guess what is going on, and possibly theorize ways to reproduce it more reliably |
|
The core dump is unfortunately missing some function pointer. |
majestic.2440.11.1723254891.core.gz also updated majestic.yaml (as I was disabling second video1 stream) |
|
For me it hasn't crashed yet since 32 hours, however I've seen some majestic versions work better than others in the past, so this could be coincidence. I'll give you an update in a few days how it's going. |
|
It looks like it's fixed on hi3516ev300 but my hi3518ev100 keeps crashing. |
|
The buffer size has now been increased to |
|
Unfortunately I'm still seeing crashes, now also on hi3516ev300 again. I'm running master+11648f1, 2024-08-14. |
|
@Flole998 same here w/ 8-13 build of majestic Curious: I'm using Blue Iris 5. What rtsp streamer are you using? |
|
also @skilurius thanks for looking into this. Not sure if it's helpful but here is some data from OEM firmware on my device, which has been up for over 19 days streaming rtsp without issue. on my majestic test device (same hardware): |
|
ok and here is one after majestic crashed after just about 20 minutes of running stream: logread: |
|
It is a strange issue and difficult to repeatedly reproduce (it takes around 40-80min on my test device). |
|
If we can't really find the source of the issue, we may just need to live with frequently crashing majestic. In which case we could either try to just have a supervisor process restart majestic, or maybe something more simple, like letting /etc/init.d/S95majestic retry restarting majestic in some loop after exit with non-zero status (i.e. after crash). But this would need some sane retry/backoff limit, like 5 or 10 times - after which, we need the hardware watchdog to kick in and reboot the device. The theory is if majestic got stuck in a bad state, i.e. due to some hardware/registers/etc corruption - we would not want to try to infinitely restart majestic, but at some point backoff and just let wdt timer expire and reset the cpu. But I think this brings us back to a secondary concern, which is that currently, the wdt does not work for some reason, but as @Flole998 pointed out, we need to uncomment I'm hoping this is a common problem (wdt not working) with all hi3516cv100, such that submitting a PR to fix this will work for everyone with this cpu family. |
zerog2k
added a commit
to zerog2k/openipc-firmware
that referenced
this issue
Aug 22, 2024
per wdt.ko: ``` parm=nodeamon:By default, a kernel deamon feed watchdog when idle, set 'nodeamon=1' to disable this. (default=0) parmtype=nodeamon:int ``` as documented in OpenIPC/majestic#171, when majestic crashes, without `nodeamon=1` param to wdt kmod, camera does not automatically reboot. Expected behavior is for majestic to feed the watchdog, and if majestic crashes, watchdog timer would expire and reset cpu, forcing host reboot.
This was referenced Sep 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
make BOARD=hi3518ev100_lite) + sysupgradeIssue description
Majestic crashes (segfault) after some time (minutes to hours) streaming rtsp h264. (No crashes if streaming only mjpeg.)
Another observation is that when majestic dies, even though watchdog is enabled, the camera never resets. OS stays up and available but majestic is gone.
(I'm certain watchdog was working properly on majestic/openipc versions from at least as recent as February. Majestic did crash, but I believe watchdog was keeping this from being a terminal issue.)
might be similar to #159
Expectations
no crash
Steps to reproduce
rtsp://CAMERAIP/?stream=0Information to attach
majestic.4731.11.1720275013.core.zip
majestic.yaml.zip
The text was updated successfully, but these errors were encountered: