Containers continuously deactivated, system is almost useless #3021
Labels
Comments
|
Is there maybe more in the logs of Supervisor when the add-on shuts down? E.g. is it the Supervisor shutting down the add-on or is it the add-on itself which shuts itself down? I'd guess an out-of-memory situation being the issue here. You can monitor memory usage on your system using https://www.home-assistant.io/integrations/systemmonitor? |
|
These |
|
Ah yeah right the health checks cause this. So yes, this is regular operation and unlikely the culprit of the issue you are seeing. The slowdown it is likely caused by something else. |
Resource doesn’t seems to be an issue (right now RAM 15%, CPU 4%, disk 6%). |
Only few users reported the “deactivated successfully” logs in the issue you referred. They make the logs useless. How can I be sure those are “good” logs, and how can I investigate more my issue? |
Sounds like NodeRED sends some invalid websocket message? 🤔 You filed this as a Home Assistant OS issue, what makes you belive this is a OS issue? I think a new version/configuration change of another component of the system made this error to start appear. |
|
Actually: This are independent API calls from the Supervisor to the Core. It looks more like your Core is having troubles. I'd check it's logs. Also make sure to disable custom components to see if any of these cause the issue. |
How can you say this logs are related to NodeRED? Regarding your other update, this is one line of log that sound suspicious in the HA Core log, but it doesn't give that many information I filed this issue as HA OS issue because from my initial troubleshooting it looked to me that:
I also have lots of error in the DNS logs I don't know if this is meant to be like this, or are a sign of some sort of failure. I think my point is: my HA OS installation is not working properly, there are evidence of issue in the logs but to me (I do this for a living) the logs are not enough to understand where is the issue and what caused it. Do you know where to point me to get more information on the issue I'm experiencing? Thanks |
|
In the end it may be related to a custom component... I added through HACS a component to connect to my ABB Solar Inverter, and it is not working (configuration error). After I removed it from the configuration, everything started to work fine again. I tried to add the configuration again, and the screen froze with a spinning wheel, the configuration was never added, and the usual logs reappeared. This is the error thrown by the component Immediately followed by The entire UI froze for around 1 minute then everything started working again. Since it wasn't added correctly, the issue was momentarily. Before I had the configuration (it was working at some point) and I guess HA core kept retrying to setup the component, creating all the mess. How can it be that the fail of an integration cause so much issue to the entire system? |
|
Try delete the custom component completely from the file system, restart, and try to re add it. |
Tried but it didn't helped, the issue is still there every time I re-enable the custom component. I guess I can close this issue and see if the owner of the component do something about it. |
|
Close since it was an issue related to a custom component, even if not sure on how to solve it. |
sairon
added a commit
that referenced
this issue
Jul 24, 2024
Reduce verbosity from deactivated Docker mounts, triggered by the Docker healthcheck. These messages do not carry any value for us and logs supplied by users are often spammed mostly with these. Moreover, they sometimes cause confusion that something is wrong, see for example #3021. Unfortunately, it's not possible to use LogFilterPatterns= here, because it's not applied to these messages, as explicitly said in the docs: > Filtering is based on the unit for which LogFilterPatterns= is defined, > meaning log messages coming from systemd(1) about the unit are not taken > into account. runc 1.2.0 supposedly should fix this, but it's unclear when it would be available, so let's stick to this solution (reducing verbosity from debug to notice for all units `run-docker-*.mount`) for the time being.
sairon
added a commit
that referenced
this issue
Jul 24, 2024
Reduce verbosity from deactivated Docker mounts, triggered by the Docker healthcheck. These messages do not carry any value for us and logs supplied by users are often spammed mostly with these. Moreover, they sometimes cause confusion that something is wrong, see for example #3021. Unfortunately, it's not possible to use LogFilterPatterns= here, because it's not applied to these messages, as explicitly said in the docs: Filtering is based on the unit for which LogFilterPatterns= is defined meaning log messages coming from systemd(1) about the unit are not taken into account. runc 1.2.0 supposedly should fix this, but it's unclear when it would be available, so let's stick to this solution (reducing verbosity from debug to notice for all units `run-docker-*.mount`) for the time being.
sairon
added a commit
that referenced
this issue
Jul 24, 2024
Reduce verbosity from deactivated Docker mounts, triggered by the Docker healthcheck. These messages do not carry any value for us and logs supplied by users are often spammed mostly with these. Moreover, they sometimes cause confusion that something is wrong, see for example #3021. Unfortunately, it's not possible to use LogFilterPatterns= here, because it's not applied to these messages, as explicitly said in the docs: Filtering is based on the unit for which LogFilterPatterns= is defined meaning log messages coming from systemd(1) about the unit are not taken into account. runc 1.2.0 supposedly should fix this, but it's unclear when it would be available, so let's stick to this solution (reducing verbosity from debug to notice for all units `run-docker-*.mount`) for the time being.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the issue you are experiencing
Hi,
since several days, my HA installation became really slow, almost useless. UI crashes every other minute, Alexa and Apple Home have problem in sending command to UI.
I’m running HASS OS on a x64 minipc, I have HA from 5 year, and more than 1 year on this HW.
I did some investigation, and I think the reason of all of this is related to the containers behind HA. I have a lot of identical logs in the Host registry
and these
These logs, thanks to another post in the community, led me to believe that some (or all) containers are constantly failing and being reactivated by the Watchdog.
I managed to understand which are the docker that keep getting deactivate while I was performing some troubleshooting:
b4f04e homeassistant/amd64-addon-configurator:5.7.0
8546a3 ghcr.io/hassio-addons/node-red/amd64:16.0.2
fc8db1 ghcr.io/hassio-addons/adguard/amd64:5.0.1
at this moment these are the only ones that are getting deactivated and activate all over again, but during the last 24 hours I noticed also others having the same behavior. How can I investigate more this issue? Where I can find the root cause of this activation/deactivation? Logs currently doesn't give any information on why everything seems to be failing.
There must be something, maybe on docker side, that can points me in the right direction.
Thanks
Matteo
What operating system image do you use?
generic-x86-64 (Generic UEFI capable x86-64 systems)
What version of Home Assistant Operating System is installed?
11.2
Did you upgrade the Operating System.
Yes
Steps to reproduce the issue
I wish I knew, it just keep happening.
Anything in the Supervisor logs that might be useful for us?
Anything in the Host logs that might be useful for us?
System information
System Information
Home Assistant Community Store
AccuWeather
Home Assistant Supervisor
Dashboards
Recorder
Additional information
No response
The text was updated successfully, but these errors were encountered: