Crashing home assistant on many added entities #809

elupus · 2022-12-18T19:40:25Z

Describe the bug

When an integration is added that adds many entities. Node red seem to overload the home assistant core system and crashes home assistant.

This was noticed with an integration that adds 1200 entities from one integration setup in home-assistant/core#83759

A likely cause is no debouncing or throttling of device registry update. Standard home assistant frontend will debounce device registry updates as cen be seen here: https://github.com/home-assistant/frontend/blob/b31a9d590ef53916cc6f481efa877a57b6f756af/src/data/device_registry.ts#L103

To Reproduce

Add nibe_heatpump integration with an S1255 pump (requires responding pump sadly)

Expected behavior

Home assistant keeps running.

Screenshots

No response

Example Flow

No response

This package's version (not the Home Assistant add-on version)

Unknown

Is Node-RED running in Docker?

Yes

Node-RED version

Unknown

Node.js version

Unknown

Additional context

Upstream bug report: home-assistant/core#83759

elupus · 2022-12-18T19:40:45Z

I will try to ask original reporters for more info, not running node red myself.

elupus · 2022-12-18T19:49:47Z

This is where debouncing is likely needed:

node-red-contrib-home-assistant-websocket/src/homeAssistant/collections.ts

Line 44 in 5103c78

conn.subscribeEvents(async () => {

Ps. Not sure why this crashes HA, seems like it shouldn't do that anyway. Its probably filling the response queue with 1200 copies of the list of devices in HA.

AndreFjeld · 2022-12-18T20:36:00Z

I can confirm that this issue is there as long as node red is running. If i stop node red, the integration can successfully ble installed.

remcoselles · 2022-12-19T15:33:47Z

I also experience the same issue @AndreFjeld is describing.

zachowj · 2022-12-20T19:51:31Z

Adding a throttle to the subscribeEvents is probably a good thing. I'll get that added to a release in the coming days.

I created a component so I could create a large number of entities or devices when it is added to HA. Testing with ranges of 1000 to 10,000 entities I couldn't get HA to crash with NR connected, this is also probably machine dependent. It did kill any websocket connection, not just NR, that was connected due to overflowing the pending message queue but HA never stop running.

elupus · 2022-12-20T21:20:50Z

I think the supervisor in hassos will kill core if unresponsive for some given time. A watchdog. So core likely didnt crash, just not available on coms for too long.

zachowj · 2022-12-21T06:33:48Z

0.47.2 contains the throttle changes.

Without NR connected to HA and only a single browser connected to UI, it is getting overloaded when 1000 entities are created in a short amount of time. The state_changed event is flooding the weboscket connection causing the pending buffer to overflow. So not really sure if this change is going to help the heat pump integration loading issue.

elupus · 2022-12-21T06:46:30Z

In this case its 1000 disabled entities. So no state_changed i think.

zachowj · 2022-12-21T07:10:38Z

Nope, looks like it still sends the event just has their state set to unavailable

websocket log

2022-12-21 07:03:22.553 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [140018645366096] Sending

[
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1301":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1301",
                  "supported_features":0
               },
               "c":"01GMSQWW20QAZMGR8T3FG2VZNZ",
               "lc":1671606202.43267
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1302":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1302",
                  "supported_features":0
               },
               "c":"01GMSQWW20P23X4T6B4T9RJ34M",
               "lc":1671606202.432766
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1303":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1303",
                  "supported_features":0
               },
               "c":"01GMSQWW200MCWV7Z5A2GWNH6R",
               "lc":1671606202.432814
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1304":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1304",
                  "supported_features":0
               },
               "c":"01GMSQWW20N8W8X5KSYZGYQP1F",
               "lc":1671606202.432856
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1305":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1305",
                  "supported_features":0
               },
               "c":"01GMSQWW20D4YCP8JPK5R8YQ5F",
               "lc":1671606202.432895
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1306":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1306",
                  "supported_features":0
               },
               "c":"01GMSQWW20CKGS9MVW7XNGWMCM",
               "lc":1671606202.432934
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1307":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1307",
                  "supported_features":0
               },
               "c":"01GMSQWW20CGM1HXZ89BD9BJA7",
               "lc":1671606202.432973
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1308":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1308",
                  "supported_features":0
               },
               "c":"01GMSQWW21EQW43TVSBNGK6YJ4",
               "lc":1671606202.433011
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1309":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1309",
                  "supported_features":0
               },
               "c":"01GMSQWW21542QQFE5JNNAJ22Q",
               "lc":1671606202.433047
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1310":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1310",
                  "supported_features":0
               },
               "c":"01GMSQWW217MAKVGH1ZBCN660E",
               "lc":1671606202.433145
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1311":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1311",
                  "supported_features":0
               },
               "c":"01GMSQWW21RR3EN886Q967GPXH",
               "lc":1671606202.433187
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1312":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1312",
                  "supported_features":0
               },
               "c":"01GMSQWW21DBRREDSV5P0H5GGH",
               "lc":1671606202.433225
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1313":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1313",
                  "supported_features":0
               },
               "c":"01GMSQWW212JD2861HFZTDM6X5",
               "lc":1671606202.433262
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1314":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1314",
                  "supported_features":0
               },
               "c":"01GMSQWW21SVYFGFH0ZNV27878",
               "lc":1671606202.433297
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1315":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1315",
                  "supported_features":0
               },
               "c":"01GMSQWW212FFKHM6C764MS1XN",
               "lc":1671606202.433334
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1316":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1316",
                  "supported_features":0
               },
               "c":"01GMSQWW211DK1VPM8Q53Q4Z3P",
               "lc":1671606202.433373
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1317":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1317",
                  "supported_features":0
               },
               "c":"01GMSQWW2150GE247095HSJSA4",
               "lc":1671606202.433426
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1318":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1318",
                  "supported_features":0
               },
               "c":"01GMSQWW21QNT21GMK1MBH0XN6",
               "lc":1671606202.43348
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1319":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1319",
                  "supported_features":0
               },
               "c":"01GMSQWW21PXC7284WE323V0Q7",
               "lc":1671606202.433528
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1320":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1320",
                  "supported_features":0
               },
               "c":"01GMSQWW21A6S252781266R64T",
               "lc":1671606202.433565
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1321":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1321",
                  "supported_features":0
               },
               "c":"01GMSQWW21HDW1VV9J8C5731A3",
               "lc":1671606202.433645
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1322":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1322",
                  "supported_features":0
               },
               "c":"01GMSQWW21C2BQ9GREP4EJTECV",
               "lc":1671606202.433911
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1323":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1323",
                  "supported_features":0
               },
               "c":"01GMSQWW218QK905FWFT70SFVT",
               "lc":1671606202.43397
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1324":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1324",
                  "supported_features":0
               },
               "c":"01GMSQWW228NASYB9R49W1MVG6",
               "lc":1671606202.434009
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1325":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1325",
                  "supported_features":0
               },
               "c":"01GMSQWW22583GS6R1K0192MTE",
               "lc":1671606202.434037
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1326":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1326",
                  "supported_features":0
               },
               "c":"01GMSQWW2221SP2AF25B2EPTKH",
               "lc":1671606202.434063
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1327":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1327",
                  "supported_features":0
               },
               "c":"01GMSQWW228T4Z3GB2V76RYA57",
               "lc":1671606202.434108
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1328":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1328",
                  "supported_features":0
               },
               "c":"01GMSQWW228NCQHMHX6EBHDG4V",
               "lc":1671606202.434213
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1329":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1329",
                  "supported_features":0
               },
               "c":"01GMSQWW221D82CD0AJB04RYJ1",
               "lc":1671606202.434269
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1330":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1330",
                  "supported_features":0
               },
               "c":"01GMSQWW22X4HFRVDXT5MDS99Z",
               "lc":1671606202.434311
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1331":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1331",
                  "supported_features":0
               },
               "c":"01GMSQWW22ENB9CV5B2KJE0WXY",
               "lc":1671606202.434365
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1332":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1332",
                  "supported_features":0
               },
               "c":"01GMSQWW222Y0KRSGGW8FJPPVB",
               "lc":1671606202.434404
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1333":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1333",
                  "supported_features":0
               },
               "c":"01GMSQWW22A8G7KM9A8931XBJC",
               "lc":1671606202.434441
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1334":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1334",
                  "supported_features":0
               },
               "c":"01GMSQWW22NH6C1KAV6X69A2VN",
               "lc":1671606202.43449
            }
         }
      }
   },
   {
      "id":2,
      "type":"event",
      "event":{
         "a":{
            "sensor.many_sensor_1335":{
               "s":"unavailable",
               "a":{
                  "restored":true,
                  "friendly_name":"Many sensor 1335",
                  "supported_features":0
               },
               "c":"01GMSQWW22ZYQ8M72FM5Y7D8VZ",
               "lc":1671606202.434538
            }
         }
      }
   }
]

elupus · 2022-12-21T08:32:39Z

Ok, that seem weird. They should not go into state machine. Will check on that.

elupus · 2022-12-22T15:29:49Z

I just tested here. I don't seem to be getting these events here for disabled entities. At least not for a reload of the integration. Just to double check. Your test code did set entity_registry_enabled_default to false right?

elupus · 2022-12-22T15:44:18Z

Also, is the node red websocket connection making use of "permessage-deflate" websocket extension to keep the size of the websocket messages down?

bdraco · 2022-12-24T20:42:21Z

0.47.2 contains the throttle changes.

Without NR connected to HA and only a single browser connected to UI, it is getting overloaded when 1000 entities are created in a short amount of time. The state_changed event is flooding the weboscket connection causing the pending buffer to overflow. So not really sure if this change is going to help the heat pump integration loading issue.

@zachowj If you enable message coalescing you'll get one or two messages instead of 1000s

home-assistant/home-assistant-js-websocket#306

bdraco · 2022-12-24T20:44:00Z

https://github.com/home-assistant/home-assistant-js-websocket/pull/306/files?w=1#diff-f0e64910289a49966c99ceadaa5637404e3439ec0812a4bdb7003fe1e5a33d1cR116

When you open the connection sending the following will tell the websocket backend to coalesce messages:

{
    type: "supported_features",
    id: 1, // Always the first message after auth
    features: { coalesce_messages: 1 },
};

bdraco · 2022-12-24T20:47:42Z

You'll get both types at that point:

Non-coalesced

{"id":3,"type":"event","event":{"c":{"sensor.gate_trigger_32_uptime_2":{"+":{"s":"2766354","lc":1671914769.480852,"c":"01GN2Y5K28ZQZXBXT4MAJFJNRP"}}}}}

Coalesced (just wrapped in [])

[{"id":3,"type":"event","event":{"c":{"binary_sensor.8th_st_west_object_detected":{"+":{"s":"off","lc":1671914833.423401,"c":"01GN2Y7HGFMHJ5ZEAX78HC30N5"},"-":{"a":["event_id","event_score"]}}}}},{"id":3,"type":"event","event":{"c":{"binary_sensor.8th_st_west_vehicle_detected":{"+":{"s":"off","lc":1671914833.423687,"c":"01GN2Y7HGF3Y134QWWSPF16DT2"},"-":{"a":["event_id","event_score"]}}}}},{"id":3,"type":"event","event":{"c":{"sensor.8th_st_west_detected_object":{"+":{"s":"none","lc":1671914833.424701,"c":"01GN2Y7HGGA7R7T4810ZJXW2QP"},"-":{"a":["event_id","event_score"]}}}}}]

bdraco · 2022-12-24T20:49:08Z

Also newer versions of home-assistant-js-websocket use subscribeEntities ({"type":"subscribe_entities","id":3}) for entity state changed events which uses the compressed format here:

https://github.com/home-assistant/home-assistant-js-websocket/blob/master/lib/entities.ts

https://github.com/home-assistant/core/blob/8819634b613f6bfd55885283bab86c3852ae40c4/homeassistant/components/websocket_api/messages.py#L114

zachowj · 2022-12-25T00:07:59Z

When you open the connection sending the following will tell the websocket backend to coalesce messages:

I recall you adding that but missed the part that it needs to be activated.

#809

github-actions · 2023-01-24T06:40:23Z

This issue has been automatically marked as abandoned because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

elupus mentioned this issue Dec 18, 2022

HA crashes while installing Nibe Heat Pump integration home-assistant/core#83759

Closed

bdraco mentioned this issue Dec 24, 2022

Permanent error: Client exceeded max pending messages [2]: 512 home-assistant/core#23938

Closed

zachowj added a commit that referenced this issue Dec 25, 2022

feat: Enable message coalescing

d235c35

#809

elupus mentioned this issue Dec 25, 2022

Avoid flushing whole entity registry cache home-assistant/core#84559

Closed

19 tasks

github-actions bot added the Stale label Jan 24, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jan 31, 2023

Habbie mentioned this issue Mar 19, 2024

maybe use message coalescing voorkant/voorkant-core#69

Open

This was referenced Mar 30, 2024

coalesce_feature not documented in websocket api home-assistant/home-assistant.io#32092

Open

document coalesce feature home-assistant/developers.home-assistant#2128

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crashing home assistant on many added entities #809

Crashing home assistant on many added entities #809

elupus commented Dec 18, 2022

elupus commented Dec 18, 2022

elupus commented Dec 18, 2022

AndreFjeld commented Dec 18, 2022

remcoselles commented Dec 19, 2022

zachowj commented Dec 20, 2022

elupus commented Dec 20, 2022

zachowj commented Dec 21, 2022

elupus commented Dec 21, 2022

zachowj commented Dec 21, 2022 •

edited

Loading

elupus commented Dec 21, 2022

elupus commented Dec 22, 2022

elupus commented Dec 22, 2022

bdraco commented Dec 24, 2022

bdraco commented Dec 24, 2022

bdraco commented Dec 24, 2022 •

edited

Loading

bdraco commented Dec 24, 2022 •

edited

Loading

zachowj commented Dec 25, 2022

github-actions bot commented Jan 24, 2023

Crashing home assistant on many added entities #809

Crashing home assistant on many added entities #809

Comments

elupus commented Dec 18, 2022

Describe the bug

To Reproduce

Expected behavior

Screenshots

Example Flow

This package's version (not the Home Assistant add-on version)

Is Node-RED running in Docker?

Node-RED version

Node.js version

Additional context

elupus commented Dec 18, 2022

elupus commented Dec 18, 2022

AndreFjeld commented Dec 18, 2022

remcoselles commented Dec 19, 2022

zachowj commented Dec 20, 2022

elupus commented Dec 20, 2022

zachowj commented Dec 21, 2022

elupus commented Dec 21, 2022

zachowj commented Dec 21, 2022 • edited Loading

elupus commented Dec 21, 2022

elupus commented Dec 22, 2022

elupus commented Dec 22, 2022

bdraco commented Dec 24, 2022

bdraco commented Dec 24, 2022

bdraco commented Dec 24, 2022 • edited Loading

bdraco commented Dec 24, 2022 • edited Loading

zachowj commented Dec 25, 2022

github-actions bot commented Jan 24, 2023

zachowj commented Dec 21, 2022 •

edited

Loading

bdraco commented Dec 24, 2022 •

edited

Loading

bdraco commented Dec 24, 2022 •

edited

Loading