homekit_controller: Support events so HA can respond to accessory state changes in real time

[tomascrespo]

Home Assistant release with the issue: 0.97.2
Last working Home Assistant release (if known): 0.96.5
Operating environment (Hass.io/Docker/Windows/etc.): Raspberry Pi with Raspbian / Debian Linux 9 Stretch
Component/platform: Homekit
Description of problem:
Hi! I have a very basic alarm system, only an Aqara Hub and an Aqara window/door sensor. I installed Home Assistant 0.96.5 and added the Aqara Hub as a HomeKit accesory (the UI detected it automatically, I only had to enter the code in the label). All was working perfectly, when I pressed ARM AWAY or DISSARM button it was almost instantaneously, and when someone opened the door the state changed to TRIGGERD (and I got the notification) very quickly.

Now I have installed from scratch (the SD card died) with latest version, 0.97.2, and I have added the alarm, but I having an tremendous lag. It takes about one minute any change, ARM, DISSARM or TRIGGER.

Any clue?

Lot of thanks

[tomascrespo]

I have downgraded to 0.96.5 and all works instantaneously, so the lag is only present newer versions

[Jc2k]

It sounds like this is about homekit_controller rather than homekit (the homekit integration is for using the HomeKit protocol between your iPhone etc and HomeAssistant, homekit_controller is for using the HomeKit protocol between a device (like the Aqara Hub) and home assistant). Can you confirm that is the case? I can retag to the right area if so.

Also, if its homekit_controller:

The homekit_controller integration is poll only, and it polls once a minute. It has only ever polled once a minute. If it used to seem faster then that was just luck - it must have been triggered near the 1 minute timer running out.

On newer builds there is some times a race on start up where the first poll is missed. But this has no other impact on a running system.

The dev branch i'm working on will replace the blocking homekit client with an asyncio one. The main aim of this work is support for HomeKit events which means that things like this will update near instantaneously, but the current synchronous client is incapable of supporting events. This is all working in my dev environment but relies on a PR getting accepted upstream, then a release happening, then a PR to home assistant to use the new release, then a library, and a good amount of QA needs to be done at each stage. So it will be a few releases probably.

[deetox]

Hello Jc2k !

Wouow ! It looks like wonderful !

Is there any chance you make your dev branch public for the ones who want to work with ? I just can’t wait to test it ! 😋

Anyway: a huge Thank you !

[Jc2k]

Does that mean you agree this ticket is for homekit_controller and not homekit? :-)

The code should all be on my GitHub but it involves dev versions of HA and homekit_python so i'm reluctant to describe or support how to try it out just yet! When the homekit_python part is released onto pypi i'll open up a PR on the home-assistant side and ping this ticket.

[deetox]

Hello again.

From my side, I talk about homekit_controller but I am not the first author: your projet sounds like interesting and I wonder if I could try it somewhere.

I understand about support problem. So I will be patient.

Thank you again.

[adrum]

@Jc2k I paired a Honeywell Lyric Controller via HomeKit Controller and noticed the same behavior. The updates would take some time. When I paired them with my phone, the state changes showed instantly.

Just chiming in to add in my experience with homekit_controller.

[gargomoma]

Hi everyone,
Having the same issue here.
In my case updates are slow, but also, after a couple of hours (2 to 3) I stop receiving ANY update from any of the devices.

The only solution I've come up with is to restart the container to have it back to normal.

v0.100.2 -- Docker
Xiaomi Aqara Bridge [China Server]
(3 - Temp sensors; 1 - Motion sensor; 1 - Door sensor)

[Jc2k]

@gargomoma can you please confirm what you mean by slow? As explained, a delay of about a minute is normal and won't be changing for a while until the work to support events is ready. Longer than that is a sign of a problem.

The 2nd part of your comment - stopping after a few hours is definitely not normal, could you please raise a seperate ticket for that though. It's a seperate problem. Please could you check your logs too and post those on the new ticket. It's likely that the connection is timing in a way that we haven't seen before and we'll need to handle that case better than we currently do. Feek free to @Jc2k on the new ticket so I get notified.

[brucysNZ]

@Jc2k Hey mate. Any update on this slow response issue with HomeKit controller with regards to sensors taking up to 1min to respond?
I am also waiting for the fix. Thanks heaps for your efforts

[Jc2k]

Hey bud. it’s not really a fix, the code is working as designed. The plan is to moving from a simple synchronous polling model to a fully asynchronous event based model. While I have done a lot of work to prepare for this that is merged and the remaining patch won’t be that large the effect will be like I rewrote the thing entirely. With that level of change is the risk that every bug I fixed for the old code might be reintroduced.

I’ve run out of spare time and while I have this working for myself more or less it would be irresponsible to drop it on HA and run.

So for now you are kinda stuck, sorry.

[brucysNZ]

Hey bud. Thanks heaps for your efforts for sure. I am not a programmer so I can only imagine the challenges you face with coming up with a solution. Sounds like a huge amount of work you have done. I know you will do the right thing and only release something when you can and when you are confident of success. It would be amazing if other knowledgeable programmers could assist you.
I paired my hub directly to iOS app initially and everything was perfect with instant response from all door sensors and motion sensors, 8 devices operating perfectly and instant. The only issue was getting devices into HA. So unpaired and paired to HA successful with all devices showing in HA, only problem not, Speed of detection, around 1 min to detect sadly.....
I sure hope something can be achieved soon as it is basically unusable in its current state
Also
Tried unpairing from HA to connect directly back to HomeKit iOS and now it won’t connect or pair back to HomeKit iOS, so now definitely unusable.
I know the community of users having this issue would appreciate a resolution when one ready to deploy. Well done and good luck bud

[Jc2k]

Yeah, if you are using door and motion sensors homekit_controller is really not for you until the async stuff lands. There is only a motion sensor at all so that I can do dev work on support for HomeKit BLE (I have an Eve bluetooth motion sensor). A workaround is to use the homekit integration to export switches to your iOS devices and iOS automations to trigger the switches when something is sensed. I use a similar trick so that I can use HomeKit home arrival triggers in HA.

You have to unpair on both sides. On simpler devices this might mean a full reset, but hubs that speak multiple protocols (like Hue) often have a "reset homekit" menu option somewhere. We should be able to tell the devices when HA removes it pairing data, but it's low down the list - I think most users would have e.g. async / events rather than a silightly easier unpair.

[brucysNZ]

Hey bud. Yes I agree most users would want Async/events rather than easier unpair. Async is priority. I cannot work out why I cannot setup new pair with IOS device again, as the little trick of using HA Switches is exactly what I was initially doing to get actions into HA. Worked OK just not ideal.
BLE however is another option. I use Xiaomi Bluetooth Temp and Humidity sensors with BLE Tracker. I do not have any other Bluetooth devices other than these temp/hum and they work flawlessly. They are very solid and reliable using ESP32 board as BLE tracker.
I will try unpair both sides, that is Homekit, and HA and try to setup Aqara Hub directly with IOS Aqara app. Hope it works as using the switches will be fine until and solution is found.
Will look at the Eve Bluetooth Motion Sensors also

[Jc2k]

To be clear Eve / Bluetooth homekit stuff doesn’t work with HA natively yet, it’s next on my list after async.

[bazorante14]

Firstly, I want to thank you for your work, it is a great contribution to the community. Are you predicting when the update will be available for us to have an instant update?

[Jc2k]

No, the past 2 months have seen the time I have slow to a near stand still. Async and events is a big chunk of work. Even though it’s working locally in my dev environment more or less there is still a big gap before it’s ready to inflict on hundreds of HA users.

[probot-home-assistant]

Hey there @Jc2k, mind taking a look at this issue as its been labeled with a integration (homekit_controller) you are listed as a codeowner for? Thanks!

[brucysNZ]

@Jc2k Still following this buddy, Keep up the great work you are doing for the community, it is greatly appreciated for sure.

[eugr]

@Jc2k - please let me know if you need any help with the development and/or testing. Looks like you’ve done most of the work already... I see your pull request to homekit_python is still pending - any blockers there?

[Jc2k]

@eugr sorry for the slow response. Your comment pushed me to find a 25th hour in one of my days last week and I added a CLI tool to that homekit_python branch. It would be great if you could find time to try it out. I posted examples of using it on the PR. Right now knowing that those commands work on a diverse range of networks and accessories would be a big help.

[rodriguezst]

@Jc2k I have installed both your branch of homeassistant with HomeKit controller events and homekit_python with aioclient and now everything works instantaneously. Thank you!! I am hoping to see this included in the master branch soon

[Jc2k]

@rodriguezst thanks for testing. Great to hear. Can you share what devices you are testing with? The biggest "risk" is how it handles inevitable TCP connection timeouts. With the old code they could manifest as about 10 different exceptions and we were still finding new ones. So i'm interested to hear how stable it has been and whether you have seen any errors in your logs after running for a few days.

It would also be really great if you could comment on the homekit_python and let the maintainer know it's working for you in HA. If you are also able to try the CLI commands i posted here and confirm that on the PR too that would be great. Post the devices you tested with there too.

To anyone watching this issue, the biggest thing you can do to help right now is to try the commands here and report how it went and with what devices on that PR.

[eugr]

@Jc2k - that's fine, now I'm being slow... I'm going to give it a try and will report back

[eugr]

@Jc2k - is there any way to make it work with existing pairings from my HA instance? Also, I've noticed that it complains about asyncio.all_tasks which is not supported in Python 3.6. Is it expected behavior? It doesn't seem to affect anything though.

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/eugr/src/homekit_python/homekit/aio/__main__.py", line 464, in <module>
    run(main())
  File "/home/eugr/src/homekit_python/homekit/aio/__main__.py", line 54, in run
    _cancel_all_tasks(loop)
  File "/home/eugr/src/homekit_python/homekit/aio/__main__.py", line 21, in _cancel_all_tasks
    to_cancel = asyncio.all_tasks(loop)
AttributeError: module 'asyncio' has no attribute 'all_tasks'

[Jc2k]

I only have 3.7 and 3.8 to hand so hadn't noticed this one :( Thanks for pointing it out. It's only used when exiting so not an issue for the testing I need right now. It's also only applicable to the CLI so not a worry for HA either.

As for reusing your pairings - yes. You need to find your configuration/.storage/ folder and in there there should be a file named something like core.config_entries? In there you should see a bit of the JSON that looks like:

            {
                "connection_class": "local_poll",
                "data": {
                    "AccessoryIP": "192.168.1.sss",
                    "AccessoryLTPK": "sdfsdf",
                    "AccessoryPairingID": "2fsdfsdfC",
                    "AccessoryPort": 51827,
                    "Connection": "IP",
                    "iOSDeviceLTPK": "dddd",
                    "iOSDeviceLTSK": "dddd",
                    "iOSPairingId": "sssss"
                },
                "domain": "homekit_controller",
                .... snip ...

You want to take the bit inside data and make a new pairing.json like this:

{
  "myalias": {
                    "AccessoryIP": "192.168.1.sss",
                    "AccessoryLTPK": "sdfsdf",
                    "AccessoryPairingID": "2fsdfsdfC",
                    "AccessoryPort": 51827,
                    "Connection": "IP",
                    "iOSDeviceLTPK": "dddd",
                    "iOSDeviceLTSK": "dddd",
                    "iOSPairingId": "sssss"
  }
}

You should now be able to use the get_events/get_accessories/get_characteristic/put_characteristic commands.

[sarevok1]

If i understand right, this update was already released. I am running HA 0.106.5. I've reset and reconnected Aqara Hub through HomeKit controller integration but i see state changes for various motion sensors connected to Aqara with a big delay, much like before. Any ideas?

[eugr]

@sarevok1 - I don't see this commit merged into either master or rc branch, so it's not released to the public yet (unless you run the dev branch)

[sarevok1]

i'am not. is this the right place to check if it was released: https://github.com/home-assistant/core/releases, PR#32213, then? i am new to this... thanks.

[eugr]

yep, or in the official release notes. It likely won't appear in the official release until at least 0.107.0

[Jc2k]

@eugr is correct. It was merged into dev shortly before 0.106.x was released, but rc had already been updated for 0.106.x so it "missed the boat". I'm running dev homekit_controller on my main home-assistant and motion sensors are just working for me with my Hue bridge and hue motion sensors, there is no need to re-pair when this is released. If your device supports it, it should just start working.

I'm not expecting it to go out before 0.107.0 now.

If you aren't a dev the best bet is the release notes here. I did a bunch of homekit stuff in 0.104.0 as an example - and you can see it by searching for homekit_controller on that post. So the async PR should show up in the same format in the release notes for 0.107.0.

I am expecting #32141 #32158 #32175 #32177 #32188 #32212 #32213 #32228 #32404 #32526 and #32610 to debut in 0.107.0. I'm not sure if stateless remotes will make it into the next release at this point.

[sarevok1]

awesome; will be eagerly waiting, thank u for your help.

[Mohannadshariefhome]

hello @Jc2k , I installed the pre-release and stopped getting any updates from my devices, it this normal?

[Jc2k]

@Mohannadshariefhome is it a Tado device of some kind?

[Mohannadshariefhome]

@Jc2k no it is Xiaomi v3 hub and yes it si confirmed no updates from my devices

[Mohannadshariefhome]

@Jc2k no updates from contact sensors, temp sensors or motions sensors, it only gets the latest status before the home assistant update

[Jc2k]

@Mohannadshariefhome are you able to ping me on discord and try some debug steps with me?

[Mohannadshariefhome]

@Jc2k yes sure, what is your ID

[Mohannadshariefhome]

@Jc2k I am ready whenever I get your tag

[Jc2k]

Jc2k#6068

[Mohannadshariefhome]

Added @Jc2k

[eugr]

Hey @Jc2k - I'm running 0.107.0 with your asyncio implementation and my HomeKit garage opener (Insignia) doesn't respond to commands or respond to events anymore. I can ping it and I can connect to it using your aiolclient branch of homekit_python library (at least the version I was testing a couple of months back), but I'm getting connection timeouts in HA:

2020-03-18 18:39:44 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/pairing.py", line 90, in _ensure_connected
    await asyncio.wait_for(self.connection.ensure_connection(), 10)
  File "/usr/local/lib/python3.7/asyncio/tasks.py", line 449, in wait_for
    raise futures.TimeoutError()
concurrent.futures._base.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1257, in _execute_service
    await handler.func(service_call)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 198, in handle_service
    self._platforms.values(), func, call, required_features
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 402, in entity_service_call
    future.result()  # pop exception if have
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 599, in async_request_call
    await coro
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 433, in _handle_entity_call
    await result
  File "/usr/src/homeassistant/homeassistant/components/homekit_controller/cover.py", line 105, in async_open_cover
    await self.set_door_state(STATE_OPEN)
  File "/usr/src/homeassistant/homeassistant/components/homekit_controller/cover.py", line 114, in set_door_state
    {CharacteristicsTypes.DOOR_STATE_TARGET: TARGET_GARAGE_STATE_MAP[state]}
  File "/usr/src/homeassistant/homeassistant/components/homekit_controller/__init__.py", line 98, in async_put_characteristics
    return await self._accessory.put_characteristics(payload)
  File "/usr/src/homeassistant/homeassistant/components/homekit_controller/connection.py", line 342, in put_characteristics
    results = await self.pairing.put_characteristics(characteristics)
  File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/pairing.py", line 246, in put_characteristics
    await self._ensure_connected()
  File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/pairing.py", line 93, in _ensure_connected
    "Timeout while waiting for connection to device"
aiohomekit.exceptions.AccessoryDisconnectedError: Timeout while waiting for connection to device

Tried to restart HA, but nothing changed. Hasn't tried to re-pair yet as it's in hard to reach area, but can try tomorrow if needed. Any advise on how to debug it?

[Jc2k]

Don't try repairing, none of the changes should need that and especially if we had it working before it's probably going to be a small change to get it working. Might be best to spin up a new ticket though.

The timeout is added here:

  File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/pairing.py", line 90, in _ensure_connected
    await asyncio.wait_for(self.connection.ensure_connection(), 10)

So bump that 10 to 30 and see if that helps, and tell me either in a new ticket or on discord.

[eugr]

@Jc2k - thanks, pinged you on Discord

[stoprocent]

Hi @Jc2k

I'm getting

2020-03-22 12:48:26 ERROR (MainThread) [homeassistant.components.homekit_controller.config_flow] Pairing attempt failed with an unhandled exception

Traceback (most recent call last):

  File "/usr/src/homeassistant/homeassistant/components/homekit_controller/config_flow.py", line 318, in async_step_pair

    self.finish_pairing = await discovery.start_pairing(self.hkid)

  File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/discovery.py", line 61, in start_pairing

    await self._ensure_connected()

  File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/discovery.py", line 47, in _ensure_connected

    await self.connection.ensure_connection()

  File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/connection.py", line 230, in ensure_connection

    await asyncio.shield(self._connector)

concurrent.futures._base.CancelledError

Any idea how to tackle this?

[Jc2k]

@stoprocent any chance you could open a new ticket and include all the details listed there. Also try increasing your log level for homeassistant.components.homekit_controller and aiohomekit and include the log before the exception not just the exception. And most importantly what kind of device so i know if its something i've already triaged or something different. Also did it work previously? If you are re-pairing something that was previously working that might be a useful diagnostic. Feel free to @ me on new ticket.

[stoprocent]

@Jc2k I'm not sure if I should :) I already found a solution. The problem was on the homebridge side related to this issue: homebridge/homebridge#2098

When homebridge is dockerized it is advertising wrong ip address. Applying thefix from workaround on this issue made aiohomekit working perfect.

[Jc2k]

Ah nice! We've always struggled with homebridge support! Any chance you could detail what plugins you are using with homebridge? I'm trying to keep track of known working devices and setups.

[stoprocent]

I'm using just 2 plugins:

1. It's a modified version of blinds plugin: https://github.com/stoprocent/homebridge-gpio-blinds

2. It's something that can easily be done in python for hass but the only nice implementation is done so far only for homebridge: https://github.com/mgcrea/homebridge-tydom

My modified blinds plugin works just great. There is a 0.5 sec delay comparing to using it straight from homebridge.

The second plugin is losing states sometimes, but for what I use it it's functional.

[Jc2k]

Nice. Thanks for that.

[l4ci]

Was this released to the public yet?

[Jc2k]

homekit_controller has supported events since about 0.107, with some bug fixes in 0.108.

[l4ci]

I mean the 1 minute polling delay others mentioned. It seems to still be the case for me in my aqara -> Homekit integration to HA

[Jc2k]

Everything discussed in this old closed ticket is released to the public and I think has been since 0.107.0. I'd normally say you need to open a new ticket but as it happens, you are probably hitting issue #33511. This is an Aqara specific bug and sporadic. For some users its working fine. For some it rarely works. Please move this discussion to #33511, or if you don't think its the same issue please open a new ticket.

[l4ci]

Thanks @Jc2k, will do that!