Walkie talkie/Broadcast recorded voice message

[MattFellows]

Hi, I got my Box today, and installed Willow and hooked it up to Home Assistant, and it almost works perfectly (that wake word is tricksy, but I have seen this discussed elsewhere)... One feature I use on my Google homes is the 'broadcast to X' feature. I use them like a hands free walkie talkie or intercom.

I was going to start having a play with the code in a fork, to look at how hard it would be to implement, but wanted to check if such a thing was already in the works or if there was some dependency that would need working on first etc?

Happy to put this on the back burner if I could be of use somewhere else in some kind of quid pro quo, for some help later on with this (I realise 1.0 is your priority right now). I'm a jack of all trades software engineer with 20 years xp and a quick study...

[kristiankielhofner]

Which wake word are you using? Have you tried configuring for 2CH 95 under Advanced Configuration? With our initial developer "release" we opted for very conservative default values - 95 is more aggressive.

We have no plans for such a feature but it is interesting. I think the most challenging aspect will be forking the audio to the recipient devices and knowing which ones. Then the next issue is our current operating paradigm is to sit and do nothing until wake is initiated on a device by the user. We're also extremely reluctant to listen on any network sockets because of the potential security issues it could expose (we get quite a few security questions) and the fact it's yet another running task.

I haven't given it much thought but you would need to do something along the lines of establishing a websocket and piping audio to each device and playing it out (we don't support this currently) -or- some kind of socket listening on a multicast group to play out received frames automatically -or- some other kind of socket to do the same. Multicast with WiFi can be very tricky and this would also give you less granularity in terms of which device(s) audio can be directed to.

This is all on a microcontroller with a real time operating system so you would need to be very, very careful in how you go about this to not adversely impact other functionality (or even do this with clean audio). We have left plenty of headroom for future functionality but if you haven't done development under these conditions before it is challenging. This device after all (at this price point) is a dual core 240 MHz CPU with some tricky memory and task management.

I also have no idea what the latency would be like with this functionality (or what your expectations are).

[MattFellows]

I was using Hi ESP, but found I had to really separate the phonemes(?)/syllables with a fairly large gap between them, that felt quite unnatural. I switched to Alexa for now.

I was considering some kind of correlative update process for box identification. Each box could open up a websocket periodically (assuming I can get some Cron based functionality to happen somehow) and listen for updates of IP addresses/hostnames and device ids/names. During the config process, we could store a secret which is used for authentication between the devices (PGP if not too resource intensive or something else fairly standard, preference would be SSL with certs set up during config, but again not sure if that would be too resource intensive)

I've managed to do some trivial changes so far with updating the UI based on button presses etc, I am gonna see what I can do going depth first and seeing what bottlenecks come up.

Do yell if I am stepping on any toes, I think I read somewhere you had some plans for device ids and groups?

[kristiankielhofner]

HI ESP is a horrendous wake word for the reasons you highlight. When we have "Hey Willow" it will overwhelmingly be the default recommended wake word. Alexa does eliminate the annunciation issue but a lot of people don't like it because of the obvious association with Amazon.

We have the fundamentals of what you are describing with Willow Application Server (repo soon to be made public). Each device maintains a persistent websocket connection and we push config changes and commands for OTA down it. The devices also use it to register with WAS so they can be enumerated in the dashboard for these pushes.

You will still have to augment several things with these fundamentals to implement what you are describing.

Unfortunately, given the fundamental hardware limitations it's unlikely that your approach with per-session authentication/encryption will provide a timely interactive experience. We use a lot of websockets because the initial TLS handshake is relatively expensive and we only want to do it once as part of the boot process (and occasional reconnects when the connection is closed for whatever underlying reasons - service restart, wifi hiccup, etc). X.509 certificate based validation and everything it entails is outside the capability/interest of many users so these connections are actually unencrypted by default. Although we do make it easy via checkboxes and URL format (WS vs WSS, HTTP vs HTTPS).

PGP is probably a non-starter because you would need component libraries for the ESP, which to my knowledge don't exist because it's a pretty weird use case/application requirement.

Additionally, I don't want to present a cavalier attitude towards security but depending on your latency expectations anything other than raw UDP audio streams or using existing established websockets likely won't provide a good user experience.

It's open source! You're not stepping on any toes, you can do whatever you want/are able to. I'm just trying to communicate our perspective on what is likely to be maintained/supported in tree. If we were to eventually add this functionality it would likely be implemented conceptually in the way I am describing.

Yes, we have plans for device IDs and groups with the thinking being groups in proximity that could wake at the same time for the same command (we need to avoid that) and adding the ability to associate device id/group with a given zone/location (in HA or similar) so that generic commands like "turn on the light" can be mapped to take action on the associated devices based on group/zone.

However, this like many things depends heavily on command endpoint support (HA). What we are able to provide via API to Home Assistant is really quite limited - their API essentially only supports STT output text and doesn't include things we already have today like dynamic language detection. Unless this changes the only way we could possibly support this in practice is to have some wildly convoluted mapping in WAS to modify STT output to merge in zone/location so HA intents can associate and process it correctly.