Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Main UI] Voice support #2275

Open
GiviMAD opened this issue Jan 17, 2024 · 1 comment
Open

[Main UI] Voice support #2275

GiviMAD opened this issue Jan 17, 2024 · 1 comment
Labels
enhancement New feature or request main ui Main UI

Comments

@GiviMAD
Copy link
Member

GiviMAD commented Jan 17, 2024

Hello,

I want to add to the UI a client for the websocket protocol added by these PR openhab/openhab-core#4032 to use the voice features from the UI.

I upgraded Webpack to v5 in these PR #2267 due to the its improved asset handling and over it I have created a functional POC that I will publish as a WIP PR so you can take a look to the code added.

The highlight design:

I have a class AudioMain, this class is responsable for:

  • Initialize a webworker (AudioWorker) that holds the persistent websocket connection.
  • Setup AudioWorkets and transfer its MessagePort to the AudioWorker.
  • Notify UI about connection state, source state and sink state.
  • Enable client keyword spotting functionality and propagate the detection to the worker.

Then the AudioWorker is in charge of:

  • Create a persistent connection to the audio-pcm websocket.
  • Instruct AudioMain to setup the source or sinks and transfer their MessagePort to the AudioWorker.
  • Transfer the audio from the WebSocket to the appropriate MessagePort or viceversa, doing reencode and resample to the audio data as needed.
  • Propagate to AudioMain the connection state changes.

I need a lot of feedback.

Initially I have placed the mentioned code under src/js/voice and I have added the options to the src/components/theme-switcher.vue. Also I have created a src/components/dialog-mixin.js for the src/components/app.js and from there the initialization is done when the option is enabled.

I will like to add some kind of icon that inform the user about the recording state and also that can be used to trigger the dialog by clicking on it, but I don't know where to place it or what kind of design I should use.

I will open the WIP PR when I have a moment and mention these issue because I think it will be easier to comment over what it's done.

Best Regards

@GiviMAD GiviMAD added enhancement New feature or request main ui Main UI labels Jan 17, 2024
@GiviMAD
Copy link
Member Author

GiviMAD commented Jan 17, 2024

POC mobile video:
https://github.com/openhab/openhab-webui/assets/9007708/987e3fac-2ea8-43ae-98f3-83e646add580

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request main ui Main UI
Projects
None yet
Development

No branches or pull requests

1 participant