Tracking: Web Chat + new Cognitive Services Speech SDK

[compulim]

This issue is for tracking the status on the work around Speech SDK.

Background

Web Chat use W3C Web Speech API to provide speech recognition and synthesis capability. Cognitive Services Speech Services has published their own API to provide speech recognition and synthesis capability.

This work is to create an adapter that turns Speech SDK into a W3C Web Speech API compliant so Web Chat (and other web apps) can consume.

For the deprecated Bing Speech, we have a version of the adapter. But that adapter no longer works with the new Speech Services. And we need to build a new one.

To-do

The to-do list below may expand from time to time to give more details on the actual work.

• Speech recognition
  • 11 stories in interactive mode (non-continuous mode)
    • Happy path with interim results
    • Happy path without interim results
    • Stop before first recognition is made
    • Stop after some speech is recognized
    • Abort before first recognition is made
    • Abort after some speech is recognized
    • Network issues due to airplane mode
    • Network issues due to bad credentials
    • Microphone muted
    • No speech is recognized
    • Not authorized to use microphone
  • 11 stories in continuous mode (Blocked, punted to New speech feature from Cognitive Services Speech Services #1400)
  • Recognition hints
    • Reference grammar ID (passed from Direct Line) (Blocked, punted to New speech feature from Cognitive Services Speech Services #1400)
    • Word list (specified by user) (Blocked, punted to New speech feature from Cognitive Services Speech Services #1400)
  • Text normalization options (ITN/Masked ITN/Lexical)
• Speech synthesis
  • Support pause() and resume()
• Switch between Bing Speech and Speech Services
• Integrate into Web Chat bundle

FAQs

We will update this issue when we have more information.

When will this feature land?

We are targeting 12/15 (version 4.2). But there are multiple risks related to this work. We are aware that this feature has very high demand and we are working with Cognitive Services Speech team to land this feature as soon as possible.

Will this feature support Custom Speech (CRIS)?

We want to support both Speech Service and Custom Speech Service. But information on our hands are limited. If we are not able to launch the first adapter with CRIS, we will schedule it as a point release (patch release).

Why I cannot use speech in Web Chat now?

Cognitive Services team deprecated Bing Speech and no new subscriptions can be created. Web Chat currently only support Bing Speech but not Speech Services. If you already have a Bing Speech subscription, you can continue to use speech in Web Chat until 2019 October. If you do not have a Bing Speech subscription, you will need to wait until this work is done.

What if my question is not in this FAQs?

Please post it in the comment below. We will try our best to answer it and put it up here for readability.

References

• Quickstart on Speech SDK in a browser
• Download Speech SDK version
• Speech SDK API

[andersponders]

i would like to help with this, have you guys found the source for the new SDK? when are they gonna throw it on NPM?

I am unable to find anywhere where i can get a subscription key to the old API, so it looks like i can't have voice in my azure bot until this issue is fixed, so I would like to help get it done asap

[compulim]

Their new SDK is on a ZIP file now, and it's a Webpack bundle. I am working on it in another repo and progress is pretty good. Speech recognition works as expected (plus automated tests), and now working on speech synthesis.

I am targeting 12/15 for a development build. I probably can't release as a production build until they release their non-bundled code on NPM.

I have updated the FAQs above based on your question. I am sorry to say you will need to wait until this work is done.

[corinagum]

This bug is to bring the speech adapter for v4 on par with current v3. On par is on track for 4.2.
Other features in addition to this will be added to a bug tracked for 4.3. You can view these items on #1400