Capture unmixed individual attendees audio streams

[abettke]

What are you trying to do?

We are looking for a way to additionally capture the audio streams before they are mixed into the main meeting audio channel. This is so that each speaker's audio can be recorded in isolation from other attendees, similar to how Chime is doing this with the live transcription. Is this officially supported by the SDK? If not, is the a place in the source code where we can hook in to grab the individual attendees audio stream manually?

How can the documentation be improved to help your use case?

• Point to where in the documentation this API exists.
• A demo example of how this can be done.

What documentation have you looked at so far?

• Amazon Chime SDK Developer Guide
• Amazon Chime SDK API Reference
• Github Issues

[avinashmidathada]

Hello, the SDK currently doesn't support receiving unmixed audio streams, but if you're interested in audio capture of the unmixed streams we recently launched support to publish to Kinesis Video Streams using Amazon Chime SDK media pipelines. You can find the developer guide for this feature here.

Please feel free to ask any questions.

[euro-bYte]

I have a question to add to this - How do I know where the audio is and how to get them? The docs point to using the GetMedia API, but that gets the chunks of data that is being received. Does KVS concat the chunks like the CreateMediaConcatenationPipeline API? Or do we need to setup KVS to point to a S3 bucket?

BTW thanks for the links to the docs, very helpful 😄

[avinashmidathada]

Hello @euro-bYte, we publish events to your EventBus when we start/stop streaming an attendee's audio to KVS. The "Using Event Bridge notifications" section of the documentation points to the sample events for your reference.

We currently don't have an API to concatenate the chunks from your KVS stream. There's also no provision from KVS to directly write to an S3 bucket, but they do provide sample libraries that you can refer to. Here's one for python: https://github.com/aws-samples/amazon-kinesis-video-streams-consumer-library-for-python

If you're only interested in processing the audio after the meeting ends, you can just listen to the Amazon Chime Media Stream Pipeline Kinesis Video Stream End event and use the GetMediaForFragmentList API. Note that, you need to first get the list of fragments using the ListFragments API. All the required parameters for doing this are present in the event payload we publish.

[euro-bYte]

@avinashmidathada Thank you for the quick answer!

[euro-bYte]

@avinashmidathada I am hoping you can help me answer one last question this is a bit more technical.

I implemented the APIs needed to get the fragments once the meeting ends, but the GetMediaForFragmentList API returns a payload and when writing to a file is empty.

Here is the basic flow:

1. Stream Pipeline Kinesis Video Stream End event is received
2. I create a data endpoint and instantiate a KVSArchiveMediaService class

    const listFragmentURL = await getKVSDataEndpoint(detailedInfo.kinesisVideoStreamArn, 'LIST_FRAGMENTS')
    const archiveClient = new KVSArchiveMediaService(listFragmentURL)

3. I get the list of fragments

const fragmentResponse = await archiveClient.getListOfFragments(
        detailedInfo.kinesisVideoStreamArn,
        detailedInfo.startTime,
        detailedInfo.endTime,
        nextToken
      )

4. Then I filter the Fragment array to only contain the FragmentNumber property
5. I create another endpoint and create another instance of KVSArchiveMediaService class to use the GetMediaForFragmentList API

const getMediaURL = await getKVSDataEndpoint(detailedInfo.kinesisVideoStreamArn, 'GET_MEDIA_FOR_FRAGMENT_LIST')
    const mediaClient = new KVSArchiveMediaService(getMediaURL)
    const payloadReponse = await mediaClient.getMediaForFragmentList(
      detailedInfo.kinesisVideoStreamArn,
      fragmentArr
    )

7. I proceed to write to a file which ends up empty

      const filePath = './downloadedMedia'
      const fileStream = fs.createWriteStream(filePath)
      const readableStream = payloadReponse.Payload.pipe(fileStream)

8. I close the KVS connection with the destroy method

What am I doing incorrectly or missing in this flow?

[avinashmidathada]

Hey @euro-bYte Can you confirm the following:

1. Can you first check Cloudwatch metrics for the Stream ARN and validate that PutMedia.IncomingBytesis a non-zero value? This will confirm whether there was any issue with data being published to KVS from the meeting.
2. Can you check if the fragments list returned by the getListOfFragments call is non-empty? And if yes can you sort the fragment numbers in ascending order before you make the getMediaForFragmentList call?

[justinIs]

Hi @avinashmidathada will there be any support for video with KVS Media Stream Pipelines? Is there any reason why video isn't supported now?

How would you recommend collecting combined video and audio on a per user basis for archiving? It seems like we could use Media Capture to get individual video, but mixed audio. KVS integration does only audio. Would it make sense to use the two and then use ffmpeg to combine the audio and video?

[euro-bYte]

@avinashmidathada The problem was the way I was processing the payloadReponse.Payload into a file. Everything else works as you suggested. Thank you!

2. Can you check if the fragments list returned by the getListOfFragments call is non-empty? And if yes can you sort the fragment numbers in ascending order before you make the getMediaForFragmentList call?

In my implementation I ordered it by the serverTimestamp does that matter? (it still plays fine after transcoding it)