The Amazon Chime SDK for JavaScript, in conjunction with WebRTC, allows you to configure quality and bandwidth options for meeting sessions. It provides both automatic and manual controls to adapt to changing network conditions. Finding the right balance of quality and performance can be challenging when either the client's network or CPU is constrained. As network reliability and available bandwidth decrease, quality and functionality trade-offs must be made in real time to preserve the viability of the meeting. This guide gives an overview of challenges associated with devices and networks, degradation detection mechanisms, and mitigations for various levels of degradation.
- Quality, Bandwidth, and Connectivity
When designing an application that shares audio and video you should determine which functionality is most important to your end users. Just as responsive web design allows a single web application to adapt to changing screen sizes, the audio-video component of your application can also be responsive to changing device and network conditions. For example, in a collaborative meeting it may be acceptable to gracefully degrade to an audio-only experience. However, in a presentation where the screen is shared, the presenter audio and screen share video may be the most important.
To join an Amazon Chime SDK meeting, each client traverses the public internet to connect to the media services for that meeting which are hosted by the AWS Global Infrastructure in one of the available SDK media regions. The following key factors influence the client experience:
- Application performance: if your client application is consuming a lot of CPU, then it may hinder the device from simultaneously processing media and transmitting and receiving network packets.
- Client device performance: transcoding audio and video in real-time is CPU intensive and may both decrease performance and further constrain the device's network adapter. Browsers may or may not support hardware acceleration depending on the codec profile and device hardware. Furthermore, browsers do not allow you to directly monitor CPU usage, so it can be difficult to tell whether CPU consumption is a factor in performance issues for your end-user devices.
- Differences in uplink and downlink: In order for an attendee to send audio or video to a remote attendee, both the sender's uplink and the receiver's downlink must have enough bandwidth. However, most end users will be on a network that has more downlink (receiving) bandwidth than uplink (sending) bandwidth. It can be challenging to determine whether a drop in quality is due to constraints of the sender, the receiver, or both.
- Last mile network connectivity: issues with WiFi, local area network, or the internet service provider may limit the effective bandwidth due to packet loss, bandwidth caps, or other hardware limitations.
- Complexity and length of the network path: meeting attendees that are geographically distant from the SDK media region may experience higher packet loss, jitter, and round-trip times due to the number of intermediate networks, which may vary in quality.
The Amazon Chime SDK for JavaScript produces several kinds of events on the AudioVideoObserver to monitor connectivity and quality. Use the following events and key health metrics to monitor the performance of the meeting session in real time. For code snippets showing how to subscribe to these events, see Monitoring and Alerts.
Metrics derived from WebRTC stats are not guaranteed to be present in all browsers. In such cases the value may be missing.
For the browser support columns below, "All" refers to the browsers officially supported by the Chime SDK.
| Event | Notes | Browsers |
|---|---|---|
| connectionDidSuggestStopVideo | Indicates that the audio connection is experiencing packet loss. Stopping local video and pausing remote video tiles may help the connection recover by reducing CPU usage and network consumption. | All |
| connectionDidBecomeGood | Indicates that the audio connection has improved. | All |
| connectionDidBecomePoor | Similar to the previous metric, but is fired when local video is already turned off. | All |
| Event | Notes | Browsers |
|---|---|---|
| realtimeSubscribeToVolumeIndicator | The signalStrength field indicates whether the server is receiving the remote attendee's audio. A value of 1 indicates a good connection, a value of 0.5 or 0 indicates some or total packet loss. Since each attendee receives the signal strength for all attendees, this event can be used to monitor the ability of attendees to share their audio in real-time. |
All |
| Event | Notes | Browsers |
|---|---|---|
| metricsDidReceive | Exposes the WebRTC getStats metrics. There may be differences among browsers as to which metrics are reported. | All |
Note: We have built a video WebRTC statistics widget in the demo browser meeting app. This widget is shown as an overlay on the video, when the video is enabled.
You get the following WebRTC stats in the video stats widget:
Upstream video information: Frame height, Frame width, Bitrate (bps), Packets Sent, Frame Rate (fps), Audio Jitter.
Downstream video information: Frame height, Frame width, Bitrate (bps), Packet Loss (%), Frame Rate (fps).
You can also use this event to get the raw
RTCStatsReportthroughclientMetricReport.getRTCStatsReport()You can check the implementation in the demo app to build your own custom widget (look for
getObservableVideoMetricsmethod in the demo). This video stats widget is built using the getObservableVideoMetrics and the metricsDidReceive APIs. Through the information provided in these stats, the application can monitor key attributes and take action. For instance if the bitrate or resolution falls below a certain threshold, the user could be notified in some manner, or diagnostic reporting could take place.
| Event | Notes | Browsers |
|---|---|---|
| encodingSimulcastLayersDidChange | Called when simulcast is enabled and simulcast uplink encoding layers are changed. Check the simulcast guide for more information on simulcast. | Chromium-based |
WebRTC will automatically reduce video frame rate, resolution, and bandwidth if it detects that it is unable to send video at the specified rate due to bandwidth or CPU.
Use the browser's built-in developer tools to profile your application and determine whether there are any hotspots. When handling real-time events (prefixed with realtime) ensure that you are doing as little processing as possible. See the API Overview section on building a roster for more information. In particular, look out for expensive DOM updates (such as when manipulating the roster or video tile layout).
When possible, profile on devices that have similar performance characteristics to the ones you expect to be used by your end users.
Sometimes it is better to sacrifice video quality in order to prioritize audio. You can call chooseVideoInputQuality(width, height, frameRate) and setVideoMaxBandwidthKbps(maxBandwidthKbps) to lower the maximum bandwidth in real-time. You can also adjust the resolution and frame rate if you call the method before starting local video (or stop and then restart the local video). See the section below on values you can use for chooseVideoInputQuality.
Calling pauseVideoTile on remote video tiles will reduce the amount of CPU consumed due to decoding remote attendee video.
In the absence of packet loss, keep in mind that the sender uplink and receiver downlink consume the same bandwidth for each video tile. Mitigations affecting one sender's uplink can benefit all receiver's downlinks. This means that in order to help receiver's, sometimes the best course of action is to lower the bandwidth consumed by the sender.
You can choose a video quality of up to 1280x720 (720p) at 30 fps using chooseVideoInputQuality(width, height, frameRate) and setVideoMaxBandwidthKbps(maxBandwidthKbps) APIs before the meeting session begins. However, in some cases it is not necessary to send the highest quality and you can use a lower values. For example, if the size of the video tile is small then the highest quality may not be worth the additional bandwidth and CPU overhead.
Example:
// 360p
meetingSession.audioVideo.chooseVideoInputQuality(640, 360, 15);
meetingSession.audioVideo.setVideoMaxBandwidthKbps(600);
// 540p
meetingSession.audioVideo.chooseVideoInputQuality(960, 540, 15);
meetingSession.audioVideo.setVideoMaxBandwidthKbps(1400);
// 720p
meetingSession.audioVideo.chooseVideoInputQuality(1280, 720, 15);
meetingSession.audioVideo.setVideoMaxBandwidthKbps(1400);The default resolution in the SDK is 540p at 15 fps and 1400 Kbps. Lower resolutions can be set if you anticipate a low bandwidth situation. Browser and codec support for very low resolutions may vary.
The value maxBandwidthKbps is a recommendation you make to WebRTC to use as the upper limit for upstream sending bandwidth. The Chime SDK default is 1400 Kbps for this value. The following table provides recommendations of minimum bandwidth value per resolution for typical video-conferencing scenarios. Note that when low values are selected the video can be appear pixelated. Using 15 fps instead of 30 fps will substantially decrease the required bit rate and may be acceptable for low-motion content.
| Resolution | Frame Rate Per Sec | Min Kbps |
|---|---|---|
| 180p | 30 | 100 |
| 360p | 30 | 250 |
| 480p | 30 | 400 |
| 540p | 30 | 500 |
| 720p | 30 | 1400 |
Setting a frame rate below 15 is not recommend and will cause the video to appear jerky and will not significantly improve the bandwidth consumed since key frames will still be sent occasionally. It would be better to adjust the resolution than set a very low frame rate.
In some situations it may be best to turn off the local video tile to improve audio uplink, CPU consumption, or remote attendee downlink.
You can also observe the connectionDidSuggestStopVideo event to monitor audio packet loss and use that as a cue to turn off local video using stopLocalVideoTile.
The SDK by default uses the NScaleVideoUplinkBandwidthPolicy which monitors number of participants in the meeting and automatically scales down the maxBandwidthKbps as the number of remote video tiles increases. This can be customized by implementing a VideoUplinkBandwidth Policy and setting it in the MeetingSessionConfiguration class.
You can use the active speaker detector to show only the video of the active speakers and pause other videos.
By default the SDK uses the AllHighestVideoBandwidthPolicy which will not adapt to local receive network conditions and always subscribes to the highest quality video of all participant, but VideoPriorityBasedPolicy will downgrade or pause videos in response to network constraints. Additional configuration of that policy is required to select which remote videos to subscribe to, see the guide for more information. Downgrades are only possible if simulcast or SVC is enabled.
You can customize the default downlink policy by setting the VideoDownlinkBandwidthPolicy in MeetingSessionConfiguration class.
Simulcast is a standardized technique in WebRTC to enhance video quality in a multi-party call. Amazon Chime SDK for JavaScript provides a video simulcast solution for Chromium-based browsers. To use simulcast, please refer to the documentation here.