Report encode/decode time and implementation metric #2434

shi-su · 2022-09-13T17:35:42Z

add implementation name as a dimension.

Issue #:

Description of changes:
Report encode/decode time and if encoder/decoder is in hardware, and add encoder/decoder name as a metric dimension

Testing:
Added unit tests to cover the changes

Can these tested using a demo application? Please provide reproducible step-by-step instructions.
Join a meeting with Chrome and select log level as DEBUG, turn on video on one user. Use the following string to filter console logs [DEBUG] SDK - sending:. Verify the log contains the following fields on sender and receiver side, respectively.

Sender side:

Message has VIDEO_ENCODE_MS metric field. Example:

{"type":"VIDEO_ENCODE_MS","value":49.850448654035056}

Message has VIDEO_ENCODER_NAME dimension. Example:

"dimensions":[{"type":"VIDEO_ENCODER_NAME","value":{"stringValue":"OpenH264"}}]

Message example:

2022-09-20T23:25:14.763Z [DEBUG] SDK - sending: {"timestampMs":"1663716314763","type":"CLIENT_METRIC","clientMetric":{"globalMetrics":[{"type":"STUN_RTT_MS","value":83},{"type":"SOCKET_DISCARDED_PPS","value":0},{"type":"VIDEO_AVAILABLE_SEND_BANDWIDTH","value":2200176}],"streamMetricFrames":[{"metrics":[{"type":"RTC_SPK_JITTER_MS","value":5},{"type":"RTC_SPK_FRACTION_PACKET_LOST_PERCENT","value":0},{"type":"RTC_SPK_PPS","value":49},{"type":"RTC_SPK_BITRATE","value":25547}]},{"metrics":[{"type":"RTC_MIC_PPS","value":49},{"type":"RTC_MIC_BITRATE","value":24263},{"type":"RTC_MIC_JITTER_MS","value":20},{"type":"RTC_MIC_FRACTION_PACKET_LOST_PERCENT","value":0},{"type":"RTC_MIC_RTT_MS","value":62}]},{"metrics":[{"type":"VIDEO_SENT_PPS","value":161},{"type":"VIDEO_SENT_BITRATE","value":1420115},{"type":"VIDEO_ENCODE_FPS","value":14},{"type":"VIDEO_ENCODE_MS","value":49.850448654035056},{"type":"VIDEO_ENCODE_WIDTH","value":960},{"type":"VIDEO_ENCODE_HEIGHT","value":540},{"type":"VIDEO_INPUT_FPS","value":15},{"type":"VIDEO_FIRS_RECEIVED","value":0},{"type":"VIDEO_PLIS_RECEIVED","value":0},{"type":"VIDEO_NACKS_RECEIVED","value":0},{"type":"VIDEO_SENT_QP_SUM","value":231},{"type":"VIDEO_SENT_FRACTION_PACKET_LOST_PERCENT","value":0},{"type":"VIDEO_SENT_RTT_MS","value":92}],"dimensions":[{"type":"VIDEO_ENCODER_NAME","value":{"stringValue":"OpenH264"}}]}]}}

Receiver side:

Message has VIDEO_DECODE_MS metric field. Example:

{"type":"VIDEO_DECODE_MS","value":40.120361083249755}

Message has VIDEO_DECODER_NAME dimension. Example:

"dimensions":[{"type":"VIDEO_DECODER_NAME","value":{"stringValue":"ExternalDecoder (VDAVideoDecoder)"}}]

Message example:

2022-09-20T23:21:21.581Z [DEBUG] SDK - sending: {"timestampMs":"1663716081581","type":"CLIENT_METRIC","clientMetric":{"globalMetrics":[{"type":"STUN_RTT_MS","value":70},{"type":"SOCKET_DISCARDED_PPS","value":0},{"type":"VIDEO_AVAILABLE_SEND_BANDWIDTH","value":300000},{"type":"VIDEO_AVAILABLE_RECEIVE_BANDWIDTH","value":1335167}],"streamMetricFrames":[{"metrics":[{"type":"RTC_SPK_JITTER_MS","value":3},{"type":"RTC_SPK_FRACTION_PACKET_LOST_PERCENT","value":0},{"type":"RTC_SPK_PPS","value":49},{"type":"RTC_SPK_BITRATE","value":19875}]},{"streamId":1,"metrics":[{"type":"VIDEO_RECEIVED_JITTER_MS","value":41},{"type":"VIDEO_RECEIVED_FRACTION_PACKET_LOST_PERCENT","value":0},{"type":"VIDEO_RECEIVED_PPS","value":149},{"type":"VIDEO_RECEIVED_BITRATE","value":1315979},{"type":"VIDEO_RECEIVED_FPS","value":75},{"type":"VIDEO_DECODE_FPS","value":15},{"type":"VIDEO_DECODE_MS","value":40.120361083249755},{"type":"VIDEO_FIRS_SENT","value":0},{"type":"VIDEO_PLIS_SENT","value":0},{"type":"VIDEO_NACKS_SENT","value":0},{"type":"VIDEO_DECODE_WIDTH","value":960},{"type":"VIDEO_DECODE_HEIGHT","value":540}],"dimensions":[{"type":"VIDEO_DECODER_NAME","value":{"stringValue":"ExternalDecoder (VDAVideoDecoder)"}}]},{"metrics":[{"type":"RTC_MIC_PPS","value":50},{"type":"RTC_MIC_BITRATE","value":25853},{"type":"RTC_MIC_JITTER_MS","value":19.25},{"type":"RTC_MIC_FRACTION_PACKET_LOST_PERCENT","value":0},{"type":"RTC_MIC_RTT_MS","value":63}]}]}}

Checklist:

Have you successfully run npm run build:release locally?
y
Do you add, modify, or delete public API definitions? If yes, has that been reviewed and approved?
n
Do you change the wire protocol, e.g. the request method? If yes, has that been reviewed and approved?
n

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

hensmi-amazon

lgtm just some naming nits

hensmi-amazon · 2022-09-14T21:34:54Z

src/clientmetricreport/GlobalMetricReport.ts

@@ -4,4 +4,5 @@
 export default class GlobalMetricReport {
  previousMetrics: { [id: string]: number } = {};
  currentMetrics: { [id: string]: number } = {};
+  stringValues: { [id: string]: string } = {};


maybe currentDimensions or just dimensions instead of stringValues?

I do not think they are just dimension, it can also be metric derived from string value. For example, whether decoder/encoder is in hardware is one of the derived metric.

Updated it to currentStringMetrics. Also added comments that this is a workaround in case metric values are strings.

hensmi-amazon · 2022-09-14T21:38:00Z

src/clientmetricreport/StreamMetricReport.ts

@@ -10,4 +10,5 @@ export default class StreamMetricReport {
  direction: Direction;
  previousMetrics: { [id: string]: number } = {};
  currentMetrics: { [id: string]: number } = {};
+  stringValues: { [id: string]: string } = {};


Updated it to currentStringMetrics. Also added comments that this is a workaround in case metric values are strings.

hensmi-amazon · 2022-09-14T21:39:25Z

src/clientmetricreport/ClientMetricReport.ts

@@ -297,6 +333,14 @@ export default class ClientMetricReport {
    },
    frameHeight: { transform: this.identityValue, type: SdkMetric.Type.VIDEO_DECODE_HEIGHT },
    frameWidth: { transform: this.identityValue, type: SdkMetric.Type.VIDEO_DECODE_WIDTH },
+    totalDecodeTime: {


should this name be changed?

This key is just about the field name in webrtc report where the metric is derived from, we cannot change that naming as it comes from webrtc..

hensmi-amazon · 2022-09-14T21:41:45Z

src/clientmetricreport/ClientMetricReport.ts

+  /**
+   *  Dimensions derived from metric
+   */
+  readonly streamMetricDimensionMap: {


Nit: Don't think we need 'Metric' in any of this or related naming?

Updated naming as suggested.

add implementation name as a dimension.

hensmi-amazon

lgtm

ltrung · 2022-09-16T22:07:20Z

src/clientmetricreport/ClientMetricReport.ts

@@ -126,6 +126,35 @@ export default class ClientMetricReport {
    return Number(metricReport.currentMetrics[metricName] * 1000);
  };

+  millisecondsPerSecond = (metricName?: string, ssrc?: number): number => {


Can we name this to be clearer what it does? milliseconds per second is kinda confusing

Thanks for the comment. Created #2443 to update function naming

Report encode/decode time and if encoder/decoder is in hardware, and add encoder/decoder name as a metric dimension Co-authored-by: Shi Su <[email protected]>

shi-su force-pushed the report_hw branch 2 times, most recently from 2103581 to c731c87 Compare September 13, 2022 17:56

shi-su requested a review from hensmi-amazon September 13, 2022 18:40

shi-su marked this pull request as ready for review September 13, 2022 20:25

shi-su requested a review from a team as a code owner September 13, 2022 20:25

chime-sdk-js-bot previously approved these changes Sep 13, 2022

View reviewed changes

hensmi-amazon reviewed Sep 14, 2022

View reviewed changes

shi-su dismissed chime-sdk-js-bot’s stale review via 0cbf2a9 September 14, 2022 22:45

Shi Su added 2 commits September 14, 2022 22:48

Report encode/decode time and if encoder/decoder is in hardware. Also

6f3e24a

add implementation name as a dimension.

Fix naming

23e6a99

shi-su force-pushed the report_hw branch from 0cbf2a9 to 23e6a99 Compare September 14, 2022 22:48

hensmi-amazon approved these changes Sep 14, 2022

View reviewed changes

shi-su merged commit 9d7d94e into main Sep 16, 2022

shi-su deleted the report_hw branch September 16, 2022 19:39

ltrung reviewed Sep 16, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Report encode/decode time and implementation metric #2434

Report encode/decode time and implementation metric #2434

shi-su commented Sep 13, 2022 •

edited

Loading

hensmi-amazon left a comment

hensmi-amazon Sep 14, 2022

shi-su Sep 14, 2022

shi-su Sep 14, 2022

hensmi-amazon Sep 14, 2022

shi-su Sep 14, 2022

hensmi-amazon Sep 14, 2022

shi-su Sep 14, 2022

hensmi-amazon Sep 14, 2022

shi-su Sep 14, 2022

hensmi-amazon left a comment

ltrung Sep 16, 2022

shi-su Sep 16, 2022

Report encode/decode time and implementation metric #2434

Report encode/decode time and implementation metric #2434

Conversation

shi-su commented Sep 13, 2022 • edited Loading

hensmi-amazon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hensmi-amazon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shi-su commented Sep 13, 2022 •

edited

Loading