Skip to content

Commit

Permalink
Report encode/decode time and implementation metric (#2434)
Browse files Browse the repository at this point in the history 
Report encode/decode time and if encoder/decoder is in hardware, and add encoder/decoder name as a metric dimension

Co-authored-by: Shi Su <[email protected]>
  • Loading branch information
@shi-su @manasisurve
2 people authored and manasisurve committed Sep 19, 2022
1 parent ae297c9 commit 33b8fa3
Show file tree
Hide file tree
Showing 11 changed files with 3,165 additions and 2,743 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added

- Add encoder/decoder is in hardware metric and stream dimension to signaling protocol
- Report encode/decode time and if encoder/decoder is in hardware, and add encoder/decoder name as a metric dimension

### Removed

Expand Down
5,482 changes: 2,759 additions & 2,723 deletions docs/assets/js/search.js
View file

Large diffs are not rendered by default.

145 changes: 129 additions & 16 deletions docs/classes/clientmetricreport.html
View file

Large diffs are not rendered by default.

22 changes: 22 additions & 0 deletions docs/classes/globalmetricreport.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ <h3>Constructors</h3>
<h3>Properties</h3>
<ul class="tsd-index-list">
<li class="tsd-kind-property tsd-parent-kind-class"><a href="globalmetricreport.html#currentmetrics" class="tsd-kind-icon">current<wbr>Metrics</a></li>
<li class="tsd-kind-property tsd-parent-kind-class"><a href="globalmetricreport.html#currentstringmetrics" class="tsd-kind-icon">current<wbr>String<wbr>Metrics</a></li>
<li class="tsd-kind-property tsd-parent-kind-class"><a href="globalmetricreport.html#previousmetrics" class="tsd-kind-icon">previous<wbr>Metrics</a></li>
</ul>
</section>
Expand Down Expand Up @@ -130,6 +131,24 @@ <h5><span class="tsd-signature-symbol">[</span>id: <span class="tsd-signature-ty
</ul>
</div>
</section>
<section class="tsd-panel tsd-member tsd-kind-property tsd-parent-kind-class">
<a name="currentstringmetrics" class="tsd-anchor"></a>
<h3>current<wbr>String<wbr>Metrics</h3>
<div class="tsd-signature tsd-kind-icon">current<wbr>String<wbr>Metrics<span class="tsd-signature-symbol">:</span> <span class="tsd-signature-symbol">{}</span><span class="tsd-signature-symbol"> = {}</span></div>
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/clientmetricreport/GlobalMetricReport.ts#L8">src/clientmetricreport/GlobalMetricReport.ts:8</a></li>
</ul>
</aside>
<div class="tsd-type-declaration">
<h4>Type declaration</h4>
<ul class="tsd-parameters">
<li class="tsd-parameter-index-signature">
<h5><span class="tsd-signature-symbol">[</span>id: <span class="tsd-signature-type">string</span><span class="tsd-signature-symbol">]: </span><span class="tsd-signature-type">string</span></h5>
</li>
</ul>
</div>
</section>
<section class="tsd-panel tsd-member tsd-kind-property tsd-parent-kind-class">
<a name="previousmetrics" class="tsd-anchor"></a>
<h3>previous<wbr>Metrics</h3>
Expand Down Expand Up @@ -171,6 +190,9 @@ <h5><span class="tsd-signature-symbol">[</span>id: <span class="tsd-signature-ty
<li class=" tsd-kind-property tsd-parent-kind-class">
<a href="globalmetricreport.html#currentmetrics" class="tsd-kind-icon">current<wbr>Metrics</a>
</li>
<li class=" tsd-kind-property tsd-parent-kind-class">
<a href="globalmetricreport.html#currentstringmetrics" class="tsd-kind-icon">current<wbr>String<wbr>Metrics</a>
</li>
<li class=" tsd-kind-property tsd-parent-kind-class">
<a href="globalmetricreport.html#previousmetrics" class="tsd-kind-icon">previous<wbr>Metrics</a>
</li>
Expand Down
22 changes: 22 additions & 0 deletions docs/classes/streammetricreport.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ <h3>Constructors</h3>
<h3>Properties</h3>
<ul class="tsd-index-list">
<li class="tsd-kind-property tsd-parent-kind-class"><a href="streammetricreport.html#currentmetrics" class="tsd-kind-icon">current<wbr>Metrics</a></li>
<li class="tsd-kind-property tsd-parent-kind-class"><a href="streammetricreport.html#currentstringmetrics" class="tsd-kind-icon">current<wbr>String<wbr>Metrics</a></li>
<li class="tsd-kind-property tsd-parent-kind-class"><a href="streammetricreport.html#direction" class="tsd-kind-icon">direction</a></li>
<li class="tsd-kind-property tsd-parent-kind-class"><a href="streammetricreport.html#mediatype" class="tsd-kind-icon">media<wbr>Type</a></li>
<li class="tsd-kind-property tsd-parent-kind-class"><a href="streammetricreport.html#previousmetrics" class="tsd-kind-icon">previous<wbr>Metrics</a></li>
Expand Down Expand Up @@ -133,6 +134,24 @@ <h5><span class="tsd-signature-symbol">[</span>id: <span class="tsd-signature-ty
</ul>
</div>
</section>
<section class="tsd-panel tsd-member tsd-kind-property tsd-parent-kind-class">
<a name="currentstringmetrics" class="tsd-anchor"></a>
<h3>current<wbr>String<wbr>Metrics</h3>
<div class="tsd-signature tsd-kind-icon">current<wbr>String<wbr>Metrics<span class="tsd-signature-symbol">:</span> <span class="tsd-signature-symbol">{}</span><span class="tsd-signature-symbol"> = {}</span></div>
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/clientmetricreport/StreamMetricReport.ts#L14">src/clientmetricreport/StreamMetricReport.ts:14</a></li>
</ul>
</aside>
<div class="tsd-type-declaration">
<h4>Type declaration</h4>
<ul class="tsd-parameters">
<li class="tsd-parameter-index-signature">
<h5><span class="tsd-signature-symbol">[</span>id: <span class="tsd-signature-type">string</span><span class="tsd-signature-symbol">]: </span><span class="tsd-signature-type">string</span></h5>
</li>
</ul>
</div>
</section>
<section class="tsd-panel tsd-member tsd-kind-property tsd-parent-kind-class">
<a name="direction" class="tsd-anchor"></a>
<h3>direction</h3>
Expand Down Expand Up @@ -204,6 +223,9 @@ <h3>stream<wbr>Id</h3>
<li class=" tsd-kind-property tsd-parent-kind-class">
<a href="streammetricreport.html#currentmetrics" class="tsd-kind-icon">current<wbr>Metrics</a>
</li>
<li class=" tsd-kind-property tsd-parent-kind-class">
<a href="streammetricreport.html#currentstringmetrics" class="tsd-kind-icon">current<wbr>String<wbr>Metrics</a>
</li>
<li class=" tsd-kind-property tsd-parent-kind-class">
<a href="streammetricreport.html#direction" class="tsd-kind-icon">direction</a>
</li>
Expand Down
64 changes: 62 additions & 2 deletions src/clientmetricreport/ClientMetricReport.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// SPDX-License-Identifier: Apache-2.0

import Logger from '../logger/Logger';
import { SdkMetric } from '../signalingprotocol/SignalingProtocol.js';
import { SdkMetric, SdkStreamDimension } from '../signalingprotocol/SignalingProtocol.js';
import VideoStreamIndex from '../videostreamindex/VideoStreamIndex';
import Direction from './ClientMetricReportDirection';
import MediaType from './ClientMetricReportMediaType';
Expand Down Expand Up @@ -126,6 +126,35 @@ export default class ClientMetricReport {
return Number(metricReport.currentMetrics[metricName] * 1000);
};

millisecondsPerSecond = (metricName?: string, ssrc?: number): number => {
const metricReport = ssrc ? this.streamMetricReports[ssrc] : this.globalMetricReport;
let intervalSeconds = (this.currentTimestampMs - this.previousTimestampMs) / 1000;
if (intervalSeconds <= 0) {
return 0;
}
if (this.previousTimestampMs <= 0) {
intervalSeconds = 1;
}
const diff =
metricReport.currentMetrics[metricName] - (metricReport.previousMetrics[metricName] || 0);
if (diff <= 0) {
return 0;
}
return (diff * 1000) / intervalSeconds;
};

isHardwareImplementation = (metricName?: string, ssrc?: number): number => {
const metricReport = this.streamMetricReports[ssrc];
const implName = String(metricReport.currentStringMetrics[metricName]);
const hasHwName =
implName.includes('ExternalDecoder') ||
implName.includes('ExternalEncoder') ||
implName.includes('EncodeAccelerator') ||
implName.includes('DecodeAccelerator');
const isFallback = implName.includes('fallback from');
return hasHwName && !isFallback ? 1 : 0;
};

/**
* Canonical and derived metric maps
*/
Expand Down Expand Up @@ -252,6 +281,14 @@ export default class ClientMetricReport {
jitter: {
transform: this.secondsToMilliseconds,
},
totalEncodeTime: {
transform: this.millisecondsPerSecond,
type: SdkMetric.Type.VIDEO_ENCODE_MS,
},
encoderImplementation: {
transform: this.isHardwareImplementation,
type: SdkMetric.Type.VIDEO_ENCODER_IS_HARDWARE,
},
};

readonly videoDownstreamMetricMap: {
Expand All @@ -261,7 +298,6 @@ export default class ClientMetricReport {
source?: string;
};
} = {
totalDecodeTime: { transform: this.identityValue, type: SdkMetric.Type.VIDEO_DECODE_MS },
packetsReceived: { transform: this.countPerSecond, type: SdkMetric.Type.VIDEO_RECEIVED_PPS },
packetsLost: {
transform: this.packetLossPercent,
Expand Down Expand Up @@ -297,6 +333,14 @@ export default class ClientMetricReport {
},
frameHeight: { transform: this.identityValue, type: SdkMetric.Type.VIDEO_DECODE_HEIGHT },
frameWidth: { transform: this.identityValue, type: SdkMetric.Type.VIDEO_DECODE_WIDTH },
totalDecodeTime: {
transform: this.millisecondsPerSecond,
type: SdkMetric.Type.VIDEO_DECODE_MS,
},
decoderImplementation: {
transform: this.isHardwareImplementation,
type: SdkMetric.Type.VIDEO_DECODER_IS_HARDWARE,
},
};

getMetricMap(
Expand Down Expand Up @@ -329,6 +373,22 @@ export default class ClientMetricReport {
}
}

/**
* Dimensions derived from metric
*/
readonly streamDimensionMap: {
[id: string]: SdkStreamDimension.Type;
} = {
encoderImplementation: SdkStreamDimension.Type.VIDEO_ENCODER_NAME,
decoderImplementation: SdkStreamDimension.Type.VIDEO_DECODER_NAME,
};

getStreamDimensionMap(): {
[id: string]: SdkStreamDimension.Type;
} {
return this.streamDimensionMap;
}

/**
* media Stream metrics
*/
Expand Down
2 changes: 2 additions & 0 deletions src/clientmetricreport/GlobalMetricReport.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,6 @@
export default class GlobalMetricReport {
previousMetrics: { [id: string]: number } = {};
currentMetrics: { [id: string]: number } = {};
// As metric values do not necessarily be number, this is a workaround in case metric value is string
currentStringMetrics: { [id: string]: string } = {};
}
2 changes: 2 additions & 0 deletions src/clientmetricreport/StreamMetricReport.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,6 @@ export default class StreamMetricReport {
direction: Direction;
previousMetrics: { [id: string]: number } = {};
currentMetrics: { [id: string]: number } = {};
// As metric values do not necessarily be number, this is a workaround in case metric value is string
currentStringMetrics: { [id: string]: string } = {};
}
36 changes: 34 additions & 2 deletions src/statscollector/StatsCollector.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@ import IntervalScheduler from '../scheduler/IntervalScheduler';
import SignalingClient from '../signalingclient/SignalingClient';
import {
SdkClientMetricFrame,
SdkDimensionValue,
SdkMetric,
SdkStreamDimension,
SdkStreamMetricFrame,
} from '../signalingprotocol/SignalingProtocol.js';
import { Maybe } from '../utils/Types';
Expand Down Expand Up @@ -253,8 +255,16 @@ export default class StatsCollector {

for (const rawMetric in rawMetricReport) {
if (rawMetric in metricMap) {
metricReport.previousMetrics[rawMetric] = metricReport.currentMetrics[rawMetric];
metricReport.currentMetrics[rawMetric] = rawMetricReport[rawMetric];
if (typeof rawMetricReport[rawMetric] === 'number') {
metricReport.previousMetrics[rawMetric] = metricReport.currentMetrics[rawMetric];
metricReport.currentMetrics[rawMetric] = rawMetricReport[rawMetric];
} else if (typeof rawMetricReport[rawMetric] === 'string') {
metricReport.currentStringMetrics[rawMetric] = rawMetricReport[rawMetric];
} else {
this.logger.error(
`Unknown metric value type ${typeof rawMetricReport[rawMetric]} for metric ${rawMetric}`
);
}
}
}
}
Expand Down Expand Up @@ -300,6 +310,26 @@ export default class StatsCollector {
this.clientMetricReport.print();
}

/**
* Add stream metric dimension frames derived from metrics
*/
private addStreamMetricDimensionFrames(
streamMetricFrame: SdkStreamMetricFrame,
streamMetricReport: StreamMetricReport
): void {
const streamDimensionMap = this.clientMetricReport.getStreamDimensionMap();
for (const metricName in streamMetricReport.currentStringMetrics) {
if (metricName in streamDimensionMap) {
const dimensionFrame = SdkStreamDimension.create();
dimensionFrame.type = streamDimensionMap[metricName];
const dimensionValue = SdkDimensionValue.create();
dimensionValue.stringValue = streamMetricReport.currentStringMetrics[metricName];
dimensionFrame.value = dimensionValue;
streamMetricFrame.dimensions.push(dimensionFrame);
}
}
}

/**
* Packages a metric into the MetricFrame.
*/
Expand Down Expand Up @@ -350,11 +380,13 @@ export default class StatsCollector {
const streamMetricFrame = SdkStreamMetricFrame.create();
streamMetricFrame.streamId = streamMetricReport.streamId;
streamMetricFrame.metrics = [];
this.addStreamMetricDimensionFrames(streamMetricFrame, streamMetricReport);
clientMetricFrame.streamMetricFrames.push(streamMetricFrame);
const metricMap = this.clientMetricReport.getMetricMap(
streamMetricReport.mediaType,
streamMetricReport.direction
);

for (const metricName in streamMetricReport.currentMetrics) {
this.addMetricFrame(metricName, clientMetricFrame, metricMap[metricName], Number(ssrc));
}
Expand Down
101 changes: 101 additions & 0 deletions test/clientmetricreport/ClientMetricReport.test.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,107 @@ describe('ClientMetricReport', () => {
});
});

describe('millisecondsPerSecond', () => {
const metricName = 'metric-name';

it('returns 0 if the interval is 0', () => {
const report = new GlobalMetricReport();
clientMetricReport.globalMetricReport = report;
clientMetricReport.currentTimestampMs = 0;
clientMetricReport.previousTimestampMs = 0;
expect(clientMetricReport.millisecondsPerSecond(metricName)).to.equal(0);
});

it('returns 0 if the diff is 0', () => {
const report = new GlobalMetricReport();
report.currentMetrics[metricName] = 0;
clientMetricReport.globalMetricReport = report;
clientMetricReport.currentTimestampMs = 2000;
clientMetricReport.previousTimestampMs = 0;
expect(clientMetricReport.millisecondsPerSecond(metricName)).to.equal(0);
});

it('interval is 1 second if pervious timestamp is 0', () => {
const report = new GlobalMetricReport();
report.currentMetrics[metricName] = 0.1;
clientMetricReport.globalMetricReport = report;
clientMetricReport.currentTimestampMs = 2000;
clientMetricReport.previousTimestampMs = 0;
expect(clientMetricReport.millisecondsPerSecond(metricName)).to.equal(100);
});

it('returns the count from the stream metric reports', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentMetrics[metricName] = 0.1;
clientMetricReport.streamMetricReports[ssrc] = report;
clientMetricReport.currentTimestampMs = 2000;
clientMetricReport.previousTimestampMs = 1000;
expect(clientMetricReport.millisecondsPerSecond(metricName, ssrc)).to.equal(100);
});
});

describe('isHardwareImplementation', () => {
const metricName = 'metric-name';

it('return 0 for software implementation FFmpeg', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] = 'FFmpeg';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(0);
});

it('return 0 for software implementation OpenH264', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] = 'OpenH264';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(0);
});

it('return 0 for software implementation libvpx', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] = 'libvpx';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(0);
});

it('return 1 for hardware implementation with ExternalDecoder', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] = 'ExternalDecoder (VDAVideoDecoder)';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(1);
});

it('return 1 for hardware implementation with ExternalEncoder', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] = 'ExternalEncoder';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(1);
});

it('return 1 for hardware implementation with HardwareAccelerator', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] = 'MediaFoundationVideoEncodeAccelerator';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(1);
});

it('return 0 for implementation fallback from hardware', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] =
'FFmpeg (fallback from: ExternalDecoder (VDAVideoDecoder)';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(0);
});
});

describe('getMetricMap', () => {
it('returns the global metric map if not passing the media type and the direction', () => {
expect(clientMetricReport.getMetricMap()).to.equal(clientMetricReport.globalMetricMap);
Expand Down
Loading

0 comments on commit 33b8fa3

Please sign in to comment.