Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Report encode/decode time and implementation metric #2434

Merged
merged 2 commits into from
Sep 16, 2022
Merged

Report encode/decode time and implementation metric #2434

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6f3e24a
Report encode/decode time and if encoder/decoder is in hardware. Also
Sep 9, 2022
23e6a99
Fix naming
Sep 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added

- Add encoder/decoder is in hardware metric and stream dimension to signaling protocol
- Report encode/decode time and if encoder/decoder is in hardware, and add encoder/decoder name as a metric dimension

### Removed

Expand Down
5,482 changes: 2,759 additions & 2,723 deletions docs/assets/js/search.js
View file
Open in desktop

Large diffs are not rendered by default.

145 changes: 129 additions & 16 deletions docs/classes/clientmetricreport.html
View file
Open in desktop

Large diffs are not rendered by default.

22 changes: 22 additions & 0 deletions docs/classes/globalmetricreport.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ <h3>Constructors</h3>
<h3>Properties</h3>
<ul class="tsd-index-list">
<li class="tsd-kind-property tsd-parent-kind-class"><a href="globalmetricreport.html#currentmetrics" class="tsd-kind-icon">current<wbr>Metrics</a></li>
<li class="tsd-kind-property tsd-parent-kind-class"><a href="globalmetricreport.html#currentstringmetrics" class="tsd-kind-icon">current<wbr>String<wbr>Metrics</a></li>
<li class="tsd-kind-property tsd-parent-kind-class"><a href="globalmetricreport.html#previousmetrics" class="tsd-kind-icon">previous<wbr>Metrics</a></li>
</ul>
</section>
Expand Down Expand Up @@ -130,6 +131,24 @@ <h5><span class="tsd-signature-symbol">[</span>id: <span class="tsd-signature-ty
</ul>
</div>
</section>
<section class="tsd-panel tsd-member tsd-kind-property tsd-parent-kind-class">
<a name="currentstringmetrics" class="tsd-anchor"></a>
<h3>current<wbr>String<wbr>Metrics</h3>
<div class="tsd-signature tsd-kind-icon">current<wbr>String<wbr>Metrics<span class="tsd-signature-symbol">:</span> <span class="tsd-signature-symbol">{}</span><span class="tsd-signature-symbol"> = {}</span></div>
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/clientmetricreport/GlobalMetricReport.ts#L8">src/clientmetricreport/GlobalMetricReport.ts:8</a></li>
</ul>
</aside>
<div class="tsd-type-declaration">
<h4>Type declaration</h4>
<ul class="tsd-parameters">
<li class="tsd-parameter-index-signature">
<h5><span class="tsd-signature-symbol">[</span>id: <span class="tsd-signature-type">string</span><span class="tsd-signature-symbol">]: </span><span class="tsd-signature-type">string</span></h5>
</li>
</ul>
</div>
</section>
<section class="tsd-panel tsd-member tsd-kind-property tsd-parent-kind-class">
<a name="previousmetrics" class="tsd-anchor"></a>
<h3>previous<wbr>Metrics</h3>
Expand Down Expand Up @@ -171,6 +190,9 @@ <h5><span class="tsd-signature-symbol">[</span>id: <span class="tsd-signature-ty
<li class=" tsd-kind-property tsd-parent-kind-class">
<a href="globalmetricreport.html#currentmetrics" class="tsd-kind-icon">current<wbr>Metrics</a>
</li>
<li class=" tsd-kind-property tsd-parent-kind-class">
<a href="globalmetricreport.html#currentstringmetrics" class="tsd-kind-icon">current<wbr>String<wbr>Metrics</a>
</li>
<li class=" tsd-kind-property tsd-parent-kind-class">
<a href="globalmetricreport.html#previousmetrics" class="tsd-kind-icon">previous<wbr>Metrics</a>
</li>
Expand Down
22 changes: 22 additions & 0 deletions docs/classes/streammetricreport.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ <h3>Constructors</h3>
<h3>Properties</h3>
<ul class="tsd-index-list">
<li class="tsd-kind-property tsd-parent-kind-class"><a href="streammetricreport.html#currentmetrics" class="tsd-kind-icon">current<wbr>Metrics</a></li>
<li class="tsd-kind-property tsd-parent-kind-class"><a href="streammetricreport.html#currentstringmetrics" class="tsd-kind-icon">current<wbr>String<wbr>Metrics</a></li>
<li class="tsd-kind-property tsd-parent-kind-class"><a href="streammetricreport.html#direction" class="tsd-kind-icon">direction</a></li>
<li class="tsd-kind-property tsd-parent-kind-class"><a href="streammetricreport.html#mediatype" class="tsd-kind-icon">media<wbr>Type</a></li>
<li class="tsd-kind-property tsd-parent-kind-class"><a href="streammetricreport.html#previousmetrics" class="tsd-kind-icon">previous<wbr>Metrics</a></li>
Expand Down Expand Up @@ -133,6 +134,24 @@ <h5><span class="tsd-signature-symbol">[</span>id: <span class="tsd-signature-ty
</ul>
</div>
</section>
<section class="tsd-panel tsd-member tsd-kind-property tsd-parent-kind-class">
<a name="currentstringmetrics" class="tsd-anchor"></a>
<h3>current<wbr>String<wbr>Metrics</h3>
<div class="tsd-signature tsd-kind-icon">current<wbr>String<wbr>Metrics<span class="tsd-signature-symbol">:</span> <span class="tsd-signature-symbol">{}</span><span class="tsd-signature-symbol"> = {}</span></div>
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/clientmetricreport/StreamMetricReport.ts#L14">src/clientmetricreport/StreamMetricReport.ts:14</a></li>
</ul>
</aside>
<div class="tsd-type-declaration">
<h4>Type declaration</h4>
<ul class="tsd-parameters">
<li class="tsd-parameter-index-signature">
<h5><span class="tsd-signature-symbol">[</span>id: <span class="tsd-signature-type">string</span><span class="tsd-signature-symbol">]: </span><span class="tsd-signature-type">string</span></h5>
</li>
</ul>
</div>
</section>
<section class="tsd-panel tsd-member tsd-kind-property tsd-parent-kind-class">
<a name="direction" class="tsd-anchor"></a>
<h3>direction</h3>
Expand Down Expand Up @@ -204,6 +223,9 @@ <h3>stream<wbr>Id</h3>
<li class=" tsd-kind-property tsd-parent-kind-class">
<a href="streammetricreport.html#currentmetrics" class="tsd-kind-icon">current<wbr>Metrics</a>
</li>
<li class=" tsd-kind-property tsd-parent-kind-class">
<a href="streammetricreport.html#currentstringmetrics" class="tsd-kind-icon">current<wbr>String<wbr>Metrics</a>
</li>
<li class=" tsd-kind-property tsd-parent-kind-class">
<a href="streammetricreport.html#direction" class="tsd-kind-icon">direction</a>
</li>
Expand Down
64 changes: 62 additions & 2 deletions src/clientmetricreport/ClientMetricReport.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// SPDX-License-Identifier: Apache-2.0

import Logger from '../logger/Logger';
import { SdkMetric } from '../signalingprotocol/SignalingProtocol.js';
import { SdkMetric, SdkStreamDimension } from '../signalingprotocol/SignalingProtocol.js';
import VideoStreamIndex from '../videostreamindex/VideoStreamIndex';
import Direction from './ClientMetricReportDirection';
import MediaType from './ClientMetricReportMediaType';
Expand Down Expand Up @@ -126,6 +126,35 @@ export default class ClientMetricReport {
return Number(metricReport.currentMetrics[metricName] * 1000);
};

millisecondsPerSecond = (metricName?: string, ssrc?: number): number => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we name this to be clearer what it does? milliseconds per second is kinda confusing

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comment. Created #2443 to update function naming

const metricReport = ssrc ? this.streamMetricReports[ssrc] : this.globalMetricReport;
let intervalSeconds = (this.currentTimestampMs - this.previousTimestampMs) / 1000;
if (intervalSeconds <= 0) {
return 0;
}
if (this.previousTimestampMs <= 0) {
intervalSeconds = 1;
}
const diff =
metricReport.currentMetrics[metricName] - (metricReport.previousMetrics[metricName] || 0);
if (diff <= 0) {
return 0;
}
return (diff * 1000) / intervalSeconds;
};

isHardwareImplementation = (metricName?: string, ssrc?: number): number => {
const metricReport = this.streamMetricReports[ssrc];
const implName = String(metricReport.currentStringMetrics[metricName]);
const hasHwName =
implName.includes('ExternalDecoder') ||
implName.includes('ExternalEncoder') ||
implName.includes('EncodeAccelerator') ||
implName.includes('DecodeAccelerator');
const isFallback = implName.includes('fallback from');
return hasHwName && !isFallback ? 1 : 0;
};

/**
* Canonical and derived metric maps
*/
Expand Down Expand Up @@ -252,6 +281,14 @@ export default class ClientMetricReport {
jitter: {
transform: this.secondsToMilliseconds,
},
totalEncodeTime: {
transform: this.millisecondsPerSecond,
type: SdkMetric.Type.VIDEO_ENCODE_MS,
},
encoderImplementation: {
transform: this.isHardwareImplementation,
type: SdkMetric.Type.VIDEO_ENCODER_IS_HARDWARE,
},
};

readonly videoDownstreamMetricMap: {
Expand All @@ -261,7 +298,6 @@ export default class ClientMetricReport {
source?: string;
};
} = {
totalDecodeTime: { transform: this.identityValue, type: SdkMetric.Type.VIDEO_DECODE_MS },
packetsReceived: { transform: this.countPerSecond, type: SdkMetric.Type.VIDEO_RECEIVED_PPS },
packetsLost: {
transform: this.packetLossPercent,
Expand Down Expand Up @@ -297,6 +333,14 @@ export default class ClientMetricReport {
},
frameHeight: { transform: this.identityValue, type: SdkMetric.Type.VIDEO_DECODE_HEIGHT },
frameWidth: { transform: this.identityValue, type: SdkMetric.Type.VIDEO_DECODE_WIDTH },
totalDecodeTime: {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this name be changed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This key is just about the field name in webrtc report where the metric is derived from, we cannot change that naming as it comes from webrtc..

transform: this.millisecondsPerSecond,
type: SdkMetric.Type.VIDEO_DECODE_MS,
},
decoderImplementation: {
transform: this.isHardwareImplementation,
type: SdkMetric.Type.VIDEO_DECODER_IS_HARDWARE,
},
};

getMetricMap(
Expand Down Expand Up @@ -329,6 +373,22 @@ export default class ClientMetricReport {
}
}

/**
* Dimensions derived from metric
*/
readonly streamDimensionMap: {
[id: string]: SdkStreamDimension.Type;
} = {
encoderImplementation: SdkStreamDimension.Type.VIDEO_ENCODER_NAME,
decoderImplementation: SdkStreamDimension.Type.VIDEO_DECODER_NAME,
};

getStreamDimensionMap(): {
[id: string]: SdkStreamDimension.Type;
} {
return this.streamDimensionMap;
}

/**
* media Stream metrics
*/
Expand Down
2 changes: 2 additions & 0 deletions src/clientmetricreport/GlobalMetricReport.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,6 @@
export default class GlobalMetricReport {
previousMetrics: { [id: string]: number } = {};
currentMetrics: { [id: string]: number } = {};
// As metric values do not necessarily be number, this is a workaround in case metric value is string
currentStringMetrics: { [id: string]: string } = {};
}
2 changes: 2 additions & 0 deletions src/clientmetricreport/StreamMetricReport.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,6 @@ export default class StreamMetricReport {
direction: Direction;
previousMetrics: { [id: string]: number } = {};
currentMetrics: { [id: string]: number } = {};
// As metric values do not necessarily be number, this is a workaround in case metric value is string
currentStringMetrics: { [id: string]: string } = {};
}
36 changes: 34 additions & 2 deletions src/statscollector/StatsCollector.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@ import IntervalScheduler from '../scheduler/IntervalScheduler';
import SignalingClient from '../signalingclient/SignalingClient';
import {
SdkClientMetricFrame,
SdkDimensionValue,
SdkMetric,
SdkStreamDimension,
SdkStreamMetricFrame,
} from '../signalingprotocol/SignalingProtocol.js';
import { Maybe } from '../utils/Types';
Expand Down Expand Up @@ -253,8 +255,16 @@ export default class StatsCollector {

for (const rawMetric in rawMetricReport) {
if (rawMetric in metricMap) {
metricReport.previousMetrics[rawMetric] = metricReport.currentMetrics[rawMetric];
metricReport.currentMetrics[rawMetric] = rawMetricReport[rawMetric];
if (typeof rawMetricReport[rawMetric] === 'number') {
metricReport.previousMetrics[rawMetric] = metricReport.currentMetrics[rawMetric];
metricReport.currentMetrics[rawMetric] = rawMetricReport[rawMetric];
} else if (typeof rawMetricReport[rawMetric] === 'string') {
metricReport.currentStringMetrics[rawMetric] = rawMetricReport[rawMetric];
} else {
this.logger.error(
`Unknown metric value type ${typeof rawMetricReport[rawMetric]} for metric ${rawMetric}`
);
}
}
}
}
Expand Down Expand Up @@ -300,6 +310,26 @@ export default class StatsCollector {
this.clientMetricReport.print();
}

/**
* Add stream metric dimension frames derived from metrics
*/
private addStreamMetricDimensionFrames(
streamMetricFrame: SdkStreamMetricFrame,
streamMetricReport: StreamMetricReport
): void {
const streamDimensionMap = this.clientMetricReport.getStreamDimensionMap();
for (const metricName in streamMetricReport.currentStringMetrics) {
if (metricName in streamDimensionMap) {
const dimensionFrame = SdkStreamDimension.create();
dimensionFrame.type = streamDimensionMap[metricName];
const dimensionValue = SdkDimensionValue.create();
dimensionValue.stringValue = streamMetricReport.currentStringMetrics[metricName];
dimensionFrame.value = dimensionValue;
streamMetricFrame.dimensions.push(dimensionFrame);
}
}
}

/**
* Packages a metric into the MetricFrame.
*/
Expand Down Expand Up @@ -350,11 +380,13 @@ export default class StatsCollector {
const streamMetricFrame = SdkStreamMetricFrame.create();
streamMetricFrame.streamId = streamMetricReport.streamId;
streamMetricFrame.metrics = [];
this.addStreamMetricDimensionFrames(streamMetricFrame, streamMetricReport);
clientMetricFrame.streamMetricFrames.push(streamMetricFrame);
const metricMap = this.clientMetricReport.getMetricMap(
streamMetricReport.mediaType,
streamMetricReport.direction
);

for (const metricName in streamMetricReport.currentMetrics) {
this.addMetricFrame(metricName, clientMetricFrame, metricMap[metricName], Number(ssrc));
}
Expand Down
101 changes: 101 additions & 0 deletions test/clientmetricreport/ClientMetricReport.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,107 @@ describe('ClientMetricReport', () => {
});
});

describe('millisecondsPerSecond', () => {
const metricName = 'metric-name';

it('returns 0 if the interval is 0', () => {
const report = new GlobalMetricReport();
clientMetricReport.globalMetricReport = report;
clientMetricReport.currentTimestampMs = 0;
clientMetricReport.previousTimestampMs = 0;
expect(clientMetricReport.millisecondsPerSecond(metricName)).to.equal(0);
});

it('returns 0 if the diff is 0', () => {
const report = new GlobalMetricReport();
report.currentMetrics[metricName] = 0;
clientMetricReport.globalMetricReport = report;
clientMetricReport.currentTimestampMs = 2000;
clientMetricReport.previousTimestampMs = 0;
expect(clientMetricReport.millisecondsPerSecond(metricName)).to.equal(0);
});

it('interval is 1 second if pervious timestamp is 0', () => {
const report = new GlobalMetricReport();
report.currentMetrics[metricName] = 0.1;
clientMetricReport.globalMetricReport = report;
clientMetricReport.currentTimestampMs = 2000;
clientMetricReport.previousTimestampMs = 0;
expect(clientMetricReport.millisecondsPerSecond(metricName)).to.equal(100);
});

it('returns the count from the stream metric reports', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentMetrics[metricName] = 0.1;
clientMetricReport.streamMetricReports[ssrc] = report;
clientMetricReport.currentTimestampMs = 2000;
clientMetricReport.previousTimestampMs = 1000;
expect(clientMetricReport.millisecondsPerSecond(metricName, ssrc)).to.equal(100);
});
});

describe('isHardwareImplementation', () => {
const metricName = 'metric-name';

it('return 0 for software implementation FFmpeg', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] = 'FFmpeg';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(0);
});

it('return 0 for software implementation OpenH264', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] = 'OpenH264';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(0);
});

it('return 0 for software implementation libvpx', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] = 'libvpx';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(0);
});

it('return 1 for hardware implementation with ExternalDecoder', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] = 'ExternalDecoder (VDAVideoDecoder)';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(1);
});

it('return 1 for hardware implementation with ExternalEncoder', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] = 'ExternalEncoder';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(1);
});

it('return 1 for hardware implementation with HardwareAccelerator', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] = 'MediaFoundationVideoEncodeAccelerator';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(1);
});

it('return 0 for implementation fallback from hardware', () => {
const ssrc = 1;
const report = new StreamMetricReport();
report.currentStringMetrics[metricName] =
'FFmpeg (fallback from: ExternalDecoder (VDAVideoDecoder)';
clientMetricReport.streamMetricReports[ssrc] = report;
expect(clientMetricReport.isHardwareImplementation(metricName, ssrc)).to.equal(0);
});
});

describe('getMetricMap', () => {
it('returns the global metric map if not passing the media type and the direction', () => {
expect(clientMetricReport.getMetricMap()).to.equal(clientMetricReport.globalMetricMap);
Expand Down
Loading