Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with signal detection in RTCPeerConnection-helper.js #23340

Open
foolip opened this issue Apr 30, 2020 · 1 comment
Open

Issues with signal detection in RTCPeerConnection-helper.js #23340

foolip opened this issue Apr 30, 2020 · 1 comment
Assignees

Comments

@foolip
Copy link
Member

foolip commented Apr 30, 2020

This issue is to track two things from #22779.

  1. Using OffscreenCanvas works, but it's not enabled in Firefox or Safari yet, so the tests will fail because of that. It's probably possible to use a canvas element instead, with a TODO to replace it later.
  2. The signal detection allows only for an error of 1, but due to quantization error in video codecs this could probably be larger. The margin of error needs to be large enough, but not so large that anything will pass.

Alternative idea for signal: send a specific color like 00FF00 green, and detect a signal if it's "almost" green. This should allow for a bigger margin without risking false detection, as long as there's nothing almost green elsewhere in the frame.

See #22779 (comment) for original discussion.

@alvestrand
Copy link
Contributor

On signal: humans are usually seen to be better able to percieve luma (brightness) than color - so codecs tend to optimize for luma fidelity. It's been suggested that instead of taking the average across the R/G/B values, we should use the ITU luma formula:

  // Use Luma as in Rec. 709: Y′709 = 0.21R + 0.72G + 0.07B;
  luma += 0.21 * data[i] + 0.72 * data[i + 1] + 0.07 * data[i + 2];

(code from https://github.com/webrtc/testrtc/blob/master/src/js/videoframechecker.js)
This way, shakiness in the blue would have almost no effect on the output signal.

But the easy thing to do to allay the concern is to write a test for the framework that scans through a range of "signal" values for a given and sees if it can reliably detect the changes; if there are ranges of values where detection doesn't work, we should prevent them from being used.

chromium-wpt-export-bot pushed a commit that referenced this issue Apr 30, 2020
This also checks that the video luma settles to a reasonable value
for a large range of values, so could be considered a codec test.
WPT bug: #23340

Bug: none
Change-Id: I6ce0537b90903d823945e2fe3c5389fa4c132608
chromium-wpt-export-bot pushed a commit that referenced this issue Apr 30, 2020
This also checks that the video luma settles to a reasonable value
for a large range of values, so could be considered a codec test.
WPT bug: #23340

Bug: none
Change-Id: I6ce0537b90903d823945e2fe3c5389fa4c132608
chromium-wpt-export-bot pushed a commit that referenced this issue May 8, 2020
This also checks that the video luma settles to a reasonable value
for a large range of values, so could be considered a codec test.
WPT bug: #23340

Bug: none
Change-Id: I6ce0537b90903d823945e2fe3c5389fa4c132608
chromium-wpt-export-bot pushed a commit that referenced this issue May 8, 2020
This also checks that the video luma settles to a reasonable value
for a large range of values, so could be considered a codec test.
WPT bug: #23340

Bug: none
Change-Id: I6ce0537b90903d823945e2fe3c5389fa4c132608
chromium-wpt-export-bot pushed a commit that referenced this issue May 11, 2020
This also checks that the video luma settles to a reasonable value
for a large range of values, so could be considered a codec test.
WPT bug: #23340

Bug: none
Change-Id: I6ce0537b90903d823945e2fe3c5389fa4c132608
chromium-wpt-export-bot pushed a commit that referenced this issue May 11, 2020
This also checks that the video luma settles to a reasonable value
for a large range of values, so could be considered a codec test.
WPT bug: #23340

Bug: none
Change-Id: I6ce0537b90903d823945e2fe3c5389fa4c132608
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2172971
Commit-Queue: Harald Alvestrand <[email protected]>
Reviewed-by: Philip Jägenstedt <[email protected]>
Cr-Commit-Position: refs/heads/master@{#767270}
blueboxd pushed a commit to blueboxd/chromium-legacy that referenced this issue May 11, 2020
This also checks that the video luma settles to a reasonable value
for a large range of values, so could be considered a codec test.
WPT bug: web-platform-tests/wpt#23340

Bug: none
Change-Id: I6ce0537b90903d823945e2fe3c5389fa4c132608
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2172971
Commit-Queue: Harald Alvestrand <[email protected]>
Reviewed-by: Philip Jägenstedt <[email protected]>
Cr-Commit-Position: refs/heads/master@{#767270}
chromium-wpt-export-bot pushed a commit that referenced this issue May 12, 2020
This also checks that the video luma settles to a reasonable value
for a large range of values, so could be considered a codec test.
WPT bug: #23340

Bug: none
Change-Id: I6ce0537b90903d823945e2fe3c5389fa4c132608
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2172971
Commit-Queue: Harald Alvestrand <[email protected]>
Reviewed-by: Philip Jägenstedt <[email protected]>
Cr-Commit-Position: refs/heads/master@{#767270}
moz-v2v-gh pushed a commit to mozilla/gecko-dev that referenced this issue May 26, 2020
…elper function., a=testonly

Automatic update from web-platform-tests
Add a test to check the video detector helper function.

This also checks that the video luma settles to a reasonable value
for a large range of values, so could be considered a codec test.
WPT bug: web-platform-tests/wpt#23340

Bug: none
Change-Id: I6ce0537b90903d823945e2fe3c5389fa4c132608
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2172971
Commit-Queue: Harald Alvestrand <[email protected]>
Reviewed-by: Philip Jägenstedt <[email protected]>
Cr-Commit-Position: refs/heads/master@{#767270}

--

wpt-commits: cdd4b43f2f89422f36d81546df70eb09f6aa8ed0
wpt-pr: 23341
xeonchen pushed a commit to xeonchen/gecko that referenced this issue May 26, 2020
…elper function., a=testonly

Automatic update from web-platform-tests
Add a test to check the video detector helper function.

This also checks that the video luma settles to a reasonable value
for a large range of values, so could be considered a codec test.
WPT bug: web-platform-tests/wpt#23340

Bug: none
Change-Id: I6ce0537b90903d823945e2fe3c5389fa4c132608
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2172971
Commit-Queue: Harald Alvestrand <[email protected]>
Reviewed-by: Philip Jägenstedt <[email protected]>
Cr-Commit-Position: refs/heads/master@{#767270}

--

wpt-commits: cdd4b43f2f89422f36d81546df70eb09f6aa8ed0
wpt-pr: 23341
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants