Add support for group calls using MSC3401 #2553

* Add group call events to EventType Signed-off-by: Šimon Brandner <[email protected]> * Use EventType instead of a const Signed-off-by: Šimon Brandner <[email protected]> * Make logging around sending group call member state event a bit better Signed-off-by: Šimon Brandner <[email protected]> * Fix m.calls elements being null Signed-off-by: Šimon Brandner <[email protected]>

) This avoids a race condition where the other side would first receive the to-device messages and only then the member state event which would result in the call being ignored Signed-off-by: Šimon Brandner <[email protected]>

and assert that olm sessions are open to the destination devices

…js-sdk into robertlong/group-call

…-call-disable-retries

No particular reason to worry about old user agents here

Seems chrome at least will give you a disabled audio track if you already had another user media audio track and disabled it, so make sure our tracks are enabled when we add them. We already did this on one code path but it didn't get moved over when a new code path was added. On the plus side, we now know the reason for the ancient code that had the comment asking what it was for, so update that.

If a renogotiation ends up with one track being removed, we removed the whole stream, which would cause us to lose, for example, audio rather than just video.

* Add PTT call mode & mute by default in PTT calls (#2311) No other parts of PTT calls implemented yet * Make the tests pass again (#2316) 3280394 made call use a bunch of methods that weren't mocked in the tests. * Add maximum trasmit time for PTT (#2312) on sender side by muting mic after the max transmit time has elapsed. * Don't allow user to unmute if another user is speaking (#2313) * Add maximum trasmit time for PTT on sender side by muting mic after the max transmit time has elapsed. * Don't allow user to unmute if another user is speaking Based on #2312 For element-hq/element-call#298 * Fix createGroupCall arguments (#2325) Comma instead of a colon...

as I can't work out why it was needed, so I can't justify keeping it in the group calls merge. It should be PRed to develop separately if needed.

Now we know what that bit in the crypto unit test was for...

…js-sdk into dbkr/group-call-merge

This was also in Element Call, and whilst js-=sdk might be a more sensible place, EC has all the information to do it properly (this impl didn't take admin talk-over into account).

await on the async operation so the promise we return resolves once everything's actuall complete

Fetch the device info once at the start of the cal and cache it rather than fetching every time, and throw if we're supposed to be using e2e but the other end has no e2e keys.

This waits until the mute metadata update is sent to all the calls before telling the user they're unmuted, when in PTT mode (and only when starting to talk, ie. unmuting). This should help avoid situations where the signalling connection is slow enough that the unmute event takes long enough to reach the other side that you hear someone speak before they've apparently unmuted. Involves splitting out the method to send the metadata update.

When we lose ICE connection to peers, set the status of their feeds to muted so to end their talking session. For element-hq/element-call#331

And not always to true. This was causing audio & video to start muted sometimes on normal calls because the ICE connection state would change to 'checking', causing the feeds to be muted.

…2447) * Emit unknown device errors for group call participants without e2e There are a number of different cases here: there were some before when dealing with versions that didn't send deviceId. This catches all of them and makes all these cases emit the same error. * Add type

Only enable the stream cloning behaviour on Safari: it was causing the audio renderer on Chrome (both desktop and Android) to hang, causing audio to fail sometimes in Element Call and other Chrome tabs (eg. YouTube) to fail to play audio. Fixes element-hq/element-call#267

To reduce review requests going to the "wrong" team. The team has been mirrored from the vector-im side.

* Disable playback of audio for muted users As hopefully explained by the comment * forEach instead of map

* Expire call member state events after 1 hour * Fix lints * Avoid a possible race

…#2466) to match updates to MSC3401

* Don't block muting on determining whether the device exists * Add comments

Regressed by #2352 (you can just use RTCStatsReport as an iterator directly (which was was what that code was doing before) which uses entries( which gives you key/value pairs, but using forEach gives you just the value.

At the moment it looks like its more valuable to get the audio from people even if they're not actually shown as speaking. We can always re-introduce it later.

This greatly reduces the amount of bandwidth used when transmitting silence.

* Fix some MatrixCall leaks and use a shared AudioContext These leaks, combined with the dozens of AudioContexts floating around in memory across different CallFeeds, could cause some really bad performance issues and audio crashes on Chrome. * Fully release the AudioContext in CallFeed's dispose method * Fix tests

Reverts #2450 Looks like this wasn't really the problem (although may have made it happens faster) and the actual problem was multiple audio contexts and/or leaking peer connections as fixed in #2484

* Remove empty decryption listener This listener looks like it was left over from something as it just did nothing at all. The todevice event gets put into the call event buffer which awaits on decryption for each event before processing, so it should already wait for decryption. More info: element-hq/element-call#428 * Unused import * Unused function!

This clause was causing all PTT calls to mute the remote side immediately upon ICE connection status changing to 'checking'.

* Set max average bitrate on PTT calls Via SDP munging. Also makes the SDP munging a bit more generic and codec-specific (we were previously adding usedtx to any codec that had an fmtp line already, which was probably not really the intention). * Make SDP munging for codecs that don't already have fmtp lines * Use sensible typescript syntax Co-authored-by: Šimon Brandner <[email protected]> Co-authored-by: Šimon Brandner <[email protected]>

Audio & video mute status were set in separate calls but share a mute status changed event, so you'd always get two mute status changed events emitted. We could suppress events where the mute status didn't change, but this would still get two events saying the same thing when they both changed. Instead, merge setAudioMuted & setVideoMuted into a single call that sets either or both.

* Send call version `1` as a string (#2471) * test typescriptification - backup.spec (#2468) * renamed: spec/unit/crypto/crypto-utils.js -> spec/unit/crypto/crypto-utils.ts * ts fixes in crypto-utils * renamed: spec/unit/crypto/backup.spec.js -> spec/unit/crypto/backup.spec.ts * ts fixes in backup.spec * remove fit * remove debug * Prepare changelog for v19.0.0-rc.1 * v19.0.0-rc.1 * Update jest monorepo (#2476) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update all (#2475) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency @types/jest to v28 (#2478) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Fix call.collectCallStats() (#2480) Regressed by #2352 (you can just use RTCStatsReport as an iterator directly (which was was what that code was doing before) which uses entries( which gives you key/value pairs, but using forEach gives you just the value. * Go back to forEach in collectcallstats (#2481) Older typescript library doesn't know about .values() on the stats object, so it was failing in react sdk which had an older typescript. matrix-org/matrix-react-sdk#8935 was an attempt to upgrade it but did not seem to be helping on CI, despite being fine locally. * Update babel monorepo to v7.18.6 (#2477) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Expose KNOWN_SAFE_ROOM_VERSION (#2474) * Fix return type on funcs in matrixClient to be optionally null (#2488) * Update pull_request.yaml (#2490) * Lock file maintenance (#2491) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Prepare changelog for v19.0.0 * v19.0.0 * Resetting package fields for development * Improve VoIP integrations testing (#2495) * Remove MSC3244 support (#2504) * Actually store the identity server in the client when given as an option (#2503) * Actually store the identity server in the client when given as an option * Update requestRegisterEmailToken to a modern spec version too * Properly re-insert room ID in bundled thread relation messages from sync (#2505) Events returned by the `/sync` endpoint, including relations bundled with other events, may have their `room_id`s stripped out. This causes decryption errors if the IDs aren't repopulated. Fixes element-hq/element-web#22094. * Remove `setNow` from `realtime-callbacks.ts` (#2509) Signed-off-by: Šimon Brandner <[email protected]> * Remove dead code (#2510) * Don't crash with undefined room in `processBeaconEvents()` (#2500) * Add a basic PR checklist for all PRs (#2511) It'll be mildly annoying for core developers who have to constantly remove or edit this, but it'll also serve as a good reminder to do these things. Note that signoff is not required for core developers. * Fix tests Co-authored-by: Šimon Brandner <[email protected]> Co-authored-by: Kerry <[email protected]> Co-authored-by: RiotRobot <[email protected]> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Michael Weimann <[email protected]> Co-authored-by: texuf <[email protected]> Co-authored-by: Michael Telatynski <[email protected]> Co-authored-by: Travis Ralston <[email protected]> Co-authored-by: Faye Duxovni <[email protected]>

* Make SDP munging media type specific We were trying to apply modifications to all media types which led to confusing warning messages saying opus wasn't present (when it was for the video stream). Make the modifications media-type specific to avoid this. * Make codec * mediatype into enums

* Prepare changelog for v19.2.0-rc.1 * v19.2.0-rc.1 * Sliding sync: add missing filters from latest MSC * Gracefully handle missing room_ids * Prepare changelog for v19.2.0 * v19.2.0 * Resetting package fields for development * Use EventType enum values instead of hardcoded strings (#2557) * Retry to-device messages (#2549) * Retry to-device messages This adds a queueToDevice API alongside sendToDevice which is a much higher-level API that adds the messages to a queue, stored in persistent storage, and retries them periodically. Also converts sending of megolm keys to use the new API. Other uses of sendToDevice are nopt converted in this PR, but could be later. Requires matrix-org/matrix-mock-request#17 * Bump matrix-mock-request * Add more waits to make indexeddb tests pass * Switch some test expectations to queueToDevice * Stop straight away if the client has been stopped Hopefully will fix tests being flakey and logging after tests have finished. * Add return types & fix constant usage * Fix return type Co-authored-by: Germain <[email protected]> * Fix return type Co-authored-by: Germain <[email protected]> * Fix return type Co-authored-by: Germain <[email protected]> * Stop the client in all test cases Co-authored-by: Germain <[email protected]> * Add support for sending user-defined encrypted to-device messages (#2528) * Add support for sending user-defined encrypted to-device messages This is a port of the same change from the robertlong/group-call branch. * Fix tests * Expose the method in MatrixClient * Fix a code smell * Fix types * Test the MatrixClient method * Fix some types in Crypto test suite * Test the Crypto method * Fix tests * Upgrade matrix-mock-request * Move useRealTimers to afterEach * Remove stream-replacement (#2551) * Reintroduce setNewStream method, fix test, update yarn.lock Co-authored-by: RiotRobot <[email protected]> Co-authored-by: Kegan Dougal <[email protected]> Co-authored-by: Germain <[email protected]> Co-authored-by: Robin <[email protected]> Co-authored-by: Šimon Brandner <[email protected]>

These had somehow got mixed up so the type check was failing. Nothing uses the response return type, so just return void.

* Add basic creation / entering tests for group calls * Missing space Co-authored-by: Robin <[email protected]> * Assert more of the group call member event and also move call leaving to a finally so it doesn't leaving a call hagning if it fails. Co-authored-by: Robin <[email protected]>

* WIP RoomWidgetClient * Wait for the widget API to become ready before backfilling * Add support for sending user-defined encrypted to-device messages This is a port of the same change from the robertlong/group-call branch. * Fix tests * Emit an event when the client receives TURN servers * Expose the method in MatrixClient * Override the encryptAndSendToDevices method * Add support for TURN servers in embedded mode and make calls mostly work * Don't put unclonable objects into VoIP events RoomWidget clients were unable to send m.call.candidate events, because the candidate objects were not clonable for use with postMessage. Converting such objects to their canonical JSON form before attempting to send them over the wire solves this. * Fix types * Fix more types * Fix lint * Upgrade matrix-widget-api * Save lockfile * Untangle dependencies to fix tests * Add some preliminary tests * Fix tests * Fix indirect export * Add more tests * Resolve TODOs * Add queueToDevice to RoomWidgetClient

Duplicate 'call' imports

* Test placing a call in a group call Refactors a bit of the call testing stuff Fixes element-hq/element-call#521 * Unused imports * Use expect.toHaveBeenCalledWith() * Types * More types * Add comment on mock typing * Use toHaveBeenCalledWith() * Initialise groupcall & room in beforeEach * Initialise mockMediahandler sensibly * Add type params to mock * Rename mute tests * Move comment * Join / leave in parallel * Remove leftover expect

* Add tests for incoming calls in group calls Inspiration wwlecome for the renamed describe group which we're really abusing for a bunch of things that happen to have the same dependencies. Fixes element-hq/element-call#532 * Extract incoming call tests out into their own describe and get the lexicographical ordering to match who should be calling who * Trailing space

Fixes element-hq/element-call#523

* Add unit tests for hangup / reject Fixes element-hq/element-call#537 * Fix some bugs where we carried on with the call after it had been ended

Fixes element-hq/element-call#547

Fixes element-hq/element-call#548

Pushed to wrong branch

* 1:1 screenshare tests Fixes element-hq/element-call#548 * Always hang up calls after tests to prevent hanging tests Also fix a null dereference as we may not have an invitee or opponent member when sending voip events if not using to-device messages. * use mockImplementationOnce Co-authored-by: Robin <[email protected]> * use mockImplementationOnce Co-authored-by: Robin <[email protected]> * Add type on mock * Add corresponding call.off * Merge enable & disable screenshare tests Co-authored-by: Robin <[email protected]>

* Fix ICE end-of-candidates messages We were casting a POJO to an RTCIceCandidate for the dummy end-of-candidates candidate, but #2473 started calling .toJSON() on these objects. Store separately whether we've seen the end of candidates rather than adding on a dummy candidate object. A test for this will follow, but a) I want to get this fix out and b) I'm currently rewriting the call test file to add typing. Fixes element-hq/element-call#553 * Remove hacks for testing * Switch if branches

* Add types to the call unit test suites Still involves quite a few casts to any unfortunately as it turns out we access quite a few private methods on the Call class in these tests. * Remove commented line & use better expect syntax * Replace more calls.length with toHaveBeenCalled * Remove mistakenly added id field

For element-hq/element-call#559

* Add GroupCalEventHandlerEvent.Room Emit an event when the group call event handler has processed all pending group calls. * Remove unused return value * Add void return type Co-authored-by: Šimon Brandner <[email protected]> Co-authored-by: Šimon Brandner <[email protected]>

As hopefully explained in comment Fixes #2589

Also add a few more types

Add some types & use mock-typed versions directly - it's clearer which client we're making assertions about.

* Test that calls in a group call are retried * Add new flushpromises file

See comment, although this still feels like a poor solution to the problem. Might be better if the js-sdk processed everything internally before emitting the 'Room' event (or indeed before joinRoom resolved) so the app knows everything is ready when it gets that event.

as otherwise it recurses and logs the entire client + store

* MediaHandler Tests, part 1 Haven't got through all the methods yet For element-hq/element-call#544 * Didn't need these in the end * Rest of the media handler tests * getUserMediaStream takes args * use mockResolvedValue * Add .off & reuse the mock we already made * Re-use mock handler again * Move updateLocalUsermediaStream to beforeEach * add .off * Add types * Add more .offs

* GroupCallEventhandler Tests Fixes element-hq/element-call#545 * Fix long line * Fix strict mode error Co-authored-by: Robin <[email protected]> * Fix typo Co-authored-by: Robin <[email protected]> Co-authored-by: Robin <[email protected]>

* Fix GroupCallEventHandler in matryoshka mode GroupCallEventHandler needs to see a 'Syncing' event before it starts handling any events, so emit one immediately in matryoshka mode. * Implement joinRoom on RoomWidgetClient Element Call has undergone some changes to how it loads rooms, meaning that this method must be implemented for the app to work in matryoshka mode. * Allow audio and video to be muted before local call feed exists This is desirable for the Element Web integration of Element Call, because we need to be able to mute our devices before ever joining the call or creating a call feed, if the users requests it. * Fix a strict mode error

Fixes element-hq/element-call#527

* Test fallback screensharing * Test replacetrack is called * Unused import * Return type * Fix other test after new track IDs

* Add test for removing RTX codec * Use mocked to cast

* Fix race in creating calls We ran an async function between checking for an existing call and adding the new one to the map, so it would have been possible to start creating another call while we were placing the first call. This changes the code to add the call to the map as soon as we've created it. Also adds more logging. * Switch to logger.debug * Fix unit tests

Fixes element-hq/element-call#578

* Add tests for call answering / candidate sending * Remopve unused stuff * Capitalise Co-authored-by: Šimon Brandner <[email protected]> * Capitalisation * Capitalise * Fix typescript strict error * Actually fix TS strict error(?) * TS strict mode try 3 Co-authored-by: Šimon Brandner <[email protected]>

Otherwise, the RoomWidgetClient class can end up accidentally sending and receiving events from rooms it didn't intend to, if it's an always-on-screen widget.

Co-authored-by: Robin <[email protected]>

…sk/gc-merge Signed-off-by: Šimon Brandner <[email protected]>

Co-authored-by: Robin <[email protected]>

* Add `CallEventHandler` tests Signed-off-by: Šimon Brandner <[email protected]> * Avoid tests hanging Signed-off-by: Šimon Brandner <[email protected]> Signed-off-by: Šimon Brandner <[email protected]>

Signed-off-by: Šimon Brandner <[email protected]> Signed-off-by: Šimon Brandner <[email protected]>

We didn't reset the 'seen end of candidates' flag when doign an ICE restart, so we would have ignored all locally gathered candidates on an ICE restart.

Merge develop

(PTT mode will still block on them, as expected)

Don't block muting/unmuting on network requests

* Fix screenshare failing after several attempts Re-use any existing transceivers when screen sharing. This prevents transceivers accumulating and making the SDP too big: see linked bug. This also switches from `addTrack()` to `addTransceiver ()` which is not that large of a change, other than having to explicitly find the transceivers after an offer has arrived rather than just adding tracks and letting WebRTC take care of it. Fixes element-hq/element-call#625 * Fix tests * Unused import * Use a map instead of an array * Add comment * more comment * Remove commented code * Remove unintentional debugging * Add test for screenshare transceiver re-use * Type alias for transceiver map

* Fix connectivity regressions Switches back to addTrack, digging the transceivers out manually to re-use, because the only way to group tracks into streams re-using trasceivers from the offer is to use setStreams which FF doesn't implement. * Remove comments

* Add throwOnFail to groupCall.setScreensharingEnabled For element-hq/element-call#652 * Update mediaHandler.ts

Add event and message capabilities to RoomWidgetClient

Merge develop into group-call branch

* TS strict mode compliance in the call / groupcall code * Also the test * Fix initOpponentCrypto to not panic if it doesn't actually need to init crypto

Merge changes from develop again

More TS strict mode fixes

A few more strict mode fixes

Let leave requests outlive the window

Unfortunately there are still other methods that could race with leave and result in broken group call state, such as enter and terminate. For the future, should consider writing a more careful specification of how the whole group call state machine is meant to work.

Resolve races between `initLocalCallFeed` and `leave`

Don't remove our own member for a split second when entering a call

Merge changes from develop

Commits on Nov 3, 2021

Revert changes to gotCallFeedsForAnswer

robertlong committed Nov 3, 2021

Configuration menu

View commit details

Copy full SHA for bdc12a2

Browse repository at this point

Copy the full SHA

bdc12a2 View commit details

Browse the repository at this point in the history

Commits on Nov 24, 2021

Set initial audio/video input ids

robertlong committed Nov 24, 2021

Configuration menu

View commit details

Copy full SHA for d1a62ed

Browse repository at this point

Copy the full SHA

d1a62ed View commit details

Browse the repository at this point in the history

Commits on Jan 13, 2022

Add user id to all send voip events

robertlong committed Jan 13, 2022

Configuration menu

View commit details

Copy full SHA for 42fef0e

Browse repository at this point

Copy the full SHA

42fef0e View commit details

Browse the repository at this point in the history

Commits on Jan 14, 2022

Add sender/dest session ids

robertlong committed Jan 14, 2022

Configuration menu

View commit details

Copy full SHA for 015eb5d

Browse repository at this point

Copy the full SHA

015eb5d View commit details

Browse the repository at this point in the history

Commits on Feb 4, 2022

Fix restartIce on FF Android

robertlong committed Feb 4, 2022

Configuration menu

View commit details

Copy full SHA for 1a78301

Browse repository at this point

Copy the full SHA

1a78301 View commit details

Browse the repository at this point in the history

Commits on Feb 19, 2022

Add callId to all logs

robertlong committed Feb 19, 2022

Configuration menu

View commit details

Copy full SHA for 0ef6c2e

Browse repository at this point

Copy the full SHA

0ef6c2e View commit details

Browse the repository at this point in the history

Commits on Mar 2, 2022

Fix shouldRequestAudio logging

robertlong committed Mar 2, 2022

Configuration menu

View commit details

Copy full SHA for 96ba061

Browse repository at this point

Copy the full SHA

96ba061 View commit details

Browse the repository at this point in the history

Commits on Aug 1, 2022

Delint group calls (#2554 )

SimonBrandner authored Aug 1, 2022

Configuration menu

View commit details

Copy full SHA for 22c5999

Browse repository at this point

Copy the full SHA

22c5999 View commit details

Browse the repository at this point in the history

Commits on Aug 18, 2022

Add screensharing tests (#2598 )

SimonBrandner authored Aug 18, 2022

Configuration menu

View commit details

Copy full SHA for 448a5c9

Browse repository at this point

Copy the full SHA

448a5c9 View commit details

Browse the repository at this point in the history

Add support for group calls using MSC3401 #2553

Add support for group calls using MSC3401 #2553

Commits on Oct 19, 2021

Commits on Oct 20, 2021

Commits on Oct 21, 2021

Commits on Oct 22, 2021

Commits on Oct 25, 2021

Commits on Oct 26, 2021

Commits on Oct 28, 2021

Commits on Nov 2, 2021

Commits on Nov 3, 2021

Commits on Nov 4, 2021

Commits on Nov 5, 2021

Commits on Nov 8, 2021

Commits on Nov 9, 2021

Commits on Nov 15, 2021

Commits on Nov 18, 2021

Commits on Nov 19, 2021

Commits on Nov 20, 2021

Commits on Nov 22, 2021

Commits on Nov 24, 2021

Commits on Nov 29, 2021

Commits on Nov 30, 2021

Commits on Jan 6, 2022

Commits on Jan 7, 2022

Commits on Jan 10, 2022

Commits on Jan 11, 2022

Commits on Jan 12, 2022

Commits on Jan 13, 2022

Commits on Jan 14, 2022

Commits on Jan 18, 2022

Commits on Feb 4, 2022

Commits on Feb 7, 2022

Commits on Feb 10, 2022

Commits on Feb 12, 2022

Commits on Feb 15, 2022

Commits on Feb 16, 2022

Commits on Feb 17, 2022

Commits on Feb 18, 2022

Commits on Feb 19, 2022

Commits on Feb 23, 2022

Commits on Feb 24, 2022

Commits on Feb 25, 2022

Commits on Mar 1, 2022

Commits on Mar 2, 2022

Commits on May 3, 2022

Commits on May 9, 2022

Commits on May 10, 2022

Commits on May 11, 2022

Commits on May 13, 2022

Commits on May 18, 2022

Commits on May 19, 2022

Commits on May 26, 2022

Commits on May 31, 2022

Commits on Jun 8, 2022

Commits on Jun 10, 2022

Commits on Jun 13, 2022

Commits on Jun 16, 2022

Commits on Jun 21, 2022

Commits on Jun 29, 2022

Commits on Jul 1, 2022

Commits on Jul 2, 2022

Commits on Jul 5, 2022

Commits on Jul 6, 2022

Commits on Jul 8, 2022

Commits on Jul 12, 2022

Commits on Jul 25, 2022

Commits on Jul 28, 2022

Commits on Jul 29, 2022

Commits on Aug 1, 2022

Commits on Aug 4, 2022

Commits on Aug 5, 2022

Commits on Aug 9, 2022

Commits on Aug 10, 2022

Commits on Aug 16, 2022

Commits on Aug 17, 2022

Commits on Aug 18, 2022

Commits on Aug 19, 2022

Commits on Aug 22, 2022

Commits on Aug 23, 2022