Add end-to-end tests for End-to-end encryption

[ara4n]

As part of our epic quest against UISIs/UTDs (https://github.com/vector-im/riot-web/issues/2996\), we should use our fancy new E2E testing framework over at https://github.com/matrix-org/matrix-react-end-to-end-tests to exercise E2EE end-to-end and try to locate some failing tests.

Tests to write:

• User A on the test synapse makes an E2E room and invites user B on the same test synapse
  * [x] Can A decrypt B's messages? Start adding Cypress tests for crypto matrix-org/matrix-react-sdk#8577
  * [ ] Can B decrypt A's messages?
  * [ ] sending messages before the users join (currently shouldn't work at all if A and B both join a room)
  * [ ] or adding devices on both a & b's side and see if they can then participate correctly
  * [ ] having b leave the room, change device, rejoin the room, and see if they can participate correctly
  * [ ] Having A invite B, and seeing if B can see A's messages
  * [ ] Having A invite B, having B then change device, then B join, see if B can see A's message.
  * [ ] Seeing what happens and if we can recover if a device runs out of published OTKs
  * [ ] Seeing what happens if devices race on claiming OTKs and end up claiming the same OTK (https://github.com/vector-im/riot-web/issues/3868\)

• All of these with A and B on different servers (if Cypress can spin up multiple federating synapse instances within our CI), as the federated & non-federated code paths have entirely different flows.

• another interesting one is where you deliberately break federation between a & b and see how rapidly it recovers

(in terms of missing e2e keys or unsynced dev lists)

fwiw sytest tries to test all of ^ already, but when you add the whole js-sdk stack into the mix too i'm not so confident there aren't still bugs.
(plus protecting from regressions in js-sdk ftw)

[bwindels]

the basic case is tested already at https://github.com/matrix-org/matrix-react-end-to-end-tests/blob/master/src/scenario.js#L60 but adding federation, and extended case would indeed be great.

[duxovni]

Rewriting this as a to-do list for Cypress crypto testing :)

[Johennes]

@richvdh do any of the test cases listed happen to have been added during the ER project?

[richvdh]

Hrm. https://github.com/matrix-org/matrix-react-end-to-end-tests is dead (and superceded by Cypress).

I'm don't think any of the unchecked tests have been added explicitly. Most of the tests we've added have centered on things like verification, cross-signing, and event shields.

Some cases would be easy enough to add; others are virtually impossible to test with an e2e test (such as racing to claim OTKs). And our cypress test infrastructure doesn't easily cater for federated homeservers, so those are hard to cover.

Not entirely sure what to do with this issue. It feels a bit outdated.

[Johennes]

How about we reduce it to the ones you'd deem reasonable to add into our current E2E testing infrastructure?

[richvdh]

• Can B decrypt A's messages?

This is easy, but of questionable value. We cover it in other tests anyway, and if E2E breaks this badly we'll notice pretty quickly.

• sending messages before the users join (currently shouldn't work at all if A and B both join a room)

Don't know what this means. How can you send a message before you join?

• or adding devices on both a & b's side and see if they can then participate correctly

We could probably usefully do something like this. It's slightly tricky because (obviously) you only actually run a single instance of EW and the other "devices" are effectively "bots" controlled directly by javascript. And you can only really have a max of 2 (3 at a stretch) active devices in cypress otherwise you hit Chrome's concurrent request limit.

• having b leave the room, change device, rejoin the room, and see if they can participate correctly

I think this would work, though there are various subtests: with and without working key backup; with and without other active devices.

• Having A invite B, and seeing if B can see A's messages

I think this is maybe about historical room key sharing. Likely we'll add tests for this when we come to that piece of work anyway.

• Having A invite B, having B then change device, then B join, see if B can see A's message.

We could test this, but I don't think it works currently...

The other stuff I think would be hard to test in Cypress.

[Johennes]

-[ ] sending messages before the users join (currently shouldn't work at all if A and B both join a room)

Don't know what this means. How can you send a message before you join?

I think this might refer to joining a room and sending a message before another user joins.

[...] you only actually run a single instance of EW and the other "devices" are effectively "bots" controlled directly by javascript

Do you think test cases like the ones above would be better handled via https://github.com/matrix-org/trafficlight than Cypress / Playwright?

[richvdh]

-[ ] sending messages before the users join (currently shouldn't work at all if A and B both join a room)
Don't know what this means. How can you send a message before you join?

I think this might refer to joining a room and sending a message before another user joins.

Ok, so: A sends a message, then B joins without an invite, and we see if B can see the message (which he can't, that being the point of E2E?)

[...] you only actually run a single instance of EW and the other "devices" are effectively "bots" controlled directly by javascript

Do you think test cases like the ones above would be better handled via https://github.com/matrix-org/trafficlight than Cypress / Playwright?

Not that familiar with trafficlight to be honest. Possibly some of the "multiple client" tests could usefully be done that way. I don't think it makes it easy to do things like running multiple servers though.

[Johennes]

Ok, so: A sends a message, then B joins without an invite, and we see if B can see the message (which he can't, that being the point of E2E?)

Err, yeah something doesn't add up there. 🤔

[richvdh]

I think we probably need to discuss the best approach for E2E tests of E2EE as a wider team. I know @kegsay is thinking about this sort of thing, and no doubt folks like @michaelkaye would have useful input too.

[michaelkaye]

So trafficlight's usefulness depends a bit on where you want to cut the tests - if you want UI to UI tests then it's probably better than everything else we have; but if you don't want to pay the overhead of managing the UI interface (like kegan is thinking about) then it's not worth the overhead.

In terms of running multiple servers; it did use to have a complement homerunner to start up servers and sets of servers alongside trafficlight, but practically that was a lot of work to make available for eg emulators in the cloud type services so has slipped by the wayside. At the moment we start with new users on a long-running server.

[kegsay]

My thoughts are now written up at element-hq/element-meta#2165 which feels like a better home than element-web.

[kegsay]

For the record, this ended up resulting in https://github.com/matrix-org/complement-crypto/ which as of this writing supports:

User A on the test synapse makes an E2E room and invites user B on the same test synapse

• Can A decrypt B's messages? Start adding Cypress tests for crypto matrix-org/matrix-react-sdk#8577
• Can B decrypt A's messages?
• sending messages before the users join (currently shouldn't work at all if A and B both join a room)
• or adding devices on both a & b's side and see if they can then participate correctly
• having b leave the room, change device, rejoin the room, and see if they can participate correctly
• Having A invite B, and seeing if B can see A's messages
• FAILS: Having A invite B, having B then change device, then B join, see if B can see A's message.
• Seeing what happens and if we can recover if a device runs out of published OTKs
• Seeing what happens if devices race on claiming OTKs and end up claiming the same OTK