Matrix.org has broken the federation connection

[artenax]

The default and largest server on the Matrix network has broken the key exchange between messenger users. Users with accounts on matrix.org do not receive the key from their contacts with accounts on other servers, so they cannot decrypt messages. /discardsession does not help.

Users on matrix.org have in chat instead of message text: “unable to decrypt message” (in Element client) and “message could not be decrypted due to missing key” (in Nheko).
Users of any other servers can read messages from matrix.org users - they get the keys.
Communication between users of any other servers goes without problems.

[roman90sv]

This problem has existed since at least the end of July.

[ImMALWARE]

I also can't get messages from nitro[.]chat user, same problem

[sigseg5]

Knock, knock
Who's there?
Three-letter agency:
Create a corral so people don't spread out on uncontrolled servers 🌚

[ara4n]

we’re not aware of federation problems on matrix.org atm. generally when other servers can’t federate (eg nitro.chat) it’s because they are coughing up errors to requests from matrix.org for whatever reason, which means they then get marked down - ie a problem on the receiver not the sender.

in order for this bug report to actually be actionable, you may want to:

• give an example of a server which matrix.org is failing to federate with
• give details of a request that’s failed (eg client logs for an event that couldn’t be decrypted on the sender & receiver side)

…otherwise we’ll have to close this as unactionable.

[kegsay]

Users on matrix.org have in chat instead of message text: “unable to decrypt message”

I've been the developer looking at messages which are unable to decrypt (UTD) in the ecosystem. There's a few reasons why this may happen. This typically (though not exclusively) means matrix.org was unable to claim one-time keys (OTKs) for those remote users, or it failed to send room keys, particularly if it used to work. This is governed by the endpoints /keys/claim for OTKs over both the client-server and federation APIs, and /sendToDevice with m.direct_to_device for room keys over the client-server API and federation API respectively.

/discardsession won't help if these endpoints are not working correctly, which implies a communication problem with your server for those endpoints (given the messages themselves seem to make it there okay, but messages use a different endpoint). Without knowing the server domain, along with the dates/times it happened, we can't help diagnose this any further, but typical reasons I've seen are:

• those endpoints (particularly /keys/claim) times out, which is more likely if you have lots of users on your server,
• the sending server (in this case matrix.org) is backing off sending to-device messages.

However, if you are sending traffic to matrix.org, the backoff timer should be reset, so it's most likely timeouts on /keys/claim. This isn't anything to do with matrix.org, it's your server being slow. Assuming this user is on the same server:

This problem has existed since at least the end of July.

then I would check your logs at the end of July for timeouts/non 2xx status codes for /keys/claim. That's basically all I can analyse with the information you've provided unfortunately.

[artenax]

So YOU have a federation working?
The point of my post was that it looks like the federation is broken for ALL people. Since there have been many complaints (and I confirm), including those with private personal servers... Although all of them are from Russia...

Well, if you don't use the federation... register two accounts on different servers and check the logs. I used a German server.
Prove that federation works...

[ara4n]

oh! if you're in Russia (which, of course, you didn't bother to say either here or on HN), then you are probably suffering from your government censoring access to matrix.org. We are aware of folks in Russia being unable to connect to the CS API on matrix.org for the last few weeks, but this is the first we've heard of the SS API being blocked too.

You may wish to complain to your own 3-letter agencies...

[ara4n]

Looking at OONI - it doesn't have stats on matrix-federation.matrix.org (https://explorer.ooni.org/domain/matrix-federation.matrix.org?since=2024-08-25&until=2024-09-25&probe_cc=RU) but it does for the CS API: https://explorer.ooni.org/domain/matrix-client.matrix.org?since=2024-08-25&until=2024-09-25&probe_cc=RU

So I'd assume that SS API is impacted too.

The workaround is to use censorship circumvention, as you would on any other service.

Separately, Matrix could do more native censorship circumvention - either via p2p (arewep2pyet.com) or by rearchitecting E2EE so it can bounce transitively via other servers (like plaintext traffic can). But both of those are long-term projects which just don't exist yet today.

[artenax]

matrix-client.matrix.org was blocked recently. But matrix.org works and the messengers that connect to it also work (even though it is not standard). I mean, when communicating within the matrix.org server.
But the connection to the federation was broken earlier.

However, something doesn't add up for you.

I tested under Cloudflare WARP VPN, it is uncensored, although the IP is considered Russian (I think it comes out of Cloudflare servers somewhere from Finland). And there was still no connection to the federation (in app.element.io in the browser). It looks more like a geo block on your side.

You still haven't said if federation is working for you.
I can certainly check under VPN later...

[kegsay]

So YOU have a federation working?

Yes.

[ara4n]

As I said:

we’re not aware of federation problems on matrix.org atm

We are not blocking anything on the matrix.org side, nor are any servers currently defederated.

If you can actually give us something to go on (e.g. the hostname of an impacted server) then we try to understand what's actually going on, but so far this continues to be completely unactionable...

[skobkin]

I've just checked from my personal server (skobk.in) which is NOT in Russia (although I am) that federation is working.

I've wrote to the chat hosted on matrix.org and matrix.org users saw that and replied.

But. When I tried to check the key exchange via DM with my friend who has account on matrix.org it showed a problem with the key.

Here's my message to her from Nheko:

Here's how she saw my message on mobile:

Here's how I see her reply:
Nheko:

Element Web (self-hosted):

@ara4n, @kegsay, @artenax
So here's what I personally can conclude:

• The federation is working.
• There are probably some issues with key exchange.

I really don't like the attitude of some people who accuse matrix.org of something based on nothing. We should probably dig deeper into the issue and try to find out what's really going on instead of pointing fingers at each other.

I can't provide server logs right now. If someone can replicate my experiment, please do that. If not, then I'll try to repeat it later and provide logs myself.

If possible, please reopen the issue so it won't be lost.

UPD: At the same time, another matrix.org user:

So... no reliable results for now.

[artenax]

Well, I managed to create an encrypted room between @nakity:matrix.org and @nakita:matrix.im
However:
No [offline] messages go from matrix.org to matrix.im
But messages are going from matrix.im to matrix.org
Client logs:
matrix.org.zip
matrix.im.zip

[artenax]

I want to add that I tested it in VPN (or rather, shadowsocks proxy from Finland) without censorship.
Cloudflare warp vpn (aka 1.1.1.1) and shadowsocks via Cloudflare warp vpn - same results. I think it doesn't matter.
Sessions are not simultaneous, i.e. it's not live chat, but offline messages (I changed browser profiles and restarted it with cache clearing, in the web versions: app.element.io and element.matrix.im).
Browser: ungoogled-chromium-128.0.6613.137.AppImage

[artenax]

I've seen the same problem before between transfem.dev (now dead) and matrix.im. That is, without involving communication with matrix.org contacts.
Message edited.

[artenax]

Logically it seems that matrix.im has the problem. But there are complaints on different servers, as you understand.

[skobkin]

Small update from me.

After some time, I've got the message from my friend (mentioned above) successfully decrypted on all devices.

I'm not really sure what changed since then 🤷