Skip to content
This repository has been archived by the owner on Nov 14, 2024. It is now read-only.

Fixing Presence Conflicts #3320

Merged
merged 24 commits into from
Aug 3, 2024
Merged

Conversation

jjj333-p
Copy link
Contributor

@jjj333-p jjj333-p commented Feb 6, 2024

This is meant to cache client presence for a moment so that it doesn't oscillate.

Currently Dendrite just federates out whatever presence it gets from the sync loop, which means if theres any clients attempting to sync without setting the user online, and there is an online client, it will just flip back and forth each time one of the clients polls /sync.

This pull request essentially stores in a map when the client last set online ideally to allow the online client to sync again and set an online presence before setting idle or offline.

I am not great at programming nor am I familiar with this codebase so if this pr is just shitwater feel free to discard, just trying to fix an issue that severely bothers me. If it is easier you can also steal the code and write it in yourself. I ran the linter, not sure that it did anything, the vscode go extension seems to format and lint anyways.

I tried to run unit tests but I have no idea any of this thing. it errors on TestRequestPool_updatePresence/same_presence_is_not_published_dummy2 (10m0s) which I think making this change broke. I am unsure how to comply, if y'all point me in the right direction ill try to fix it. I have tested it with all the situations I can think of on my personal instance pain.agency, and this seems to stand up under all the previously bugged situations.

My go also decided to update a bunch of the dependencies, I hate git and github and have no idea how to fix that, it was not intentional. i just overwrote them with the ones from the main repo and committed it, seems to have done what was needed.

Pull Request Checklist

Signed-off-by: Joseph Winkie <[email protected]>

@jjj333-p jjj333-p requested a review from a team as a code owner February 6, 2024 05:54
@jjj333-p
Copy link
Contributor Author

jjj333-p commented Feb 6, 2024

heres the unit test full output:

joseph@f ~/jendrite (presence-optimization) [1]> DENDRITE_TEST_SKIP_NODB=1 go test --race ./...
# github.com/matrix-org/dendrite/internal/sqlutil
internal/sqlutil/sqlutil_test.go:10:2: missing go.sum entry for module providing package github.com/DATA-DOG/go-sqlmock (imported by github.com/matrix-org/dendrite/internal/sqlutil); to add:
        go get -t github.com/matrix-org/dendrite/internal/sqlutil
FAIL    github.com/matrix-org/dendrite/internal/sqlutil [setup failed]
# maunium.net/go/mautrix/crypto/olm
../go/pkg/mod/maunium.net/go/[email protected]/crypto/olm/account.go:6:11: fatal error: olm/olm.h: No such file or directory
    6 | // #include <olm/olm.h>
      |           ^~~~~~~~~~~
compilation terminated.
?       github.com/matrix-org/dendrite/appservice/api   [no test files]
?       github.com/matrix-org/dendrite/appservice/consumers     [no test files]
?       github.com/matrix-org/dendrite/appservice/query [no test files]
?       github.com/matrix-org/dendrite/build/dendritejs-pinecone        [no test files]
?       github.com/matrix-org/dendrite/build/gobind-yggdrasil   [no test files]
?       github.com/matrix-org/dendrite/clientapi/api    [no test files]
?       github.com/matrix-org/dendrite/clientapi/auth/authtypes [no test files]
?       github.com/matrix-org/dendrite/clientapi/httputil       [no test files]
?       github.com/matrix-org/dendrite/clientapi/producers      [no test files]
?       github.com/matrix-org/dendrite/clientapi/threepid       [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone       [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/conn  [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/defaults      [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/embed [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/monolith      [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/rooms [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/users [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil      [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/embed        [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/signing      [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/yggconn      [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/yggrooms     [no test files]
?       github.com/matrix-org/dendrite/cmd/dendrite-upgrade-tests       [no test files]
?       github.com/matrix-org/dendrite/cmd/furl [no test files]
?       github.com/matrix-org/dendrite/cmd/generate-config      [no test files]
?       github.com/matrix-org/dendrite/cmd/generate-keys        [no test files]
?       github.com/matrix-org/dendrite/cmd/resolve-state        [no test files]
?       github.com/matrix-org/dendrite/federationapi/api        [no test files]
?       github.com/matrix-org/dendrite/federationapi/producers  [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/cache      [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/postgres   [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/postgres/deltas    [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/shared     [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/shared/receipt     [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/sqlite3    [no test files]
?       github.com/matrix-org/dendrite/federationapi/storage/sqlite3/deltas     [no test files]
?       github.com/matrix-org/dendrite/federationapi/types      [no test files]
?       github.com/matrix-org/dendrite/internal/eventutil       [no test files]
?       github.com/matrix-org/dendrite/internal/hooks   [no test files]
?       github.com/matrix-org/dendrite/mediaapi [no test files]
?       github.com/matrix-org/dendrite/mediaapi/fileutils       [no test files]
?       github.com/matrix-org/dendrite/mediaapi/storage/postgres        [no test files]
?       github.com/matrix-org/dendrite/mediaapi/storage/shared  [no test files]
?       github.com/matrix-org/dendrite/mediaapi/storage/sqlite3 [no test files]
?       github.com/matrix-org/dendrite/mediaapi/storage/tables  [no test files]
?       github.com/matrix-org/dendrite/mediaapi/thumbnailer     [no test files]
?       github.com/matrix-org/dendrite/mediaapi/types   [no test files]
?       github.com/matrix-org/dendrite/relayapi/api     [no test files]
?       github.com/matrix-org/dendrite/relayapi/storage [no test files]
?       github.com/matrix-org/dendrite/relayapi/storage/postgres        [no test files]
?       github.com/matrix-org/dendrite/relayapi/storage/shared  [no test files]
?       github.com/matrix-org/dendrite/relayapi/storage/sqlite3 [no test files]
ok      github.com/matrix-org/dendrite/appservice       3.425s
ok      github.com/matrix-org/dendrite/build/gobind-pinecone    2.418s
FAIL    github.com/matrix-org/dendrite/clientapi [build failed]
ok      github.com/matrix-org/dendrite/clientapi/auth   (cached)
ok      github.com/matrix-org/dendrite/clientapi/routing        (cached)
ok      github.com/matrix-org/dendrite/clientapi/userutil       (cached)
ok      github.com/matrix-org/dendrite/cmd/create-account       1.079s
ok      github.com/matrix-org/dendrite/cmd/dendrite     1.123s
ok      github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/conduit       (cached)
ok      github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/relay (cached)
ok      github.com/matrix-org/dendrite/federationapi    (cached)
ok      github.com/matrix-org/dendrite/federationapi/consumers  (cached)
ok      github.com/matrix-org/dendrite/federationapi/internal   (cached)
ok      github.com/matrix-org/dendrite/federationapi/queue      (cached)
ok      github.com/matrix-org/dendrite/federationapi/routing    (cached)
ok      github.com/matrix-org/dendrite/federationapi/statistics (cached)
ok      github.com/matrix-org/dendrite/federationapi/storage    (cached)
ok      github.com/matrix-org/dendrite/federationapi/storage/tables     (cached)
ok      github.com/matrix-org/dendrite/internal (cached)
ok      github.com/matrix-org/dendrite/internal/caching (cached)
ok      github.com/matrix-org/dendrite/internal/fulltext        (cached)
ok      github.com/matrix-org/dendrite/internal/httputil        (cached)
ok      github.com/matrix-org/dendrite/internal/pushgateway     (cached)
ok      github.com/matrix-org/dendrite/internal/pushrules       (cached)
ok      github.com/matrix-org/dendrite/internal/transactions    (cached)
ok      github.com/matrix-org/dendrite/mediaapi/routing (cached)
ok      github.com/matrix-org/dendrite/mediaapi/storage (cached)
ok      github.com/matrix-org/dendrite/relayapi (cached)
ok      github.com/matrix-org/dendrite/relayapi/internal        (cached)
ok      github.com/matrix-org/dendrite/relayapi/routing (cached)
ok      github.com/matrix-org/dendrite/relayapi/storage/tables  (cached)
?       github.com/matrix-org/dendrite/roomserver/internal      [no test files]
?       github.com/matrix-org/dendrite/roomserver/internal/perform      [no test files]
?       github.com/matrix-org/dendrite/roomserver/producers     [no test files]
?       github.com/matrix-org/dendrite/roomserver/storage       [no test files]
?       github.com/matrix-org/dendrite/roomserver/storage/postgres      [no test files]
?       github.com/matrix-org/dendrite/roomserver/storage/sqlite3       [no test files]
?       github.com/matrix-org/dendrite/roomserver/version       [no test files]
?       github.com/matrix-org/dendrite/setup    [no test files]
?       github.com/matrix-org/dendrite/setup/jetstream  [no test files]
?       github.com/matrix-org/dendrite/setup/mscs       [no test files]
?       github.com/matrix-org/dendrite/setup/process    [no test files]
?       github.com/matrix-org/dendrite/syncapi/consumers        [no test files]
?       github.com/matrix-org/dendrite/syncapi/producers        [no test files]
?       github.com/matrix-org/dendrite/syncapi/storage/postgres [no test files]
?       github.com/matrix-org/dendrite/syncapi/storage/postgres/deltas  [no test files]
?       github.com/matrix-org/dendrite/syncapi/storage/sqlite3  [no test files]
?       github.com/matrix-org/dendrite/syncapi/storage/sqlite3/deltas   [no test files]
?       github.com/matrix-org/dendrite/syncapi/streams  [no test files]
?       github.com/matrix-org/dendrite/test     [no test files]
?       github.com/matrix-org/dendrite/test/testrig     [no test files]
?       github.com/matrix-org/dendrite/userapi/api      [no test files]
?       github.com/matrix-org/dendrite/userapi/producers        [no test files]
?       github.com/matrix-org/dendrite/userapi/storage/postgres [no test files]
?       github.com/matrix-org/dendrite/userapi/storage/postgres/deltas  [no test files]
?       github.com/matrix-org/dendrite/userapi/storage/shared   [no test files]
?       github.com/matrix-org/dendrite/userapi/storage/sqlite3  [no test files]
?       github.com/matrix-org/dendrite/userapi/storage/sqlite3/deltas   [no test files]
?       github.com/matrix-org/dendrite/userapi/types    [no test files]
ok      github.com/matrix-org/dendrite/roomserver       6.391s
ok      github.com/matrix-org/dendrite/roomserver/acls  (cached)
ok      github.com/matrix-org/dendrite/roomserver/api   (cached)
ok      github.com/matrix-org/dendrite/roomserver/auth  (cached)
ok      github.com/matrix-org/dendrite/roomserver/internal/helpers      (cached)
ok      github.com/matrix-org/dendrite/roomserver/internal/input        (cached)
ok      github.com/matrix-org/dendrite/roomserver/internal/query        (cached)
ok      github.com/matrix-org/dendrite/roomserver/state (cached)
ok      github.com/matrix-org/dendrite/roomserver/storage/postgres/deltas       (cached)
ok      github.com/matrix-org/dendrite/roomserver/storage/shared        (cached)
ok      github.com/matrix-org/dendrite/roomserver/storage/sqlite3/deltas        (cached)
ok      github.com/matrix-org/dendrite/roomserver/storage/tables        (cached)
ok      github.com/matrix-org/dendrite/roomserver/types (cached)
ok      github.com/matrix-org/dendrite/setup/base       (cached)
ok      github.com/matrix-org/dendrite/setup/config     (cached)
ok      github.com/matrix-org/dendrite/setup/mscs/msc2836       (cached)
ok      github.com/matrix-org/dendrite/syncapi  4.661s
ok      github.com/matrix-org/dendrite/syncapi/internal (cached)
ok      github.com/matrix-org/dendrite/syncapi/notifier (cached)
ok      github.com/matrix-org/dendrite/syncapi/routing  1.168s
ok      github.com/matrix-org/dendrite/syncapi/storage  (cached)
ok      github.com/matrix-org/dendrite/syncapi/storage/shared   (cached)
ok      github.com/matrix-org/dendrite/syncapi/storage/tables   (cached)
panic: test timed out after 10m0s
running tests:
        TestRequestPool_updatePresence (10m0s)
        TestRequestPool_updatePresence/same_presence_is_not_published_dummy2 (10m0s)

goroutine 620 [running]:
testing.(*M).startAlarm.func1()
        /usr/lib/golang/src/testing/testing.go:2259 +0x259
created by time.goFunc
        /usr/lib/golang/src/time/sleep.go:176 +0x45

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000007a00, {0xef5917, 0x1e}, 0xf2ddb0)
        /usr/lib/golang/src/testing/testing.go:1649 +0x871
testing.runTests.func1(0x0?)
        /usr/lib/golang/src/testing/testing.go:2054 +0x85
testing.tRunner(0xc000007a00, 0xc0002b1af8)
        /usr/lib/golang/src/testing/testing.go:1595 +0x262
testing.runTests(0xc0000b7720?, {0x13ee6e0, 0x1, 0x1}, {0x1c?, 0x1e?, 0x1459fe0?})
        /usr/lib/golang/src/testing/testing.go:2052 +0x8ae
testing.(*M).Run(0xc0000b7720)
        /usr/lib/golang/src/testing/testing.go:1925 +0xcd8
main.main()
        _testmain.go:47 +0x2be

goroutine 19 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x0?)
        /home/joseph/go/pkg/mod/github.com/golang/[email protected]/glog.go:882 +0x8b
created by github.com/golang/glog.init.0 in goroutine 1
        /home/joseph/go/pkg/mod/github.com/golang/[email protected]/glog.go:410 +0x258

goroutine 35 [sleep]:
time.Sleep(0x2faf080)
        /usr/lib/golang/src/runtime/time.go:195 +0x125
github.com/matrix-org/dendrite/syncapi/sync.(*RequestPool).cleanPresence(0xc000230000, {0xffd378?, 0x23c0620}, 0x2faf080)
        /home/joseph/jendrite/syncapi/sync/requestpool.go:119 +0x152
created by github.com/matrix-org/dendrite/syncapi/sync.TestRequestPool_updatePresence in goroutine 6
        /home/joseph/jendrite/syncapi/sync/requestpool_test.go:129 +0x45a

goroutine 6 [chan receive]:
testing.(*T).Run(0xc000007ba0, {0xefb650, 0x25}, 0xc0002281a0)
        /usr/lib/golang/src/testing/testing.go:1649 +0x871
github.com/matrix-org/dendrite/syncapi/sync.TestRequestPool_updatePresence(0x0?)
        /home/joseph/jendrite/syncapi/sync/requestpool_test.go:131 +0x49b
testing.tRunner(0xc000007ba0, 0xf2ddb0)
        /usr/lib/golang/src/testing/testing.go:1595 +0x262
created by testing.(*T).Run in goroutine 1
        /usr/lib/golang/src/testing/testing.go:1648 +0x846

goroutine 40 [sync.Mutex.Lock]:
sync.runtime_SemacquireMutex(0x12eb090?, 0x80?, 0xc00022e540?)
        /usr/lib/golang/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xc00022c010)
        /usr/lib/golang/src/sync/mutex.go:171 +0x213
sync.(*Mutex).Lock(0xc00022c010)
        /usr/lib/golang/src/sync/mutex.go:90 +0x55
github.com/matrix-org/dendrite/syncapi/sync.TestRequestPool_updatePresence.func1(0xc0002269c0)
        /home/joseph/jendrite/syncapi/sync/requestpool_test.go:132 +0x59
testing.tRunner(0xc0002269c0, 0xc0002281a0)
        /usr/lib/golang/src/testing/testing.go:1595 +0x262
created by testing.(*T).Run in goroutine 6
        /usr/lib/golang/src/testing/testing.go:1648 +0x846
FAIL    github.com/matrix-org/dendrite/syncapi/sync     600.041s
ok      github.com/matrix-org/dendrite/syncapi/synctypes        (cached)
ok      github.com/matrix-org/dendrite/syncapi/types    (cached)
ok      github.com/matrix-org/dendrite/userapi  (cached)
ok      github.com/matrix-org/dendrite/userapi/consumers        (cached)
ok      github.com/matrix-org/dendrite/userapi/internal (cached)
ok      github.com/matrix-org/dendrite/userapi/storage  (cached)
ok      github.com/matrix-org/dendrite/userapi/storage/tables   (cached)
ok      github.com/matrix-org/dendrite/userapi/util     (cached)
FAIL
⏎                                                                                                                                                                                                                        
joseph@f ~/jendrite (presence-optimization) [1]> 

@jjj333-p
Copy link
Contributor Author

jjj333-p commented Feb 6, 2024

Oh I should also mention don't look too closely at the commit history, this one is a squash and merge (if it gets merged)

@jjj333-p
Copy link
Contributor Author

jjj333-p commented Feb 6, 2024

im fixing it, dont review yet I guess

@jjj333-p
Copy link
Contributor Author

jjj333-p commented Feb 6, 2024

eh im in over my head and not in a mental state where it makes sense to work on this. bit of a waste of a day but 🤷 i got out of bed

@jjj333-p jjj333-p closed this Feb 6, 2024
@jjj333-p jjj333-p reopened this Feb 6, 2024
@jjj333-p
Copy link
Contributor Author

jjj333-p commented Feb 6, 2024

I've hopefully fixed any potential race conditions, thanks DiamondBurned for the assistance

@jjj333-p
Copy link
Contributor Author

I could be seeing things, but this seems to have made presence with maunium bridges work correctly.

@jjj333-p
Copy link
Contributor Author

one interesting limitation ive discovered is this means presence set through sync doesnt override explicitly set presence through the api.

the only time this matters is for some reason element ios explicitly sets presence to offline when you swipe out of the app to the homescreen. im marking this down as a client bug more than anything because why the hell would it do that

@pcouy
Copy link

pcouy commented Jul 16, 2024

I'm not a Dendrite user myself, but I'm very interested in seeing some progress on this, as my homeserver is getting presence spam from a few Dendrite instances from time to time.

I did not review your code (I don't usually write or even read go), but thank you for trying and keeping this up to date with the main branch. Have you been running this since February ? Is it going well ?

If a maintainer reads this : how can I help with making this move forward ?

PS : Disabling presence on my side does not seem to prevent receiving a deluge of presence updates.

@jjj333-p
Copy link
Contributor Author

jjj333-p commented Jul 16, 2024 via email

Copy link

codecov bot commented Jul 24, 2024

Codecov Report

Attention: Patch coverage is 65.38462% with 9 lines in your changes missing coverage. Please review.

Project coverage is 68.24%. Comparing base (4d116ff) to head (ed1f7cb).
Report is 1 commits behind head on main.

Files Patch % Lines
syncapi/sync/requestpool.go 64.00% 8 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3320   +/-   ##
=======================================
  Coverage   68.24%   68.24%           
=======================================
  Files         513      513           
  Lines       46865    46888   +23     
=======================================
+ Hits        31984    32000   +16     
- Misses      10892    10899    +7     
  Partials     3989     3989           
Flag Coverage Δ
unittests 53.41% <65.38%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@qwqtoday
Copy link

This sounds useful for me, hopefully the maintainers can spend some of their time on reviewing this soon

@S7evinK
Copy link
Contributor

S7evinK commented Jul 28, 2024

I'm basically happy with it, I just want CI to be green before merging, this needs a few other PRs to be merged before.

Copy link
Contributor

@S7evinK S7evinK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#3397 is going to fix the media tests, so approving this one.
Thanks a lot! And sorry for the delay.

@S7evinK S7evinK merged commit 8c6cf51 into matrix-org:main Aug 3, 2024
15 of 20 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants