Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Separated Statistics (Target) #5879

Closed
wants to merge 91 commits into from
Closed

Separated Statistics (Target) #5879

wants to merge 91 commits into from

Conversation

reivilibre
Copy link
Contributor

Signed-off-by: Olivier Wilkinson (reivilibre) [email protected]

Do not merge this yet. This acts as an intermediate PR for several PRs which will be merged into this one

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file
  • Pull request includes a sign off

@reivilibre
Newsfile
8374bcb 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre reivilibre mentioned this pull request Aug 19, 2019
Merged
reivilibre and others added 3 commits August 20, 2019 13:22
@reivilibre
Tear out current room & user statistics (#5880)
8de9ebe 
* Tear out current room & user statistics.

Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>

* Black is back with more linting complaints

Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Add schema for Separated Statistics
d7675e7 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Add storage function for storing stats deltas
80a1c6e 
Old collection is not included in this commit

Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre reivilibre mentioned this pull request Aug 20, 2019
Merged
@reivilibre
Handle state deltas and turn them into stats deltas
e4cbea6 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre reivilibre mentioned this pull request Aug 20, 2019
Merged
@reivilibre
Ack, isort!
1819563 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre reivilibre mentioned this pull request Aug 20, 2019
Closed
reivilibre and others added 20 commits August 20, 2019 15:02
@reivilibre @erikjohnston
Update synapse/storage/stats.py
b5573c0 
Co-Authored-By: Erik Johnston <[email protected]>
@reivilibre @erikjohnston
Update synapse/storage/stats.py
4a97eef 
Co-Authored-By: Erik Johnston <[email protected]>
@reivilibre
Add room and user statistics documentation.
6a19f7e 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Sanitise accepted fields in _update_stats_delta_txn
981c6cf 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Clarify _update_stats_delta_txn
977310e 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Unify name of 'stats regenerator' in schema comments.
eafa8d3 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Remove needless defaults.
18a4c03 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Simplify table structure
7b657f1 
This obviates the need for old collection, but comes at the minor cost
of not being able to track historical stats or per-slice fields until
after the statistics regenerator is finished.

Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Fix up SQL schema delta
e8fc180 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Fix up historical stats support.
79252d1 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Allow schema deltas to be engine-specific
c3d2bf2 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Use engine-specific delta SQL files rather than delta written in Python.
1ecd1a6 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Merge branch 'develop' into rei/rss_target
baeaf00
@reivilibre
Merge branch 'rei/rss_target' into rei/rss_inc2
5043ef8
@reivilibre @erikjohnston
Apply suggestions from code review
4b7bf2e 
Co-Authored-By: Erik Johnston <[email protected]>
@reivilibre
Clarify _update_stats_delta_txn by adding code comments and kwargs.
81c5289 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Apply minor suggestions from review
544ba2c 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Lock tables in upsert fall-backs.
a6c1020 
Should not be too much of a performance concern as this code won't be
hit on Postgres, which large deployments should be using.

Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Code formatting (Black)
736ac58 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Switch to milliseconds in room/user stats for consistency.
09cbc3a 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
reivilibre and others added 28 commits August 30, 2019 14:39
@reivilibre
Merge pull request #5924 from matrix-org/rei/rss_inc6
8c02602 
Separated Statistics [6/7ish]
@reivilibre
Merge branch 'rei/rss_inc7' into rei/rss_inc8
1d6cf15
@reivilibre
Some fixes that have become necessary due to changes in other PRs
440c60e 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Code formatting and typo pointed out by Erik.
065042c 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Merge branch 'rei/rss_inc7' into rei/rss_inc8
7dc387e
@reivilibre
Fix that became apparent after unit testing
0f2e59f 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Merge branch 'rei/rss_inc7' into rei/rss_inc8
bf6d45f
@reivilibre
users table's ID field is actually called name.
b379a11 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Merge branch 'rei/rss_inc7' into rei/rss_inc8
425d445
@reivilibre
Whoops, took out a line there...
4ecc62b 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Merge branch 'rei/rss_inc7' into rei/rss_inc8
97b2035
@reivilibre
Ambiguous room_id
d39c09c 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Merge branch 'rei/rss_inc7' into rei/rss_inc8
eba432e
@reivilibre
Adapt to use renamed room_state
50c321d 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Merge branch 'rei/rss_inc7' into rei/rss_inc8
98a8928
@reivilibre
Fix incremental processor when there are no deltas.
b928909 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Whoopsies; these things come in order…
ab11c0a 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Fixes to counting and stats deltas
7b977bd 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Various fixes
6f5e543 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Add stats tests
d49457b 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Add stats docs
fc5d118 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Linting
21593fe 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Fix to use milliseconds
fca3a9c 
Signed-off-by: Olivier Wilkinson (reivilibre) <[email protected]>
@reivilibre
Merge branch 'develop' into rei/rss_target
ffc30b8
@reivilibre
Merge pull request #5941 from matrix-org/rei/rss_inc7
e893214 
Separated Statistics [7/7ish]
@erikjohnston
Merge branch 'rei/rss_target' of github.com:matrix-org/synapse into r…
84532a4 
…ei/rss_inc8
@erikjohnston
Renamve get_room_state
02f759e
@erikjohnston
Merge pull request #5946 from matrix-org/rei/rss_inc8
745f2da 
Separated Statistics [8/7ish EOF]
@erikjohnston
Copy link
Member

This landed in #5971 \o/

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@reivilibre @erikjohnston