Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Process previously failed backfill events in the background #15585

Merged
merged 31 commits into from
May 25, 2023
Merged

Process previously failed backfill events in the background #15585

merged 31 commits into from
May 25, 2023

Commits on May 12, 2023

  1. Process previously failed backfill events in the background 

    Fix #13623

Follow-up to #13621 and #13622

Part of making `/messages` faster: #13356
    @MadLittleMods
    MadLittleMods committed May 12, 2023
    Configuration menu
    Copy the full SHA
    fd26164 View commit details
    Browse the repository at this point in the history

  2. Add changelog

    @MadLittleMods
    MadLittleMods committed May 12, 2023
    Configuration menu
    Copy the full SHA
    c5dc746 View commit details
    Browse the repository at this point in the history

  3. Add consideration

    @MadLittleMods
    MadLittleMods committed May 12, 2023
    Configuration menu
    Copy the full SHA
    8fc47d8 View commit details
    Browse the repository at this point in the history

  4. Fix lints

    @MadLittleMods
    MadLittleMods committed May 12, 2023
    Configuration menu
    Copy the full SHA
    b5d95f7 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2023

  1. Merge branch 'develop' into madlittlemods/process-previously-failed-e… 

    …vents-in-background
    @MadLittleMods
    MadLittleMods committed May 16, 2023
    Configuration menu
    Copy the full SHA
    ebc93be View commit details
    Browse the repository at this point in the history

  2. Always check for failed attempts 

    This function is used for backfill and when another server sends us a PDU. In
either case, we're trying to avoid the costly state calculations to see if
it's allowed so I think we should always do this check

See #15585 (comment)
    @MadLittleMods
    MadLittleMods committed May 16, 2023
    Configuration menu
    Copy the full SHA
    e13f5a9 View commit details
    Browse the repository at this point in the history

  3. Add comments and concern about maybe queue

    @MadLittleMods
    MadLittleMods committed May 16, 2023
    Configuration menu
    Copy the full SHA
    70f5911 View commit details
    Browse the repository at this point in the history

  4. Process all failed events as a sequential task in the background 

    To avoid events stacking up in parallel exhausting our resources.

See #15585 (comment)
    @MadLittleMods
    MadLittleMods committed May 16, 2023
    Configuration menu
    Copy the full SHA
    45934fe View commit details
    Browse the repository at this point in the history

  5. Merge branch 'develop' into madlittlemods/process-previously-failed-e… 

    …vents-in-background
    @MadLittleMods
    MadLittleMods committed May 16, 2023
    Configuration menu
    Copy the full SHA
    b1998d7 View commit details
    Browse the repository at this point in the history

  6. Better comments

    @MadLittleMods
    MadLittleMods committed May 16, 2023
    Configuration menu
    Copy the full SHA
    93de856 View commit details
    Browse the repository at this point in the history

  7. Add test for separate_event_ids_with_failed_pull_attempts

    @MadLittleMods
    MadLittleMods committed May 16, 2023
    Configuration menu
    Copy the full SHA
    631d7db View commit details
    Browse the repository at this point in the history

Commits on May 17, 2023

  1. Avoid doing extra work if the list is empty

    @MadLittleMods
    MadLittleMods committed May 17, 2023
    Configuration menu
    Copy the full SHA
    beeccc3 View commit details
    Browse the repository at this point in the history

  2. Make sure to retain the same order they were given in case the depth … 

    …is the same
    @MadLittleMods
    MadLittleMods committed May 17, 2023
    Configuration menu
    Copy the full SHA
    7eabc60 View commit details
    Browse the repository at this point in the history

  3. Add comments why OrderedDict

    @MadLittleMods
    MadLittleMods committed May 17, 2023
    Configuration menu
    Copy the full SHA
    7583c2c View commit details
    Browse the repository at this point in the history

  4. Make test more robust around ordering

    @MadLittleMods
    MadLittleMods committed May 17, 2023
    Configuration menu
    Copy the full SHA
    e101318 View commit details
    Browse the repository at this point in the history

  5. Add test description

    @MadLittleMods
    MadLittleMods committed May 17, 2023
    Configuration menu
    Copy the full SHA
    899fc34 View commit details
    Browse the repository at this point in the history

  6. Same order separated results

    @MadLittleMods
    MadLittleMods committed May 17, 2023
    Configuration menu
    Copy the full SHA
    b5aec4f View commit details
    Browse the repository at this point in the history

  7. Refactor to get_event_ids_with_failed_pull_attempts(...)

    @MadLittleMods
    MadLittleMods committed May 17, 2023
    Configuration menu
    Copy the full SHA
    6edd126 View commit details
    Browse the repository at this point in the history

  8. Update comment doc

    @MadLittleMods
    MadLittleMods committed May 17, 2023
    Configuration menu
    Copy the full SHA
    d4b8ff7 View commit details
    Browse the repository at this point in the history

Commits on May 18, 2023

  1. Merge branch 'develop' into madlittlemods/process-previously-failed-e… 

    …vents-in-background
    @MadLittleMods
    MadLittleMods committed May 18, 2023
    Configuration menu
    Copy the full SHA
    6a0ec9d View commit details
    Browse the repository at this point in the history

  2. Use List 

    See #15585 (comment)
    @MadLittleMods
    MadLittleMods committed May 18, 2023
    Configuration menu
    Copy the full SHA
    d843557 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2023

  1. Merge branch 'develop' into madlittlemods/process-previously-failed-e… 

    …vents-in-background

Conflicts:
	synapse/handlers/federation_event.py
    @MadLittleMods
    MadLittleMods committed May 23, 2023
    Configuration menu
    Copy the full SHA
    75bec52 View commit details
    Browse the repository at this point in the history

  2. Trace differentiaed events

    @MadLittleMods
    MadLittleMods committed May 23, 2023
    Configuration menu
    Copy the full SHA
    c4e1533 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2023

  1. Prefer plain language 

    Co-authored-by: Patrick Cloke <[email protected]>
    @MadLittleMods @clokep
    MadLittleMods and clokep authored May 24, 2023
    Configuration menu
    Copy the full SHA
    ec230a3 View commit details
    Browse the repository at this point in the history

  2. Use a set for efficient lookups 

    Since we only use the result from `get_event_ids_with_failed_pull_attempts(...)`
in lookups, we don't need to care about the order.

See:

 - #15585 (comment)
 - #15585 (comment)
    @MadLittleMods
    MadLittleMods committed May 24, 2023
    Configuration menu
    Copy the full SHA
    22a69be View commit details
    Browse the repository at this point in the history

  3. Add some context 

    See #15585 (comment)
    @MadLittleMods
    MadLittleMods committed May 24, 2023
    Configuration menu
    Copy the full SHA
    65febed View commit details
    Browse the repository at this point in the history

  4. Use dedicated partition function to separate list 

    See https://matrix.to/#/!vcyiEtMVHIhWXcJAfl:sw1v.org/$3ygksoSclc6ZlWNCKqysasK3oU_lrLkdeOL6nveDliA?via=matrix.org&via=element.io&via=pixie.town
    @MadLittleMods
    MadLittleMods committed May 24, 2023
    Configuration menu
    Copy the full SHA
    6474b4e View commit details
    Browse the repository at this point in the history

  5. Add context for why source order for MSC2716 

    See #15585 (comment)
    @MadLittleMods
    MadLittleMods committed May 24, 2023
    Configuration menu
    Copy the full SHA
    15527f7 View commit details
    Browse the repository at this point in the history

  6. Add sanity check test that failed pull attempt events are still proce… 

    …ssed even if it is in the background

See #15585 (comment)
    @MadLittleMods
    MadLittleMods committed May 24, 2023
    Configuration menu
    Copy the full SHA
    d59615f View commit details
    Browse the repository at this point in the history

Commits on May 25, 2023

  1. Use obvious type 

    See #15585 (comment)
    @MadLittleMods
    MadLittleMods committed May 25, 2023
    Configuration menu
    Copy the full SHA
    95ffa7c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'develop' into madlittlemods/process-previously-failed-e… 

    …vents-in-background
    @MadLittleMods
    MadLittleMods committed May 25, 2023
    Configuration menu
    Copy the full SHA
    50acf6a View commit details
    Browse the repository at this point in the history