feat: add streaming updates core functionality and project streaming

[corban-beaird]

Description

Add Streaming Updates core functionality with project streaming & testing. This also provides the framework for adding additional streamable types as time goes on. It also provides a master/stream.py demo that can be used to test out the feature.

Offline & Out of Scope testing will need to be added back in due to those test not using streamable types that will be included in this initial merge.

Test Plan

• Automated testing (stream/system_intg_test.go)

Commentary (optional)

Checklist

• Changes have been manually QA'd
• User-facing API changes need the "User-facing API Change" label.
• Release notes should be added as a separate file under docs/release-notes/.
  See Release Note for details.
• Licenses should be included for new code which was copied and/or modified from any external code.

Ticket

DET-10056

[netlify]

✅ Deploy Preview for determined-ui canceled.

Name	Link
🔨 Latest commit	880d884
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/65c68cf3b6353c0008d60218

[codecov]

Codecov Report

Attention: 333 lines in your changes are missing coverage. Please review.

Comparison is base (cfffe96) 47.79% compared to head (880d884) 47.82%.
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8669      +/-   ##
==========================================
+ Coverage   47.79%   47.82%   +0.03%     
==========================================
  Files        1050     1061      +11     
  Lines      167359   168109     +750     
  Branches     2240     2239       -1     
==========================================
+ Hits        79982    80394     +412     
- Misses      87219    87557     +338     
  Partials      158      158              

Flag	Coverage Δ
backend	43.85% <55.71%> (+0.24%)	⬆️
harness	64.32% <ø> (-0.01%)	⬇️
web	42.52% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
master/internal/db/migrations.go	59.57% <100.00%> (ø)
master/internal/db/postgres.go	53.21% <100.00%> (ø)
master/pkg/model/task.go	15.59% <ø> (ø)
master/internal/stream/authz_basic_impl.go	75.00% <75.00%> (ø)
master/pkg/syncx/errgroupx/errgroupx.go	63.63% <0.00%> (-6.37%)	⬇️
master/internal/stream/subscription.go	92.68% <92.68%> (ø)
master/internal/core.go	2.46% <0.00%> (-0.02%)	⬇️
master/internal/stream/socketlike.go	33.33% <33.33%> (ø)
master/internal/stream/messages.go	0.00% <0.00%> (ø)
master/internal/stream/util.go	67.74% <67.74%> (ø)
... and 6 more

... and 3 files with indirect coverage changes

[eecsliu]

Looks good! My only question is whether or not to include stuff in this PR - comments and extra files meant for developing

[eecsliu]

I'll defer to @rb-determined-ai, but I'm ok with this change. I feel like it'd be rare to bump up against the 64 bit int overflow to begin with (but maybe in a few years I'll be changing my tune...)

[gt2345]

ProcessKnown might not be entirely correct if known and/or exist is not sorted, while it is highly possible for exist to be unsorted

[corban-beaird]

I'll add in the assumption to the comment.

known is created by the streaming client, which keeps track of the known keys and builds the known string, and exist is populate by a query for each streamable type (currently just projects).

The project select includes an ORDER <PK> ASC; so we should be be fine in terms of making sure that exists is sorted.

As long as the streaming client does not build an improper known string and any additional streamable types are also sorting the results of their queries properly, this shouldn't be an issue.

[gt2345]

Since exist is basically a concatenation of oldEventsQuery scan and newEventsQuery scan, both queries sorted does not guarantee exist also sorted

[corban-beaird]

Good catch!

[erikwilson]

Just leaving a comment here so that I can be notified properly.
From a slack conversation with Corban it sounds like the code is being re-worked to deal with the character limit.
If the code is actually ready now, or whenever it does become ready, please re-request a review.
Thanks!

[corban-beaird]

Just leaving a comment here so that I can be notified properly. From a slack conversation with Corban it sounds like the code is being re-worked to deal with the character limit. If the code is actually ready now, or whenever it does become ready, please re-request a review. Thanks!

@erikwilson
In an effort to get the core functionality available for the transition with the re-org, the transition over to only communicating using filterable columns will be considered a separate unit of work

[rb-determined-ai]

great work! It's been a long haul