-
Notifications
You must be signed in to change notification settings - Fork 56
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix: ensure that during resumption of a scan, rows that have not been…
… observed by the caller are re-requested (#1444) * fix: dropping buffered rows during a retry of a scan createReadStream() creates a pipeline of streams that converts a stream of row chunks into a stream of logical rows. It also has logic to handle stream resumption when a single attempt fails. The pipeline can be split into 2 parts: the persistent operation stream that the caller sees and the transient per attempt segment. When a retry attempt occurs, the per attempt segment is unpiped from the operation stream and is discarded. Currently this includes any buffered data that each stream might contain. Unfortunately, when constructing the retry request, createReadStream() will use the last row key from the last buffered row. This will cause the buffered rows to be omitted from the operation stream. This PR fixes the missing rows part by only referencing the row keys that were seen by the persistent operation stream when constructing a retry attempt. In other words, this will ensure that we only update the lastSeenRow key once the row has been "committed" to the persistent portion of the pipeline * Add a test that sends rows back # Conflicts: # system-test/read-rows.ts * Create an instance with dummy server. # Conflicts: # system-test/read-rows.ts * Add test parameters for sending back the right chu * Omit server start * Run a test against the server in the old code * Add logging to the chunk transformer * Add logging to indicate that the server received r * chunk transformer logs and other logs * Add a log * Don’t change the old server * Add another guard against the logs * Add setImmediate everywhere that it needs to be * Remove the logging * Remove more logging * Adjust header * Add the high watermarks back in * Remove the at accessor * Eliminate the watermark adjustments * Introduce the watermarks back in * Reduce the number of watermark removals to 1. * Reverted the streamEvents pipeline * Add some comments for introducing the new waterma * Remove comments and console logs. Add TODO. * Add TODO * refactor the test with a helper method * Fix linting issue * Adding a comment about the mock * readable comment change * Update test/readrows.ts Co-authored-by: Leah E. Cole <[email protected]> * It uses 150 rows not 1000 rows * Add a TODO for making more specific typing * Add some TODOs for better factoring * Add interface: server writable stream Use it to replace any --------- Co-authored-by: Igor Berntein <[email protected]> Co-authored-by: Leah E. Cole <[email protected]>
- Loading branch information
1 parent
c86d456
commit 2d8de32
Showing
5 changed files
with
418 additions
and
15 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.