Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: ensure that during resumption of a scan, rows that have not been observed by the caller are re-requested #1444

Commits on Jul 9, 2024

  1. fix: dropping buffered rows during a retry of a scan

    createReadStream() creates a pipeline of streams that converts a stream of row chunks into a stream of logical rows. It also has logic to handle stream resumption when a single attempt fails.
    The pipeline can be split into 2 parts: the persistent operation stream that the caller sees and the transient per attempt segment. When a retry attempt occurs, the per attempt segment is unpiped from the operation stream and is discarded. Currently this includes any buffered data that each stream might contain. Unfortunately, when constructing the retry request, createReadStream() will use the last row key from the last buffered row. This will cause the buffered rows to be omitted from the operation stream.
    
    This PR fixes the missing rows part by only referencing the row keys that were seen by the persistent operation stream when constructing a retry attempt. In other words, this will ensure that we only update the lastSeenRow key once the row has been "committed" to the persistent portion of the pipeline
    igorbernstein2 committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    9e2c037 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. Add a test that sends rows back

    # Conflicts:
    #	system-test/read-rows.ts
    danieljbruce committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    2a4d928 View commit details
    Browse the repository at this point in the history
  2. Create an instance with dummy server.

    # Conflicts:
    #	system-test/read-rows.ts
    danieljbruce committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    d8e2e91 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d803684 View commit details
    Browse the repository at this point in the history
  4. Omit server start

    danieljbruce committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    a8bebb1 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    fdc5c0d View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    80f4fca View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    580d65f View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    b08288a View commit details
    Browse the repository at this point in the history
  9. Add a log

    danieljbruce committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    e4d9b86 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    9d41781 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    2d9b603 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    5bad7c0 View commit details
    Browse the repository at this point in the history
  13. Remove the logging

    danieljbruce committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    e61343d View commit details
    Browse the repository at this point in the history
  14. Remove more logging

    danieljbruce committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    138d105 View commit details
    Browse the repository at this point in the history
  15. Adjust header

    danieljbruce committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    f35fc3e View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    bf74092 View commit details
    Browse the repository at this point in the history
  17. Remove the at accessor

    danieljbruce committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    ff13036 View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    87c57a9 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    09d9155 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    89c7c53 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. Configuration menu
    Copy the full SHA
    f60e790 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d07aa9e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    537760b View commit details
    Browse the repository at this point in the history
  4. Add TODO

    danieljbruce committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    96f98d7 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    7b03b84 View commit details
    Browse the repository at this point in the history
  6. Fix linting issue

    danieljbruce committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    cffcd57 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    52675d1 View commit details
    Browse the repository at this point in the history
  8. readable comment change

    danieljbruce committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    c62ed72 View commit details
    Browse the repository at this point in the history
  9. Update test/readrows.ts

    Co-authored-by: Leah E. Cole <[email protected]>
    danieljbruce and leahecole authored Jul 11, 2024
    Configuration menu
    Copy the full SHA
    d78d119 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    8c0726c View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    eba2195 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    0a86df7 View commit details
    Browse the repository at this point in the history
  13. Merge branch 'fix-missing-rows-with-test-and-fix-for-node-14-plus-wat…

    …ermark-removal' of https://github.com/danieljbruce/nodejs-bigtable into fix-missing-rows-with-test-and-fix-for-node-14-plus-watermark-removal
    danieljbruce committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    85901e5 View commit details
    Browse the repository at this point in the history
  14. Add interface: server writable stream

    Use it to replace any
    danieljbruce committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    88236d1 View commit details
    Browse the repository at this point in the history