Here we go...

[stephenplusplus]

Re-worked some things, comments inline;

[stephenplusplus]

Instead of:

• For every pull from this stream, emit 200 rows at once

I changed it to:

• For every pull from this stream, emit 1 row

Then, you don't have to get into the flow control aspect of Node stream internals with highWatermark-- we will let the requesting stream pull as it needs and get one thing at a time.

[chrishiestand]

That is way better thank you.

Obviously it's not important in this case, but do you have any idea if reading one item at a time would have performance implications in other projects?

[stephenplusplus]

I can't say definitively, but in general, I would trust Node's internals to handle as much as possible. So switching to the pattern of pull (1) -> get (1) where possible should show performance improvements. If you aren't seeing good performance, play with highWaterMark (16kb default / 16 objects default for objectMode), but still return one thing at a time from read.

(If you find that to be bad advice, let me know!)

[chrishiestand]

Thank you, this sounds like good advice and I'll write readable streams this way in the future.

[stephenplusplus]

This isn't necessary, but using the job's events, we can skip the optimistic delay later in the test, and instead know for certain that the rows were inserted.

This also gives us what turned out to be the key component of this issue, the logging of the job object's state. Most of the time, everything works as expected. However, an occasional test run returns:

  Rejected promise returned by test. Reason:

  ErrorClass {
    code: 503,
    errors: [
      {
        domain: 'global',
        message: 'Error encountered during execution. Retrying may solve the problem.',
        reason: 'backendError',
      },
    ],
    message: 'Error encountered during execution. Retrying may solve the problem.',
    response: undefined,
  }

  Object.parseHttpRespBody (node_modules/@google-cloud/common/src/util.js:192:30)
  Object.handleResp (node_modules/@google-cloud/common/src/util.js:132:18)
  Request._callback (node_modules/@google-cloud/common/src/util.js:255:14)
  Request.self.callback (node_modules/request/request.js:186:22)
  Request.<anonymous> (node_modules/request/request.js:1163:10)
  IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
  Test.__dirname [as fn] (test/many-streams.test.js:189:38)

I'm going to take this finding back to the original issue and discuss what we should do about it.

[chrishiestand]

Perfect, thank you. I didn't quite understand how to use job from the documentation but this makes much more sense.