streams - request for docs: can _transform or _write(v) be called in parallel? #3208
Comments
|
I'm fairly sure you're correct - this is a guarantee for streams. I'm surprised that it isn't documented, actually. |
|
/cc @nodejs/documentation? Provisionally adding a doc label. |
bnoordhuis
added
doc
|
also CC @nodejs/streams |
|
@ronkorving correct. For each stream there is only one My module throughv does parallel transform (and it's compatible with streams3). |
addaleax
added
the
good first issue
sam-github
added
doc
|
This has been open for quite some time with the |
Trott
added
help wanted
Add a note to the stream docs specifying that at most a single call to _transform can happen, and the provided callback() should be used to process another chunk. Fixes: nodejs#3208
|
Fixed in 0e5283b. |
Add a note to the stream docs specifying that at most a single call to _transform can happen, and the provided callback() should be used to process another chunk. Fixes: #3208 PR-URL: #14321 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>
|
@mcollina Appreciate the documentation update. But it doesn't quite answer the question I had. Maybe my phrasing was simply unclear. So let me be more specific: During every transform, I write a 10 byte buffer. The contents of that buffer will be different each time. Is it safe to reuse the same 10 byte Buffer instance each transform, or will the 2nd transform change the contents of the 1st transform? My guess is that this is unsafe, but .. that's really where this question originated. Apologies for the lack of clarity. |
|
You cannot reuse buffers in any part of the streams API. |
Add a note to the stream docs specifying that at most a single call to _transform can happen, and the provided callback() should be used to process another chunk. Fixes: #3208 PR-URL: #14321 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>
Add a note to the stream docs specifying that at most a single call to _transform can happen, and the provided callback() should be used to process another chunk. Fixes: #3208 PR-URL: #14321 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>
Add a note to the stream docs specifying that at most a single call to _transform can happen, and the provided callback() should be used to process another chunk. Fixes: #3208 PR-URL: #14321 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>
Add a note to the stream docs specifying that at most a single call to _transform can happen, and the provided callback() should be used to process another chunk. Fixes: #3208 PR-URL: #14321 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>
Add a note to the stream docs specifying that at most a single call to _transform can happen, and the provided callback() should be used to process another chunk. Fixes: #3208 PR-URL: #14321 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>
Add a note to the stream docs specifying that at most a single call to _transform can happen, and the provided callback() should be used to process another chunk. Fixes: #3208 PR-URL: #14321 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>
The reason I ask is because I have a transform-stream that receives objects and turns those into a bunch of buffers for each single transform. Some of these buffers I have to push out always have the same length. If possible (safe) I would like to create those buffers in my transform stream constructor and simply stick new data in during each transform.
So my question: as long as I don't call the callback in _transform, is it safe to assume _transform will not be called again? If my transform is piped down to disk for example, is there a chance that me changing the buffer in the next _transform call could have an effect on the outcome of the first write to disk?
In either case, documentation on this would be most welcome. I think stream implementers (myself in this case) often really need to squeeze out all performance, and understanding how this behaves would help. Also, as long as it's not documented one way or the other, I would be scared to depend on the behavior (even if it behaves the way I hope it does), because it may change tomorrow.
The text was updated successfully, but these errors were encountered: