Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mention merge* in parallel doc #689

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

dvisztempacct
Copy link

Wasted a good deal of my time trying to figure out why parallel was not performing as efficiently as I expected.

Hopefully this will save others from the same tar pit!

@richardscarrott
Copy link

@dvisztempacct I wonder if you could share any more insight into the difference between parallel and merge?... I have a scenario where I don't care about order, but I do want to constrain the amount of parallelism. e.g.

_(nodeReadableStream)
   .map(wrapAsync(async (obj) => {
          const transformedObj = await transformObj(obj);
          return transformedObj;
   }))
   .parallel(100) // I want to limit the concurrent reads to 100, but would rather it was pushed to the writable stream ASAP
   .pipe(nodeWritableStream); // Doesn't care what order the `transformedObj`s are written in

I was wondering if Highland.prototype.merge would:

a) honour backpressure, or is it like turning on the tap full blast?
b) prevent Highland.prototype.map from being called over n times in parallel?

@richardscarrott
Copy link

I think Highland.prototype.mergeWithLimit is exactly what I'm after -- I had glossed over it for some reason 🤷‍♂️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants