Roadmap for v1.0

[thomasdondorf]

I'm thinking about what kind of functionality this library should provide before it should be released as v1. I might edit the list in the future:

My goals:

• (Long-term runs of puppteer-cluster #25) Make sure it's reliable and crawl more than 10 million pages with it (so far the maximum I crawled was ~800k pages)
• (Use TLD.js for sameDomainDelay #9) Improve sameDomainDelay and skipDuplicateUrls. Detection of domains should use TLD.js for example. Documentation should be better. And there should be a way to provide the URL without using data or { url: ... } Not a goal for 1.0 anymore
• (Split up Cluster.ts #28) Optimize the code, fix code smells
• More tests, get code coverage up to > 90%
• More documentation on the concurrency types. Maybe make CONCURRENCY_BROWSER the default as it is more robust?
• More code snippets in the documentation page (for Cluster.queue for example)
• Provide a cluster.execute function which executes the job with higher priority (does not queue it at the end) and returns a Promise which is resolved when the job is finished. Might also solve this confusion: why await?  #10 (comment)
• Statistics API: How many jobs in queue, how many jobs processes, etc.
• Use "puppeteer-core" instead of "puppeteer" #41 Offer more functionality, maybe provide a way to use puppeteer-extra?
• is there a way to use different proxies for every instances? #36 Sandbox Offer a way to run code from users in a sandbox, maybe even Docker? => This can now be implemented via custom concurrency implementations (although there are now custom implementations right now)
• Minor type checking improvement for cluster.queue method #70 Improve types

Maybe:

• Provide a simple but robust data store with the library
• Rename API: Some parts of API are rather unfortunate
  • concurrency should be concurrencyType
  • maxConcurrency maybe maxWorkers?
• Provide queue function to the task function for a more functional syntax (so that you don't need to access cluster from inside the task

Not planned (for now):

• Roadmap for v1.0 #8 (comment) Mixed concurrency models
  • Reason: It does not work well together with the idea of having a sandbox (which part of the browser/page/context stuff should be sandboxed then)

[barpaw]

I have a question. How many browsers I can spawn in parallel for processor core? Lets Say my server has processor with 4 cores. How many browsers I can spawn in one time for my tests to pass?

[thomasdondorf]

Next time, please open a separate issue if it has nothing to do with this issue.

Regarding your question: It depends on your use case. For simple DOM handling I was able to run ~10 worker on my machine (i5 quad core). Just give it a try with the option (monitor: true) and see how your machine is handling the tasks.

[j-manu]

1. Add a mixed concurrency model. i.e for PAGE or CONTEXT concurrency model, have the option to distribute the jobs to more than one browser instance. So a crash won't affect all jobs and this offers a good balance between reliability and resource usage.

2. Add API to return the length of queue, time when the oldest item in queue was added and Number of jobs processed in the last minute. For a continuously operating cluster i.e jobs being added continuously, this information is valuable.

[cyxou]

Unfortunately, the current implementation of custom concurrency doesn't address the case when you need to provide custom puppeteer parameters to jobInstances. IMHO this would effectively solve the #36 with puppeteer args: [ '--incognito', '--proxy-server=${proxyServer}' ] and await page.authenticate(credentials).

@thomasdondorf , what do you think about this?

[thomasdondorf]

I'm currently thinking about completely reworking the concurrency implementations. Then there would be no more "WorkerInstance" and "JobInstance". Just one function that is called when a page is needed. Then the concurrency implementation would have 100% flexibility when a puppeteer instance is started and when one is reused.

Expect some code changes in the next two weeks ;)

[cyxou]

Cool, glad to hear that. Feel free to ping me if you need any help)

[strarsis]

+1 for Docker container support.
https://github.com/skalfyfan/dockerized-puppeteer

[ermolaev1337]

Is there a way to connect the puppeteer-cluster to a remote instance of chromium? (“connect” instead of “launch”)

[generic11]

Hello - just wanted to get a feel for how active this project is. I see puppeteer cluster as being useful for several projects I'd like to work on. However, I'm hesitant to use it if development will be abandoned. Is development still happening? Thanks!

[rennokki]

(Long-term runs of puppteer-cluster #25) Make sure it's reliable and crawl more than 10 million pages with it (so far the maximum I crawled was ~800k pages)

I use k6 benchmarks in my CI tests for soketi, making sure all releases are passing benchmarks in most of the cases.

Would it be a great idea to set it up for you for page rendering testing?