is there a way to use different proxies for every instances?

[lxwang42]

some websites have ip blocking measures, i have to change ip for each instance after a while, is there a way i can use puppeteer-cluster to do that without start a new cluster?

[imfht]

I will suggest you handle proxy problem out of headless chrome.
build a proxy-switch-server and set you proxy server to proxy-switch-server.
It looks like

[thomasdondorf]

Yes, perfect answer given by @imfht (thanks!). This is currently not build into this library.

Ether do it with a custom proxy server or develop a solution without puppeteer-cluster.

[lxwang42]

@imfht Thank you for your suggestion, this load balancing strategy is really good for scrapping, yet i want each browser instance stick to one specific socks5 proxy(like tor) so sessions wont get blocked for swapping IP. But I guess puppeteer-cluster is not designed to do this kind of task?

[thomasdondorf]

That is currently not supported, but I could modify the library to support your use case (as it is trivial to implement).

What you need to do is copy one of the browser concurrency files and implement a custom launch function:
https://github.com/thomasdondorf/puppeteer-cluster/blob/master/src/browser/ConcurrencyBrowser.ts#L15

If that works for you, I can change the current implementation to support custom concurrency implementations.

[thomasdondorf]

I added a way in version 0.12 to provide any object as concurrency option. That way, this could be implemented very easy.

The concurrency implementation can be changed to call the launch function with custom arguments (like proxy) for each worker.

Documentation is coming soon.

[thomasdondorf]

Added tests, fixed a bug and added documentation. This feature is now ready to use. Just copy one of the concurreny implementations (like Browser) and provide the class to the concurrency option.

[cyxou]

That is currently not supported, but I could modify the library to support your use case (as it is trivial to implement).

What you need to do is copy one of the browser concurrency files and implement a custom launch function:
https://github.com/thomasdondorf/puppeteer-cluster/blob/master/src/browser/ConcurrencyBrowser.ts#L15

If that works for you, I can change the current implementation to support custom concurrency implementations.

In my case, I don't know all the proxies IP addresses at cluster instantiation. Proxy IPs and corresponding credentials are obtained from a database and once fetched, a cluster worker needs to be created with these proxy credentials and currently, there is no way to pass any arguments to custom ClusterImplementation on queuing a worker task. Any thoughts on this use case?