Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No option to wait until page is loaded #46

Open
kylefrost opened this issue Jan 29, 2020 · 8 comments
Open

No option to wait until page is loaded #46

kylefrost opened this issue Jan 29, 2020 · 8 comments

Comments

@kylefrost
Copy link

I am trying to screenshot a webpage that takes a few moments to fully load, and the script is simply running and screenshotting before the page can load. Is there an option to set a sleep period? Or could one be added?

@maaaaz
Copy link
Owner

maaaaz commented Jan 29, 2020

Hello, try to play with --ajax-max-timeouts.
And take a look at this

@mtdeguzis
Copy link

mtdeguzis commented Mar 4, 2020

My use case is Icingaweb2 (Icinga2 web front end), the page loads, but the screenshot is taken so quick the alerts are not loaded:
image

It would be nice to have the engine load the page, and allow the scrape request to wait X seconds.

Possibly related:

@sbermind
Copy link

sbermind commented May 1, 2020

Hello, try to play with --ajax-max-timeouts.
And take a look at this

Hello!
--ajax-max-timeouts works only for PhantomJS, isn't it?

I have the same problem. Some pages don't load because screenshots is done too fast. (i'm using -r chromium)
It is good idea to add a request waiting. If you add this function i will be very grateful.
Thank you.

@zonicdoe
Copy link

zonicdoe commented May 8, 2020

I'm working on a fork that will use the Devtools protocol to manage the screenshot process for Chromium based browsers such as Chrome. I've ran into the same issue in the past, specially with those sites that have some sort of WAF protection that is not very agressive, but it requires to wait for at least 5 seconds to render the requested page.
With the aid of the Devtools protocol running the browser in remote debugging mode, some of the PhantomJS exclusive functions can be implemented in Chrome, such as image cropping, screenshot delay, http headers and cookies control, etc, etc. The only caveat I see with this implementation is the use of the "websockets" lib needed to interact with the browser, because it requires python 3, and it will break backwards compatibility with python 2 :(
Do you guys know any websocket library compatible with python 2?

@maaaaz
Copy link
Owner

maaaaz commented May 9, 2020

Python 2 compatibility does not have to maintained, particularly for a specific feature.

@zonicdoe
Copy link

zonicdoe commented May 9, 2020

Awesome :)

@zonicdoe
Copy link

zonicdoe commented May 10, 2020

Hello @maaaaz .
Here is my suggested approach to implement the Devtools protocol in the script.
It's not complete yet, and there are a lot of tests to run, but it will give you a general idea of where this is going:
Running Chrome / Chromium in remote debugging mode: Line 325
Screenshot function (currently supporting PDF, JPG and PNG formats, image quality for JPG and screenshot delay): Line 765
I'll be implementing other functions from the PhantomJS renderer and then I will start the heavy testing, but for now I will be uploading new changes to the experimental branch.
Regards ;)
Edit:
Added --crop option support: Line 950
Edit:
Added --cookie option support: Line 828
Edit:
Added --header option support: Line 815

@MarcoLeder-zz
Copy link

Dear maaaaz

Really looking forward for the implementation of this feature - hopefully you'll be able to add it soon :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants