-
Notifications
You must be signed in to change notification settings - Fork 314
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improved stability and error logging #449
Conversation
This also better handles errors if we still fail after retrying (so the aggregator process doesn't crash).
The current inter-process log handler discards `exc_info` and adds the arguments to the exception message. This prevents Sentry from properly grouping log messages by their exception and call information. We send the information if possible (or fall back to the previous approach if not).
@motin This PR is ready for review. I ran a 10k site crawl with this branch and here are the improvements: First, some positives:
Remaining problems:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very good news! I am happy with merging this and having the remaining issues be tackled in separate PRs
@@ -133,7 +133,7 @@ def get_website(url, sleep, visit_id, webdriver, | |||
alert = webdriver.switch_to_alert() | |||
alert.dismiss() | |||
time.sleep(1) | |||
except TimeoutException: | |||
except (TimeoutException, WebDriverException): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will completely suppress the exceptions that are reported in #404 and prevent being able to properly address these errors as per #404 (comment). When addressing #404, we must remember to change this back to throwing exceptions, so that we can reproduce the root cause via tests, report upstream etc.
This is addressing some of the errors observed in a recent large crawl.