Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements to initialization logs and monitoring #1040

Closed
istreeter opened this issue Sep 1, 2022 · 0 comments
Closed

Improvements to initialization logs and monitoring #1040

istreeter opened this issue Sep 1, 2022 · 0 comments
Milestone

Comments

@istreeter
Copy link
Contributor

We had a problem where a loader could not move beyond the initialization block. The log messages and error messages made it hard to see exactly where the error happened. There are a few small changes we can make:

Log a message when the target check is successful. The [target check]https://github.com/snowplow/snowplow-rdb-loader/blob/4.2.1/modules/loader/src/main/scala/com/snowplowanalytics/snowplow/rdbloader/loading/TargetCheck.scala#L36] is the thing that just runs SELECT 1. Currently we log a message on a failure, but I think it would be helpful to see in the logs that the loader has successfully connected to the warehouse.

Log a message if there are no changes needed to the manifest table. Referring to this line. It will help to know when the manifest initilialization step is complete and the flow of execution has moved onto the next step.

The alert error message should be more specific where the error occurred. Currently we wrap the entire initialization block in an error handler. Which means the alert we send is simply "Exception from init block" (here). I think we need to wrap each step of the initialization separately, so the alert error messages can be more helpful like "Error waiting for target to be ready" or "Error initializing manifest table".

@istreeter istreeter added this to the 4.2.2 milestone Sep 1, 2022
@spenes spenes closed this as completed in 8e774e3 Sep 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant