-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
629 organize the daily wys dag #1037
Conversation
…ityofToronto/bdit_data-sources into 629-organize-the-daily-wys-dag
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great! Thanks for speeding this chonk task up. Only have one minor comment on sql.literal.
wys/api/README.md
Outdated
|
||
## `api_main` Process | ||
|
||
The main function in the puller script `api_main` performs the following steps: | ||
|
||
1. Parse the `config.cfg` file (for API key and database credentials) and any input parameters. | ||
2. Retrieve the list of all signs from the `signs` endpoint by calling `location_id()` function, if `location_flag` is `0`. Insert/update into `wys.locations` using `wys.daily_intersections` as a staging table to identify new and updated signs. | ||
1. Retrive API key and database credentials from Airflow variables (must be run in `airflow_venv`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
retrieve typo and a missing )
wys/api/python/wys_api.py
Outdated
DELETE FROM wys.raw_data | ||
WHERE datetime_bin >= {}::timestamp AND datetime_bin < {}::timestamp + interval '1 day'; | ||
""") | ||
delete_sql = delete_sql.format(sql.Literal(str(start_date)), sql.Literal(str(start_date))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should try not to use sql.Literal
as much as we can and use %s
as the placeholder instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
:gabe-approved:
What this pull request accomplishes:
signs
instead ofstatistics
)wys.aggregate_speed_counts_one_hour_5kph
failing with duplicate key error #720)pull_wys
task into 4 parts:get_signs
,pull_wys
(pull data for each sign),update_wys_locations
,agg_speed_counts_hr
read_google_sheets
(see testing in#airflow_pipelines-dev
). (Simplify the custom Slack message for WYS pull #715)Issue(s) this solves:
wys.aggregate_speed_counts_one_hour_5kph
failing with duplicate key error #720, Improve response handling from WYS API #816What, in particular, needs to reviewed:
wys_api.py
: the changes are pretty significant so may want to review without looking at diff.What needs to be done by a sysadmin after this PR is merged