Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DSRE-9] - Upgrade Airflow wtmo to 1.10.15 bridge release prior to 2.0 #1335

Merged
merged 1 commit into from
Jul 20, 2021
Merged

[DSRE-9] - Upgrade Airflow wtmo to 1.10.15 bridge release prior to 2.0 #1335

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions bin/run
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,11 +156,11 @@ init_variables() {

case $1 in
flower)
exec airflow flower
exec airflow celery flower
;;
web)
airflow initdb
airflow upgradedb
airflow db init
airflow db upgrade

# Only init connections in dev
[ ! -z ${DEVELOPMENT+check} ] && init_connections && init_variables
Expand All @@ -169,7 +169,7 @@ case $1 in
exec airflow webserver -p ${PORT} --workers 4
;;
worker)
exec airflow worker
exec airflow celery worker
;;
scheduler)
exec airflow scheduler
Expand Down
2 changes: 1 addition & 1 deletion bin/test-parse
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ function get_errors_in_listing {
# Parse the logs for ERROR messages, these typically correspond to python
# exceptions in the DAG. In general, there should NOT be any errors when
# runnning the local environment.
docker-compose exec web airflow list_dags | grep "ERROR"
docker-compose exec web airflow dags list | grep "ERROR"
}


Expand Down
4 changes: 2 additions & 2 deletions plugins/backfill/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,9 +162,9 @@ def background(self):
# Prepare the command and execute in background
background_cmd = "screen -dmS {} ".format(screen_id)
if clear == 'true':
background_cmd = background_cmd + 'airflow clear -c '
background_cmd = background_cmd + 'airflow tasks clear -c '
elif clear == 'false':
background_cmd = background_cmd + 'airflow backfill '
background_cmd = background_cmd + 'airflow dags backfill '

if use_task_regex == 'true':
background_cmd = background_cmd + "-t {} ".format(task_regex)
Expand Down
15 changes: 12 additions & 3 deletions requirements.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
boto3
kombu==4.6.3 # CeleryExecutor issues with 1.10.2 supposedly fixed in 1.10.5 airflow, but still observed issues on 1.10.7
boto3==1.15.18
botocore<1.19.0,>=1.18.0
kombu==4.6.10 # CeleryExecutor issues with 1.10.2 supposedly fixed in 1.10.5 airflow, but still observed issues on 1.10.7
importlib-metadata==2.1.0
argcomplete==1.12.2
pandas-gbq==0.14.1
# removed hdfs
apache-airflow[celery,postgres,hive,jdbc,async,password,crypto,github_enterprise,datadog,statsd,s3,mysql,google_auth,gcp_api,kubernetes]==1.10.12
apache-airflow[celery,postgres,hive,jdbc,async,password,crypto,github_enterprise,datadog,statsd,s3,mysql,google_auth,gcp_api,kubernetes]==1.10.15
apache-airflow-upgrade-check
# Airflow 2.0 backported providers
apache-airflow-backport-providers-google
apache-airflow-backport-providers-amazon
apache-airflow-backport-providers-http
cryptography>=3.2
mozlogging
retrying
Expand Down
Loading