Add @now, @latest, @start_date, & @end_date exec date short-cuts to a… #1408

r39132 · 2016-04-20T06:17:21Z

Dear Airflow Maintainers,

Please accept the following PR that

Addresses the following issues (links to issues addressed by this PR) : Add @latest, @now, @start_date, @end_date to the airflow test CLI #1398

Reminder to contributors:

You must add an Apache License header to all new files
Please squash your commits when possible and follow the 7 rules of good Git commits

…irflow test

landscape-bot · 2016-04-20T06:20:54Z

Repository health decreased by 0.07% when pulling 0670f16 on r39132:master into c1f485f on airbnb:master.

6 new problems were found (including 0 errors and 1 code smell).
No problems were fixed.

criccomini · 2016-04-20T06:35:47Z

airflow/bin/cli.py

@@ -607,6 +636,10 @@ class CLIFactory(object):
        'execution_date': Arg(
            ("execution_date",), help="The execution date of the DAG",
            type=parsedate),
+        'execution_date_enhanced': Arg(


Why not support this as part of execution_date, rather than having two different params?

I went that way first. At the point where the type converter (e.g. int, parsedate, etc...) is used from the argparse module, I don't have access to the dag. Hence, I need to execute these conversions later in code execution

You can remove the type argument in execution_date and take charge. That would ensure consistent behavior of execution_date across the CLI.

@mistercrunch
I see execution date used in 4 CLI commands currently : render, run, test, and task_state. task_state seems possible, but I'm not familiar with how run is used internally. Also, what does render do? I can possibly look at providing uniform handling if you think this feature makes sense for all of the commands above.

coveralls · 2016-04-20T06:40:02Z

Coverage increased (+0.1%) to 67.172% when pulling 0670f16a1ebd06f9cc1882540cd60c045c3a2ca9 on r39132:master into c1f485f on airbnb:master.

landscape-bot · 2016-04-20T16:05:30Z

Repository health decreased by 0.20% when pulling eb3e098 on r39132:master into c1f485f on airbnb:master.

8 new problems were found (including 2 errors and 1 code smell).
No problems were fixed.

landscape-bot · 2016-04-20T16:14:09Z

Repository health decreased by 0.26% when pulling 348cd1f on r39132:master into c1f485f on airbnb:master.

9 new problems were found (including 3 errors and 1 code smell).
No problems were fixed.

landscape-bot · 2016-04-20T16:33:44Z

Repository health decreased by 0.19% when pulling 7b8eb7d on r39132:master into c1f485f on airbnb:master.

7 new problems were found (including 2 errors and 1 code smell).
No problems were fixed.

landscape-bot · 2016-04-20T17:01:14Z

Repository health decreased by 0.07% when pulling 02e94d0 on r39132:master into c1f485f on airbnb:master.

7 new problems were found (including 0 errors and 1 code smell).
No problems were fixed.

landscape-bot · 2016-04-20T22:13:51Z

Repository health decreased by 0.05% when pulling 659caaf on r39132:master into c1f485f on airbnb:master.

5 new problems were found (including 0 errors and 1 code smell).
No problems were fixed.

landscape-bot · 2016-04-20T22:24:04Z

Code quality remained the same when pulling 7f60cd3 on r39132:master into c1f485f on airbnb:master.

landscape-bot · 2016-04-20T23:27:54Z

Code quality remained the same when pulling 67c8388 on r39132:master into c1f485f on airbnb:master.

landscape-bot · 2016-04-20T23:41:05Z

Code quality remained the same when pulling 09d9461 on r39132:master into c1f485f on airbnb:master.

coveralls · 2016-04-20T23:48:37Z

Coverage increased (+0.07%) to 67.098% when pulling 67c838890f70ad1a2d1307172c96ec46cbae1553 on r39132:master into c1f485f on airbnb:master.

jlowin · 2016-04-20T23:57:28Z

airflow/bin/cli.py

+    '''
+    timestr = args.execution_date
+    now = datetime.now()
+    start_date = dag.default_args['start_date'] if \


I think default args is always a dict, so maybe just start_date = dag.default_args.get('start_date')? Same for end_date.

Thought # 2 -- there are actually a few different "start dates"

dag.start_date (could be None)

dag.default_args['start_date'] (could be None or not even exist)

dag.tasks[i].start_date (could be None)

In the scheduler's logic, if a DAG has a start date, that takes precedence. If it doesn't, then min(t.start_date for t in dag.tasks) is checked. Each task's start_date is either explicitly provided OR taken from default_args.

I'm not sure if that level of detail is required, but it occurs to me that default_args might not be a robust place to look for the start date

coveralls · 2016-04-21T00:08:37Z

Coverage increased (+0.07%) to 67.098% when pulling 09d94614c29811cb503d2cea14e9a616d1d05701 on r39132:master into c1f485f on airbnb:master.

landscape-bot · 2016-04-21T04:49:17Z

Repository health decreased by 0.01% when pulling afee7b9 on r39132:master into c1f485f on airbnb:master.

2 new problems were found (including 0 errors and 1 code smell).
No problems were fixed.

landscape-bot · 2016-04-21T04:53:12Z

Repository health increased by 0.03% when pulling 5a8251c on r39132:master into c1f485f on airbnb:master.

1 new problem was found (including 0 errors and 0 code smells).
No problems were fixed.

coveralls · 2016-04-21T05:12:39Z

Coverage increased (+0.08%) to 67.105% when pulling 5a8251c43d66f68e93913fd2392314ecad297ef0 on r39132:master into c1f485f on airbnb:master.

coveralls · 2016-04-21T05:23:52Z

Coverage increased (+0.08%) to 67.105% when pulling 5a8251c43d66f68e93913fd2392314ecad297ef0 on r39132:master into c1f485f on airbnb:master.

landscape-bot · 2016-04-21T05:42:31Z

Repository health increased by 0.03% when pulling f4c0b3a on r39132:master into c1f485f on airbnb:master.

2 new problems were found (including 0 errors and 0 code smells).
No problems were fixed.

coveralls · 2016-04-21T06:02:26Z

Coverage increased (+0.09%) to 67.121% when pulling f4c0b3abfc5233ec71854bdf4222148349b18173 on r39132:master into c1f485f on airbnb:master.

mistercrunch · 2016-04-21T15:36:00Z

airflow/bin/cli.py

@@ -336,6 +336,83 @@ def list_tasks(args, dag=None):
        print("\n".join(sorted(tasks)))


+def get_start_date(dag):


Shouldn't this be a method or @Property of airflow.models.DAG?

It's a good idea to centralize this logic, since it's also used by the scheduler. I think a private method like dag._get_start_date() would be best since we already have a start_date attribute that would be returned by the method unless start_date is None.

landscape-bot · 2016-04-25T06:46:33Z

Repository health increased by 0.09% when pulling 82fbb4e on r39132:master into 12b542d on airbnb:master.

4 new problems were found (including 0 errors and 0 code smells).
2 problems were fixed (including 1 error and 1 code smell).

coveralls · 2016-04-25T07:06:26Z

Coverage decreased (-0.02%) to 67.129% when pulling 82fbb4eac2ed156cba7fc7db83eaaec0689585d9 on r39132:master into 12b542d on airbnb:master.

landscape-bot · 2016-04-28T01:38:57Z

Repository health increased by 0.08% when pulling c1f485f on r39132:master into 1909790 on airbnb:master.

No new problems were introduced.
2 problems were fixed (including 1 error and 1 code smell).

criccomini reviewed Apr 20, 2016
View reviewed changes

r39132 force-pushed the master branch from 0670f16 to eb3e098 Compare April 20, 2016 16:03

r39132 force-pushed the master branch from eb3e098 to 348cd1f Compare April 20, 2016 16:11

r39132 force-pushed the master branch from 348cd1f to 7b8eb7d Compare April 20, 2016 16:31

r39132 force-pushed the master branch from 7b8eb7d to 02e94d0 Compare April 20, 2016 16:58

r39132 force-pushed the master branch from 02e94d0 to 659caaf Compare April 20, 2016 22:10

r39132 force-pushed the master branch from 659caaf to 7f60cd3 Compare April 20, 2016 22:20

r39132 force-pushed the master branch from 7f60cd3 to 67c8388 Compare April 20, 2016 23:24

jlowin reviewed Apr 20, 2016
View reviewed changes

r39132 force-pushed the master branch from 09d9461 to afee7b9 Compare April 21, 2016 04:47

r39132 force-pushed the master branch from afee7b9 to 5a8251c Compare April 21, 2016 04:51

r39132 force-pushed the master branch from 5a8251c to f4c0b3a Compare April 21, 2016 05:32

mistercrunch reviewed Apr 21, 2016
View reviewed changes

r39132 force-pushed the master branch from f4c0b3a to 82fbb4e Compare April 25, 2016 06:44

r39132 closed this Apr 28, 2016

r39132 force-pushed the master branch from 82fbb4e to c1f485f Compare April 28, 2016 01:35

asfgit pushed a commit that referenced this pull request Dec 26, 2017

closes incubator-airflow/#1408

9e5fbad

asfgit pushed a commit that referenced this pull request Dec 26, 2017

closes #1408 *Already Merged

56f4219

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add @now, @latest, @start_date, & @end_date exec date short-cuts to a… #1408

Add @now, @latest, @start_date, & @end_date exec date short-cuts to a… #1408

r39132 commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

criccomini Apr 20, 2016

r39132 Apr 20, 2016

mistercrunch Apr 21, 2016

r39132 Apr 25, 2016 •

edited

Loading

coveralls commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

coveralls commented Apr 20, 2016

jlowin Apr 20, 2016

jlowin Apr 21, 2016 •

edited

Loading

coveralls commented Apr 21, 2016

landscape-bot commented Apr 21, 2016

landscape-bot commented Apr 21, 2016

coveralls commented Apr 21, 2016

coveralls commented Apr 21, 2016

landscape-bot commented Apr 21, 2016

coveralls commented Apr 21, 2016

mistercrunch Apr 21, 2016

jlowin Apr 21, 2016

landscape-bot commented Apr 25, 2016

coveralls commented Apr 25, 2016

landscape-bot commented Apr 28, 2016

		@@ -336,6 +336,83 @@ def list_tasks(args, dag=None):
		print("\n".join(sorted(tasks)))


		def get_start_date(dag):

Add @now, @latest, @start_date, & @end_date exec date short-cuts to a… #1408

Add @now, @latest, @start_date, & @end_date exec date short-cuts to a… #1408

Conversation

r39132 commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

criccomini Apr 20, 2016

Choose a reason for hiding this comment

r39132 Apr 20, 2016

Choose a reason for hiding this comment

mistercrunch Apr 21, 2016

Choose a reason for hiding this comment

r39132 Apr 25, 2016 • edited Loading

Choose a reason for hiding this comment

coveralls commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

landscape-bot commented Apr 20, 2016

coveralls commented Apr 20, 2016

jlowin Apr 20, 2016

Choose a reason for hiding this comment

jlowin Apr 21, 2016 • edited Loading

Choose a reason for hiding this comment

coveralls commented Apr 21, 2016

landscape-bot commented Apr 21, 2016

landscape-bot commented Apr 21, 2016

coveralls commented Apr 21, 2016

coveralls commented Apr 21, 2016

landscape-bot commented Apr 21, 2016

coveralls commented Apr 21, 2016

mistercrunch Apr 21, 2016

Choose a reason for hiding this comment

jlowin Apr 21, 2016

Choose a reason for hiding this comment

landscape-bot commented Apr 25, 2016

coveralls commented Apr 25, 2016

landscape-bot commented Apr 28, 2016

r39132 Apr 25, 2016 •

edited

Loading

jlowin Apr 21, 2016 •

edited

Loading