Material on luigi presentation from LPUG meeting on 10/11/2015. Slides.
Create some sample input data first:
$ make plays.ldj
$ head plays.ldj
{"album": "0", "track": "10", "date": "2015-10-26", "artist": "Aluminica"}
{"album": "1", "track": "8", "date": "2015-10-23", "artist": "Aluminica"}
{"album": "2", "track": "6", "date": "2015-10-03", "artist": "The Carpenters"}
{"album": "3", "track": "10", "date": "2015-10-17", "artist": "Aluminica"}
{"album": "2", "track": "8", "date": "2015-10-29", "artist": "Aluminica"}
{"album": "0", "track": "8", "date": "2015-10-23", "artist": "The Carpenters"}
{"album": "2", "track": "6", "date": "2015-10-23", "artist": "Aluminica"}
{"album": "1", "track": "0", "date": "2015-10-21", "artist": "The Carpenters"}
{"album": "1", "track": "3", "date": "2015-10-19", "artist": "Rolling Waters"}
{"album": "1", "track": "3", "date": "2015-10-17", "artist": "Aluminica"}
The date values range from 2015-10-01
to 2015-10-30
.
Create a virtualenv and install deps:
$ mkvirtualenv lpug-luigi
(lpug-luigi) $ pip install luigi
There are four tasks in main.py
. Run a single task with local scheduler only:
(lpug-luigi) $ python main.py DailyPlays --date 2015-10-08 --local-scheduler
[2015-11-11 ...] Informed scheduler that task DailyPlays(date=2015-10-08) ... PENDING
[2015-11-11 ...] Informed scheduler that task InputFile() has status DONE
[2015-11-11 ...] Done scheduling tasks
[2015-11-11 ...] Running Worker with 1 processes
[2015-11-11 ...] Starting pruning of task graph
[2015-11-11 ...] Done pruning task graph
[2015-11-11 ...] [pid 87712] Worker Worker(...) running DailyPlays(date=2015-10-08)
[2015-11-11 ...] [pid 87712] Worker Worker(...) done DailyPlays(date=2015-10-08)
[2015-11-11 ...] Informed scheduler that task DailyPlays(date=2015-10-08) ... DONE
[2015-11-11 ...] Starting pruning of task graph
[2015-11-11 ...] Done pruning task graph
[2015-11-11 ...] Done
[2015-11-11 ...] There are no more tasks to run at this time
[2015-11-11 ...] Worker Worker(...) was stopped. Shutting down Keep-Alive thread
[2015-11-11 ...]
===== Luigi Execution Summary =====
Scheduled 2 tasks of which:
* 1 present dependencies were encountered:
- 1 InputFile()
* 1 ran successfully:
- 1 DailyPlays(date=2015-10-08)
This progress looks :) because there were no failed tasks or ... dependencies
===== Luigi Execution Summary =====
(lpug-luigi) $ python main.py DailyTopArtists --date 2015-10-08 --local-scheduler
...
(lpug-luigi) $ cat DailyTopArtists/2015-10-08.tsv
Aluminica 13776
Ono 11052
Red Floyd 2824
Rolling Waters 2818
The Carpenters 2796
To run with luigi daemon, start luigid
in a separate terminal.