Add ability to list jobs

[rojer]

With filtering by state and tags.

ListJobs call is added to the Storage interface and implemented for both
in-memory and RDBMS storage plugins. RDBMS implementation requires
database schema changes to allow efficient lookups:

• "state" column is added to the jobs table
• "job_tags" table is added to map tags to job_id

Database migration script is provided to create and populate the new
columns. Going forward, they will be maintained by the server.

[codecov]

Codecov Report

Merging #213 (e1b68f1) into master (1a15df7) will increase coverage by 0.50%.
The diff coverage is 70.40%.

@@            Coverage Diff             @@
##           master     #213      +/-   ##
==========================================
+ Coverage   61.35%   61.86%   +0.50%     
==========================================
  Files          84       88       +4     
  Lines        4117     4345     +228     
==========================================
+ Hits         2526     2688     +162     
- Misses       1270     1324      +54     
- Partials      321      333      +12     

Flag	Coverage Δ
integration	56.71% <66.97%> (+0.43%)	⬆️
integration_storage	100.00% <ø> (ø)
unittests	49.74% <13.15%> (-1.73%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/api/api.go	58.13% <0.00%> (-8.83%)	⬇️
pkg/api/event.go	62.50% <0.00%> (-8.93%)	⬇️
pkg/api/response.go	0.00% <0.00%> (ø)
pkg/jobmanager/jobmanager.go	54.58% <0.00%> (-0.51%)	⬇️
pkg/jobmanager/list.go	0.00% <0.00%> (ø)
pkg/jobmanager/status.go	0.00% <0.00%> (ø)
pkg/storage/events.go	50.00% <ø> (ø)
pkg/storage/storage.go	33.33% <ø> (ø)
pkg/transport/http/http.go	0.00% <0.00%> (ø)
plugins/storage/rdbms/request.go	59.32% <25.00%> (-6.72%)	⬇️
... and 19 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a15df7...e1b68f1. Read the comment docs.

[tfg13]

Mostly nits, 2 main things:

• I am really scared by the SQL stuff, see inline
• Will the list API call return all jobs that ever ran and match the filters? I.e. will this return 100000s of jobs potentially? Should it? Do we know approx how many jobs until it OOMs or the DB aborts the query because it is too large?

[rojer]

forgot to respond to this one, so here goes:

Mostly nits, 2 main things:

• I am really scared by the SQL stuff, see inline

addressed

• Will the list API call return all jobs that ever ran and match the filters? I.e. will this return 100000s of jobs potentially? Should it? Do we know approx how many jobs until it OOMs or the DB aborts the query because it is too large?

so, yes, the API has no pagination support at the moment. yes, unfiltered query will return all 100000 or so jobs, but for now it's just 100K integers, so it's not so bad. the first real use case for this will be to find all the paused jobs at the instance startup, of which there will never be more than a single instance can handle anyway, so it's fine.
i can add an explicit TODO to add pagination to API and put a hard limit on the number of returned results.

[rojer]

rebased

[rojer]

addressed 2 comments i somehow missed before