[AIRFLOW-5285] Pylint pre-commit filters out pylint_todo files

[potiuk]

We have now all pylint checks incorporated into pre-commit hooks including filtering out the files from todo list. Also all checks are now moved to a single job in Travis, utilising multiple processors so it should be even faster and less overhead to run all the tests.

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-5285

Description

• Here are some details about my PR, including screenshots of any UI changes:

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

• Passes flake8

[potiuk]

CC : @nuclearpinguin

[potiuk]

@nuclearpinguin @mik-laj - I'd love if you test in on Mac running for --all-files and for a small change. I enabled parallel execution for the tests and previously when I tested it on Mac I had a feeling it is very slow but likely this was because of image rebuilding in parallel. Now we have a single test at the beginning which rebuilds the images once if needed (and REBUILD=true used) and all the other checks should run fast with already rebuilt image.

[turbaszek]

Sure I will test it ✅

[potiuk]

Also I tried another variant (fixup) - where travis still runs pylint separately. I think this will be best for now as running all tests on Travis took 16 minutes https://travis-ci.org/apache/airflow/jobs/575437986 (we have two CPU machines there only and with limited memory). By having parallel pre-commit on a fast 8CPU machine this will go down to 3-4 minutes, but for now splitting the pylint jobs separately might be much better (but still we run it all via pre-commit). https://travis-ci.org/apache/airflow/builds/575447025

[potiuk]

Just to give a heads-up - seems that it works fine -> it takes < 5 minutes on my Mac to run 'pre-commit run --all-files' :)

[potiuk]

Trying more sensible split trying to optimize image builds and Travis.

[potiuk]

Seems like perfect split now:

[turbaszek]

Running pre-commit run --all-files took 4m32s.

Small change in code and docs took 51s while only change in docs took 32s (I checked this because of feeling that check-apache-license take a relatively long time).

Change in operator, hook and tests took 1m25s so I think it's a quite acceptable time.

[ashb]

Out of interest how long does it take if you delete the check-apache-license step from your local config file?

[turbaszek]

@ashb cira 4m10s, so same difference of ~20 seconds.

[potiuk]

Then for the Travis build we also save time on downloading the image. In the latest version we only pull/build checklicence image for checklicence step, and only pull/build slim-ci image for the rest. That also saves a bit of time. Locally when you run the pre-commit both images will normally be already built so we only save time for running the test. It's highly optimised in this version.

[potiuk]

Anything else needed to merge this one ?