[AIRFLOW-5373] Super fast pre-commit check for basic python2 compatib…

[potiuk]

…ility

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-5373

Description

• Here are some details about my PR, including screenshots of any UI changes:

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

• Passes flake8

[potiuk]

@kaxil @ashb -> that might be a life-saver for cherry-picking to v1-10-test branch. Not perfect but I think even such simple regexp can save us hours of waiting for CI.

[potiuk]

Running pre-commit run python2-fastcheck --all-files takes milliseconds

[potiuk]

I also figured that we can simply do py_compile on all files. It works great too.

[potiuk]

I think both together works beautifully and can save us many hours https://travis-ci.org/apache/airflow/jobs/579595829 - if you configure them as pre-commit checks especially all those problems will not even leave your local sources (I have both pre-commit and pre-push configured and it works great as long as we merge #5972

[potiuk]

We can even add python3 py_compile in master as well - I think it's rather nice way of catching basic syntax errors.

[potiuk]

@kaxil @ashb -> WDYT ? I think it can be really helpful for further cherry-picking.

[ashb]

This one is a false positive -- this is Scala code :D

[potiuk]

Aaaaah indeed. So I need to add a a possibility to exclude such false positives. Will do soon.

[ashb]

Looks like https://pre-commit.com/#regular-expressions to exclude the whole file is the only way to do this one. A bit overly broad but this file isn't changed very often so that's probably okay. That or adjust the regex to not allow ) { on the line (trickier regex to get that working)

[potiuk]

Actually I thought about writing a bit more complex regex to exclude lines which have the standard # noqa in the same line - so that you can explicitly disable it if needed.

Here is the regex that will likely work :)

^(?!.*# noqa)(.*\.super\(\)|^\s*def\s*\S*\([^):]*:.*|^\sdef\s*\S*\(.*\):\s*\-\>\s*\S*).*$

[potiuk]

And likely with (?x) I can write in multiple lines to make it easier to read :)

[potiuk]

But I also remember about "you have one problem - introduce regexp - then you will have two problems" saying, so maybe it's even worth to write our own simple hook for that purpose if you think it's too "nerdy" with the regexp.

[ashb]

I used to work as a Perl developer for about 5 years, so I am in favour of regeps :D You can take the programmer out of perl, but not the perl out of the programmer.

[ashb]

If each pygrep test is quick, might it be easier to split the super case from the type annotation case into separate rules?

[potiuk]

Actually if you are a perl programmer - then you will find the final result well.. refreshing :).

I found that with (?x) extension you can not only write elegant and pretty "modular" code but also one that can contain comments (!).

Also the exclusion rules can be rather elegant :).

Surprisingly I have now regexp that I am actually proud of!

[potiuk]

Hold on. Just checking super() matching :)

[potiuk]

Now it's perfect. It detects 846 Python3-specific lines in master and 0 in v1-10-test :)

[kaxil]

LGTM