Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: Convert typings to mypy #311

Merged
merged 28 commits into from
Aug 18, 2020
Merged

chore: Convert typings to mypy #311

merged 28 commits into from
Aug 18, 2020

Commits on Aug 18, 2020

  1. requirements: add mypy 

    This version is a few ahead of the other amundsen services, but includes
better defaults.

Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    f72506a View commit details
    Browse the repository at this point in the history

  2. setup.cfg: update mypy config for python3 

    warn_no_return and strict_optional are now default-on

Also try disabling ignore_missing_imports

Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    784d2a5 View commit details
    Browse the repository at this point in the history

  3. setup.cfg: ignore ignore_missing_imports for now 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    20bc26f View commit details
    Browse the repository at this point in the history

  4. makefile: add mypy task, include in test 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    da39c23 View commit details
    Browse the repository at this point in the history

  5. README: type comment -> annotation 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    3687fa4 View commit details
    Browse the repository at this point in the history

  6. chore: move most type comments to annotations (automated via com2ann) 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    35f3c73 View commit details
    Browse the repository at this point in the history

  7. typings: start fixing comments left by com2ann 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    b582ea5 View commit details
    Browse the repository at this point in the history

  8. typings: add missing type imports 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    afa1b16 View commit details
    Browse the repository at this point in the history

  9. typings: fix syntax errors caused by com2ann 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    197fb11 View commit details
    Browse the repository at this point in the history

  10. typings: fix typo'd Iterator pattern 

    This was either messed up by com2ann, or was incorrect before.

Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    8fd7550 View commit details
    Browse the repository at this point in the history

  11. typings: fix some parse ordering issues with type annotations 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    adfb73f View commit details
    Browse the repository at this point in the history

  12. typing: fix other compile errors left by com2ann 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    875907c View commit details
    Browse the repository at this point in the history

  13. typing: fix some leftover broken annotations 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    d6c1e9f View commit details
    Browse the repository at this point in the history

  14. typings: add missing typings checkpoint (197 errors remain) 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    043ef47 View commit details
    Browse the repository at this point in the history

  15. tests: HACK to preserve trailing whitespace in query test 

    I edit with trim_trailing_white_space_on_save enabled in Sublime, editing
anything else in this file will cause this sensitive test to break.

This isn't the prettiest way to fix this, but I think this is enough of a
gotcha that the previous status quo is not acceptable.

Very open to feedback if there's a better way or established pattern to fix
this footgun.

Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    3802832 View commit details
    Browse the repository at this point in the history

  16. typing: checkpoint mypy fixes (124 errors remain) 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    8d425b9 View commit details
    Browse the repository at this point in the history

  17. typings: checkpoint for fixed mypy errors (84 remain) 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    5112f6f View commit details
    Browse the repository at this point in the history

  18. bigquery: work around typing fault in library 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    9b31891 View commit details
    Browse the repository at this point in the history

  19. cassandra: specify types more accurately 

    This was tricky because the code conflates the objects themselves and their
names. We also have classes named `TableMetadata` and `ColumnMetadata` which
shadow those in the Cassandra package

Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    e1e3ef1 View commit details
    Browse the repository at this point in the history

  20. typing: fix remaining easy type errors (12 remain) 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    9d440f0 View commit details
    Browse the repository at this point in the history

  21. bigquery: fix types and move method to subclass 

    `_iterate_over_tables` was attached to the super class but reached into
methods that only the subclass had. Move that over, and add best-guess
typings to the rest.

This class is really a mess type wise, there's a ton of dict-typing going on,
should clean it up later.

Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    0931b5c View commit details
    Browse the repository at this point in the history

  22. bigquery: fix typing errors in BigQueryTableUsageExtractor 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    3734d86 View commit details
    Browse the repository at this point in the history

  23. bigquery: move _iterate_over_tables back to base class 

    Am going to unwind this later

Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    d8b1545 View commit details
    Browse the repository at this point in the history

  24. fix a few loose ends and syntax errors 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    0120539 View commit details
    Browse the repository at this point in the history

  25. tests: fix HttpFailureSkipOnStatus semantics 

    I fixed the original incorrectly, reading the `or` as an `and`, which
thankfully caused a test failure.

However, I believe the original code which tested `isinstance(exception, HTTPError)`
to be incorrect -- we don't need that guard if we're just grabbing the value
regardless.

Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    bc74aa0 View commit details
    Browse the repository at this point in the history

  26. chore: fix lint errors introduced during typing 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    2fb6e14 View commit details
    Browse the repository at this point in the history

  27. lint: remove extraneous comma in list literals 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    2766e08 View commit details
    Browse the repository at this point in the history

  28. tests: fix typing in TestHiveTableLastUpdatedExtractor util 

    Signed-off-by: Dorian Johnson <[email protected]>
    @dorianj
    dorianj committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    81d93f4 View commit details
    Browse the repository at this point in the history