Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Replace default file_cache usage with simple memory cache #4184

Merged
merged 7 commits into from
Jul 1, 2020

Conversation

aawilson
Copy link
Contributor

@aawilson aawilson commented Jun 26, 2020

The original main.py creates noisy log messages when the function is run file_cache is unavailable when using oauth2client >= 4.0.0:

2020-06-26T19:34:03.459Z datastore_export file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export Traceback (most recent call last): E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export   File "/env/local/lib/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in <module> E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export     from oauth2client.contrib.locked_file import LockedFile E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export ModuleNotFoundError: No module named 'oauth2client' E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export  E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export During handling of the above exception, another exception occurred: E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export  E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export Traceback (most recent call last): E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export   File "/env/local/lib/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in <module> E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export     from oauth2client.locked_file import LockedFile E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export ModuleNotFoundError: No module named 'oauth2client' E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export  E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export During handling of the above exception, another exception occurred: E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export  E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export Traceback (most recent call last): E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export   File "/env/local/lib/python3.7/site-packages/googleapiclient/discovery_cache/__init__.py", line 44, in autodetect E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export     from . import file_cache E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export   File "/env/local/lib/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in <module> E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export     "file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth" E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export ImportError: file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth E  datastore_export 

While they don't interfere with the function's operation, they are disconcerting and not necessary. This implements the workaround suggested in googleapis/google-api-python-client#325 (which creates an in-memory cache object to use in replacement).

Description

Fixes #4185

Checklist

The original main.py creates noisy log messages when the function is run file_cache is unavailable when using oauth2client >= 4.0.0:

```
2020-06-26T19:34:03.459Z datastore_export file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export Traceback (most recent call last): E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export   File "/env/local/lib/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in <module> E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export     from oauth2client.contrib.locked_file import LockedFile E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export ModuleNotFoundError: No module named 'oauth2client' E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export  E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export During handling of the above exception, another exception occurred: E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export  E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export Traceback (most recent call last): E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export   File "/env/local/lib/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in <module> E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export     from oauth2client.locked_file import LockedFile E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export ModuleNotFoundError: No module named 'oauth2client' E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export  E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export During handling of the above exception, another exception occurred: E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export  E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export Traceback (most recent call last): E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export   File "/env/local/lib/python3.7/site-packages/googleapiclient/discovery_cache/__init__.py", line 44, in autodetect E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export     from . import file_cache E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export   File "/env/local/lib/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in <module> E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export     "file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth" E  datastore_export 
2020-06-26T19:34:03.459Z datastore_export ImportError: file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth E  datastore_export 
```

While they don't interfere with the function's operation, they are disconcerting and not necessary. This implements the workaround suggested in googleapis/google-api-python-client#325 (which creates an in-memory cache object to use in replacement).
@aawilson aawilson requested a review from a team as a code owner June 26, 2020 20:01
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jun 26, 2020
@leahecole
Copy link
Collaborator

PTAL @BenWhitehead or @crwilcox to see how you feel about it from a Datastore point of view

@crwilcox crwilcox added the kokoro:run Add this label to force Kokoro to re-run the tests. label Jun 29, 2020
@kokoro-team kokoro-team removed the kokoro:run Add this label to force Kokoro to re-run the tests. label Jun 29, 2020
Add a comment explaining the inclusion of MemoryCache instance in the `build` invocation.
@aawilson aawilson requested a review from crwilcox June 29, 2020 18:21
@leahecole leahecole added kokoro:force-run Add this label to force Kokoro to re-run the tests. automerge Merge the pull request once unit tests and other checks pass. labels Jun 30, 2020
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 30, 2020
@leahecole
Copy link
Collaborator

@aawilson it looks like this is failing lint - there needs to be one more blank line in main.py line 20. You can also test this locally with nox -s lint (see Test Enviroment Setup)

@gcf-merge-on-green
Copy link
Contributor

Your PR has attempted to merge for 3 hours. Please check that all required checks have passed, you have an automerge label, and that all your reviewers have approved the PR

@gcf-merge-on-green
Copy link
Contributor

Merge-on-green attempted to merge your PR for 6 hours, but it was not mergeable because either one of your required status checks failed, or one of your required reviews was not approved. Learn more about your required status checks here: https://help.github.com/en/github/administering-a-repository/enabling-required-status-checks. You can remove and reapply the label to re-run the bot.

@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Jun 30, 2020
@aawilson
Copy link
Contributor Author

Yup got it, my local linter would have caught it but I've been editing this entirely in GitHub's interface, will add

aawilson and others added 2 commits June 30, 2020 18:04
PEP8-mandated two newlines between top-level elements
@leahecole leahecole added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 1, 2020
@leahecole
Copy link
Collaborator

@aawilson I've been there! Thanks for the edits 😄

@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 1, 2020
@leahecole leahecole added the automerge Merge the pull request once unit tests and other checks pass. label Jul 1, 2020
@leahecole leahecole merged commit 22788da into GoogleCloudPlatform:master Jul 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automerge Merge the pull request once unit tests and other checks pass. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ImportError shows in logs when implementing datastore export sample
5 participants