File caching for the oauth2client >= 4.0.0

[xmedeko]

@tmatsuo proposed in #110 to cache discovery.build(), however, #294 has stopped using cache outside of the GAE (appengine) environment (see also #299). E.g. the discovery.build() is not cached in the CGE environment. Is it possible to bring the file caching back, or provide some example of file cache (subclass of googleapiclient.discovery_cache.Cache)?

[theacodes]

Sure, mostly what needs to happen is that the current file cache needs to be updated to not use oauth2client's locked file.

…

On Sun, Jan 8, 2017, 2:14 PM Ondrej Medek ***@***.***> wrote: @tmatsuo <https://github.com/tmatsuo> proposed in #110 <#110> to cache discovery.build(), however, #294 <#294> has stopped using cache outside of the GAE (appengine) environment (see also #299 <#299>). E.g. the discovery.build() is not cached in the CGE environment. Is it possible to bring the file caching back, or provide some example of file cache (subclass of googleapiclient.discovery_cache.Cache)? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#325>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc2WtGDCjrz4H1QhftQxypDuOidnpks5rQV-6gaJpZM4Ld0fj> .

[Schweigi]

Is there a known workaround in the meantime?

[Schweigi]

I created a small workaround by using a in memory cache (see example below). Does anyone know if there should be a regular cache invalidation? Currently the memory cache is cleared on each deployment, I hope that's ok for the time being.

In memory cache:

class MemoryCache(Cache):
    _CACHE = {}

    def get(self, url):
        return MemoryCache._CACHE.get(url)

    def set(self, url, content):
        MemoryCache._CACHE[url] = content

Usage:

http = ...
service = discovery.build('calendar', 'v3', http=http, cache=MemoryCache())

[xmedeko]

AFAIK the Google production API should not change (the same version of API), so the you do not need to invalidate the cache. However, the discovery_cache/__init__.py defines the limit DISCOVERY_DOC_MAX_AGE as one day.

+1 for a neat, simple MemoryCache. IMO it's good when you have a single, long running process.

If you want to share the cache among processes, (like running scripts by the cron,) you may copy the old good oauth2client locked_file.py and then to plug it somehow with discovery_cache/file_cache.py. Beware, the locked_file.py is considered not 100% save, so you take the risk. (But works well in practice.) By "plugin somehow" I mean either copy & change the discovery_cache/file_cache.py to take your locked_file.py or to trick the Python to think your locked_file.py is from the oauth2client.locked_file package.

[xmedeko]

@jonparrott what about to change file lock to the multiprocessing.RLock() in the original locked_file.py? Would be it sufficient solution to accept back to the repo?

[theacodes]

I think it's likely possible to do this without any sort of locking.

…

On Wed, May 17, 2017, 12:02 AM Ondrej Medek ***@***.***> wrote: @jonparrott <https://github.com/jonparrott> what about to change file lock to the multiprocessing.RLock() in the original locked_file.py? Would be it sufficient solution to accept back to the repo? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#325 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc-ne-ZS8p4rfQbV1qs3RtjXgc7Opks5r6puOgaJpZM4Ld0fj> .

[xmedeko]

You mean a file cache, shared among multiple processes, without any locking? How to prevent data corruption, when 2 process write to the same file?

[theacodes]

Likely using a folder containing a file for each cached discovery doc. No locking needed. Worst case scenario two processes write the same file twice.

…

On Wed, May 17, 2017, 12:08 AM Ondrej Medek ***@***.***> wrote: You mean a file cache, shared among multiple processes, without any locking? How to prevent data corruption, when 2 process write to the same file? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#325 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc9PWEHEBR_dWKW4KmNhmgV4lH8qiks5r6pzbgaJpZM4Ld0fj> .

[xmedeko]

FYI to use the locked_file from oauth2client 3.0.0 without that package (it's not required since 1.7.3), just use the gist https://gist.github.com/xmedeko/e5293707e075d18c9dde8a574d2670e5 and call the shim() once.

[Chronial]

Based on Schweigi's solution, I wrote a simple file based cache (unix-only) to also cache across processes.

import os.path
import hashlib
import tempfile

class DiscoveryCache:
    def filename(self, url):
        return os.path.join(
            tempfile.gettempdir(),
            'google_api_discovery_' + hashlib.md5(url.encode()).hexdigest())

    def get(self, url):
        try:
            with open(self.filename(url), 'rb') as f:
                return f.read().decode()
        except FileNotFoundError:
            return None

    def set(self, url, content):
        with tempfile.NamedTemporaryFile(delete=False) as f:
            f.write(content.encode())
            f.flush()
            os.fsync(f)
        os.rename(f.name, self.filename(url))

[busunkim96]

I'm opting to close this as it's unlikely we will implement it in the future.

These client libraries are officially supported by Google. However, the libraries are considered complete and are in maintenance mode. This means that we will address critical bugs and security issues but will not add any new features.

[aawilson]

I don't know if this changed at some point, but I had to import Cache from googleapiclient.discovery_cache.base (presumably googleapiclient.discovery_cache.file_cache also works fine, but inheriting from the Base class is almost certainly the better idea)

[Kache]

An equivalent & simpler workaround is to set cache_discovery=False:

discovery.build(api, version, http=http, cache_discovery=False)

Since according to the source the "cache" is never reused in the "single, long running process" case.

[neomafo88]

I created a small workaround by using a in memory cache (see example below). Does anyone know if there should be a regular cache invalidation? Currently the memory cache is cleared on each deployment, I hope that's ok for the time being.

In memory cache:

class MemoryCache(Cache):
    _CACHE = {}

    def get(self, url):
        return MemoryCache._CACHE.get(url)

    def set(self, url, content):
        MemoryCache._CACHE[url] = content

Usage:

http = ...
service = discovery.build('calendar', 'v3', http=http, cache=MemoryCache())

googleapiclient.discovery_cache.base.Cache