Dynamic configuration

[ingenieroariel]

GeoNode has a use case for Tegola where configuration should be passed at runtime.

It would be great to be able to use something other than files on a disk to support this.

[gdey]

@ARolek @jj0hns0n What do you think about this? I think it makes sense, but we have to think about things like Security, and how do we persist the changes.

[ARolek]

Random idea, what if the --config-file flag supported an HTTP path? You could then store the config in one location, and fire a "reload" signal to the tegola server and it would fetch the config file.

[gdey]

We could tell the different from a regular file by looking to see that it starts with https?:// to determine if it's a url or not. Is that what you are thinking. Otherwise we assume it's a file? Or do you think we should force it to always be a url, and files start with file://?

[ARolek]

For ease of use we would assume it's a local file unless it has a protocol in front of it (i.e. http(s))

[ingenieroariel]

I tested this in my local machine:
curl file:///code/hhypermap/101.diff
and it worked fine. Curl already handles all kinds of urls (local filesystem or remote) nicely.

[ingenieroariel]

I am trying to think through how it would work with docker, this solution leads to something that may not make sense in that context (having to let my webapp know about tegola and tegola about my webapp).

# docker-compose.yml
version: '2'
services:
  postgres:
    image: postgres
    ports:
      - "5432:5432”

   django:
     image: django
     volumes:
       - .:/code
     command: python manage.py runserver 0.0.0.0:8000
     ports:
       - “8000:8000”
     links:
       - postgres
       - tegola
     environment:
       - TEGOLA_RELOAD_URL=http://tegola/reload

  tegola:
     image: terranodo/tegola
     command: -config-file=http://django/tegola/config.yml
     links:
       - postgres
       - django
     ports:
        - "9090:9090”

[george-silva]

Hello all! Just chiming in here.

I've built something like this for Tilestache, but it fetched from a http(s) endpoint. This is extremely useful and it's one of the features preventing me to switch to tegola.

I would actually change django-tilestache to support both servers.

[ARolek]

@george-silva tegola supports loading a config over http(s), just pass in a URL to the --config flag. We don't have runtime reloading in yet though. Do you need reloading the config for your use case?

[george-silva]

Yes. That's one of the main usecases. Because there are some apps of ours that allows users to define their own tables/layers (within some limitations, of course), but it would be great if it was dynamic, instead of needing to restart the whole service.

In django-tilestache I just added a timer that re fetches the config from the http(s) endpoint configured, for example.

On the other hand, I just saw this: https://github.com/go-spatial/tegola/tree/master/cmd/tegola_lambda which might work for my usecase.

One thing does not rule out the other because lambdas are a whole new department (and only works on AWS), but it's very cool to known that you already support it.

[ARolek]

@george-silva got it. In an ideal world would how would you like to issue the reload command?

And yep, Lambda support is recently released. I just heard GCP has native Go functions coming soon too.

[george-silva]

There are two possibilities: on demand, like an endpoint on tegola itself, like "reload" or a timed operation. I know 0 things about go, perhaps with some mentoring this is something you can help me contribute? Em qua, 29 de ago de 2018 18:23, Alexander Rolek <[email protected]> escreveu:

…

@george-silva <https://github.com/george-silva> got it. In an ideal world would how would you like to issue the reload command? And yep, Lambda support is recently released. I just heard GCP has native Go functions coming soon too. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#58 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA0rP2_JcXJieQKZ7T5SVx6mVHKFASGwks5uVwZugaJpZM4JbNSb> .

[ARolek]

@george-silva I was thinking more along the lines of sending a signal to a running tegola process. The endpoint idea is something I'm trying to avoid as I don't like the idea of having a reload endpoint publicly available. If we did want to support an endpoint we could potentially support a configurable secret that would be required to invoke the endpoint. I'm open for ideas on this.

Regarding implementing this feature, I'm happy to provide codebase direction if you want to work on it. It's going to take some thinking to figure out the best strategy. We have to consider remote loading (so possible network errors), config validation and then a swap of the map configs in the atlas system. The good thing is the design of atlas did consider this situation so it shouldn't be too much work to get it in. I would start by looking into the way the config is loaded & validated first and then understand how the subsystems are instantiated.

We could also look into the the UX for nginx's reload functionality as they provide something similar.

[george-silva]

But how can we inform, from the outside, that the config changed?

Since there is support for http(s), it would be simple to implement a timed operation for this reload, right? Could be an option to pass along to run the process.

The endpoint for sure would need a secret token for that. What we could do is: when asked to reload, the server tries to reload the new config (from file or from http(s)) and if it's invalid, we just return a bad response and keep the old config.

Both options seem feasible. A timed reload would be simpler to implement and basically configures the server to pull data from a URL. An endpoint could have similar behavior or could be more complex (like providing a new config).

Thoughts?

[digitaltopo]

I'm also interested if there's a way to make dynamic queries to the postgres database with tegola, and generate tiles in "realtime"

[ARolek]

@digitaltopo this thread is about dynamic configuration. Are you interested in changing the configuration (i.e. the queries) or just serving data directly from the database rather than a cache?

[digitaltopo]

@ARolek I am interested in both:

1. Serve data directly from the database
2. Queries (request tiles with a query... aka define a layer with a query then request tiles)

Apologies if this is the wrong thread to post in, I'm new to Tegola, and looking at the docs, it appeared that in order to setup specific layers (using SQL queries) you have to have them available in the configuration (https://tegola.io/documentation/configuration/#postgis-1). I am trying to understand if a query can be sent at runtime in order to dynamically generate a new set of tiles, by running an arbitrary SQL query. I don't know how the cache would fit into this scenario, but I'd love to hear an outline of how this works. Thank you!

[dBitech]

at this time no you can not do dynamic (ad hoc) queries via a uri request. when I have a need to something similar I fall back to a different tool, namely https://github.com/tobinbradley/dirt-simple-postgis-http-api

…

On Thu, Mar 14, 2019 at 3:08 PM Orestis Herodotou ***@***.***> wrote: @ARolek <https://github.com/ARolek> I am interested in both: 1. Serve data directly from the database 2. Queries (request tiles with a query... aka define a layer with a query then request tiles) Apologies if this is the wrong thread to post in, I'm new to Tegola, and looking at the docs, it appeared that in order to setup specific layers (using SQL queries) you have to have them available in the configuration ( https://tegola.io/documentation/configuration/#postgis-1). I am trying to understand if a query can be sent at runtime in order to dynamically generate a new set of tiles, by running an arbitrary SQL query. I don't know how the cache would fit into this scenario, but I'd love to hear an outline of how this works. Thank you! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#58 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ1FThvLJZWoGnbdFeA15HzkTqtnx-g8ks5vWsgLgaJpZM4JbNSb> .

[digitaltopo]

I understand that Tegola's server capabilities don't allow for queries via a uri request, but could I build a an http server in front of Tegola that generated new provider.layer configs and passed them to Tegola? It sounds like right now once Tegola is running, the configurations cannot be swapped out, correct?

[dBitech]

That is correct, as of today, the only way to dynamically change the configuration of tegola, is to change it's config on disk and restart the service.

…

On Thu, Mar 14, 2019 at 3:21 PM Orestis Herodotou ***@***.***> wrote: I understand that Tegola's server capabilities don't allow for queries via a uri request, but could I build a an http server in front of Tegola that generated new provider.layer configs and passed them to Tegola? It sounds like right now once Tegola is running, the configurations cannot be swapped out, correct? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#58 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ1FTqmHYbLePqtZXv9J3WfK3e2HkFxIks5vWssmgaJpZM4JbNSb> .

[digitaltopo]

@dBitech Thank you for clarifying. So next question, would it be appropriate to try to replicate the desired behavior by using Tegola as a serverless function, with AWS Lambda or Google Cloud Functions?

[dBitech]

I've never tried tegola in a "serverless" environment so I can not say, though I expect with the current startup overhead (stand up a pg connection pool, make connections for all the various maps, perform sanity checks on the underlying db structure/data types etc) that moving it to a serverless environment may not pose any benefit, and will come at a performance loss.

…

On Thu, Mar 14, 2019, 15:34 Orestis Herodotou, ***@***.***> wrote: @dBitech <https://github.com/dBitech> Thank you for clarifying. So next question, would it be appropriate to try to replicate the desired behavior by using Tegola as a serverless function, with AWS Lambda or Google Cloud Functions? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#58 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ1FTqKzIk7j3WnVa3lHoj0W6Pv0e6xHks5vWs46gaJpZM4JbNSb> .

[ARolek]

@digitaltopo there is currently a version of tegola for lambda. We have the prebuilt binaries on the release page and instructions in the tegola_lambda README. It's fairly easy to get going but there are some gotchas I tried to capture in the README. As @dBitech pointed out there is startup overhead using serverless, but depending on how you configure your infrastructure over time your caching layers (i.e. CDN) will be handling a lot of the traffic. Ultimately it depends on your requirements if this is a viable strategy or not.

Regarding dynamic configuration, I played with some config hot swap strategies awhile back and it's completely doable. There will be a moment where requests will be locked as the new config is parsed and tested prior to overwriting the previous config. This has just not been a priority to implement so far. Would you be interested in tackling it?

[stdmn]

Moved to #613