Add ability to reload config at runtime

[ingenieroariel]

This is required to support changing a configuration file during development.

If possible, Tegola should finish serving existing requests with the old config and serve new requests with the new config.

[ARolek]

@gdey and I discussed this and we think the best way to accomplish this is to support an an endpoint (i.e. /reload) which would require a "reload token" in the query string (i.e. /reload?token=....). The "reload token" would be set in the config file and if the provided token matched the config file token, the config would reload.

Thoughts?

[ingenieroariel]

nice!

[JivanAmara]

Why a reload endpoint instead of reloading when the config file has changed?

[ARolek]

@JivanAmara we allow for remote configuration files (i.e. fetch a file on S3 or some other HTTP endpoint). This allows for easier management of a centralized config file across a cluster of servers instead of requiring the config to be reloaded on all servers. I need to document this.

[stdmn]

@ARolek Can you provide a brief example of how to do that? I've tried using the TEGOLA_CONFIG for AWS Lambda (pulling the config.toml file from a S3 Cloudfront distribution) and it doesn't seem to be working. Not sure if I'm overthinking it.

[ARolek]

@stdmn do you have any logs you can provide? If you set TEGOLA_CONFIG to a URL, tegola will attempt a remote fetch. A log is written when this routine starts as shown here.

[stdmn]

Looks like it may be a CORS issue based on this line: Client.Timeout exceeded while awaiting headers. Not sure exactly, though.

- 2019-06-17 14:24:03 [INFO] config.go:183: loading remote config (**PUBLIC CLOUDFRONT URL**)
- START RequestId: 10de3e0b-3171-43cb-9c7b-6c1233cf9cfd Version: $LATEST
- 14:24:13 2019-06-17 14:24:13 [INFO] config.go:183: loading remote config (**PUBLIC CLOUDFRONT URL**)
- 14:24:23 2019/06/17 14:24:23 main.go:40: error fetching remote config file (**PUBLIC CLOUDFRONT URL**): Get **PUBLIC CLOUDFRONT URL**: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
- 2019/06/17 14:24:23 main.go:40: error fetching remote config file (**PUBLIC CLOUDFRONT URL**): Get **PUBLIC CLOUDFRONT URL**: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 
- 2019/06/17 14:24:23 exit status 1
- END RequestId: 10de3e0b-3171-43cb-9c7b-6c1233cf9cfd
- REPORT RequestId: 10de3e0b-3171-43cb-9c7b-6c1233cf9cfd	Duration: 10335.33 ms	Billed Duration: 10400 ms Memory Size: 704 MB	Max Memory Used: 37 MB
- RequestId: 10de3e0b-3171-43cb-9c7b-6c1233cf9cfd Process exited before completing request
- 2019-06-17 14:24:24 [INFO] config.go:183: loading remote config (**PUBLIC CLOUDFRONT URL**)

[ARolek]

Is your lambda function running inside of a VPC? From the looks of it, the request is timing out, which happens after 10 seconds: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

If your function is in a VPC then you will need to configure the VPC to allow the lambda function to make public requests to the internet. Relevant article.

[stdmn]

Doh! You're right. Was able to connect directly to the S3 instance by allowing the VPC to connect to my S3 endpoint. Sorry for my ignorance and thanks again for your patience!

[ARolek]

@stdmn beautiful! I'm glad it's working for you! If you have any suggestions to help make the instructions easier to follow, feel free to send in a PR with any documentation updates. This is an interesting "gotcha" that is not currently documented.

[stdmn]

@ARolek One related question, it seems that when I change the S3 config object but don't update the Lambda function, Lambda doesn't recognize that the config file has been changed and retains some kind of cache on the previous config file. As far as I can tell, this is server side, not client side (tested with clearing cache and using incognito). When I remove the TEGOLA_CONFIG save and then re-add with the same config location it will update.

Any idea how to invalidate in order to make sure that the Lambda function is pulling the new S3 config?

Sorry, I recognize this stuff is somewhat out of your jurisdiction, but my goal is to hot-swap the config file when a user uploads a new table to a DB to add the new uploaded layer.

[stdmn]

UPDATE: seems that it's on the API Gateway (unsurprisingly): https://stackoverflow.com/questions/39303410/can-aws-api-gateway-cache-invalidate-specific-entries-based-on-the-response-cont.

Still working on a solution.

[ARolek]

@stdmn you could also look into not using API Gateway now that Application Load Balancers can Invoke Lambda Functions.