Update DB dockerfile to use the postgis image

[hiddewie]

For local development, the Dockerfiles are used to set up a development environment.

The database is created with the Dockerfile.db image, which is based on the image mdillon/postgis (https://hub.docker.com/r/mdillon/postgis/). This base image has not been updated for 2 years. Moreover, the Dockerhub link links to the official PostGIS image postgis/postgis (https://github.com/postgis/docker-postgis).

This PR updates the base image for the Dockerfile.db. The official image of PostgreSQL 10 with PostGIS 2.5 is used. The alpine version is used which results in a smaller image to download.

Using this base image will allow updating to more modern Postgres versions as well (see the full list in https://registry.hub.docker.com/r/postgis/postgis/tags). In the future, newer PostgreSQL and PostGIS versions can be used. The upgrade is out of scope of this PR.

Finally, the Docker COPY command is used instead of the ADD command. The ADD command can also untar files, or fetch remote URLs which introduces complexity.

Tested by running the database, importing a dataset and viewing it in the Kosmtik previewer.

[jeisenbe]

To review this I updated to the latest version of Docker Desktop 3.1.0 (on MacOS 10.14.6)

I checked out this branch after using docker image ls and docker image rm -f <kosmitik_image_id> to remove all the existing containers, and then used docker-compose build import and docker-compose build kosmtik to rebuild the containers, and finally docker-compose up import and docker-compose up kosmtik to start everything.

The import container has a problem with osm2pgsql:

Setting up table: planet_osm_point
node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: -nan%
Osm2pgsql failed due to ERROR: CREATE UNLOGGED TABLE planet_osm_point (osm_id int8,"access" text,"addr:housename" text,"addr:housenumber" text,"admin_level" text,"aerialway" text,"aeroway" text,"amenity" text,"barrier" text,"boundary" text,"building" text,"highway" text,"historic" text,"junction" text,"landuse" text,"layer" int4,"leisure" text,"lock" text,"man_made" text,"military" text,"name" text,"natural" text,"oneway" text,"place" text,"power" text,"railway" text,"ref" text,"religion" text,"shop" text,"tourism" text,"water" text,"waterway" text,"tags" hstore,way geometry(POINT,3857) ) WITH ( autovacuum_enabled = FALSE ) failed: ERROR:  could not access file "$libdir/postgis-2.5": No such file or directory
LINE 1: ...xt,"water" text,"waterway" text,"tags" hstore,way geometry(P...
                                                             ^

And also kosmtik gets this error:

kosmtik_1  |     at ProjectServer.raise (/usr/lib/node_modules/kosmtik/src/back/ProjectServer.js:261:13)
kosmtik_1  |     at /usr/lib/node_modules/kosmtik/src/back/ProjectServer.js:75:30
kosmtik_1  |     at /usr/lib/node_modules/kosmtik/node_modules/generic-pool/lib/generic-pool.js:283:11
kosmtik_1  |     at loaded (/usr/lib/node_modules/kosmtik/node_modules/mapnik-pool/index.js:23:37)
kosmtik_1  | Postgis Plugin: ERROR:  could not access file "$libdir/postgis-2.5": No such file or directory
kosmtik_1  | in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM planet_osm_polygon WHERE "way" IS NOT NULL LIMIT 1;'
kosmtik_1  |   encountered during parsing of layer 'landcover-low-zoom' in Layer

When you tested this, did you remove all the existing containers and rebuild them as above?

[jeisenbe]

I completely removed all docker containers, images and volumes, then tried it again. I got this error:

Attaching to openstreetmap-carto_import_1
import_1   | Waiting for PostgreSQL to be running
import_1   | Timeout while waiting for PostgreSQL to be running
import_1   | psql: could not translate host name "db" to address: Name or service not known
import_1   | createdb: could not connect to database template1: could not translate host name "db" to address: Name or service not known
import_1   | osm2pgsql version 1.2.2 (64 bit id space)
import_1   | 
import_1   | Allocating memory for dense node cache
import_1   | Allocating dense node cache in one big chunk
import_1   | Allocating memory for sparse node cache
import_1   | Sharing dense sparse
import_1   | Node-cache: cache=4096MB, maxblocks=65536*65536, allocation method=11
import_1   | Mid: pgsql, cache=4096
import_1   | Connection to database failed: could not translate host name "db" to address: Name or service not known
import_1   | 
import_1   | Error occurred, cleaning up
import_1   | Traceback (most recent call last):
import_1   |   File "scripts/get-external-data.py", line 280, in <module>
import_1   |     main()
import_1   |   File "scripts/get-external-data.py", line 183, in main
import_1   |     password=password) as conn:
import_1   |   File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 130, in connect
import_1   |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
import_1   | psycopg2.OperationalError: could not translate host name "db" to address: Name or service not known
import_1   | 
openstreetmap-carto_import_1 exited with code 1

PostgreSQL seems not to have run properly and the database did not connect.

[hiddewie]

I completely removed all docker containers, images and volumes, then tried it again. I got this error:

Strange, that is the same way I tried. I will try again tomorrow.

[hiddewie]

I updated the PR. Apparently I did not clean up my Docker environment as much as I thought. There were indeed some missing PostGIS configuration changes.

[jeisenbe]

@hiddewie what is the reason for this PR at the current time?

You said: "Using this base image will allow updating to more modern Postgres versions as well (see the full list in https://registry.hub.docker.com/r/postgis/postgis/tags). In the future, newer PostgreSQL and PostGIS versions can be used. The upgrade is out of scope of this PR.”

But we are not upgrading the PostgreSQL or PostGIS version right now. So is it necessary to change the docker image for this release? Are there some benefits?

[jeisenbe]

@hiddewie I could review this PR again, but it takes about an hour to make the changes to my docker setup each time, so I would like to understand the benefit we will gain from this PR before working on it further.

[hiddewie]

@jeisenbe Thanks for your comments.

I would argue that upgrading a Docker image should be easy if there will ever be an infrastructure change in the deployed version of this style. The (Docker) development environment should be as close as possible to the production environment.

And when the infrastructure versions are upgraded, you do not want that change blocked by having to wait for an up-to-date image.

As mentionend in the PR description, the currently used PostGIS image https://hub.docker.com/r/mdillon/postgis has not had updates for two years. You can choose to accept this fact, and let the development environment 'fall behind' on updates. If you choose that, feel free to close this PR.

After looking around in Github it seems the person responsible for the mdillon/postgis is the same person currently maintaining the official PostGIS Docker images in https://github.com/postgis/docker-postgis. There are many more versions (flavours) of the postgis/postgis image available than of the mdillon/postgis.

An additional benefit of the official PostGIS Docker image is that the Alpine version is much smaller to download.

[pnorman]

I agree with moving to the official postgis image, but don't use docker so can't review this PR

[pnorman]

I tested this, but am concerned that we don't have enough people using this code who are reviewing PRs.

[hiddewie]

Thank you! Do not hesitate to contact/tag me in new issues that arise for the Docker images. I trill try to help if I am able to.