Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use flex backend #4431

Closed
wants to merge 67 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
d7be154
Switch to the new osm2pgsql flex backend
pnorman Apr 27, 2020
1e38b85
Refactor creation of tables to reduce duplication
pnorman Apr 27, 2020
717d549
Remove unused lua function
pnorman Apr 27, 2020
c0f759b
Create a route table
pnorman Apr 28, 2020
6a6e69d
Skip Travis osm2pgsql tests
pnorman Apr 29, 2020
edbfe32
Merge branch 'master' v5.2.0 into flex/master
jeisenbe May 8, 2020
e6045ed
Merge tag 'v5.3.1' into flex/master
pnorman Feb 5, 2021
924e71f
Update README with osm2pgsql version requirements
pnorman Feb 7, 2021
4448c69
Switch from multi to new split_at in transforms
pnorman Feb 7, 2021
8c06ed9
Correctly split tags for route_member table
pnorman Feb 7, 2021
9df706e
Add tests for layer function
pnorman Feb 7, 2021
25421a4
Add an admin table
pnorman Jun 26, 2021
05005ee
Compare admin levels as numbers
pnorman Jun 26, 2021
e491c0a
Add a transport_line table
pnorman Jun 28, 2021
2e47c05
Add a transport_polygon table
pnorman Jun 28, 2021
c5c021b
Don't repeat layer tag in columns and hstore
pnorman Jun 29, 2021
d378bf6
Use new planet_osm_admin table
pnorman Jun 29, 2021
fe8c4b6
Use the new admin table for admin boundaries.
pnorman Jun 29, 2021
0f9d5af
Correctly use the transport cols to figure out hstore
pnorman Jul 1, 2021
b82ba63
Don't add route relations to line and road tables
pnorman Jul 22, 2021
ab745c3
Fix failing tests from previous two commits
pnorman Jul 22, 2021
5b4f81b
Switch default database name
pnorman Aug 29, 2021
7a8871e
Also switch DB name for loading external data
pnorman Aug 29, 2021
7e7ac1b
Correctly add ways back in stage 2 flex processing
pnorman Dec 30, 2021
834e5f5
Merge tag 'v5.4.0' into flex/master
pnorman Dec 30, 2021
5adf64f
Use ST_LineMerge on admin boundaries
pnorman Dec 30, 2021
0068392
Add a z_order for busways
pnorman Dec 30, 2021
e7af9bd
Use correct caps on bbox token
pnorman Dec 31, 2021
9741ef0
Remove unused .style file
pnorman Jul 4, 2022
b2b253c
Refactor to take objects instead of object tags in remainder of add_*
pnorman Jul 4, 2022
25a5a37
Move all layer handling to split_tags
pnorman Jul 4, 2022
1744b84
Avoid hard-coding construction z_order
pnorman Jul 4, 2022
26f234c
Code style, comment fixes
pnorman Jul 9, 2022
efb05f6
Don't handle boundary MPs
pnorman Jul 9, 2022
29447e3
Merge tag 'v5.5.0' into flex/master
pnorman Jul 10, 2022
fc0d624
Suggested change to code comment
mboeringa Jul 10, 2022
dba7c3d
Suggested change to code comment (2)
mboeringa Jul 10, 2022
b22ce24
Merge pull request #4594 from mboeringa/patch-2
pnorman Jul 10, 2022
684797e
Fix missing parameter definitions in code comments, fix case of boolean
mboeringa Jul 10, 2022
fa2f014
Fix to comment based on feedback
mboeringa Jul 10, 2022
bba21ac
Merge pull request #4597 from mboeringa/patch-5
pnorman Jul 10, 2022
41545d8
Remove missed merge conflict
pnorman Jul 11, 2022
5680832
Specify minimum PostGIS version
pnorman Jul 11, 2022
619c4f1
Merge pull request #4595 from mboeringa/patch-3
pnorman Jul 11, 2022
3283fd8
Document flex callbacks and supporting functions
pnorman Jul 11, 2022
912c7ae
Reformat layer and admin_level checks
pnorman Jul 11, 2022
543e6a9
Merge tag 'v5.5.1' into flex/master
pnorman Jul 13, 2022
9c4c9c2
Treat unknown waterway tags as linestrings
pnorman Jul 14, 2022
40683e1
Refactor phase 2 data structures
pnorman Jul 16, 2022
de95dee
Correct geometry column name
pnorman Jul 16, 2022
f8d3098
Merge tag 'v5.6.0' into flex/master
pnorman Aug 3, 2022
7c75f34
Merge tag 'v5.6.1' into flex/master
pnorman Aug 12, 2022
f8117a3
Fix for #3504
mboeringa Jan 9, 2023
aa9b75a
Adding test data for removal of "virtual" borders on the antimeridian
mboeringa Jan 10, 2023
598116f
Switch to flex output in CI
pnorman Jan 19, 2023
49c5624
Merge tag 'v5.7.0' into flex/master
pnorman Jan 19, 2023
8623e4b
Rework layer/admin conversion expressions to be clearer
pnorman Feb 25, 2023
4d5abbd
Fix lua syntax error
pnorman Feb 25, 2023
abc204e
Merge branch '4766' into flex/master
pnorman Feb 25, 2023
ea8cfe0
Fix merge errors in docs
pnorman Feb 25, 2023
816e006
Use postgres DB for system commands
pnorman Feb 25, 2023
800996e
Fix merge errors in ci
pnorman Feb 25, 2023
72879d5
Fix index script when reindexing
pnorman Feb 25, 2023
14f668d
Always return something on layer and admin conversion
pnorman Feb 25, 2023
ba887bd
More fixes of database name in docs
pnorman Feb 25, 2023
cc263e6
Potential fix for the issue raised in https://github.com/gravitystorm…
mboeringa Jul 28, 2023
c7066c8
Merge pull request #4846 from mboeringa/patch-7
pnorman Aug 25, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,12 +59,12 @@ jobs:
- name: Wait for database
run : sudo pg_ctlcluster 14 main start; until pg_isready; do sleep 0.5; done
- name: Setup database
run: sudo -i -u postgres createuser -s $USER && createdb -E utf8 gis && psql -Xq -d gis -c "CREATE EXTENSION postgis; CREATE EXTENSION hstore;"
run: sudo -i -u postgres createuser -s $USER && createdb -E utf8 flex && psql -Xq -d flex -c "CREATE EXTENSION postgis; CREATE EXTENSION hstore;"
- name: Import empty file
run: |
osm2pgsql -G --hstore --style openstreetmap-carto.style --tag-transform-script openstreetmap-carto.lua -d gis -r xml <(echo '<osm version="0.6"/>')
osm2pgsql --output flex --style openstreetmap-carto.lua -d flex -r xml <(echo '<osm version="0.6"/>')
- name: Create indexes
run: psql -1Xq -v ON_ERROR_STOP=1 -d gis -f indexes.sql
run: psql -1Xq -v ON_ERROR_STOP=1 -d flex -f indexes.sql
- name: Load empty shapefiles
run: scripts/get-external-data.py --no-update --cache -D scripts/empty_files
- name: Test queries are valid
Expand Down
7 changes: 7 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,13 @@ Because SQL within JSON or YAML will not generally be syntax highlighted, indent
* Hstore queries tested for NULL should be enclosed in parentheses, e.g. `(tags->'foo') IS NULL`.
* To check if a tag is in the tags hstore, use `tags @> 'foo=>bar'`, relying on automatic conversion from `text` to `hstore`.

## Lua style guidelines

* Four space indents, no tabs
* No external dependencies
* Use `a["name"]` where name is a tag key, and `a.name` otherwise.


## Map icon guidelines

* All new icons must be SVG format only. The SVG must be saved as standards compliant SVG without any proprietary tags. In Inkscape software, you will need to "Save As..." and choose the format Optimized SVG (preferable) or Plain SVG.
Expand Down
2 changes: 1 addition & 1 deletion DOCKER.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,5 +99,5 @@ Docker stores its disk image by default in the home directories of the user. If
When working with the style's database tables after an import, it can be helpful to log in at the [console](https://www.postgresql.org/docs/current/app-psql.html) to inspect the table structure or view imported data. The following command will open a psql console on the database:

```
docker-compose exec -e PGUSER=postgres -e PGDATABASE=gis db psql
docker-compose exec -e PGUSER=postgres -e PGDATABASE=flex db psql
```
20 changes: 10 additions & 10 deletions INSTALL.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,25 @@
This document describes how to manually configure your system for running OpenStreetMap Carto. If you prefer quick, platform independent setup for a development environment, without the need to install and configure tools by hand, follow a Docker installation guide in [DOCKER.md](DOCKER.md).

## OpenStreetMap data
You need OpenStreetMap data loaded into a PostGIS database (see below for [dependencies](#dependencies)). These stylesheets expect a database generated with osm2pgsql using the pgsql backend (table names of `planet_osm_point`, etc), the default database name (`gis`), and the [lua transforms](https://osm2pgsql.org/doc/manual.html#lua-tag-transformations) documented in the instructions below.
You need OpenStreetMap data loaded into a PostGIS database (see below for [dependencies](#dependencies)). These stylesheets expect a database generated with osm2pgsql using the flex backend with the supplied Lua scripts. This requires osm2pgsql 1.4.1 or later.

Start by creating a database

```sh
sudo -u postgres createuser -s $USER
createdb gis
createdb flex
```

Enable PostGIS and hstore extensions with

```sh
psql -d gis -c 'CREATE EXTENSION postgis; CREATE EXTENSION hstore;'
psql -d flex -c 'CREATE EXTENSION postgis; CREATE EXTENSION hstore;'
```

then grab some OSM data. It's probably easiest to grab an PBF of OSM data from [Geofabrik](https://download.geofabrik.de/). Once you've done that, import with osm2pgsql:

```sh
osm2pgsql -G --hstore --style openstreetmap-carto.style --tag-transform-script openstreetmap-carto.lua -d gis ~/path/to/data.osm.pbf
osm2pgsql --output flex --style openstreetmap-carto.lua -d flex ~/path/to/data.osm.pbf
```

You can find a more detailed guide to setting up a database and loading data with osm2pgsql at [switch2osm.org](https://switch2osm.org/serving-tiles/manually-building-a-tile-server-16-04-2-lts/).
Expand All @@ -32,19 +32,19 @@ We do not recommend [PostgreSQL JIT](https://www.postgresql.org/docs/current/jit

Disabling JIT is **essential** for use with Kosmtik and other style development tools.

JIT can be disabled with `psql -d gis -c 'ALTER SYSTEM SET jit=off;' -c 'SELECT pg_reload_conf();'` or any other means of adjusting the PostgreSQL config.
JIT can be disabled with `psql -d postgres -c 'ALTER SYSTEM SET jit=off;' -c 'SELECT pg_reload_conf();'` or any other means of adjusting the PostgreSQL config.

### Custom indexes
Custom indexes are required for rendering performance and are essential on full planet databases. These are generated by the `scripts/indexes.py` script, see `scripts/indexes.py --help` for various advanced options, but the command below will work to create the indexes on a new installation:

```sh
psql -d gis -f indexes.sql
psql -d flex -f indexes.sql
```

The indexes can be created in parallel with

```sh
scripts/indexes.py -0 | xargs -0 -P0 -I{} psql -d gis -c "{}"
scripts/indexes.py -0 | xargs -0 -P0 -I{} psql -d flex -c "{}"
```

## Scripted download
Expand Down Expand Up @@ -81,9 +81,9 @@ For development, a style design studio is needed.
To display any map a database containing OpenStreetMap data and some utilities are required

* [PostgreSQL](https://www.postgresql.org/)
* [PostGIS](https://postgis.net/)
* [osm2pgsql](https://github.com/openstreetmap/osm2pgsql#installing) to [import your data](https://switch2osm.org/serving-tiles/updating-as-people-edit/) into a PostGIS database
* Python 3 with the psycopg2, yaml, and requests libraries (`python3-psycopg2`, `python3-yaml`, `python3-requests` packages on Debian-derived systems)
* [PostGIS](https://postgis.net/) 3.1.0 or later with GEOS 3.9.0 or later
* [osm2pgsql](https://github.com/openstreetmap/osm2pgsql#installing) 1.4.1 or later to import your data into a PostGIS database
* Python 3 with the psycopg2, yaml, and requests libraries (`python3-psycopg2` `python3-yaml` `python3-requests` packages on Debian-derived systems)
* `ogr2ogr` for loading shapefiles into the database (`gdal-bin` on Debian-derived systems)

### Optional development dependencies
Expand Down
2 changes: 1 addition & 1 deletion external-data.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ settings:
temp_schema: loading
schema: public
data_dir: data
database: gis
database: flex
metadata_table: external_data
sources:
simplified_water_polygons:
Expand Down
1 change: 1 addition & 0 deletions indexes.sql
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,4 @@ CREATE INDEX planet_osm_polygon_way_area_z6 ON planet_osm_polygon USING GIST (wa
CREATE INDEX planet_osm_roads_admin ON planet_osm_roads USING GIST (way) WHERE boundary = 'administrative';
CREATE INDEX planet_osm_roads_admin_low ON planet_osm_roads USING GIST (way) WHERE boundary = 'administrative' AND admin_level IN ('0', '1', '2', '3', '4');
CREATE INDEX planet_osm_roads_roads_ref ON planet_osm_roads USING GIST (way) WHERE highway IS NOT NULL AND ref IS NOT NULL;
CREATE INDEX planet_osm_route_member_id ON planet_osm_route USING btree (member_id);
5 changes: 5 additions & 0 deletions indexes.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,8 @@ roads:
where: boundary = 'administrative'
roads_ref:
where: highway IS NOT NULL AND ref IS NOT NULL
route:
# The route table has no geospatial data, so it's index is a btree
member_id:
function: member_id
type: btree
Loading