Host vector tiles generated by tilemaker #565
Comments
|
I know you indicate this to be an experiment but still - may i ask what the strategic vision is behind this or in other words: Where do you intend this to go if the experiment should be successful (by whatever definition of successful)? I mean is it supposed to lead to the OSMF providing a vector tile service for other parties (and if yes: how does it intend to position that relative to commercial services in that domain)? Or is it supposed to be a non-monetary support program/grant for tilemaker development (and if yes: on what basis was it decided to choose tilemaker to receive that support and not other software in the domain of map rendering/data preprocessing for map rendering)? As you know i have been saying for quite some time that the OSMF needs to strategically invest in development of technologies suitable for scalable high quality cartography based on OSM data and usable for cooperative community projects so in principle i would see favorably any steps in that direction. But i would like to understand how this step fits into such a strategy or if not what other strategic goals it is meant to serve. |
|
The immediate goal is to support the development of vector tiles for openstreetmap.org. |
|
Trying to understand what you mean by that - so this is a feasibility study to see if tilemaker can be practically managed operationally to generate world wide vector tiles (with no requirements what that entails specified so far, presumably within the current capabilities of tilemaker) with a weekly update cycle and then recruit some volunteer style developers to develop a map style (or several - presumably using Maputnik) to produce maps targeting yet to be specified needs on openstreetmap.org. I see a number of issues with that (in particular in the long term regarding the strategic needs for community map styles providing meaningful constructive feedback for mappers). But this is evidently not a very suitable venue to discuss such matters (although i would very much suggest you to have such a discussion with a broader reach - not many people follow this tracker and few would be inclined to comment here). Independent of the specifics discussed here i would suggest to clearly distinguish between operational work (stuff that serves concrete and well specified immediate needs) and strategic endeavors and programs. When you state the goal to support the development of vector tiles for openstreetmap.org it is not clear if that is an operational goal (in which case it seems a bit underspecified - see above) or strategic endeavor (in which case a more long term vision what strategic goals this should serve would be highly advisable IMO). |
|
What I said on the operations call yesterday (and this is my personal view) is that I hoped that providing them would break the impasse that seems to exist where people can't work on a style because there's no tiles but don't work on tiles because there's no style. My hope would be that people will start developing a vector style we can use and that there can then be a feedback cycle between that and any necessary changes to the tile schema until we have something we feel is deployable. At that point we can worry about what stack to use in production and how best to generate and serve vector tiles for that. |
|
Understood. Just keep in mind that when the OSMF rolls out something like that there will be a message of endorsement (of the tools used, their capabilities and their limitations and the standards they are built upon) and a message of strategic direction (in what direction community cartography in the OSM community should develop) perceived even if no such message is intended. |
This comment has been minimized.
This comment has been minimized.
|
Please can we keep this ticket on topic and not go off on crazy tangents. This ticket is just about getting an experimental vector tile source deployed. Writing a vector style is a matter for other people to be discussed in other places. Deploying that style to openstreetmap.org is a matter for operations and/or the web site developers once it exists, which is a logn way off. Once we have the tiles we will probably put some sort of minimal style up on a separate domain as a way of visualising them in the short term, with the hope that somebody will then start to develop a full scale style that we can then put in it's place until it's ready to consider a production deployment. We are all well aware of the additional opportunities that vector tiles present, which is why we are trying to encourage people to work in that direction, but this is not the place to discuss wider issues. |
|
Tilemaker explicitly states:
Has this concern been addressed, or is there a workaround of sorts that might not be good for others? While we could say that the stack discussion should be postponed, the specific settings to convert download into tiles is stack-specific, so deploying a solution that is not optimal for others could in theory lead to duplicated efforts. P.S. This is a great news regardless, thank you for working on getting MVT to OSM! |
|
We literally had the author of tilemaker on the operations call on Wednesday to discuss this so yes, we have discussed it. |
|
RAM estimate requirement is between 100GB and 200GB. Estimate time to generate for the full planet is around 4 days runtime on a 2x Intel Xeon X5650 using 144GB RAM system. @systemed would be able to provide more clarity. |
|
Yes, that's about right; hopefully a bit less time than that. Currently finishing off a set of performance optimisations which will give some harder numbers. |
|
If I understand it well, the goal of the experimental vector tile service is to boost the OSM vector tile style development and to improve the schema for OSM vector tiles - and allow regular iteration on development by the community. The community around @openmaptiles would be very happy to help with these efforts. For the upcoming OMT v4 these things may be of interest to OSMF:
If OSMF is interested - we could provide a weekly updated downloadable tile package for self-hosting and/or live tile endpoint (for proxy via your CDN or direct use on your DNS), based on the latest OSM data and powered by the latest modifications of the schema in GitHub. You could also run the open-source components on-prem with Kubernetes. Just let us know. Delay on pre-generating an entire planet is about a day - less if materializing only upper zoom levels, with real time on lower zoom levels served from an endpoint, or if only diffs are generated. |
|
Hosting tilemaker as an experiment does not preclude us from hosting other vector tile solutions. This is something we're planning on hosting using resources that will become available later this year. It will not be a featured tile layer on the website to start, and if it should be is a question the OWG will consider at the time. We have in the past hosted experiments that did not end up on osm.org, and looked at multiple approaches for a goal. For using a live tile endpoint, are you thinking of that on osm.org? We had spoken to someone back in March with the call for featured layers, but communication died out. If you're interested in that, please email information about it and meeting the criteria of the featured layer policy to the OWG. If you're thinking of instead asking the OWG to host an instance of the OMT stack, please open a new issue so this one remains about maptiler. |
|
Maybe tiles can/should be hosted on
effect? |
This PR adds support for several features in the `natural` key, including `saddle`, `ridge`, `arete`, and `cliff`. These features all currently render in openstreetmap-carto, so adding these features would advance OpenMapTiles towards its goal of parity with the standard renderer, as indicated in openstreetmap/operations#565 (comment). This PR also adds the features requested in #274, with the exception of valleys, which I've left out of this PR because there are different complexities to mapping valleys that should be addressed as a separate (future) PR. ### Examples of mountain features in paper maps These features are regularly found in American-style maps and thus is of interest to the openstreetmap-americana map style. Below are examples of these mountain-related features found in general purpose maps: <img src="https://user-images.githubusercontent.com/3254090/131270340-af80c7bf-9416-40a3-9b64-56418abb2aef.png" width=400/> <img src="https://user-images.githubusercontent.com/3254090/131270375-b9eb2095-7708-443d-b7bc-22bf8adab721.png" width=400/> <img src="https://user-images.githubusercontent.com/3254090/131270423-64447ad3-d90a-4615-82e1-91175a8f8a6b.png" width=400/> <img src="https://user-images.githubusercontent.com/3254090/131270506-2248db9f-ded5-443f-82ed-b0e8d4d12a70.png" width=400/> ### Approach This PR extends the existing `mountain_peak` layer by adding other mountain features. We may want to consider renaming this layer in version 4.0 to be more inclusive of other mountain features including the potential future addition of valleys. However, the features added in this PR are associated with mountains and so their inclusion in this layer is the most appropriate location. A new `osm_mountain_linestring` mapping maps the new linear mountain features, with similar ranking logic as is used for the existing `peak`/`volcano` features. Additionally, `natural=saddle` is added to the `osm_peak_point` mapping and ranked using the formula for peaks. Since saddles will have lower elevations than peaks, important saddles will be preempted by important peaks. The new linestring features are rendered only at zoom 13-14, in order to match the zoom at which they appear in openstreetmap-carto. However, it may be appropriate in a future PR to extend the rendering of these features as generalized linestrings at lower zooms. ### Test Renderings Below is a test rendering showing aretes and peaks in Austria, just north of the Swiss border, followed by a screen shot of the [same location](https://www.openstreetmap.org/#map=14/46.8682/10.0863) in openstreetmap-carto: **OpenMapTiles, zoom 14** <img src="https://user-images.githubusercontent.com/3254090/131271258-5cd90bdb-cac2-41d8-887f-b4bf6be83673.png" width=400/> **openstreetmap-carto, zoom r14/v13:** <img src="https://user-images.githubusercontent.com/3254090/131271332-32d5bcfc-41c6-4625-829c-df063b7af523.png" width=400/>
|
@matkoniecz it is rather clear that what the OSMF runs as its "standard stack" has in the past and likely will in the future have a big effect on the market for such solutions. Just consider, for example, if something else than the openmaptiles schema is chosen (a distinct possibility) for the tiles. |
The openstreetmap-americana project is presently in the early stages of developing a vector style for the US mapping community, with a small team of active contributors and nearly 100 folks dialed into our project's Slack channel. The focus on our project is to develop a regional style with features of interest to the US community, most notably comprehensive highway shield rendering with concurrencies. This project is currently based on the OpenMapTiles schema. If the OWG-hosted vector server is using an OpenMapTiles configuration, we would be more than happy to collaborate for this purpose. |
|
I'm pleased to report we've managed to perform a one-time planet build of the OpenMapTiles schema at full zoom (z14) using planetiler, and are hosting it on AWS on a temporary basis. I'm providing a description of this test planet build, costs, and technical details in case it is useful for anyone working on cloud vector tile hosting with OpenMapTiles. I also have notes that I took during this process that I'm happy to share with anyone looking to create a similar setup. The tile server is running here The key advance of planetiler is the use of in-memory transformations for mbtiles generation. As a result, planetiler is able to execute OpenMapTiles builds orders of magnitude faster than the out-of-the box scripts in the OpenMapTiles distribution. The planet render was done on an AWS m5dn.8xlarge instance, which has 128GB RAM, 32 CPUs, and 2x600GB SSDs (of which we only used 1x600GB). We executed the render on a variably-priced "spot" instance which was running approximately 40 cents per hour at the time of the render. The render took 2h 0m 6s to run on this hardware, plus an additional 6 minutes to transfer the resulting 100GB planet mbtiles file to an AWS Elastic File System (EFS) volume ($10/month). Thus we were able to generate a planet mbtiles file for a bit less than $1. However, this may cost as much as $5 for the same render at peak data center times. Because there is an OSM planet file hosted on AWS, the download of the planet pbf file takes less than three minutes, so this may result in additional download time on a different hosting provider. Next, for the tile server we spun up a separate AWS t2.nano instance (0.5GB RAM, 1 CPU) and connected it to the EFS volume, installed docker, and ran tileserver-gl with docker's restart-always setting. The t2.nano instance runs at a cost of $0.0065 per hour (a bit less than $5/month), and costs $0.09 per GB of outbound file transfer. While this is a low-powered setup, it demonstrates the most basic possible infrastructure that can run a planet vector tile build. An enterprise-level vector tile server would ideally also deploy a CDN and load balancer with higher powered servers in multiple availability zones. Planetiler is nearly complete, however, it is missing an implementation of OpenMapTiles' street abbreviation functionality. In addition, for this render I turned off wikidata integration, which would have added an additional 15 minutes to the render time. This setup could be conceptually extended to a cyclic build as follows:
|
|
Just a brief question about zoom levels. Currently the max zoom level supported by each of the osm.org maps are respectively 19, 20, 21, 21, 18 and 20. What would the max zoom level (either technical or practical) of a layer created using either tilemaker , openmaptiles, planetiler or anything else? I appreciate that with vector tiles you've effectively got "almost infinite overzoom" - you can zoom in as much as you like; but what I'm asking here is "at what level can new features appear" - can they start to be shown at the equivalent of raster zoom 18, 19 or 20? |
|
That's entirely up to the author of the style. The actual vector tiles normally stop at z14 or so and obviously you're limited by what data your schema includes in the high zoom tiles but beyond that it's up to the style which features in the tile it renders at each zoom level. |
|
Planetiler currently limits to z14 but it should be pretty straightforward to increase if people want to go higher. The z14 limit was to support openmaptiles schema which only needs z14. The main reason to go further is if your style starts including more detail and the z14 tiles get too big. The biggest z14 tiles with the openmaptiles schema are about 1.7mb (before gzip compression) and there are only a handful over 1mb so there wouldn't be much benefit for that schema. Generating more zoom levels will also make the tile render time longer, and mbtiles file bigger. The most recent run I did on planetiler on a beefy ec2 instance took 47 minutes and rendering z14 took 10 of those minutes, so z15 would probably add at least ~40 minutes and z16 would add at least another ~160 minutes. |
|
It's also worth pointing out that OpenMapTiles omits quite a few features of the "micro mapping" variety that are supported by osm-carto, so if the goal is a vector schema that supports everything that osm-carto does, we could be looking at very substantial mbtiles size inflation over the current 100GB full-zoom mbtiles produced from OpenMaptiles. |
|
I'd suggest to look also at Shortbread https://shortbread.geofabrik.de/ , a lean Vector Tile Schema, which is CC0/FTWPL licensed. |
The challenge at the moment is that somebody needs to develop the tilemaker .lua stylesheet or planetiler extension to support generating this or any other schema in vector tiles. Both tools are able to support OpenMapTiles out of the box, other schemas need developer time. |
|
It would be advisable to keep in mind that most non-trivial cartography, especially at higher zoom levels/larger map scales, requires scale/zoom level dependent geometry processing. And the practical feasibility and efficiency of client side map rendering as it is common today is strongly tied to decidedly not transferring geometry data separately for the higher zoom levels. I know this issue is not meant to be concerned with actual map design. But the question from @SomeoneElseOSM is pointing in the right direction. The bottom line is if you contemplate the question of choosing a server side data preprocessing framework for client side rendering without regards to the needs of community map design you will likely end up providing a basis exclusively for the the dead end of cartographic primitivism a la mapbox/openmaptiles. It is no coincidence that many of the most innovative studies in high zoom level OSM based map design done more recently (like https://github.com/SupaplexOSM/strassenraumkarte-neukoelln, https://github.com/enzet/map-machine) are not based on any of the more established tools you are contemplating here. |
This is basic functionality in all of the tools we're discussing here, including out-of-the-box openmaptiles. |
|
Link to zoom level specific geometry processing in openmaptiles at z>14 (or any other framework fwiw)? |
|
Geometry generalization is basic functionality in ALL vector tile renderers regardless of which zoom levels you choose to apply it to, whether it's the PostGIS functions used by openmaptiles or the Java Topology Suite used by planetiler, or whatever tilemaker does. It would be helpful if you could provide an example of what you mean by "scale/zoom level dependent geometry processing" because clearly that is done today. |
|
Well - i don't want to get into what would truly be an off-topic discussion here - I was pointing to the need for zoom level specific geometry processing at the high zoom levels, you claimed this to be available routinely in openmaptiles, i was asking for links. If you wonder about practical use cases - links i gave above show some - as does various stuff on https://github.com/imagico/osm-carto-alternative-colors. But again: This is off-topic here. My point was not discussing specific map design questions, my point was pointing out the inherent limitations of the setups discussed here so far. No one doubts that you can in principle produce a vector tile set with separate tiles for each zoom level up to 24 or higher. But the practical feasibility of all of the client side rendered vector tile systems is based on the fact that this is not done, because - as you said yourself - this would explode the volume of data to be stored and transferred quite massively. |
|
Can we please keep this on topic people. The goal here is to get a basic set of vector tiles up to encourage the creation of a project to develop a full schema and style that we can use to run a production service. We don't need to discuss the technical minutiae of what a production quality style might look like at this point! |
|
My aim was specifically not to discuss technical minutiae but to point out strategic issues that could well decide if what is encouraged in terms of style is going to be map design primitivism or avant-garde. |
|
No problem. I'll just unsubscribe from this ticket and let you get on. |
Can anyone give us an update on the status of this proposal? |
https://github.com/geofabrik/shortbread-tilemaker.git FYI |
|
To get this back on topic, some results from my experiments so far on the hardware we're planning to use for this test:
So planetiler is clearly significantly faster (and much less demanding on the machine's I/O especially) but is currently limited to the single builtin schema though that is being worked on I'm told - how much that might slow it down is unknown of course. A weekly built test would be entirely possible with any of them, and planetiler could probably do daily if we wanted. |
|
@tomhughes Nice! Glad you were able to get planetiler running on your setup. My initial goal building planetiler last year was to build out the utilities needed just for the openmaptiles schema to validate that the approach was sound, with a goal of supporting more schemas in the future. I'm working on a few more performance optimizations to make planetiler runnable on lower-end machine (<32GB ram) and utilize higher-end machines better, but after that I plan to look into how to support configurable schemas without needing to recompile the tool. We've had some initial disscussions here: onthegomap/planetiler#81, and I'm tracking my progress here: https://github.com/onthegomap/planetiler/projects/2. Tilemaker's lua profiles are a great option for iterating on schema definitions quickly! |
|
In case anyone is looking for a status update, there is significant activity around modifying planetiler to be able to configure the tile output (i.e. the data schema, mapping of OSM tags to mvt attributes) happening in the following places:
This is fairly complicated and performance-sensitive work so it's taking a bit of time to get it right, but progress is moving along quite nicely and I believe we're very close to having an initial capability that we can start testing with and incrementally improving. |
|
|
I thought the goal was to start with basically any schema and change as needed. Planetiler already is mostly compatible with OpenMapTiles schema which seemed like most suitable candidate for v1 anyway. So is the deployment actually held up by waiting for these functionalities? Feels like work on setting up the rest could continue since most of it would be useful regardless of schema. |
The goal of modifying planetiler to be arbitrarily configurable is that it could work with any schema without having to be a Java programmer to generate it. You'd just create a YAML configuration file and planetiler could generate an mbtiles from this. It remains to be seen whether or not this approach could be used in more complex cases, but we'll cross that bridge as we come to it. I haven't done a deep dive into the shortbread schema yet but I suspect after onthegomap/planetiler#160 is merged we'd be able to produce most if not all of it. The initial goal is a "trivial" output of land, water, and roads as a demonstrar.
I don't really agree that anything is "held up" and I've been working pretty hard in my spare time on making planetiler configurable. Do you follow the minutes of the OWG? Here are the minutes from the meeting I last attended where this effort was discussed, and it mentions the concern that the OWG has about using OpenMapTiles as their production schema. This is why there is currently interest in Shortbread or something like it which has a more open license (shortbread is CC0 as noted above in thread). Or, are you looking for an OpenMapTiles planet vector tile server? Here you go: That server is one that I personally host for a cost of around $10 a month. It could go away any time as I'm only using it for Americana demo purposes but it's pretty straightforward to host your own vector tile server. Here's a github repo where I keep the scripts for generating that tile server on AWS (would need to modify to change server locations and so forth to replicate): So bottom line is that a lot is happening, it's just that none of it is really within the openstreetmap-carto organization. I'm personally really excited about vector tiles and what planetiler brings to the table. |
Sorry I wasn't sure from the post if OWG is waiting for custom schemas in planetiler or not. It's great that you are working on this and this will surely be a great step for vector tile technology that will allow more people to create their own schemas/hosting. Also thanks for publishing the scripts and your server url ❤️
I read this one from march and they were the last to mention vector tiles afaik. I was under the impression that comment about OpenMapTiles was just one person's observation/opinion not official stance/decision. Actually still I am not 100% sure what's the status on OWG side. Minutes mentioned that test setup of TileMaker would be implemented and minimal schema would be selected. Since then Tom tested all 3 generators and made a point about Planetiler being significantly faster. So was the decision made to use Planetiler instead of TileMaker? And if so is OWG waiting on the custom schema with just the most basic features instead of built in OMT schema? Or is this just a matter of lack of admin time to make the experimental setup with weekly updates (on OMT schema)? As an aside I packaged some icons into sprites for my use when playing with vector tile maps in dev env, maybe it will be useful for someone else's tests: https://github.com/ttomasz/sample-icon-sprites |
|
We will not be using tilemaker in the end, but tilekiln. There will be separate tickets for that. |
|
Cool. When you make those tickets, can you link them from here so watchers can re-follow? |
|
Sure! Here's the announcement for this vector tile project: https://blog.openstreetmap.org/2024/02/11/2024-announcing-the-year-of-the-openstreetmap-vector-maps/ |
The OWG is planning to host vector tiles generated by tilemaker as an experiment. The goal is to get something others can work with to see where more work needs to be done
The intended architecture is
We have some resources we can potentially reuse for this, but the immediate priority is scaling up tile.osm.org with the new dublin and umea servers.
Items that need addressing are
The text was updated successfully, but these errors were encountered: