openstreetmap-carto project file for TagInfo #3151

ImreSamu · 2018-03-28T23:52:20Z

STATUS (2018May08) : As I understand and accept that the maintainence problem is not optimal for my "toy code", so I will move my code to the separate repo/project, and I will close this pull requests in the next days.

background: #961

This is a minimalistic approach: only the osm keys parsed and reported.

I have created a separated directory ./taginfo-project/

New files:

./taginfo-project/README.md : basic readme
./taginfo-project/generate-taginfo-project-file.py
./taginfo-project/taginfo-openstreetmap-carto.json -> this is the result , for the taginfo!

Status:

First working version
add ARRAY parsing ( tags @> ARRAY['building',...] )
the result imported to the Taginfo dev system, and everything is OK.
(https://github.com/gravitystorm/openstreetmap-carto/blob/master/CONTRIBUTING.md#sql-style-guidelines)
integrate to the workflow ( .travis ? )

…yle"; "project.mml"

kocio-pl · 2018-03-29T00:38:07Z

integrate with the CONTRIBUTING.md#sql-style-guidelines

I'm against adding anything to do by hand, especially if it could be easily automated. I still don't have the idea how to make it efficiently after each release, though. I think that while we can include this script, as it is closely related to our code, running this tool is another story.

ImreSamu · 2018-03-29T16:49:38Z

@kocio-pl :

integrate with the CONTRIBUTING.md

I need to re-think, but you can expect some extra documentation line ...

Testing

temporary "taginfo-project" test

please help:

the "Objects" detections is correct? ( node,way,area, realation)
Now this is the text - but I am not satisfied, what is your suggestions?
- Used in the osm2pgsql database backend, see more in the github repo
- Used as a hstore tags-> in the database backend, see more in the github repo
Please test the "More->" part ( github search )

kocio-pl · 2018-03-29T17:24:46Z

I need to re-think, but you can expect some extra documentation line ...

Sorry, I've made a mistake. It's OK to make the style more strict and consistent (it might be even preferable). I meant RELEASES.md, which you didn't even mentioned...

Still it means that this script should be triggered somehow and the best moment seems to be release time.

kocio-pl · 2018-04-01T02:45:12Z

taginfo-project/generate-taginfo-project-file.py

+
+# -------------------------------------------------------------------------------
+#
+# This code generate a taginfo project list file (see more https://wiki.openstreetmap.org/wiki/Taginfo/Projects )


generate->generates

Thank you, fixed!

kocio-pl · 2018-04-02T09:49:53Z

I think JSON file should not be included in the final version - it will be automatically generated only when needed. It should be always listed in .gitignore file.

ImreSamu · 2018-04-02T14:02:47Z

I think JSON file should not be included in the final version -

I don't think I've understood all that,
Could you clarify why don't want the final JSON file? ( = taginfo project file )

According to the "Taginfo/Projects/Integration into Taginfo"
"Taginfo will periodically (usually daily) poll the project file and add the information to its own database."

And I have planned the project file should be here :

https://raw.githubusercontent.com/gravitystorm/openstreetmap-carto/master/taginfo-project/taginfo-openstreetmap-carto.json

And I have not added a timestamp inside the JSON, so my git status don't detect as a change if the content is the same.

kocio-pl · 2018-04-02T15:52:28Z

I guess this file will be created by Travis on next merge, isn't it?

…/openstreetmap-carto"

kocio-pl · 2018-04-02T22:04:32Z

The script and documentation should be placed inside the scripts directory, just like lua.

ImreSamu · 2018-04-02T22:26:24Z

I guess this file will be created by Travis on next merge, isn't it?

I am using the Travis only for checking.

testing: my code is running?
and a simple git diff - what has been changed in the taginfo project file.

if you not need, you can remove.

imho: some human checking is useful because my code is not well tested. ( code parsing is hard )

The script and documentation should be placed inside the scripts

ok, no problem.

like this?

old files	new files
./taginfo-project/generate-taginfo-project-file.py	./scripts/taginfo-project/generate-taginfo-project-file.py
./taginfo-project/README.md	./scripts/taginfo-project/README.md

imho: than better if we move the taginfo JSON project file to the root. ( so no need a special folder )

my suggestion:

old project file	new project file
./taginfo-project/taginfo-openstreetmap-carto.json	./taginfo-openstreetmap-carto.json

The taginfo expecting a "static/fix url" for the project file, so it can be any url - as you like.
so If you don't want to store the project JSON file in the repo, no problem.

kocio-pl · 2018-04-02T22:31:38Z

like this?
old files | new files

Yes, this is what I meant.

imho: than better if we move the taginfo JSON project file to the root. ( so no need a special folder )

This would be proper location for a JSON file IMO, if we want to store it in this repo. However I'm still not sure how do you want to run the script?

HolgerJeromin · 2018-04-02T22:56:05Z

I guess this file will be created by Travis on next merge, isn't it?

We can not access travis files, so Travis can not generate the really used files. Ref #3012

ImreSamu · 2018-04-02T22:59:02Z

However I'm still not sure how do you want to run the script?

now I will generate for the current taginfo JSON file - with this pull request.
- after accepted
  - I try to add the url to the https://github.com/taginfo/taginfo-projects/blob/master/project_list.txt
every release
- re -RUN the python script
- and IF the git status shows a difference ( and it seems correct )
  * THEN git commit the new taginfo JSON project file.
AND every day the taginfo will process and show the info.

IMO, if we want to store it in this repo.

what is the alternatives?

HolgerJeromin · 2018-04-02T22:59:20Z

However I'm still not sure how do you want to run the script?

Imo a maintainer should generate the chicago preview png and this json file locally and commits both to this repo if there is a major change

kocio-pl · 2018-04-02T23:03:40Z

I said before that I don't like to make more manual steps if they can be done automatically - releasing is already quite complicated and checking the script output differences would be even worse.

This script could be run from some other place and I guess even automatically checking the new release is possible, but as simple thing as running it every night for example would do too.

ImreSamu · 2018-04-03T16:46:23Z

@kocio-pl :

I have finished.

Ready to final review, merge, adapt, delete, reject, modify ... as you like

This script could be run from some other place and I guess even automatically

Sorry, I can't help on this topic,

kocio-pl · 2018-04-04T15:24:33Z

Thanks. Unfortunately I can't do the merge here, since I'm not even a programmer. Some coder has to do the code review. Anybody willing?

kocio-pl · 2018-04-08T01:24:13Z

@matthijsmelissen What do you think about this PR?

pnorman

I've done a partial review of the code, and found it difficult to follow. More comments explaining what's going on would help, as well as a clearer flow between functions.

Document your functions with docstrings explaining what they do and what they return.

I am against any approach that requires an additional step during releases, or checking in generated files. When project.mml was generated JSON it was a big headache.

The first step might be to run this script yourself and make the results accessible to others.

pnorman · 2018-04-22T05:09:49Z

scripts/taginfo-project/generate-taginfo-project-file.py

+                    osmtype='area'
+                elif 'planet_osm_line'  in ds_table.lower():
+                    osmtype='way'
+                elif 'planet_osm_ways'  in ds_table.lower():


We don't use the ways table, and can't without a schema change.

pnorman · 2018-04-22T05:09:51Z

scripts/taginfo-project/generate-taginfo-project-file.py

+                    osmtype='way'
+                elif 'planet_osm_ways'  in ds_table.lower():
+                    osmtype='way'
+                else:


planet_osm_roads hasn't been processed

pnorman · 2018-04-22T05:11:00Z

scripts/taginfo-project/generate-taginfo-project-file.py

+
+with open( cartocss_project_file , 'r') as f:
+  newf = yaml.load(f.read())
+f.closed


Doesn't the with block make sure it's closed?

pnorman · 2018-04-22T05:12:32Z

scripts/taginfo-project/README.md

+* `tags @> '"generator:source"=>wind'`
+* `tags @> 'capital=>yes'"]`
+* `tags -> 'leaf_type'`
+* `tags -> 'wetland'`


There's already an example of this type one line up

pnorman · 2018-04-22T05:15:28Z

scripts/taginfo-project/generate-taginfo-project-file.py

+#  tags -> ARRAY['wheelchair',ramp:wheelchair']
+#  tags ?& ARRAY['wheelchair',ramp:wheelchair']
+#  tags ?| ARRAY['wheelchair',ramp:wheelchair']
+re_tags_array = re.compile(  r"[^a-zA-Z0-9_]tags\s*[@\?-][>&\|]\s*[aA][rR][rR][aA][yY]\[.+?\]"  )


Use re.IGNORECASE as a flag for re.compile to simplify the expression

pnorman · 2018-04-22T05:20:28Z

.travis.yml

@@ -1,22 +1,32 @@
 language: node_js
 dist: trusty
-sudo: false
+sudo: required


This will have a significant impact on how long it takes us to get a Travis instance

pnorman · 2018-04-22T05:23:23Z

.travis.yml

  - npm install carto@$CARTO
  - mkdir -p data/world_boundaries data/simplified-land-polygons-complete-3857 data/ne_110m_admin_0_boundary_lines_land data/land-polygons-split-3857
  - touch data/simplified-land-polygons-complete-3857/simplified_land_polygons.shp data/ne_110m_admin_0_boundary_lines_land/ne_110m_admin_0_boundary_lines_land.shp data/land-polygons-split-3857/land_polygons.shp
 script:
  # We're using pipes in the checks, so fail if any part fails
  - set -o pipefail
+  # Check the Taginfo-project
+  - cd ./scripts/taginfo-project && ./generate-taginfo-project-file.py  && cd ../..


it should be callable as scripts/taginfo-project/generate-taginfo-project-file.py

pnorman · 2018-04-22T05:23:43Z

.travis.yml

  - npm install carto@$CARTO
  - mkdir -p data/world_boundaries data/simplified-land-polygons-complete-3857 data/ne_110m_admin_0_boundary_lines_land data/land-polygons-split-3857
  - touch data/simplified-land-polygons-complete-3857/simplified_land_polygons.shp data/ne_110m_admin_0_boundary_lines_land/ne_110m_admin_0_boundary_lines_land.shp data/land-polygons-split-3857/land_polygons.shp
 script:
  # We're using pipes in the checks, so fail if any part fails
  - set -o pipefail
+  # Check the Taginfo-project
+  - cd ./scripts/taginfo-project && ./generate-taginfo-project-file.py  && cd ../..
+  - git diff taginfo-openstreetmap-carto.json


It's better to generate to a different file and diff the existing taginfo-openstreetmap-carto.json against the new one, or have the script write to stdout and use <()

matthijsmelissen · 2018-05-07T22:34:01Z

Thanks for proposing this, @ImreSamu.

The thing I'm worrying about is that this adds fairly complicated code that will break easily under many unsuspected code. Also it is quite a lot of code that achieves relatively little (it only parses the keys). Moreover, checking whether a key is used in the mml does not give any guarantees that the key is actually displayed.

So I doubt this is the right way to go. I'm again inclined now towards manually maintaining a list of interpreted tags (which might also serve as a way of specification for us).

ImreSamu · 2018-05-08T07:36:07Z

@matthijsmelissen

Thank you for you feedback.

Imho: My proposal is not guaranteed to be optimal or perfect, but sufficient for my immediate goals. Just creating a "minimal" taginfo project file with a minimal "business value" for the OSM community.

As I understand and accept that the maintainence problem is not optimal for my "toy code",
so I will move my code to the separate repo/project, and I will close this pull requests in the next days.

Now I am testing my code with other carto forks ( osmfr-cartocss, openstreetmap-carto-hiking-cz, ... )

From the "minimal" taginfo project files I can create some new "business value" :
for example:

osmfr-cartocss extra keys ( compare to this project ) :

CEMT aerodrome alt_name basin boat bus 
capacity:disabled conveying conveyor_dir craft crossing crossing:bollard ele:local footway fuel:lpg golf 
heritage importance incline information int_name is_capital level line location 
maritime medical motorboat mountain_pass name:fr network organic 
parking piste:difficulty piste:type post_office:type ramp:wheelchair ref:FR:LaPoste ref:INSEE room 
school:FR shelter_type short_name short_name:fr sloped_curb stars 
tactile_paving tidal type:RATP 
wall water wheelchair

carto-hiking-cz extra keys:

kct_blue kct_green kct_red kct_yellow network official_name:cs osmc:symbol

So I doubt this is the right way to go. I'm again inclined now towards manually maintaining a list of interpreted tags
(which might also serve as a way of specification for us).

My proposal is just a temporary ( quick&dirty ) solution. It is not perfect and I accept that it has no 'business value' for the current project members.
So I will be very happy If somebody create and maintain an improved version of taginfo project file with more business value.

Generating a simple taginfo project file from "openstreetmap-carto.st…

024a279

…yle"; "project.mml"

ImreSamu mentioned this pull request Mar 28, 2018

osm-carto for TagInfo #961

Closed

ImreSamu added 15 commits March 29, 2018 20:44

replace project name to "OpenStreetMap Carto keys"

54b3739

regenerated - taginfo project file

0b32ba4

change description to "Default OpenStreetMap.org style using CartoCSS"

4e87eb9

some refactoring in parsing

4deb542

add ./generate-taginfo-project-file.py to .travis check

e6fad5c

add "pip install pyyaml" to .travis ( missing yaml )

6490f80

fix PyYAML install

766db86

fix travis python

f563927

fix python travis

19c222e

fix python travis

cb48f81

fix travis python3

93eeb75

fix python3 travis

7cbf6a1

fix travis

3544f6c

fix travis python3

0d0794d

fix python3 yaml

2960203

kocio-pl reviewed Apr 1, 2018

View reviewed changes

ImreSamu added 2 commits April 2, 2018 16:32

fix typos, improve python code, add '-v"

b59ab9a

add "git diff taginfo-openstreetmap-carto.json" to the travis check

74db3d3

ImreSamu added 2 commits April 2, 2018 18:22

sort 'taginfo-openstreetmap-carto.json' by osm key

fdd5694

change suggested link to " https://taginfo.openstreetmap.org/projects…

79ef7f5

…/openstreetmap-carto"

ImreSamu added 2 commits April 3, 2018 16:45

move taginfo files to the new place

fec1f49

fix taginfo-project scripts path

6a5fb1b

ImreSamu changed the title ~~[WIP] openstreetmap-carto project file for TagInfo~~ openstreetmap-carto project file for TagInfo Apr 3, 2018

kocio-pl added the input needed label Apr 4, 2018

pnorman requested changes Apr 22, 2018

View reviewed changes

ImreSamu added 6 commits April 22, 2018 13:04

using re.IGNORECASE

b39eace

remove unnecessary close

461b3db

remoove duplicated example

8df146a

taginfo: improve documentation / add extra parameters / refactor .travis

07ea490

taginfo: fix .travis diff

82b272f

fix travis

b4476ac

kocio-pl mentioned this pull request Apr 28, 2018

Migration to vector tiles #3201

Open

ImreSamu closed this May 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

openstreetmap-carto project file for TagInfo #3151

openstreetmap-carto project file for TagInfo #3151

ImreSamu commented Mar 28, 2018 •

edited

Loading

kocio-pl commented Mar 29, 2018 •

edited

Loading

ImreSamu commented Mar 29, 2018

kocio-pl commented Mar 29, 2018

kocio-pl Apr 1, 2018

ImreSamu Apr 2, 2018

kocio-pl commented Apr 2, 2018

ImreSamu commented Apr 2, 2018

kocio-pl commented Apr 2, 2018

kocio-pl commented Apr 2, 2018

ImreSamu commented Apr 2, 2018

kocio-pl commented Apr 2, 2018 •

edited

Loading

HolgerJeromin commented Apr 2, 2018

ImreSamu commented Apr 2, 2018

HolgerJeromin commented Apr 2, 2018

kocio-pl commented Apr 2, 2018

ImreSamu commented Apr 3, 2018

kocio-pl commented Apr 4, 2018

kocio-pl commented Apr 8, 2018

pnorman left a comment

pnorman Apr 22, 2018

pnorman Apr 22, 2018

pnorman Apr 22, 2018

pnorman Apr 22, 2018

pnorman Apr 22, 2018

pnorman Apr 22, 2018

pnorman Apr 22, 2018

pnorman Apr 22, 2018

matthijsmelissen commented May 7, 2018

ImreSamu commented May 8, 2018

openstreetmap-carto project file for TagInfo #3151

openstreetmap-carto project file for TagInfo #3151

Conversation

ImreSamu commented Mar 28, 2018 • edited Loading

kocio-pl commented Mar 29, 2018 • edited Loading

ImreSamu commented Mar 29, 2018

Testing

kocio-pl commented Mar 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kocio-pl commented Apr 2, 2018

ImreSamu commented Apr 2, 2018

kocio-pl commented Apr 2, 2018

kocio-pl commented Apr 2, 2018

ImreSamu commented Apr 2, 2018

kocio-pl commented Apr 2, 2018 • edited Loading

HolgerJeromin commented Apr 2, 2018

ImreSamu commented Apr 2, 2018

HolgerJeromin commented Apr 2, 2018

kocio-pl commented Apr 2, 2018

ImreSamu commented Apr 3, 2018

kocio-pl commented Apr 4, 2018

kocio-pl commented Apr 8, 2018

pnorman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthijsmelissen commented May 7, 2018

ImreSamu commented May 8, 2018

ImreSamu commented Mar 28, 2018 •

edited

Loading

kocio-pl commented Mar 29, 2018 •

edited

Loading

kocio-pl commented Apr 2, 2018 •

edited

Loading