Accept-encoding not sent by MapBox Studio

[pnorman]

I finally did the packet logging to verify the MBS bug mentioned in mapbox/vector-tile-spec#27 (comment)

MapBox Studio does not send Accept-Encoding with its requests for vector tiles, but gzipped tiles served with Content-Encoding: gzip are expected (mapbox/vector-tile-spec#27 (comment)).

Without an Accept-Encoding field, a server SHOULD use the identity encoding, so most HTTP servers will require some setup to serve compressed data where there is no accept-encoding and include a content-encoding header indicating that it is compressed.

cc @bcamper @zerebubuth

[bcamper]

Thanks for the cc

[nyurik]

This bug has hit us as well - without the 'Accept-Encoding': 'gzip' in the tilejson/lib/tilejson.js - TileJSON.prototype.get, our caching server (Varnish) automatically sends an uncompressed response PBF tile, even though the backend performs the nasty hack of always returning the compressed pbf even if accept-encoding was not set.

From the looks of it, tilejson lib could use request lib's ability to uncompress transparently, unless the tile uncompression happens in c++, in which case it should simply send the accept-encoding for all pbf tile requests.

[wilhelmberg]

@nyurik We are already testing tilejson with request https://github.com/mapbox/node-tilejson/tree/request

Is there a public endpoint for me to try with the latest Mapbox Studio dev build?
You could also mail it to me: my-first-name AT mapbox DOT com

[nyurik]

@BergWerkGIS i have created a test point at http://ns512621.ip-167-114-156.net/osm/pbfinfo.json -- it points to the port 4000, which is NOT behind varnish. If you remove the port, it will go via varnish, and will uncompress the data on the fly. Thanks!
P.S. I am testing with the github master version of the studio, because the 2.7 removes the host name.

[nyurik]

One more thing - I enabled it via Varnish on https://maps.wikimedia.org/osm/pbfinfo.json, but because Varnish cached the old data for today, it has to be cache busted with https://maps.wikimedia.org/osm/pbfinfo.json?a -- but query params are not supported by mapbox studio for some reason.

[wilhelmberg]

@nyurik Thanks, I'm able to replicate:
via 4000 ✅
via 80: ❌

vs2015 is the branch I'm currently working on for the next release.
https://github.com/mapbox/mapbox-studio/tree/vs2015
Would be great, if you could take a look and let me know, if you stumble upon any other bumps with that branch.

[nyurik]

@BergWerkGIS I pulled that version, but its not working -- image_reader: can't determine type from input data (which is the same error i got when mapnik gets uncompressed data i think).

You can use https://maps.wikimedia.org/osm/a/pbfinfo.json as more stable (behind varnish)
ATTENTION: The link would not work with any non wmf referrer - so copy paste it instead of clicking it

P.S. Feel free to ping me on IRC -- "yurik" (in many channels on freenode)

[wilhelmberg]

@nyurik sorry, for any misunderstanding.
I'm able to replicate the problem, but haven't solved it yet.
vs2015 is the branch I'm currently working on for the next release.
I just thought you might be interested in trying it, as it contains some other updates, like [email protected] and others.

[nyurik]

Gotcha, will try it, thanks! Are you on IRC?

[wilhelmberg]

Not really on IRC, mainly communicating via Github issues.

[nyurik]

ok, thx. I have updated this server to always use nonVarnished (hacked) method, and the official maps.wikimedia.org will always use varnish

[wilhelmberg]

@nyurik here's a dev build, could you try it and let me know, how it works for you?
Both your endpoints worked for me.

Please empty the Mapbox Studio cache folder before you use one of those versions:

*nix: ~/.mapbox-studio/cache
Windows: %USERPROFILE%\.mapbox-studio\cache

https://mapbox.s3.amazonaws.com/mapbox-studio/mapbox-studio-linux-x64-vs2015.zip
https://mapbox.s3.amazonaws.com/mapbox-studio/mapbox-studio-darwin-x64-vs2015.zip
https://mapbox.s3.amazonaws.com/mapbox-studio/mapbox-studio-win32-x64-vs2015.7z

[nyurik]

Is that the same code as in github? I would rather try it by "git pull"

[nyurik]

(unrelated to the discussion, but confusing nevertheless)

[nyurik]

I tried custom build with the https://maps.wikimedia.org/osm/pbfinfo.json -- works fine, thanks! github seems to be behind the custom build

[wilhelmberg]

@nyurik sorry for the confusion. It's the first dev build of the next version.
If you prefer pull its in the vs2015 branch.

[wilhelmberg]

Great that it works for you, too.
Thanks for testing.

[wilhelmberg]

It's working for Mapbox Studio with branch https://github.com/mapbox/node-tilejson/tree/request-mbs

Closing as the integration of requestjs into node-tilejson still seems to need some more time and testing (evaluation of consequences/breaking changes in other modules):

mapbox/node-tilejson#37
mapbox/node-tilejson#39
mapbox/node-tilejson#41

[pnorman]

@BergWerkGIS Does this mean that mapbox studio now has this bug fixed, or that a different branch not yet merged does?

[nyurik]

@pnorman, the latest mapbox studio classic works correctly with the maps.wikimedia.org, without requiring any special setup on the server. I do not know if the solution was to compress the pbfs on the fly (bad), or to always send the "accepted-encoding: gzip" (good). Nor do i know if they handle it if the server returns identity encoding despite receiving "accepted-encoding: gzip" (e.g. if its a weird sample server that ignores that header)

[wilhelmberg]

@pnorman Mapbox Studio Classic kind of works now, as long as the vector tiles are served with the extension pbf or mvt.

I think you are well aware of that "solution", as you've also commented (https://github.com/mapbox/node-tilejson/pull/41/files#r45290744) on that exact code location.

My comment when closing this issue here still holds true: a proper solution according to the HTTP specs would break other modules in the pipeline.

Maybe @yhahn could comment about a time frame if and when this might be solved/merged?