Glyphs file format

[schriftgestalt]

I’m working on a future version of Glyphs. That will bring some changes to the file format. I will help implementing the changes in glyphsLib in time.

• The axes will be stored in dedicated key and the weight/width/custom keys will be gone.
• One big change will be a more compact format to store outlines. Each path will store the nodes in a string similar to a SVG path.
• There are some structural changes mostly about font info. Will publish that as soon as it is finalised.
• next to the .appVersion, key, there will be a .formatVersion = 2 key. glyphsLib could already look for that key and stop processing if .formatVersion > 1.

Do you have any suggestions/requests about the file format?

[belluzj]

Having worked with format v1, here are my recommendations:

• Please use a standard & currently supported format like JSON/XML, and do not add any tricks or extensions to it (like the quotes that are sometimes there or not). Also about quotes, if we were able to use a standard parser/serializer from the python stdlib, all the quoting would be handled automatically.
• Please use a standard date format as well, and don't try to drop the timezone sometimes to save bytes.
• For the outline data, I think relying more on strings goes in the wrong direction, instead I would suggest that each node in the path should be an object with attributes "x", "y", "smooth", "userData", "hints" etc. That would reduce considerably the number of exceptions that currently exist around the userData of a single node, for example.
• Same for all the data elements that are an object or array disguised as a string, like the list of unicodes or some other lists of numbers that currently look like "{1,2}". Make them real arrays. That would suppress several double parsing steps.
• In general, I think life would be easier for everyone involved if everything that is not part of Glyphs' "core business" of font editing was standard. Like, we could spend time on interpolating smart components instead of trying to guess whether some numbers are hexadecimal or base64.
• And since a lot of changes to the current format are driven by "saving bytes", maybe provide an option to gzip the output? For example have ".glyphs" which is standard plain text JSON/XML and ".glyphz" which is the same plain text but gziped? That way, saving bytes does not come at the cost of working around custom parsing stuff, and can be implemented trivially using stdlib components of any language.

[schriftgestalt]

Many thanks for the comments. You are right about the exceptions. I try to get rid of them.

[schriftgestalt]

I have played around with json a bit. It produces just a bit bigger files (150MB as .glyphs file) when written as minified. Writing nodes as objects, produces 80% bigger files. ;( For big CJK files like this, the raw file size has a big impact on the saving speed. And another 100MB will take some time.

Zipping helps with the file size, for sure but it is quite slow. Only zipping the .json file takes 5 second.

I need the file writing to be really quick, as it happens quite often for autosaving. And waiting for 2 seconds every 10 or 30 second is "annoying".

[belluzj]

Hum, maybe try faster/more compact standard formats like:

• MessagePack http://msgpack.org/index.html (almost drop-in replacement of JSON but compact/"binary")
• Protobuf https://developers.google.com/protocol-buffers/ (needs to define a schema of the data (would be nice to have for glyphsLib :))
• ... probably others?

Also there are faster compression algorithms than gzip:

• Zstandard https://facebook.github.io/zstd/
• LZ4 https://lz4.github.io/lz4/
• ... lots of others!

Also about the wait time, it might be possible to "mask" it by running the serialization/compression in a different thread than the UI? (I don't know how Glyphs work internally so maybe you already do that)

[schriftgestalt]

Protobuff looks very promising but has the same problem than .zip. It doesn’t play well with git.

I will investigate this and will make it as compatible as possible.

[madig]

Please also consider or give the option to write out the file into a UFO-like file-directory. The biggest annoyance in my design work is that I have to use external tools to split the changes up. With UFOs, I can stage single files (= glyphs). Doing it this way may also reduce the writing pressure, as you don't have to re-dump those 150 MB again and again -- just the glyphs that changed ;) This would make the format super git-friendly.

Edit: Maybe this would even kill two stones with one bird: file directory for normal design work, compressed single file for e-mailing it to someone?

[madig]

Actually, I have to word this more strongly: going the file-directory route is essential when more than one person is working on a font. You don't want to make font designers deal with merge conflicts just because something unrelated to their changes flipped around.

[schriftgestalt]

I heard that there are plans to have a one file approach with .ufo. The multiple file route has advantages but also problems. Both have to do with tools that use the files. Git is a bit easier but Dropbox sucks.

[madig]

It might make sense to give people the option to do both -- e.g. give them a ".glyphsdir" and ".glyphs" save option so they take whatever suits their workflow best. Both formats should just be different ways to save the exact same information.

[schriftgestalt]

Or, what I wanted to add fix the tools ;)

[madig]

I suggest fixing Dropbox then 😁

[davelab6]

Capn proto is a nice evolution of protobufs, may be interesting

[schriftgestalt]

I would love to use any of the Proto things. But then git is out of the door.

[madig]

I guess you can use text representations if you really want to? https://stackoverflow.com/questions/18873924/what-does-the-protobuf-text-format-look-like

But then you might as well experiment with JSON some more I guess. What if you represent nodes as simple lists with the line drawing operations represented as numbers? What did your node objects look like?

[schriftgestalt]

I played around a bit. It seems like that plist is still the best choice. It is very similar to JSON just uses some different characters. It can be ‘minified’ and doesn’t need quotes.

My current approach is to use shorter keys and collaps the element objects to one line. I thought about using list for the node, too. That would mean I could just replace the quotes on the node string with parentheses ;)

[madig]

Hm. Can you paste a snippet that shows how standard JSON increases file size by 80%? Really curious to see what the difference is.

[anthrotype]

It seems like that plist is still the best choice

really? you mean xml plist, or your own customized NeXTStep-like ascii plist?
It's would be so much easier if you opted for some standard serialization format.

[schriftgestalt]

Writing nodes as objects, produces 80% bigger files.

It was not about JSON but about writing "proper" node objects instead of the "string" format that is currently used. Something like this:"476 99 LINE SMOOTH" > {x = 476; y = 99; type = "line smooth"} (the type could be made a bit better, i know)

I checked again with some smaller and bigger files. After minifying it the same way the difference between plist and JSON is not that much, actually. So the JSON don't look that bad after all.

you mean xml plist,

That produces huge files in comparison.

[madig]

You could try assigning integers to types. That would save a few bytes and maybe up parsing performance?

[madig]

One more thing: adding a fixed version number to the file might be a good idea, seeing as build numbers overlap between stable and dev releases (#352 (comment)). So, every time you make even the tiniest change to the file format, you increase the number. That would make it much easier for everyone and you to correctly parse things now and in the future.

[schriftgestalt]

What about 3.1-1203? Or is that to complicated?

[madig]

Well, if there's a difference between 3.1-1203 and 3.1-1204, you might as well call it 3.2 ;)

This would require that you really only change the file format if it is necessary for a feature or it gets unwieldy quickly.

[schriftgestalt]

I do not change the version number for each build or cutting edge release.

[madig]

You don't have to if you do not change the format with each build :) if over the span of at least a minor version you could keep just eg 3.1, that would be great already.

[madig]

I suppose this can be closed, as the format won't change soon?

[schriftgestalt]

I’m heavily working on the new version. I hope I have an early beta by the end of the year.

[madig]

Alright, then I'll reopen this? Please keep us in the loop about any format change :)

[schriftgestalt]

I just made a JSON schema for the Glyphs 3 file format.
https://github.com/schriftgestalt/GlyphsSDK/tree/Glyphs3/GlyphsFileFormat/validator
The file will still be plist but the script demonstrates how to use it. I hope that makes it easier to implement the new format.

I can help with implementing the new format. It would help a lot if someone could give me some hints how it would best fit in the code base.

[madig]

Interesting. Do you use the same schema internally in G3? I'm still a bit sad that we will still have to have a plist parser :(

[schriftgestalt]

I don’t use it internally as I just made it. I’ll use it in my test as it already found some problems. And also to keep the file spec up to date.

The parser is there and works well. When building the schema I wrote a bunch of JSON and it has it quirks (leave a comma after the last element and it will not load, sometimes without a proper error message).

And I removed deviations from the plist format (unicode is stored as plain integers now).