Modular approach to subcommand functionality (and associated dependencies)

[chrissimpkins]

fontbakery is a true Swiss army knife for post build font issues and I love it. You have put in a ton of work here and there is a great deal of useful functionality that our team has benefit from. We would love to use this project for both scripted post build fixes and CI testing in our Hack project. The issue for us is that there are numerous unnecessary (for our purposes) and reasonably time consuming dependency installs required in order to implement the relatively simple and largely Python builtin + fontTools library tasks that we need such as the dsig table fix:

https://github.com/googlefonts/fontbakery/blob/c52332f182181fcb194fb88faddc6b5ebdfd0d2b/bin/fontbakery-fix-dsig.py

For local developer/development installs, it isn't a huge issue. We go through the install once and it's there, available for use whenever we need it. This works well in a model where the font binaries are released to end users by the project authors. As we transition to a scripted build from source, in part prompted by a few Linux distros that want to release Hack and highly suggest/require builds from open source prior to redistribution in their package managers, we would like to implement some of the fontbakery fixes as part of the scripted build approach pushing these dependency installs on others. This is where a lighter weight set of mandatory dependencies comes into play for us.

I was wondering if you would be willing to split out some of the general (as in not Google Fonts specific) testing and fix subcommands in fontbakery into lighter weight isolated executables that can be installed and executed separate from the fontbakery project, or if there might be a long term goal to create a more modular structure to the fontbakery project that would allow users to determine which "subcommands/modules" they would like to install so that it is possible to obtain only the source and associated dependencies that you need without those that you do not. This turns fontbakery into a reporting and fixing "front end" tool with a "backend" that is composed of separate repositories that meet your testing/fixing interface requirements. Open that up as a public interface for other developers and it might offer all of us a way to eliminate the time required to create new one off executables for odds and ends that are required for typeface development. We just install the new script as a module within fontbakery. The MkDocs project is an example of a Python project that uses this approach via individual Github repository modules (for project extensions that can be developed by anyone) along with PyPI distribution to users.

I understand that these suggestions may fall well outside of the bounds of your needs for Google Fonts management. Just a couple of ideas to support developers outside of the Google Fonts project with the useful code that has been developed here.

[graphicore]

Hi Chris, thanks for your feedback and this request. I'm glad you didn't ask this a few month ago, when the modular testrunner was not ready for me to point at :-) There is some good food for thought in this, so I rather use it for brain storming now.

I was wondering if you would be willing to split out some of the general (as in not Google Fonts specific) testing and fix subcommands in fontbakery into lighter weight isolated executables that can be installed and executed separate from the fontbakery project,

I don't like this very much, because we'd get a lot of fragmentation to our sources, which (even worse) wouldn't be a semantic splitting e.g. one thing is Font Bakery related and the other is rather fontTools or something completely different. Instead the splitting would be based on the dependencies as per your request.

A discussion of critical dependencies would be interesting though.

or if there might be a long term goal to create a more modular structure to the fontbakery project […]

We are working on things like this, see the new testrunner, which is now separate from the specifications/googlefonts module. You can write your own specification and use our testrunner and reporters. (Does not yet make the dependencies go away though.)

[…] that would allow users to determine which "subcommands/modules" they would like to install so that it is possible to obtain only the source and associated dependencies that you need without those that you do not.

This sounds like creating kind of a meta-fontbakery and have a build script to create subset-releases out of fontbakery.

I wonder if it can be done without all this hassle, mainly because I think the effort is not justified, even by your use case. Maybe we can short term start with finding a way to make some of the dependencies optional and create well understandable error messages if an optional dependency is missing.

This turns fontbakery into a reporting and fixing "front end" tool with a "backend" that is composed of separate repositories that meet your testing/fixing interface requirements. We just install the new script as a module within fontbakery.

This sounds reasonable. We can't do it all at once though. Thus, a plan that progresses in that direction would be needed. We already have a separate module for the tests specifications/googlefonts but, if we'd put it into a separate project, then you wouldn't be able to reuse some of our tests easily in your own "backend" without the same dependency issues you are describing right now (would you do a copy and paste job then?). We could make a per-test based central registry, which is mad but would also be cool.

Defining which tests are google-fonts-specification and which tests are "generally useful" (or even worse, based on their dependencies without any semantics) sounds like a lot of headaches to me, although not totally mad.

Open that up as a public interface for other developers and it might offer all of us a way to eliminate the time required to create new one off executables for odds and ends that are required for typeface development. We just install the new script as a module within fontbakery.

Could you get more specific on this? I wonder how it would actually work. Our "one off executables" are nothing more than a collection of simple python scripts as well (despite check-googlefonts probably, which is not simple). So I wonder how you plan to create benefit for all by fragmenting everyones work into their own "backends".

We could add a full blown package manager (that's what you actually suggest, or?) to fontbakery: fontbakery search {command}, fontbakery install {command} etc. but that's a big effort. "a public interface for other developers" something like apt-get, pip, npm or bower? I'd like to have this and I'd love to make it if someone pays for it ;-)

The MkDocs project is an example of a Python project that uses this approach via individual Github repository modules (for project extensions that can be developed by anyone) along with PyPI distribution to users.

I'm interested in an example, but I couldn't quickly find it in the docs of that project.

I understand that these suggestions may fall well outside of the bounds of your needs for Google Fonts management.

Well, yes and no. We also depend on good tools and if we can share effort that's good. However, I think we'd need to have the perspective that all the work we put into these generic things actually helps us with getting better tools or if it just creates effort we could spend smarter. At the moment, this is my personal perspective, we are rather trying to get into a good shape for our future needs from a google-fonts and fontbakery centered view, gettng rid of a lof technical debt, enable designers to use fontbakery easily (via a web interface) etc.

Just a couple of ideas to support developers outside of the Google Fonts project with the useful code that has been developed here.

Would a concept of optional dependencies help you short to mid term? Other than that, I don't see how we don't support developers "outside".

[chrissimpkins]

we'd get a lot of fragmentation to our sources,

I completely understand. No worries. Just a thought.

We already have a separate module for the tests specifications/googlefonts but, if we'd put it into a separate project, then you wouldn't be able to reuse some of our tests easily in your own "backend"

Could you get more specific on this? I wonder how it would actually work. Our "one off executables" are nothing more than a collection of simple python scripts as well

"a public interface for other developers" something like apt-get, pip, npm or bower?

So I wonder how you plan to create benefit for all by fragmenting everyones work into their own "backends".

I tend to apply the triple --verbose flag in my writing style and tried to scale back to only double --verbose in the initial post :) I'll attempt to explain this in a bit more detail and better.

Definitely nothing as extensive as a package manager. You don't need to maintain, or even know about the extensions. I am suggesting an approach that permits extension of your application by anyone in a distributed fashion. fontbakery acts as the chef and provides a set of common font baking utensils. The 'extensions' then are the flour, yeast, cinnamon, sprinkles, and bacon that fontbakery cooks with. Essentially they would be fontbakery libraries with some form of agreed upon coding interface between the 'external' extension code and your 'internal' fontbakery code so that extension authors can take advantage of common tasks that would be provided by fontbakery. Things like read/write access to OT tables, ttf builds/writes, otf builds/writes, web font builds/writes, standard output formatting, standard error formatting, exit status code handling, command line argument handling. It is all in fontbakery already, it would just need to be public facing for the extensions. Extensions then utilize the central set of fontbakery tools to address small or not so small specific issues distributed across any repository by any author. Perhaps they are very project specific and not relevant to a wider audience, others could be very generalizable and of great value to other teams. This wouldn't be your concern. I am not suggesting that you play any role in maintaining or even knowing about all of this extension code. You have plenty to do with your own set of tests provided here. I suggested a look at the MkDocs project because this is what prompted the idea. I haven't looked into their code (and theirs is a static site generator tool, the approach may not apply here, but perhaps it could) but they maintain a consistent PyPI package naming scheme for extension libraries and are somehow able to register this within the application to modify the execution of their code. It appears to be an approach to project extensions that builds on PyPI as the package manager for the extensions. The tests and font manipulation scripts that you currently maintain within the project could be turned into extensions in order to isolate the dependencies and you maintain, test, develop them as you currently do. To address your specific question, no one builds another test that you have currently developed. They use yours and they build new extension types for anyone to use when/as needed. When someone needs to bake a cake instead of a loaf of bread, they develop their own extension that is maintained, tested, documented, distributed on their end using the fontbakery architecture that you provide.

If this is possible, then the install looks something like this:

$ pip install fontbakery
$ pip install fbakery-libflour
$ pip install fbakery-libyeast
$ pip install fbakery-libsugar
$ pip install fbakery-libcheck-googlefonts

and execution is like this:

$ fontbakery flour [args]
$ fontbakery yeast [args]
$ fontbakery sugar [args]
$ fontbakery check-googlefonts [args]

Perhaps a pie (pun fully intended) in the sky idea but one that would make for an incredibly cool tool that could be simple for extension authors to build upon and greatly expand, while isolating the testing dependency structure so that only necessary elements (as defined by the user for a given project) are included for tests/checks/fixes.

Other than that, I don't see how we don't support developers "outside".

You absolutely do! This was definitely not intended to be a criticism. I understand that this is a tool for Google Fonts and its primary objective is to maintain the standards you use in the project. There are likely to be financial, administrative, etc issues that you have to take into consideration and that guide the project goals. Please take this suggestion as a 'dream' idea from a user for a group who has the capacity to build such a thing, not as a criticism of the project as is.

Maybe we can short term start with finding a way to make some of the dependencies optional and create well understandable error messages if an optional dependency is missing.

Would a concept of optional dependencies help you short to mid term?

Absolutely. Sounds great!

[graphicore]

Ok, thanks for clarification, looks like we're talking about PEP 420 - Implicit Namespace Packages see also Packaging namespace packages.

The APIs would need some thought but if python packaging tools can do the heavy lifting for us I think this is nice. Also, it looks like we could keep all of our code in this repository and create different packages for pypi, which would be ace but still some extra work.

[chrissimpkins]

👍 be happy to create some test extensions or contribute in any way that is helpful to this effort if there is anything that I can do.

Thanks for considering!

[chrissimpkins]

It struck me this morning that I totally missed an opportunity to follow your project theme. The extensions would of course be called 'recipes' ;)

[felipesanches]

I like a lot the idea of having other people using Fontbakery for their own font projects and customizing the set of tests to run.

When we refactored the codebase prior to the 0.3.x release series, we introduced the concept of uuids for the tests. All of the tests we currently have are using Google Fonts uuids like com.google.font/test/031. Other organizations could add new custom tests to Fontbakery (in the typical git workflow of keeping customized forks and preferably sending us pull-requests) with their own uuid namespaces. A naive approach for running only your set of tests could involve adding a command-line attribute to fontbakery to specify a uuid regex and only those tests would run. That's naive because I would assume that people would actually want to run a subset of other organization's tests as well. So the real need would be then to have the ability to specify custom selections of tests.

We also renamed the check-ttf target into check-googlefonts. In retrospect I think this was bad for at least 2 reasons:

• It gives people the impression that that project is only useful for Google.
• It mixes two independent semantic axes: (a) which org originally introduced the test (b) what category the test belongs to (such as "ttf binary tests" versus "upstream repo" tests)

We also created a Lib/fontbakery/specifications/ directory, supposedly meant to be the place for adding one python file for each organization to keep their respective test collections. I now think it would be better to keep all tests targeting TTF binaries (or more broadly, tests targeting binary distributions of fonts and families) under a single script with tests authored from many orgs packed together. And then to organize the collection of tests by category (binary-fonts vs. upstream-font-project vs. something-else).

If people wanted to run subsets of the full test collection, then, in that case a specifications folder would really make sense. Not to host the test implementations themselves, but instead to keep lists of uuids of which tests to run for each project. So we would hypothetically have a googlefonts_spec.py specifying that all com.google.fonts/test/* tests would be run, and also cherry-picking a few tests that may have been contributed by others (so those would have non-google uuids) but that may have been good (or important) enough that may have ended up at some point being adopted by the Google Fonts team's revised quality standards.

To help illustrate my point, think again of all the OpenType tables which were created by different vendors, but are supposed to be honored by all. I think that the "ultimate font testing tool" should aim at aggregating everyone's tests, instead of encouraging fragmentation. My fear is that by not keeping things tied together, we may have fragmentation and chaos. I'd rather keep Fontbakery as the ultimate/canonical upstream collection of tests whith every external collaborator aiming at integrating their contributions back into FB.

[felipesanches]

short version:

If we can get a good solution for the dependency-hell (assuring it is fault tolerant, never hanging/crashing due to missing import modules) then a upstream/downstream workflow could work well and make everyone happy.

Downstream is not obliged to send pull requests (even though there usually are sinergy/collaboration benefits in doing so), they can simply run the custom collection from their forks.

[chrissimpkins]

A naive approach for running only your set of tests could involve adding a command-line attribute to fontbakery to specify a uuid regex and only those tests would run.

I'm not certain that I follow what you mean here Felipe? How does this differ from the approach that you document in your README:

$ fontbakery check-googlefonts \
    -c com.google.fonts/test/034 \
    -c com.google.fonts/test/044

Do you mean rather than individual test calls in the fashion above, you could group test "collections" under a single argument linked in some way to a regular expression that pulls all required "test collection" uuid into a single command line argument? This would definitely make for a nice concise command line approach.

We also renamed the check-ttf target into check-googlefonts. In retrospect I think this was bad for at least 2 reasons:

I tend to disagree with you here, at least on the first point. The project is to help you manage Google Fonts. If I were using it from that vantage, I would want a concise call to perform the entire series of repetitive tests/fixes that I need too. And a number of the tests in the check-googlefonts subcommand as it currently stands are Google Fonts specific so it seems very appropriate. My challenge here was simply to see if there is a way to offer access to the test execution/font fix architecture that you've built in a similarly concise fashion and extensible way so that it is not necessary to reinvent the wheel each time we need an edge case project-specific or unavailable "tester / fixer" style executable.

I think that the "ultimate font testing tool" should aim at aggregating everyone's tests, instead of encouraging fragmentation. My fear is that by not keeping things tied together, we may have fragmentation and chaos. I'd rather keep Fontbakery as the ultimate/canonical upstream collection of tests whith every external collaborator aiming at integrating their contributions back into FB.

Point well taken. There is definitely value in a central collection of generally useful tests/fixes and central documentation site for them. With the model that you suggest (and that is currently available for individual test calls) you can craft project specific testing from the collection of broadly useful tests (and fixes). There are bound to be project specific edge cases that would benefit from the approach that I suggested but it is hard to use that as justification for such a model and your upstream/downstream approach seems to address them for both local and remote CI style testing.

Downstream is not obliged to send pull requests (even though there usually are sinergy/collaboration benefits in doing so),

When ready, buff those contributing docs with what meets your standards for inclusion, how to implement new tests with your uuid registration naming scheme/module structure, and the argument 'hook' that you intend to use for the test calls, and you will get them!

[graphicore]

We also renamed the check-ttf target into check-googlefonts.

Well, it runs specifications/googlefonts, nothing else.

I want to make a generic test runner command, where one can specify a custom python module, that exports a specification. That way you could create your organization specific tests in your own repo and run it with fontbakery vanilla. Also, if you want to re-use tests from specifications/googlefonts, just import them and register them in your spec.

supposedly meant to be the place for adding one python file for each organization to keep their respective test collections

No, it's just for specifications and sections, there's no other semantic level. I'd prefer more semantic modules/names though. The spec that we have is in fact googlefonts-specific at many instances e.g. font naming rules or allowed os/2 weights. Its name is justified.

I now think it would be better to keep all tests targeting TTF binaries (or more broadly, tests targeting binary distributions of fonts and families) under a single script with tests authored from many orgs packed together. And then to organize the collection of tests by category (binary-fonts vs. upstream-font-project vs. something-else).

I agree that reordering is a good idea. Luckily nothing is set in stone and we can progress towards a better code organization. It's easier than ever! 😄 I disagree with your proposal for reordering. Why don't order tests on a per OTF/SFNT-Table base? Most of TTF/OTF is identical after all.

Also, we don't have any upstream-font tests yet ;-)

My fear is that by not keeping things tied together, we may have fragmentation and chaos. I'd rather keep Fontbakery as the ultimate/canonical upstream collection of tests whith every external collaborator aiming at integrating their contributions back into FB.

I agree in so far that it would be great to have a set of industry standard tests in one place. Still, I see that different organizations and use cases may need custom, non-universal tests. Also, we have chaos already, everyone uses their own testing tools and processes ... can only get better.

[davelab6]

Why don't order tests on a per OTF/SFNT-Table base? Most of TTF/OTF is identical after all.

Indeed, https://github.com/googlefonts/fontbakery/blob/master/BRIEF.md#v010-planning says

Each tool should be structured table by table (not followed this)

[davelab6]

@chrissimpkins thanks for all your thoughtful input here, really :)

I understand that these suggestions may fall well outside of the bounds of your needs for Google Fonts management. Just a couple of ideas to support developers outside of the Google Fonts project with the useful code that has been developed here.

In fact, I would like to consider what the max bounds of the problem space that FB is working in are... I wonder, would you be interested to write a small document that essays: If you had unlimited resources to make a font update and release system like the world has never seen, what should we do?

[chrissimpkins]

I wonder, would you be interested to write a small document that essays: If you had unlimited resources to make a font update and release system like the world has never seen, what should we do?

I am taking this as a not so subtle request to stfu. Again, these are enhancement requests. Take them for what they are worth.

On a note related to the dependency issue raised in the original post that does have significance to our capacity to build from source with Font Bakery, please see source-foundry/Hack#227 (comment) re: a new Debian package build tool issue that was raised this week. Build applications (and their dependencies) for all fonts that are built from source for Debian / Ubuntu packages must be included in Debian main, an issue that I was not aware of until Paride raised it this week. As of now, I don't know of a way to get around this issue with fontbakery fixes other than to do the following: source-foundry/Hack@feeb927 . fonttools is currently part of Debian main. We poked Cosimo on this issue as well as it affects fontmake too.

Food for thought if you want to reach projects that build on Debian and Ubuntu distros with Font Bakery.

[graphicore]

I am taking this as a not so subtle request to stfu.

I can assure you it's not that. We have been discussing this question. @davelab6 was only trying to get some new perspective on the matter.

[davelab6]

LOL indeed no, I am sincere in this request. We'll be publishing an update to BRIEF.md soon and you can make a PR to that :)

[davelab6]

I was wondering if you would be willing to split out some of the general (as in not Google Fonts specific) testing and fix subcommands in fontbakery into lighter weight isolated executables that can be installed and executed separate from the fontbakery project

I guess this is done now via #1639 although I'm not sure if this is exactly what @chrissimpkins wanted.

Chris, what do you think?

[felipesanches]

@chrissimpkins please take a look at the dependency footprint of gftools: https://pypi.python.org/pypi/gftools

Is that set of dependencies acceptable? If not, please open a new issue on the gftools repo issue tracker.

Also, please let us know how you feel about the split of the tools into the new gftools package and if you still want to keep this issue open here for some other purpose or whether we're good to close it now.

[felipesanches]

[chrissimpkins]

👍 This is really helpful! Thanks all for making this change. It is greatly appreciated.

Two quick related questions before close though:

1. are you accepting PR for additional scripts that fall in a similar line of font work on that repo?
2. does the following information on the README:

These tools are intended to work with Python 2.7 systems. While these tools may work with Python 3.x, if so, that's a happy accident.

apply to the scripts that were transitioned there from Font Bakery?

OK to close this thread from my standpoint. Thanks again.

[felipesanches]

1. Yes. Feel free to submit pull requests!

2. I have tested them with python 2. I'll probably do the same with python 3 soon, but right now I cannot guarantee it will be ok in both versions. That's been postponed several times and was never considered a top priority. I would be happy to get bug reports if any of those scripts fail to work when run explicitly with python 3, though.

Thanks, @chrissimpkins !

[chrissimpkins]

👍 thanks Felipe!