Addition of a build system generator

[elfring]

I suggest to reuse a higher level build system than your current small make file so that powerful checks for software features will become easier.

• CMake
• Autotools

[johnkerl]

This is a really great idea. The current is definitely simple and quick for a dev box; since the release announcement with more people building on more platforms, it'll quickly become essential. For configuring install dir, if nothing else.

[johnkerl]

If you've used cmake or autotools before (I have not) and have any sage advice, bring it on! :)

[elfring]

• Does a tool like "autoscan" show any software dependencies that you find worth for further considerations?
• How do you think about to support more development environments by the reuse of build system generators?

[johnkerl]

Marking as wishlist for now, just relative to other things more pressing -- namely, RFC-CSV and packaging. In the medium term, though, this will be a should-do rather than a wishlist item.

[0-wiz-0]

I've started an autoconf/automake based build structure in https://github.com/0-wiz-0/miller
mlr itself builds; I haven't tested it.
There is still lots of stuff to do:
tests
doc/
custom targets
distribution

To try it out:
install autoconf, automake, libtool
checkout my fork https://github.com/0-wiz-0/miller
autoreconf -fiv
./configure
make

[elfring]

checkout my fork

• I suggest to add the call of the macro "AC_CONFIG_AUX_DIR" to your configuration script so that auxiliary build tools will be stored in a dedicated directory.
• Do further parameters make sense for the call of the macro "AM_INIT_AUTOMAKE"?

[0-wiz-0]

elfring, thanks for the suggestions. I did add an AC_CONFIG_AUX_DIR call.
I also read all automake options but didn't find any I would add to the AM_INIT_AUTOMAKE call.
Let me know if I overlooked anything.

[elfring]

Let me know if I overlooked anything.

• How do you think about the reuse of parameters like "--foreign" and "-Wall"?
• Would you like to drop the call of the obsolescent macro "AM_PROG_CC_C_O"?

[0-wiz-0]

foreign is in the top level Makefile.am; there's no point in adding it to AM_INIT_AUTOMAKE, since it doesn't do anything in the other Makefiles.
Wall is only interesting for people developing the Makefile.ams. You can add it from the command line. Again, there is no point in other Makefiles than the top-level one.
I've removed the AM_PROG_CC_C_O, thanks for pointing out it's on the way out!

[elfring]

…; there's no point in adding it to …

I guess that there is a trade-off involved. How many settings should be moved from the file "Makefile.am" to the primary script "configure.ac"?

[johnkerl]

Nice!

[johnkerl]

This is cool. There's no push request yet but this is something to work with. I'm currently doing what I consider higher-pri work on file formats & read performance. But, this is important. Some of the feedback I've found off Github (by searching for it) is that my failure to provide a standard ./configure && make experience out of the box was off-putting for some.

[johnkerl]

... by which I mean, if someone else finishes the last bits, great, else I will. So we'll get there either way.

[0-wiz-0]

I've worked on it some more.
All the unit tests are now run with 'make check'.
'make distcheck' succeeds, i.e. 'make dist' generates tarballs that can be extracted, configured, built, and finish 'make check' successfully.
The man page is now installed (but only generated on the system generating the tarball, so not everyone needs to have asciidoc installed).

Can you please take a closer look and recommend which files should be distributed that currently aren't?
In particular, in doc (but consider that not everyone should have to install poki).
but also for data (but see #61).

I know that there is one more test, test/run, that should be added.

One issue I found is that the lemon executable needs its input files in ".". Perhaps it can be extended to support being called with an input file in ${srcdir} while being in ${objdir}, i.e. being called with "lemon ../../c/dsls/inputfile"?

[0-wiz-0]

I've worked on test/run a bit. I have the problem that this script has paths in the expected output. Can you please modify it so it doesn't, e.g. by calling basename on executable and input(s) (just for the debugging output)?
Otherwise out-of-src-tree builds will never run tests successfully, and 'make distcheck' uses these for testing.
Except for that it seems easy to add this script as well, I've done it locally but not pushed yet (because of -- see above).

[johnkerl]

Out-of-dir builds, very good idea in any case.

OK I think I'll fork your fork ... get it where I want it to be ... then make you a pull request. Then you can make me a pull request to master -- sound OK?

[elfring]

I think I'll fork your fork …

Are you aware of better Git development techniques?

[johnkerl]

Apparently not :^/. I'll take a read.

[elfring]

@0-wiz-0, @johnkerl: How do you think about to improve the software development collaboration by the reuse of topic branches?

[johnkerl]

sure, forks and branches are morally the same, either will work. whatever's easiest. 99% of the time i prefer unbranched iteration in master, but here either forks or branches will be called for since the changes could temporarily break master.

[elfring]

I find that a topic branch should usually get an other name than "master".
To which branch would you like to contribute the next updates here?

[jungle-boogie]

Hi @johnkerl, yes, develop on trunk/master!

http://www.tedunangst.com/flak/post/branchless-development

[johnkerl]

All -- thanks for the software-engineering discussion. Over my career, on the whole I've become a strong advocate for developing in master almost all the time, with frequent merges. This is good for small projects; it's a make-or-break for large projects/companies. That said, every once in a while a short-lived branch is useful -- and the implementation details don't matter too much -- which is the case here since autoconfigure testing will break master, and/or just confuse people, until it's complete.

[johnkerl]

Pushed out-of-directory test/run to https://github.com/0-wiz-0/miller. Ran as:

• ./test/run
• cd ../; ./c/test/run
• cd; ~/tmp/wiz/miller/c/test/run

[0-wiz-0]

Thanks!
But isn't that only half the solution? When I run this from "c/build" with "../test/run", I have the following diff in my source tree afterwards:

-mlr having-fields --at-least a,b,i,x,y /Users/kerl/tmp/wiz/miller/c/test/input/abixy
+mlr having-fields --at-least a,b,i,x,y ../test/input/abixy

In other words, the path to the input files also needs sanitizing, I think.

[johnkerl]

That case was included in the diff, using "diff -I '^mlr '" in test/run. It
works for me from all sorts of directories, on OSX. I'll investigate more,
late this evening.

On Thu, Sep 10, 2015 at 9:33 AM, Thomas Klausner [email protected]
wrote:

Thanks!
But isn't that only half the solution? When I run this from "c/build" with
"../test/run", I have the following diff in my source tree afterwards:

-mlr having-fields --at-least a,b,i,x,y /Users/kerl/tmp/wiz/miller/c/test/input/abixy
+mlr having-fields --at-least a,b,i,x,y ../test/input/abixy

In other words, the path to the input files also needs sanitizing, I think.

—
Reply to this email directly or view it on GitHub
#9 (comment).

John Kerl
[email protected]
http://johnkerl.org
https://github.com/johnkerl http://johnkerl.github.com

[0-wiz-0]

Ah, I had overlooked that. Thanks.
I have two issues:

1. When run in-tree, it overwrites c/test/output/out
2. When run out-of-tree during 'make distcheck' the source tree is read-only, which makes 'run' fail with:

mkdir: ../../../../c/test/output: Permission denied

[johnkerl]

ok. test/output/out should not be checked into source control; it should
be created by test/run. this was my oversight. i'll take a look at make distcheck.

thank you!!!! :)

On Thu, Sep 10, 2015 at 10:17 AM, Thomas Klausner [email protected]
wrote:

Ah, I had overlooked that. Thanks.
I have two issues:

1. When run in-tree, it overwrites c/test/output/out
2. When run out-of-tree during 'make distcheck' the source tree is
   read-only, which makes 'run' fail with:

mkdir: ../../../../c/test/output: Permission denied

—
Reply to this email directly or view it on GitHub
#9 (comment).

John Kerl
[email protected]
http://johnkerl.org
https://github.com/johnkerl http://johnkerl.github.com

[johnkerl]

Wiz, thanks very much for your ongoing expert help!! I'm new to autoconf, so just to be sure I know the steps:

• autoreconf -fiv
• Then the resulting ./configure gets checked into source control so not all end-users need to have autoconf, automake, and libtool installed
• ./configure run by end-user
• Interesting dev targets are make, make check, and make distcheck -- ?
• Interesting end-user targets are make, make check, and make install -- ?
• Are there important other make targets and if so where in the workflow do they belong?

[johnkerl]

I'll be reading through automake.pdf on today's flight as well.

[johnkerl]

Another test case for myself is VPATH build with read-only source tree

[0-wiz-0]

The steps are good, and you mentioned all the main targets.
There is a tags target as well, I think.
You can give debugging etc. flags on the configure command line like this:

./configure CFLAGS="-g -O0"

The configure script itself is usually not checked in. If you did, you would have to check in many more files as well (all Makefile.in and the support scripts like libtool, config.guess...).
autoconf is more aimed at a distfile workflow: someone with autotools creates a distfile (make distcheck) which contains all the necessary files. The user of a distfile does not need autotools or libtool installed.
Hope this helps. Feel free to ask more!

[0-wiz-0]

Oh, and you probably are interested in the "--disable-shared" configure argument -- that should give you a static executable without dependencies, that you can easily copy to other machines.

[johnkerl]

ALso the pdm and fdm executables in c/dsls -- I should turn these into unit-test code, or abandon them.

[johnkerl]

The merge is in. To do:

• Have the Travis build go through autoconfig
• User-level docs
• Clearly delineate in the docs what needs to be done at package-build time vs. at package-install time
• Try to reduce the number of dependencies for package install: manpage/mlrdoc/etc. should be artifacts at that point in time with no need for xmllint, poki, etc.

Thank you @0-wiz-0 !!!! :D

[johnkerl]

This is complete with docs at http://johnkerl.org/miller/doc/build.html.

There is some separate manpage-related work tracked on #56.

If there are any subsequent build errors, feel free to open a new issue as always. And thanks again @0-wiz-0. :)