make failing

[jungle-boogie]

Hello,

On a gnu/linux system, make is failing with a simple make:

make: *** No targets specified and no makefile found.  Stop.

cc --version
gcc-4.8.real (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Is there a new method to make mlr now?

[olorin]

Looks like miller switched to autoconf in #67. Try this:

autoreconf -if
./configure
make

[jungle-boogie]

there's also this:
https://travis-ci.org/johnkerl/miller/jobs/81385366

 % autoreconf -if
autoreconf2.50: 'configure.ac' or 'configure.in' is required

[olorin]

Looks like that's fixed in fc3db29.

[jungle-boogie]

And does autoreconf -if work for you?

[olorin]

Yup, using autoconf v2.69 and automake v1.15 on Linux. If yours is still broken with current master, what error are you getting?

[jungle-boogie]

Hello,

% autoconf --version
Autoconf version 2.13

% automake --version
automake (GNU automake) 1.14.1

---

% cd ~/bin/miller 

% git pull
Already up-to-date.

% cd c

% autoreconf -if
autoreconf2.50: 'configure.ac' or 'configure.in' is required

You're on a very newer version of autoconf than I am, so that's something worth looking into.

However....

FreeBSD  11.0-CURRENT FreeBSD 11.0-CURRENT #0 r287930: Thu Sep 17 20:40:12 UTC 2015

% autoconf --version
autoconf (GNU Autoconf) 2.69

% automake --version
automake (GNU automake) 1.15

% pwd
/home/vagrant/bin/miller/c

% autoreconf -if
autoreconf-2.69: 'configure.ac' or 'configure.in' is required

Are you sure autoreconf -ifis the right command?

[olorin]

Ah right, sorry, run autoreconf -if from the repository root, not c/. configure.ac should be in that directory.

[jungle-boogie]

ugly:

https://gist.github.com/jungle-boogie/8da8bf72194f6d42a07d

[olorin]

That looks like you don't have libtool installed; try installing that and trying again?

[jungle-boogie]

Not really any better...
https://gist.github.com/jungle-boogie/49b38c8daac24980586d

https://gist.github.com/jungle-boogie/9c0a20ae0d0cfd0a40e1

make -e CC=clang in millier and c directory immediately fail:
https://gist.github.com/jungle-boogie/cc46e900ff75dc73dd83

make generates many of the warnings that previously existed and eventually fails with this:
https://gist.github.com/jungle-boogie/62544a083794897ccc92

I know autoconf and automake can do wonderful things but now the dependencies are increased, the warnings are not corrected and portability seems smashed to pieces.

[jungle-boogie]

just fyi, this build installs:
https://github.com/johnkerl/miller/tree/v2.2.0

This is Miller version >= v2.2.0.

so to re-cap, people will now need to have:

autoconf
libtool
automake
and basically gcc compiler and/or gnu/linux.

[olorin]

In an autotools workflow you'd usually specify the C compiler at the configure stage, something like CC=clang ./configure && make. (I don't have a clang on my current machine and haven't tested this.)

[jungle-boogie]

If you take a look at this build output: https://travis-ci.org/johnkerl/miller/jobs/81531717
clang is set in the environment and if I do that, followed by make -f Makefile.no-autoconfig -e yes, it will work.

so technically, it will build but it's much more cumbersome.

[olorin]

Just tested with clang; from a fresh checkout, autoreconf -if && CC=clang ./configure && make works for me without jumping through any additional hoops.

[johnkerl]

My apologies. This is a bit in flux with current status on #9.

The short of it:

• cd /path/to/miller/c; make -f Makefile.no-autoconfig is the same build as always. No new dependencies.
• cd /path/to/miller; autoreconf -fiv; ./configure; make is the way to build from master.
• If you've got a release tarball (the next will be mlr-2.3.0.tar.gz although I haven't released that yet) then I hope you'll be able to get away with ./configure && make with no need for autotools.

The long of it:

• I'm a veteran of the software industry in many ways but am brand-new to internals of autoconfig.
• The fine work done by @0-wiz-0 is a fantastic gift to Miller, which will greatly enhance its portability.
• At this point my most important work (detailed on Addition of a build system generator #9) is to clearly and cleanly iron out what the dependencies are for master builds and for tarball builds, and to document everything for the benefit of myself and others.

[johnkerl]

Also @jungle-boogie, I do agree with you about autoconf -- I do (personally) find its error-handling confusing. Yet, many folks find it familiar and essential -- and in fact will be reluctant to use any open-source software which doesn't have autoconf support. These are two different use-cases and they both need supporting.

For those reasons:

• I'll make autoconf the default (and will connect it into the Travis build), since when it works, it works on all sorts of platforms for all sorts of people;
• I'll keep Makefile.no-autoconfig around, since it's little more than gcc -I. *.c */*.c, it's easier to figure out in cases of confusion, it builds quite quickly and transparently, and it's adequate on various Linuxes and OSX
• I'll document both alternatives
• Most importantly I'll include up front what the dependencies are for various use-cases

[johnkerl]

Lastly, I don't like to break head -- but effectively I did with last night's push (I woke up to find this conversation), by failing to communicate then what I just wrote now. This is my bad.

[johnkerl]

Fourthly, @olorin is correct that you can pick your C compiler with options to configure -- likewise with install-dir and so on. So part of the miller+autoconf doc effort will be to tabulate for easy reference some of the most-used autoconf options. (The automake.pdf manual is 100+ pages; as a service to users I certainly should distill/curate a short list of most-needed information out of that.)

[jungle-boogie]

Hi @olorin and @johnkerl,

Breaking head is better than regressions in a release so I'm happy to help find all these things before the next release. ;)

I'd give the slight recommendation that documentation get updated (when it's known to be wrong) when commits like this are made.

With what was said above, I'm still getting a build failure with
autoreconf -if && CC=clang ./configure && make:
https://gist.github.com/jungle-boogie/ffce6ee4b3a0ed3da244

autoreconf -fiv; ./configure; make also fails:
https://gist.github.com/jungle-boogie/9bf651cc084958dea36c

I project I use that works with automake and autoconf is tmux:
https://github.com/tmux/tmux

When building tmux, I don't need to specify the compiler or anything outside of the few steps on the readme once I have the dependencies installed. Will it ALWAYS be necessary to specify the compiler when it's not gcc?

[0-wiz-0]

The error output looks like you built your libtool for gcc, not clang.
Make sure you use the same compiler for both, you can't switch.

For configure, it's better to give the compiler on the command line instead of the environment, i.e.
./configure CC=clang
instead of
CC=clang ./configure

then configure remembers it when rebootstrapping due to changed autotools files.

[johnkerl]

@jungle-boogie, agreed about doc update lagging build-change -- when I said "this was my bad" above, that's what i meant.

[jungle-boogie]

Hi @0-wiz-0,

so do I simply do ./configure CC=clang or autoreconf -if && ./configure CC=clang && make

Output of first:
https://gist.github.com/jungle-boogie/70eb3f87588750e9e228

Output of second:
https://gist.github.com/jungle-boogie/1c5c4056308fde4c87b0

these were both performed in miller directory, not the miller/c directory.

@johnkerl,

Got it. thank you for your project!

[0-wiz-0]

Please make sure, as I already said above, that libtool was compiled for clang. Or just use default cc for compiling miller (i.e., remove CC=clang).

[jungle-boogie]

Hi @0-wiz-0,

Yes, I did see you mentioned that and I'll check into how libtool was compiled.

By saying just use default cc, you mean gcc?

[0-wiz-0]

Whatever 'cc' is. It seems you're running FreeBSD 10, so it might even be clang.
"cc --version" will probably tell you.

[jungle-boogie]

Yes, I'm testing this on freebsd and it is clang:

cc --version
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Target: x86_64-unknown-freebsd10.2
Thread model: posix

I'm pretty sure the libtool package for freebsd was built with clang but I'm checking.

[jungle-boogie]

Hi @0-wiz-0,

Just confirmed that libtool package is built with clang.

Thanks!

[0-wiz-0]

And just as a last datapoint, what does:

make distclean
autoreconf -fiv
./configure
make check

report?

I.e. blow away previous build, run autoconf and configure again (without CC) and make?

[jungle-boogie]

Here's the report:
https://gist.github.com/jungle-boogie/2d2186c27660932b2abf

I first did a git pull followed by make clean and then the commands above. make check is what's reported.

[0-wiz-0]

That's a different problem now. A circular dependency was introduced by
fc3db29

The cli library currently depends on the container library which contains on the mapping library, which depends on the cli library. John, can you please clean this up?

[johnkerl]

sure

[johnkerl]

It's still unclear to me how this got past Travis ...

[0-wiz-0]

Does travis start from a clean checkout every time?
If the libraries previously existed from an earlier build, it probably works.

[jungle-boogie]

Here's the most recent failure log:
https://gist.github.com/jungle-boogie/832e7a5a56703bc980b8

[johnkerl]

Yes, Travis is starting from git clone. Unless it is retaining artifacts in some way I don't understand.

I'm fixing this, but there is a deeper issue with cyclic subpackage dependencies involving the join function. @jungle-boogie you might avoid master until I resolve this issue.

[jungle-boogie]

Hi @johnkerl,

Thanks for the tip and your efforts to correct this!

[johnkerl]

meanwhile, #dayjob ;)

[0-wiz-0]

The last error jungle-boogie reported is quite similar to the first one.
My current best guess is that there is a problem with FreeBSD's clang and -pg.
Can you try building "mlrp" using the old Makefile? It is not built by default in the pre-autoconf Makefile, so perhaps this issue existed before and just became more visible up now.

[jungle-boogie]

Hi @0-wiz-0,

Is that something I can do or does it need to be done on master?

[0-wiz-0]

cd /path/to/miller/c; make -f Makefile.no-autoconfig mlrp should do it.

[jungle-boogie]

% cd c; make -f Makefile.no-autoconfig mlrp
make -C dsls -f Makefile.no-autoconfig put_dsl_parse.o
gcc -o lemon lemon.c
make[1]: exec(gcc) failed (No such file or directory)
*** Error code 1

Stop.
make[1]: stopped in /usr/home/sean/bin/miller/c/dsls
*** Error code 1

Stop.
make: stopped in /usr/home/sean/bin/miller/c

[0-wiz-0]

Well, of course, please also add the CC=clang part you always have. So.
make -f Makefile.no-autoconfig CC=clang mlrp
Thanks.

[jungle-boogie]

Hello,

https://gist.github.com/jungle-boogie/cc7e50db46084af2df81

[0-wiz-0]

Ok, so it's a problem with the mlrp executable and independent of the build system. One step forward.
Thanks for the test!

[jungle-boogie]

@0-wiz-0,

Thank YOU for walking me though all of this.

[johnkerl]

I believe the originally reported problem is resolved by my latest commit (51f89ef), although #9 is still open re html/doc info.

Re the second problem, mlrp -- I don't doubt that BSD semantics for gprof are different. @jungle-boogie I appreciate your help (if you really want to run the profiled version on your system) since I don't have access to a BSD system. (Well, OSX ;)) Can you try what the error message suggested, namely, after running your commands, follow that with simply pasting in at the command prompt

clang -g -std=gnu99 -I. -I.. -Wall -Werror -g -pg *.c cli/*.c lib/*.c containers/*.c stream/*.c input/*.c mapping/*.c output/*.c ./dsls/put_dsl_parse.o    ./dsls/put_dsl_lexer.o    ./dsls/put_dsl_wrapper.o ./dsls/filter_dsl_parse.o ./dsls/filter_dsl_lexer.o ./dsls/filter_dsl_wrapper.o -lc -lm -o mlrp

? If so that'll indicate that profiling on BSD needs -lc on the link line.

[jungle-boogie]

@johnkerl,

Will I run that in c/ or millier/?

[jungle-boogie]

After running make -f Makefile.no-autoconfig CC=clang mlrp I ran what's above and the output:

/tmp/mlr_test_util-162c18.o: In function `write_temp_file_or_die':
/home/sean/bin/miller/c/lib/mlr_test_util.c:9: warning: warning: mktemp() possibly used unsafely; consider using mkstemp()

[johnkerl]

OK so it looks like there were no errors, so if you ls -l mlrp & see it's as old as the command you typed, then that suggests that it linked OK.

So two things: one, for me, is the data point that to use gprof on FreeBSD, one needs to link with -lc. The other, for you, is if it successfully created the mlrp executable, you can go ahead and do whatever you intended to do with a gprof-enabled Miller executable.

[jungle-boogie]

Yes, mlrp has the same timestamp as when I ran the command you provided above.

[johnkerl]

Great, you're good to go then! I'm happy to hear whatever you find out about profiling Miller on FreeBSD.

[jungle-boogie]

mlrp aside, should this occur with sudo make -f Makefile.no-autoconfig CC=clang mlr install: ?

https://gist.github.com/jungle-boogie/c87f1ca07be45b4adc26

[johnkerl]

My bad. Resolving the circular-dependency issue you found, but didn't compile with/without autoconfig before pushing.

670c5c5
711c300

[jungle-boogie]

👍