proposed standard for nonlinear expression graphs

[mlubin]

In the discussion in #32 we postponed defining a standard interface for accessing expression graphs of a nonlinear model. Now that the nonlinear ecosystem is a bit more developed with support for Ipopt, NLopt, and Mosek as NLP solvers and JuMP and AMPL (JuliaSmoothOptimizers/AmplNLReader.jl#4) as modeling interfaces, I'd like to push forward on this. The proposed format uses plain scalar Julia expression objects. There are definitely possibilities to include more structure like sum{} and prod{} or matrix operations, but I think that this is a least common denominator that we'll be able to get a lot of mileage out of to start with.

In particular, the possibilities include:

• In-memory interfaces to (MI)NLP solvers (SCIP/Xpress) that have decent APIs for specifying a model using expression trees
• NL reader (exposing the expression graph interface from AMPL.jl)
• NL writer, gives us access to ASL-based solvers
• OSiL writer, gives us access to a number of solvers via OS, maybe easier than NL writer
• OSiL reader, combined with the writer gives us an open file format for exchanging problems

CC @IainNZ @joehuchette @dpo @juan-pablo-vielma @tkelman @baharev @vitaut

[dpo]

Accessing the expression graphs yields tremendous possibilities. Years ago, Bob Fourer and I devised walks through the AMPL DAG to prove and disprove convexity (http://dx.doi.org/10.1287/ijoc.1090.0321). An evolution of the idea lead to DrAMPL (http://dx.doi.org/10.1007/s10287-009-0101-z), which additionally recognizes quadratic functions, diff-convex functions, etc., and recommends appropriate solvers. If I understand well, a generic interface to the expression graph would allow you to do this in a generic way, across modeling interfaces. (See here if you don't have access to the journals.)

[mlubin]

@dpo, yes, this would enable one to implement techniques that you've proposed in a way that's generic and independent of the modeling interface, and at a much higher level than directly interfacing with asl.

[tkelman]

Readers and writers for NL and OSiL would be very valuable for us, for generating optimization problem formulations from high-level JuMP code but being able to deploy and run them on more restricted environments where getting all of Julia running would be a challenge (more than say just compiling Ipopt and its NL reader, or the OS stack).

I've worked with OSiL before and a really naive mapping back and forth between Julia expression trees and XML representation should be straightforward. (Edit: here's a really simple operator overloading based prototype in Matlab that I wrote last year https://github.com/tkelman/OptimizationServices.m/) I've read the documentation of the NL format several times but was never successful at creating properly formed problem representations from scratch. It is more widely supported and more compact than OSiL, but not human-readable (well maybe it is to @dpo or @vitaut but likely not many others) and hence harder to debug.

As I said in #32 and @jaeandersson agreed with, we should probably look at the "prior art" of how Pyomo manages multiple backends NL vs OSiL vs ... and conversion between the different formats. If Pyomo has conversions already implemented between NL and OSiL, an indirect and probably-slow-but-functional way of getting NL support could be using their code via PyCall or porting some pieces of it from Python to Julia. I think Pyomo's BSD so license should be not be a major concern.

[vitaut]

ASL is not so low-level any more =). With the C++ API we now have type-safe access to expression trees, visitors and stuff like that. All constraint programming solver connections like Gecode use this functionality to convert to their representation and it will be used for other stuff similar to what @dpo mentioned (thanks for the links BTW, great stuff).

[vitaut]

I don't know what Pyomo uses, but ASL includes an .nl writer. Of course to use it you need to build an ASL problem representation first. AMPL/MP includes ASLBuilder which allows building an ASL problem programmatically. There are some missing features like common subexpressions, but otherwise it is almost complete.

[tkelman]

@vitaut ooh that does sound really useful, did not know about ASLBuilder. And last time I was looking at this stuff I was working in Matlab and didn't want to get into the business of writing mex files, but that's no longer a problem. Is ASLBuilder usable from a C API? That's all we can reliably access from Julia yet.

[mlubin]

Here's pyomo's code for writing NL: https://projects.coin-or.org/Coopr/browser/coopr.pyomo/trunk/coopr/pyomo/plugins/io/ampl/ampl_.py. Doesn't appear to use ASL directly.

[mlubin]

It seems like there are no objections to the proposed standard format, so I'll go ahead and merge this. Discussion on implementing readers/writers/solver interfaces is welcome to continue.

[IainNZ]

Tagging in @JackDunnNZ as well

[jaeandersson]

About writing .nl: I wrote a .nl reader for CasADi. I was planning to write a writer as well (casadi/casadi#755) but so far it hasn't been high priority.

Anyway, I just used David Gay's papers for implementing it (I think http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.60.9659). I don't think there is a need to use AMPL's .nl writer. IMO, the .nl format is quite easy to understand. It's also easy to parse or to export to. It's basically a list of instructions for a stack-based virtual machine.

Note that there is also the Standard Input Format (SIF). I wouldn't use it as-is, but might be used for inspiration.

[JackDunnNZ]

Agree with @jaeandersson that the .nl format isn't too hard to follow. I have written a .nl writer in VBA for OpenSolver to convert our internal Excel models into .nl files, and it wasn't too difficult.

David Gay's paper linked above is the main resource, and in addition I needed another of his papers Hooking Your Solver to AMPL for more information on some of the nuances of the .nl file.

Basically it wouldn't be too hard to avoid using the ASL if desired IMO.

[vitaut]

Yet another .nl reader, which I blogged about recently, is included in the AMPL/MP library. It doesn't require ASL either.

[tkelman]

I might just be dense then, but I think there's a lot of value in having a human-readable serialization/interchange format for optimization problems.

@JackDunnNZ OpenSolver doesn't do nonlinear problems, does it?

There's not really much reason to want to avoid ASL, I think the biggest issue with it is that the currently written Julia-ASL bindings are GPL licensed which is somewhat problematic.

[JackDunnNZ]

@tkelman We have a new OpenSolver release coming in the next day or so with support for the COIN-OR nonlinear solvers via .nl files, and also support for Excel on OSX

[mlubin]

One option for a human-readable format is just to write out the plain Julia expressions, with a bit of decoration to separate the objective, constraints, bounds etc. This is useful for debugging, but obviously wouldn't be supported by other solvers.

For writing NL at least, I'd prefer to have a pure Julia solution just to avoid issues with binary dependencies and debugging ASL. The GPL license of AMPL.jl is also an issue; I'm sure @dpo has valid reasons for choosing this license, but for our own reasons we couldn't rely on any GPL code from JuMP.

[tkelman]

@JackDunnNZ Very cool, good to hear.

One option for a human-readable format is just to write out the plain Julia expressions, with a bit of decoration to separate the objective, constraints, bounds etc.

@mlubin that wouldn't satisfy my original use case at all. If you express the Julia syntax as S-expressions but replace parens with xml tags, you basically have the nonlinear part of OSiL.

Of course it's very tractable to do both starting from the Julia syntax. Once this is implemented I'll probably do the OSiL part.

[jaeandersson]

Would writing out the Julia expressions mean vector/matrix-valued operations in the expressions? If so, I'd like that. Our optimization tool (CasADi) supports matrix-valued expression graphs and one of the reasons I haven't been pushing the .nl import is that AMPL only has scalar-valued operations in the expression graphs.

[mlubin]

@jaeandersson, currently we're just talking about scalar expressions, but yes, potentially this could include matrix-valued expressions.

[baharev]

I personally find the NL file format a very well thought out format. It is also fairly human readable if the comments are turned on with option nl_comments 1; (emits a comment at each node) and you are familiar with the Polish prefix notation.

My only objection against the NL file format is exactly what jaeandersson writes: "It's basically a list of instructions for a stack-based virtual machine." This means that if the user wants to write a parser, then the user also has to implement a stack-based virtual machine. So the users will keep re-inventing the wheel.

I have implemented an NL file parser myself 5 years ago and it was fun, although I would not force my users to do the same (for example those users who insist on a pure Fortran implementation).

If I designed the format, it would look something like: For y[1] = u*v[1] + 0.2; which in the .nl file has the following V segment:

V6 0 0  #y[1]
o0  # + 
o2  #*
v1  #u
v2  #v[1]
n0.2

I would emit something like

t7 = v1*v2
t8 = t7 + 0.2
v6 = t8

but I would only keep the indices (assuming that the first temporary variable t has an index that is bigger than all other variable indices in the model) and I would rearrange it in such a way that the operation is right up front after the target node index:

7 * 1 2
8 + 7 n0.2
6 = 8

It makes parsing and building the expression graph super easy. Now, I realize that this format is wasteful with respect to storage space. (In my case, it has never been an issue.)

As far as I am concerned, the NL format is a decent solution to serialize a nonlinear expression graph.

[mlubin]

We've implemented this expression interface in JuMP: jump-dev/JuMP.jl#278.