Symbolic Ring to Maxima via EclObject

[nbruin]

With maxima-as-an-ecl-library and ecl accessible as a library, we can start interfacing with maxima via a binary library interface. This should be more efficient and more robust, because expressions can be transmitted in a much richer format than text and parsing does not have to recognise error messages and questions (since communication does not go via STDIN/STDOUT anymore).

Applies cleanly to 4.7.alpha3.

Depends on: #10766, #10773, #10743. #10818 desirable but not essential.

Apply:

• attachment: trac_7377-abstract-maxima_p2.patch
• attachment: trac_7377-maximalib_p2.patch
• attachment: trac_7377-fastcalculus_p2.patch
• attachment: trac_7377-better-ask-error_p2.patch
• attachment: trac_7377-split_and_refactor-p3.patch
• attachment: trac_7377-lazy-maxlib.p2.patch
• attachment: trac_7377-floatcast.patch
• attachment: trac_7377-unicode_to_ecl-p1.patch
• attachment: trac_7377-assumptions-p3.patch
• attachment: trac_7377-doctests-p3.patch

CC: @burcin @saliola @jpflori

Component: symbolics

Author: Robert Bradshaw, Nils Bruin, Jean-Pierre Flori, Karl-Dieter Crisman

Reviewer: Jean-Pierre Flori, François Bissey, Karl-Dieter Crisman, Nils Bruin

Merged: sage-4.7.1.alpha0

Issue created by migration from https://trac.sagemath.org/ticket/7377

[nbruin]

Attachment: sagemax.py.gz

example package for how to accomplish said interface

[nbruin]

I think the following email might be of interest to other people wanting to hack maxima:

Since there is already a decent string parser for taking Sage stuff to
Maxima and (especially) getting back Sage stuff from Maxima, would
there be a way to modify the above - or ecl_eval - to give back the
Maxima expression directly? Perhaps this is already in the
lib/ecl.pyx file, though I didn't see it on the first reading.

lib/ecl.pyx has nothing to do with maxima-specific stuff. That is all in
sagemax.py. It is actually a little bit important to keep that distinction.
Maxima is not the only lisp program we might want to interface with.

There is max_read(S) = cadadr(EclObject("#$%s"%S)), which reads the string
S as a maxima expression and returns its parsed result.

So meval(max_read(S)) would do the trick. I do not know what routines maxima
has to convert expressions to a string. If I wanted to know, I would find it
out by looking at something like

max_read("maxima-command-to-print-to-string(x)")

and see what token the maxima reader puts in. I imagine you get something
like

<ECL: ((MAX-PRINT-STRING) x)>

back, So do:

maxprint:=EclObject("MAX-PRINT-STRING") #do this once globally
meval(EclObject([[maxprint], maxexpression])).python()

Indeed:

sage: max_read("string(x)")
<ECL: (($STRING) $X)>
sage: meval(max_read("string(x)"))
<ECL: "x">
sage: meval(max_read("string(x)")).python()
'"x"'
sage: meval(max_read("string(x)")).python()[1:-1]
'x'

So

sage: maxprint=EclObject("$STRING")
sage: def max_to_string(s):
....:      return meval(EclObject([[maxprint],s])).python()[1:-1]

would give you a string representation. Be aware that #$...$ suffers from
performance loss, so if you repeatedly use it, your timings will go south
quickly. See Dodier's comments on this a while ago. Apparently you could
patch Maxima to lose the performance-losing routine (at the expense of
losing the wonderful linear-search history the maxima parser offers now).

Sage command -> calls Maxima command -> Maxima command evaluated, but
in the ECL library version, not the pexpect version -> Maxima output
(a string) converted back using the stuff in Sage

Yes, that would be an approach virtually orthogonal to the one taken in sagemax.py.

> sage: str(ecl_eval("#$load(to_poly_solver)$"))
> append: argument must be a non-atomic expression; found false
> -- an error.  To debug this try debugmode(true);

To illustrate the inner workings:

sage: from sagemax import *
sage: e=max_read('load("to_poly_solver")')
sage: e
<ECL: (($LOAD) "to_poly_solver")>
sage: meval(e)
append: argument must be a non-atomic expression; found false
 -- an error.  To debug this try debugmode(true);

sage: meval(max_read("append(a,b)"))
append: argument must be a non-atomic expression; found a
 -- an error.  To debug this try debugmode(true);

So my guess is that "load" is trying to append to an improperly initialized
maxima list. I guess that means that in the current setting, our maxima
environment hasn't properly been set up yet. That is no surprise. It is
actually surprising that so much of maxima works without any initialization
at all.

[robertwb]

Attachment: 7377-abstract-maxima.patch.gz

[nbruin]

comment:3

Finally success! Recipe for getting calculus use
maxima in ecl library mode do the following

• apply the patch from Library access to ecl #6781
• apply Robert's 7377-abstract-maxima.patch
• apply maxlib.patch
  The calculus doctests currently fail because maxima asking questions does not get handled yet.

[nbruin]

comment:4

Updated maxlib.patch. Changes made:

• Code catches (throw 'macsyma-quit) and tries to recognise errors (often successful)
• suppress questions and throw errors instead
• suppress ";;; load ..." lines and load warnings
• set environment appropriately for calculus instance
  There is a nasty problem that "forget(facts())" gets executed by sage, but throws errors. These get caught by the classic interface, but mess up the printing.

Doctesting calculus/calculus.py leads to 10 failures. All have to do with either forget() acting up or the fact that "maxima asks a question" gets reported differently by the two interfaces.

(the uploaded code has some diagnostic output in maxima_eval to try to debug the "forget" issues. The "ask" errors should just consist of updating the doctest.

Go at it!

[nbruin]

Maxima via library; used by calculus

[nbruin]

comment:5

Attachment: maxlib.patch.gz

Another update:

• change to symbolic/assumptions.py to not try: ... error: pass (gentler removal of assumptions makes the interface behave a bit better)

• monkey patch maxima to throw error on divergent integral instead of printing "Principal Value" and giving an answer.
  (maxima does behave as documented, so it is a matter of taste to change maxima's behaviour)

• slightly more pretty/informative error messages

Doctests failures on calculus.py are only 5 and are restricted to whitespace and differently formatted error messages (in other words, for real use, the two interfaces shouldn't be distinguishable anymore).

[kcrisman]

comment:7

Very sadly I do not have time right now to look at this properly, but if it behaves as advertised, the layout is very understandable and I think might even improve the previous interface in terms of future customization.

Do you think you can post some sample timings, just for information? After all, one (not the only) reason for this would be to significantly speed up repeated use of Maxima for certain applications.

Thanks to nbruin and robertwb for working so hard on this!

[nbruin]

Patch to get faster calculus routines

[nbruin]

comment:8

Attachment: fastcalculus.patch.gz

The latest patch must be applied after abstract-maxima and maxlib. It adds routines sr_to_max and max_to_sr that try to translate between maxima's internal datastructure and SR while avoiding string transformations as much as possible. It falls back to the old interface to establish many of its symbol and operator translations, but takes note of them and uses a dictionary directly the next time around. It also avoids the overhead of binding expressions to maxima variables.

to go back and forth, maxima_lib.MaximaElement() as a new function {ecl()} which passes back the underlying EclObject. Conversely, maxima_lib.maxima accepts EclObjects and returns a corresponding expect interface object.

This patch also rewrite calculus integral, limit and sum to directly pass back and forth to maxima. There is a lot of overhead in SR itself, so speed gains are not that big. New times are:

sage: timeit("integral(cos(x),x)")
625 loops, best of 3: 1.1 ms per loop
sage: timeit("integral(cos(x^2),x)")
5 loops, best of 3: 31.2 ms per loop

where old were

sage: timeit("integral(cos(x),x)")
25 loops, best of 3: 8.08 ms per loop
sage: timeit("integral(cos(x^2),x)")
5 loops, best of 3: 37 ms per loop

It is easy to swap the old and new interfaces: simply comment/uncomment the appropriate lines in calculus.py to define the appropriate calculus.maxima

There are 7 doctest failures in calculus.py. All of them are whitespace, errors and changed precision in floats:
the binary interface passes doubles directly, so there are no digits rounded. This affects some doctests. In other words, as far is calculus.py is concerned the two interfaces are functionally equivalent.

[kiwifb]

comment:11

Does this need any rebasing? I am willing to give it a spin in sage-on-gentoo
to see if there are any obvious problems on linux.

[burcin]

comment:12

Replying to @kiwifb:

Does this need any rebasing?

Unfortunately, it does. Since Robert split off the Maxima pexpect interface, there were a few minor changes to sage/interfaces/maxima.py. His patch needs to be rebased.

AFAIR, the only reason that held this ticket back was a mix-up in the updates to the ecl/maxima packages. The version that properly built maxima as a library got lost since there were several new packages at the same time. I don't know if that problem is fixed in the latest release.

[sagetrac-drkirkby]

comment:13

Note that there's a ticket to update both Maxima and ECL open just now. See #10187

At this point in time, it does not have positive review, but I think it is pretty close to getting it. The packages work together, but there's a doctest failure which needs to be resolved. As far as I can see, its just that Maxima is outputing results in a different form (yet again).

I would suggest you use the packages at #10187 and make this dependent on that, which I think will be merged in the next alpha.

[jpflori]

comment:14

I tried to update the patches so that they apply on Sage 4.6.1 with Maxima 5.22.1, some things moved since the last patches were produced, but there was nothing too terrible. I'll up them right now.

However, there is now a problem when loading Maxima as a library. Maybe some changes in the spkg (in init-cl.lisp where set-pathnames is defined ?) are responsible for this. Without that line, there is a problem later.

/home/jp/boulot/sage/sage-current/local/lib/python2.6/site-packages/sage/interfaces/maxima_lib.py in <module>()
    487 ecl_eval("(defvar *MAXIMA-LANG-SUBDIR* NIL)")
    488 ecl_eval("(set-locale-subdir)")
--> 489 ecl_eval("(set-pathnames)")
    490 ecl_eval("(defun add-lineinfo (x) x)")
    491 ecl_eval("""(defun retrieve (msg flag &aux (print? nil))(error (concatenate 'string "Maxima asks:" (meval (list '($string) msg)))))""")

/home/jp/boulot/sage/sage-current/local/lib/python2.6/site-packages/sage/libs/ecl.so in sage.libs.ecl.ecl_eval (sage/libs/ecl.c:5616)()

/home/jp/boulot/sage/sage-current/local/lib/python2.6/site-packages/sage/libs/ecl.so in sage.libs.ecl.ecl_eval (sage/libs/ecl.c:5570)()

/home/jp/boulot/sage/sage-current/local/lib/python2.6/site-packages/sage/libs/ecl.so in sage.libs.ecl.ecl_safe_eval (sage/libs/ecl.c:2403)()

RuntimeError: ECL says: In function PATHNAME, the value of the only argument is
  NIL
which is not of the expected type (OR FILE-STREAM STRING PATHNAME)
Error importing ipy_profile_sage - perhaps you should run %upgrade?
WARNING: Loading of ipy_profile_sage failed.

I don't know anything about Lisp, ECL and Maxima, so I was not able to fix that in the little time I tried.

As far as the patches as concerned, I feel that more work is needed. Deleting duplicate examples and methods, moving the library interface to libs directory, not sure where maxima abstract should go (if it is kept, should something be done like MaximaLib/Maxima as for Pari/GP ? no idea if those two "interfaces" share something...), getting rid of the inheritance on Expect in Maxima_lib and Maxima_abstract.

I'm ready to have a look at all of this, if someone helps me launching Maxima as library with latest versions of everything.

[jpflori]

Attachment: trac_7377-abstract-maxima-rebased.patch.gz

Rebased on Sage 4.6.1

[jpflori]

Rebased on Sage 4.6.1

[kiwifb]

comment:15

Attachment: trac_7377-maximalib-rebased.patch.gz

What happens if you just remove that line?
I just noticed a typo - marking makes you terrible at that kind of things :)
"We begin here by initializing maxima in library more"
^

[jpflori]

comment:16

It fails later, here is the piece of code involved:

/home/jp/boulot/sage/sage-current/local/lib/python2.6/site-packages/sage/interfaces/maxima_lib.py in __init__(self, script_subdirectory, logfile, server, init_code)
    586         ecl_eval("(setf *standard-output* *dev-null*)")
    587         for l in init_code:
--> 588             ecl_eval("#$%s$"%l)
    589         ecl_eval("(setf *standard-output* original-standard-output)")
    590 

/home/jp/boulot/sage/sage-current/local/lib/python2.6/site-packages/sage/libs/ecl.so in sage.libs.ecl.ecl_eval (sage/libs/ecl.c:5616)()

/home/jp/boulot/sage/sage-current/local/lib/python2.6/site-packages/sage/libs/ecl.so in sage.libs.ecl.ecl_eval (sage/libs/ecl.c:5570)()

/home/jp/boulot/sage/sage-current/local/lib/python2.6/site-packages/sage/libs/ecl.so in sage.libs.ecl.ecl_safe_eval (sage/libs/ecl.c:2403)()

RuntimeError: ECL says: THROW: The catch MACSYMA-QUIT is undefined.
Error importing ipy_profile_sage - perhaps you should run %upgrade?
WARNING: Loading of ipy_profile_sage failed.

Tried to google that, but couldn't find anything meaningful to me.

[kiwifb]

comment:17

me neither :(

[jpflori]

comment:18

I don't know what happened but I rebuilt ecl and maxima while trying to add debug information and everything is fine now. I surely did something wrong at some point...

There are some problems left with what I uploaded earlier in sage.symbolic.integration.external I'll upload fixed version of the patches soon.

[jpflori]

Attachment: trac_7377-fastcalculus-rebased.patch.gz

Rebased on Sage 4.6.1, fixed changes in sage.symbolic.integration.external

[kiwifb]

comment:19

Ok I will give it a spin in sage-on-gentoo.

[kiwifb]

comment:156

I'll admit that I was surprised you took that "patch bomb" in at all initially in alpha4 then alpha5. I am not completely sure but I think the new interface.py makes a number of components chatty. When I reviewed the test logs from my experiments with python-2.7 I have seen plenty of chatter of stuff going by pexpect that I don't remember being there before.

[kcrisman]

comment:157

Well, if that is the case then I guess this is wise. I do think that it's not appropriate to ask JP or Nils or anyone else to make more fixes until it's tested on a more wide basis by being in an alpha - as long as JP has time to rebase to 4.7, if that's necessary! 150+ comments is enough.

[jpflori]

comment:158

Replying to @kiwifb:

I'll admit that I was surprised you took that "patch bomb" in at all initially in alpha4 then alpha5. I am not completely sure but I think the new interface.py makes a number of components chatty. When I reviewed the test logs from my experiments with python-2.7 I have seen plenty of chatter of stuff going by pexpect that I don't remember being there before.

Could you tell me where to have a look for those chatty things ? Don't have much time to investigate that alone right now.

It is also a little bit surprising because the splitting of the expect class into expect and interface classes should be kind of trivial (just take a bunch of methods in expect and paste them in a superclass).

At least for most classes inheriting expect it should not have dramatic consequences. Of course I could have screwed up something in the way.

Both Maxima classes are more problematic because of multiple inheritance. It "works" now thanks to python MRO, but IIRC, that MRO will change with python 3.0 and could break everything.

[jpflori]

Work Issues: investigate pexpect refactoring

[jdemeyer]

comment:160

I get the following error on SunOS 5.10-32 (marks) with a trial sage-4.7.alpha5. I'm sure whether the error is repeatable and whether it is related to this ticket, but I'm copying it here anyway:

sage -t -long  -force_lib devel/sage/sage/misc/trace.py
**********************************************************************
File "/home/buildbot/build/sage/mark2-1/mark_full/build/sage-4.7.alpha5/devel/sage-main/sage/misc/trace.py", line 54:
    sage: _ = s.expect('100')
Exception raised:
    Traceback (most recent call last):
      File "/home/buildbot/build/sage/mark2-1/mark_full/build/sage-4.7.alpha5/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/home/buildbot/build/sage/mark2-1/mark_full/build/sage-4.7.alpha5/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/home/buildbot/build/sage/mark2-1/mark_full/build/sage-4.7.alpha5/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_1[6]>", line 1, in <module>
        _ = s.expect('100')###line 54:
    sage: _ = s.expect('100')
      File "/home/buildbot/build/sage/mark2-1/mark_full/build/sage-4.7.alpha5/local/lib/python/site-packages/pexpect.py", line 912, in expect
        return self.expect_list(compiled_pattern_list, timeout, searchwindowsize)
      File "/home/buildbot/build/sage/mark2-1/mark_full/build/sage-4.7.alpha5/local/lib/python/site-packages/pexpect.py", line 989, in expect_list
        raise TIMEOUT (str(e) + '\n' + str(self))
    TIMEOUT: Timeout exceeded in read_nonblocking().
    <pexpect.spawn instance at 0x32b4990>
    version: 2.0 ($Revision: 1.151 $)
    command: /home/buildbot/build/sage/mark2-1/mark_full/build/sage-4.7.alpha5/sage
    args: ['/home/buildbot/build/sage/mark2-1/mark_full/build/sage-4.7.alpha5/sage']
    patterns:
        100
    buffer (last 100 chars): 
    before (last 100 chars):                          *

    **********************************************************************


    after: <class 'pexpect.TIMEOUT'>
    match: None
    match_index: None
    exitstatus: None
    flag_eof: 0
    pid: 24906
    child_fd: 4
    timeout: 30
    delimiter: <class 'pexpect.EOF'>
    logfile: None
    maxread: 2000
    searchwindowsize: None
    delaybeforesend: 0.1
**********************************************************************
File "/home/buildbot/build/sage/mark2-1/mark_full/build/sage-4.7.alpha5/devel/sage-main/sage/misc/trace.py", line 61:
    sage: print s.before[s.before.find('-'):]
Expected:
    ---...
    ipdb> c
    2 * 5
Got:
    ----------------------------------------------------------------------

    | Sage Version 4.7.alpha5, Release Date: 2011-04-13                  |

    | Type notebook() for the GUI, and license() for information.        |

    ----------------------------------------------------------------------

    **********************************************************************

    *                                                                    *

    * Warning: this is a prerelease version, and it may be unstable.     *

    *                                                                    *

    **********************************************************************

    <BLANKLINE>
**********************************************************************
1 items had failures:
   2 of  11 in __main__.example_1

[kiwifb]

comment:161

Replying to @jpflori:

Replying to @kiwifb:

I'll admit that I was surprised you took that "patch bomb" in at all initially in alpha4 then alpha5. I am not completely sure but I think the new interface.py makes a number of components chatty. When I reviewed the test logs from my experiments with python-2.7 I have seen plenty of chatter of stuff going by pexpect that I don't remember being there before.

Could you tell me where to have a look for those chatty things ? Don't have much time to investigate that alone right now.

I have posted a log at #9958 (sage-4.7.alpha4-test.log.bz2) look for mwrank.py for example.

[jpflori]

comment:162

Replying to @kiwifb:

I have posted a log at #9958 (sage-4.7.alpha4-test.log.bz2) look for mwrank.py for example.

Are you sure that mwrank is related in any way to the expect.py/interface.py we've modified here ?

mwrank.pyx provides access to a C++ library and the interface.py in the same folder just provides direct python access to methods defined in the first file or am I completely missing something here ?

[jpflori]

comment:163

I've search through the logs for "interfaces" (the main directory wa are concerned with), "calculus" and "symbolic".

The only noise I found so far is:

File "/usr/share/sage/devel/sage-main/sage/symbolic/expression.pyx", line 7728:
    sage: f = 1 + x + sqrt(x+2); f.find_root(-2,10)
Expected:
    -1.6180339887498949
Got:
    -1.618033988749895
#0: simplify_sum(expr='sum(q^k,k,0,inf))
#1: simplify_sum(expr=a*'sum(q^k,k,0,inf))

and:

File "/usr/share/sage/devel/sage-main/sage/calculus/calculus.py", line 223:
    sage: float(z)
Expected:
    4.6467837624329356
Got:
    4.646783762432936
#0: simplify_sum(expr='sum(q^k,k,0,inf))
#1: simplify_sum(expr=a*'sum(q^k,k,0,inf))

Which does not cause any problem if the doctest passes (i.e. if we set the expected answer to the correct precision).

It's caused in some way by ECL (I've tried to redirect as much as possible toward "/dev/null" slightly modifying the first version of the patches but could not get better result, I'm no expert in Lisp) and not by the refactoring of expect.py into expect.py/interface.py

[kiwifb]

comment:164

OK, no problem Jean-Pierre. There are a lot of things going on with python 2.7 that may just be one of them. I thought the mwrank executable was called by the expect interface. There probably wouldn't be anything to spot if it weren't for all these doctests failures. It doesn't cause the test failure either I think.

[jpflori]

comment:165

I definitely think the mwrank problem is complete unrelated to this ticket.

Could someone repoduce Jeroen bug ? I've got no problems on my debian amd64 with 4.7.alpha3, but I won't have the time to build the latest alpha today or next week, nor working on this ticket, but is there anything preventing it to go back to needs_review or even positive_review ?

[jdemeyer]

comment:166

This bug is most likely not related to this ticket.

[jpflori]

Changed work issues from investigate pexpect refactoring to none

[kiwifb]

comment:168

I think we are now clear. The chattiness is likely to come from somewhere else and Jeroen says his bug is probably unrelated. I'll put that back to positive review - again.

[jdemeyer]

Merged: sage-4.7.1.alpha0