Python linsys solver. #11

bamos · 2019-04-03T17:49:59Z

This is a nearly-finished PR for a linear system solver that calls back up into Python. It passes an empty A matrix into SCS and defers all operations involving A to Python. Here is some of the remaining work (if you're fine with the core content of this PR, it may make sense to merge it in sooner and move some of these to the issue tracker)

Add support for other precision.
Decide if anything else should be passed back up to Python.
Add normalization.
Add some docs and a description somewhere.
Add more checks to the functions passed in to make sure they are reasonable/take the correct number of arguments since otherwise errors coming from C calling back into Python can be uninformative/unnecessarily long
Windows support.

I've added a simple test in the same format as the other tests in test/test_scs_python_linsys.py and here's an example use of the interface:

ncon_cone, nz_cone = A.shape
rho = 1e-3
M = sp.bmat([[rho*sp.eye(nz_cone), A.T], [A, -sp.eye(ncon_cone)]])
Msolve = sla.factorized(M)

def solve_lin_sys_cb(b, s, i):
    b[:] = Msolve(b)

def accum_by_a_cb(x, y):
    y += A.dot(x)

def accum_by_atrans_cb(x, y):
    y += A.T.dot(x)

sol = scs.solve(
    data, cones, verbose=True, use_indirect=False,
    normalize=False, max_iters=10,
    linsys_cbs=(solve_lin_sys_cb,accum_by_a_cb,accum_by_atrans_cb)
)

bamos · 2019-04-03T18:06:48Z

The tests on windows are currently giving NaNs so I've disabled the test and added windows support to the issue list above

bamos · 2019-04-03T19:46:34Z

The travis tests are failing since cvxgrp/scs#110 hasn't yet been merged, I'll re-run them once that's been merged in. This may also fix the appveyor tests.

Slight consistency update to setup.py

bodono

One thing to check is for memory leaks. Try running many solves sequentially with new data each time and make sure that the memory usage doesn't climb over time.

bodono · 2019-04-04T10:24:15Z

src/scsmodule.c

 /* Note, Python3.x may require special handling for the scs_int and scs_float
 * types. */
-static int get_int_type(void) {


Is there a problem with these being static?

I'm using them as extern in my linsys solver, although I could just copy them over there or share them in a cleaner way if you prefer

Ok that's fine, probably good to prefix them with something like 'scs_' just so they don't collide with any other possible functions in the namespace.

Added bamos@67aa92e

bodono · 2019-04-04T10:24:46Z

test/test_scs_python_linsys.py

+import sys
+import gen_random_cone_prob as tools
+
+if platform.system() == 'Windows':


We'll figure out how to fix the windows tests

bodono · 2019-04-04T10:31:20Z

src/scsmodule.c

+#endif
+
+
+#ifndef PYTHON_LINSYS
  /* release the GIL */
  Py_BEGIN_ALLOW_THREADS;


We could release the GIL here and then reacquire once we enter each callback (then release at the end of each callback), that would probably be better for multi-threading.

Hmm, I thought about this earlier. This would require sharing the python thread state from this part with the linsys calls. I think the easiest way of adding this would be to have a global _save variable in scs-module.c that private.c modifies as it's acquiring/releasing the GIL around the Python callbacks. What do you think?

Ref: https://docs.python.org/3/c-api/init.html#c.Py_BEGIN_ALLOW_THREADS

Hmm, I just tried doing this and am getting a segfault and am not sure what else to try. I put this around the main scs call:

/* release the GIL */ scs_thread_state = PyEval_SaveThread(); /* Solve! */ scs(d, k, &sol, &info); /* reacquire the GIL */ PyEval_RestoreThread(scs_thread_state);

And this around all of the callbacks:

PyEval_RestoreThread(scs_thread_state); PyObject_CallObject(scs_solve_lin_sys_cb, arglist); scs_thread_state = PyEval_SaveThread();

Weird, ok let's just go with what you have for now then.

bodono · 2019-04-04T10:36:57Z

Once you resolve the minor comments I have (only major question being to check for memory leakage) I will merge this and then we can iterate on it some more.

bamos · 2019-04-04T14:15:37Z

One thing to check is for memory leaks. Try running many solves sequentially with new data each time and make sure that the memory usage doesn't climb over time.

First I tracked the memory for multiple solves with the same data and there aren't any apparent leaks here:

print('--- vanilla scs ---')
gc.collect(); print('  + {} bytes'.format(process.memory_info().rss))
for _ in range(10):
    sol = scs.solve(
        data, cones, verbose=False,
        use_indirect=False, normalize=False,
        max_iters=10,
    )
    gc.collect(); print('  + {} bytes'.format(process.memory_info().rss))

...

print('\n\n--- scs with python cbs---')
gc.collect(); print('  + {} bytes'.format(process.memory_info().rss))
for _ in range(10):
    sol = scs.solve(
        data, cones, verbose=False, use_indirect=False,
        normalize=False, max_iters=10,
        linsys_cbs=(solve_lin_sys_cb,accum_by_a_cb,accum_by_atrans_cb)
    )
    gc.collect(); print('  + {} bytes'.format(process.memory_info().rss))

--- vanilla scs ---
  + 108814336 bytes
  + 112439296 bytes
  + 112439296 bytes
  + 112447488 bytes
  + 112447488 bytes
  + 112451584 bytes
  + 112451584 bytes
  + 112451584 bytes
  + 112451584 bytes
  + 112451584 bytes
  + 112451584 bytes


--- scs with python cbs---
  + 112701440 bytes
  + 112852992 bytes
  + 112852992 bytes
  + 112852992 bytes
  + 112857088 bytes
  + 112857088 bytes
  + 112857088 bytes
  + 112857088 bytes
  + 112857088 bytes
  + 112857088 bytes
  + 112857088 bytes

Next I tracked the memory for multiple solves with the different data every time and there aren't any apparent leaks here:

print('--- vanilla scs ---')
init_m = process.memory_info().rss
gc.collect(); print('  + {} bytes'.format(init_m))
for _ in range(20):
    _G.value = npr.randn(nineq, nz)
    data, chain, inv_data = prob.get_problem_data('SCS')
    sol = scs.solve(
        data, cones, verbose=False,
        use_indirect=False, normalize=False,
        max_iters=10,
    )
    gc.collect()
    m = process.memory_info().rss
    print('  + {} bytes (+ {} bytes)'.format(m, m-init_m))
    init_m = m

...

print('\n\n--- scs with python cbs---')
init_m = process.memory_info().rss
gc.collect(); print('  + {} bytes'.format(init_m))
for _ in range(20):
    _G.value = npr.randn(nineq, nz)
    data, chain, inv_data = prob.get_problem_data('SCS')
    sol = scs.solve(
        data, cones, verbose=False, use_indirect=False,
        normalize=False, max_iters=10,
        linsys_cbs=(solve_lin_sys_cb,accum_by_a_cb,accum_by_atrans_cb)
    )
    gc.collect()
    m = process.memory_info().rss
    print('  + {} bytes (+ {} bytes)'.format(m, m-init_m))
    init_m = m

--- vanilla scs ---                                                  [9/1847]
  + 108982272 bytes
  + 112644096 bytes (+ 3661824 bytes)
  + 112693248 bytes (+ 49152 bytes)
  + 112750592 bytes (+ 57344 bytes)
  + 112812032 bytes (+ 61440 bytes)
  + 112861184 bytes (+ 49152 bytes)
  + 112910336 bytes (+ 49152 bytes)
  + 112943104 bytes (+ 32768 bytes)
  + 112967680 bytes (+ 24576 bytes)
  + 112975872 bytes (+ 8192 bytes)
  + 112979968 bytes (+ 4096 bytes)
  + 112984064 bytes (+ 4096 bytes)
  + 112984064 bytes (+ 0 bytes)
  + 112984064 bytes (+ 0 bytes)
  + 112988160 bytes (+ 4096 bytes)
  + 112996352 bytes (+ 8192 bytes)
  + 113025024 bytes (+ 28672 bytes)
  + 113037312 bytes (+ 12288 bytes)
  + 113037312 bytes (+ 0 bytes)
  + 113037312 bytes (+ 0 bytes)
  + 113037312 bytes (+ 0 bytes)

--- scs with python cbs---
  + 113299456 bytes
  + 113463296 bytes (+ 163840 bytes)
  + 113504256 bytes (+ 40960 bytes)
  + 113504256 bytes (+ 0 bytes)
  + 113524736 bytes (+ 20480 bytes)
  + 113541120 bytes (+ 16384 bytes)
  + 113573888 bytes (+ 32768 bytes)
  + 113577984 bytes (+ 4096 bytes)
  + 113577984 bytes (+ 0 bytes)
  + 113610752 bytes (+ 32768 bytes)
  + 113614848 bytes (+ 4096 bytes)
  + 113618944 bytes (+ 4096 bytes)
  + 113623040 bytes (+ 4096 bytes)
  + 113623040 bytes (+ 0 bytes)
  + 113627136 bytes (+ 4096 bytes)
  + 113631232 bytes (+ 4096 bytes)
  + 113635328 bytes (+ 4096 bytes)
  + 113639424 bytes (+ 4096 bytes)
  + 113639424 bytes (+ 0 bytes)
  + 113643520 bytes (+ 4096 bytes)
  + 113647616 bytes (+ 4096 bytes)

bodono · 2019-04-04T14:23:48Z

Cool, is this ready to be merged then?

bamos · 2019-04-04T14:28:44Z

Almost -- Python=3.6 on OSX is passing the tests for this PR now but the earlier versions of python on OSX aren't. I just tried changing the import_array call (which seems necessary somewhere in private.c for some reason, otherwise it will segfault)

bamos · 2019-04-04T15:05:03Z

Hmm, the OSX Python builds can now at least compile this without errors but are now failing with /Users/travis/.travis/functions: line 104: nosetests: command not found? https://travis-ci.org/bodono/scs-python/jobs/515741429

Seems weird since the master branch is currently running/passing https://travis-ci.org/bodono/scs-python/jobs/515100158

bamos · 2019-04-04T15:10:01Z

Hmm, in that travis build that's failing, it's setting PYTHON=3.4.7 but appears to be using pip with Python 2.

bamos · 2019-04-04T15:10:27Z

The pyenv install command is failing for some reason on the builds for this PR, but not for master.

bamos · 2019-04-04T17:27:29Z

Hmm, I've tried to fix the Python install part of the OSX travis build but it's still failing for reasons that appear to be beyond this PR

bodono · 2019-04-05T10:30:58Z

Yeah, the tests can be a real pain, I'll merge this in then and we can try and fix them later.

Initial commit of Python linsys solver.

173b32b

bamos mentioned this pull request Apr 3, 2019

Allow zero A matrices as a special case. cvxgrp/scs#110

Merged

Brandon Amos added 2 commits April 3, 2019 14:00

Add python_linsys test to travis

451b6ad

Disable test_scs_python_linsys test on windows

7e63cbc

bamos mentioned this pull request Apr 3, 2019

Re-using matrix factorization #1

Closed

Brandon Amos and others added 2 commits April 3, 2019 22:39

Remove use_python_linsys arg. Instead infer it from linsys_cbs

4079ebd

Update setup.py

f10f85a

Slight consistency update to setup.py

bodono reviewed Apr 4, 2019

View reviewed changes

Brandon Amos added 5 commits April 4, 2019 09:33

Update scs submodule to the latest.

1fe7295

Prefix get_{int,float}_type with scs

67aa92e

Re-enable windows tests.

594575a

Prefix callbacks.

3c24d2c

Merge remote-tracking branch 'upstream/master'

a22a032

bamos force-pushed the master branch from 79c730f to a22a032 Compare April 4, 2019 14:17

Try using _import_array

dcd6249

Brandon Amos added 3 commits April 4, 2019 11:19

Travis: Try installing openssl.

392ed6c

Travis: Try setting CFLAGS/LDFLAGS.

e7a720d

Travis: Add semicolons.

ae93e41

bodono merged commit 3f45183 into bodono:master Apr 5, 2019

bamos mentioned this pull request Apr 8, 2019

Python linsys: Add normalization #15

Merged

bamos mentioned this pull request Jul 15, 2019

[feature request] Implementing Block Sparse Operations pytorch/pytorch#9222

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python linsys solver. #11

Python linsys solver. #11

bamos commented Apr 3, 2019 •

edited

Loading

bamos commented Apr 3, 2019

bamos commented Apr 3, 2019

bodono left a comment

bodono Apr 4, 2019

bamos Apr 4, 2019

bodono Apr 4, 2019

bamos Apr 4, 2019

bodono Apr 4, 2019

bodono Apr 4, 2019

bamos Apr 4, 2019

bamos Apr 4, 2019 •

edited

Loading

bodono Apr 4, 2019

bodono commented Apr 4, 2019

bamos commented Apr 4, 2019

bodono commented Apr 4, 2019

bamos commented Apr 4, 2019 •

edited

Loading

bamos commented Apr 4, 2019

bamos commented Apr 4, 2019

bamos commented Apr 4, 2019 •

edited

Loading

bamos commented Apr 4, 2019

bodono commented Apr 5, 2019

Python linsys solver. #11

Python linsys solver. #11

Conversation

bamos commented Apr 3, 2019 • edited Loading

bamos commented Apr 3, 2019

bamos commented Apr 3, 2019

bodono left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bamos Apr 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bodono commented Apr 4, 2019

bamos commented Apr 4, 2019

bodono commented Apr 4, 2019

bamos commented Apr 4, 2019 • edited Loading

bamos commented Apr 4, 2019

bamos commented Apr 4, 2019

bamos commented Apr 4, 2019 • edited Loading

bamos commented Apr 4, 2019

bodono commented Apr 5, 2019

bamos commented Apr 3, 2019 •

edited

Loading

bamos Apr 4, 2019 •

edited

Loading

bamos commented Apr 4, 2019 •

edited

Loading

bamos commented Apr 4, 2019 •

edited

Loading