Move optional sage optimization backends (COIN, CPLEX, Gurobi) to separate Cython packages to remove OptionalExtension problems

[mkoeppe]

sage.numerical.backends depends on very little from Sage. We propose to split out the optional backends (COIN, CPLEX, Gurobi) to separate Cython packages.

With this ticket, sagelib no longer has the optional dependency on cbc. Instead, these new packages depend on the sage package (i.e., sagelib).

This makes it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this will also simplify packaging binary distributions.

The new packages are on PyPI:

wget -P upstream https://files.pythonhosted.org/packages/29/8a/65fd90a890fcfb56166c4071aa7a4fb5b865d313304f60474870f0e89998/sage_numerical_backends_cplex-9.0b12.tar.gz
wget -P upstream https://files.pythonhosted.org/packages/2c/5c/e1532bb6cde28cf86ebffd408ccb500d7b846bc29bbd1c5e479a2f18acb6/sage_numerical_backends_gurobi-9.0b12.tar.gz
wget -P upstream https://files.pythonhosted.org/packages/17/f4/308d3e151d82daf2c58ca26bce157fac31f763b2b60b5173243c09291e0a/sage_numerical_backends_coin-9.0b12.tar.gz

Development here:

• https://github.com/mkoeppe/sage-numerical-backends-cplex
• https://github.com/mkoeppe/sage-numerical-backends-coin
• https://github.com/mkoeppe/sage-numerical-backends-gurobi

See also:

• Follow-up: Move sage optimization backend framework (sage.numerical.backends) to separate Cython packages #28920: Move sage optimization backend framework (sage.numerical.backends) to separate Cython packages
• Meta-ticket: Use Python optimization interfaces: CVXPY, SCIP, or-tools, PuLP, Pyomo, cylp... #26511 - Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp...
• https://groups.google.com/forum/#!topic/sage-packaging/_FLDkqD2X8c

CC: @isuruf @saraedum @kiwifb @dimpase @sagetrac-tmonteil @videlec @seblabbe

Component: numerical

Author: Matthias Koeppe

Branch: 1279897

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/28175

[dcoudert]

comment:1

Can you be more precise on your plans.

These backends are used in graphs. So what will happen if linear programming backends are in a separate package ?

[mkoeppe]

comment:2

Replying to @dcoudert:

These backends are used in graphs. So what will happen if linear programming backends are in a separate package ?

In my plan, the backends would be a standard package of the sage distribution, and the cython modules it provides would be automatically imported when MixedIntegerLinearProgram is used.

[kiwifb]

comment:5

For glpk it looks like https://github.com/biosustain/swiglpk/releases is the best maintained. Excluding sage and cvxopt.

[mkoeppe]

comment:8

I have prepared a first module, https://github.com/mkoeppe/sage-numerical-backends-coin.
It depends on Sage (as it inherits from sage.numerical.backends.generic_backend.GenericBackend).

[dimpase]

comment:10

Is there a git branch to try, replacing Sage's current COIN-OR backend with this one?

[mkoeppe]

comment:11

Not yet (working on it right now), but the 3 backends already pass their test suites separately.

[mkoeppe]

Branch: u/mkoeppe/move_optional_sage_optimization_backends_to_separate_cython_packages_to_remove_optionalextension_problems

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

437fec5

Fixup dependencies

[sagetrac-git]

Commit: 437fec5

[mkoeppe]

comment:14

Now the packages can be installed using sage -i -c... please test

[dimpase]

comment:15

remains of copypaste, I see PyNormaliz in the branch...

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

846ca9a

Replace CoinBackend, CPLEXBackend, GurobiBackend by their versions imported from sage_numerical_backends_*

[sagetrac-git]

Changed commit from 437fec5 to 846ca9a

[mkoeppe]

Author: Matthias Koeppe

[sagetrac-git]

Changed commit from 846ca9a to 3c557b4

[sagetrac-git]

Changed commit from 53f3774 to 7caa365

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

7caa365

Update sage_numerical_backends_gurobi and sage_numerical_backends_coin to 9.0b12

[mkoeppe]

comment:51

There's a new version of the gurobi package now. I have done some testing on Linux now, but I could not do a full test with a Gurobi license because it does not like to generate license keys on a Docker container. I have revised thee installation instructions to include Linux:
https://github.com/mkoeppe/sage-numerical-backends-gurobi/blob/master/README.md

[mkoeppe]

comment:52

Replying to @dimpase:

Even on a Mac, with a multi-user installation one might want to hardcode GUROBI_HOME and the license file location (GRB_LICENSE_FILE).

Sure. The package looks at GUROBI_HOME first.

[sagetrac-git]

Changed commit from 7caa365 to 2110ed3

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

2110ed3

Update sage_numerical_backends_cplex to 9.0b12

[mkoeppe]

comment:55

Just updated the cplex backend.
Ready for review.

[isuruf]

comment:56

I suggest using a namespace package for this. See https://packaging.python.org/guides/packaging-namespace-packages/

diff --git a/setup.py b/setup.py
index 176e75e..6dcffe2 100755
--- a/setup.py
+++ b/setup.py
@@ -44,8 +44,8 @@ cbc_library_dirs = cbc_pc['library_dirs']
 cbc_include_dirs = cbc_pc['include_dirs']
 
 
-ext_modules = [Extension('sage_numerical_backends_coin.coin_backend',
-                         sources=[os.path.join('sage_numerical_backends_coin',
+ext_modules = [Extension('sage.numerical.backends.coin_backend',
+                         sources=[os.path.join('sage/numerical/backends',
                                     'coin_backend.pyx')],
                          libraries=cbc_libs,
                          include_dirs=sage_include_directories() + cbc_include_dirs,
@@ -81,7 +81,7 @@ except CompileError:
 print("Using compile_time_env: {}".format(compile_time_env), file=sys.stderr)
 
 setup(
-    name="sage_numerical_backends_coin",
+    name="sage.numerical.backends.coin_backend",
     version=readfile("VERSION").strip(),
     description="COIN-OR backend for Sage MixedIntegerLinearProgram",
     long_description = readfile("README.md"), # get the long description from the README
@@ -109,9 +109,8 @@ setup(
                             compile_time_env=compile_time_env),
     cmdclass = {'test': SageTest}, # adding a special setup command for tests
     keywords=['milp', 'linear-programming', 'optimization'],
-    packages=['sage_numerical_backends_coin'],
-    package_dir={'sage_numerical_backends_coin': 'sage_numerical_backends_coin'},
-    package_data={'sage_numerical_backends_coin': ['*.pxd']},
+    packages=['sage.numerical.backends'],
+    package_data={'sage.numerical.backends': ['*.pxd']},
     install_requires = [# 'sage>=8',    ### On too many distributions, sage is actually not known as a pip package
                         'sphinx'],
     setup_requires   = ['pkgconfig'],

[mkoeppe]

comment:57

Replying to @isuruf:

I suggest using a namespace package for this. See https://packaging.python.org/guides/packaging-namespace-packages/

Thanks a lot! I was briefly looking into this before I created the new packages, but was discouraged by various warnings on this page that seemed to imply one needs control of all packages sharing a namespace simultaneously. But I want the packages to work also with versions of sagelib that are already released.
Which of the three approaches mentioned on the page do you specifically recommend?

[isuruf]

comment:58

The diff above is for "1. native namespace package".

But I want the packages to work also with versions of sagelib that are already released.

This should work in the case of sagelib already released and if the optional extension is already installed, it will be overriden I think.

[mkoeppe]

comment:59

Replying to @isuruf:

The diff above is for "1. native namespace package".

Thanks. This seems to be for Python 3 only, which would conflict with the goal of my packages to support older released versions such as sage 8.1 on Ubuntu bionic. But I'll keep it in mind for the future.

[mkoeppe]

comment:60

Thanks for the pull request at sagemath/sage-numerical-backends-coin#4 . Let's continue the discussion regarding namespace packages there.

[sagetrac-git]

Changed commit from 2110ed3 to 1279897

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

1279897

build/pkgs/sage_numerical_backends_*/dependencies: Add some .pxd and .h as dependencies

[mkoeppe]

comment:62

Replying to @mkoeppe:

Thanks for the pull request at sagemath/sage-numerical-backends-coin#4 . Let's continue the discussion regarding namespace packages there.

I consider the idea of using namespace packages beyond the scope of the present ticket. I have created ticket #28925 (Modify clean_stale_files to support modularization of sagelib by namespace packages) for this direction.

[dimpase]

comment:64

OK, coin and gurobi work, I can't test cplex, but I think it should be OK.
It won't make it into 9.0, so I bumped up the milestone.

[dimpase]

Reviewer: Dima Pasechnik

[mkoeppe]

comment:66

Thank you!

[vbraun]

Changed branch from u/mkoeppe/move_optional_sage_optimization_backends_to_separate_cython_packages_to_remove_optionalextension_problems to 1279897

[seblabbe]

comment:68

Sorry, I was late testing this. It works for me with Ubuntu 16.04 running Gurobi 8.1.1 and CPLEX 12.9.

Question: how can I make sure that the tests involing Gurobi and cplex backends continue to work in the next dev versions of SageMath now that those files are gone from the sage library?

[seblabbe]

Changed commit from 1279897 to none

[mkoeppe]

comment:69

Replying to @seblabbe:

It works for me with Ubuntu 16.04 running Gurobi 8.1.1 and CPLEX 12.9.

Thanks for testing!

Question: how can I make sure that the tests involing Gurobi and cplex backends continue to work in the next dev versions of SageMath now that those files are gone from the sage library?

The tests are run when you install the new packages with sage -i -c.