Skip to content

Latest commit

 

History

History
109 lines (73 loc) · 4.34 KB

build.org

File metadata and controls

109 lines (73 loc) · 4.34 KB
Raw

PyPI

Packaging and uploading source files

To package the source files, run:

python setup.py sdist

To upload to pypi, do something like the following (but replacing with the appropriate version):

twine upload dist/momi-2.1.19.tar.gz

Building and uploading binary wheels

We no longer build binary wheels for pypi to avoid the complexity of building for many platforms (linux, osx). Also, it’s not straightforward to make a generic build for linux (see [1]).

Nevertheless, I record the steps for building a binary wheel here:

python setup.py bdist_wheel # build binary dist
twine upload dist/*

[1] https://stackoverflow.com/questions/59451069/binary-wheel-cant-be-uploaded-on-pypi-using-twine

conda

conda-build command

To build for conda, run the following from a separate conda environment with conda-build installed:

conda build -c conda-forge -c bioconda conda/

meta.yaml

conda-build-3 syntax

Ideally, we could use the conda-build=3 syntax in meta.yaml, like so:

requirements:
  build:
    - {{ compiler('c') }}
  host:
    - python
    - etc...

However, when I tried this, conda-build used the wrong C compiler, causing the following errors on Linux:

gcc: error: unrecognized command line option ‘-fstack-protector-strong’
gcc: error: unrecognized command line option ‘-fno-plt’
error: command 'gcc' failed with exit status 1

Apparently, this is a conda-forge issue, which these links sort-of explain: conda/conda-build#2523 (comment) https://conda-forge.readthedocs.io/en/latest/meta.html#compilers

There should be a way to use the conda-build-3 syntax and build with conda-forge, but I wasn’t able to figure it out. Instead I found 2 possible workarounds:

  1. Prioritize the defaults channel above conda-forge; then we use the correct C compilers from the defaults channel. (We still need conda-forge because of the msprime dependency).
  2. Don’t use the conda-build-3 syntax, just stick with the conda-build-2 syntax.

It generally isn’t recommended to mix defaults and conda-forge as in Option 1 (for example, Bioconda recommends having priorities be conda-forge, then bioconda, then defaults).

So, we stick with the conda-build-2 syntax. The main advantage of the conda-build-3 syntax has to do with cross-compiling but we’re not using that, and the conda-build-2 syntax is a lot easier to understand and better documented.

When we look at this again, the following link might help us to get the conda-forge C compilers working: conda-forge/python-feedstock#203 (comment)

This reference is also useful for explaining on conda defaults build things: https://conda.io/docs/user-guide/tasks/build-packages/compiler-tools.html

OSX and openmp

To build for OSX we need openmp as both a build and a run dependency. We also have clangdev as a build dependency, it’s probably required, but I haven’t tested this extensively.

Previously I used llvm-openmp, which is in the defaults channel. This no longer worked after I switched to using conda-build-2 syntax and setting conda-forge as the highest priority channel.

Reference: xgboost

I found it helpful to look at the conda-forge package for xgboost, as it also requires OpenMP. Here is an older version of their meta.yaml that uses the conda-build-2 syntax:

https://github.com/conda-forge/xgboost-feedstock/blob/cbd3078becfe5bf8f55c35144cfd0080353cca96/recipe/meta.yaml conda-forge/xgboost-feedstock#4

Note the latest version of the conda-forge xgboost recipe uses the conda-build-3 syntax, and also uses llvm-openmp dependency – but I couldn’t get this working when I tried it out, as noted above. Here is the latest version of their meta.yaml as of this writing:

https://github.com/conda-forge/xgboost-feedstock/blob/f06d4542163cc3fe62670bad9a56ddb3a06b68b8/recipe/meta.yaml

openblas

We list openblas as a runtime dependency, otherwise msprime will cause the following error:

from _msprime import FORWARD  # NOQA
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

The downside of this is that installing openblas may disable MKL.