Trouble executing DragoNN

[hussius]

Thanks for making this package available! However, I am having some trouble running it on an academic computing cluster where I have an account.

First, I tried to install it through python setup.py install. That appeared to work, but then I ran into the error below when trying to execute:

b-an01 [~/dragonn]$ dragonn
Traceback (most recent call last):
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/bin/dragonn", line 11, in <module>
    load_entry_point('dragonn==0.1.3', 'console_scripts', 'dragonn')()
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 565, in load_entry_point
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 2598, in load_entry_point
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 2258, in load
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 2264, in resolve
  File "build/bdist.linux-x86_64/egg/dragonn/__main__.py", line 5, in <module>
  File "build/bdist.linux-x86_64/egg/dragonn/utils.py", line 6, in <module>
  File "build/bdist.linux-x86_64/egg/simdna/__init__.py", line 3, in <module>
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 1203, in resource_filename
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 1716, in get_resource_filename
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 1737, in _extract_resource
KeyError: 'simdna/resources/HOCOMOCOv10_HUMAN_mono_homer_format_0.001.motif.gz'

On the other hand, using the Anaconda installer by conda install dragonn -c kundajelab, I run into this when executing:

(conda_env_dragonn) b-an01 [~/dragonn]$ dragonn
Traceback (most recent call last):
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/envs/conda_env_dragonn/bin/dragonn", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/envs/conda_env_dragonn/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 2985, in <module>
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/envs/conda_env_dragonn/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 2971, in _call_aside
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/envs/conda_env_dragonn/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 2998, in _initialize_master_working_set
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/envs/conda_env_dragonn/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 662, in _build_master
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/envs/conda_env_dragonn/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 675, in _build_from_requirements
  File "/pfs/nobackup/home/m/mikaelhu/anaconda2/envs/conda_env_dragonn/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 854, in resolve
pkg_resources.DistributionNotFound: The 'pydot_ng==1.0.0' distribution was not found and is required by dragonn

I used a Python 2.7 virtual environment because it seems DragoNN has extra functionalities with 2.7 as opposed to 3.x.

[jisraeli]

@hussius: Thanks for reporting this issue. I updated the simdna tag which should fix the missing package data error you encountered with python setup.py install, I updated the travis script to run the dragonn binary and it seems to work now.

Can you pull from master, install with python setup.py install and run dragonn again?

The missing pydot dependency error with the conda install suggests that I left it out from the latest conda package. I will take a closer look later today.

[hussius]

@jisraeli Thanks for the reply. I now have a running version of dragonn installed via Anaconda (I installed pydot-ng separately), but the one installed via python setup.py install still gives me the same error. However, I have been working with two virtual environments (conda and virtualenv) so it's possible that I have tangled things up somehow. I did make an effort to start from scratch with a new virtual environment, git pull and python setup.py install but that did not change things.

[agitter]

I'll create a new issue if this is unrelated, but I also had problems installing with conda install dragonn -c kundajelab. I tried a linux-64 and win-64 system, both with Python 3.5.2.

linux-64 made it seem like there is not support for Python 3.5:

Fetching package metadata ...........
Solving package specifications: .
UnsatisfiableError: The following specifications were found to be in conflict:
  - dragonn -> deeplift 0.2 -> keras ==0.3.2 -> python 2.7*|3.4*
  - dragonn -> deeplift 0.2|==0.3 -> python 2.7*
  - dragonn -> keras ==0.3.2 -> python 2.7*|3.4*
  - dragonn -> pydot-ng -> python 2.7*|3.4*
  - dragonn -> python 2.7*
  - python 3.5*
Use "conda info <package>" to see the dependencies for each package.

win-64 may have had problems installing shapely:

Fetching package metadata ...........
Solving package specifications: .
PackageNotFoundError: Package not found: '' Dependencies missing in current win-64 channels:
  - dragonn -> keras ==0.3.2
  - dragonn -> shapely
  - dragonn -> deeplift 0.2
You can search for packages on anaconda.org with
    anaconda search -t conda shapely
(and similarly for the other packages)

I've been able to install shapely for Windows from the scitools channel https://anaconda.org/scitools/shapely

I'm working on installing from source and will add an update if anything interesting comes from that. I'll also try a Python 2.7 environment with the conda install, which seemed to work for @hussius.

[jisraeli]

@agitter thank you for highlighting these issues. I think a python 3.5 build test in travis CI would be useful for catching these errors. I may be able to work through these issues next week, I would be happy to review a PR with travis test with python 3.5 so please feel free to contribute.

On a separate note, we are in the process of forming a DragoNN organization which will offer this repo for workshops/tutorials and a more advanced repo that would be more appropriate for research. The more advanced repo will be designed based on patterns that have worked well for our lab, greater flexibility, and smaller set of dependencies. I look forward to making it public soon.

[agitter]

@jisraeli I made a branch on my fork that runs a Travis CI test in Python 2.7 and 3.5 in parallel by creating different conda environments: https://github.com/agitter/dragonn/tree/travis

The install worked in 2.7.13 and 3.5.3. However, the dragonn unit tests failed in 3.5.3: https://travis-ci.org/agitter/dragonn/jobs/211924331

I don't think I'll have time to dig in and see why the tests failed. Should I make a pull request to merge this into a new Travis CI branch so that you can check out the Python 3.5 tests when you have time?

The DragoNN organization sounds great. I'm currently interested in the simpler version actually. I'd like to use this in my bioinformatics class, which is why I want to see how hard it will be for students to install.

[jisraeli]

@agitter I looked briefly over the unit test failures, it looks like something I could fix sometime next week. Let's go ahead and open a PR with this. Thanks!

[agitter]

For anyone else who had trouble installing and found this issue, I was able to install from source in Python 2.7 on my linux-64 server. I followed the install steps used in .travis.yml, but there were problems with pydot-ng. Uninstalling pydot-ng and reinstalling with pip install pydot-ng==1.0.0 before installing dragonn worked.

[dtrudg]

We also see the pydot-ng issue. For us the full working procedure to install is...

conda create -n dragonn python
source activate dragonn
conda install -c kundajelab dragonn

conda remove pydot-ng
pip install pydot-ng==1.0.0
git clone [email protected]:kundajelab/dragonn.git
cd dragonn
python setup.py install

# Then it runs
dragonn train --pos-sequences examples/example_pos_sequences.fa --neg-sequences examples/example_neg_sequences.fa --prefix training_example

[jisraeli]

@dctrud: Thanks for posting your fix. Which operating system did you use?

I'm on vacation this week, will look through the pydot dependency in the conda package next week.

[dtrudg]

@jisraeli - sorry for the delay. We are on Red Hat Enterprise Linux 6.8 in this case.