automatically deploy development docs

[orbeckst]

Following Automatically Update Github Pages with Travis Example seems straighforward enough in order to have travis-ci build the sphinx docs and then push them to the gh-pages branch. (See also the discussion at the bottom of #350.)

Potential solutions

• Automatically Update Github Pages with Travis Example
• Auto-deploying built products to gh-pages with Travis
• awestruct gh deployment notes

Progress so far

• generated OAUTH token (limit scope to public_repo access; I used the tokens web interface but can also be done via curl as described in the awestruct gh deployment notes). DONE

• encrypt for travis with the travis client:

  gem2.0 install travis
  travis encrypt GH_TOKEN=`cat ./github_OAUTH_token`
  

  yields the entry for the .travis.yml file:

   secure: "HIj3p+p2PV8DBVg/KGUx6n83KwB0ASE5FwOn0SMB9zxnzAqe8sapwdBQdMdq0sXB7xT1spJqRxuxOMVEVn35BNLu7bxMLfa4287C8YXcomnvmv9xruxAsjsIewnNQ80vtPVbQddBPxa4jKbqgPby5QhhAP8KANAqYe44pIV70fY="
  

  (This is encrypted with the travis public key. For more details, see Travis CI: Encryption keys.)

• add lines to .travis.yml along the lines of

  script:
    - ./testsuite/MDAnalysisTests/mda_nosetests -v --with-coverage --cover-package MDAnalysis --processes=2 --process-timeout=120 --with-memleak
    - coveralls
    - test $TRAVIS_PULL_REQUEST == "false" && test $TRAVIS_BRANCH == "develop" && cd package/doc/sphinx && make clean html
  
  after_success:
    - test $TRAVIS_PULL_REQUEST == "false" && test $TRAVIS_BRANCH == "develop" && bash maintainer/deploy.sh
  
  env:
     global:
        secure: "HIj3p+p2PV8DBVg/KGUx6n83KwB0ASE5FwOn0SMB9zxnzAqe8sapwdBQdMdq0sXB7xT1spJqRxuxOMVEVn35BNLu7bxMLfa4287C8YXcomnvmv9xruxAsjsIewnNQ80vtPVbQddBPxa4jKbqgPby5QhhAP8KANAqYe44pIV70fY="

  (We want to build the developer docs when we push an update to the developer branch.)

  I don't know if an error during doc building will then also signal a failure of a build. Perhaps doc building should be put in a sub shell (e.g., - (cd package/doc/sphinx && make html); true) and always return success?

• Write a maintainer/deploy.sh script (or rake or whatever):

  • It will need to init a git repo in the package/doc/html dir and push this cleanly to the gh-pages branch. (This will also have the advantage that the docs will appear nicely as http://www.mdanalysis.org/mdanalysis/index.html )
  • As commit author we could use the fake Travis CI <[email protected]>.

• test on a feature branch without screwing up our repo...

[richardjgowers]

Looks like it didn't work?

https://travis-ci.org/MDAnalysis/mdanalysis/builds/79970890

/home/travis/build/MDAnalysis/mdanalysis/package/MDAnalysis/lib/transformations.py:1811: UserWarning: failed to import module _transformations

  warnings.warn("failed to import module " + module_name)

Exception occurred:

  File "/home/travis/build/MDAnalysis/mdanalysis/package/MDAnalysis/lib/distances.py", line 40, in <module>

    from ._distances import (calc_distance_array,

ImportError: No module named _distances

The full traceback has been saved in /tmp/sphinx-err-5NdJCZ.log, if you want to report the issue to the developers.

Please also report this if it was a user error, so that a better error message can be provided next time.

A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!

make: *** [html] Error 1

It looks like sphinx isn't seeing the Cython modules?

[kain88-de]

They are not installed locally in the package folder. You have to build them with

Python setup.py build_ext --inplace 

Then the shared object files will be where Sphinx and the Python interpreter can pick them up

[orbeckst]

I just pushed 80bc2a2 (directly to develop), which includes @kain88-de 's fix. It means that we're rebuilding the library a second time but that's hopefully not too bad. However, just failed with:

$ test $TRAVIS_PULL_REQUEST == "false" && test $TRAVIS_BRANCH == "develop" && bash ./maintainer/deploy_docs.sh

bash: ./maintainer/deploy_docs.sh: No such file or directory

I guess this means that the cd hopping is remembered, i.e. each travis.yml line is not executed in a sub shell. Will fix... but feel free to make this more robust when you look at it. (Sorry for cluttering the history...)

A bit more annoying is that as soon as we're running multiple CI workers, each one will deploy the docs as well. Maybe there's a way to mark one worker as special or perhaps Travis sets some environment variables that we could use?

[richardjgowers]

57eebe1#diff-354f30a63fb0907d4ad57269548329e3R27

WRT the last point, once we're running multiple jobs, we can nominate one as the "master" one that does the doc push (if TRAVIS_PYTHON == 2.7: push docs)

[orbeckst]

For multiple jobs, I found a solution at steveklabnik/automatically_update_github_pages_with_travis_example#5

I am currently trying out things on the test-docdeploy branch.

[kain88-de]

When the module are already build my fix will just cause a copy to the right place and not a full rebuild.