Website building process needs improvement

[jlstevens]

At the minimum, the dev/Tutorials should be automatically updated.

[jlstevens]

I know why everything is configured this way (we don't want to bloat the code repositories) but the process is fairly miserable even just for updating the content in /dev/Tutorials:

1. Stash whatever you are doing.
2. Checkout gh-pages.
3. Make sure to pull the latest version of gh-pages.
4. Run git submodule update --init (take a long time and downloads around 1.8GB!)
5. Checkout holoviews-tutorials then pull the submodule you want to update (typically dev/Tutorials)
6. Make the commit(s) needed to update the submodule reference(s).
7. Push to gh-pages.
8. Trigger a documentation build on buildbot.
9. Switch back to master and delete the Tutorials and dev/Tutorials directories.
10. Pop the stash.

Note that some of these steps are not required if you keep a copy of holoviews somewhere on your hard drive just for this purpose!

Unfortunately there are a lot of steps but I want them all automated (at least for dev/Tutorials).

Edit 1: Philipp tells me that fewer steps are required if you work on doozy directly. He'll explain more shortly...

Edit 2: Now I find out doozy won't push so all these steps are still insufficient. You apparently have to work via doozy.

[jlstevens]

I think we should consider a new approach - our website is huge due to the notebooks and the gh-pages branch is bloating our code repository. I would consider removing the gh-pages branch and keeping the website in an entirely separate repository.

As we have enough repositories, maybe we could have a holoviews-website subfolder on the master branch of the ioam.github.com repository?

[philippjfr]

I totally agree that now we a dev section of the homepage this process should be automated. The list you gave there isn't the currently the way it works however.

1. Make your changes
2. Tell buildbot to build the docs
3. Go to doozy then:

cd /var/lib/buildbot/slaves/holoviews_docs/build/doc/reference_data
git checkout holoviews-tutorials
git push
cd ../..
git checkout gh-pages
git submodule update --init
cd (dev/)Tutorials
git checkout holoviews-tutorials
git pull
cd ..
git add Tutorials
git commit -m "Updated (dev/)Tutorials submodule reference"
git push

This final step is what could be automated now that we have the dev folder on gh-pages.

[jlstevens]

Ok thanks! At least we now have an 'official' set of instructions for the current procedure...

What do you think about the idea of using a subfolder of ioam.github.com? I'm sure we could arrange the redirects to make the website links work the same way they do now...

Edit: For reference, this page on setting the CNAME might help...

[jlstevens]

Some notes to add to Philipp's instructions:

• You need to log in as user topo-buildbot on doozy.
• I had to git reset HEAD --hard to switch to the gh-pages branch for some reason.
• The submodule update step is still very slow!
• I had to use git remote set-url origin https://github.com/ioam/holoviews.git before the final git push.

As for moving the gh-pages branch somewhere else, this is the most useful page I've found so far

[jlstevens]

Ok. The tutorial links are finally fixed to point to the archive for 1.2.0!

This was much too much effort for something that should happen automatically nightly (again, for the dev version).

[jlstevens]

The website has been updated for 1.3.0 - once again, it has been a pretty painful process.

In addition to the automation, we should also think about keeping snapshots of the website (including the tutorials!) for each version. We could do this with tags, but you would need to tag both gh-pages in the holoviews repository as well as the holoviews-tutorials branch of the ioam.github.com repository (and some other branches as well).

Adding these git tags is easy but making a snapshot of the whole website is a bit more effort...

Edit: A single separate repository just for the website would make this much easier...

[jlstevens]

This issue has finally been addressed in PR #429 together with many other improvements including moving the websiteto S3, the travis.holoviews.org page and the buildbot.holoviews.org page.

Other than some polish, this work is now complete and the process of updating the website and test data is now much easier. For an example of one of the first PR's that was updated using the new system, see PR #436.

Closing the issue.

[jbednar]

Hooray!