Most problems deploying/upgrading BinderHub can be solved by checking indentation

[sgibson91]

Hello!

So 9 times out of 10 when my BinderHub is acting weird, it's because I've indented something incorrectly in config.yaml. There was a (brief) discussion in #835 around trying to implement some kind of linting check to catch these types of mistakes, but there isn't a concrete plan yet.

For the time being, I suggest updating the Debugging BinderHub docs to advise users to thoroughly check their config files before diving into k8s debugging.

Are there any thoughts or further ideas around this?

[minrk]

This is a general problem with helm, silently ignoring misspelled or mis-structured config instead of warning about it. It's feasible to handle certain common mistakes like verifying the list of top-level keys in values.yaml, but general linting is going to be tough without a dedicated command that's not run on helm upgrade, which is of significantly reduced utility.

[sgibson91]

Agreed. I guess what I want to do is raise awareness that this is probably the most likely cause of a problem and to check that before deep-diving into any k8s stuff.

This might be an ok resource to link to. It doesn't seem perfect, but better than nothing. I might play with it over the next week or so and see how I get along with it.

[minrk]

Yeah. It would be really awesome if helm could record which items in values.yaml were visited and then show you what's been ignored. That might be pretty hard, though.

helm lint is useful, but I think it doesn't help with the ignored-config case. It does help catch things like producing invalid templates. If you are exploring it, maybe check out the jupyterhub lint script, which we use to run yamllint, helm lint, and kubeval to test the validity of the chart.

[minrk]

Ah, what might be useful for ignored config is helm diff, which shows you what would change. If you think you're changing something and the diff is empty, that's probably a good clue!

[sgibson91]

Thanks, those look like great resources!

[betatim]

I think adding some text to the documentation (or even just linking to this issue) would already help get people unstuck. Should we do that while we investigate more automatic/tooling based approaches?

Another thing to do is add a paragraph to the docs about the difference between using the BinderHub chart (in which case you (probably) have a config.yaml) and the case where you make a chart that uses the BinderHub chart (a la https://github.com/jupyterhub/mybinder.org-deploy/). Because now you need an extra level of indentation.

I think our docs are consistent in using the config.yaml approach but people might find snippets around the internet or look at the mybinder.org config files and then end up stumped.

[sgibson91]

Another thing to do is add a paragraph to the docs about the difference between using the BinderHub chart (in which case you (probably) have a config.yaml) and the case where you make a chart that uses the BinderHub chart (a la https://github.com/jupyterhub/mybinder.org-deploy/). Because now you need an extra level of indentation.

The third flavour being you have a BinderHub config.yaml file and you are customising it using the JupyterHub guide. In which case, everything you see on those pages will need to go under the jupyterhub key in config.yaml. But it's not correct to change those docs as they are correct for people deploying a JupyterHub only.

[sgibson91]

@minrk just gave a great answer here that could be written up

[minrk]

Yeah, docs on common gotchas (e.g. extra indent) and troubleshooting (helm diff) would be a great start.

[sgibson91]

I'm having a headache with connecting an Azure container registry to my BinderHub. So if I don't have any breakthroughs with that, I can start a PR for this 😄

[consideRatio]

@sgibson91 ❤️ 🎉 you are great! :D

[sgibson91]

PR open at #847 🎉