Internal build issue: PR builds intermittently fail with no space left on device

[ryandawsonuk]

Seen for example at #1298 (comment)
and #1315 (comment)

Happens when you run multiple builds in quick succession. If you delete the old build jobs then the next one works. Seems to be the ephemeral storage that is being used for logging. Istio logs quite a lot and we log a lot anyway in our E2E tests, which we run multiple times within kind for different platforms.

Looks like a fairly common one to hit with JX anyway - jenkins-x/jx#3243 and jenkins-x/jx#5622

[ryandawsonuk]

Could also be that https://github.com/SeldonIO/seldon-core/blob/master/jenkins-x-integration.yml#L49 means the images are being stored on ephemeral storage

[ryandawsonuk]

I was listing and deleting completed pipeline jobs one by one but possibly jx gc would do it. Didn't actually try that before.

[ryandawsonuk]

Also hitting this on #1316 Happening even after cleaning up old jobs.

[ryandawsonuk]

This seems to have worked as tests passed on #1323. Didn't do any cleanup prior to that so the changes in that PR seem to be what has fixed it.

Re cleanup, we may occasionally want to run jx gc and also jx get build pods followed by kubectl delete pod <pod> for any completed ones. Unfortunately the jx gc command alone doesn't do this. Old pods do get automatically cleaned up but typically not fast enough.

[ryandawsonuk]

Separate issue - the gh-pages branch seems to somehow be tracking master.

[axsaucedo]

@ryandawsonuk should we close this issue?