Use --bootstrap to wait for remote to become active

[jedevc]

Previously, the --bootstrap flag didn't do anything with the remote driver - now it should wait until the remote becomes active, with a small linear backoff.

This should help improve scenarios where the remote may not come up instantly if it has been started just before as was the case in the issue that caused #1094.

[crazy-max]

@jedevc Can we remove this step now since these changes:

buildx/.github/workflows/e2e.yml

Lines 150 to 163 in 43968ff

	-
	name: Check remote buildkitd
	if: matrix.driver == 'remote'
	run: |
	try=0
	max=10
	until [ "$(docker container inspect remote-buildkit --format '{{ .State.Health.Status }}')" = "healthy" ]; do
	if [ $try -gt $max ]; then
	echo >&2 "healthcheck failed after $max trials"
	exit 1
	fi
	sleep $(awk "BEGIN{print (100 + $try * 20) * 0.002}")
	try=$(expr $try + 1)
	done

?

[tonistiigi]

I think this should still return error and we should check for "network error" on the caller side.

[jedevc]

I agree about preserving the error to be able to display - I'm not sure about how much to change the caller side. Info gets called before Boot, which requires some fiddly logic changes in generic code to ignore network errors in that case.

I think there's two scenarios for errors here:

• Failure to connect to the endpoint providing the driver (can't connect to kubernetes, can't connect to docker daemon, etc) - this kind of failure means it's not worth trying to bootstrap, something is wrong.
• Failure to connect to the individual driver node (as in this case) - bootstrapping here is fine

Could we maybe rework status to be a struct and add an Err field to it? Then we could express a status of a node that's inactive because of an err. It currently feels like the error returned from here is quite tied to it's status, and is a little inconsistent across the drivers.