Update Dockerfiles and CI

[ruffsl]

Spinning out non-dumenten related CI changes from #1731 into own seperate PR.

• Use build kit for persistent caching across docker builds
• Clean up dummy files for better scripting
• Simple ergonomic improvements and file clarity

[peci1]

I just saw the ROSCon lightning talk and wanted to have a look at the implementation. Great job digging through all the Docker docs and tutorials and distilling them into a thing like this ;) It's definitely an inspiration. I just wondered if I'm the only one who's lifting his eyebrows in front of the huge inefficiency of most Docker builds. I'm lucky I'm not alone =)

I also used the mount=type=cache technique in our Dockerfiles, and I found out it's also good to cache /root/.cache/pip because rosdep usually pulls some pip dependencies.

I'm not sure if it's really needed, but at some point I added rw to the cache definitions, like

--mount=type=cache,target=/root/.cache/pip,rw

I was also experimenting with Git caching, but one has to be careful there to not break when people force-push or do some other silly stuff. My first idea was to cache bare git repos and utilize them using something like git clone --reference, but I couldn't find a way to integrate that with vcstool (or, frankly speaking, wstool we're still using). I ended up running wstool in a mount=type=cache directory and copying the files to workspace. I think that works as long as wstool/vcstool is able to update the cached state of repos to the new desired state. But I'm pretty sure there are cases where this would break.

[SteveMacenski]

It might be good for you to join our Slack group and talk to Ruffin or the rest of the team there ;-)

[peci1]

Not another Slack group, please :-D But if you feel these ideas could be leveraged, feel free to open an issue in this repo and invite me in. Or maybe rather in some tooling repo?

[ruffsl]

I'm not sure if it's really needed, but at some point I added rw to the cache definitions, like

That looks new. What nuance effects does being explicit about the rewrite permissions to the cash volume have? By default, or when permissions are omitted, rewrite permissions are used, no? I may need to fix that then.

[peci1]

Ah, my bad, rw is for bind mounts. cache mounts have ro instead (so I assume rw is the default). https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md#run---mounttypecache . When I was setting up our dockerfile, I had the impression that rw prevented some cache busts, but I can't tell for sure.