Enable nested virtualization

[felipecrs]

I don't know which is the hypervisor used to spin the VMs, but enabling nested virtualization would be a great enhancement. For example, this way we could build Vagrant boxes in GitHub Actions.

[felipecrs]

This feature is enabled by default in Travis CI. So I wonder why not here?

[dalehamel]

It looks like this is likely because azure only supports nested virtualization on dv3 or above https://azure.microsoft.com/en-ca/blog/nested-virtualization-in-azure/ but azure runs on ds2 instances: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/virtual-environments-for-github-hosted-runners#cloud-hosts-for-github-hosted-runners

This seems to be older than dv3 from the catalogue: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-general#dsv2-series vs https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-general#dv3-series-1

I haven't checked the exact CPU features, but i wonder if Hyper-V powering azure needs to work with a specific CPU family / featureset?

Has anyone tried just running KVM on azure? What happens? Is there even support for it in the stock azure / ubuntu kernel for bionic?

[dalehamel]

My use-case is I'd like to run an android emulator on Github Actions btw, like with https://github.com/google/android-emulator-container-scripts

[dalehamel]

When I run:

$egrep -c '(vmx|svm)' /proc/cpuinfo
0

This indicates there isn't CPU support, at least not passed through to the guest.

I installed kvm packages and was able to modprobe kvm, but there is no /dev/kvm, which is what I would expect.

Unless Azure adds support for this, or github actions switches / offers other machine classes, I don't think nested virtualization can be done.

[felipecrs]

Travis supports not only nested virtualization but also xvfb, that I guess you also need to run your Android tests.

https://github.com/felipecassiors/ubuntu1804-4dev/blob/master/.travis.yml

The support for nested virtualization is out-of-the-box there, where I use to build a Vagrant box using VirtualBox.

[AkihiroSuda]

https://github.com/actions/virtual-environments/blob/master/images/macos/macos-10.15-Readme.md

Intel x86 Emulator Accelerator (HAXM installer) | 7.5.1

macOS instances seem to support nested virtualization?
(I couldn't install VirtualBox though: #429)

[hannes]

+1, I would also really like to see this. I wonder what went wrong, if I manually spin up an equivalent Azure VM, it has vmx set and VirtualBox works fine with hardware acceleration.

[felipecrs]

Seems like it is possible indeed: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/nested-virtualization

[dalehamel]

Yes azure supports it but not the instance type GitHub actions runs on

[hannes]

Works fine on Travis ^^

[AjkayAlan]

+1 - Building vagrant boxes using packer is not currently possible with GitHub Actions, unless we self-host a runner. Same with Azure DevOps (which appears to use the same hosted images as GitHub Actions).

So, if this got prioritized, you would maybe solve this on both platforms.

[dalehamel]

I suspect that with self-hosted runners on azure you could circumvent this issue, as jobs could be dispatched to a host that supports nested virtualization.

The current worker that actions runs on in their free offering just doesn’t have a processor that supports this.

I suspect that if this feature is ever offered, it will be when a compatible processor is bumped into this low-tier instance class. Until then, self-hosted runners could be feasible but only really for corporate users.

[njsmith]

I wouldn't mind a self-hosted runner, but I need nested virtualization for a public project, and those aren't really usable with self-hosted runners: https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories

[hannes]

I wouldn't mind a self-hosted runner, but I need nested virtualization for a public project, and those aren't really usable with self-hosted runners: https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories

Yes, exactly, that's also the reason we can't use them.

[mvgijssel]

Would also love this to be able to use packer to build images.

[Andrei-Pozolotin]

+1

[alepauly]

Unfortunately we can't support this feature at this time, that said the feedback here is valuable and we will definitely keep it in mind . Thanks all.

[sbates130272]

I am another +1 on this but I do understand that GitHub prefers to run its runners (sic) on hardware that might not support nested virtualization out of the box. Of course anyone can do a self-hosted runner on hardware that does support nested-virtualization and do it that way. Any input from tyhe GitHub team on when their runner hardware will support nested?

[mattjohnsonpint]

Of course anyone can do a self-hosted runner on hardware that does support nested-virtualization and do it that way.

@sbates130272 - Unfortunately that's not quite true, because self-hosted runners are a security risk for public repos.
https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security

[nebuk89]

on the larger runners if you add yourself to the KVM user group on Linux, you should be good to go.

            sudo adduser $USER kvm
            sudo chown -R $USER /dev/kvm

Though this is not formally supported yet as we are still sorting the bits for Windows to work, though that should come in the next few weeks along with a change log documenting this as in support for runners >4 cores.

If anyone has any feedback on this or concerns or has issues I would <3 to hear.

[mrk-han]

on the larger runners if you add yourself to the KVM user group on Linux, you should be good to go.

            sudo adduser $USER kvm
            sudo chown -R $USER /dev/kvm

Though this is not formally supported yet as we are still sorting the bits for Windows to work, though that should come in the next few weeks along with a change log documenting this as in support for runners >4 cores.

If anyone has any feedback on this or concerns or has issues I would <3 to hear.

Thank you so much! Will experiment with this and get back to you. This is very exciting.

[mattjohnsonpint]

... for runners >4 cores.

Did you mean >= 4 cores ? Or will we need an 8 core runner minimum?

[maxkratz]

on the larger runners if you add yourself to the KVM user group on Linux, you should be good to go.

I first misread this and tried it on the normal (base) runners.

[...]
jobs:
  vagrant-provision:
    runs-on: ubuntu-22.04
    steps:
    - uses: actions/checkout@v3
    - name: set up KVM
      run: |
        sudo adduser $USER kvm
        sudo chown -R $USER /dev/kvm
[...]

resulting in:

Adding user `runner' to group `kvm' ...
Adding user runner to group kvm
Done.
chown: cannot access '/dev/kvm': No such file or directory
Error: Process completed with exit code 1.

Are there any plans on supporting KVM on the "normal" runner VMs @nebuk89?

[Gustl22]

Now Flutter removes OpenGL support in 3.7.0 which was testable through GH-CI software rendering at least on macOS. How developers are supposed to test their GUI applications in GH actions?

[nebuk89]

@mattjohnsonpint I did mean >= ! So 4-core will work.

@maxkratz not right now :( this is a test for us to see how people use it and to ensure our ability to detect bad actors abusing Actions when we have nested virt as another tool in the mix.

@Gustl22 a good question, @Steve-Glass is doing research into the future of what we offer on the Mac runners specifically :) I suspect we don't have an answer quite yet as 3.7 has only been out a week or so. It could be worth breaking that out into a separate issue?

[Gustl22]

@nebuk89 You mean a separate issue for GPU support on MacOS (maybe with Apple Silicon / M1) or for the specific Flutter 3.7 issue? I think it all comes down to put in some effort for GPU support in general from GH side. This is on the table for far too long and other CI providers provide much better conditions for GUI testing on all platforms. Idk, if a separation from this issue will solve the general problem.

Edit: seems like there's a light at the end of the tunnel by supporting Linux and Windows nested virtualisation, let's hope it is usable on the free plan as well some day :)
And yes, you're right, sorry: nested virtualization and GPU are two different things, although they are strongly connected as stated here.

[maxkratz]

@maxkratz not right now :( this is a test for us to see how people use it and to ensure our ability to detect bad actors abusing Actions when we have nested virt as another tool in the mix.

Thank you for the clarification.

[Steve-Glass]

👋 @Gustl22 With regard to your prior comment:

@nebuk89 You mean a separate issue for GPU support on MacOS (maybe with github/roadmap#528) or for the specific Flutter 3.7 issue?

I would be great if a separate issue is filed for GPU support on macOS. Having customer submitted issues for this feature helps with establishing the priority of upcoming investigations. You can tag me in it as well 😄

[gsauthof]

@nebuk89 wrote:

sudo adduser $USER kvm
sudo chown -R $USER /dev/kvm

Hm, this is suboptimal:

• On Ubuntu, /dev/kvm is owned by group kvm - thus, adding your user to that group is a valid strategy for using it as a normal user - iff you are able to re-login with that user - which is usually cumbersome in a CI environment.
• The second part is a hack to work-around the re-login requirement. First, it includes the -R recursive option for no reason. Second, this change might be reversed by udev. I guess one can live with such a unstable hack to some extend, but then you don't have to add your user to the kvm group, in the first place - as well as calling chown recursively.

The better alternative is to widen the permission via udev, since it doesn't require a re-login nor such a hack.

See my Stackoverflow answer for details.

[alexellis]

Hey folks, hope you don't mind us chiming in here.

actuated supports launching VMs within CI, so long as Linux + KVM are all you need.

Launching Firecracker in a GitHub Action: https://twitter.com/welteki/status/1625487015650410496?s=20

Running a NixOS build in a GitHub Action https://twitter.com/welteki/status/1625419864029466624?s=20

Nested virtualisation extensions are required on the host, so this only works on x86_64 for now.

[nebuk89]

https://github.blog/changelog/2023-02-23-hardware-accelerated-android-virtualization-on-actions-windows-and-linux-larger-hosted-runners/ << announced as live

and thank you @gsauthof for the recommendation on how to better approach the KVM group :) it works and is what I have in the change log (I hope it was ok to credit you there!)

[alexellis]

That's so cool @nebuk89, congrats on shipping a much requested feature!