bug: macos sonoma hang during startup

[ctamietto]

I have installed quickget cloning from github

I downloaded sonoma with quickget with the command
quickget macos sonoma
see the video
https://youtu.be/07JLZCy5zlo

then I installed sonoma with the command
quickemu --vm macos-sonoma.conf
see the vidoes of the begin of installation end the end
start : https://youtu.be/MpbozxjQACw
end : https://youtu.be/Z5H0FT6Karo

it seem to me that almost at the end of the installation ( second startup ) the vm rebooted
so probably the installation in not complete ( you can view that in https://youtu.be/Z5H0FT6Karo )

when I try to boot i get a blank screen ( after i have chosen the disk of installation ) and after sometime it restart
see the video
https://youtu.be/MpbozxjQACw

my system is fedora 39

the log is void

what can i check ?
P.S: i have tried do install also big-sur monterey ventura with the same result

only the installation of catalina is working ( but is too old for what i have to do )

thank you
for any response

[TuxVinyards]

It will do that. There's a some stuff about it in the qqX wiki. And a link to walk through for Ventura. Pretty much the same ...

[ctamietto]

i have read and applied the suggestions of this thread

#937

but nothing changed

in this thread you talk about

"The Ventura VM that I had was built using the new, under-development, version."

is this new version already present in quickemu 4.9.3 ( i'm using that ) o is to be released in the future ?

Anyway thanks for your help

[TuxVinyards]

Ciao Claudio.

I have two installs of Sonoma already. You need patience to install them but my memory is that they were much the same as with Ventura.

I was running a VM of Fedora 40 last night and tried to install Sonoma on that. It threw up complaints about graphics memory, so I just put that down to it being a VM. You can normally 'telescope' VM's though. ie install machines inside other machines. It's normally not a problem.

I have just tried installing Sonoma, again, a third copy, on my main machine. I use Ubuntu Mate on that one. It didn't work as well as I had expected. Didn't get the graphics memory error though. Something to do with virtio on that one probably.

I am starting to question the Mac distro which is getting downloaded. It doesn't have the 'installer' icon like you can see in the screenshots in #937 and as mentioned in the Quickemu instructions. I initially put this down to an Apple renaming exercise, but I am not sure that this is entirely the case.

I also tried starting Sonoma from the command line, by copying quickemu out of the 'freebird' builtins to the VM folder and using ./quickemu -vm macos-sonoma.conf but got the same results. Note the dot at the front ...

At this point in time, FreeBird is identical to the current Quickemu 'master' and it runs my two previously installed Sonama distros without fault.

@lj3954 did 95% of the work on adding in Sonoma. I am wondering if he might have some ideas?

[ctamietto]

Ciao Alex ,
do you think that changing distro ( for example using ubuntu 22.04 ) the installation of mac os vm can work ?
i have found this video on you tube https://www.youtube.com/watch?v=wykyreQxVV8
where you can see a succesfull installation of ventura using a linux mint host ( i think based on ubuntu 22.04 )
using https://github.com/kholia/OSX-KVM ( quickemu use OVMF firmware from that project )
i'm guessing to install ubuntu 22.04 on a host and try to create the mac os vm on that system
what do you think about ?

[TuxVinyards]

Might be worth trying an install of Ventura. See what happens there. Might give us some clues.

[TuxVinyards]

I think this is distro. The Ventura front end has changed as well.

This how it used to look:

[lj3954]

@lj3954 did 95% of the work on adding in Sonoma. I am wondering if he might have some ideas?

I did very little work inside quickemu. I added sonoma to a couple of switch cases and blocked macOS guests from using some core counts they just won't boot with. Ventura and Sonoma should be identical in nearly every way. Have you been able to install a macOS guest after you modified the CPU argument?

I am starting to question the Mac distro which is getting downloaded.

The image is downloaded in exactly the same way that macrecovery does it, and as far as I can tell, exactly the same way an Apple computer would. Also, the image is verified using a chunklist file which also contains Apple's EFI key. There is no doubt in my mind that the image is legitimate.

I'll check this out tonight. I haven't done much testing with macOS in quickemu since I finished my PR on it.

[TuxVinyards]

I'll check this out tonight. I haven't done much testing with macOS in quickemu since I finished my PR on it.

Appreciated 👍

[lj3954]

I can't get the (sonoma) installer to even boot with your new CPU arg, nor can I get my preexisting VM to boot. Ventura kernel panics with it. Whether it works correctly on Intel CPUs or not doesn't matter, silencing a couple of irrelevant errors in the console is not worth whatever regression this has caused.

After reverting the CPU to Haswell, the installation was painless. I just selected the 2nd option "macOS base system" on the first boot, and the 3rd option on reboots. It was quite surprising to me how painless it was.

This doesn't solve the issue, though, since macOS releases prior to Ventura will use the Penryn CPU instead. I'm currently downloading Monterey to test whether this works.

[lj3954]

@ctamietto Please provide the output of lscpu. It appears to me that macOS guests work very differently depending on the host's CPU, more data points would be helpful.

[lj3954]

I completed the install of macOS Monterey with no changes to the CPU. Once again, there was no issues. Different systems clearly have very different outcomes, much more data is needed to find a solution.

[TuxVinyards]

After reverting the CPU to Haswell, the installation was painless

I had wondered whether the Haswell line was actually correct when it added sse_4.2 ?

The original code only grepped for 4.1 and the notes in the code said 4.1.

What happens if you change the new CPU line to 4.2 on your Ryzens?

For me, on Intel, Sonoma runs faultlessly.

[lj3954]

No dice. That emulated broadwell CPU just doesn't work here

[cyrileek]

I tried to install ventura and sonoma en Debian Trixie. It hang or reboot and go back to the "EFI" and "macOS Base System" menu. I used to have the ventura working on this Debian with the quickemu previous version. I agree to the fact that the installation process is not the same as the startup menu is different.

[TuxVinyards]

No dice. That emulated broadwell CPU just doesn't work here

So, I just changed the new CPU instruction to 4.2 and it all works, for me.

And the old one. But it's plain wrong to grep for one thing and to add another.

The question now is how do fix the CPU line so that it works for everyone, Ryzen or Intel.

[TuxVinyards]

@lj3954 Try this hybrid line. The old one with Haswell but tidied up.

CPU="-cpu Haswell-noTSX-IBRS,kvm=on,vendor=GenuineIntel,+sse3,+sse4.2,+aes,+xsave,+avx,+xsaveopt,+xsavec,+xgetbv1,+avx2,+bmi2,+smep,+bmi1,+fma,+movbe,+invtsc"

[lj3954]

That works for me.

[TuxVinyards]

Yay. I'll push that to a PR, if that's okay. 🥳

[ctamietto]

this is the output requested

I have tried again to install big-sur monterey ventura sonoma but nothing works

something is changed , practically all the images hang on first boot ( after selecting the base system )

[TuxVinyards]

@ctamietto try qqX 1.8.03 that I have just posted 🤞

From the main https://github.com/TuxVinyards/qqX/archive/refs/heads/main.zip

[TuxVinyards]

SUCCESS 🥳

            case ${macos_release} in
                ventura|sonoma)
                    if check_cpu_flag sse4_2 && check_cpu_flag avx2; then
                        CPU="-cpu Haswell-v2,kvm=on,vendor=GenuineIntel,+avx,+avx2,+sse,+sse2,+ssse3,+sse4.1,+sse4.2,+rdtscp"
                        #CPU="-cpu Skylake-Server-v3,kvm=on,vendor=GenuineIntel,+avx,+avx2,+sse,+sse2,+ssse3,+sse4.1,+sse4.2,+rdtscp"
                        #if check_cpu_flag pdpe1gb ; then CPU+=",+pdpe1gb" ; fi
                        #if check_cpu_flag smap ; then CPU+=",+smap" ; fi
                        #if check_cpu_flag clwb ; then CPU+=",+clwb" ; fi
                        #if check_cpu_flag adx ; then CPU+=",+adx" ; fi
                    else
                        echo "ERROR! macOS ${macos_release} requires a CPU with SSE 4.2 and AVX2 support."
                        exit 1
                    fi ;;
                catalina|big-sur|monterey)
                    if check_cpu_flag sse4_2; then
                        CPU="-cpu Haswell-v2,kvm=on,vendor=GenuineIntel,+avx,+sse,+sse2,+ssse3,+sse4.1,+sse4.2,+rdtscp"
                    else
                        echo "ERROR! macOS ${macos_release} requires a CPU with SSE 4.2 support."
                        exit 1
                    fi ;;
                *)
                    if check_cpu_flag sse4_1; then
                        CPU="-cpu Penryn,kvm=on,vendor=GenuineIntel,+sse,+sse2,+ssse3,+sse4.1"
                    else
                        echo "ERROR! macOS ${macos_release} requires a CPU with SSE 4.1 support."
                        exit 1
                    fi ;;
            esac

            local FLAG=""

            if [[ $CPU == *'Penryn'* ]]; then
                for FLAG in tsc vme fxsr mmx clflush pse36 pat cmov mca pge mtrr sep apic cx8 \
                            mce pae msr pse de fp87 cx16 lm nx syscall ; do
                    if check_cpu_flag "${FLAG}"; then CPU+=",+${FLAG}" ; fi
                done
                if check_cpu_flag lahf_lm ; then CPU+=",+lahf-lm" ; fi
            else
                if ! check_cpu_flag fma || ! check_cpu_flag rdtscp; then
                    echo "ERROR! macOS requires a CPU with FMA and TSC support."
                    exit 1
                fi
                for FLAG in vmx abm aes bmi1 bmi2 cx8 eist f16c fma \
                            mmx movbe mpx popcnt smep vaes vbmi2 vpclmulqdq \
                            fxsr clflush pse36 pat cmov mca pge mtrr sep apic cx8 \
                            mce pae msr tsc pse de fp87 x2apic cx16 \
                            pcid f16c rdrand lm nx syscall fsgsbase erms invpcid arat \
                            xgetbv1 xsave xsaveopt ; do
                    if check_cpu_flag "${FLAG}"; then CPU+=",+${FLAG}" ; fi
                done
                if check_cpu_flag tsc_adjust ; then CPU+=",+tsc-adjust" ; fi
                if check_cpu_flag lahf_lm ; then CPU+=",+lahf-lm" ; fi
                if check_cpu_flag amd_ssbd ; then CPU+=",+amd-ssbd" ; fi
            fi

Tested with FULL complete installs of Sonoma, Catalina and Mojave. Full setup with login and preferences set. Shutdown and re-login as working.

Much faster than previously with 4.9.2 through 4.9.4 etc. Fully working, no hangs, no stalls.

Of new, now requires the installer icon to be pressed on the restarts. This now happens a couple of times. But this all flows smoothly.

CAVEAT: not yet tested on Ryzen. Volunteers please ...

Additionally note the Skylake version is also working. This is commented out but has been tested on Sonoma. cpu_cores="8" should be placed in the .conf before running.

A conf selector for skylake should be considered, or at least noted and archived in case of future problems.

Also see notes and comments in #1214

[lj3954]

A couple of CPU flags are named differently in QEMU, beware. vbmi2, for example, is avx512vbmi2

[lj3954]

Note that PCID appears to be enabled by default with this cpu arg. CPUs which don't support this legacy flag, including the latest Ryzen CPUs, will get warnings. I've included disabling PCID on CPUs which don't support it in my rust quickemu.

[TuxVinyards]

Note that PCID appears to be enabled by default with this cpu arg. CPUs which don't support this legacy flag, including the latest Ryzen CPUs, will get warnings. I've included disabling PCID on CPUs which don't support it in my rust quickemu.

@lj3954 Thank you for that.

I wasn't fully sure about the inclusion of if check_cpu_flag amd_ssbd ; then CPU+=",+amd-ssbd" ; fi but Qemu docs say this makes things run faster and it follows along in the line of removal of IBRS via haswell-v2, which was key to getting things working for me.

@flexiondotorg you are missing the point of the one-liners. As @lj3954 has also pointed out, some of the flags are need special treatment. This is why I hesitated to write a loop in my initial work. I hadn't determined what was required at that stage. lscpu output sse4_1 needs translating to sse4.1 for Qemu, for example. But also

                if check_cpu_flag tsc_adjust ; then CPU+=",+tsc-adjust" ; fi
                if check_cpu_flag lahf_lm ; then CPU+=",+lahf-lm" ; fi

need the the underscores translating to hyphens.

Edit: also needed for the underscore in the amd_ssbd flag, so this flag won't have worked when it was:

" Tested full installs of Mojave, Catalina, Big Sur, Monterey, Ventura and Sonoma on AMD Ryzen 5950X. "

unfortunately.

[TuxVinyards]

The following flags are not found in Haswell

                        #if check_cpu_flag pdpe1gb ; then CPU+=",+pdpe1gb" ; fi
                        #if check_cpu_flag smap ; then CPU+=",+smap" ; fi
                        #if check_cpu_flag clwb ; then CPU+=",+clwb" ; fi
                        #if check_cpu_flag adx ; then CPU+=",+adx" ; fi

Which is why they were treated separately.

Edit: The commit has `pdpe1gb' with a minus but probably could just be omitted, to avoid confusion

smap and clwb are not present but adx is.

[TuxVinyards]

CPU="-cpu ${CPU_MODEL},vendor=GenuineIntel,-pdpe1gb,+avx,+sse,+sse2,+sse3,vmware-cpuid-freq=on"

Again sse3 is not a valid qemu parameter:

 qemu-system-x86_64 -cpu help | grep -A 100 flags | tr ' ' '\n' | grep sse | cat -s 

misalignsse
sse
sse2
sse4.1
sse4.2
sse4a
ssse3

[lj3954]

Again sse3 is not a valid qemu parameter:

QEMU appears to refer to sse3 as pni. For many of these flags, though, I think it's likely they're enabled by default when you specify Haswell or whatever other architecture. SSE3, for example, was introduced in 2004. Surely anything expecting a Haswell CPU would require it.

[TuxVinyards]

Again sse3 is not a valid qemu parameter:

QEMU appears to refer to sse3 as pni. For many of these flags, though, I think it's likely they're enabled by default when you specify Haswell or whatever other architecture. SSE3, for example, was introduced in 2004. Surely anything expecting a Haswell CPU would require it.

It works the other way round. ssse3 includes sse3 Try lscpu | grep sse or :

lscpu | grep -i flags | tr ' ' '\n' | grep sse

sse
sse2
ssse3
sse4_1
sse4_2

[TuxVinyards]

A couple of CPU flags are named differently in QEMU, beware. vbmi2, for example, is avx512vbmi2

Well spotted. Not sure where that came from.

lscpu | grep -i flags | tr ' ' '\n' | grep bmi
bmi1
bmi2
avx512vbmi
avx512_vbmi2

qemu-system-x86_64 -cpu help | grep -A 100 flags | tr ' ' '\n' | grep bmi | cat -s
avx512vbmi
avx512vbmi2
bmi1
bmi2

[flexiondotorg]

I've incorporated the feedback in my feature branch, macos-host and created a new function, configure_cpu_flag(), which converts the appropriate flag to QEMU format.

What I have implemented does deviate slightly from the discussion because it also works for running macOS VMs on macOS hosts, and therefore handles macOS CPU flag conversions too.

If the host CPU is "GenuineIntel" then quickemu passes the host CPU model features, model, stepping, exactly to the guest. This is what the broader macOS on QEMU community recommends. I've tested this on Linux with Intel Xeon E-2176M and on macOS with Intel Core i7-4770HQ. If the host is not "GenuineIntel" then configure_cpu_flag() does its thing, including outright disabling pcid on "AuthenticAMD". I've tested this in Linux with Ryzen 5950X and 5900X using VMs of macOS Mojave, Catalina, Big Sur, Monterey, Ventura and Sonoma. All works well.

Fully disabling huge pages (,-pdpe1gb) is absolutely required on macOS to prevent crashes on every attempted boot:

• https://stackoverflow.com/questions/60231203/qemu-qcow2-mmu-gva-to-gpa-crash-in-mac-os-x

Therefore, because it is now possible to run a macOS VM that, in turn, can run a VM, huge pages are always disabled on macOS hosts and guests.

[TuxVinyards]

Wow

[TuxVinyards]

@flexiondotorg A couple of things:

I am finding that -cpu host is making my CPU churn badly.

If I edit the following line to read as "xxGenuineIntel" and force AMD/Haswell, things run a lot smoother.

macos)
            # If the host has an Intel CPU, passes the host CPU model features, model, stepping, exactly to the guest.
            # Disable huge pages (,-pdpe1gb) on macOS to prevent crashes
            # - https://stackoverflow.com/questions/60231203/qemu-qcow2-mmu-gva-to-gpa-crash-in-mac-os-x
            if [ "${HOST_CPU_VENDOR}" == "xxGenuineIntel" ] && [ -z "${HYPERVISOR}" ]; then
                CPU_MODEL="host"

2. There were a few overlooked flags that weren't quite right. I did make a comment but I think you had had enough at that point and these got missed. They are not critical but they shouldn't be there ....

#1225 (comment)

I also tried a PR but the bot kept blocking things and saying that I hadn't made any changes. Maybe because no lines were added. That's another issue to look at possibly.

[flexiondotorg]

Please provide some performance metrics for running a VM with host and Haswell so we have an objective comparison.

[TuxVinyards]

Every time I click on a GUI icon in -cpu host mode the physical Host monitors show the cores maxing out. Doesn't happen with Haswell.

On my 3 year old mid-range Intel, the objective times are as follows:

1. Time from clicking the Mac start disk icon that shows after initial boot, until the logon screen is presented:
   Haswell mode 15 secs Host mode 21s

2. Time to complete a 10k repeated bc calculation:

time for i in {1..10000} ; do  bc <<< "scale=10; $i*22/7" ; done

Physical Host: 32 secs and Sonoma/Haswell 28s but Sonoma/Host 3min 10s (7x slower)

[TuxVinyards]

Also, from 4.9.5 code:

# A CPU with invtsc is required for macOS to boot

Not true:

[TuxVinyards]

Just noticed flag "eist" which is not Haswell.

I think a few things crept in from the Tech PowerUp links that I referenced, before I started using Qemu source instead.

https://www.techpowerup.com/cpu-specs/xeon-w-2140b.c2953

From the transitional work above #1114 (comment)

[TuxVinyards]

@flexiondotorg @lj3954

Something NEW

TuxVinyards/qqX@79d2886

Subtractive CPU methods. Totally the reverse of Nick Sherlock's and Kholia's methods that have been used in Quickemu in the past.

Outperforms the additive methods at runtime. 26 secs for the bc test. Very smooth installation with Ventura but having a few problems with the installer on Sonoma that need sorting, if anyone has ideas ....

on tests, both haswell & skylake -ve perform well
but skylake3 seems to manage IBRS (spectre-mitigation)
and is still faster than Haswell +ve i=on timed tests
use cpu_cores="8" in .conf

Use "sysctl -a | grep machdep.cpu" at terminal in MacOS to see flags