add UID migration script for macOS Sequoia 15

[abathur]

Motivation

Unless Apple fixes the current behavior of the macOS Sequoia 15 beta updaters (which is ~taking UIDs 301-304 from our _nixbld1-4 users) before public beta, we'll likely get a lot of support requests from users with broken installs.

Unless it's feasible for Nix itself to automatically migrate them, it's probably worth having a migration script that people can manually run before or after their update.

Early iterations of this migrated to 331+, but I've more recently relocated to 350+ on suggestion from an Apple devrel.

Context

• macOS 15 Sequoia clobbers _nixbld1-4 users #10892
• I split this out of install-darwin: fix _nixbld uids for macOS sequoia #10919 on request so that one can potentially go a little faster once we settle on a UID range.

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[lloeki]

I suspect the fact that the update leaves some _nixbld users with no UID at all

It does.

may prevent it from working.

IIUC the script seems to migrate a set of existing users to a safe spot, but if like me you had a few users overwritten by Apple they need to be recreated.

In my specific case I had _nixbld1 to _nixbld4 missing, so this helped me get out of the hole I was in (I picked 401 as a starting point):

# check your _nixbld users for who's missing
dscl . list /Users UniqueID | grep _nixbld | sort -n -k2

# check what are the non-nix used ones, so that you can find a hole in there
dscl . list /Users UniqueID | grep -v _nixbld | sort -n -k2

# fill in the blanks, mine were 1 to 4:
for i in {1..4}; do
  sudo dscl . -create "/Users/_nixbld${i}" UniqueID $(( 400 + ${i} ))
  sudo dscl . -create "/Users/_nixbld${i}" PrimaryGroupID 30000
  sudo dscl . -create "/Users/_nixbld${i}" IsHidden 1
  sudo dscl . -create "/Users/_nixbld${i}" RealName "_nixbld${i}"
  sudo dscl . -create "/Users/_nixbld${i}" NFSHomeDirectory '/var/empty'
: sudo createhomedir -cu "_nixbld${i}"
  sudo dscl . -create "/Users/_nixbld${i}" UserShell /sbin/nologin
done

# that's just for having the survivors be consistent
for i in {5..32}; do
  id="$(dscl . -read "/Users/_nixbld${i}" UniqueID | cut -d' ' -f2)"
  sudo dscl . change "/Users/_nixbld${i}" UniqueID $id $(( 400 + ${i} ))
done

[abathur]

@lloeki Any chance you can include output from running those on your system? (It'll be really helpful to know what the first command outputs, and I'm also a little surprised "create" works on the users since as far as I can tell from other reports they do still exist, they just don't have a UID set anymore.)

[lloeki]

Before (recreated, but 200% sure it's exact because I stared at it multiple times over for quite a while):

dscl . list /Users UniqueID | grep _nixbld | sort -n -k2

_nixbld5                 305
_nixbld6                 306
_nixbld7                 307
_nixbld8                 308
_nixbld9                 309
_nixbld10                310
_nixbld11                311
_nixbld12                312
_nixbld13                313
_nixbld14                314
_nixbld15                315
_nixbld16                316
_nixbld17                317
_nixbld18                318
_nixbld19                319
_nixbld20                320
_nixbld21                321
_nixbld22                322
_nixbld23                323
_nixbld24                324
_nixbld25                325
_nixbld26                326
_nixbld27                327
_nixbld28                328
_nixbld29                329
_nixbld30                330
_nixbld31                331
_nixbld32                332

dscl . list /Users UniqueID | grep -v _nixbld | sort -n -k2

nobody                   -2
root                     0
daemon                   1
_uucp                    4
_taskgated               13
_networkd                24
_installassistant        25
_lp                      26
_postfix                 27
_scsd                    31
_ces                     32
_appstore                33
_mcxalr                  54
_appleevents             55
_geod                    56
_devdocs                 59
_sandbox                 60
_mdnsresponder           65
_ard                     67
_www                     70
_eppc                    71
_cvs                     72
_svn                     73
_mysql                   74
_sshd                    75
_qtss                    76
_cyrus                   77
_mailman                 78
_appserver               79
_clamav                  82
_amavisd                 83
_jabber                  84
_appowner                87
_windowserver            88
_spotlight               89
_tokend                  91
_securityagent           92
_calendar                93
_teamsserver             94
_update_sharing          95
_installer               96
_atsserver               97
_ftp                     98
_unknown                 99
_softwareupdate          200
_coreaudiod              202
_screensaver             203
_locationd               205
_trustevaluationagent    208
_timezone                210
_lda                     211
_cvmsroot                212
_usbmuxd                 213
_dovecot                 214
_dpaudio                 215
_postgres                216
_krbtgt                  217
_kadmin_admin            218
_kadmin_changepw         219
_devicemgr               220
_webauthserver           221
_netbios                 222
_warmd                   224
_dovenull                227
_netstatistics           228
_avbdeviced              229
_krb_krbtgt              230
_krb_kadmin              231
_krb_changepw            232
_krb_kerberos            233
_krb_anonymous           234
_assetcache              235
_coremediaiod            236
_launchservicesd         239
_iconservices            240
_distnote                241
_nsurlsessiond           242
_displaypolicyd          244
_astris                  245
_krbfast                 246
_gamecontrollerd         247
_mbsetupuser             248
_ondemand                249
_xserverdocs             251
_wwwproxy                252
_mobileasset             253
_findmydevice            254
_datadetectors           257
_captiveagent            258
_ctkd                    259
_applepay                260
_hidd                    261
_cmiodalassistants       262
_analyticsd              263
_fpsd                    265
_timed                   266
_nearbyd                 268
_reportmemoryexception   269
_driverkit               270
_diskimagesiod           271
_logd                    272
_appinstalld             273
_installcoordinationd    274
_demod                   275
_rmd                     277
_accessoryupdater        278
_knowledgegraphd         279
_coreml                  280
_sntpd                   281
_trustd                  282
_mmaintenanced           283
_darwindaemon            284
_notification_proxy      285
_avphidbridge            288
_biome                   289
_backgroundassets        291
_mobilegestalthelper     293
_audiomxd                294
_terminusd               295
_neuralengine            296
_eligibilityd            297
_systemstatusd           298
_aonsensed               300
_modelmanagerd           301
_reportsystemmemory      302
_swtransparencyd         303
_naturallanguaged        304
_oahd                    441
lloeki                   501

After:

dscl . list /Users UniqueID | grep _nixbld | sort -n -k2

_nixbld1                 401
_nixbld2                 402
_nixbld3                 403
_nixbld4                 404
_nixbld5                 405
_nixbld6                 406
_nixbld7                 407
_nixbld8                 408
_nixbld9                 409
_nixbld10                410
_nixbld11                411
_nixbld12                412
_nixbld13                413
_nixbld14                414
_nixbld15                415
_nixbld16                416
_nixbld17                417
_nixbld18                418
_nixbld19                419
_nixbld20                420
_nixbld21                421
_nixbld22                422
_nixbld23                423
_nixbld24                424
_nixbld25                425
_nixbld26                426
_nixbld27                427
_nixbld28                428
_nixbld29                429
_nixbld30                430
_nixbld31                431
_nixbld32                432

For the for loop commands, there was no output at all.

Note: the first dscl command I have in my shell history is a plain sudo dscl . -list /Users (without UniqueID), presumably it should have listed all users whether or not they have an id, but the missing ones were not to be seen. It's only afterwards seeing them missing that I found these posts and looked into how to create these users again.

[lloeki]

For the heck of it I ran this again, looks like create doesn't care if things exist:

$ sudo dscl . -create "/Users/_nixbld1" UniqueID 401
$ echo $?
0

And going deeper into the rabbit hole I'd say create is actually idempotent:

# let's get some baseline
$ dscl . read /Users/_nixbld1 
dsAttrTypeNative:accountPolicyData:
 <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>creationTime</key>
	<real>1720819125.8711572</real>
</dict>
</plist>

dsAttrTypeNative:IsHidden: 1
dsAttrTypeNative:record_daemon_version: 9040000
AppleMetaNodeLocation: /Local/Default
GeneratedUID: C7C57E37-BBB0-45E0-ADEA-21B4C0348991
NFSHomeDirectory: /var/empty
Password: ********
PrimaryGroupID: 30000
RealName: _nixbld1
RecordName: _nixbld1
RecordType: dsRecTypeStandard:Users
UniqueID: 401
UserShell: /sbin/nologin

# changing to id 400
$ sudo dscl . -create "/Users/_nixbld1" UniqueID 400         

# and the id changed but nothing else
$ dscl . read /Users/_nixbld1                       
dsAttrTypeNative:accountPolicyData:
 <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>creationTime</key>
	<real>1720819125.8711572</real>
</dict>
</plist>

dsAttrTypeNative:IsHidden: 1
dsAttrTypeNative:record_daemon_version: 9040000
AppleMetaNodeLocation: /Local/Default
GeneratedUID: C7C57E37-BBB0-45E0-ADEA-21B4C0348991
NFSHomeDirectory: /var/empty
Password: ********
PrimaryGroupID: 30000
RealName: _nixbld1
RecordName: _nixbld1
RecordType: dsRecTypeStandard:Users
UniqueID: 400
UserShell: /sbin/nologin

[dhess]

Thanks for this script. I wasn't relishing the idea of reinstalling Nix with a fixed installer before upgrading my Macs to Sequoia.

As I understand it, the Determinate Systems installer starts from uid 450 from Sequoia and later:

https://github.com/DeterminateSystems/nix-installer/blob/c5471f6dcb2853d6b297dd4249f209d55a37f424/src/settings.rs#L252

This is based on a comment from @ahcm:

#10892 (comment)

It would be nice to keep the two installers (and this script) in sync.

[abathur]

@dhess My understanding is that their use of 450+ is a sequoia-specific short term fix and that they intend to sync up with the decisions we make ~here.

(I intend to follow up w/ Apple to see if we can get any guidance on range. I was hoping to send that mon/tues this week but we're having a time in Houston, though that was probably net good since we have learned a little that I should include in the summary this week.)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-07-10-nix-team-meeting-minutes-160/49101/1

[abathur]

I force-pushed to (hopefully) fix the script for the uid-less users that people are seeing post-upgrade.

If anyone is in the position to take this update whether in a VM or on bare metal, it would be good to confirm it runs clean and does the right thing when run both pre-upgrade and post-upgrade.

[cole-h]

I just tested this in a VM, and unfortunately, the current approach doesn't work. Before I get into the issues, I'll first describe how I tested it:

• I installed UTM: https://mac.getutm.app/
• I created a new macOS 12+ VM and left the IPSW blank (to get the current latest macOS)
• I gave it 8192MB of memory and 128GB of disk (overkill, but 64GB was not enough to be able install the beta from an existing macOS 14.5 install -- the installer said I was ~2GiB short, so I doubled it :)
• I started up the VM, let the initial macOS install finish, and installed Nix via curl -L https://nixos.org/nix/install | sh
• In the background, I had downloaded the macOS 15 Beta 3 Update (build version 24A5289h) InstallAssistant.pkg from https://swcdn.apple.com/content/downloads/00/55/062-36085-A_SVM8I0RSF8/xqj8j50pwf49z53toqmdsej6od5u2s0kmw/InstallAssistant.pkg and made it available to the VM via a shared folder
• I ran the beta installer and waited a while for it to complete
• When I booted back in, of course macOS nuked the shell profiles, but after I manually sourced nix-daemon.sh, I was able to access my previously-installed hello just fine
• This is where the trouble occurs: dscl . read /Users/_nixbld1 through 4 no longer exist at all. _nixbld5 still exists just fine, however.
• Just to double-check that this is problematic, I downloaded the migration script: curl https://raw.githubusercontent.com/abathur/nix/macos_sequoia_proto_migration_script/scripts/sequoia-nixbld-user-migration.sh -O and ran it.

It bailed out at the very first iteration:

[abathur]

@cole-h Thanks for giving it a shake. Sigh. Not ideal :)

If they aren't there at all, we'll need full logic to create them correctly. I guess I could change heuristics again to try and figure out how to find the highest-numbered nixbld user and then go create/migrate all of them up to that number.

Here's where the assumption that they exist but lack a UID comes from:

#10948 (comment)

For grins, can you try the command I suggested there (edit: Nevermind. I see from Matrix that you did):

/usr/bin/dscl . -read "/Users/_nixbld1"; echo $?

Now that I review that issue, I suppose it isn't absolutely sworn there that Sequoia is the cause, so aside from the possibility that they've "fixed" this issue by actually fully deleting the users, it's also possible that's actually some Nth issue.

[cole-h]

@abathur Sorry for the delay, here's the output you asked for:

(To summarize the screenshot: I get a 56 exit code when looking up the nixbld1-that-no-longer-exists user)

[abathur]

Force-pushed to try to scaffold an approach that can backfill missing users and migrate the rest. Not generally tested on my end.

I did find at least one bug in the script as it existed while revising for this.

[mkenigs]

I followed a similar process as @cole-h, and I think it mostly worked. I think it ran into _nixbld users with IDs 331 and 332, so it started at 333. I'm guessing we could just delete those _nixbld users first and start at 331? Here's what I ended up with:

_nixbld1                 333
_nixbld2                 334
_nixbld3                 335
_nixbld4                 336
_nixbld5                 337
_nixbld6                 338
_nixbld7                 339
_nixbld8                 340
_nixbld9                 341
_nixbld10                342
_nixbld11                343
_nixbld12                344
_nixbld13                345
_nixbld14                346
_nixbld15                347
_nixbld16                348
_nixbld17                349
_nixbld18                350
_nixbld19                351
_nixbld20                352
_nixbld21                353
_nixbld22                354
_nixbld23                355
_nixbld24                356
_nixbld25                357
_nixbld26                358
_nixbld27                359
_nixbld28                360
_nixbld29                361
_nixbld30                362
_nixbld31                363
_nixbld32                364

[abathur]

@mkenigs Good catch. I think excluding _nixbld users from the ID availability check should be enough. Force-pushed.

[mkenigs]

@abathur here are the logs logs.txt

[tomberek]

@abathur this would be a fast way to check and bypass migration if not necessary.

diff --git a/scripts/sequoia-nixbld-user-migration.sh b/scripts/sequoia-nixbld-user-migration.sh
index 45a6f1bfc..858aaabeb 100755
--- a/scripts/sequoia-nixbld-user-migration.sh
+++ b/scripts/sequoia-nixbld-user-migration.sh
@@ -128,11 +128,33 @@ change_nixbld_uids(){
 		echo "dscl . list /Users UniqueID | grep _nixbld | sort -n -k2"
 	fi
 }
+needs_migration(){
+	local name uid next_id user_n
+
+	((next_id=NEW_NIX_FIRST_BUILD_UID))
+	((user_n=1))
+
+	while read -r name uid; do
+		expected_name="$(nix_user_n "$user_n")"
+		if [[ "$expected_name" != "$name" ]]; then
+			return 0
+		fi
+		if [[ "$next_id" != "$uid" ]]; then
+			return 0
+		fi
+		((next_id++))
+		((user_n++))
+	done < <(dscl . list /Users UniqueID | grep _nixbld | sort -n -k2)
+	return 1
+}
+
 
 if any_nixbld; then
-	echo "Attempting to migrate _nixbld users."
-	temporarily_move_existing_nixbld_uids
-	change_nixbld_uids
+	if needs_migration; then
+		echo "Attempting to migrate _nixbld users."
+		temporarily_move_existing_nixbld_uids
+		change_nixbld_uids
+	fi
 else
 	echo "Didn't find any _nixbld users. Perhaps you have single-user Nix?"
 	exit 1

Another check to be done is to ensure we have the correct number of builders.

[mkenigs]

@abathur worked for me with that most recent change!

[tomberek]

@abathur I'm inclined to have this run on all OSX installs + upgrade. Perhaps not unconditionally, but only if we detect UIDs in a dangerous range (either _nixbld or otherwise, both indicate potential problems). People without a conflict may have already moved them or are doing something custom.

[abathur]

@abathur this would be a fast way to check and bypass migration if not necessary.

I've applied this in the latest force-push.

Another check to be done is to ensure we have the correct number of builders.

Can you elaborate on what you have in mind? I'm not sure if you mean as a sanity/work-avoidance check similar to above or as a confirmation at the end, but one reason for the cumbersome implementation is that I'm trying to ~handle installs with a non-default number of users via --daemon-user-count N or export NIX_USER_COUNT=N. (As long as N was 5+.)

---

That said, this currently doesn't do anything ~smart for installs that overrode NIX_FIRST_BUILD_UID for a few reasons:

• Because that implies an opinionated decision about a UID range, I haven't been able to convince myself there's any correct action the script can take without making the user disambiguate. (Aside, perhaps, from bailing.)
• I'm hoping (lazily...) to avoid needing to accept interactive input or add invocation options and the help/documentation they imply.
• While supporting these installs would be nice if it's simple enough, I don't feel obliged since the installer doesn't have a flag for customizing it.
• The current approach assumes any missing users will be the first few, but fully supporting arbitrary custom UID ranges will entail coping with missing users at the middle and end.
• There aren't many overrides of this env visible in a GH code search. None of the search hits (at least as of this writing) use a UID range that would require more complex logic for coping with missing users in the middle or end of the range.

I had been thinking we might just message (everywhere we announce this migration) that these installs aren't supported and leave decisions about whether/how to migrate up to those users?

I have thought about bailing if the existing UIDs are unexpected, but this would make it harder to recommend the script to people who have been DIYing their first UIDs to work around this problem and get them back on ~normal UIDs.

Summing these up, I feel like:

• Users who DIY their UID range for other opinionated reasons (i.e., not working around sequoia) are likely rare, more likely to pay attention to what the script's going to do before running it, and equipped to reconcile anything they didn't like without being a support burden.
• The subset of these who chose a UID range that won't work with the heuristics here (i.e., lower than our existing 301 default, but still overlapping 301-304) will be even rarer.

[tomberek]

The situation I mean is that they somehow lost some users at the end of the range, either in uid or in names. We would not recreate them, or ly any lost in the beginning. This is not very likely, nor such a critical problem.

[emilazy]

I guess we should probably get #10919 finished soon if this is ready to go.

I missed getting an official response about numbering! Did the DevRel specify an opinion on 350 vs. 351? I think having an off‐by‐one from the Nix builder numbers, and from the numbering before the migration, is a bit unfortunate.

Edit: Ah, I see #10919 has been updated (although it still uses the old UID range).

[abathur]

I missed getting an official response about numbering! Did the DevRel specify an opinion on 350 vs. 351? I think having an off‐by‐one from the Nix builder numbers, and from the numbering before the migration, is a bit unfortunate.

We didn't put that specific 350/351 question to them.

Our reasoning for changing the open PRs to their recommendation during the installer meeting last week was roughly:

• They might be ever so slightly more likely to be helpful in the future if they eventually encroach on a number they suggested as reasonable.
• Cole also had some supporting evidence in that they've got a Mac Studio in service as a CI system and haven't felt any strong need to give it more than 32 build users.

I didn't mean to suggest that the UID choice was settled beyond debate by marking these ready for review--just that human testing had indicated that the logic might finally be ~right (minus some edge cases around installs that used custom first UIDs which I don't plan to address).

Not sure I have a super strong opinion on 350/351, but I agree it would be more consistent and if you think we're making a mistake here I don't see any reason not to center that question for debate in a followup PR as soon as #10919 lands?

Edit: Ah, I see #10919 has been updated (although it still uses the old UID range).

Good catch. I did update that PR with the new range for the actual installer, but I forgot to also apply it in the old bigsur migration script (which unfortunately is the first file GH shows :). I've force-pushed to fix that.

[emilazy]

Not sure I have a super strong opinion on 350/351, but I agree it would be more consistent and if you think we're making a mistake here I don't see any reason not to center that question for debate in a followup PR as soon as #10919 lands?

Yeah, sure, not trying to bikeshed all this even more, and I’m very glad Apple gave us a recommendation :) I prefer 351 because it preserves last digit of _nixbld* = last digit of UID, and GID not overlapping with any UID, but… it’s marginal, of course. I’m just happy to see this resolved.

Sorry for never getting around to doing extensive VM testing here; I had too much on the stack. I really appreciate you carrying this awkward migration through despite my nitpicking, and please do let me know if I can do anything to amplify the message about migration when Sequoia is released.

[abathur]

No worries. I introduced the consistency issue by not thinking a step beyond their suggestion.

I think we should try to settle the UID before we start putting effort into communicating, but the script should work whether you've taken the macOS update or not--so we can start messaging as soon as we're confident rather than waiting for Apple to release it.

It may also be helpful to start towards ~temporarily documenting the problem with notes/warnings anywhere we think people may land when flailing around with a broken install? I know I need to update the open pinned issue to describe how to run the script (and maybe reference missing _nixbld1 in the title), but it seems like a fair number of people don't look for open issues before trying to reinstall.

[emilazy]

We should at least backport the installer changes all the way back to 2.18, I think (there’s NixOS/nixpkgs#335342, but 24.05 is still a thing and will presumably default to 2.18 for its lifetime).

Unfortunately I’m not really sure where we can reliably message stuff other than the places we run code on user machines. Certainly we should put it in the release notes, but nobody reads those. A Discourse post would be a good idea, but I was planning to just spend a lot of time sending rote replies on Matrix mostly :)

FWIW I’m happy to open a PR to bump the UIDs by 1 myself if you think it sounds reasonable (and then I guess we should wait for the Determinate Systems installer to deploy the same UID/GID values before we start messaging?), though since the installer PR hasn’t landed yet maybe it should be adjusted there first. No desire to bikeshed the value beyond that because a recommendation from Apple makes me very happy!

[tomberek]

@abathur Thoughts on migrating the GID as well in this script?

[abathur]

Thoughts on migrating the GID as well in this script?

We discussed during the installer wg meeting I think.

I had tentatively added this at one point because I generally like minimizing differences across the install base, but someone made a good point about the risk of introducing file ownership issues (i.e., the group is the GID and not symbolic) on existing installs.

[edolstra]

I guess this will need to be backported?

[emilazy]

I expect most users will get this script via curl | bash, so I don’t think a backport is strictly required. That’s how the issue pointed to by the various docs admonitions and the nix-darwin error offer it, at least. The manual changes could probably use it.

[emilazy]

(What would be really good to have backported into a release is #11415, so that the Nix version used by Nixpkgs 24.05 and (for now) 24.11-pre can be installed correctly on macOS Sequoia. Currently nix-darwin has to do ugly CI hacks to work around that.)

[abathur]

I guess this will need to be backported?

I expect most users will get this script via curl | bash, so I don’t think a backport is strictly required. That’s how the issue pointed to by the various docs admonitions and the nix-darwin error offer it, at least. The manual changes could probably use it.

Agree with Emily that the migration script does not need to be backported for now. (Just the actual installer fixes.)