CI: Another macos FileWatching failure

[Keno]

Separate from #45982, we have another macos-specific FileWatching failure:

FileWatching                                      (7) \|         failed at 2022-07-26T18:18:33.436
  | Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-6.0/build/default-macmini-x64-6-0/julialang/julia-master/julia-ff1b563e3c/share/julia/stdlib/v1.9/FileWatching/test/runtests.jl:323
  | Expression: events.changed && (!(events.timedout) && !(events.renamed))

This looks a lot like #26725 in that we may be getting these events (or rather these events and the events from the previous loop) out-of-order. We should first determine if we should we expect the kernel to deliver these in order. If not, we should adjust the tests accordingly. If so, we should complain to Apple.

[Keno]

@vtjnash thoughts on what the expected behavior should be?

[Keno]

If I'm reading the kernel source correctly, it does look like events should show up in order: https://github.com/apple/darwin-xnu/blob/2ff845c2e033bd0ff64b5b6aa6063a1f8f65aa32/bsd/vfs/vfs_fsevents.c. Though there's a userspace fseventsd daemon in the middle that does who-knows-what.

[JeffBezanson]

Another one (https://buildkite.com/julialang/julia-master/builds/14598#01827ba7-bbb0-4ff8-a462-5475b793a4f9):

Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-5.0/build/default-macmini-x64-5-0/julialang/julia-master/julia-b91aeb331d/share/julia/stdlib/v1.9/FileWatching/test/runtests.jl:294
--
  | Expression: fname == "$(F_PATH)$(i)"
  | Evaluated: "afile.txt" == "afile.txt1"

[Keno]

I think the hypothesis that fseventsd is re-ordering events is pretty well validated at this point, and we'll just have to adjust the test or disable it on macos.

[gbaraldi]

maybe this is relevant dotnet/runtime#30415

[Keno]

Yeah, same issue. We can keep tracking this, but I think for CI purposes, we just need to turn off the test or make it robust to re-ordering. @gbaraldi want to take a crack at it?

[gbaraldi]

Sure, will see if I can get it to reproduce easier

[gbaraldi]

Recent failures

• x86
  https://buildkite.com/julialang/julia-master/builds/15023#0182d065-7b3e-463a-bb87-09c1ce9735ba

julia/stdlib/FileWatching/test/runtests.jl

Lines 189 to 191 in 36aab14

	@test c.first == F_PATH
	@test c.second.changed ⊻ c.second.renamed
	@test !c.second.timedout

• arm
  https://buildkite.com/julialang/julia-master/builds/14883#0182b509-4661-459c-a530-f4145638161e

julia/stdlib/FileWatching/test/runtests.jl

Line 35 in 36aab14

@test fd_in_limits

and https://buildkite.com/julialang/julia-master/builds/15032#0182d266-b194-4cf3-b0f4-c05eab8ac7e7

julia/stdlib/FileWatching/test/pidfile.jl

Line 292 in 36aab14

@test maximum(lock_times) > 2

This last one is interesting because it doesn't seem event related @Keno

[vtjnash]

#46497 didn't entirely fix this, but should have avoided the CI issues now

[wegank]

Recent failures

• x86
  https://buildkite.com/julialang/julia-master/builds/15023#0182d065-7b3e-463a-bb87-09c1ce9735ba

julia/stdlib/FileWatching/test/runtests.jl

Lines 189 to 191 in 36aab14

	@test c.first == F_PATH
	@test c.second.changed ⊻ c.second.renamed
	@test !c.second.timedout

• arm
  https://buildkite.com/julialang/julia-master/builds/14883#0182b509-4661-459c-a530-f4145638161e

julia/stdlib/FileWatching/test/runtests.jl

Line 35 in 36aab14

@test fd_in_limits

and https://buildkite.com/julialang/julia-master/builds/15032#0182d266-b194-4cf3-b0f4-c05eab8ac7e7

julia/stdlib/FileWatching/test/pidfile.jl

Line 292 in 36aab14

@test maximum(lock_times) > 2

This last one is interesting because it doesn't seem event related @Keno

Same failures on aarch64-darwin.

https://logs.nix.ci/?attempt_id=66e4325a-4d1f-40af-888f-fb276ed411d9&key=nixos%2Fnixpkgs.205584

Error in testset FileWatching:
Test Failed at /nix/store/v90b9l5gk8vc6l91f08zsdawa6l4lj69-julia-bin-1.8.3/share/julia/stdlib/v1.8/FileWatching/test/runtests.jl:187
  Expression: c.first == F_PATH
   Evaluated: "" == "afile.txt"
Error in testset FileWatching:
Test Failed at /nix/store/v90b9l5gk8vc6l91f08zsdawa6l4lj69-julia-bin-1.8.3/share/julia/stdlib/v1.8/FileWatching/test/runtests.jl:188
  Expression: c.second.changed ⊻ c.second.renamed
Error in testset FileWatching:
Test Failed at /nix/store/v90b9l5gk8vc6l91f08zsdawa6l4lj69-julia-bin-1.8.3/share/julia/stdlib/v1.8/FileWatching/test/runtests.jl:189
  Expression: !(c.second.timedout)
Error in testset FileWatching:
Test Failed at /nix/store/v90b9l5gk8vc6l91f08zsdawa6l4lj69-julia-bin-1.8.3/share/julia/stdlib/v1.8/FileWatching/test/runtests.jl:292
  Expression: fname == "$(F_PATH)$(i)"
   Evaluated: "afile.txt" == "afile.txt1"