Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JIT CI abort on aarch64-darwin in LinearAlgebra/addmul uv_mutex_lock #47254

Closed
vtjnash opened this issue Oct 20, 2022 · 3 comments · Fixed by #47419
Closed

JIT CI abort on aarch64-darwin in LinearAlgebra/addmul uv_mutex_lock #47254

vtjnash opened this issue Oct 20, 2022 · 3 comments · Fixed by #47419
Labels
ci Continuous integration system:apple silicon Affects Apple Silicon only (Darwin/ARM64) - e.g. M1 and other M-series chips system:mac Affects only macOS

Comments

@vtjnash
Copy link
Sponsor Member

vtjnash commented Oct 20, 2022

      From worker 5:
      From worker 5:	[95755] signal (6): Abort trap: 6
      From worker 5:	in expression starting at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/LinearAlgebra/test/addmul.jl:133
      From worker 5:	__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 5:	pthread_kill at /usr/lib/system/libsystem_pthread.dylib (unknown line)
      From worker 5:	abort at /usr/lib/system/libsystem_c.dylib (unknown line)
      From worker 5:	uv_mutex_lock at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	_ZN12_GLOBAL__N_117alloc_shared_pageEmPmb at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-codegen.1.9.dylib (unknown line)
      From worker 5:	_ZN12_GLOBAL__N_116DualMapAllocatorILb1EE11alloc_blockEm at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-codegen.1.9.dylib (unknown line)
      From worker 5:	_ZN12_GLOBAL__N_121RTDyldMemoryManagerJL19allocateCodeSectionEmjjN4llvm9StringRefE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-codegen.1.9.dylib (unknown line)
      From worker 5:	_ZN4llvm15RuntimeDyldImpl11emitSectionERKNS_6object10ObjectFileERKNS1_10SectionRefEb at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm15RuntimeDyldImpl17findOrEmitSectionERKNS_6object10ObjectFileERKNS1_10SectionRefEbRNSt3__13mapIS5_jNS8_4lessIS5_EENS8_9allocatorINS8_4pairIS6_jEEEEEE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm15RuntimeDyldImpl14loadObjectImplERKNS_6object10ObjectFileE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm16RuntimeDyldMachO10loadObjectERKNS_6object10ObjectFileE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm11RuntimeDyld10loadObjectERKNS_6object10ObjectFileE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm13jitLinkForORCENS_6object12OwningBinaryINS0_10ObjectFileEEERNS_11RuntimeDyld13MemoryManagerERNS_17JITSymbolResolverEbNS_15unique_functionIFNS_5ErrorERKS2_RNS4_16LoadedObjectInfoENSt3__13mapINS_9StringRefENS_18JITEvaluatedSymbolENSF_4lessISH_EENSF_9allocatorINSF_4pairIKSH_SI_EEEEEEEEENS9_IFvS3_NSF_10unique_ptrISD_NSF_14default_deleteISD_EEEESA_EEE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc24RTDyldObjectLinkingLayer4emitENSt3__110unique_ptrINS0_29MaterializationResponsibilityENS2_14default_deleteIS4_EEEENS3_INS_12MemoryBufferENS5_IS8_EEEE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc14IRCompileLayer4emitENSt3__110unique_ptrINS0_29MaterializationResponsibilityENS2_14default_deleteIS4_EEEENS0_16ThreadSafeModuleE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc16IRTransformLayer4emitENSt3__110unique_ptrINS0_29MaterializationResponsibilityENS2_14default_deleteIS4_EEEENS0_16ThreadSafeModuleE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN9JuliaOJIT12OptSelLayerT4emitENSt3__110unique_ptrIN4llvm3orc29MaterializationResponsibilityENS1_14default_deleteIS5_EEEENS4_16ThreadSafeModuleE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-codegen.1.9.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeENSt3__110unique_ptrINS0_29MaterializationResponsibilityENS2_14default_deleteIS4_EEEE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc19MaterializationTask3runEv at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm6detail18UniqueFunctionBaseIvJNSt3__110unique_ptrINS_3orc4TaskENS2_14default_deleteIS5_EEEEEE8CallImplIPFvS8_EEEvPvRS8_ at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc16ExecutionSession17OL_completeLookupENSt3__110unique_ptrINS0_21InProgressLookupStateENS2_14default_deleteIS4_EEEENS2_10shared_ptrINS0_23AsynchronousSymbolQueryEEENS2_8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISG_vEEEENSH_ISE_vEENS_6detail12DenseMapPairISE_SJ_EEEEEEE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc25InProgressFullLookupState8completeENSt3__110unique_ptrINS0_21InProgressLookupStateENS2_14default_deleteIS4_EEEE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ENSt3__110unique_ptrINS0_21InProgressLookupStateENS2_14default_deleteIS4_EEEENS_5ErrorE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKNSt3__16vectorINS3_4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEEENS3_9allocatorIS9_EEEENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISK_vEENS_6detail12DenseMapPairISK_SL_EEEEEEEEENS3_8functionIFvRKNSJ_IS7_NS_8DenseSetISK_SN_EENSM_IS7_vEENSP_IS7_SX_EEEEEEE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc16ExecutionSession6lookupERKNSt3__16vectorINS2_4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEEENS2_9allocatorIS8_EEEERKNS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateENS2_8functionIFvRKNS_8DenseMapIS6_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISM_vEEEENSN_IS6_vEENS_6detail12DenseMapPairIS6_SP_EEEEEEE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc16ExecutionSession6lookupERKNSt3__16vectorINS2_4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEEENS2_9allocatorIS8_EEEENS0_15SymbolStringPtrENS0_11SymbolStateE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS0_15SymbolStringPtrENS0_11SymbolStateE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS_9StringRefENS0_11SymbolStateE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libLLVM.dylib (unknown line)
      From worker 5:	_ZN9JuliaOJIT9addModuleEN4llvm3orc16ThreadSafeModuleE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-codegen.1.9.dylib (unknown line)
      From worker 5:	_ZL12jl_add_to_eeRN4llvm3orc16ThreadSafeModuleERNS_9StringMapIPS1_NS_15MallocAllocatorEEERNS_8DenseMapIS4_iNS_12DenseMapInfoIS4_vEENS_6detail12DenseMapPairIS4_iEEEERNSt3__16vectorINSH_IS4_NSG_9allocatorIS4_EEEENSI_ISK_EEEEi at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-codegen.1.9.dylib (unknown line)
      From worker 5:	_ZL20_jl_compile_codeinstP19_jl_code_instance_tP15_jl_code_info_tmN4llvm3orc17ThreadSafeContextE at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-codegen.1.9.dylib (unknown line)
      From worker 5:	jl_generate_fptr_impl at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-codegen.1.9.dylib (unknown line)
      From worker 5:	jl_compile_method_internal at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	mul! at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-6.0/build/default-macmini-x64-6-0/julialang/julia-master/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:303
      From worker 5:	unknown function (ip: 0x1837c2666)
      From worker 5:	ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/LinearAlgebra/test/addmul.jl:190 [inlined]
      From worker 5:	macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-6.0/build/default-macmini-x64-6-0/julialang/julia-master/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1584 [inlined]
      From worker 5:	macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/LinearAlgebra/test/addmul.jl:179 [inlined]
      From worker 5:	macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-6.0/build/default-macmini-x64-6-0/julialang/julia-master/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1496 [inlined]
      From worker 5:	macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/LinearAlgebra/test/addmul.jl:179 [inlined]
      From worker 5:	macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-6.0/build/default-macmini-x64-6-0/julialang/julia-master/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1584 [inlined]
      From worker 5:	macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/LinearAlgebra/test/addmul.jl:150 [inlined]
      From worker 5:	top-level scope at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-6.0/build/default-macmini-x64-6-0/julialang/julia-master/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1584 [inlined]
      From worker 5:	top-level scope at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/LinearAlgebra/test/addmul.jl:0
      From worker 5:	jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	ijl_toplevel_eval_in at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	eval at ./boot.jl:370 [inlined]
      From worker 5:	include_string at ./loading.jl:1523
      From worker 5:	ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	_include at ./loading.jl:1583
      From worker 5:	include at ./Base.jl:450 [inlined]
      From worker 5:	macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/test/testdefs.jl:29 [inlined]
      From worker 5:	macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-6.0/build/default-macmini-x64-6-0/julialang/julia-master/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1496 [inlined]
      From worker 5:	macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/test/testdefs.jl:23 [inlined]
      From worker 5:	macro expansion at ./timing.jl:474 [inlined]
      From worker 5:	#runtests#3 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/test/testdefs.jl:21
      From worker 5:	runtests at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/test/testdefs.jl:5 [inlined]
      From worker 5:	runtests at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/test/testdefs.jl:5
      From worker 5:	unknown function (ip: 0x10d7de92d)
      From worker 5:	ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	jl_f__call_latest at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	do_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	#invokelatest#2 at ./essentials.jl:812
      From worker 5:	ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	do_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	invokelatest at ./essentials.jl:807
      From worker 5:	ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	do_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	#110 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-6.0/build/default-macmini-x64-6-0/julialang/julia-master/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285
      From worker 5:	run_work_thunk at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-6.0/build/default-macmini-x64-6-0/julialang/julia-master/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
      From worker 5:	macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-6.0/build/default-macmini-x64-6-0/julialang/julia-master/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285 [inlined]
      From worker 5:	#109 at ./task.jl:514
      From worker 5:	unknown function (ip: 0x10d7d83af)
      From worker 5:	ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/lib/julia/libjulia-internal.1.9.dylib (unknown line)
      From worker 5:	Allocations: 2726094196 (Pool: 2721630192; Big: 4464004); GC: 3738
LinearAlgebra/addmul                             (5) |         failed at 2022-10-20T06:47:52.299
ProcessExitedException(5)
Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
    @ Base ./task.jl:909
  [2] wait()
    @ Base ./task.jl:973
  [3] wait(c::Base.GenericCondition{ReentrantLock})
    @ Base ./condition.jl:124
  [4] take_buffered(c::Channel{Any})
    @ Base ./channels.jl:456
  [5] take!(c::Channel{Any})
    @ Base ./channels.jl:450
  [6] take!(::Distributed.RemoteValue)
    @ Distributed /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:726
  [7] remotecall_fetch(::Function, ::Distributed.Worker, ::String, ::Vararg{String}; kwargs::Base.Pairs{Symbol, UInt128, Tuple{Symbol}, NamedTuple{(:seed,), Tuple{UInt128}}})
    @ Distributed /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:461
  [8] kwcall(::NamedTuple{(:seed,), Tuple{UInt128}}, ::typeof(remotecall_fetch), ::Function, ::Distributed.Worker, ::String, ::Vararg{String})
    @ Distributed /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:454
  [9] remotecall_fetch(::Function, ::Int64, ::String, ::Vararg{String}; kwargs::Base.Pairs{Symbol, UInt128, Tuple{Symbol}, NamedTuple{(:seed,), Tuple{UInt128}}})
    @ Distributed /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:492
 [10] kwcall(::NamedTuple{(:seed,), Tuple{UInt128}}, ::typeof(remotecall_fetch), ::Function, ::Int64, ::String, ::Vararg{String})
    @ Distributed /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:492
 [11] macro expansion
    @ /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/test/runtests.jl:260 [inlined]
 [12] (::var"#45#57"{Vector{Task}, var"#print_testworker_errored#53"{ReentrantLock, Int64, Int64}, var"#print_testworker_stats#51"{ReentrantLock, Int64, Int64, Int64, Int64, Int64, Int64}, Vector{Any}, Dict{String, DateTime}})()
    @ Main ./task.jl:514
Worker 5 terminated.
Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::TCPSocket, nb::Int64)
    @ Base ./stream.jl:410
  [2] (::Base.var"#wait_locked#717")(s::TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base ./stream.jl:949
  [3] unsafe_read(s::TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base ./stream.jl:955
  [4] unsafe_read
    @ ./io.jl:761 [inlined]
  [5] unsafe_read(s::TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base ./io.jl:760
  [6] read!
    @ ./io.jl:762 [inlined]
  [7] deserialize_hdr_raw
    @ /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/Distributed/src/messages.jl:167 [inlined]
  [8] message_handler_loop(r_stream::TCPSocket, w_stream::TCPSocket, incoming::Bool)
    @ Distributed /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:172
  [9] process_tcp_streams(r_stream::TCPSocket, w_stream::TCPSocket, incoming::Bool)
    @ Distributed /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-2.0/build/default-macmini-x64-2-0/julialang/julia-master/julia-87fd6d9d0c/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
 [10] (::Distributed.var"#103#104"{TCPSocket, TCPSocket, Bool})()
    @ Distributed ./task.jl:514

https://buildkite.com/julialang/julia-master/builds/17193#0183f487-4d43-4d2c-bb10-b6441505bc35

@vtjnash vtjnash added the ci Continuous integration label Oct 20, 2022
@quinnj
Copy link
Member

quinnj commented Oct 20, 2022

I was actually going to open an issue this morning where we're seeing a very similar backtrace like:

signal (6): Abort trap: 6
in expression starting at REPL[3]:1
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
pthread_kill at /usr/lib/system/libsystem_pthread.dylib (unknown line)
abort at /usr/lib/system/libsystem_c.dylib (unknown line)
uv_mutex_lock at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-codegen.1.8.dylib (unknown line)
_ZN12_GLOBAL__N_117alloc_shared_pageEmPmb at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-codegen.1.8.dylib (unknown line)
_ZN12_GLOBAL__N_116DualMapAllocatorILb1EE11alloc_blockEm at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-codegen.1.8.dylib (unknown line)
_ZN12_GLOBAL__N_121RTDyldMemoryManagerJL19allocateCodeSectionEmjjN4llvm9StringRefE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-codegen.1.8.dylib (unknown line)
_ZN4llvm15RuntimeDyldImpl11emitSectionERKNS_6object10ObjectFileERKNS1_10SectionRefEb at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm15RuntimeDyldImpl17findOrEmitSectionERKNS_6object10ObjectFileERKNS1_10SectionRefEbRNSt3__13mapIS5_jNS8_4lessIS5_EENS8_9allocatorINS8_4pairIS6_jEEEEEE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm15RuntimeDyldImpl14loadObjectImplERKNS_6object10ObjectFileE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm16RuntimeDyldMachO10loadObjectERKNS_6object10ObjectFileE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm11RuntimeDyld10loadObjectERKNS_6object10ObjectFileE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm13jitLinkForORCENS_6object12OwningBinaryINS0_10ObjectFileEEERNS_11RuntimeDyld13MemoryManagerERNS_17JITSymbolResolverEbNS_15unique_functionIFNS_5ErrorERKS2_RNS4_16LoadedObjectInfoENSt3__13mapINS_9StringRefENS_18JITEvaluatedSymbolENSF_4lessISH_EENSF_9allocatorINSF_4pairIKSH_SI_EEEEEEEEENS9_IFvS3_NSF_10unique_ptrISD_NSF_14default_deleteISD_EEEESA_EEE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm3orc24RTDyldObjectLinkingLayer4emitENSt3__110unique_ptrINS0_29MaterializationResponsibilityENS2_14default_deleteIS4_EEEENS3_INS_12MemoryBufferENS5_IS8_EEEE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm3orc14IRCompileLayer4emitENSt3__110unique_ptrINS0_29MaterializationResponsibilityENS2_14default_deleteIS4_EEEENS0_16ThreadSafeModuleE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeENSt3__110unique_ptrINS0_29MaterializationResponsibilityENS2_14default_deleteIS4_EEEE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm3orc19MaterializationTask3runEv at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm6detail18UniqueFunctionBaseIvJNSt3__110unique_ptrINS_3orc4TaskENS2_14default_deleteIS5_EEEEEE8CallImplIPFvS8_EEEvPvRS8_ at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupENSt3__110unique_ptrINS0_21InProgressLookupStateENS2_14default_deleteIS4_EEEENS2_10shared_ptrINS0_23AsynchronousSymbolQueryEEENS2_8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISG_EEEENSH_ISE_EENS_6detail12DenseMapPairISE_SJ_EEEEEEE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm3orc25InProgressFullLookupState8completeENSt3__110unique_ptrINS0_21InProgressLookupStateENS2_14default_deleteIS4_EEEE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ENSt3__110unique_ptrINS0_21InProgressLookupStateENS2_14default_deleteIS4_EEEENS_5ErrorE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKNSt3__16vectorINS3_4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEEENS3_9allocatorIS9_EEEENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISK_EENS_6detail12DenseMapPairISK_SL_EEEEEEEEENS3_8functionIFvRKNSJ_IS7_NS_8DenseSetISK_SN_EENSM_IS7_EENSP_IS7_SX_EEEEEEE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKNSt3__16vectorINS2_4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEEENS2_9allocatorIS8_EEEERKNS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateENS2_8functionIFvRKNS_8DenseMapIS6_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISM_EEEENSN_IS6_EENS_6detail12DenseMapPairIS6_SP_EEEEEEE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKNSt3__16vectorINS2_4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEEENS2_9allocatorIS8_EEEENS0_15SymbolStringPtrENS0_11SymbolStateE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS0_15SymbolStringPtrENS0_11SymbolStateE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS_9StringRefENS0_11SymbolStateE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN9JuliaOJIT9addModuleENSt3__110unique_ptrIN4llvm6ModuleENS0_14default_deleteIS3_EEEE at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-codegen.1.8.dylib (unknown line)
_ZL12jl_add_to_eeRNSt3__110unique_ptrIN4llvm6ModuleENS_14default_deleteIS2_EEEERNS1_9StringMapIPS5_NS1_15MallocAllocatorEEERNS1_8DenseMapIPS2_iNS1_12DenseMapInfoISD_EENS1_6detail12DenseMapPairISD_iEEEERNS_6vectorINSL_IS8_NS_9allocatorIS8_EEEENSM_ISO_EEEEi at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-codegen.1.8.dylib (unknown line)
_ZL20_jl_compile_codeinstP19_jl_code_instance_tP15_jl_code_info_tm at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-codegen.1.8.dylib (unknown line)
jl_generate_fptr_impl at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-codegen.1.8.dylib (unknown line)
jl_compile_method_internal at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)

This is from a fairly complex RelationalAI workload that only occurs (as far as we can tell) on Julia 1.8 Intel Apple (but not M1 w/ rosetta it seems? We're still trying to confirm this).

Does this seem like the same root cause or I can open up a separate issue?

@vtjnash
Copy link
Sponsor Member Author

vtjnash commented Oct 20, 2022

Those look similar, though seems very strange (it is a very simple mutex), or it is general memory corruption that somehow targets that

@ViralBShah ViralBShah added system:mac Affects only macOS system:arm ARMv7 and AArch64 system:apple silicon Affects Apple Silicon only (Darwin/ARM64) - e.g. M1 and other M-series chips and removed system:arm ARMv7 and AArch64 labels Oct 31, 2022
@quinnj
Copy link
Member

quinnj commented Nov 1, 2022

I can confirm that #47419 fixes the original issue I posted above. @vtjnash will be better positioned to say whether the original CI issue would also be fixed by that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci Continuous integration system:apple silicon Affects Apple Silicon only (Darwin/ARM64) - e.g. M1 and other M-series chips system:mac Affects only macOS
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants