Build fails intermittently with "Command exited with code -1073741819"

[Methuselah96]

We've gotten seven build failures (out of 75 builds) so far this week each with "Command exited with code -1073741819." This has been happening for at least a month, but was occurring less frequently before this week. So far it's always failed while running Jest. I investigated other related issues, but they seemed specific to x86 (#3407).

Environment:

• Node 14.15.1 (x64)
• [email protected]

One of them dumped an error message before failing:

#
# Fatal error in , line 0
# unreachable code
#
#
#
#FailureMessage Object: 00000038086F9BB0
 1: 00007FF73D95021F napi_wrap+109311
 2: 00007FF73D8848DF std::basic_ostream<char,std::char_traits<char> >::operator<<+57151
 3: 00007FF73E4BD442 V8_Fatal+162
 4: 00007FF73E001C7C v8::internal::ItemParallelJob::NumberOfTasks+3292
 5: 00007FF73E001ECD v8::internal::ItemParallelJob::NumberOfTasks+3885
 6: 00007FF73E0023D3 v8::internal::ItemParallelJob::NumberOfTasks+5171
 7: 00007FF73E040052 v8::internal::ItemParallelJob::Task::RunInternal+18
 8: 00007FF73E03FFE1 v8::internal::ItemParallelJob::Run+641
 9: 00007FF73DFFF3A0 v8::internal::ItemParallelJob::AddTask+4496
10: 00007FF73E05C6EE v8::internal::Heap::RightTrimFixedArray+878
11: 00007FF73E0576CF v8::internal::Heap::PageFlagsAreConsistent+799
12: 00007FF73E04C971 v8::internal::Heap::CollectGarbage+2033
13: 00007FF73E04AB75 v8::internal::Heap::AllocateExternalBackingStore+1317
14: 00007FF73E064E16 v8::internal::Factory::AllocateRaw+166
15: 00007FF73E0783CC v8::internal::FactoryBase<v8::internal::Factory>::NewDescriptorArray+76
16: 00007FF73DE94A46 v8::internal::FixedArray::CopyTo+486
17: 00007FF73DE949CE v8::internal::FixedArray::CopyTo+366
18: 00007FF73DEB3B6C v8::internal::Map::EnsureDescriptorSlack+156
19: 00007FF73DEB799D v8::internal::Map::SetPrototype+829
20: 00007FF73DEB226C v8::internal::Map::CompleteInobjectSlackTracking+1964
21: 00007FF73DEB3356 v8::internal::Map::CopyWithField+358
22: 00007FF73DEB8A15 v8::internal::Map::TransitionToDataProperty+837
23: 00007FF73DEC0286 v8::internal::LookupIterator::PrepareTransitionToDataProperty+486
24: 00007FF73DE8E5E3 v8::internal::StringSet::Add+1427
25: 00007FF73DED4CAC v8::internal::JSObject::DefineAccessor+1580
26: 00007FF73DEE349D v8::internal::JSObject::SetOwnPropertyIgnoreAttributes+109
27: 00007FF73DD94A0C v8::internal::Runtime::SetObjectProperty+54668
28: 00007FF73DD9583B v8::internal::Runtime::SetObjectProperty+58299
29: 00007FF73E248EFD v8::internal::SetupIsolateDelegate::SetupHeap+463949
30: 00007FF73E2CBF68 v8::internal::SetupIsolateDelegate::SetupHeap+1000632
31: 00007FF73E1E18E2 v8::internal::SetupIsolateDelegate::SetupHeap+40498
32: 000000A605698516 
Command exited with code -1073741819

I can provide links to private builds if those would help.

[FeodorFitsner]

Could be related to this issue: nodejs/node#34927

[FeodorFitsner]

Honestly, it's unclear how to deal with these issues. Would you be able to provide a simplest repro in a public repository? Like that issue I was able to reproduce on my workstation and on VMs in all major clouds. Try Node 15.x to see if it makes it better?

[Methuselah96]

@FeodorFitsner I can't even reproduce it outside of a running AppVeyor build. I've RDP'd into Appveyor and still was not able to reproduce when running the command in Command Prompt 50 times. Any ideas?

[FeodorFitsner]

Have you tried Node 15.x? Would you be able to do a minimalistic repro in a public repository I can play with?

[Methuselah96]

@FeodorFitsner Yes, Node 15.x has the same issue. I've tried extensively to try to create a smaller example, but have been unsuccessful so far.

[FeodorFitsner]

For the sake of experiment could you try running your build with a new .NET Core-baed build agent and see if it makes it better. You'll need to add this into your appveyor.yml:

environment:
  AGENT_VERSION: 7.0.2883
  AGENT_INSTALL_DIR: 'C:\Program Files\AppVeyor\BuildAgentCore'
  BUILD_AGENT_MODE: HyperV
  APPVEYOR_CONSOLE_DISABLE_PTY: true
 
init:
- appveyor version
- ps: $core_agent_installed = (Test-Path $env:AGENT_INSTALL_DIR)
- ps: |
    if (-not $core_agent_installed) {
      iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/build-images/master/scripts/Windows/install_appveyor_build_agent_core.ps1'))
      $appveyorPath = $env:AGENT_INSTALL_DIR
      $pwshProfileFilename = "$env:USERPROFILE\Documents\PowerShell\Microsoft.PowerShell_profile.ps1"
      if (Test-Path $pwshProfileFilename) {
          $lines = Get-Content $pwshProfileFilename
          for($i = 0; $i -lt $lines.length; $i++) {
            if ($lines[$i] -contains '*AppVeyor.BuildAgent.PowerShell.dll*') {
              $lines[$i] = ''
            }
          }
          $lines += "try { Import-Module '$appveyorPath\AppVeyor.BuildAgent.PowerShell.dll' } catch {}"
          $lines | Set-Content $pwshProfileFilename
      }
    }
# Restart VM
- ps: if (-not $core_agent_installed) { Start-Sleep -s 5; Restart-Computer }
- ps: if (-not $core_agent_installed) { Start-Sleep -s 5 }

So, the agent is re-installed on init phase of the build and then VM is restarted.

Let me know if that worked.

[Methuselah96]

@FeodorFitsner Thanks for your response! Unfortunately this fails with the same error.

[FeodorFitsner]

Alright, thanks for trying. What about x86 version of Node? Have you tried with x86?

[Methuselah96]

@FeodorFitsner Same result.

[FeodorFitsner]

Hm, honestly, at this stage I'm out of ideas :-/

[Methuselah96]

@FeodorFitsner Alright, thanks for your help. I've already spent a couple days digging into it and it happens just infrequently enough that I haven't been able to justify digging into it more.

I'll let you know if it ever gets resolved or if I find more useful info.

[tniessen]

Is this still an issue from Node.js 16 onward? (See nodejs/node#34927 (comment).)

[FeodorFitsner]

Haven't heard about it for a while.