Develop a new CI build running under GitHub Actions

[CharliePoole]

AppVeyor is continuing to have issues so creating an alternative approach is necessary. This should be relatively simple since all it has to do is kick off our build.cake with the proper options and handle the results.

@veleek Are you familiar with GitHub actions? Would you be willing to take this?

[CharliePoole]

It appears that @OsirisTerje has started this already. We can use this as a basis. I am inclined to leave the logic of what actions to take to the Cake recipe rather than calling cake multiple times for a push, but that's open for discussion.

[CharliePoole]

@veleek Any progress on this?

[veleek]

Minimal progress, mostly spending time digging through the existing code and familiarizing myself with build stuff.

I spent a bit of time getting stuff setup as well as trying to see if I could use act to run the actions locally, but that ended up being a dead end because it doesn't work great with Windows hosts (what the current action uses).

[CharliePoole]

I moved it the next release in order to get 3.18 out.

Act sounds interesting. Too bad it didn't work out. So what next? Are you able to continue on this?

I'm wondering if it isn't as simple (initially) as looking at what we do in our appveyor.yml and emulating it. The cake script is basically a black box to AppVeyor except for the fact that it has an AppVeyor target, which does what we want done on AppVeyor. I can create an initial target GitHubActions, and set it up initially to do just Build and Test.

Going beyond those two steps introduces questions of authorization of course, but there is a --nopush option in the script to use in testing without actually publishing new releases. Cake itself has a --dry-run option (possibly mis-spelled) we could also use for tests - it merely lists what would have been done.

[veleek]

That works for me! I had a bit of time to spend on this tonight and I'm testing a bunch of stuff out in my fork, but I'm running into some unexpected issues.

I have the action setup to just run the exact same build.ps1 that is run by the Appveyor script and it's failing on a later packaging step. See https://github.com/veleek/nunit-console/actions/runs/10002323808/job/27647413350 for details on the error.

Attempting to gather dependency information for package 'NUnit.Extension.NUnitProjectLoader.3.7.1' with respect to project 'D:\a\nunit-console\nunit-console\package\tests', targeting 'Any,Version=v0.0'
An error occurred while retrieving package metadata for 'NUnit.Extension.NUnitProjectLoader.3.7.1' from source 'https://community.chocolatey.org/api/v2/'.
  Failed to fetch results from V2 feed at 'https://community.chocolatey.org/api/v2/Packages(Id='NUnit.Extension.NUnitProjectLoader',Version='3.7.1')' with following message : Response status code does not indicate success: 504 (Gateway Time-out).
  Response status code does not indicate success: 504 (Gateway Time-out).

I'm surprised by a Gateway Time-out, but I also can't find that package at all, since https://community.chocolatey.org/packages/nunit-extension-nunit-project-loader/3.7.1 definitely exists. Do you have any insight about what to do here?

Mostly unrelated thing, I also wanted to see whether we could use the GitHub actions "matrix" strategy to build/test a bunch of the different versions in parallel. It'd probably need to happen later because to do it right we'd have to pull a bunch of the logic out of cake and into the GitHub action, but before I spend time on it i'm interested in hearing whether there's a reason that we need to build them all together or could we pass in a "framework version" attribute and have it build/test that one. Or are there tests that depend on having multiple different versions SxS?

[CharliePoole]

That package is not a chocolatey package... I think nuget is just looking on all the feeds for it and happens to encounter that gateway timeout.

I've never found a problem with building sequentially. I'm in the "make it work first" camp.

[CharliePoole]

@veleek As I mentioned, I'll be (mostly) out of action for the next three weeks. My sister and her husband are visiting us.

I'll do a bit, perhaps late at night, with my priorities being...

1. Any new critical bugs that pop up for the 3.0.1 NUnit-Console release
2. Some experimental scripting changes on TestCentric.

My suggestion for some sub-goals for you on this issue:

1. Get an action working that specifies target "Test" (which implies Build as well). Based on your latest output, that should work easily, because Package would not have executed if Test had not succeeded.

2. Try a new one, which t targets either "BuildTestAndPackage" or just "Package" (implying Build).

3. Move on to "Publish", which implies BuildAndPackge. For safety, you should add "--nopush" to the arguments for build.ps1 so that no actual publishing occurs. You can verify that it ran correctly by looking at the output for a message that tells you pushing was suppressed.

You should rebase your own branch on the latest main, which matches the 3.0.1 release. Since I wont be changing the script for a few weeks it will remain stable. Of course, if something critical requires changing the script itself, I'll tell you,
but that's unlikely.

Time allowing, I'm going to port some script changes I made on NUnit to TestCentric. In particular, I added code to allow package tests to be run against multiple versions of the console, which is useful for testing extensions. I may also shadow some of your work on TestCentric.

[veleek]

Created an initial PR to make some tweaks to the github action, as well as a separate one to update the Cake recipe to enable binlogs.

Next step is going to be further troubleshooting the package steps to try and identify the problems.

[veleek]

I've identified the reason that the package verification tests are failing. We are missing X86 versions of the .NET Runtime (or SDK?) on the machines so they will fail with errors similar to this failing run: https://github.com/veleek/nunit-console/actions/runs/10328774992/job/28595729078

I haven't had a chance to look at the setup-dotnet docs to figure out how to install the right things. I had similar error on my local dev box when running x86 versions of tests and I had to explicitly install the x86 versions of the SDKs to resolve them. But interestingly, on the GH action run linked above, the x86 on .NET 8 job succeeded even though we don't explicitly install an x86 version of that. I suppose it's possible that version is already available by default?

---------------------------------------------
Run mock-assembly-x86.dll under .NET Core 3.1
---------------------------------------------
Running package test
NUnit Console 3.18.0-pr00002-3984+3ed5f2aeadfb71c55290ee54c84dabf632f8f359 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Saturday, August 10, 2024 4:03:54 AM

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    netcoreapp3.1/mock-assembly-x86.dll


Unhandled Exception: NUnit.Engine.NUnitEngineException: Remote test agent exited with non-zero exit code -2147450730
   at NUnit.Engine.Services.TestAgency.OnAgentExit(Process process, Guid agentId)
   at NUnit.Engine.Services.TestAgency.<>c__DisplayClass13_0.<GetAgent>b__0(Object sender, EventArgs e)
   at System.Diagnostics.Process.OnExited()
   at System.Diagnostics.Process.RaiseOnExited()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(Object state, Boolean timedOut)

Errors, Failures and Warnings

1) Error : 
NUnit.Engine.NUnitEngineException : Unable to acquire remote process agent
--NUnitEngineException
Unable to acquire remote process agent
   at NUnit.Engine.Runners.ProcessRunner.CreateAgentAndRunnerIfNeeded()
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)

Test Run Summary
  Overall result: Failed
  Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2024-08-10 04:03:54Z
    End time: 2024-08-10 04:03:55Z
    Duration: 0.468 seconds

Results (nunit3) saved as TestResult.xml

ERROR: Test Result not as expected!

[CharliePoole]

I'd be OK with commenting out the X86 tests for now or installing. I can work on this while you are traveling as well. We should figure out what we want to have working immediately vs longer term.

WRT .NET 8, I have noticed it seems to be available without installing. My workflow for the recipe worked without any step to install .NET.

[veleek]

If you look in the Setup Job portion of the action run there's a link:

that lists all of the included software on the runner image. So for the windows runners, they've already got all of these .NET tools installed.

[CharliePoole]

So probably X86 is installed as well for .NET 6, 7 and 8. Surprising it isn't included when you install core 3.1. We can test all that of course.

For ubuntu-latest, you get these tools. .NET 6, 7 and 8 are all there as well.

[CharliePoole]

This issue has been resolved in version 3.18.2

The release is available on:
GitHub.
NuGet packages are also available NuGet.org and
Chocolatey Packages may be found at Chocolatey.org