Add Aspire.Hosting.Testing to facilitate integration testing

[ReubenBond]

Addresses #1704

This PR adds two types to aid in integration testing of Aspire AppHost projects.

• DistributedApplicationTestingBuilder<TEntryPoint>

This type allows you to write tests following the builder pattern. For example, if this is your app host's Program.cs:

var builder = DistributedApplication.CreateBuilder(args);
builder.AddRedis("redis1");
builder.AddProject<Projects.TestingAppHost1_MyWebApp>("mywebapp1");
builder.AddProject<Projects.TestingAppHost1_MyWorker>("myworker1")
    .WithEndpoint(name: "myendpoint1");
builder.AddPostgres("postgres1");
builder.Build().Run();

Then you can write tests which orchestrate and modify it like so:

[Fact]
public async Task CanRemoveResources()
{
    // Create a builder for the above AppHost - the type, Program, is any type in the app host's assembly.
    // Constructing the builder executes the AppHost on its own thread up until the point that its `Build()`
    // is called. At that point, the AppHost is suspended until our test builder's Build() is called, giving
    // the developer the opportunity to modify the app model before build.
    var appHost = new DistributedApplicationTestingBuilder<Program>();

    // Remove the redis resource from the app host - eg because we don't need it for this test.
    appHost.Resources.Remove(appHost.Resources.Single(r => r.Name == "redis1"));

    // Calling Build() allows the AppHost to resume building, returning the app.
    // The AppHost is suspended again it calls RunAsync or StartAsync. It will be resumed once
    // StartAsync is called in test code. When the app is disposed, the AppHost is terminated.
    await using var app = appHost.Build();
 
    // Allow the AppHost to start and wait for it to complete starting.
    await app.StartAsync();

    // Ensure that the resource which we added is present in the model.
    var appModel = app.Services.GetRequiredService<DistributedApplicationModel>();
    Assert.DoesNotContain(appModel.GetContainerResources(), c => c.Name == "redis1");
    Assert.Contains(appModel.GetProjectResources(), p => p.Name == "myworker1");

    // Get an endpoint from a resource
    var workerEndpoint = app.GetEndpoint("myworker1", "myendpoint1");
    Assert.NotNull(workerEndpoint);
    Assert.True(workerEndpoint.Host.Length > 0);

    // Get a connection string from a resource
    var pgConnectionString = app.GetConnectionString("postgres1");
    Assert.NotNull(pgConnectionString);
    Assert.True(pgConnectionString.Length > 0);

  // As the 'app' gets disposed, the AppHost is terminated and the resources it created are deleted.
}

For this to work, you currently need to add this snippet to your app host project (eg, at the bottom of Program.cs):

public partial class Program
{
}

• DistributedApplicationTestingHarness<TEntryPoint>

This type is akin to WebApplicationFactory<TEntryPoint>, but for DistributedApplicationBuilder instead of host builders.

Here is an example which adds an xUnit fixture which uses the testing harness:

public class TestingHarnessTests(DistributedApplicationFixture<Program> appHostFixture) : IClassFixture<DistributedApplicationFixture<Program>>
{
    private readonly HttpClient _httpClient = appHostFixture.CreateHttpClient("mywebapp1");

    [Fact]
    public async Task HttpClientGetTest()
    {
        var result1 = await _httpClient.GetFromJsonAsync<WeatherForecast[]>("/weatherforecast");
        Assert.NotNull(result1);
        Assert.True(result1.Length > 0);
    }

    private sealed record WeatherForecast(DateOnly Date, int TemperatureC, string? Summary)
    {
        public int TemperatureF => 32 + (int)(TemperatureC / 0.5556);
    }
}

The fixture:

public sealed class DistributedApplicationFixture<TEntryPoint> : DistributedApplicationTestingHarness<TEntryPoint>, IAsyncLifetime where TEntryPoint : class
{
    protected override void OnBuilderCreating(DistributedApplicationOptions applicationOptions, HostApplicationBuilderSettings hostOptions)
    {
        // This does nothing, but shows what can be intercepted.
        base.OnBuilderCreating(applicationOptions, hostOptions);
    }

    protected override void OnBuilderCreated(DistributedApplicationBuilder applicationBuilder)
    {
        // This does nothing, but shows what can be intercepted.
        base.OnBuilderCreated(applicationBuilder);
    }

    protected override void OnBuilding(DistributedApplicationBuilder applicationBuilder)
    {
        // This does nothing, but shows what can be intercepted.
        base.OnBuilding(applicationBuilder);
    }

    public async Task InitializeAsync() => await base.InitializeAsync();

    async Task IAsyncLifetime.DisposeAsync() => await DisposeAsync();
}

The harness base class offers members to:

• Get the DistributedApplication via the DistributedApplication property
• Get an HttpClient, endpoint, or connection string for a resource

TODO:

• Pipe console output from orchestrated resources & app host to test output
• VS support for debugging the orchestrated resources

Microsoft Reviewers: Open in CodeFlow

[davidfowl]

@ReubenBond do you think there's anything incremental thats worth getting in for p4?

[ReubenBond]

Yes, it's good to have the basics available in p4. It has caveats like: The app projects won't be launched under the debugger, and ports aren't randomized, so your tests need to be part of the same xunit test collection to make them run sequentially

[davidfowl]

Moving to P5

[ReubenBond]

@DamianEdwards please take a look / mess about with this

[mitchdenny]

???

[mitchdenny]

@davidfowl awareness ... this will break with your incoming async changes.

[mitchdenny]

This is really good comprehensive stuff. Impressed with the approach of using the DiagnosticSource/Listener to extract internal state from the entry point. It is a shame that we need to use the partial class trick to coerce the right accessibility on the entry point.

BTW, I am not sure that resource removal is a great example. In your sample you AddRedis(...) but you don't reference it anywhere. If you did reference it you would find that the WithReference(...) call would have resulted in a callback which captures the RedisResource.

Your code would result in DCP not spinning up the container, but the callbacks would likely blow up.

[ReubenBond]

BTW, I am not sure that resource removal is a great example.

Agreed, I've been discussing the inevitable issue you pointed out with @DamianEdwards and @davidfowl. Ideally, we'd have some way to filter out parts of the app model so that smaller bits of the application can be tested. The same issue will arise with replacing parts.

Anyhow, I'll remove it from the test - it was mostly there as a discussion point.

[BrennanConroy]

This just hangs forever. Are the tests missing using the fixture or something?

[ReubenBond]

I introduced this issue when trying to make CI tests work. There is a fix PR: #2575