Add Multi repo support

[dosper7]

In several companies that use microservices have their own cloud platform teams and normally this "translates" into having a service per git repos.

In this cases how to use .NET Aspire? Meaning, different projects from different repos? It's not common to have multiple services within the same repo (unless you have a monorepo like Google 😆)

What is your suggestion?

[davidfowl]

Clone multiple repos locally, create an Uber solution and add the apphost to it, wiring up all of the projects required.

That's where I would start.

it you don’t want to pull and run the other repos locally, then I’m not sure exactly what the workflow would be (point to something already running).

[massina]

Can someone explain what is the Uber solution?

[En3Tho]

@massina A big solution that contains all these cloned projects. Hence "uber".

[atrauzzi]

@davidfowl - Spending lots of time thinking about what you're saying about the ubersolution. I'm going to try it out with my tinkering locally as a vessel to learn dapr. I think it makes sense...

(some terminology here is bound to shift from person to person, but I sketched out a quick diagram to give the gist of it)

I feel like there's a little bit of opportunity for some Aspire magic to smooth things over in the ubersolutions?

[DamianEdwards]

Some patterns I imagine might emerge:

• git sub-modules for service repos and a parent repo with the .sln file and AppHost project, if you want to build and run all services locally
• Services published as container images that are then composed into an AppHost project as containers, if you don't want to build the services locally but you do want to run them locally
• Dev-instances of services deployed to dedicated environments and/or locally that are then composed into an AppHost project as HTTP endpoints, if you don't want to build or run the services locally

[bbrandt]

I believe we would need to publish our service defaults as a NuGet package, published to our own private NuGet server (Azure Artifacts in our case). Then projects throughout all the different repos could add a reference to this service defaults NuGet package.

[contione]

using NuGet might complicate debugging and hinder quick updates.

[DGAISmith]

Although this is true, how often would you realistically want to change/debug the service defaults? And when you do - when we want to debug the code delivered via nuget package, we just take a local clone of the repo and swap the nuget reference out for a project reference in the solution so it's fairly simple to get a local setup ready for debugging. So it is a complication, but a small one and the benefit of sharing the defaults via nuget outweighs it.

[scabana]

Always dreamed of having a primitive that would look something like

<ProjectReference Include="../Foo/Foo.csproj">
  <FallbackPackageReference Include="Foo" Version="1.0.0" />
</ProjectReference>

Although, I do also think it would make things much harder to have a mental model of what's going on, have good build reproducibility.

Maybe something like this:

<PackageReference Include="Foo" Version="1.0.0" ProjectReference="../Foo/Foo.csproj" />

And a parameter like dotnet run --use-local-projects. But I digress, back to the main topic here:

With that said, having the service defaults as a project reference for smaller codebases makes sense, but I feel it doesn't grow well into a large system of composed services. Also, you may not want to push a fundamental service default change across the whole system at once since those changes tend to have impact. The solution of moving it to a NuGet enables the gradual rollout of those changes across a system while re-using knowledge, behaviors and expectations of devs towards NuGet upgrades in their system.

[atrauzzi]

Yeah, I'd want the project reference to be supplementary to a more reliable package reference. That way if the project doesn't exist, no problem. But I don't think Microsoft has taken the reality of multiple solutions seriously enough yet.

[DamianEdwards]

I believe we would need to publish our service defaults as a NuGet package, published to our own private NuGet server (Azure Artifacts in our case). Then projects throughout all the different repos could add a reference to this service defaults NuGet package.

Yep, or you could put the ServiceDefaults in a sub-module and include it that way. There are lots of ways for folks to share code and assets among teams and I don't think we want to be too prescriptive about it. We should ensure we're flexible enough to support different patterns.

[artyom-p]

@DamianEdwards would be nice to have some samples on how to proceed with multi-repo scenarios. It is definitely a very cool and useful tool, but a huge part of microservice solutions are split across different repos, and in this case its not that obvious how to utilize it in local environment.

[dosper7]

I also don't understand how the deployment story is in this scenario, where we "run" our k8s cluster with a custom cloud platform team.

For instance, we use github + tekton pipelines + argo CD to deploy to k8s using multiple repos (one per service), all things k8s related are generated/managed by the platform (I've worked in previous companies with similar setup bu using different tools like gitlab as an example)

How Aspire can be helpful in this cases?

[davidfowl]

@artyom-p

@DamianEdwards would be nice to have some samples on how to proceed with multi-repo scenarios. It is definitely a very cool and useful tool, but a huge part of microservice solutions are split across different repos, and in this case its not that obvious how to utilize it in local environment.

Are you cloning everyone's repositories and running them today?

[davidfowl]

@dosper7

I also don't understand how the deployment story is in this scenario, where we "run" our k8s cluster with a custom cloud platform team.

It sounds like you don't need to use the deployment part of aspire because you have already figured it out! It's still great for local development, using components, and service discovery!

[artyom-p]

@davidfowl Yes, we clone all the repositories locally and debug service communication having multiple studio instances opened. Its doable, but its a pain.

[davidfowl]

@artyom-p What does the pain free solution look like? What would you ideal experience be?

[artyom-p]

@davidfowl Let's say we have 2 microservices that need to communicate with each other, they live in their own git repositories.

Each microservice can have its own or shared dependencies, for example:
ServiceA depends on ServiceB, Redis and MongoDB
ServiceB depends on MongoDB

There are cases when you want to debug only one service, but you still need another one to be running with all its dependencies, in this case, we can just somehow reference that second service and make it run, I think there was something similar in Tye before if I'm not mistaken.

The second case is more complex - when you want to debug both services. I'm not sure if it's doable without referencing projects of ServiceB in ServiceA solution.

[DGAISmith]

We also have the same multi-repo approach here, and this is where Tye was really strong for local development. A tye.yaml file could be relatively pathed to target x services, all cloned locally. Tye was also useable as a completely external tool, no changes were required to any service, and they could also target a variety of different dotnet versions.

Aspire looks so much better in many ways to Tye, and I can see it's trying to achieve different things, but being able to use Tye as a non-invasive external local dev orchestrator remains a huge benefit and time saver for multi-repo scenarios.

[davidfowl]

There are cases when you want to debug only one service, but you still need another one to be running with all its dependencies, in this case, we can just somehow reference that second service and make it run, I think there was something similar in Tye before if I'm not mistaken.

You can do a similar thing with the app host. Clone multiple repositories and create a project file that references individual projects. Maybe we need to enable referencing app hosts themselves 🤔.

Tye was also useable as a completely external tool, no changes were required to any service, and they could also target a variety of different dotnet versions.

Aspire does require .NET 8 and above.

but being able to use Tye as a non-invasive external local dev orchestrator remains a huge benefit and time saver for multi-repo scenarios.

Can you show me your tye setup?

[scabana]

You can do a similar thing with the app host. Clone multiple repositories and create a project file that references individual projects. Maybe we need to enable referencing app hosts themselves 🤔.

That could be a solution! In our case, we have git repositories per service in a system. We expect to do the following:

• For a specific service:
  • Run/Debug locally
  • Run tests in an isolated way, having its dependencies handled through Testcontainers
  • Generate a GitOps (argocd) application yaml (trying to keep it simple)
• For a group of service, or a system:
  • Generate a graph of dependencies across services
  • Stop deployment (which are independent per service) if a dependency is broken/missing

I have the impression we could generate a strongly typed dll/nuget that would expose another app host's resources which would enable us to have compile time errors. This would come with a challenge for handling circular dependencies, generated constants representing the resources from another app host could be enough.

I'd love to be able to use Aspire here. We have a few hundreds of services, thinking about having them all cloned with the right version is not practical. I'd rather have Renovate / Dependabot update nuget refrences and have build pipelines validate the integrity of the system.

[davidfowl]

What are you trying to accomplish though? Is the goal to run with another team's dependencies?

[scabana]

The end goal is running services with their dependencies in production for sure! With that said, when services are ran locally, they either have mocks for dependencies or they act a one of the services within an actual deployment (far from ideal since you can get traffic from other people on that system). One of the reason to interconnect the services from different repositories/teams is to have a build time check that all interconnections between the service are valid, ie: Foo service that required Bar service's url should fail at build time if the Bar service has not been declared in the system. In a distributed system, it would also enable having a shared redis cache deployment while keeping the ownership of that part outside of the team's boundary (for example, a platform engineering team could expose their resources and service teams could consume those in a safe way, not just a magic configuration key that comes from who knows where).

[DGAISmith]

Many thanks for the reply, really enjoying the Conf this year.

Tye was also useable as a completely external tool, no changes were required to any service, and they could also target a variety of different dotnet versions.

Aspire does require .NET 8 and above.

You say that, and of course you're totally correct if you want to use everything in the Aspire stack (and who am I to argue with the mad professors who came up with this amazing stuff), but I have been able to hack something together today from the starter application template using AddExecutable():

var builder = DistributedApplication.CreateBuilder(args);

//var apiservice = builder.AddProject<Projects.AspireApp2_ApiService>("apiservice");
var apiservicePort = 5323;
var apiserviceScheme = "http";
var apiserviceHostname = "localhost";

var apiservice = builder
    .AddExecutable("apiservice", "dotnet", @"..\AspireApp2.ApiService6", "run", $"--urls={apiserviceScheme}://{apiserviceHostname}:{apiservicePort}")
    .WithServiceBinding(apiservicePort, apiserviceScheme)
    .WithOtlpExporter();

builder.AddProject<Projects.AspireApp2_Web>("webfrontend");
    // .WithReference(apiservice);

builder.Build().Run();

In this case, ApiService6 is a dotnet6 api, inside which I've added OpenTelemetry support based on the code in the ServiceDefaults project. You do then end up with traces and (partial) metrics in the Asipre dashboard. So this is potentially useful-ish (I should try this with func.exe as well to see if we can spin up Functions in a decent way while Function support isn't quite ready yet). Service discovery is obvs broken here, and Aspire Components are presumably off the table too.

This serves two purposes though - allows some use of older dotnet applications, and also allows targeting of different repos outside of the solution the Aspire AppHost lives in. So it falls back to a similar kind of experience to Tye in these respects.
Probably not what you had in mind when you designed Aspire of course, but it's really nice to get some of the new dashboard experience with older projects.

but being able to use Tye as a non-invasive external local dev orchestrator remains a huge benefit and time saver for multi-repo scenarios.

Can you show me your tye setup?

Sure, we have a lot of tye files, each targeting a specific surface area of our microservices (which are all in individual repos). Redacted example below:

name: Portal Backend Frontend
extensions:
- name: zipkin
- name: seq
services:
- name: Portal-UI
  executable: cmd
  args: '/c npm start'
  workingDirectory: ./MyCompany.Portal/src/Portal
  bindings:
    - host: localhost
      port: 80
      protocol: https
- name: Identity-Api
  project: ./MyCompany.IdentityApi/src/MyCompany.IdentityApi/MyCompany.IdentityApi.csproj
  bindings:  
   - port: 5000
     protocol: https	  
- name: Portal-Api
  project: ./MyCompany.PortalApi/src/MyCompany.PortalApi/MyCompany.PortalApi.csproj
  bindings:  
   - port: 19081
     protocol: http
- name: DataPull-Function
  azureFunction: ./MyCompany.Functions/src/MyCompany.DataPull.Functions
  bindings:
  - port: 7071
    protocol: http      
- name: DataMapper-Function
  azureFunction: ./MyCompany.Functions/src/MyCompany.DataMapper.Functions
  bindings:
  - port: 7072
    protocol: http      

Key thing here is that we haven't made any code changes to any of these services, and they're spread over 4 different repos.

[DamianEdwards]

@artyom-p I've opened dotnet/aspire-samples#35 to track adding some samples of multi-repo orchestration from an AppHost project.

[davidfowl]

My idea was more of a concept, not an actual definition.

I hear you, but I think we need to explore something in the realm of the feature set of the app host. The app host is never deployed, it's a local orchestrator only. It can run executables, containers, projects, cloud and reference remote resources (deployed elsewhere). The dashboard is also an OTLP server so telemetry is pushed to it with a single env variable config. With those primitives, it should be possible to build various workflows with these primitives.

We're still working through a couple of issues like enabling docker networking and making it easier to reference things within containers (so you can run your another team's container images for e.g.). Hopefully that'll be enough for you all to experiment and come back with feedback about what works and what doesn't work.

AppHost is a service that can run locally or in a centralised environment.
If running in a central place, each team can connect to the same host and execute the registration of each service. This means that each team would have their services up and running for each other. This could be an integration testing environment.
Developers can hook their development versions to that host and work like dev containers. Since Aspire controls service discovery, it can also control routing to multiple instances and versions like a service mesh solution.

I think we're pretty far away from this based on the current set of capabilities. That sad, maybe you could build something on top of these primitives to get the next level of understanding for what this workflow looks like.

[luizbon]

The app host is never deployed, it's a local orchestrator only.

That makes complete sense.
I want aware of this premise. Not sure if it is documented and I missed that.

Regarding my idea, I don't think it is aligned with the project goals then, but also can be implemented as an extension on top of it.

[davidfowl]

Regarding my idea, I don't think it is aligned with the project goals then, but also can be implemented as an extension on top of it.

Maybe consider other interesting ways to connect services like using an external service discovery system (dapr, consul) and the app host to bridge local and remote development.

Maybe you can use a shared configuration server (something like azure app config in the azure case) as a way to store service discovery information across teams.

The problem is the code needs to be executed (that means access to another teams' code or assets), or it needs to be running remotely.

[atrauzzi]

👋 -- Hello! Chiming in late here, but I was thinking: Why not make it possible to make a reference a git repo URL? With something like that, Aspire could clone and then either discover or be explicitly told what type of project is within for it to manage?

This would be a game changer, particularly for onboarding scenarios. I've followed a pattern of having a "home" repo that houses a docker-compose.yml definition that presumes the presence of other repositories in the past. Aspire is in a great position to improve on such a concept.

[dosper7]

@atrauzzi that seems good idea (just don't know about the execution part), but would definitely be helpful since the repos url is "always" there. It seems an approach like Go or Deno do with the "usings/imports".

Maybe something like this pseudo code:

var builder = DistributedApplication.CreateBuilder(args);
var apiservice = builder.AddGitProject("www.github.com/FooCompany/BaaRepoName").WithName("apiservice");

What do you think? Crazy stupid idea?

[davidfowl]

👋 -- Hello! Chiming in late here, but I was thinking: Why not make it possible to make a reference a git repo URL? With something like that, Aspire could clone and then either discover or be explicitly told what type of project is within for it to manage?

Because I don't think aspire should manage git repos. Pulling, updating, picking the right branch etc etc. That should be managed externally.

This would be a game changer, particularly for onboarding scenarios. I've followed a pattern of having a "home" repo that houses a docker-compose.yml definition that presumes the presence of other repositories in the past. Aspire is in a great position to improve on such a concept.

Right, I think it should be possible to have an aspire app host reference source anywhere but it shouldn't be responsible for the cloning and updating of the git repo.

[atrauzzi]

Yeah, I get what you're saying, it even makes sense. But I guess then it takes away some of that snappy convenience of being able to obtain the Aspire project itself and have things self-assemble.

If something like this were to exist, I don't think I'd want to see aspire go as far as branch management. Just clone and then put the developer back in the drivers seat, that's all.

[davidfowl]

If something like this were to exist, I don't think I'd want to see aspire go as far as branch management. Just clone and then put the developer back in the drivers seat, that's all.

That feels very incomplete IMO. I'm not sure we would want to ship a feature like that. I'm sure it could be built on top of our existing primitives, so if that's something interesting, somebody should build that and put up a nuget package.

[atrauzzi]

Oh! So that is to say, there's support for different source types to be added? And I guess then the rest of Aspire would (hopefully) pick up, assuming all went well?

[DamianEdwards]

The hosting primitives for DistributedApplicationBuilder are extensible, as long as any added resource eventually boils down to a container or an executable. So you could add overloads of AddProject() that accept git URLs or create a new AddGitProject() method that adds all the cloning logic but ultimately just adds normal project resources to the model.

[atrauzzi]

Love it. Something like .AddGithubProject("someoneororg/blabla") and then it would go cd ../../; mkdir "someoneororg/blabla"; cd "someoneororg/blabla"; git clone... (oversimplified!), and then do the regular add.

Although I agree it has an out-of-scopey flavour, I could see it being a very popular feature or extension. We'll see 🙂

[atrauzzi]

So what if I had multiple projects within the same git repository? Would it know to not try and re-clone it?

[WhiteOlivierus]

You can set a different clone target path. Than you can have 1 repo 2 different project states

[atrauzzi]

Hmm, not sure I follow. I'm just saying, if I have two "entrypoints" in a single repository, can that be supported without having to clone the repo multiple times?

[WhiteOlivierus]

I have updated again because in our workflow this works better, and maybe the documentation is better now too?

[Xenofono]

I'm encountering some resistance at my workplace when I sugggested trying out Aspire. This seems like it could be an easy way for me and other curious devs to try it out without disturbing the workflow for the old schoolers that just want to work in the specific repo?

[bUKaneer]

Hey, I've been working on a project that I think may be useful for some here. Please feel free to send feedback and let me know what you think.

https://github.com/bUKaneer/white-label.starthere

[Beelzenef]

So... after a few months, no further progress on multi-repo support? Is it already implemented?

[davidfowl]

There's a pretty cool implementation #1137 (reply in thread), that you can try out.

[WhiteOlivierus]

I'm working on this at the moment, so it is still in flux. But if anyone wants to look at it and suggest any features that I missed that would be amazing :).

[Beelzenef]

thanks for the awesome answers!

[jornhd]

Not directly a multi-repo problem, but I have multiple solutions in the same repo. To reference an "external" project in the repo, I tried to use:
builder.AddProject("otherapi", "<relative path to .csproj>");
But when I start the app host, I get the message: Either the project type is not supported or the project is not loaded in the solution.
With Tye there was no problem referencing projects in multiple solutions.

[vaspopinex]

Experiencing same issue here. Did you manage to resolve the issue? @jornhd

[jornhd]

Unfortunately no. I've tried debugging it, and it does actually find and read the launchSettings.json file, but I still get this error. Could be a bug in Aspire.

[vaspopinex]

FYI, i just got it working using CLI rather than visual studio. Possibly a visual studio issue?
dotnet run --project .\Aspire.AppHost.csproj

[DamianEdwards]

#2154

[isaranghi]

Hello,

Currently, our team uses a microservice architecture where each microservice resides in its own repository and is maintained by a dedicated team. We also use Docker Compose for each microservice to manage running the API, database, Redis, RabbitMQ, etc. Most of the time, each team deals only with their own microservice, but occasionally, they run all the microservices locally for end-to-end testing. To facilitate this, we have created another Docker Compose file that runs all the microservices along with their dependencies.

Now, my question is: if we decide to go with Aspire, the use case where we want to run all the microservices locally works fine using one Uber AppHost solution and sharing ServiceDefaults as a NuGet package. But how would you suggest we run individual microservices (that currently use Docker Compose)? Besides the Uber AppHost, should we also have an AppHost per microservice that stays in the same repository as the microservice? Or is there another solution anyone would suggest?

Thanks.

[DamianEdwards]

Are you planning to always run the microservices from other repos using container images that have been published, or would you be getting the source code and running them as regular projects? Either way, you could use separate AppHost projects with each only configuring the resources you want, or you could use one AppHost project with separate launch profiles (configured in Properties/launchSettings.json), with each launch profile setting a different environment name (DOTNET_ENVIRONMENT) and then having your AppHost code conditionally add the desired resources depending on the environment (or some other environment variable).

[isaranghi]

@DamianEdwards Thanks for your response...

Currently, we are getting the source code of each microservice and running them using Docker Compose. This is how we plan to proceed going forward. However, most of the time, each team is only working on their own microservice. Based on your suggestion, I believe I have two options:

1. Create an AppHost project for each microservice that takes care of all the dependencies of that microservice. This would also be checked into the same repository along with the microservice.

2. Create an Uber AppHost project and conditionally set the dependencies and microservices to run.

I am leaning more towards Option 1, as we already have Docker Compose for each microservice. Additionally, I think it would be easier for each team to just clone their microservice and work with it using its own AppHost.