[Issue] Figure out flow for custom domain names

[pamelafox]

• Make sure you've installed the latest version using instructions in the wiki

Output from azd version

azd version 0.7.0-beta.1 (commit 9ce7165)

Describe the bug

Everything is working as currently designed, so this is more a bug in the process than the code.

I would like to be able to use azd to deploy a website that has a custom domain attached (on Static Web Apps, App Service, or Container Apps). However, if I specify a domain name in Bicep using 'Microsoft.Web/staticSites/customDomains@2022-03-01' that has not yet been verified, then I get an error during the provision stage. I cannot verify a domain name until I know the website endpoint, however, since the DNS record is a CNAME for 'www' with the value of the website endpoint.

This is how I ended up doing it for my personal website, www.pamelafox.org:

1. I set up the infra with domainName as a parameter defaulting to empty string, and only conditionally create the custom domain if the parameter is non-empty:
   https://github.com/pamelafox/pamelafox-site/blob/main/infra/swa.bicep#L24
2. I then run azd up with domainName still empty in main.parameters.json
3. When I see the deployed endpoint URI, I go to my registrar and add a CNAME of www->endpoint
   and wait a few minutes
4. I set domainName in main.parameters.json and re-deploy.

That seemed to work, but it's possible that I also intervened in the Portal in between those steps.
Maybe there'd be a more straightforward approach if I used Azure's registrar? I already have a registrar I pay for, so I set it up there.

[weikanglim]

Not sure if it helps, but I usually think of setting up DNS for an application as two separate steps:

1. Get the app up-and-running with a private endpoint (run provision)
2. At some point when the code is deployed, and everything is healthy, use DNS records to point to the private endpoint. You could choose to use the Azure offering here. It can either be done through the Portal manually, or via a separate bicep deployment to be automated.

I don't think you'd want to set up DNS immediately as part of the app provisioning, until you have everything confirmed working.

[rajeshkamal5050]

We need to provide some guidance around this before GA.

[rajeshkamal5050]

@weikanglim @pamelafox is there any existing Azure documentation which already exists which we can point to as part of Azd FAQ?

cc @savannahostrowski @ellismg

[savannahostrowski]

I managed to setup a custom domain via Bicep for an ACA template but it's not perfect and it's also a bit of a chicken and an egg problem.

I have my certificate as a managed certificate in the Container App Environment (added after first provision). Then I just use the certificateId as part of the customDomain ingress property in the Container App. It's probably harmless but less than ideal as I'd rather have no one know anything about my deployments but I hardcoded the managed certificate name. This was just the simplest thing I could think of.

So at the end of the day, the UX flow for someone consuming this template would be:

• Comment out the custom domain and managed certificate resource bits for first provision
• Add a cert as a managed certificate in the CAE
• Grab certificate name
• Uncomment the custom domain Bicep and pop in the cert name
• Re-provision

https://github.com/savannahostrowski/terminal-personal-site/blob/07e32659fde54dc82ac45acc9544931d33fd0be5/infra/core/host/container-app.bicep#L31

[pamelafox]

It may be possible to do a slightly nicer workflow by using conditions in the Bicep? i.e. customDomain is in parameters.json, only create the resources in the Bicep if customDomain is !empty. Or is there a reason commenting out is necessary vs. Bicep conditionals?

[savannahostrowski]

I'm no Bicep expert so perhaps a conditional statement will suffice! I will investigate this a bit further and circle back!

[weikanglim]

I'd love for the flow to look sequential:

1. Provision infrastructure provision
2. Provision domain names provision public-dns

But if we find the single IaC easier to reason about, that'd be fine too.

From an advanced IaC authoring perspective, creating bicep conditionals based on Azure state may make it harder to test and maintain over time.

[mip1983]

At the moment to do this, I have to use 'azd infra synth'. In the containerApp.tmpl.yaml files for my web projects I add lines for custom domain under 'ingress':

    ingress:
      external: true
      targetPort: {{ targetPortOrDefault 8080 }}
      transport: auto
      allowInsecure: false
      customDomains: 
      - name: {{ .Env.WEB_URL }}
        bindingType: SniEnabled
        certificateId: {{ .Env.SSL_CERT_ID }}

I then add those environment variables as input and output in main.bicep and main.parameters.json so they can be fed by variables in my CI/CD environment.

To deploy, I need to do the 'azd provision' step first. Once the container app environment is up, I can manually upload my cert and configure DNS as required. Then azd deploy after that. (My cert and domain aren't Azure managed at the moment but that's the direction I'd like to go eventually.)

If I don't put those lines in the yaml, the SSL cert binding gets removed by azd deploy, so it's the only way I've found so far that I can deploy with this can keep the custom domain in place.

It would be great if in Aspire you could specify that a project will have a custom domain, something like:

pseudocode:

var web= builder.AddProject<Projects.Web>("web")
	.WithExternalHttpEndpoints(hasCustomDomain: true, userProvidedCertificate: true)

If 'userProvidedCertificate' is true:
Then it could put the 'customDomains' part into the yaml automatically and generate environment variables for the user to provide for url and certificate id. Then azd deploy could prompt you to provide values for these environment variables as part of the deploy process if you're running locally, or pick up from your CI/CD env.

If 'userProvidedCertificate' is false:
Generate an azure managed certificate (and domain?) as part of the provisioning/deploy process.

I'd love to get away from needing 'azd infra synth' so that everything is driven by code from my aspire project and this is one of the main puzzle pieces preventing that at the moment for me.

[ellismg]

I'd love to get away from needing 'azd infra synth' so that everything is driven by code from my aspire project and this is one of the main puzzle pieces preventing that at the moment for me.

The medium-term plan (post the initial aspire GA, but high priority after that) is to provide a way to allow full customization of the ContainerApp object (i.e. what you can control via the .yaml file today with azd infra synth, but via a strongly typed object in C#). @davidfowl has been prototyping some ideas there. Once we have that, I think we'd be able to build what you want. We already have support for adding parameters to infrastructure via the app host, and I think that combined with the ability to modify the ContainerApp in the app host would allow you to wire these things up in a way that azd deploy would be able to handle

[mip1983]

@ellismg That's great to know the vision and this pain point will go away, thanks for sharing that. That'll really simplify things, looking forward to it.

[softwaretirol]

This is a blocker for us for adapting aspire.

[relounge]

@mip1983 how did you manage customDomains to work? I added the same to my service yaml file inside ingress settings, but azd deply seems to ignore this lines and drops network settings again and again. So annoying

[mip1983]

@mip1983 how did you manage customDomains to work? I added the same to my service yaml file inside ingress settings, but azd deply seems to ignore this lines and drops network settings again and again. So annoying

There's now a command in azd to get it to preserve your domain settings:

azd config set alpha.aca.persistDomains on

So in my build pipeline (Azure DevOps):

- pwsh: |
     azd config set alpha.aca.persistDomains on
     displayName: Configure AZD to persist domain name configuration on container app environment.

The domain and cert are manually set up in the Azure portal after deploy, but this command ensures they're preserved.

I'm using an externally managed domain and certificate at the moment, eventually would like this to all be azure managed and generated via aspire.

[relounge]

I use GutHub actions for deployment. I added a stet with this setting but nothing has changed.
`
- name: Log in with Azure (Client Credentials)
if: ${{ env.AZURE_CREDENTIALS != '' }}
run: |
$info = $Env:AZURE_CREDENTIALS | ConvertFrom-Json -AsHashtable;
Write-Host "::add-mask::$($info.clientSecret)"

      azd auth login `
        --client-id "$($info.clientId)" `
        --client-secret "$($info.clientSecret)" `
        --tenant-id "$($info.tenantId)"
    shell: pwsh
    env:
      AZURE_CREDENTIALS: ${{ secrets.AZURE_CREDENTIALS }}

  - name: Setup azd
    run: azd config set aca.persistDomains on

  - name: Provision Infrastructure
    run: azd provision --no-prompt
    env:
      AZD_INITIAL_ENVIRONMENT_CONFIG: ${{ secrets.AZD_INITIAL_ENVIRONMENT_CONFIG }}

  - name: Deploy Application
    run: azd deploy --no-prompt`

Does anyone know what is wrong here?

I did the same locally from the dev machine and used "azd up" and it seem to take the customDomain section but it end up with error "Invalid certificateId". @mip1983 please, can you tell me what is this certificate Id? I use azure managed certificated and in the portal I see only certificate names. I tried to use them but no luck

[mip1983]

@relounge If you're using 'azd config set alpha.aca.persistDomains on' you can remove the custom domains bit from your YAML file. That's only something I used along with output variables from bicep before this alpha.aca.persistDomains existed as a work around to stop it clearing the domain config each time.

Unless you're doing any other customization, you can just take the YAML that infra synth generates (or not use it at all if this is your only customization) along with this azd config command.

[relounge]

Thanks it worked. Added this to my workflow file.

  - name: Setup custom domains
    run: azd config set alpha.aca.persistDomains on

The only one thing left is to understand how to stop dropping ingress settings all the time, I mean allow an endpoint to be external after deployment. But this is out of this topic

[mip1983]

Thanks it worked. Added this to my workflow file.

  - name: Setup custom domains
    run: azd config set alpha.aca.persistDomains on

The only one thing left is to understand how to stop dropping ingress settings all the time, I mean allow an endpoint to be external after deployment. But this is out of this topic

Hopefully should be a simple one liner 'WithExternalHttpEndpoints' in Program.cs of your Aspire.AppHost project e.g.:

var webApp = builder.AddProject<Projects.web>("<your-web-project>")
	.WithExternalHttpEndpoints();

[mip1983]

@rajeshkamal5050

Is there any guidance in the works for this area? Would be good to switch over to an aspire/azure based dns and certificate solution

[rajeshkamal5050]

@mip1983 this depends on the work being done as part of #3292 which is currently being worked on.

cc @vhvb1989 @ellismg @davidfowl @mitchdenny

[mitchdenny]

We are just about to backport a change into release/9.0 of Aspire to allow custom domains to be set when using PublishAsAzureContainerApp. We still need to support the current behavior for people that aren't using this however.