Azure Application Gateway with end-to-end SSL

[hashibot]

This issue was originally opened by @edsonmarquezani as hashicorp/terraform#16896. It was migrated here as a result of the provider split. The original body of the issue is below.

---

I'm currently having a hard time trying to setup an Application Gateway with end-to-end SSL on Azure.

First of all, I noticed the configuration (and documentation as well) is a bit confusing. The backend_http_settings block expects an authentication_certificate nested object/block, instead of a reference to it like all the other blocks (frontend_ip_configuration_name, backend_address_pool_name, etc).

I could get it to work with a configuration like this.

  backend_http_settings {
    name                  = "https"
    cookie_based_affinity = "Disabled"
    port                  = "${var.master_port}"
    protocol              = "Https"
    request_timeout       = 5
    probe_name            = "default"

    authentication_certificate {
      name = "master"
    }
  }

I'm also declaring a separate authentication_certificate block.

  authentication_certificate {
    name = "master"
    data = "${base64encode(file("files/keys/master.cer"))}"
  }

Documentation seems to indicate there's something to be fixed regarding this.

Anyway, applying it, I get a generic error from the API, and the resource ends up stuck with a Failed status on panel.

Error Creating/Updating ApplicationGateway network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=200 -- Original Error: Long running operation terminated with status 'Failed': Code="InternalServerError" Message="An error occurred."

Any clues on this?

The complete resource configuration is below:

resource "azurerm_application_gateway" "master" {
  name                = "${var.master_lb_name_prefix}"
  location            = "${var.location}"
  resource_group_name = "${azurerm_resource_group.openshift.name}"

  sku {
    name     = "Standard_Small"
    tier     = "Standard"
    capacity = 2
  }

  gateway_ip_configuration {
    name      = "${var.master_lb_name_prefix}"
    subnet_id = "${var.application_gateway_subnet}"
  }

  ssl_certificate {
    name     = "default"
    data     = "${base64encode(file("files/keys/master.pfx"))}"
    password = ""
  }

  authentication_certificate {
    name = "master"
    data = "${base64encode(file("files/keys/master.cer"))}"
  }

  frontend_port {
    name = "https"
    port = "${var.master_port}"
  }

  frontend_ip_configuration {
    name                          = "default"
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = "${var.application_gateway_subnet}"
  }

  backend_address_pool {
    name            = "${var.master_lb_name_prefix}-instances"
    ip_address_list = ["${azurerm_network_interface.master-if0.*.private_ip_address}"]
  }

  probe {
    name                = "default"
    protocol            = "Https"
    path                = "/"
    interval            = 5
    host                = "master.${var.cluster_domain}"
    timeout             = 4
    unhealthy_threshold = 3
  }

  backend_http_settings {
    name                  = "https"
    cookie_based_affinity = "Disabled"
    port                  = "${var.master_port}"
    protocol              = "Https"
    request_timeout       = 5
    probe_name            = "default"

    authentication_certificate {
      name = "master"
    }
  }

  http_listener {
    name                           = "https"
    frontend_ip_configuration_name = "default"
    frontend_port_name             = "https"
    protocol                       = "Https"
    ssl_certificate_name           = "default"
  }

  request_routing_rule {
    name                       = "default"
    rule_type                  = "Basic"
    http_listener_name         = "https"
    backend_address_pool_name  = "${var.master_lb_name_prefix}-instances"
    backend_http_settings_name = "https"
  }
}

[edsonmarquezani]

Oh, I'm sorry. 😢 Should've posted it here.

[tombuildsstuff]

@edsonmarquezani no problem at all - we can migrate the issues around as needed :)

I'm taking a more in-depth look into Application Gateways over the next few days (and updating our tests to ensure all the known use-cases are covered) - as such I'm hopeful to get back to you with a working example of how to do this next week.

Thanks!

[edsonmarquezani]

@tombuildsstuff Great! If you need any additional info, just tell me. Thanks in advance.

[rahulkp220]

Any updates on this?

[wtfiwtz]

There a few things to note about Application Gateways:

• They need their own dedicated subnet for the "gateway" IP - this subnet must be empty (or contain only other app gateways) - see https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-faq#configuration
• The "authentication certificate" seems to be for outbound SSL communications - e.g. if you re-encrypt, this is probably the cert the HTTPS listener would be expecting to connect to on your VM (... as far as I can tell)
• You will probably need a probe - the host defaults to "127.0.0.1" but you will need an endpoint such as "/"... you can check the health status of each backend VM under ❤️ Backend Health in the Azure Portal
• Changes to App Gateways take about 20 minutes, so be prepared for some frustrating delays!!
• Adding / removing hosts to the App Gateway (or Load Balancer) is difficult - I'm currently using IP addresses to provide the most flexibility. I can't actually replace a Standard Load Balancer with an Application Gateway because of the tight coupling with the VMs or the VM Scale Sets behind the Load Balancer. This means I will need to destroy the VMs upgrade them to fully detach them. Very frustrating!

Otherwise, the app gateway all seems to work as expected.

[wtfiwtz]

Nope I spoke too soon :)

There is no way I can see to attach a VM Scale Set to an Application Gateway, directly... you are basically forced to do it through a Standard Load Balancer as an intermediary. Power Shell seems to be the only way to do it, but I still haven't been able to get it to work at this time.

See these links:

• https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-upgrade-scale-set
• https://docs.microsoft.com/en-us/azure/application-gateway/tutorial-create-vmss-powershell

This is a related issue (getting the backend id to use): #727

I've tried the following using the CLI and Power Shell but they don't seem to work:

az network application-gateway show -n APPGW -g <resourcegroup> -o json
az vmss show -n VMSSNAME -g <resourcegroup> -o json
az vmss update -n VMSSNAME -g <resourcegroup> --add virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].applicationGatewayBackendAddressPools '{"id": "/subscriptions/.../resourceGroups/.../providers/Microsoft.Network/applicationGateways/.../backendAddressPools/<backend-address-pool-name>"}'

Install-Module PowerShellGet -Force
Install-Module -Name AzureRM -AllowClobber
Import-Module -Name AzureRM

$pass = "<password>"|ConvertTo-SecureString -AsPlainText -Force
$cred = New-Object System.Management.Automation.PsCredential("<user>",$pass)
Connect-AzureRmAccount -Credential $cred -ServicePrincipal -Subscription <subscrption> -TenantId <tenantid>

$vmss=Get-AzureRmVmss -ResourceGroupName <resource group> -Name VMSSNAME
$ipconf = New-AzureRmVmssIPConfig myNic -ApplicationGatewayBackendAddressPoolsId /subscriptions/.../resourceGroups/.../providers/Microsoft.Network/applicationGateways/.../backendAddressPools/<backend-pool-name> -SubnetId $vmss.VirtualMachineProfile.NetworkProfile.NetworkInterfaceConfigurations[0].IpConfigurations[0].Subnet.Id –Name $vmss.VirtualMachineProfile.NetworkProfile.NetworkInterfaceConfigurations[0].IpConfigurations[0].Name
$vmss.VirtualMachineProfile.NetworkProfile.NetworkInterfaceConfigurations[0].IpConfigurations[0] = $ipconf
Update-AzureRmVmss -ResourceGroupName <resourcegroup> -Name VMSSNAME -virtualMachineScaleSet $vmss

[bpoland]

I know for a fact that you can attach a VMSS directly to an App Gateway using ARM templates. If you have the app gateway running, you can deploy a VMSS ARM template with applicationGatewayBackendAddressPools set and it works -- no extra load balancer needed.

[tombuildsstuff]

@wtfiwtz @bpoland it's also possible via the Azure API - there's a feature request for this here: #857 and a PR: #884

[wtfiwtz]

Ok I figured it out... the VMSS has an upgrade mode which defaults to manual. You must "upgrade" each VM instance inside the scale set and then they are attached to the App Gateway.

At this time (March 2018) the Portal doesn't support changes to App Gateways with VM scale sets.

So the commands above did work, but it wasn't obvious from the Portal UI because of this manual step.

[AlexBevan]

Am I right in saying this now fixed, not tested myself but can see the documentation has been updated?

[AvinashReddyVaka]

How to create multiple front_end ports and respective http_settings sections in a single application gateway?

[tombuildsstuff]

hi @edsonmarquezani @rahulkp220 @wtfiwtz @bpoland

Given this issue is blocked on an upstream issue in the Azure API rather than keeping multiple issues open and trying to ensure they all remain up to date - I'm going to close this in favour of #1576 which is the Meta-Issue tracking the Bugs and Enhancements for the Application Gateway Resource. Once the bug in the API is fixed we should be able to take a look into this, but we'll keep track of the status of this issue there for the moment.

Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!