Attempt to create an Azure Secure Service Fabric cluster fails

[jsutch]

Terraform Version

Run terraform -v to show the version. If you are not running the latest version of Terraform, please upgrade because your issue may have already been fixed.

person@Shrdlu:~/Coding/Starbucks/Terraform/HelloWorldDemo$ terraform -v
Terraform v0.9.9 (0.9.4-0.9.9)

Affected Resource(s)

-azurerm_template_deployment

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.
Service Fabric creation With ARM Template

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.

https://www.dropbox.com/s/kcdf0zxagvasj65/terraform_secure_service_fabric.tar.pgp?dl=0

Debug Output

Please provider a link to a GitHub Gist containing the complete debug output: https://www.terraform.io/docs/internals/debugging.html. Please do NOT paste the debug output in the issue; just paste a link to the Gist.
Included
https://www.dropbox.com/s/kcdf0zxagvasj65/terraform_secure_service_fabric.tar.pgp?dl=0

Panic Output

If Terraform produced a panic, please provide a link to a GitHub Gist containing the output of the crash.log.

Expected Behavior

What should have happened?

I would have expected the creation of a Secure Service Fabric Cluster using the
secret = "${var.secret}"
sourceVaultValue = "${var.sourceVaultValue}"
certificateUrlValue = "${var.certificateUrlValue}"
certificateStoreValue = "${var.certificateStoreValue}"
certificateThumbprint = "${var.certificateThumbprint}"
clientCertificateThumbprint = "${var.clientCertificateThumbprint}"
clusterProtectionLevel = "${var.clusterProtectionLevel}"

imports from the spcdev_template.json, which pulls in variables from the variables.tf file.

Actual Behavior

What actually happened?

An unsecured Service Fabric cluster (everything but the Secure part) created.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Important Factoids

Are there anything atypical about your accounts that we should know? For example: Running in EC2 Classic? Custom version of OpenStack? Tight ACLs?
This is a best attempt to create a secure service fabric cluster using azurerm.

credentials (subscription, tenant, sp and password) are pulled in as environment variables and must be set to the vars setup in credentials.tf

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

• azurerm_public_ip doesn't change from dynamic to static #1234

[grubernaut]

Hey @jsutch, thanks for the issue!

However, it looks like the debug log that you pasted was corrupted somewhere along the way. Either between your computer->dropbox or during the download.

$ gpg --no-batch --verbose --list-packets terraform_secure_service_fabric.tar.pgp             
gpg: packet(5) with unknown version 183        
gpg: packet(5) with unknown version 116        
# off=0 ctb=c5 tag=5 hlen=2 plen=1 new-ctb     
:key packet: [unknown version]                 
# off=3 ctb=96 tag=5 hlen=5 plen=2826133868    
:key packet: [unknown version]        

Would you be able to email me directly the encrypted file? Thanks!

[jsutch]

Update sent to the Github listed email. Let me know if you need anything else.

[grubernaut]

Hey @jsutch, sorry for the delay on this one!

I've gotten the decrypted configuration to @tombuildsstuff for a fix, as he's our resident Azure expert. 😄 Thanks!

[jsutch]

@tombuildsstuff - thanks for looking at this. If you you have POC .tf configs that create a simple secure service fabric cluster (passing the certificate/thumbprint/etc vars) that would be enough to unblock me. Is there a method to do this that works currently?

[tombuildsstuff]

@jsutch sorry for the delay looking into this, I've got the decrypted files and I'm taking a look into this now :)

[jsutch]

@tombuildsstuff - thanks. Any early observations you can make about whether the functionality should be available, or obvious limitations to the approach in the code would be helpful. Happy to answer any questions I can.

[jsutch]

Checking in. Is there an expectation of Azure Secure Service Fabric to with the azurerm provider using ARM templates?

[jsutch]

Could I get an update? This is a blocker for our project.

[tombuildsstuff]

👋🏻 hey @jsutch

Apologies for the delayed response here, it's taken a little while to understand how Service Fabric works so we can diagnose this correctly. From what I can see the ARM Template being used is incorrect here - looking specifically at the Microsoft.ServiceFabric/clusters object:

{
  "apiVersion": "2016-09-01",
  "type": "Microsoft.ServiceFabric/clusters",
  "name": "[parameters('clusterName')]",
  "location": "[parameters('clusterLocation')]",
  "properties": {
    "clientCertificateCommonNames": [],
    "clientCertificateThumbprints": [],
    "clusterState": "[parameters('clusterState')]",
    "diagnosticsStorageAccountConfig": {
        "blobEndpoint": "[parameters('supportLogStorageAccountBlobEndpoint')]",
        "protectedAccountKeyName": "[parameters('protectedAccountKeyName')]",
        "queueEndpoint": "[parameters('supportLogStorageAccountQueueEndpoint')]",
        "fileEndpoint": "[parameters('supportLogStorageAccountFileEndpoint')]",
        "storageAccountName": "[parameters('supportLogStorageAccountName')]",
        "tableEndpoint": "[parameters('supportLogStorageAccountTableEndpoint')]"
    },
    "fabricSettings": [],
    "managementEndpoint": "[concat('http://',parameters('lbIPName'),':',parameters('nt0fabricHttpGatewayPort'))]",
    "nodeTypes": [
      {
        "name": "[parameters('vmNodeType0Name')]",
        "applicationPorts": {
          "endPort": "[parameters('nt0applicationEndPort')]",
          "startPort": "[parameters('nt0applicationStartPort')]"
        },
        "clientConnectionEndpointPort": "[parameters('nt0fabricTcpGatewayPort')]",
        "durabilityLevel": "Bronze",
        "ephemeralPorts": {
          "endPort": "[parameters('nt0ephemeralEndPort')]",
          "startPort": "[parameters('nt0ephemeralStartPort')]"
        },
        "httpGatewayEndpointPort": "[parameters('nt0fabricHttpGatewayPort')]",
        "isPrimary": true,
        "vmInstanceCount": "[parameters('nt0InstanceCount')]"
      }
    ],
    "sourceVaultValue": "[parameters('sourceVaultValue')]",
    "certificateUrlValue": "[parameters('certificateUrlValue')]",
    "certificateStoreValue": "[parameters('certificateStoreValue')]",
    "certificateThumbprint": "[parameters('certificateThumbprint')]",
    "clientCertificateThumbprint": "[parameters('clientCertificateThumbprint')]",
    "clusterProtectionLevel": "[parameters('clusterProtectionLevel')]",
    "provisioningState": "[parameters('provisioningState')]",
    "reliabilityLevel": "[parameters('reliabilityLevel')]",
    "upgradeMode": "[parameters('upgradeMode')]",
    "vmImage": "[parameters('vmImage')]"
  },
  "tags": {
    "resourceType": "Service Fabric",
    "clusterName": "[parameters('clusterName')]"
  }
}

When comparing this to a Secure Cluster created in the portal - there's no certificate section which is used to specify the certificate for the cluster, and no ClusterProtectionLevel which defines that this should be a Secure cluster:

"certificate": {
    "thumbprint": "[parameters('certificateThumbprint')]",
    "x509StoreName": "[parameters('certificateStoreValue')]"
}

When creating a Secure Service Fabric Cluster in the Portal - and then exporting the ARM Template - we get the relevant certificate section (and the ClusterProtectionLevel set) included in the output:

{
    "apiVersion": "2017-07-01-preview",
    "type": "Microsoft.ServiceFabric/clusters",
    "name": "[parameters('clusterName')]",
    "location": "[parameters('clusterLocation')]",
    "dependsOn": [
        "[concat('Microsoft.Storage/storageAccounts/', parameters('supportLogStorageAccountName'))]"
    ],
    "properties": {
        "addonFeatures": [],
        "certificate": {
            "thumbprint": "[parameters('certificateThumbprint')]",
            "x509StoreName": "[parameters('certificateStoreValue')]"
        },
        "clientCertificateCommonNames": [],
        "clientCertificateThumbprints": [],
        "clusterState": "Default",
        "diagnosticsStorageAccountConfig": {
            "blobEndpoint": "[reference(concat('Microsoft.Storage/storageAccounts/', parameters('supportLogStorageAccountName')), variables('storageApiVersion')).primaryEndpoints.blob]",
            "protectedAccountKeyName": "StorageAccountKey1",
            "queueEndpoint": "[reference(concat('Microsoft.Storage/storageAccounts/', parameters('supportLogStorageAccountName')), variables('storageApiVersion')).primaryEndpoints.queue]",
            "storageAccountName": "[parameters('supportLogStorageAccountName')]",
            "tableEndpoint": "[reference(concat('Microsoft.Storage/storageAccounts/', parameters('supportLogStorageAccountName')), variables('storageApiVersion')).primaryEndpoints.table]"
        },
        "fabricSettings": [
            {
                "parameters": [
                    {
                        "name": "ClusterProtectionLevel",
                        "value": "[parameters('clusterProtectionLevel')]"
                    }
                ],
                "name": "Security"
            }
        ],
        "managementEndpoint": "[concat('https://',reference(concat(parameters('lbIPName'),'-','0')).dnsSettings.fqdn,':',parameters('nt0fabricHttpGatewayPort'))]",
        "nodeTypes": [
            {
                "name": "[parameters('vmNodeType0Name')]",
                "applicationPorts": {
                    "endPort": "[parameters('nt0applicationEndPort')]",
                    "startPort": "[parameters('nt0applicationStartPort')]"
                },
                "clientConnectionEndpointPort": "[parameters('nt0fabricTcpGatewayPort')]",
                "durabilityLevel": "Bronze",
                "ephemeralPorts": {
                    "endPort": "[parameters('nt0ephemeralEndPort')]",
                    "startPort": "[parameters('nt0ephemeralStartPort')]"
                },
                "httpGatewayEndpointPort": "[parameters('nt0fabricHttpGatewayPort')]",
                "isPrimary": true,
                "vmInstanceCount": "[parameters('nt0InstanceCount')]"
            }
        ],
        "provisioningState": "Default",
        "reliabilityLevel": "None",
        "upgradeMode": "Automatic",
        "vmImage": "Windows"
    },
    "tags": {
        "resourceType": "Service Fabric",
        "clusterName": "[parameters('clusterName')]"
    }
}

Given this is an issue with an ARM Template and not specific to Terraform - I'm not sure we're the best people to help here. That said, I've posted the full ARM Template exported from the Portal for a Secure Service Fabric Cluster here which may be of some use? Instead, I think you may be better to ask this question on the ARM Templates repository - where a specialist with that technology should be able to help.

With regards to supporting Service Fabric Clusters natively in Terraform - this work has been started in #4 - but there's a number of dependencies preventing this from progressing, such as support for Key Vault Certificates and Locks. Once those resources have been added we can progress with the pull request to natively support Service Fabric Clusters in Terraform.

Given this issue doesn't appears to be Terraform specific I'm going to close this for the moment - however please feel free to re-open it if needed :)

Thanks!

[jsutch]

My question is much less related to the ARM template JSON than it is to the problem of how to ingest the ARM elements into the azurerm_template_deployment such that they can be filled with vars from the variables.tf file.

How would you recommend importing variables from those multidimensional properties into the azurerm_template_deployment parameters? Do you have examples of how this would be represented in the .tf template?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!