Support for VMSS orchestration mode

[ArcturusZhang]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

The orchestration mode of VMSS is currently in public preview: https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/orchestration-modes

With the orchestration mode, you can now choose whether the scale set should orchestrate virtual machines which are created explicitly outside of a scale set configuration model, or virtual machine instances created implicitly based on the configuration model. Scale set orchestration mode also helps you design your VM infrastructure for high availability by deploying these VMs in fault domains and Availability Zones.

Currently we have two different orchestration modes: ScaleSetVM, and VM. Their difference are listed in this doc. The orchestration mode of a VMSS cannot be changed once it is created.

New or Affected Resource(s)

The orchestration mode ScaleSetVM is just the current default VMSS behaviour. The orchestration mode VM will only create an empty VMSS without any instances, and you will have to manually add new VMs into it by specifying the VMSS ID during the creation of the VM.

The VMSS in VM mode can accepts both linux VM and windows VM, therefore I suppose this should be achieved by introducing a new resource and data source representing a VMSS in VM mode.

And the VMs will need the functionality to add themselves to a VMSS in VM mode.

• azurerm_linux_virtual_machine_scale_set
• azurerm_windows_virtual_machine_scale_set
• azurerm_linux_virtual_machine
• azurerm_windows_virtual_machine

Potential Terraform Configuration

data "azurerm_resource_group" "main" {
    name = "test-resource"
}

resource "azurerm_linux_virtual_machine_scale_set" "test" {
  name                 = "MyVMSSInVMMode"
  location             = data.azurerm_resource_group.main.location
  resource_group_name  = data.azurerm_resource_group.main.name

  orchestration_mode = "VM"
  
  platform_fault_domain_count = 2

  zones = ["1"]

  tags = {
    environment = "Terraform Deployment"
  }
}

resource "azurerm_virtual_network" "main" {
  name                = "${var.prefix}-network"
  address_space       = ["10.0.0.0/16"]
  location            = data.azurerm_resource_group.main.location
  resource_group_name = data.azurerm_resource_group.main.name
}

resource "azurerm_subnet" "internal" {
  name                 = "internal"
  resource_group_name  = data.azurerm_resource_group.main.name
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefix       = "10.0.2.0/24"
}

resource "azurerm_network_interface" "main" {
  name                = "${var.prefix}-nic"
  resource_group_name = data.azurerm_resource_group.main.name
  location            = data.azurerm_resource_group.main.location

  ip_configuration {
    name                          = "internal"
    subnet_id                     = azurerm_subnet.internal.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_linux_virtual_machine" "main" {
  name                            = "${var.prefix}-linux"
  resource_group_name             = data.azurerm_resource_group.main.name
  location                        = data.azurerm_resource_group.main.location
  size                            = "Standard_F2"
  admin_username                  = "adminuser"
  admin_password                  = "P@ssw0rd1234!"
  disable_password_authentication = false
  network_interface_ids = [
    azurerm_network_interface.main.id,
  ]

  source_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "16.04-LTS"
    version   = "latest"
  }

  os_disk {
    storage_account_type = "Standard_LRS"
    caching              = "ReadWrite"
  }

  vmss_vm_mode_id = azurerm_linux_virtual_machine_scale_set.test.id
}

References

• #0000

[ArcturusZhang]

I am working on this feature now.

[tombuildsstuff]

@ArcturusZhang why make this a separate resource rather than being a field within the linux/windows vmss resources?

[ArcturusZhang]

@ArcturusZhang why make this a separate resource rather than being a field within the linux/windows vmss resources?

I have two reasons:

1. When the VMSS is created, it is empty and can accept either a linux or windows VM.
2. Once the VMSS is created, you can add both linux and windows VMs to it.
   Therefore I suppose this is not suitable for either linux VMSS or windows VMSS.

And additionally, I think the VMSS in VM mode should have a Computed field for all the VMs it associated

[tombuildsstuff]

@ArcturusZhang can you please test adding both a selection of Linux and Windows VM to the same Scale Set to confirm that, since Azure explicitly separates the two across other services - as such it's likely this should be bundled into those resources rather than a separate resource.

Unfortunately the computed field you've mentioned wouldn't make sense since this value would be out of sync with the VM's attached to it, as such I think we'll need to omit this.

[ArcturusZhang]

Well, I have done the test whether a linux and windows VM can co-exist or not. But I just realized that the Windows VM still shows Creating for a long time. Please let me contact with the service team to confirm.

[ArcturusZhang]

Hi @tombuildsstuff I have confirmed with developers in the service team that they currently not "officially" support mixing of different operating systems. Therefore I suppose I would take your advise to make the orchestration mode inside the azurerm_linux_virtual_machine_scale_set and azurerm_windows_virtual_machine_scale_set. And I have already modified the potential schema in my first comment.

But still, there is a question needed to be solved: how we could determine which operating system this vmss belongs to when there is no VM in it yet? I suppose that we should know this information before we are importing an VMSS, like the current logic of determine whether this is a linux VMSS or windows VMSS. But when the VMSS in VM mode is empty, there is no way that I am aware of to determine this. This is also a reason to make it a separated resource (which I forgot to mention in the last comment).

[ArcturusZhang]

Hi @tombuildsstuff I have confirmed with developers in the service team that they currently not "officially" support mixing of different operating systems. Therefore I suppose I would take your advise to make the orchestration mode inside the azurerm_linux_virtual_machine_scale_set and azurerm_windows_virtual_machine_scale_set. And I have already modified the potential schema in my first comment.

But still, there is a question needed to be solved: how we could determine which operating system this vmss belongs to when there is no VM in it yet? I suppose that we should know this information before we are importing an VMSS, like the current logic of determine whether this is a linux VMSS or windows VMSS. But when the VMSS in VM mode is empty, there is no way that I am aware of to determine this. This is also a reason to make it a separated resource (which I forgot to mention in the last comment).

Well it has been pointed out by @katbyte that the user can choose which resource to import in when there is no VMs yet. Therefore I will implement the new mode into the existing VMSS resources. Thanks!

[ArcturusZhang]

Hi @tombuildsstuff and @katbyte given that the two mode have totally different schema, the VM mode forbade all of the fields in VMSS other than platform_fault_domain_count, single_placement_group and zones, therefore I am thinking that instead of making a field called orchestration_mode in enum, how about making this field as orchestration_mode_VM_enabled as a bool? In this way, we can simply just mark this attribute to be ConflictWith all other attributes (which is still a rather long list) and this message could be shown before the Create method is called.

In this way the potential config should be like:

data "azurerm_resource_group" "main" {
    name = "test-resource"
}

resource "azurerm_linux_virtual_machine_scale_set" "test" {
  name                 = "MyVMSSInVMMode"
  location             = data.azurerm_resource_group.main.location
  resource_group_name  = data.azurerm_resource_group.main.name

  orchestration_mode_vm_enabled = true
  
  platform_fault_domain_count = 2
  single_placement_group = true  # the service only accepts true and true is the default value, therefore this attribute can be negligible

  zones = ["1"]

  tags = {
    environment = "Terraform Deployment"
  }
}

What do you think about this?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!