Support for Stream Analytics Output to Synapse

[ggmahyar]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

The ouput to Azure Synapse (formerly Data Warehouse) does not exist yet as a Terraform resource. At the moment, we deploy it via an ARM template but it is sub-optimal. Furthermore, this output and the type of job which uses this output need an storage account to be set, which is also only possible via ARM template at the moment. This should be another Feature Request, but the Synapse output depends on that.

New or Affected Resource(s)

• azurerm_stream_analytics_output_synapse

The ARM Template is the same as azurerm_stream_analytics_output_mssql, but with another type:

{
      "type": "Microsoft.StreamAnalytics/streamingjobs/outputs",
      "apiVersion": "2017-04-01-preview",
      "name": "${job_name}/stg-${table}",
      "properties": {
        "datasource": {
          "type": "Microsoft.Sql/Server/DataWarehouse",
          "properties": {
            "database": "${synapse_db_name}",
            "server": "${sql_server_name}",
            "table": "stg.${table}",
            "user": "${sql_admin}",
            "password": "${sql_admin_pw}"
          }
        }
      }
    }

Potential Terraform Configuration

resource "azurerm_stream_analytics_output_synapse" "example" {
  name                      = "example-output-synapse"
  stream_analytics_job_name = azurerm_stream_analytics_job.example.name
  resource_group_name       = azurerm_stream_analytics_job.example.resource_group_name

  server   = azurerm_sql_server.example.fully_qualified_domain_name
  user     = azurerm_sql_server.example.administrator_login
  password = azurerm_sql_server.example.administrator_login_password
  database = azurerm_sql_database.example.name
  table    = "ExampleTable"
}

References

• #0000

[owenfarrell]

@ggmahyar - I took a stab at implementing this. My struggle here is that the acceptance tests pass, but I want to make sure my representative configuration (and corresponding example in the docs) actually works.

In the ARM template provided above, I'm a bit confused on how to map server and database properties to a Synapse workspace. My assumption is that:

• Serverless SQL pools use Synapse's sqlOnDemand endpoint as the server, which provides a "master" database out of the box.
• Dedicated SQL pools use Synapse's sql endpoint as the server, and use the name of the SQL pool as the database name.

Two specific asks here:

1. Could you confirm what Synapse attributes you are filling in to the above ARM template?
2. Can you point me to a representative Stream Analytics test harness? (I don't profess to be an expert on that front and could use some help testing the acceptance tests)

[github-actions]

This functionality has been released in v2.84.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.