Set-AzDiagnosticSetting - impossible to set RentionInDays for individual categories

[AdrianSanguineti]

Description

I'm trying to set diagnostic settings via the Set-AzDiagnosticSetting commandlet for an Azure Front Door resource.

I've tried a number a ways, but I have not been able to set a separate Retention policy per category. I can easily do this via the Azure portal, but I want to automate it with PowerShell. No matter how I call/chain the calls to Set-AzDiagnosticSetting command together, the last value given for RetentionInDays is applied to all Categories.

Steps to reproduce

The following are the values I am trying to set. Metrics should be stored for 180 days, the logs indefinitely.

$diagnosticMetricSettings = @{
  Name = 'FrontDoor-Diagnostics'
  ResourceId = '<enter front door resource id>'
  StorageAccountId = '<enter storage account resource id>'
  Enabled = $true
  MetricCategory = @('AllMetrics')
  RetentionInDays = 180
  RetentionEnabled = $true
}

$diagnosticLogSettings = @{
  Name = 'FrontDoor-Diagnostics'
  ResourceId = '<enter front door resource id>'
  StorageAccountId = '<enter storage account resource id>'
  Enabled = $true
  Category = @('FrontDoorAccessLog', 'FrontdoorWebApplicationFireallLog')
  RetentionInDays = 0
  RetentionEnabled = $true
}

Executing Set-AzDiagnosticSetting with these parameters in the following order:

Set-AzDiagnosticSetting @diagnosticMetricSettings 
Set-AzDiagnosticSetting @diagnosticLogSettings 

Results in all categories and metrics having a retention policy of 0 days

Executing Set-AzDiagnosticSetting with these parameters in the following order:

Set-AzDiagnosticSetting @diagnosticLogSettings
Set-AzDiagnosticSetting @diagnosticMetricSettings 

Results in all categories and metrics having a retention policy of 180 days

Piping the commands together in any order results in the same behaviour.

I've taken a quick look at the source for Set-AzDiagnosticSetting and it looks like the SetRetention function is executed against ALL categories enabled on the resource, instead of just the categories that are passed in to the commandlet. This would confirm the behaviour that I'm experiencing.

Environment data

(Sorry hand to write this out by hand - unable to cut'n paste from target environment to GitHub)

Name                                               Value
-----                                                 -----
PSVersion                                         5.1.17134.858
PSEdition                                          Desktop
PSCompatibleVersions                     {1.0, 2.0, 3.0, 4.0...}
BuildVersion                                     10.0.17134.858
CLRVersion                                       4.0.30319.42000
WSManStackVersion                        3.0
PSRemotingProtocolVersion            2.3
SerializationVersion                          1.1.0.1

Module versions

Sorry unable provide this as unable to cut'n paste from target environment to GitHub.

Debug output

Sorry unable provide this as unable to cut'n paste from target environment to GitHub.

Error output

There's no errors executing the command.

[dingmeng-xue]

Thanks for reporting. @VeryEarly , please check this issue.

[VeryEarly]

It seems like each settings is identified by its "Name",
{resourceUri}/providers/microsoft.insights/diagnosticSettings/{name}

since your two settings have identical name, the previous one will be overwrite by the later one.

Can you try give different names for these two settings?

[AdrianSanguineti]

@VeryEarly, yes that might be a workaround, but it will still mean that this commandlet can't reproduce behaviour that is possible from the Azure Portal, and Azure CLI (based on the reading I've done, not actually tested Azure CLI because it's a pain to get working through an authenticated proxy which I can't bypass in my environment.)

Here's a screenshot of the Azure Portal that shows that I can set different Retention days per category, under the single Diagnostic Setting:

If I can do this in the Azure Portal, I would expect to be able to do the same with the PowerShell commandlet.

Using multiple Diagnostic Settings, also means that I would need to use 2 Storage accounts as the one storage account can't be used for two Diagnostic settings. Whilst that is doable, and might have some other performance benefits, it adds additional undesired complexity into the solution.

[AdrianSanguineti]

@VeryEarly, and to confirm: different names for the Diagnostic Settings (pointing to different storage accounts) produces the following results, when viewed in the Azure Portal

This lets the Retention in days to be set to desired values for categories, but I would describe this as still broken due to it not supporting the same use-cases as the portal and the management REST API allows for.

I would think the Set-AzDiagnosticSetting commandlet would benefit from having some custom model objects for categories to let the retention be set per category, which also make the parameters usage less confusing.

[AdrianSanguineti]

For reference, with the Azure CLI, its syntax allows for in a single Diagnostic Setting the retention policies independently for each category can be independently set:

az monitor diagnostic-settings create --resource {ID} -n {name}
   --event-hub-rule {eventHubRuleID} --storage-account {storageAccount}
   --logs '[
     {
       "category": "WorkflowRuntime",
       "enabled": true,
       "retentionPolicy": {
         "enabled": false,
         "days": 0
       }
     }
   ]'
   --metrics '[
     {
       "category": "WorkflowRuntime",
       "enabled": true,
       "retentionPolicy": {
         "enabled": false,
         "days": 0
       }
     }
   ]'

[VeryEarly]

Hi, I can make fix to have what you desired, set retention policy per Metric/Log
but still it would be different behavior from the Azure Portal. Because in the portal, retention days for each category need to be provided while in current retention policy is "one for all" per setting.

#11635

[AdrianSanguineti]

@VeryEarly, that merged change you have made is an improvement, so thank you!

However, I still would like to see this function match the Azure Portal behavior, but also understand that requires a major rewrite as the function's parameters have to be completely redesigned to cater for the scenario.

Therefore it would be great if the documentation for Set-AzureDiagnosticSettings is also updated to explicitly state this limitation, as it will help the next person who runs into this problem, until the different behavior can be properly addressed.

[VeryEarly]

@VeryEarly, that merged change you have made is an improvement, so thank you!

However, I still would like to see this function match the Azure Portal behavior, but also understand that requires a major rewrite as the function's parameters have to be completely redesigned to cater for the scenario.

Therefore it would be great if the documentation for Set-AzureDiagnosticSettings is also updated to explicitly state this limitation, as it will help the next person who runs into this problem, until the different behavior can be properly addressed.

Thanks for your feedback.
I'll update documents and demonstrate how this cmdlet can be properly used.

The redesign involves breaking change and due to our heavy schedule it would not catch the next breaking change release (this May). However we will certainly consider refactor the cmdlet to match behavior from Azure Portal.

[VeryEarly]

fixed in #13535