Tags for resource get deleted on second azd up execution which blocks Azure Functions deploy from working

[nzthiago]

• Make sure you've installed the latest version using instructions in the wiki

Output from azd version
azd version 1.10.1 (commit 31409a3)

Describe the bug
Running azd up on a project that has a Function App (Flex Consumption) works the first time but shows the following on subsequent calls:
ERROR: error executing step command 'deploy --all': getting target resource: resource not found: unable to find a resource tagged with 'azd-service-name: processor'. Ensure the service resource is correctly tagged in your infrastructure configuration, and rerun provision

So, after running azd up the first time the app has the tags and deploys correctly:

But after attempting azd up the second time, it fails at the deploy stage because the tags are not there:

To Reproduce
This shows up from time to time. I have just reproduced this with the E2E DURABLE-FUNC-ORDER-PROCESSING sample for Azure Functions Flex Consumption when reviewing this PR so you can use that PR (or the sample from brain if the PR is already merged by the time of reviewing this).

Get the code of the sample down, run azd up, test the app as per readme, then make a small code change and run azd up again.

Expected behavior
Repeated azd up should correctly re-deploy the app.

[weikanglim]

I can see the resource is tagged correctly in the Bicep file. Will need to investigate further.

[nzthiago]

@weikanglim - I have just run into this one again, this time with the https://github.com/Azure-Samples/azure-functions-java-flex-consumption-azd sample. I had run just earlier that sample with SKIP_VNET set to true. Then I did an 'azd down' and deleted the .azure folder. Then tried azd up again but this time without setting SKIP_VNET. Got this:

Then I ran azd up again and this time it publishes fine:

It does seem like there's something strange with the tag search.

[weikanglim]

@nzthiago Thanks for reporting the second instance of the issue. I apologize but I haven't had a chance to take a real look at this.

I'm wondering if something along the following is happening:

1. For the second instance, it may be perhaps that there is a delay in ARM resource tags actually showing up when queried for after provisioning (we have not seen this reported widely though, which perhaps makes this unlikely).
2. There may be some automation in the subscription that is somehow updating tags outside of Bicep
3. There may be something related to flex-consumption functions that updates the tags outside of Bicep

I'd recommend for debugging purposes, when this happens again, to leave the Azure resources as-is in their state. Perhaps if we looked at the Activity log of the resource in the Portal, we can see if there are updates being initiated outside of the Bicep provisioning and see if tags are being modified.

[nzthiago]

I'll remember to keep the assets around next time I run into this, it does happen from time to time so it's not an isolated thing.

[rajeshkamal5050]

@weikanglim moving to backlog. Once, we get a repro/environment please pull this back into the sprint.

[nzthiago]

I have a current instance of this issue. In the resource group activity log I see this entry, that seems to removing both tags and properties.tags from the Microsoft.Web/sites resource?

The "Event initiated by" shows as empty for that specific "UpdateWebSite" operation. So does the CreateWebSite which was initiated by AZD:

[weikanglim]

@nzthiago In your Bicep files for this particular deployment, is httpsOnly specified at all?

I really do think the policy that is tied to the actor CloudGovRemediationWe.. is the likely culprit here. Investigation notes below:

1. The update on 2:05 pm PDT only set httpsOnly from false to true. While doing this, it also cleared the tags that was set on the resources.
2. The update on 2:05pm PDT is 10 minutes after the app settings update on 1:55pm PDT.

From the second screenshot, I am unable to correlate the events, but based on the how recent column named "Time", I can see that:

1. UpdateWebSite as you highlighted from azd happened at 42 minutes ago
2. UpdateWebsite from CloudGovRemediationWe.. happened at 32 minutes ago

which corresponds the 10 minutes delay interval we see from screenshot 1.

This leads me to believe that CloudGovRemediationWe.. is the policy that's causing the behavior you're seeing. A likely fix is to just set httpsOnly=true to avoid the policy's problematic update, while following up with the owning team to make sure they're preserving tags.

As a foot note, I do think you're seeing an issue as a result of azd's resource discovery mechanism relying on resource tagging -- which is susceptible to the unfortunate underlying issues that you're seeing. I think it'd be best if resources discovery just happened as part of provisioning, rather than leveraging metadata being set on these resources.

[nzthiago]

@weikanglim - Aha, you might have found the reason why this is happening then! Looks like the policies being auto applied by that application are clearing the tags from the app. I will try figuring out who owns that policy/entra app to let them know. But glad to finally understand why this was happening.

I agree that this resource discovery based on tags might not be the best approach, perhaps a way to do it by actual resource id would be better. Happy for you to close this thread this as we found the root cause, but feel free to start a separate discussion on that. Thanks!!