[Feature Request]: Introduce template tests for idempotency

[AlexanderSehr]

Description

To further validate our templates, we could introduce an 'idempotency' test. This effectively means that we'd need to update our deployment script to deploy the same template at least one more time following a preceeding successful deployment.

Note: This will likely lead to some pipelines failing as not all services / their features are idempontent. Synapse Workspace encryption comes to mind.
Following the implementation, we should run the tests for all pipelines to identify those modules and try and find a reasonable solution.

Note: Probably the trickiest part about the implementation will be to properly work together with the already existing 'retry' logic.

[eriqua]

We should also consider gh hosted runners job duration limitations (6h) for services taking multiple hours to deploy (e.g., sqlmi) although, if idempotent, the second deployment would likely take a limited time to just check that all resources are deployed already.
In case of canceled jobs, the deployment may continue in Azure without it being logged.

[AlexanderSehr]

We should also consider gh hosted runners job duration limitations (6h) for services taking multiple hours to deploy (e.g., sqlmi) although, if idempotent, the second deployment would likely take a limited time to just check that all resources are deployed already. In case of canceled jobs, the deployment may continue in Azure without it being logged.

Agreed. Something we can maybe solve by not running the 2nd deployment in the same job