-
Notifications
You must be signed in to change notification settings - Fork 5.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add apitestErrorCode doc #14749
Merged
Merged
add apitestErrorCode doc #14749
Changes from all commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
dfbaeb2
add apitestErrorCode doc
ruowan 3c69568
add armRPC doc
ruowan d97b4e6
add roundTripInconsistentProperty doc
ruowan 29b0c02
small fix
ruowan 27ab05e
update
ruowan 01cdf73
update signalR armTemplate
ruowan 7fdac41
update yaml
ruowan ffd67dc
new file
ruowan cfa5608
add testScenario with armTemplate
ruowan 7d632d1
update doc
ruowan 5a4f6d7
update readme.md
ruowan ca65d70
update doc
ruowan bd9c0ca
update NOTE section
ruowan 9c1cf46
add signalRCreateOrUpdate example file
ruowan 882668a
udpate doc
ruowan f4c3994
add generate test scenario section
ruowan 90618cc
update doc
ruowan 75535cb
add serviceFacbric test scenario file
ruowan 5c6dc7d
update managedClusters.yaml
ruowan b0af73c
update generateABasicTestScenario.md
ruowan 983d796
update features doc
ruowan 6a5f970
add run api test gif
ruowan File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,123 @@ | ||
<!-- | ||
Copyright (c) 2021 Microsoft Corporation | ||
|
||
This software is released under the MIT License. | ||
https://opensource.org/licenses/MIT | ||
--> | ||
|
||
# API testing error code | ||
|
||
OAV api testing define some rules and check whether actual service response match with example. Example is very important for downstream SDK test code generation, code samples and docs. The goal to detect example quality issues and check service behaviors consistent with example. | ||
|
||
## Rule descriptions | ||
|
||
#### INCORRECT_PROVISIONING_STATE | ||
|
||
If service return 200 status code,it means the operation is succeed and finished. So the provisioning state should be one of the terminal states ["succeeded", "failed", "canceled", "ready", "created", "deleted"]. | ||
|
||
The provisioning state is very important for downstream terraform or cli to manage resource status. For more details about provisioning state, please refer to this [ARM RPC provisioning state](https://github.com/Azure/azure-resource-manager-rpc/blob/master/v1.0/Addendum.md#provisioningstate-property) | ||
|
||
#### RESPONSE_MISSING_VALUE | ||
|
||
**Error message**: The response value is missing. Path: {}. Expected: {}. Actual: undefined | ||
|
||
The example has defined response value, but actually the server doesn't return that value. | ||
|
||
Example: | ||
|
||
```diff | ||
{ | ||
"properties":{ | ||
"targetType":"blobNfs", | ||
"junctions":[ | ||
{ | ||
"namespacePath":"/blobnfs" | ||
} | ||
], | ||
"blobNfs":{ | ||
"target":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/scgroup/providers/Microsoft.Storage/storageAccounts/blofnfs/blobServices/default/containers/blobnfs", | ||
- "usageModel":"WRITE_WORKLOAD_15" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. suggest to show what is actual response and what correct should be in 2 example There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sure. Will be covered in next PR. |
||
} | ||
} | ||
} | ||
``` | ||
|
||
#### RESPONSE_ADDITIONAL_VALUE | ||
|
||
**Error message** Return additional response value. Path: {}. Expected: undefined. Actual: {} | ||
|
||
The example doesn't define the response value, but service actually return this value. | ||
|
||
Example: | ||
|
||
```diff | ||
{ | ||
"properties":{ | ||
"targetType":"blobNfs", | ||
"junctions":[ | ||
{ | ||
"namespacePath":"/blobnfs" | ||
} | ||
], | ||
"blobNfs":{ | ||
"target":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/scgroup/providers/Microsoft.Storage/storageAccounts/blofnfs/blobServices/default/containers/blobnfs", | ||
"usageModel":"WRITE_WORKLOAD_15" | ||
+ "enableFeature": true | ||
} | ||
} | ||
} | ||
``` | ||
|
||
#### RESPONSE_INCONSISTENT_VALUE | ||
|
||
**Error message** The actual response value is different from example. Path: {}. Expected: {}. Actual: {} | ||
|
||
The service returned value is different from example value. | ||
|
||
Example: | ||
|
||
```diff | ||
{ | ||
"properties":{ | ||
"targetType":"blobNfs", | ||
"junctions":[ | ||
{ | ||
"namespacePath":"/blobnfs" | ||
} | ||
], | ||
"blobNfs":{ | ||
"target":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/scgroup/providers/Microsoft.Storage/storageAccounts/blofnfs/blobServices/default/containers/blobnfs", | ||
- "usageModel":"WRITE_WORKLOAD_15" | ||
+ "usageModel":"WORK_LOAD_14" | ||
} | ||
} | ||
} | ||
``` | ||
|
||
#### ROUNDTRIP_INCONSISTENT_PROPERTY | ||
|
||
**Error message** The property's value in the response is different from what was set in the request. Path: {}. Request: {}. Response: {} | ||
|
||
Example: The sku in request parameters is `default`, but actual return is `standard`. | ||
|
||
```diff | ||
{ | ||
"parameters":{ | ||
"properties":{ | ||
"name":"myService", | ||
"SKU":"default" | ||
} | ||
}, | ||
"responses":{ | ||
"200":{ | ||
"properties":{ | ||
"name":"myService", | ||
+ "SKU":"standard" | ||
- "SKU":"default" | ||
|
||
} | ||
} | ||
} | ||
} | ||
|
||
``` |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
# Generate a basic test scenario file | ||
|
||
## Prerequisite | ||
|
||
We use `oav` tools to generate basic test scenario. `oav` analyze swagger file and use swagger example as test scenario steps. | ||
|
||
## Introduction | ||
|
||
`oav` support rule based test scenario file generation. We use this command to generate test scenario file. | ||
|
||
`oav generate-static-test-scenario --readme <readme> --tag <tag> --rules <generated-rules>` | ||
|
||
- readme: swagger readme file. | ||
- tag: which tag to generate. oav will analyze swagger file under the tag and generate test scenario. | ||
|
||
Example: | ||
|
||
![](./genTestScenario.gif) | ||
|
||
This command will load and analyze swagger and generate a basic test scenario file. | ||
|
||
Result: the output contains two files | ||
|
||
- test-scenarios/signalR.yaml: The test scenario file. | ||
- readme.test.md: The entry for SDK test generation | ||
|
||
The generated test scenario file: The generated test scenario file contains two steps. Create signalR and delete it. It's a basic test scenario and developer can add more step based on the basic test scenario file. | ||
|
||
``` | ||
scope: ResourceGroup | ||
testScenarios: | ||
- description: Microsoft.SignalRService/signalR SignalR_CreateOrUpdate | ||
steps: | ||
- step: SignalR_CreateOrUpdate | ||
exampleFile: ../examples/SignalR_CreateOrUpdate.json | ||
- step: SignalR_Delete | ||
exampleFile: ../examples/SignalR_Delete.json | ||
``` | ||
|
||
## Reference | ||
|
||
- [oav](https://github.com/Azure/oav/tree/develop) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
104 changes: 104 additions & 0 deletions
104
documentation/test-scenario/testScenarioWithARMTemplate.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,104 @@ | ||
# Test scenario integrate with armTemplate | ||
|
||
## Background | ||
|
||
In some cases, we need to do some more complex operations before creating a resource. For example, | ||
|
||
- Create Azure SignalR service with a global unique name. Using ARMTemplate to generate a random unique string. | ||
- Create VM with a storage account. Using ARMTemplate to provision storage account and passing the storage account resourceId as VM creation parameter. | ||
|
||
## Examples | ||
|
||
Here is an example about `generate unique resource name for signalR service` | ||
|
||
#### Generate unique resource name | ||
|
||
We use `armTemplate output` to overwrite `resourceName` variable and following `createResource` step will use this variable. Below is generate unique name armTemplate. This armTemplate output `resourceName` variables, so test scenario following step will using the output variable. | ||
|
||
```json | ||
{ | ||
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", | ||
"contentVersion": "1.0.0.0", | ||
"parameters": { | ||
"resourceNamePrefix": { | ||
"type": "string", | ||
"defaultValue": "signalr-" | ||
} | ||
}, | ||
"variables": { | ||
"resourceName": "[concat(parameters('resourceNamePrefix'), uniqueString(resourceGroup().id))]" | ||
}, | ||
"resources": [], | ||
"outputs": { | ||
"resourceName": { | ||
"type": "string", | ||
"value": "[variables('resourceName')]" | ||
} | ||
} | ||
} | ||
``` | ||
|
||
After we have this armTemplate, we could define current test scenario file. We defined `resourceName` variable globally. `./generate_unique_string.json` is armTemplate. | ||
|
||
`SignalR_CreateOrUpdate.json` | ||
|
||
```json | ||
{ | ||
"parameters": { | ||
"parameters": { | ||
"tag": { | ||
"key1": "tag1" | ||
}, | ||
"properties":{ | ||
... | ||
} | ||
}, | ||
"api-version": "2020-07-01-preview", | ||
"subscriptionId": "00000000-0000-0000-0000-000000000000", | ||
"resourceGroupName": "myResourceGroup", | ||
"resourceName": "mySignalRService123xx" | ||
}, | ||
"responses": { | ||
"200": { | ||
"body": { | ||
... | ||
} | ||
}, | ||
"201": { | ||
"body": { | ||
... | ||
} | ||
}, | ||
"202": {} | ||
} | ||
} | ||
``` | ||
|
||
> NOTE: the example file `../examples/SignalR_CreateOrUpdate` has the same parameter name `resourceName`. So it will be automatically overwrite in runner. | ||
|
||
```yaml | ||
scope: ResourceGroup | ||
variables: | ||
resourceName: "" | ||
testScenarios: | ||
- description: Microsoft.SignalRService/signalR CRUD | ||
steps: | ||
- step: Generate_Unique_string | ||
armTemplateDeployment: ./generate_unique_string.json | ||
- step: SignalR_checknameAvailability | ||
exampleFile: ../examples/SignalR_CheckNameAvailability.json | ||
- step: SignalR_CreateOrUpdate | ||
exampleFile: ../examples/SignalR_CreateOrUpdate.json | ||
- step: SignalR_ListKey | ||
exampleFile: ../examples/SignalR_ListKeys.json | ||
- step: SignalR_Delete | ||
exampleFile: ../examples/SignalR_Delete.json | ||
``` | ||
|
||
**Result**: | ||
|
||
![](./armTemplate.png) | ||
|
||
## Reference | ||
|
||
- [ARMTemplate deployment script](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-script-template) |
2 changes: 1 addition & 1 deletion
2
...e-manager/Microsoft.ServiceBus/preview/2021-01-01-preview/examples/SBOperations_List.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,6 @@ | ||
{ | ||
"parameters": { | ||
"api-version": "2021-01-01-preview" | ||
"api-version": "2017-04-01" | ||
}, | ||
"responses": { | ||
"200": { | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
14 changes: 14 additions & 0 deletions
14
...er/Microsoft.ServiceFabric/preview/2020-01-01-preview/test-scenarios/managedClusters.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
scope: ResourceGroup | ||
testScenarios: | ||
- description: >- | ||
Microsoft.ServiceFabric/managedClusters Put a cluster with minimum | ||
parameters | ||
steps: | ||
- step: Put a cluster with minimum parameters | ||
exampleFile: ../examples/ManagedClusterPutOperation_example_min.json | ||
- step: Delete a cluster | ||
exampleFile: ../examples/ManagedClusterDeleteOperation_example.json | ||
- step: Put a cluster with maximum parameters | ||
exampleFile: ../examples/ManagedClusterPutOperation_example_max.json | ||
- step: Delete a cluster | ||
exampleFile: ../examples/ManagedClusterDeleteOperation_example.json |
12 changes: 12 additions & 0 deletions
12
...c/resource-manager/Microsoft.ServiceFabric/stable/2020-03-01/test-scenarios/clusters.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
scope: ResourceGroup | ||
testScenarios: | ||
- description: Microsoft.ServiceFabric/clusters Put a cluster with minimum parameters | ||
steps: | ||
- step: Put a cluster with minimum parameters | ||
exampleFile: ../examples/ClusterPutOperation_example_min.json | ||
- step: Delete a cluster | ||
exampleFile: ../examples/ClusterDeleteOperation_example.json | ||
- step: Put a cluster with maximum parameters | ||
exampleFile: ../examples/ClusterPutOperation_example_max.json | ||
- step: Delete a cluster | ||
exampleFile: ../examples/ClusterDeleteOperation_example.json |
10 changes: 10 additions & 0 deletions
10
specification/servicefabric/resource-manager/readme.test.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
|
||
### Tag: package-2020-03 | ||
|
||
These settings apply only when `--tag=package-2020-03` is specified on the command line. | ||
|
||
```yaml $(tag) == 'package-2020-03' | ||
test-resources: | ||
- test: Microsoft.ServiceFabric/stable/2020-03-01/test-scenarios/clusters.yaml | ||
- test: Microsoft.ServiceFabric/preview/2020-01-01-preview/test-scenarios/managedClusters.yaml | ||
``` |
8 changes: 8 additions & 0 deletions
8
...e-manager/Microsoft.SignalRService/preview/2018-03-01-preview/test-scenarios/SignalR.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
scope: ResourceGroup | ||
testScenarios: | ||
- description: Microsoft.SignalRService/SignalR CreateOrUpdate | ||
steps: | ||
- step: CreateOrUpdate | ||
exampleFile: ../examples/CreateOrUpdate.json | ||
- step: Delete | ||
exampleFile: ../examples/Delete.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a great starting point we have these error codes, can we add a section focus on each rule to describe how to fix? It's good to be more specific to guide service team to fix them, and it seems missing x-ms-secret or readonly are common issues spotting from the pilot
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. Maybe I need to write a doc about validation logic. It will be covered in next PR.