-
Notifications
You must be signed in to change notification settings - Fork 177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial scripts and pipeline for stress test discovery/build/push #1851
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
f3e7bb0
Initial scripts and pipeline for stress test discovery/build/push
benbp b1569a4
Improve stress test deploy script command line flags and execution model
benbp 78266be
Update stress test release pipeline arguments
benbp c5d494a
Fixes to get everything working
benbp 368bf5c
Check in stress deployment example compiled test-resources.json
benbp 5ae04d8
Change stress test script filename convention
benbp 7a239ff
Function naming and output improvements
benbp c2dd8b2
Move stress testing scripts to eng/common
benbp File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
174 changes: 174 additions & 0 deletions
174
eng/common/scripts/stress-testing/deploy-stress-tests.ps1
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,174 @@ | ||
| [CmdletBinding(DefaultParameterSetName = 'Default')] | ||
| param( | ||
| [string]$SearchDirectory, | ||
| [hashtable]$Filters, | ||
| [string]$Environment, | ||
| [string]$Repository, | ||
| [switch]$PushImages, | ||
| [string]$ClusterGroup, | ||
| [string]$DeployId, | ||
|
|
||
| [Parameter(ParameterSetName = 'DoLogin', Mandatory = $true)] | ||
| [switch]$Login, | ||
|
|
||
| [Parameter(ParameterSetName = 'DoLogin')] | ||
| [string]$Subscription | ||
| ) | ||
|
|
||
| $ErrorActionPreference = 'Stop' | ||
|
|
||
| . $PSScriptRoot/find-all-stress-packages.ps1 | ||
| $FailedCommands = New-Object Collections.Generic.List[hashtable] | ||
|
|
||
| if (!(Get-Module powershell-yaml)) { | ||
| Install-Module -Name powershell-yaml -RequiredVersion 0.4.1 -Force -Scope CurrentUser | ||
| } | ||
|
|
||
| # Powershell does not (at time of writing) treat exit codes from external binaries | ||
| # as cause for stopping execution, so do this via a wrapper function. | ||
| # See https://github.com/PowerShell/PowerShell-RFC/pull/277 | ||
| function Run() { | ||
| Write-Host "`n==> $args`n" -ForegroundColor Green | ||
| $command, $arguments = $args | ||
| & $command $arguments | ||
| if ($LASTEXITCODE) { | ||
| Write-Error "Command '$args' failed with code: $LASTEXITCODE" -ErrorAction 'Continue' | ||
| $FailedCommands.Add(@{ command = "$args"; code = $LASTEXITCODE }) | ||
| } | ||
| } | ||
|
|
||
| function RunOrExitOnFailure() { | ||
| run @args | ||
| if ($LASTEXITCODE) { | ||
| exit $LASTEXITCODE | ||
| } | ||
| } | ||
|
|
||
| function Login([string]$subscription, [string]$clusterGroup, [boolean]$pushImages) { | ||
| Write-Host "Logging in to subscription, cluster and container registry" | ||
| az account show *> $null | ||
| if ($LASTEXITCODE) { | ||
| RunOrExitOnFailure az login --allow-no-subscriptions | ||
| } | ||
|
|
||
| $clusterName = (az aks list -g $clusterGroup -o json| ConvertFrom-Json).name | ||
|
|
||
| RunOrExitOnFailure az aks get-credentials ` | ||
| -n "$clusterName" ` | ||
| -g "$clusterGroup" ` | ||
| --subscription "$subscription" ` | ||
| --overwrite-existing | ||
|
|
||
| if ($pushImages) { | ||
| $registry = (az acr list -g $clusterGroup -o json | ConvertFrom-Json).name | ||
| RunOrExitOnFailure az acr login -n $registry | ||
| } | ||
| } | ||
|
|
||
| function DeployStressTests( | ||
| [string]$searchDirectory = '.', | ||
| [hashtable]$filters = @{}, | ||
| [string]$environment = 'test', | ||
| [string]$repository = 'images', | ||
| [boolean]$pushImages = $false, | ||
| [string]$clusterGroup = 'rg-stress-test-cluster-', | ||
| [string]$deployId = 'local', | ||
| [string]$subscription = 'Azure SDK Test Resources' | ||
| ) { | ||
| if ($PSCmdlet.ParameterSetName -eq 'DoLogin') { | ||
| Login $subscription $clusterGroup $pushImages | ||
| } | ||
|
|
||
| RunOrExitOnFailure helm repo add stress-test-charts https://stresstestcharts.blob.core.windows.net/helm/ | ||
| Run helm repo update | ||
| if ($LASTEXITCODE) { return $LASTEXITCODE } | ||
|
|
||
| $pkgs = FindStressPackages $searchDirectory $filters | ||
| Write-Host "" "Found $($pkgs.Length) stress test packages:" | ||
| Write-Host $pkgs.Directory "" | ||
| foreach ($pkg in $pkgs) { | ||
| Write-Host "Deploying stress test at '$($pkg.Directory)'" | ||
| DeployStressPackage $pkg $deployId $environment $repository $pushImages | ||
| } | ||
|
|
||
| Write-Host "Releases deployed by $deployId" | ||
| Run helm list --all-namespaces -l deployId=$deployId | ||
|
|
||
| if ($FailedCommands) { | ||
| Write-Warning "The following commands failed:" | ||
| foreach ($cmd in $FailedCommands) { | ||
| Write-Error "'$($cmd.command)' failed with code $($cmd.code)" -ErrorAction 'Continue' | ||
| } | ||
| exit 1 | ||
| } | ||
| } | ||
|
|
||
| function DeployStressPackage( | ||
| [object]$pkg, | ||
| [string]$deployId, | ||
| [string]$environment, | ||
| [string]$repository, | ||
| [boolean]$pushImages | ||
| ) { | ||
| $registry = (az acr list -g $clusterGroup -o json | ConvertFrom-Json).name | ||
| if (!$registry) { | ||
| Write-Host "Could not find container registry in resource group $clusterGroup" | ||
| exit 1 | ||
| } | ||
|
|
||
| if ($pushImages) { | ||
| Run helm dependency update $pkg.Directory | ||
| if ($LASTEXITCODE) { return $LASTEXITCODE } | ||
|
|
||
| $dockerFiles = Get-ChildItem "$($pkg.Directory)/Dockerfile*" | ||
| foreach ($dockerFile in $dockerFiles) { | ||
| # Infer docker image name from parent directory name, if file is named `Dockerfile` | ||
| # or from suffix, is file is named like `Dockerfile.myimage` (for multiple dockerfiles). | ||
| $prefix, $imageName = $dockerFile.Name.Split(".") | ||
| if (!$imageName) { | ||
| $imageName = $dockerFile.Directory.Name | ||
| } | ||
| $imageTag = "${registry}.azurecr.io/$($repository.ToLower())/$($imageName):$deployId" | ||
| Write-Host "Building and pushing stress test docker image '$imageTag'" | ||
| Run docker build -t $imageTag -f $dockerFile.FullName $dockerFile.DirectoryName | ||
| if ($LASTEXITCODE) { return $LASTEXITCODE } | ||
| Run docker push $imageTag | ||
| if ($LASTEXITCODE) { | ||
| if ($PSCmdlet.ParameterSetName -ne 'DoLogin') { | ||
| Write-Warning "If docker push is failing due to authentication issues, try calling this script with '-Login'" | ||
| } | ||
| return $LASTEXITCODE | ||
| } | ||
| } | ||
| } | ||
|
|
||
| Write-Host "Creating namespace $($pkg.Namespace) if it does not exist..." | ||
| kubectl create namespace $pkg.Namespace --dry-run=client -o yaml | kubectl apply -f - | ||
|
|
||
| Write-Host "Installing or upgrading stress test $($pkg.ReleaseName) from $($pkg.Directory)" | ||
| Run helm upgrade $pkg.ReleaseName $pkg.Directory ` | ||
| -n $pkg.Namespace ` | ||
| --install ` | ||
| --set repository=$registry.azurecr.io/$repository ` | ||
| --set tag=$deployId ` | ||
| --set stress-test-addons.env=$environment | ||
| if ($LASTEXITCODE) { | ||
| # Issues like 'UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress' | ||
| # can be the result of cancelled `upgrade` operations (e.g. ctrl-c). | ||
| # See https://github.com/helm/helm/issues/4558 | ||
| Write-Warning "The issue may be fixable by first running 'helm rollback -n $($pkg.Namespace) $($pkg.ReleaseName)'" | ||
| return $LASTEXITCODE | ||
| } | ||
|
|
||
| # Helm 3 stores release information in kubernetes secrets. The only way to add extra labels around | ||
| # specific releases (thereby enabling filtering on `helm list`) is to label the underlying secret resources. | ||
| # There is not currently support for setting these labels via the helm cli. | ||
| $helmReleaseConfig = kubectl get secrets ` | ||
| -n $pkg.Namespace ` | ||
| -l status=deployed,name=$($pkg.ReleaseName) ` | ||
| -o jsonpath='{.items[0].metadata.name}' | ||
|
|
||
| Run kubectl label secret -n $pkg.Namespace --overwrite $helmReleaseConfig deployId=$deployId | ||
| } | ||
|
|
||
| DeployStressTests @PSBoundParameters |
53 changes: 53 additions & 0 deletions
53
eng/common/scripts/stress-testing/find-all-stress-packages.ps1
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| param( | ||
| [string]$searchDirectory = '.', | ||
| [hashtable]$filters = @{} | ||
| ) | ||
|
|
||
| class StressTestPackageInfo { | ||
| [string]$Namespace | ||
| [string]$Directory | ||
| [string]$ReleaseName | ||
| } | ||
|
|
||
| function FindStressPackages([string]$directory, [hashtable]$filters = @{}) { | ||
| # Bare minimum filter for stress tests | ||
| $filters['stressTest'] = 'true' | ||
|
|
||
| $packages = @() | ||
| $chartFiles = Get-ChildItem -Recurse -Filter 'Chart.yaml' $directory | ||
| foreach ($chartFile in $chartFiles) { | ||
| $chart = ParseChart $chartFile | ||
| if (matchesAnnotations $chart $filters) { | ||
| $packages += NewStressTestPackageInfo $chart $chartFile | ||
| } | ||
| } | ||
|
|
||
| return $packages | ||
| } | ||
|
|
||
| function ParseChart([string]$chartFile) { | ||
| return ConvertFrom-Yaml (Get-Content -Raw $chartFile) | ||
| } | ||
|
|
||
| function MatchesAnnotations([hashtable]$chart, [hashtable]$filters) { | ||
| foreach ($filter in $filters.GetEnumerator()) { | ||
| if (!$chart.annotations -or $chart.annotations[$filter.Key] -ne $filter.Value) { | ||
| return $false | ||
| } | ||
| } | ||
|
|
||
| return $true | ||
| } | ||
|
|
||
| function NewStressTestPackageInfo([hashtable]$chart, [System.IO.FileInfo]$chartFile) { | ||
| return [StressTestPackageInfo]@{ | ||
| Namespace = $chart.annotations.namespace | ||
| Directory = $chartFile.DirectoryName | ||
| ReleaseName = $chart.name | ||
| } | ||
| } | ||
|
|
||
| # Don't call functions when the script is being dot sourced | ||
| if ($MyInvocation.InvocationName -ne ".") { | ||
| FindStressPackages $searchDirectory $filters | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| pr: none | ||
|
|
||
| trigger: none | ||
|
|
||
| parameters: | ||
| - name: Subscription | ||
| type: string | ||
| default: 'Azure SDK Test Resources' | ||
| - name: Environment | ||
| type: string | ||
| default: prod | ||
| - name: ClusterGroup | ||
| type: string | ||
| default: rg-stress-test-cluster-prod | ||
| - name: TestRepository | ||
| displayName: Stress Test Repository | ||
| type: string | ||
| default: all | ||
| values: | ||
| - all | ||
| - examples | ||
| - javascript | ||
| - java | ||
| - net | ||
| - python | ||
| - go | ||
|
|
||
| jobs: | ||
| - job: | ||
| strategy: | ||
| matrix: | ||
| ${{ if or(eq(parameters.TestRepository, 'examples'), eq(parameters.TestRepository, 'all')) }}: | ||
| examples: | ||
| Repository: Azure/azure-sdk-tools | ||
| Filters: '@{ "example" = "true" }' | ||
| ${{ if or(eq(parameters.TestRepository, 'javascript'), eq(parameters.TestRepository, 'all')) }}: | ||
| javascript: | ||
| Repository: Azure/azure-sdk-for-js | ||
| Filters: '@{}' | ||
| ${{ if or(eq(parameters.TestRepository, 'java'), eq(parameters.TestRepository, 'all')) }}: | ||
| java: | ||
| Repository: Azure/azure-sdk-for-java | ||
| Filters: '@{}' | ||
| ${{ if or(eq(parameters.TestRepository, 'net'), eq(parameters.TestRepository, 'all')) }}: | ||
| net: | ||
| Repository: Azure/azure-sdk-for-net | ||
| Filters: '@{}' | ||
| ${{ if or(eq(parameters.TestRepository, 'python'), eq(parameters.TestRepository, 'all')) }}: | ||
| python: | ||
| Repository: Azure/azure-sdk-for-python | ||
| Filters: '@{}' | ||
| ${{ if or(eq(parameters.TestRepository, 'go'), eq(parameters.TestRepository, 'all')) }}: | ||
| go: | ||
| Repository: Azure/azure-sdk-for-go | ||
| Filters: '@{}' | ||
| pool: | ||
| vmImage: 'ubuntu-20.04' | ||
| #name: 'azsdk-pool-mms-ubuntu-2004-general' | ||
| #vmImage: 'MMSUbuntu20.04' | ||
| steps: | ||
| - template: /eng/common/pipelines/templates/steps/sparse-checkout.yml | ||
| parameters: | ||
| Repositories: | ||
| - Name: Azure/azure-sdk-tools | ||
| Commitish: $(Build.SourceVersion) | ||
| WorkingDirectory: $(System.DefaultWorkingDirectory)/Azure/azure-sdk-tools | ||
| - Name: $(Repository) | ||
| Commitish: $(Build.SourceVersion) | ||
| WorkingDirectory: $(System.DefaultWorkingDirectory)/$(Repository) | ||
| Paths: | ||
| - '/tools' | ||
| - '!sdk/**/test-recordings' | ||
| - '!sdk/**/session-records' | ||
| - '!sdk/**/SessionRecords' | ||
|
|
||
| - task: AzureCLI@2 | ||
| displayName: Build and Deploy Stress Tests | ||
| inputs: | ||
| azureSubscription: ${{ parameters.Subscription }} | ||
| scriptType: pscore | ||
| scriptPath: $(System.DefaultWorkingDirectory)/Azure/azure-sdk-tools/eng/common/scripts/stress-testing/deploy-stress-tests.ps1 | ||
| arguments: | ||
| -SearchDirectory '$(System.DefaultWorkingDirectory)/$(Repository)' | ||
| -Filters $(Filters) | ||
| -Environment '${{ parameters.Environment }}' | ||
| -Repository '$(Agent.JobName)' | ||
| -PushImages | ||
| -ClusterGroup '${{ parameters.ClusterGroup }}' | ||
| -Login | ||
| -Subscription '${{ parameters.Subscription }}' | ||
| -DeployId '$(Build.BuildNumber)' | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
9 changes: 7 additions & 2 deletions
9
...xamples/network_stress_example/Chart.yaml → ...xamples/network-stress-example/Chart.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
8 changes: 8 additions & 0 deletions
8
tools/stress-cluster/chaos/examples/network-stress-example/Dockerfile
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| FROM alpine:3.14 | ||
| RUN apk add --no-cache wget | ||
| RUN apk add --no-cache bash | ||
|
|
||
| ADD ./poll.sh /poll.sh | ||
| RUN chmod +x /poll.sh | ||
|
|
||
| CMD bash /poll.sh |
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What value are we gaining by using the AzureCLI task here? Is the service connection information enough for us to do all the deployments or don't we need to stull pass in most of the secrets? Also what about other clouds do they need other configurations or is that not a scenario at this point.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I hadn't gotten to that point yet but my thinking was to add an access policy to the buildout bicep configs to allowlist one of the devops service connections (that way there are no manual steps on buildout). I may still create a new identity just for this.
I think it's unlikely that we'll target other clouds, as our aim here is to stress the client code, rather than to test its operability across clouds. Though if we do have a need for that, I don't think we'll ever need to host the stress cluster itself in another cloud, so any relevant credentials are not configured in this context (they are auto-synced into the cluster via keyvault).