Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CircleCI + Performance Regression #925

Open
wants to merge 408 commits into
base: master
Choose a base branch
from
Open

CircleCI + Performance Regression #925

wants to merge 408 commits into from

Conversation

prashantsail
Copy link
Contributor

@prashantsail prashantsail commented Jul 2, 2020 •
edited
Loading

Description of changes:

Primary goal of this PR is -

  1. Integration of performance regression suite with CircleCI
    • Add new performance regression job in circleci config
    • Add corresponding shell script for job execution
    • Added new ci_linux_medium environment for comparing against thresholds specific to CircleCI env.
      • Renamed xlarge environment to macos_xlarge (more specific)

Along with the primary goal, we have identified and fixed following -

  1. Movement of API tests(newman) suite from test/ to tests/api/
  2. CircleCI's API tests(newman) job now continues with execution of all 3 collections(management, inference, https), even if any one of them fails.
  3. CircleCI's shell scripts tweaked to exit with error when any of the important commands fail.
  4. CircleCI Jobs now copy the entire MMS's logs directory along with console logs (instead of only copying mms console logs)
  5. Removed - /performance_regression. The directory was part of POC and is no longer required.

All Jobs are working and are successful after these changes -
Build #443
Screenshot 2020-07-09 at 15 05 33

Performance regression Job
Performance regression logs and results

Note:
The commit count is high because the previous PRs of CircleCI and Performance regression have been squashed & merged.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

prashantsail and others added 30 commits June 19, 2020 18:03
@prashantsail
Merge branch 'performance' of https://github.com/maheshambule/multi-m…
35affa7 
…odel-server into circleci-perf-regression

# Conflicts:
#	tests/performance/run_perfomance_suite.py
@prashantsail
updated file name in script
5643e8d
@prashantsail
register unregister scenario
7629ad3
@prashantsail
Updated TESTS md
7bcacb3
@prashantsail
Merge pull request #8 from prashantsail/ps_performance
79dac3a 
register unregister scenario
@maheshambule
refactor
0db993c
@maheshambule
Merge branch 'performance' of https://github.com/maheshambule/multi-m…
4fd48cf 
…odel-server into test1
@prashantsail
model description scenario
69ddfbc
@maheshambule
compare runs
e0776df
@maheshambule
remove files
6b535b3
@maheshambule
change in ini
07780b9
@maheshambule
path name cahnge
bee14d1
@maheshambule
Merge pull request #9 from maheshambule/test1
53a5747 
Test1
@prashantsail
list models scenario
7ac1cfc
@maheshambule
requirement.txt changes
825f825
@maheshambule
boto3 req.txt
04b3661
@prashantsail
api description and health check scenario
e07c438
@prashantsail
Updated test plan names
5ca9e96
@prashantsail
Merge pull request #10 from prashantsail/ps_performance
04442cf 
Ps performance
@prashantsail
Merge branch 'performance' of https://github.com/maheshambule/multi-m…
810cbb2 
…odel-server into circleci-perf-regression
@prashantsail
added missing threshold
9282f28
@prashantsail
Merge pull request #11 from prashantsail/ps_performance
d44ed67 
added missing threshold
@dhruvaray
polished README.md
9c25f8b 
Signed-off-by: Dhruva Ray <[email protected]>
@dhruvaray
Merge branch 'performance' of https://github.com/maheshambule/multi-m…
d1a7958 
…odel-server into performance
@maheshambule
changes
5b193b9
@maheshambule
Merge branch 'performance' of https://github.com/maheshambule/multi-m…
08c486b 
…odel-server into performance
@prashantsail
Update README.md
1dc558e
@maheshambule
folder
51c7f01
@maheshambule
fix issue
b405877
@prashantsail
Revert breaking changes
667738e
prashantsail and others added 28 commits July 2, 2020 18:45
@prashantsail
tweaked thresholds for cci
bc96274
@prashantsail
Merge branch 'performance' of https://github.com/maheshambule/multi-m…
ba2b83b 
…odel-server into circleci-perf-regression
@prashantsail
tweaked thresholds for circleci environment
dab2d50
@prashantsail
performance regression only on py36
99687a3
@prashantsail
performance regression only on py36
96e40bd
@prashantsail
unregister while inferencing moved to examples
b41186e
@prashantsail
tweaked thresholds for ci_linux_medium
8cf7620
@prashantsail
tweaked TCs
9d9cc6a
@prashantsail
tweaked thresholds for circleci runs
c0be394
@prashantsail
tweaked thresholds for circleci runs
a97d9de
@prashantsail
Tweaked metric thresholds for circleci
450bde2
@prashantsail
revert config.yaml to optimal config
1b5a219
@maaquib
Merge branch 'master' into performance
3eca7e5
@prashantsail
Ps ci perf (#13)
ca92cd1 
tweaks to run on py27 environment
@prashantsail
Merge branch 'performance' of https://github.com/maheshambule/multi-m…
711a70c 
…odel-server into circleci-perf-regression
@prashantsail
API tests execute from tests/api directory
ece60b1 
Execute all 3 collectiosn from API suite irrespective of faliure
spelling corrections
modelarchiver build from .whl
benchmark symlink simplified
benchmark store the entire output directory
updated build step name
@prashantsail
Psail test ci perf reg (#14)
6de300a 
* benchmark symlinks corrected 

* api tests relative paths updated
@prashantsail
Merge branch 'master' of https://github.com/awslabs/multi-model-server …
2b2e04c 
…into circleci-perf-regression

# Conflicts:
#	.circleci/README.md
#	.circleci/config.yml
#	.circleci/scripts/linux_test_api.sh
#	.circleci/scripts/linux_test_benchmark.sh
#	.circleci/scripts/linux_test_modelarchiver.sh
#	.circleci/scripts/linux_test_perf_regression.sh
#	.circleci/scripts/linux_test_python.sh
#	run_circleci_tests.py
#	tests/performance/run_performance_suite.py
#	tests/performance/tests/api_description/api_description.yaml
#	tests/performance/tests/batch_and_single_inference/batch_and_single_inference.yaml
#	tests/performance/tests/batch_inference/batch_inference.yaml
#	tests/performance/tests/examples_local_criteria/examples_local_criteria.yaml
#	tests/performance/tests/examples_local_monitoring/examples_local_monitoring.yaml
#	tests/performance/tests/examples_remote_criteria/examples_remote_criteria.yaml
#	tests/performance/tests/examples_remote_monitoring/examples_remote_monitoring.yaml
#	tests/performance/tests/examples_starter/examples_starter.yaml
#	tests/performance/tests/health_check/health_check.yaml
#	tests/performance/tests/inference_multiple_models/inference_multiple_models.yaml
#	tests/performance/tests/inference_multiple_worker/inference_multiple_worker.yaml
#	tests/performance/tests/inference_single_worker/inference_single_worker.yaml
#	tests/performance/tests/list_models/list_models.yaml
#	tests/performance/tests/model_description/model_description.yaml
#	tests/performance/tests/multiple_inference_and_scaling/multiple_inference_and_scaling.yaml
#	tests/performance/tests/register_unregister/register_unregister.yaml
#	tests/performance/tests/register_unregister_multiple/register_unregister_multiple.yaml
#	tests/performance/tests/scale_down_workers/scale_down_workers.yaml
#	tests/performance/tests/scale_up_workers/scale_up_workers.yaml
@prashantsail
removed cruft
5687c2b
@prashantsail
renamed to macos_xlarge
fdc5d7d
@prashantsail
Updated Readme
ec6dc02
@prashantsail
corrected paths after movement from test to tests/api
fb927b9
@prashantsail
dev test triggger
7868bf0
@prashantsail
updated thresholds for circlce ci
65549f3
@prashantsail
updated thresholds for circlce ci
63dd038
@prashantsail
updated thresholds for circlce ci
bcd948e
@prashantsail
updated thresholds for circlce ci
8098785
@prashantsail
Revert dev test changes
2bfb5a0
@prashantsail prashantsail changed the title Circleci perf regression CircleCI + Performance Regression Jul 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@prashantsail @maheshambule @dhruvaray @maaquib