fix issues with test_upgrade notebook

[RafalSkolasinski]

What this PR does / why we need it:

It fixes problems with notebook that tests seldon-core upgrades.

Which issue(s) this PR fixes:

Closes #2119

Special notes for your reviewer:

• increased timeout in waitStatus helper function (defined inside notebook)
• fixed re-raising exception if notebook fails
• fixed helm-target for upgrade command (makefile installs seldon, notebook was trying to upgrade seldon-core)
• install_seldon target in testing Makefile can be re-executed now as it will first attempt to remove currently installed seldon-core

Does this PR introduce a user-facing change?:

[review-notebook-app]

Check out this pull request on 

Review Jupyter notebook visual diffs & provide feedback on notebooks.

---

Powered by ReviewNB

[seldondev]

Thu Jul 16 11:55:08 UTC 2020
The logs for [pr-build] [1] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/1.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=1

[seldondev]

Thu Jul 16 11:55:23 UTC 2020
The logs for [lint] [2] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/2.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=2

[RafalSkolasinski]

/test notebooks
/test integration

[seldondev]

Thu Jul 16 11:58:38 UTC 2020
The logs for [integration] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=4

[seldondev]

Thu Jul 16 11:58:42 UTC 2020
The logs for [notebooks] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=3

[seldondev]

Thu Jul 16 12:49:32 UTC 2020
The logs for [lint] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=6

[seldondev]

Thu Jul 16 12:49:38 UTC 2020
The logs for [pr-build] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=5

[RafalSkolasinski]

@axsaucedo Wondering if it wouldn't be reasonable for testing Makefile to helm install seldon-core instead of helm install seldon?

[RafalSkolasinski]

Logs got garbage collected... starting tests again.
/test notebooks
/test integration

[seldondev]

Thu Jul 16 15:26:55 UTC 2020
The logs for [integration] [8] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/8.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=8

[seldondev]

Thu Jul 16 15:27:05 UTC 2020
The logs for [notebooks] [7] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/7.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=7

[axsaucedo]

/test integration
/test notebooks

[seldondev]

Fri Jul 17 16:04:07 UTC 2020
The logs for [integration] [10] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/10.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=10

[seldondev]

Fri Jul 17 16:04:29 UTC 2020
The logs for [notebooks] [9] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/9.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=9

[axsaucedo]

/test integration
/test notebooks

[seldondev]

Fri Jul 17 18:18:28 UTC 2020
The logs for [notebooks] [11] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/11.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=11

[seldondev]

Fri Jul 17 18:18:28 UTC 2020
The logs for [integration] [12] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/12.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=12

[RafalSkolasinski]

Failure on: test_label_update[1.0.2]
Triggering integration tests again.

/test integration

[seldondev]

Mon Jul 20 10:00:21 UTC 2020
The logs for [integration] [13] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/13.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=13

[RafalSkolasinski]

Failure on test_rolling_deployment[graph1.json-graph2.json-True-ambas] - different one
/test integration

[seldondev]

Mon Jul 20 10:55:45 UTC 2020
The logs for [integration] [14] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/14.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2145 --build=14

[RafalSkolasinski]

And now it is TestPrepack.test_text_alibi_explainer.

Three runs, three different failures -> @axsaucedo can we consider it just flakiness?

[axsaucedo]

@RafalSkolasinski yes, if you can run these three tests locally and confirm they pass I think it woudl be reasonable to go ahead, looking at cluster metrics it may be that we need to add more memory resources to the pods that run the integration + notebook tests

[RafalSkolasinski]

Nothing in this PR that could actually affect the tests in question.

Except maybe tiny modification here:
https://github.com/SeldonIO/seldon-core/pull/2145/files#diff-584998bbdce9ef98a2ec464ddfe71576R76
but this only make install_seldon target in the Makefile re-triggerable.

So basically running these three integration tests locally would be like running against master.

[RafalSkolasinski]

/test integration

[seldondev]

STARTING TEST [ integration # 15 ]

Logs will be available when test completes in the following link:
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2145/15.log

Impatient try:
jx get build logs SeldonIO/seldon-core/PR-2145 --build=15

[axsaucedo]

That's a good point, I think given that it has passed, and the notebooks test (which is the main one) has passed, we can merge
/approve

[seldondev]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: axsaucedo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [axsaucedo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[seldondev]

@RafalSkolasinski: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
integration	63a2274	link	/test integration

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.