Fix Stress Tests #4406

AndrewJGaut · 2023-02-27T23:53:18Z

This addresses: #4404

This PR does several things to alleviate issues brought to light by stress tests:

Reset ITERATIONS_PER_DISK_CHECK to be 2000 rather than 1. When it was 1, upload speed was significantly slower since the client was sending a request to server to update user_disk_used for every 16KB of data uploaded. With it now set to 2000, the user_disk_used will be updated every 32MB (and will be updated with the initial 16KB on the first iteration of the upload). Upload speeds are now approximately in line with previous speeds according to benchmarking tests.
Fix an SQL deadlock issue. When user_disk_used was incremented, we were selecting the user disk used in one statement, incrementing it, and then updating it in another statement. This was not atomic and was leading to an SQL deadlock and possible race conditions. To alleviate this, we now use select_for_update rather than just select when getting user_disk_used.
Fix bundle finalization issue. Stress tests were also taking significantly longer because bundles were staying in the "finalized" state for a long time. It turns out that the socket timeout set in the bundle_manager for the finalization message sent to workers was too low, and so we were frequently getting a socket timeout error for that message and the bundle_manager was taking a long time to progress to actual finalization of the bundle.
Update stress tests: Stress tests now include timing for each individual stress test as well as overall timing. This gives us a better handle for which tests are taking up the majority of the time.

…effiicnet

…rather than matster

…ix-deadlock

…ight be goign wrong

AndrewJGaut · 2023-02-28T19:17:52Z

I should note that stress tests are now finishing in around 4 hours as they were previously and there were no SQL deadlock errors that I saw. Once this PR is in, therefore, we should finally be able to deploy again!

AndrewJGaut · 2023-02-28T19:32:10Z

tests/stress/stress_test.py

@@ -38,34 +46,38 @@ class TestFile:

    def __init__(self, file_name, size_mb=1, content=None):
        self._file_name = file_name
+        self._file_path = temp_path(file_name)


I should note this oddity: suddenly, when running stress tests, I started getting an Errno 13 Permission Denied error when the stress tests would try to create a file. To get around this, instead of creating the file in the working directory, I create it /tmp/ and then update the rest of the code accordingly.

Not sure why this issue just started popping up, though.

epicfaace

Looks great!

epicfaace · 2023-03-01T15:28:15Z

codalab/model/bundle_model.py

        """
-        User used some time.
+        Increment disk_used for user by amount.
+        When incrementing values, we have to use a special query to ensure that we avoid


can we shorten this docstring? probably a bit too much detail for posterity

epicfaace · 2023-03-01T15:30:21Z

codalab/model/worker_model.py

                    time.sleep(
-                        0.3
+                        0.1
                    )  # changed from 0.003 to keep from rate-limiting due to dead workers


Suggested change

) # changed from 0.003 to keep from rate-limiting due to dead workers

)

epicfaace · 2023-03-01T15:30:57Z

scripts/test_util.py

@@ -179,3 +181,48 @@ def cleanup(cl, tag, should_wait=True):
            run_command([cl, 'wrm', uuid, '--force'])
            worksheets_removed += 1
    print('Removed {} bundles and {} worksheets.'.format(bundles_removed, worksheets_removed))
+
+
+class timer:


epicfaace

Looks great!

epicfaace · 2023-03-01T15:31:35Z

tests/stress/stress_test.py

@@ -393,6 +391,7 @@ def main():
    runner.run()
    duration_seconds = time.time() - start_time
    print("--- Completion Time: {} minutes---".format(duration_seconds / 60))
+    print(json.dumps(runner._runs))


let's print this before we print completion time.

…ix-deadlock2

wwwjn and others added 15 commits February 6, 2023 23:21

bump version to 1.6.1

553ea19

Merge remote-tracking branch 'origin/master' into rc1.6.1

887ca29

fix mysql db deadlock

11e970b

do upload first so we can check for deadlock

06a47b7

minor update

e54130f

Modified the code to use better iterations per disk check to be more …

71a6ae8

…effiicnet

add changes to increment time used as well

0bcbef0

revert chnages from rc1.6.1 since I acttually branched off from that …

7b81ae9

…rather than matster

minor change for debugging stress tests

1d48e8a

minor changes

ddaaf42

Merge branch 'master' of github.com:codalab/codalab-worksheets into f…

c1a8235

…ix-deadlock

update stress tests to get more fine-grained information about what m…

bf391af

…ight be goign wrong

update worker manager to log image name

57aa16f

small changes to try and debug stress test issue

f426320

get remaining tests to pass

d6d7891

AndrewJGaut changed the title ~~test draft~~ Fix Stress Tests Feb 28, 2023

AndrewJGaut marked this pull request as ready for review February 28, 2023 19:16

AndrewJGaut requested review from epicfaace and wwwjn February 28, 2023 19:16

codalab deleted a comment from mergify bot Feb 28, 2023

AndrewJGaut commented Feb 28, 2023

View reviewed changes

AndrewJGaut added 2 commits February 28, 2023 11:37

Add in better error logging for send_json_message

d047df9

minor update to help with finalization issue

2a1207a

epicfaace approved these changes Mar 1, 2023

View reviewed changes

AndrewJGaut added 2 commits March 1, 2023 19:29

Merge branch 'master' of github.com:codalab/codalab-worksheets into f…

164216e

…ix-deadlock2

Address comments

9845591

AndrewJGaut merged commit 7833e8e into master Mar 2, 2023

AndrewJGaut deleted the fix-deadlock2 branch March 2, 2023 04:19

This was referenced Mar 2, 2023

Bump version to 1.6.1 #4382

Merged

increase timeouts range in both worker and bundle-manager to help all… #4421

Merged

Stress tests running slow #4404

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Stress Tests #4406

Fix Stress Tests #4406

AndrewJGaut commented Feb 27, 2023 •

edited

Loading

AndrewJGaut commented Feb 28, 2023

AndrewJGaut Feb 28, 2023

epicfaace left a comment

epicfaace Mar 1, 2023

epicfaace Mar 1, 2023

epicfaace Mar 1, 2023

epicfaace left a comment

epicfaace Mar 1, 2023

	) # changed from 0.003 to keep from rate-limiting due to dead workers
	)

Fix Stress Tests #4406

Fix Stress Tests #4406

Conversation

AndrewJGaut commented Feb 27, 2023 • edited Loading

AndrewJGaut commented Feb 28, 2023

AndrewJGaut Feb 28, 2023

Choose a reason for hiding this comment

epicfaace left a comment

Choose a reason for hiding this comment

epicfaace Mar 1, 2023

Choose a reason for hiding this comment

epicfaace Mar 1, 2023

Choose a reason for hiding this comment

epicfaace Mar 1, 2023

Choose a reason for hiding this comment

epicfaace left a comment

Choose a reason for hiding this comment

epicfaace Mar 1, 2023

Choose a reason for hiding this comment

AndrewJGaut commented Feb 27, 2023 •

edited

Loading