chore: Increase the timeout for e2e_slurm_restart tests (FOUNDENG-470) (

#714)

The _test_master_restart_cmd tests are failing when run as part of the e2e_slurm_restart job inside the nightly workflow. The reason they are failing is that these tests pull an image as part of the test which takes more than a couple of minutes. This delay makes the experiment stay in PULLING state instead of RUNNING state and makes the check for RUNNING status inside _test_master_restart_cmd fail. I increased the timeout from 30 seconds to 300 seconds and tested the e2e_slurm_restart job manually to ensure all the tests run successfully.

e2e_tests/tests/cluster/test_master_restart.py

@@ -225,7 +225,9 @@ def test_master_restart_cmd_k8s(
 
 def _test_master_restart_cmd(managed_cluster: Cluster, slots: int, downtime: int) -> None:
     command_id = run_command(30, slots=slots)
-    wait_for_command_state(command_id, "RUNNING", 30)
+    # The timeout below is set to 300 seconds because sometimes the experiment
+    # remains in the PULLING state for long duration when pulling a new image.
+    wait_for_command_state(command_id, "RUNNING", 300)
 
     if downtime >= 0:
         managed_cluster.kill_master()