[tune] experiment_state.json doesn't get synced to gs/s3

[hartikainen]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• Ray installed from (source or binary): source
• Ray version: 0.6.4
• Python version: 3.6
• Exact command to reproduce: n/a

Describe the problem

It looks like the experiment_state.json does not get synced to gs/s3 even when upload_dir flag is specified and all the other result info gets synced normally.

Locally I see:

  total 380K
  drwxr-xr-x 18 root root 4.0K Mar  8 22:04 .
  drwxr-xr-x 21 root root 4.0K Mar  8 10:28 ..
  -rw-r--r--  1 root root 308K Mar  8 22:04 experiment_state-2019-03-08_18-28-13.json
  drwx------  2 root root 4.0K Mar  8 10:31 id=21f29b12-seed=4124_2019-03-08_18-28-14qhbvpwcc
  drwx------  2 root root 4.0K Mar  8 10:30 id=2679576f-seed=5656_2019-03-08_18-28-14aqes5rio
  drwx------  2 root root 4.0K Mar  8 10:30 id=4dd38015-seed=4750_2019-03-08_18-28-147wukpx9t
  drwx------  2 root root 4.0K Mar  8 10:30 id=61ff90e5-seed=8690_2019-03-08_18-28-15uatasnti
  drwx------  2 root root 4.0K Mar  8 10:30 id=6f1fd9c9-seed=7721_2019-03-08_18-28-14bqrtr90o
  drwx------  2 root root 4.0K Mar  8 10:30 id=76f13a03-seed=6902_2019-03-08_18-28-141937xedh
  drwx------  2 root root 4.0K Mar  8 10:30 id=9d9646e4-seed=3534_2019-03-08_18-28-140z9chdrc
  drwx------  2 root root 4.0K Mar  8 10:30 id=a32cb54a-seed=2289_2019-03-08_18-28-14xnp00di4
  drwx------  2 root root 4.0K Mar  8 10:30 id=ae84a81d-seed=8578_2019-03-08_18-28-15un50yef9
  drwx------  2 root root 4.0K Mar  8 10:30 id=b8f122a0-seed=6289_2019-03-08_18-28-14o9vy605k
  drwx------  2 root root 4.0K Mar  8 10:30 id=bbfb406f-seed=1753_2019-03-08_18-28-14us5sce3s
  drwx------  2 root root 4.0K Mar  8 10:30 id=d958e54e-seed=6092_2019-03-08_18-28-148s2dn8l6
  drwx------  2 root root 4.0K Mar  8 10:30 id=e093a3f7-seed=8295_2019-03-08_18-28-14a188y2gb
  drwx------  2 root root 4.0K Mar  8 10:30 id=e103817e-seed=1041_2019-03-08_18-28-14cca3065m
  drwx------  2 root root 4.0K Mar  8 10:30 id=e8535802-seed=474_2019-03-08_18-28-145r29jx76
  drwx------  2 root root 4.0K Mar  8 10:30 id=fe470d42-seed=7937_2019-03-08_18-28-14sjdbisc2

On google cloud:

The experiment state should be synced since for example the new tune cli uses it.

[richardliaw]

Should be closed by #4450 (albeit with some directory structure changes !)