[tune] add-note command for Tune CLI

[andrewztan]

What do these changes do?

Adds add-note command to Tune CLI. Builds on top of #3983

[andrewztan]

@richardliaw

[andrewztan]

Questions:

1. Should we limit to just txt files?
2. Should the --filename flag be the full filename (e.g. note.txt) or just name (e.g. note), or allow for both?
3. In addition to experiment folders, should add-note be allowed in project or other folders?
4. What feedback do we want to give the user during and after the note editing experience?

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12731/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12733/
Test FAILed.

[richardliaw]

Questions:

1. Should we limit to just txt files?

Well, the default should just be note.txt for now. I don't think you need to worry about otherwise.

2. Should the --filename flag be the full filename (e.g. note.txt) or just name (e.g. note), or allow for both?

Full name should be good.

3. In addition to experiment folders, should add-note be allowed in project or other folders?

Yeah, that should be fine.

4. What feedback do we want to give the user during and after the note editing experience?

What do you have in mind for this?

[hartikainen]

This is nice! I think it would also be great to be able to add notes when launching an experiment.

[AmplabJenkins]

Can one of the admins verify this patch?

[richardliaw]

I think this looks fine for now; I'll try this out tomorrow.

I think later on this should maybe somehow use a starter template?

[richardliaw]

This is nice! I think it would also be great to be able to add notes when launching an experiment.

Hm, via run_experiments as a config like note=str?

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12747/
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12750/
Test PASSed.

[hartikainen]

@richardliaw Yeah, something like note=str would be nice, although I'm not sure if adding things like that might make the api cluttered at some point.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12756/
Test PASSed.

[richardliaw]

@hartikainen I'll merge this for now (just as a preliminary first cut to get something in), but in a later PR we'll expose the contents of this note file through tune ls and then we can add a Python API for modifying after iterating.

[richardliaw]

@andrewztan nice work!