Feature/allow ensemble dir overwrite

[jlnav]

User can set libE_specs["reuse_output_dir"] to prevent intentional errors and warnings on attempting to reuse an ensemble directory.

Keep in mind "overwriting" a sim_dir with a new one really means deleting old copied and symlinked files over.

Another consequence of this approach and the current sim/gen-dir naming scheme is if subsequent runs overwrite the same ensemble directory, but different workers each time process the same sim_id, then both sim_dirs for that sim_id will be retained.

"reuse_ensemble_dir"

---

1. Set libE_specs["reuse_output_dir"] = True to allow subsequent runs to operate within a previous ensemble directory
2. workerID suffix removed from calculation directories

Addresses #1028 and #1027

[shuds13]

Name of option?
Maybe reuse_ensemble_dir

We should remove appending of worker ID to each sim_dir so they don't get re-run under different worker and produce another dir.

It is not "overwriting" a sim_dir, but overwriting input files.

[shuds13]

Suggestions:

• Modify top comment on PR (remove "allow_dir_overwrite")

also add separating log files to workflow dirs to bullet list - as there are multiple features here

• In ensemble dir exists message - add use libE_specs["reuse_ensemble_dir"] = True to allow reuse of dir.
• created dirs like test_workflow0_nworkers3_comms-local/ need to be cleaned in run_scripts.sh

I think we also need to make reuse_ensemble_dir work when there is a named workflow dir as this is how Optimas now runs.
ie. if you add libE_specs["reuse_ensemble_dir"] = True to test_workflow_dir.py it wont rerun. This could be incorporated into the test with 2 libE() calls.