To continue a training

[yernarsha]

I'm playing with the training of Connect 4 (thanks for the example).
In the params.jl I set num_iters=10.
How can I continue a training after it passed 10 iterations?
After I change num_iters to 20 and start a training, it says that training completed.

[jonathan-laurent]

This is because once an experiment is started, its initial parameters are saved in a json file at the root of the session dir and recovered from there every time the session is resumed.

I think you should be able to go around this by editing the json file manually.

[yernarsha]

Excuse me, that doesn't help.
I edited params.json file (now "num_iters": 20), but after I start a training again, it still says that training completed. Maybe there's a flag somewhere indicating that training completed?

[johannes-fischer]

I think the parameters are saved as params.data and params.json. But when resuming a training, the params.data file is loaded, the json is only to have a human-readable version.

[yernarsha]

I think the parameters are saved as params.data and params.json. But when resuming a training, the params.data file is loaded, the json is only to have a human-readable version.

Is there a way to modify params.data file?

[jonathan-laurent]

Is there a way to modify params.data file?

Unfortunately not but this should become more flexible in the future.

[smart-fr]

Is there a way to modify params.data file?

Unfortunately not but this should become more flexible in the future.

Would the following work?

1. Edit num_iters in params.jl
2. Launch a brand new (temporary) experiment
3. Overwrite params.data and params.json in the original experiment's session folder with the temp experiment's files
4. Delete the temp experiment's session folder.

[smart-fr]

I confirm that importing params.json and params.data from one session to another allows to continue a training with a modified num_iters. This works for some other parameters of the session, not all of them. I believe it works fine with self play and arena parameters, maybe with learning parameters, and certainly not with network parameters.

[A-Cepheus]

Is there a quick way to generate params.data and params.json without benchmark?