Adaptive Solvers: AdaDelta, RMSprop, and ADAM

[shelhamer]

There are open PRs for adaptive solvers that need review, revision, and merge:

• RMSprop RMSprop implementation based on G. Hinton Lecture 6  #1890: finished and ready for review in RMSprop clean up and rebase #2867
• AdaDelta AdaDelta Solver (v3) #2782: need to be aligned with the new solver interface Deduplicate solver regularization, logging, and local rates and decays #2518 Decouple the computational batch size and minibatch size by accumulating gradients #1977
• ADAM Adam solver #2856: finished and ready for review in Adam solver #2918

As a last step AdaGrad, AdaDelta, and RMSprop could likely be improved by refactoring out shared code into an AdaptiveSGDSolver

@ronghanghu these could be good warm-up projects for development.

@longjon @jeffdonahue thoughts on posting tracking issues of this kind? Does this help navigate or is it noise?

[ronghanghu]

OK, I'll go through these PRs. Also, I agree that solver_type should be string instead of enum.

[matthiasplappert]

AdaDelta should already conform to the new solver interface. Unless I missed something, I ported it over in #2782 when merging in master a couple of weeks ago.

[ronghanghu]

Since right now these 3 PRs are blocking each other by enum value in SolverType and SolverParameter next available ID, I expect to merge Adam #2856 or AdaDelta #2782 first, and RMSprop afterwards. Merged #2867 first. If necessary, I can take over and clean up #2856 and #2782.

After that, a solver refactor PR can follow to address the solver_type and split solver.cpp into different files.

[ronghanghu]

Note: #2836 and #2866 modifies solver as well as introduces new snapshot tests, so a further rebase is needed here. #2870 is also going to introduce new conflicts.

[bhack]

@ronghanghu The merging sequence It is becoming to be a "political" issue because there isn't a clear policy for reviews and merges. Fast track merges on younger PRs push adaptation work over the older one (apart of the quality, the in shape status, the historical rebasing effort a maintainer has done on this older PRs)

[ronghanghu]

@bhack Indeed, there isn't a clear policy for PR reviewing process right now. Our merge sequence is mainly based on the status of PRs, that is, whether a PR is ready to merge or needs further modification. Since these 2 out of 3 here (Adam and AdaDelta) are still not ready, I expect to wait to merge them after #2870.

[bhack]

The problem is that there isn't almost a rough policy for the community. Readiness it is influenced also by reviews timing other that proposer availability. #2836 that brake this was a fast track review. There are many staging PR in the queue that need to support rebasing efforts of fast track Pr merges.

[ronghanghu]

Now all of them done. Address solver refactor in #2890.