RMSprop implementation based on G. Hinton Lecture 6

[erogol]

I implemented RMSprop as Hinton suggests and alternatively you can apply momentum as well with RMSprop suggested by http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf

From my experiments I observed much better convergence rate compared to other implemented solver methods. Here's my another toy experiment if you like to see more empirical results. http://www.erogol.com/

Any comments are very welcome.

Here is the mnist example output, https://gist.github.com/erogol/5c5b7beae8b088bd00e9

[shelhamer]

Thanks for the solver -- RMSprop has been on our list so we'll take a look when we can.

Please drop the logs from versioning and instead post a gist in your PR description for us to read.

[erogol]

Sorry I forgot to remove these logs but now they are gone.

[erogol]

I add rmsprop example to mnist example folder also I ll give gist link as you liked.

[ducha-aiki]

@erogol, I have tested it and it works quite fine :)

[erogol]

@ducha-aiki yeah :) thank you... However there are some compilation issues I guess pointed by Travis.

[seanbell]

Are there any plans on porting this to the newest master branch?

[cdoersch]

Well, you almost fixed the typo :-)

[erogol]

yeah :)

[cbalint13]

@erogol ,

• Could update for latest upstream changes ?
• Also can please update to not use 'tabs' but use spaces instead? (this is very bad)
• Would be also nice to squash all you commits into one.

[erogol]

I will take care of it in one of the available times but in a very buzzy period right now.

[beniz]

For those interested, I've merged this PR against my own slightly modified Caffe tree which is up to date with current master, and it appears to work fine on a few applications I've tested. However, two points:

• the unit tests for the gradient do fail for RMSProp and I've been unable to fix them
• I can't tell whether the results do match those of a reference implementation

To reproduce the merge against master, you should be able to use the last two commits from https://github.com/beniz/caffe/tree/master_dd_integ_rmsprop

[shelhamer]

Closing in favor of follow-up #2867. Thanks for the first version @erogol.