Why the different type of schedulers/annealing rates for optimizer?

[brando90]

Hi Kaiyu,

I saw this code:

    if opts.no_validation:
        scheduler = StepLR(optimizer, step_size=opts.lr_reduce_steps, gamma=0.1)
    else:
        scheduler = ReduceLROnPlateau(optimizer, patience=opts.lr_reduce_patience, verbose=True)

and have been wondering why you choose that. Why not use ReduceLROnPlateau always since it detects when the metric stops improving and then anneals. Is there a reason why you decided to code it that way?

Or perhaps you are doing something special during validation that I trivially missing?

Thanks for your time!

[brando90]

Oh I just noticed there is this:

        # reduce the learning rate
        if opts.no_validation:
            scheduler.step()
        else:
            scheduler.step(loss_valid)

does that mean that it only reduces the learning rate if does not detect a change for some number of epchos but with respect to the validation loss? (btw why not just feed the validation error or acc instead)

[yangky11]

Yes, when opts.no_validation == True, the validation loss is not available so we do not use ReduceLROnPlateau . I think validation accuracy would work similarly.