Setting weights after initialization

[nwjnilsson]

Hi,
I'm trying to use DJL for a reinforcement learning project and I'm looking at Q-learning at the moment. In Q-learning, there's a step involving copying the weights from the main net to the target net. The snipped below is an example taken from https://github.com/yunjhongwu/DJL-RL-examples which does exactly this.

  protected final void syncNets() {
        for (Pair<String, Parameter> params : policy_net.getBlock().getParameters()) {
            target_net.getBlock().getParameters().get(params.getKey())
                    .setArray(params.getValue().getArray().duplicate());
        }
        target_predictor = target_net.newPredictor(new NoopTranslator());
    }

This seems to have worked in previous versions of DJL, however as far as I can see this would cause issues in 0.15.0 since setArray() cannot be used after the parameter has been initialized:

    public void setArray(NDArray array) {
        if (this.shape != null) {
            throw new IllegalStateException("array has been set! Use either setArray or setShape");
        } else {
            this.array = array;
            this.shape = array.getShape();
            array.setName(this.name);
        }
    }

I'm new to this and can't really see how to get around this issue. How do I set the weights of a net after it has been initialized?

[frankfliu]

@Applooza
Instead of calling setArray(), you can getArray() and then copy the data into it:

NDArray array = target_net.getBlock().getParameters().get(params.getKey()).getArray();
params.getValue().getArray().copyTo(array);

The reason we block the setArray() is assume the shape and data type should never change after initialized.

[nwjnilsson]

@frankfliu
Okay yes that seems good. But am I limited to using mxnet in that case? As far as I can see copyTo() is not implemented in either PyTorch or Tensorflow. Not that this causes any issues for me..

Thanks for the help!

[zachgk]

You can also use array.set(...) which is implemented in those engines

[nwjnilsson]

Brilliant. I actually ran into issues when switching to mxnet with the example I was working with because it was missing a computational graph, and I was unable to fix it. It said to use autograd.record() or setRecording(). Either way I now have a working example with PyTorch and another example with mxnet so hopefully I can get on with my project. Thanks a lot!