Transposed weights sharing == tied weights for AE

[ducha-aiki]

Hi,

Is there any possibility to use weights sharing (like in #546 ), but transposed? It is used for autoencoders and deconvnets?

[shelhamer]

Yes, with the right model definition. See #1211 (comment).

[Macbull]

@shelhamer I can't find solution to transposed weights there ! Can you point me to the particular comment you are talking about ?

[SBodenstedt]

I am also looking in transposed weight sharing, but I can't find the solution mentioned above either... is there an example? Thanks!

[Macbull]

@SBodenstedt see if #2565 is of some help to you !

[SBodenstedt]

@Macbull Thanks for the tip! I checked it out and compiled, but it doesn't seem to work. I used the Transpose layer to share weights between two InnerProduct layers, but when trying to train the net, a seg fault occurred:

I0811 18:41:12.985070  2152 net.cpp:489] Collecting Learning Rate and Weight Decay.
I0811 18:41:12.985090  2152 net.cpp:247] Network initialization done.
I0811 18:41:12.985100  2152 net.cpp:248] Memory required for data: 9931188
I0811 18:41:12.985183  2152 solver.cpp:42] Solver scaffolding done.
caffe-d: /usr/include/boost/smart_ptr/shared_ptr.hpp:653: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = caffe::Blob<float>; typename boost::detail::sp_member_access<T>::type = caffe::Blob<float>*]: Assertion `px != 0' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff699ccc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff699ccc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff69a00d8 in __GI_abort () at abort.c:89
#2  0x00007ffff6995b86 in __assert_fail_base (fmt=0x7ffff6ae6830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x7ffff7a6eb46 "px != 0", file=file@entry=0x7ffff7a6eb08 "/usr/include/boost/smart_ptr/shared_ptr.hpp", line=line@entry=653, 
    function=function@entry=0x7ffff7a6fb00 <boost::shared_ptr<caffe::Blob<float> >::operator->() const::__PRETTY_FUNCTION__> "typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = caffe::Blob<float>; typename boost::detail::sp_member_access<T>::type = caffe::Blob<float>*]") at assert.c:92
#3  0x00007ffff6995c32 in __GI___assert_fail (assertion=0x7ffff7a6eb46 "px != 0", file=0x7ffff7a6eb08 "/usr/include/boost/smart_ptr/shared_ptr.hpp", 
    line=653, 
    function=0x7ffff7a6fb00 <boost::shared_ptr<caffe::Blob<float> >::operator->() const::__PRETTY_FUNCTION__> "typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = caffe::Blob<float>; typename boost::detail::sp_member_access<T>::type = caffe::Blob<float>*]")
    at assert.c:101
#4  0x00007ffff78f56ed in boost::shared_ptr<caffe::Blob<float> >::operator-> (this=0xc992f30) at /usr/include/boost/smart_ptr/shared_ptr.hpp:653
#5  0x00007ffff7911386 in caffe::SGDSolver<float>::PreSolve (this=0x4b404e0) at /media/data/sebastian/caffe/ELM-Caffe/src/caffe/solver.cpp:450
#6  0x0000000000422f50 in caffe::SGDSolver<float>::SGDSolver (this=0x4b404e0, param=...)
    at /media/data/sebastian/caffe/ELM-Caffe/include/caffe/solver.hpp:74
#7  0x000000000042115e in caffe::GetSolver<float> (param=...) at /media/data/sebastian/caffe/ELM-Caffe/include/caffe/solver.hpp:137
#8  0x000000000041d745 in train () at /media/data/sebastian/caffe/ELM-Caffe/tools/caffe.cpp:123
#9  0x000000000041f85c in main (argc=2, argv=0x7fffffffe108) at /media/data/sebastian/caffe/ELM-Caffe/tools/caffe.cpp:307

Removing theTranpose layer fixes the issue.

[Macbull]

@SBodenstedt sorry for that, maybe I forgot to update the PR. You may try this https://github.com/Macbull/ELM-Caffe
This should work.
There is also an example of ELM in it.
Thanks

[SBodenstedt]

Thank you for your help. I cloned the master branch this time, but the error is still the same :(
This is the part of my net where I used the Transpose layer, maybe I made a mistake?

layer {
  name: "fc6"
  type: "InnerProduct"
  bottom: "conv1"
  top: "fc6"
  param {
    name: "fc6_w"
    lr_mult: 10
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 1024
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu6"
  type: "ReLU"
  bottom: "fc6"
  top: "fc6"
}
layer {
  name: "tr1"
  type: "Transpose"
  param: {
    name: "fc6_w"
  }
  param: {
    name: "fc6_w_t"
  }
}
layer {
  name: "fc7"
  type: "InnerProduct"
  bottom: "fc6"
  top: "fc7"
  param {
    name: "fc6_w_T"
    lr_mult: 10
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 63550
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}

[Macbull]

@SBodenstedt ohh! Sorry, but I am currently out of track of this project of mine (which is still incomplete). A few changes you may try :
use fc6_w_t instead if fc6_w_T in layer fc7

If this doesn't work, I would suggest you to modify inner product itself, to take input the boolean for Transpose (you will got it, once you see code of inner product layer). It should be easy and straight forward

[SBodenstedt]

Woops, fixed that typo, but still have the same error...
Do you mean to share and transpose the weights without your Transpose Layer?

[Macbull]

yes ! by explicitly mentioning transpose parameter in modified inner product layer, you can share weights without additional layer ! (set transpose = true for one and false for the other)