Python wrapper

[chuckcho]

Made pycaffe available for C3D classification with a sample python wrapper (run_c3d_classification.py)

[facebook-github-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla - and if you have received this in error or have any questions, please drop us a line at [email protected]. Thanks!

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

[gyglim]

I believe the way you read in the mean is wrong (mixed dimension). I had the problem that the mean didn't match in my Pull request to Lasagne Recipes: Lasagne/Recipes#41

It appears the values are read in a different order from the one they are written.

The elements in C3D are shaped as (channel,frameNr,y,x), but you read in the mean as (frameNr,channel,y,x).
Fix: image_mean.reshape(3,16,128,171) did the trick for me :)

Here is the mean difference between the frame means visualized:
Difference between the first channel of image 5 and image 0: https://data.vision.ee.ethz.ch/gyglim/C3D/mean_diff/5_0.png

Difference between the first channel of image 6 and image 5:
https://data.vision.ee.ethz.ch/gyglim/C3D/mean_diff/6_5.png

i.e. there is a jump in the values, which comes from reading the input wrong.

[chuckcho]

@gyglim thanks for the fix. i'll incorporate this in this PR soon.

[chuckcho]

Added some fixes / sample script for UCF-101.

[chuckcho]

I believe this commit completes all there is to support C3D in python scripts. Please let me know if anyone has a feedback or a question. Thanks.

[dutran]

@chuckcho Thank you very much for your contribution. I did not have time to fully review this for now (so it may take sometime to merge as soon as I fully try/test it).

[fengwhu]

@chuckcho I wonder whether I could directly load the pre-trained Sport1m model in run_c3d_classification.py?
I edited the prototxt file for model conv3d_deepnetA_sport1m_iter_1900000 just like conv3d_ucf101_deploy.prototxt. But I still have trouble when using net = caffe.Net(model_def_file, model_file) to load the sport1m model. The python console goes wrong like that It seems the kernel died unexpectedly. Use 'Restart kernel' to continue using this console.

[chuckcho]

@fengwhu. conv3d_deepnetA_sport1m_iter_1900000 was created with c3d_finetuning/c3d_ucf101_finetuning_train.prototxt, so the deploy proto file should be modified as such. For example, the following deploy file works fine with conv3d_deepnetA_sport1m_iter_1900000:

name: "DeepConv3DNet_ucf101_finetuning"
input: "data"
input_dim: 30
input_dim: 3
input_dim: 16
input_dim: 112
input_dim: 112
# ----------- 1st layer group ---------------
layers {
  name: "conv1a"
  type: CONVOLUTION3D
  bottom: "data"
  top: "conv1a"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 64
    kernel_size: 3
    kernel_depth: 3
    pad: 1
    temporal_pad: 1
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {
  name: "relu1a"
  type: RELU
  bottom: "conv1a"
  top: "conv1a"
}
layers {
  name: "pool1"
  type: POOLING3D
  bottom: "conv1a"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 2
    kernel_depth: 1
    stride: 2
    temporal_stride: 1
  }
}
# ------------- 2nd layer group --------------
layers {
  name: "conv2a"
  type: CONVOLUTION3D
  bottom: "pool1"
  top: "conv2a"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 128
    kernel_size: 3
    kernel_depth: 3
    pad: 1
    temporal_pad: 1
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu2a"
  type: RELU
  bottom: "conv2a"
  top: "conv2a"
}
layers {
  name: "pool2"
  type: POOLING3D
  bottom: "conv2a"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_size: 2
    kernel_depth: 2
    stride: 2
    temporal_stride: 2
  }
}
# ----------------- 3rd layer group --------------
layers {
  name: "conv3a"
  type: CONVOLUTION3D
  bottom: "pool2"
  top: "conv3a"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 256
    kernel_size: 3
    kernel_depth: 3
    pad: 1
    temporal_pad: 1
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu3a"
  type: RELU
  bottom: "conv3a"
  top: "conv3a"
}
layers {
  name: "conv3b"
  type: CONVOLUTION3D
  bottom: "conv3a"
  top: "conv3b"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 256
    kernel_size: 3
    kernel_depth: 3
    pad: 1
    temporal_pad: 1
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu3b"
  type: RELU
  bottom: "conv3b"
  top: "conv3b"
}
layers {
  name: "pool3"
  type: POOLING3D
  bottom: "conv3b"
  top: "pool3"
  pooling_param {
    pool: MAX
    kernel_size: 2
    kernel_depth: 2
    stride: 2
    temporal_stride: 2
  }
}

# --------- 4th layer group
layers {
  name: "conv4a"
  type: CONVOLUTION3D
  bottom: "pool3"
  top: "conv4a"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 512
    kernel_size: 3
    kernel_depth: 3
    pad: 1
    temporal_pad: 1
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu4a"
  type: RELU
  bottom: "conv4a"
  top: "conv4a"
}
layers {
  name: "conv4b"
  type: CONVOLUTION3D
  bottom: "conv4a"
  top: "conv4b"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 512
    kernel_size: 3
    kernel_depth: 3
    pad: 1
    temporal_pad: 1
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu4b"
  type: RELU
  bottom: "conv4b"
  top: "conv4b"
}
layers {
  name: "pool4"
  type: POOLING3D
  bottom: "conv4b"
  top: "pool4"
  pooling_param {
    pool: MAX
    kernel_size: 2
    kernel_depth: 2
    stride: 2
    temporal_stride: 2
  }
}

# --------------- 5th layer group --------
layers {
  name: "conv5a"
  type: CONVOLUTION3D
  bottom: "pool4"
  top: "conv5a"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 512
    kernel_size: 3
    kernel_depth: 3
    pad: 1
    temporal_pad: 1
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu5a"
  type: RELU
  bottom: "conv5a"
  top: "conv5a"
}
layers {
  name: "conv5b"
  type: CONVOLUTION3D
  bottom: "conv5a"
  top: "conv5b"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 512
    kernel_size: 3
    kernel_depth: 3
    pad: 1
    temporal_pad: 1
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu5b"
  type: RELU
  bottom: "conv5b"
  top: "conv5b"
}

layers {
  name: "pool5"
  type: POOLING3D
  bottom: "conv5b"
  top: "pool5"
  pooling_param {
    pool: MAX
    kernel_size: 2
    kernel_depth: 2
    stride: 2
    temporal_stride: 2
  }
}
# ---------------- fc layers -------------
layers {
  name: "fc6"
  type: INNER_PRODUCT
  bottom: "pool5"
  top: "fc6"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu6"
  type: RELU
  bottom: "fc6"
  top: "fc6"
}
layers {
  name: "drop6"
  type: DROPOUT
  bottom: "fc6"
  top: "fc6"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "fc7"
  type: INNER_PRODUCT
  bottom: "fc6"
  top: "fc7"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 4096
    weight_filler {
    type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu7"
  type: RELU
  bottom: "fc7"
  top: "fc7"
}
layers {
  name: "drop7"
  type: DROPOUT
  bottom: "fc7"
  top: "fc7"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "fc8"
  type: INNER_PRODUCT
  bottom: "fc7"
  top: "fc8"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 101
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {
  name: "prob"
  type: SOFTMAX
  bottom: "fc8"
  top: "prob"
}

[fengwhu]

@chuckcho Fine. It works now. Thanks a lot!