Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Conv layer can not work,only outputs zeros #6168

Closed
Vilour opened this issue Jan 12, 2018 · 0 comments
Closed

Conv layer can not work,only outputs zeros #6168

Vilour opened this issue Jan 12, 2018 · 0 comments

Comments

@Vilour
Copy link

Vilour commented Jan 12, 2018 •
edited
Loading

Please use the caffe-users list for usage, installation, or modeling questions, or other requests for help.
Do not post such requests to Issues. Doing so interferes with the development of Caffe.

Please read the guidelines for contributing before submitting this issue.

Issue summary

Conv layer only outputs zero whatever it takes in.I have tried to debug only using cpu and the issue remains still.I have run test and passed.

Here is the log of mnist experiment:

Log file created at: 2018/01/12 16:57:06
Running on machine: hu-System-Product-Name
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0112 16:57:06.539829 15972 caffe.cpp:218] Using GPUs 0
I0112 16:57:06.575636 15972 caffe.cpp:223] GPU 0: GeForce GTX 1080 Ti
I0112 16:57:06.767050 15972 solver.cpp:44] Initializing solver from parameters:
test_iter: 100
test_interval: 500
base_lr: 0.01
display: 100
max_iter: 10000
lr_policy: "inv"
gamma: 0.0001
power: 0.75
momentum: 0.9
weight_decay: 0.0005
snapshot: 5000
snapshot_prefix: "examples/mnist/lenet"
solver_mode: GPU
device_id: 0
net: "examples/mnist/lenet_train_test.prototxt"
train_state {
level: 0
stage: ""
}
I0112 16:57:06.767146 15972 solver.cpp:87] Creating training net from net file: examples/mnist/lenet_train_test.prototxt
I0112 16:57:06.767263 15972 net.cpp:294] The NetState phase (0) differed from the phase (1) specified by a rule in layer mnist
I0112 16:57:06.767271 15972 net.cpp:294] The NetState phase (0) differed from the phase (1) specified by a rule in layer accuracy
I0112 16:57:06.767316 15972 net.cpp:51] Initializing net from parameters:
name: "LeNet"
state {
phase: TRAIN
level: 0
stage: ""
}
layer {
name: "mnist"
type: "Data"
top: "data"
top: "label"
include {
phase: TRAIN
}
transform_param {
scale: 0.00390625
}
data_param {
source: "examples/mnist/mnist_train_lmdb"
batch_size: 64
backend: LMDB
}
}
layer {
name: "conv1"
type: "Convolution"
bottom: "data"
top: "conv1"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
convolution_param {
num_output: 20
kernel_size: 5
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "pool1"
type: "Pooling"
bottom: "conv1"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "conv2"
type: "Convolution"
bottom: "pool1"
top: "conv2"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
convolution_param {
num_output: 50
kernel_size: 5
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "pool2"
type: "Pooling"
bottom: "conv2"
top: "pool2"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "ip1"
type: "InnerProduct"
bottom: "pool2"
top: "ip1"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
inner_product_param {
num_output: 500
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu1"
type: "ReLU"
bottom: "ip1"
top: "ip1"
}
layer {
name: "ip2"
type: "InnerProduct"
bottom: "ip1"
top: "ip2"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
inner_product_param {
num_output: 10
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "loss"
type: "SoftmaxWithLoss"
bottom: "ip2"
bottom: "label"
top: "loss"
}
I0112 16:57:06.767364 15972 layer_factory.hpp:77] Creating layer mnist
I0112 16:57:06.767424 15972 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_train_lmdb
I0112 16:57:06.767441 15972 net.cpp:84] Creating Layer mnist
I0112 16:57:06.767446 15972 net.cpp:380] mnist -> data
I0112 16:57:06.767457 15972 net.cpp:380] mnist -> label
I0112 16:57:06.768013 15972 data_layer.cpp:45] output data size: 64,1,28,28
I0112 16:57:06.768980 15972 net.cpp:122] Setting up mnist
I0112 16:57:06.768988 15972 net.cpp:129] Top shape: 64 1 28 28 (50176)
I0112 16:57:06.768992 15972 net.cpp:129] Top shape: 64 (64)
I0112 16:57:06.768995 15972 net.cpp:137] Memory required for data: 200960
I0112 16:57:06.769001 15972 layer_factory.hpp:77] Creating layer conv1
I0112 16:57:06.769012 15972 net.cpp:84] Creating Layer conv1
I0112 16:57:06.769016 15972 net.cpp:406] conv1 <- data
I0112 16:57:06.769024 15972 net.cpp:380] conv1 -> conv1
I0112 16:57:06.903791 15972 net.cpp:122] Setting up conv1
I0112 16:57:06.903810 15972 net.cpp:129] Top shape: 64 20 24 24 (737280)
I0112 16:57:06.903813 15972 net.cpp:137] Memory required for data: 3150080
I0112 16:57:06.903828 15972 layer_factory.hpp:77] Creating layer pool1
I0112 16:57:06.903837 15972 net.cpp:84] Creating Layer pool1
I0112 16:57:06.903856 15972 net.cpp:406] pool1 <- conv1
I0112 16:57:06.903861 15972 net.cpp:380] pool1 -> pool1
I0112 16:57:06.903889 15972 net.cpp:122] Setting up pool1
I0112 16:57:06.903894 15972 net.cpp:129] Top shape: 64 20 12 12 (184320)
I0112 16:57:06.903898 15972 net.cpp:137] Memory required for data: 3887360
I0112 16:57:06.903899 15972 layer_factory.hpp:77] Creating layer conv2
I0112 16:57:06.903905 15972 net.cpp:84] Creating Layer conv2
I0112 16:57:06.903908 15972 net.cpp:406] conv2 <- pool1
I0112 16:57:06.903913 15972 net.cpp:380] conv2 -> conv2
I0112 16:57:06.904696 15972 net.cpp:122] Setting up conv2
I0112 16:57:06.904705 15972 net.cpp:129] Top shape: 64 50 8 8 (204800)
I0112 16:57:06.904707 15972 net.cpp:137] Memory required for data: 4706560
I0112 16:57:06.904712 15972 layer_factory.hpp:77] Creating layer pool2
I0112 16:57:06.904717 15972 net.cpp:84] Creating Layer pool2
I0112 16:57:06.904719 15972 net.cpp:406] pool2 <- conv2
I0112 16:57:06.904723 15972 net.cpp:380] pool2 -> pool2
I0112 16:57:06.904744 15972 net.cpp:122] Setting up pool2
I0112 16:57:06.904748 15972 net.cpp:129] Top shape: 64 50 4 4 (51200)
I0112 16:57:06.904750 15972 net.cpp:137] Memory required for data: 4911360
I0112 16:57:06.904753 15972 layer_factory.hpp:77] Creating layer ip1
I0112 16:57:06.904758 15972 net.cpp:84] Creating Layer ip1
I0112 16:57:06.904760 15972 net.cpp:406] ip1 <- pool2
I0112 16:57:06.904763 15972 net.cpp:380] ip1 -> ip1
I0112 16:57:06.906505 15972 net.cpp:122] Setting up ip1
I0112 16:57:06.906512 15972 net.cpp:129] Top shape: 64 500 (32000)
I0112 16:57:06.906514 15972 net.cpp:137] Memory required for data: 5039360
I0112 16:57:06.906520 15972 layer_factory.hpp:77] Creating layer relu1
I0112 16:57:06.906525 15972 net.cpp:84] Creating Layer relu1
I0112 16:57:06.906528 15972 net.cpp:406] relu1 <- ip1
I0112 16:57:06.906532 15972 net.cpp:367] relu1 -> ip1 (in-place)
I0112 16:57:06.906631 15972 net.cpp:122] Setting up relu1
I0112 16:57:06.906637 15972 net.cpp:129] Top shape: 64 500 (32000)
I0112 16:57:06.906639 15972 net.cpp:137] Memory required for data: 5167360
I0112 16:57:06.906641 15972 layer_factory.hpp:77] Creating layer ip2
I0112 16:57:06.906646 15972 net.cpp:84] Creating Layer ip2
I0112 16:57:06.906648 15972 net.cpp:406] ip2 <- ip1
I0112 16:57:06.906651 15972 net.cpp:380] ip2 -> ip2
I0112 16:57:06.907167 15972 net.cpp:122] Setting up ip2
I0112 16:57:06.907174 15972 net.cpp:129] Top shape: 64 10 (640)
I0112 16:57:06.907176 15972 net.cpp:137] Memory required for data: 5169920
I0112 16:57:06.907181 15972 layer_factory.hpp:77] Creating layer loss
I0112 16:57:06.907186 15972 net.cpp:84] Creating Layer loss
I0112 16:57:06.907188 15972 net.cpp:406] loss <- ip2
I0112 16:57:06.907191 15972 net.cpp:406] loss <- label
I0112 16:57:06.907196 15972 net.cpp:380] loss -> loss
I0112 16:57:06.907204 15972 layer_factory.hpp:77] Creating layer loss
I0112 16:57:06.907357 15972 net.cpp:122] Setting up loss
I0112 16:57:06.907363 15972 net.cpp:129] Top shape: (1)
I0112 16:57:06.907366 15972 net.cpp:132] with loss weight 1
I0112 16:57:06.907379 15972 net.cpp:137] Memory required for data: 5169924
I0112 16:57:06.907382 15972 net.cpp:198] loss needs backward computation.
I0112 16:57:06.907388 15972 net.cpp:198] ip2 needs backward computation.
I0112 16:57:06.907390 15972 net.cpp:198] relu1 needs backward computation.
I0112 16:57:06.907392 15972 net.cpp:198] ip1 needs backward computation.
I0112 16:57:06.907394 15972 net.cpp:198] pool2 needs backward computation.
I0112 16:57:06.907397 15972 net.cpp:198] conv2 needs backward computation.
I0112 16:57:06.907400 15972 net.cpp:198] pool1 needs backward computation.
I0112 16:57:06.907402 15972 net.cpp:198] conv1 needs backward computation.
I0112 16:57:06.907405 15972 net.cpp:200] mnist does not need backward computation.
I0112 16:57:06.907407 15972 net.cpp:242] This network produces output loss
I0112 16:57:06.907413 15972 net.cpp:255] Network initialization done.
I0112 16:57:06.907508 15972 solver.cpp:172] Creating test net (#0) specified by net file: examples/mnist/lenet_train_test.prototxt
I0112 16:57:06.907531 15972 net.cpp:294] The NetState phase (1) differed from the phase (0) specified by a rule in layer mnist
I0112 16:57:06.907578 15972 net.cpp:51] Initializing net from parameters:
name: "LeNet"
state {
phase: TEST
}
layer {
name: "mnist"
type: "Data"
top: "data"
top: "label"
include {
phase: TEST
}
transform_param {
scale: 0.00390625
}
data_param {
source: "examples/mnist/mnist_test_lmdb"
batch_size: 100
backend: LMDB
}
}
layer {
name: "conv1"
type: "Convolution"
bottom: "data"
top: "conv1"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
convolution_param {
num_output: 20
kernel_size: 5
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "pool1"
type: "Pooling"
bottom: "conv1"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "conv2"
type: "Convolution"
bottom: "pool1"
top: "conv2"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
convolution_param {
num_output: 50
kernel_size: 5
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "pool2"
type: "Pooling"
bottom: "conv2"
top: "pool2"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "ip1"
type: "InnerProduct"
bottom: "pool2"
top: "ip1"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
inner_product_param {
num_output: 500
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu1"
type: "ReLU"
bottom: "ip1"
top: "ip1"
}
layer {
name: "ip2"
type: "InnerProduct"
bottom: "ip1"
top: "ip2"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
inner_product_param {
num_output: 10
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "accuracy"
type: "Accuracy"
bottom: "ip2"
bottom: "label"
top: "accuracy"
include {
phase: TEST
}
}
layer {
name: "loss"
type: "SoftmaxWithLoss"
bottom: "ip2"
bottom: "label"
top: "loss"
}
I0112 16:57:06.907650 15972 layer_factory.hpp:77] Creating layer mnist
I0112 16:57:06.907681 15972 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_test_lmdb
I0112 16:57:06.907691 15972 net.cpp:84] Creating Layer mnist
I0112 16:57:06.907696 15972 net.cpp:380] mnist -> data
I0112 16:57:06.907701 15972 net.cpp:380] mnist -> label
I0112 16:57:06.907748 15972 data_layer.cpp:45] output data size: 100,1,28,28
I0112 16:57:06.908696 15972 net.cpp:122] Setting up mnist
I0112 16:57:06.908704 15972 net.cpp:129] Top shape: 100 1 28 28 (78400)
I0112 16:57:06.908707 15972 net.cpp:129] Top shape: 100 (100)
I0112 16:57:06.908710 15972 net.cpp:137] Memory required for data: 314000
I0112 16:57:06.908712 15972 layer_factory.hpp:77] Creating layer label_mnist_1_split
I0112 16:57:06.908718 15972 net.cpp:84] Creating Layer label_mnist_1_split
I0112 16:57:06.908720 15972 net.cpp:406] label_mnist_1_split <- label
I0112 16:57:06.908725 15972 net.cpp:380] label_mnist_1_split -> label_mnist_1_split_0
I0112 16:57:06.908730 15972 net.cpp:380] label_mnist_1_split -> label_mnist_1_split_1
I0112 16:57:06.908802 15972 net.cpp:122] Setting up label_mnist_1_split
I0112 16:57:06.908810 15972 net.cpp:129] Top shape: 100 (100)
I0112 16:57:06.908813 15972 net.cpp:129] Top shape: 100 (100)
I0112 16:57:06.908815 15972 net.cpp:137] Memory required for data: 314800
I0112 16:57:06.908818 15972 layer_factory.hpp:77] Creating layer conv1
I0112 16:57:06.908826 15972 net.cpp:84] Creating Layer conv1
I0112 16:57:06.908828 15972 net.cpp:406] conv1 <- data
I0112 16:57:06.908833 15972 net.cpp:380] conv1 -> conv1
I0112 16:57:06.909557 15972 net.cpp:122] Setting up conv1
I0112 16:57:06.909566 15972 net.cpp:129] Top shape: 100 20 24 24 (1152000)
I0112 16:57:06.909569 15972 net.cpp:137] Memory required for data: 4922800
I0112 16:57:06.909574 15972 layer_factory.hpp:77] Creating layer pool1
I0112 16:57:06.909586 15972 net.cpp:84] Creating Layer pool1
I0112 16:57:06.909590 15972 net.cpp:406] pool1 <- conv1
I0112 16:57:06.909593 15972 net.cpp:380] pool1 -> pool1
I0112 16:57:06.909664 15972 net.cpp:122] Setting up pool1
I0112 16:57:06.909670 15972 net.cpp:129] Top shape: 100 20 12 12 (288000)
I0112 16:57:06.909672 15972 net.cpp:137] Memory required for data: 6074800
I0112 16:57:06.909674 15972 layer_factory.hpp:77] Creating layer conv2
I0112 16:57:06.909680 15972 net.cpp:84] Creating Layer conv2
I0112 16:57:06.909682 15972 net.cpp:406] conv2 <- pool1
I0112 16:57:06.909685 15972 net.cpp:380] conv2 -> conv2
I0112 16:57:06.910446 15972 net.cpp:122] Setting up conv2
I0112 16:57:06.910455 15972 net.cpp:129] Top shape: 100 50 8 8 (320000)
I0112 16:57:06.910457 15972 net.cpp:137] Memory required for data: 7354800
I0112 16:57:06.910464 15972 layer_factory.hpp:77] Creating layer pool2
I0112 16:57:06.910468 15972 net.cpp:84] Creating Layer pool2
I0112 16:57:06.910471 15972 net.cpp:406] pool2 <- conv2
I0112 16:57:06.910476 15972 net.cpp:380] pool2 -> pool2
I0112 16:57:06.910500 15972 net.cpp:122] Setting up pool2
I0112 16:57:06.910503 15972 net.cpp:129] Top shape: 100 50 4 4 (80000)
I0112 16:57:06.910506 15972 net.cpp:137] Memory required for data: 7674800
I0112 16:57:06.910508 15972 layer_factory.hpp:77] Creating layer ip1
I0112 16:57:06.910514 15972 net.cpp:84] Creating Layer ip1
I0112 16:57:06.910516 15972 net.cpp:406] ip1 <- pool2
I0112 16:57:06.910521 15972 net.cpp:380] ip1 -> ip1
I0112 16:57:06.912299 15972 net.cpp:122] Setting up ip1
I0112 16:57:06.912308 15972 net.cpp:129] Top shape: 100 500 (50000)
I0112 16:57:06.912310 15972 net.cpp:137] Memory required for data: 7874800
I0112 16:57:06.912317 15972 layer_factory.hpp:77] Creating layer relu1
I0112 16:57:06.912322 15972 net.cpp:84] Creating Layer relu1
I0112 16:57:06.912324 15972 net.cpp:406] relu1 <- ip1
I0112 16:57:06.912328 15972 net.cpp:367] relu1 -> ip1 (in-place)
I0112 16:57:06.912431 15972 net.cpp:122] Setting up relu1
I0112 16:57:06.912436 15972 net.cpp:129] Top shape: 100 500 (50000)
I0112 16:57:06.912439 15972 net.cpp:137] Memory required for data: 8074800
I0112 16:57:06.912441 15972 layer_factory.hpp:77] Creating layer ip2
I0112 16:57:06.912448 15972 net.cpp:84] Creating Layer ip2
I0112 16:57:06.912451 15972 net.cpp:406] ip2 <- ip1
I0112 16:57:06.912454 15972 net.cpp:380] ip2 -> ip2
I0112 16:57:06.912534 15972 net.cpp:122] Setting up ip2
I0112 16:57:06.912539 15972 net.cpp:129] Top shape: 100 10 (1000)
I0112 16:57:06.912540 15972 net.cpp:137] Memory required for data: 8078800
I0112 16:57:06.912545 15972 layer_factory.hpp:77] Creating layer ip2_ip2_0_split
I0112 16:57:06.912549 15972 net.cpp:84] Creating Layer ip2_ip2_0_split
I0112 16:57:06.912551 15972 net.cpp:406] ip2_ip2_0_split <- ip2
I0112 16:57:06.912555 15972 net.cpp:380] ip2_ip2_0_split -> ip2_ip2_0_split_0
I0112 16:57:06.912560 15972 net.cpp:380] ip2_ip2_0_split -> ip2_ip2_0_split_1
I0112 16:57:06.912580 15972 net.cpp:122] Setting up ip2_ip2_0_split
I0112 16:57:06.912583 15972 net.cpp:129] Top shape: 100 10 (1000)
I0112 16:57:06.912585 15972 net.cpp:129] Top shape: 100 10 (1000)
I0112 16:57:06.912588 15972 net.cpp:137] Memory required for data: 8086800
I0112 16:57:06.912590 15972 layer_factory.hpp:77] Creating layer accuracy
I0112 16:57:06.912595 15972 net.cpp:84] Creating Layer accuracy
I0112 16:57:06.912596 15972 net.cpp:406] accuracy <- ip2_ip2_0_split_0
I0112 16:57:06.912600 15972 net.cpp:406] accuracy <- label_mnist_1_split_0
I0112 16:57:06.912605 15972 net.cpp:380] accuracy -> accuracy
I0112 16:57:06.912609 15972 net.cpp:122] Setting up accuracy
I0112 16:57:06.912612 15972 net.cpp:129] Top shape: (1)
I0112 16:57:06.912616 15972 net.cpp:137] Memory required for data: 8086804
I0112 16:57:06.912617 15972 layer_factory.hpp:77] Creating layer loss
I0112 16:57:06.912621 15972 net.cpp:84] Creating Layer loss
I0112 16:57:06.912623 15972 net.cpp:406] loss <- ip2_ip2_0_split_1
I0112 16:57:06.912626 15972 net.cpp:406] loss <- label_mnist_1_split_1
I0112 16:57:06.912637 15972 net.cpp:380] loss -> loss
I0112 16:57:06.912642 15972 layer_factory.hpp:77] Creating layer loss
I0112 16:57:06.912783 15972 net.cpp:122] Setting up loss
I0112 16:57:06.912788 15972 net.cpp:129] Top shape: (1)
I0112 16:57:06.912791 15972 net.cpp:132] with loss weight 1
I0112 16:57:06.912797 15972 net.cpp:137] Memory required for data: 8086808
I0112 16:57:06.912799 15972 net.cpp:198] loss needs backward computation.
I0112 16:57:06.912803 15972 net.cpp:200] accuracy does not need backward computation.
I0112 16:57:06.912806 15972 net.cpp:198] ip2_ip2_0_split needs backward computation.
I0112 16:57:06.912808 15972 net.cpp:198] ip2 needs backward computation.
I0112 16:57:06.912811 15972 net.cpp:198] relu1 needs backward computation.
I0112 16:57:06.912813 15972 net.cpp:198] ip1 needs backward computation.
I0112 16:57:06.912816 15972 net.cpp:198] pool2 needs backward computation.
I0112 16:57:06.912818 15972 net.cpp:198] conv2 needs backward computation.
I0112 16:57:06.912820 15972 net.cpp:198] pool1 needs backward computation.
I0112 16:57:06.912823 15972 net.cpp:198] conv1 needs backward computation.
I0112 16:57:06.912827 15972 net.cpp:200] label_mnist_1_split does not need backward computation.
I0112 16:57:06.912829 15972 net.cpp:200] mnist does not need backward computation.
I0112 16:57:06.912832 15972 net.cpp:242] This network produces output accuracy
I0112 16:57:06.912834 15972 net.cpp:242] This network produces output loss
I0112 16:57:06.912842 15972 net.cpp:255] Network initialization done.
I0112 16:57:06.912863 15972 solver.cpp:56] Solver scaffolding done.
I0112 16:57:06.913017 15972 caffe.cpp:248] Starting Optimization
I0112 16:57:06.913022 15972 solver.cpp:272] Solving LeNet
I0112 16:57:06.913023 15972 solver.cpp:273] Learning Rate Policy: inv
I0112 16:57:06.913442 15972 solver.cpp:330] Iteration 0, Testing net (#0)
I0112 16:57:06.964840 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:06.965673 15972 solver.cpp:397] Test net output #0: accuracy = 0.1513
I0112 16:57:06.965690 15972 solver.cpp:397] Test net output #1: loss = 2.32174 (* 1 = 2.32174 loss)
I0112 16:57:06.967387 15972 solver.cpp:218] Iteration 0 (0 iter/s, 0.0543487s/100 iters), loss = 2.30732
I0112 16:57:06.967401 15972 solver.cpp:237] Train net output #0: loss = 2.30732 (* 1 = 2.30732 loss)
I0112 16:57:06.967412 15972 sgd_solver.cpp:105] Iteration 0, lr = 0.01
I0112 16:57:07.095191 15972 solver.cpp:218] Iteration 100 (782.618 iter/s, 0.127776s/100 iters), loss = 0.197739
I0112 16:57:07.095214 15972 solver.cpp:237] Train net output #0: loss = 0.197739 (* 1 = 0.197739 loss)
I0112 16:57:07.095218 15972 sgd_solver.cpp:105] Iteration 100, lr = 0.00992565
I0112 16:57:07.211272 15972 solver.cpp:218] Iteration 200 (861.713 iter/s, 0.116048s/100 iters), loss = 0.121513
I0112 16:57:07.211297 15972 solver.cpp:237] Train net output #0: loss = 0.121513 (* 1 = 0.121513 loss)
I0112 16:57:07.211300 15972 sgd_solver.cpp:105] Iteration 200, lr = 0.00985258
I0112 16:57:07.321912 15972 solver.cpp:218] Iteration 300 (904.108 iter/s, 0.110606s/100 iters), loss = 0.165501
I0112 16:57:07.321940 15972 solver.cpp:237] Train net output #0: loss = 0.165501 (* 1 = 0.165501 loss)
I0112 16:57:07.321945 15972 sgd_solver.cpp:105] Iteration 300, lr = 0.00978075
I0112 16:57:07.429175 15972 solver.cpp:218] Iteration 400 (932.583 iter/s, 0.107229s/100 iters), loss = 0.0693915
I0112 16:57:07.429203 15972 solver.cpp:237] Train net output #0: loss = 0.0693915 (* 1 = 0.0693915 loss)
I0112 16:57:07.429209 15972 sgd_solver.cpp:105] Iteration 400, lr = 0.00971013
I0112 16:57:07.533810 15972 solver.cpp:330] Iteration 500, Testing net (#0)
I0112 16:57:07.579049 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:07.580595 15972 solver.cpp:397] Test net output #0: accuracy = 0.9699
I0112 16:57:07.580621 15972 solver.cpp:397] Test net output #1: loss = 0.090742 (* 1 = 0.090742 loss)
I0112 16:57:07.581598 15972 solver.cpp:218] Iteration 500 (656.215 iter/s, 0.152389s/100 iters), loss = 0.104903
I0112 16:57:07.581638 15972 solver.cpp:237] Train net output #0: loss = 0.104904 (* 1 = 0.104904 loss)
I0112 16:57:07.581645 15972 sgd_solver.cpp:105] Iteration 500, lr = 0.00964069
I0112 16:57:07.686987 15972 solver.cpp:218] Iteration 600 (949.308 iter/s, 0.10534s/100 iters), loss = 0.0830735
I0112 16:57:07.687012 15972 solver.cpp:237] Train net output #0: loss = 0.0830735 (* 1 = 0.0830735 loss)
I0112 16:57:07.687017 15972 sgd_solver.cpp:105] Iteration 600, lr = 0.0095724
I0112 16:57:07.792603 15972 solver.cpp:218] Iteration 700 (947.117 iter/s, 0.105584s/100 iters), loss = 0.162873
I0112 16:57:07.792629 15972 solver.cpp:237] Train net output #0: loss = 0.162873 (* 1 = 0.162873 loss)
I0112 16:57:07.792634 15972 sgd_solver.cpp:105] Iteration 700, lr = 0.00950522
I0112 16:57:07.899186 15972 solver.cpp:218] Iteration 800 (938.534 iter/s, 0.106549s/100 iters), loss = 0.192637
I0112 16:57:07.899211 15972 solver.cpp:237] Train net output #0: loss = 0.192637 (* 1 = 0.192637 loss)
I0112 16:57:07.899216 15972 sgd_solver.cpp:105] Iteration 800, lr = 0.00943913
I0112 16:57:08.005599 15972 solver.cpp:218] Iteration 900 (940.042 iter/s, 0.106378s/100 iters), loss = 0.131764
I0112 16:57:08.005623 15972 solver.cpp:237] Train net output #0: loss = 0.131764 (* 1 = 0.131764 loss)
I0112 16:57:08.005627 15972 sgd_solver.cpp:105] Iteration 900, lr = 0.00937411
I0112 16:57:08.041206 15985 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:08.110942 15972 solver.cpp:330] Iteration 1000, Testing net (#0)
I0112 16:57:08.156153 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:08.157608 15972 solver.cpp:397] Test net output #0: accuracy = 0.9793
I0112 16:57:08.157624 15972 solver.cpp:397] Test net output #1: loss = 0.0635092 (* 1 = 0.0635092 loss)
I0112 16:57:08.158535 15972 solver.cpp:218] Iteration 1000 (654 iter/s, 0.152905s/100 iters), loss = 0.101569
I0112 16:57:08.158547 15972 solver.cpp:237] Train net output #0: loss = 0.101569 (* 1 = 0.101569 loss)
I0112 16:57:08.158552 15972 sgd_solver.cpp:105] Iteration 1000, lr = 0.00931012
I0112 16:57:08.265123 15972 solver.cpp:218] Iteration 1100 (938.383 iter/s, 0.106566s/100 iters), loss = 0.00940876
I0112 16:57:08.265149 15972 solver.cpp:237] Train net output #0: loss = 0.00940875 (* 1 = 0.00940875 loss)
I0112 16:57:08.265153 15972 sgd_solver.cpp:105] Iteration 1100, lr = 0.00924715
I0112 16:57:08.371924 15972 solver.cpp:218] Iteration 1200 (936.634 iter/s, 0.106765s/100 iters), loss = 0.0125595
I0112 16:57:08.371950 15972 solver.cpp:237] Train net output #0: loss = 0.0125595 (* 1 = 0.0125595 loss)
I0112 16:57:08.371954 15972 sgd_solver.cpp:105] Iteration 1200, lr = 0.00918515
I0112 16:57:08.477246 15972 solver.cpp:218] Iteration 1300 (949.788 iter/s, 0.105287s/100 iters), loss = 0.0129839
I0112 16:57:08.477270 15972 solver.cpp:237] Train net output #0: loss = 0.0129839 (* 1 = 0.0129839 loss)
I0112 16:57:08.477275 15972 sgd_solver.cpp:105] Iteration 1300, lr = 0.00912412
I0112 16:57:08.583765 15972 solver.cpp:218] Iteration 1400 (939.079 iter/s, 0.106487s/100 iters), loss = 0.00636841
I0112 16:57:08.583791 15972 solver.cpp:237] Train net output #0: loss = 0.00636842 (* 1 = 0.00636842 loss)
I0112 16:57:08.583796 15972 sgd_solver.cpp:105] Iteration 1400, lr = 0.00906403
I0112 16:57:08.687963 15972 solver.cpp:330] Iteration 1500, Testing net (#0)
I0112 16:57:08.726721 15972 blocking_queue.cpp:49] Waiting for data
I0112 16:57:08.733594 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:08.734315 15972 solver.cpp:397] Test net output #0: accuracy = 0.9844
I0112 16:57:08.734333 15972 solver.cpp:397] Test net output #1: loss = 0.0497479 (* 1 = 0.0497479 loss)
I0112 16:57:08.735321 15972 solver.cpp:218] Iteration 1500 (659.962 iter/s, 0.151524s/100 iters), loss = 0.0598267
I0112 16:57:08.735337 15972 solver.cpp:237] Train net output #0: loss = 0.0598267 (* 1 = 0.0598267 loss)
I0112 16:57:08.735343 15972 sgd_solver.cpp:105] Iteration 1500, lr = 0.00900485
I0112 16:57:08.843225 15972 solver.cpp:218] Iteration 1600 (926.965 iter/s, 0.107879s/100 iters), loss = 0.0870188
I0112 16:57:08.843266 15972 solver.cpp:237] Train net output #0: loss = 0.0870188 (* 1 = 0.0870188 loss)
I0112 16:57:08.843271 15972 sgd_solver.cpp:105] Iteration 1600, lr = 0.00894657
I0112 16:57:08.949916 15972 solver.cpp:218] Iteration 1700 (937.704 iter/s, 0.106643s/100 iters), loss = 0.0331378
I0112 16:57:08.949942 15972 solver.cpp:237] Train net output #0: loss = 0.0331378 (* 1 = 0.0331378 loss)
I0112 16:57:08.949947 15972 sgd_solver.cpp:105] Iteration 1700, lr = 0.00888916
I0112 16:57:09.055101 15972 solver.cpp:218] Iteration 1800 (951.01 iter/s, 0.105151s/100 iters), loss = 0.0248283
I0112 16:57:09.055125 15972 solver.cpp:237] Train net output #0: loss = 0.0248283 (* 1 = 0.0248283 loss)
I0112 16:57:09.055131 15972 sgd_solver.cpp:105] Iteration 1800, lr = 0.0088326
I0112 16:57:09.129863 15985 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:09.162317 15972 solver.cpp:218] Iteration 1900 (932.976 iter/s, 0.107184s/100 iters), loss = 0.125227
I0112 16:57:09.162345 15972 solver.cpp:237] Train net output #0: loss = 0.125227 (* 1 = 0.125227 loss)
I0112 16:57:09.162351 15972 sgd_solver.cpp:105] Iteration 1900, lr = 0.00877687
I0112 16:57:09.266227 15972 solver.cpp:330] Iteration 2000, Testing net (#0)
I0112 16:57:09.311487 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:09.312958 15972 solver.cpp:397] Test net output #0: accuracy = 0.9858
I0112 16:57:09.312975 15972 solver.cpp:397] Test net output #1: loss = 0.0417699 (* 1 = 0.0417699 loss)
I0112 16:57:09.313905 15972 solver.cpp:218] Iteration 2000 (659.835 iter/s, 0.151553s/100 iters), loss = 0.0162155
I0112 16:57:09.313917 15972 solver.cpp:237] Train net output #0: loss = 0.0162155 (* 1 = 0.0162155 loss)
I0112 16:57:09.313922 15972 sgd_solver.cpp:105] Iteration 2000, lr = 0.00872196
I0112 16:57:09.418843 15972 solver.cpp:218] Iteration 2100 (953.144 iter/s, 0.104916s/100 iters), loss = 0.0293796
I0112 16:57:09.418869 15972 solver.cpp:237] Train net output #0: loss = 0.0293796 (* 1 = 0.0293796 loss)
I0112 16:57:09.418874 15972 sgd_solver.cpp:105] Iteration 2100, lr = 0.00866784
I0112 16:57:09.526562 15972 solver.cpp:218] Iteration 2200 (928.62 iter/s, 0.107687s/100 iters), loss = 0.0165145
I0112 16:57:09.526588 15972 solver.cpp:237] Train net output #0: loss = 0.0165145 (* 1 = 0.0165145 loss)
I0112 16:57:09.526593 15972 sgd_solver.cpp:105] Iteration 2200, lr = 0.0086145
I0112 16:57:09.633265 15972 solver.cpp:218] Iteration 2300 (937.483 iter/s, 0.106669s/100 iters), loss = 0.0957732
I0112 16:57:09.633291 15972 solver.cpp:237] Train net output #0: loss = 0.0957731 (* 1 = 0.0957731 loss)
I0112 16:57:09.633294 15972 sgd_solver.cpp:105] Iteration 2300, lr = 0.00856192
I0112 16:57:09.738709 15972 solver.cpp:218] Iteration 2400 (948.667 iter/s, 0.105411s/100 iters), loss = 0.0110309
I0112 16:57:09.738735 15972 solver.cpp:237] Train net output #0: loss = 0.0110309 (* 1 = 0.0110309 loss)
I0112 16:57:09.738741 15972 sgd_solver.cpp:105] Iteration 2400, lr = 0.00851008
I0112 16:57:09.842794 15972 solver.cpp:330] Iteration 2500, Testing net (#0)
I0112 16:57:09.888008 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:09.889173 15972 solver.cpp:397] Test net output #0: accuracy = 0.9837
I0112 16:57:09.889190 15972 solver.cpp:397] Test net output #1: loss = 0.0480748 (* 1 = 0.0480748 loss)
I0112 16:57:09.890143 15972 solver.cpp:218] Iteration 2500 (660.488 iter/s, 0.151403s/100 iters), loss = 0.01583
I0112 16:57:09.890156 15972 solver.cpp:237] Train net output #0: loss = 0.01583 (* 1 = 0.01583 loss)
I0112 16:57:09.890161 15972 sgd_solver.cpp:105] Iteration 2500, lr = 0.00845897
I0112 16:57:09.996053 15972 solver.cpp:218] Iteration 2600 (944.406 iter/s, 0.105887s/100 iters), loss = 0.0721185
I0112 16:57:09.996080 15972 solver.cpp:237] Train net output #0: loss = 0.0721185 (* 1 = 0.0721185 loss)
I0112 16:57:09.996086 15972 sgd_solver.cpp:105] Iteration 2600, lr = 0.00840857
I0112 16:57:10.102244 15972 solver.cpp:218] Iteration 2700 (942.003 iter/s, 0.106157s/100 iters), loss = 0.0683237
I0112 16:57:10.102270 15972 solver.cpp:237] Train net output #0: loss = 0.0683237 (* 1 = 0.0683237 loss)
I0112 16:57:10.102275 15972 sgd_solver.cpp:105] Iteration 2700, lr = 0.00835886
I0112 16:57:10.209354 15972 solver.cpp:218] Iteration 2800 (933.904 iter/s, 0.107077s/100 iters), loss = 0.00175662
I0112 16:57:10.209381 15972 solver.cpp:237] Train net output #0: loss = 0.00175663 (* 1 = 0.00175663 loss)
I0112 16:57:10.209388 15972 sgd_solver.cpp:105] Iteration 2800, lr = 0.00830984
I0112 16:57:10.217985 15985 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:10.316431 15972 solver.cpp:218] Iteration 2900 (934.228 iter/s, 0.10704s/100 iters), loss = 0.0155334
I0112 16:57:10.316457 15972 solver.cpp:237] Train net output #0: loss = 0.0155334 (* 1 = 0.0155334 loss)
I0112 16:57:10.316462 15972 sgd_solver.cpp:105] Iteration 2900, lr = 0.00826148
I0112 16:57:10.421016 15972 solver.cpp:330] Iteration 3000, Testing net (#0)
I0112 16:57:10.466331 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:10.467798 15972 solver.cpp:397] Test net output #0: accuracy = 0.9865
I0112 16:57:10.467816 15972 solver.cpp:397] Test net output #1: loss = 0.0400159 (* 1 = 0.0400159 loss)
I0112 16:57:10.468765 15972 solver.cpp:218] Iteration 3000 (656.593 iter/s, 0.152301s/100 iters), loss = 0.00998391
I0112 16:57:10.468778 15972 solver.cpp:237] Train net output #0: loss = 0.00998392 (* 1 = 0.00998392 loss)
I0112 16:57:10.468785 15972 sgd_solver.cpp:105] Iteration 3000, lr = 0.00821377
I0112 16:57:10.574146 15972 solver.cpp:218] Iteration 3100 (949.126 iter/s, 0.10536s/100 iters), loss = 0.0119933
I0112 16:57:10.574170 15972 solver.cpp:237] Train net output #0: loss = 0.0119933 (* 1 = 0.0119933 loss)
I0112 16:57:10.574175 15972 sgd_solver.cpp:105] Iteration 3100, lr = 0.0081667
I0112 16:57:10.679939 15972 solver.cpp:218] Iteration 3200 (945.539 iter/s, 0.10576s/100 iters), loss = 0.00804365
I0112 16:57:10.679965 15972 solver.cpp:237] Train net output #0: loss = 0.00804365 (* 1 = 0.00804365 loss)
I0112 16:57:10.679970 15972 sgd_solver.cpp:105] Iteration 3200, lr = 0.00812025
I0112 16:57:10.787699 15972 solver.cpp:218] Iteration 3300 (928.618 iter/s, 0.107687s/100 iters), loss = 0.0357003
I0112 16:57:10.787724 15972 solver.cpp:237] Train net output #0: loss = 0.0357003 (* 1 = 0.0357003 loss)
I0112 16:57:10.787730 15972 sgd_solver.cpp:105] Iteration 3300, lr = 0.00807442
I0112 16:57:10.893801 15972 solver.cpp:218] Iteration 3400 (942.783 iter/s, 0.106069s/100 iters), loss = 0.00880067
I0112 16:57:10.893827 15972 solver.cpp:237] Train net output #0: loss = 0.00880066 (* 1 = 0.00880066 loss)
I0112 16:57:10.893832 15972 sgd_solver.cpp:105] Iteration 3400, lr = 0.00802918
I0112 16:57:10.997958 15972 solver.cpp:330] Iteration 3500, Testing net (#0)
I0112 16:57:11.042796 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:11.043982 15972 solver.cpp:397] Test net output #0: accuracy = 0.9876
I0112 16:57:11.043998 15972 solver.cpp:397] Test net output #1: loss = 0.0397221 (* 1 = 0.0397221 loss)
I0112 16:57:11.044992 15972 solver.cpp:218] Iteration 3500 (661.556 iter/s, 0.151159s/100 iters), loss = 0.00571682
I0112 16:57:11.045006 15972 solver.cpp:237] Train net output #0: loss = 0.00571682 (* 1 = 0.00571682 loss)
I0112 16:57:11.045011 15972 sgd_solver.cpp:105] Iteration 3500, lr = 0.00798454
I0112 16:57:11.150439 15972 solver.cpp:218] Iteration 3600 (948.552 iter/s, 0.105424s/100 iters), loss = 0.0340838
I0112 16:57:11.150466 15972 solver.cpp:237] Train net output #0: loss = 0.0340838 (* 1 = 0.0340838 loss)
I0112 16:57:11.150472 15972 sgd_solver.cpp:105] Iteration 3600, lr = 0.00794046
I0112 16:57:11.256443 15972 solver.cpp:218] Iteration 3700 (943.661 iter/s, 0.10597s/100 iters), loss = 0.0176656
I0112 16:57:11.256472 15972 solver.cpp:237] Train net output #0: loss = 0.0176656 (* 1 = 0.0176656 loss)
I0112 16:57:11.256492 15972 sgd_solver.cpp:105] Iteration 3700, lr = 0.00789695
I0112 16:57:11.305328 15985 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:11.362747 15972 solver.cpp:218] Iteration 3800 (941.019 iter/s, 0.106268s/100 iters), loss = 0.0108183
I0112 16:57:11.362776 15972 solver.cpp:237] Train net output #0: loss = 0.0108183 (* 1 = 0.0108183 loss)
I0112 16:57:11.362782 15972 sgd_solver.cpp:105] Iteration 3800, lr = 0.007854
I0112 16:57:11.468052 15972 solver.cpp:218] Iteration 3900 (949.938 iter/s, 0.10527s/100 iters), loss = 0.0445024
I0112 16:57:11.468080 15972 solver.cpp:237] Train net output #0: loss = 0.0445024 (* 1 = 0.0445024 loss)
I0112 16:57:11.468086 15972 sgd_solver.cpp:105] Iteration 3900, lr = 0.00781158
I0112 16:57:11.572196 15972 solver.cpp:330] Iteration 4000, Testing net (#0)
I0112 16:57:11.617511 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:11.618247 15972 solver.cpp:397] Test net output #0: accuracy = 0.99
I0112 16:57:11.618264 15972 solver.cpp:397] Test net output #1: loss = 0.0300614 (* 1 = 0.0300614 loss)
I0112 16:57:11.619247 15972 solver.cpp:218] Iteration 4000 (661.539 iter/s, 0.151163s/100 iters), loss = 0.0130394
I0112 16:57:11.619262 15972 solver.cpp:237] Train net output #0: loss = 0.0130394 (* 1 = 0.0130394 loss)
I0112 16:57:11.619269 15972 sgd_solver.cpp:105] Iteration 4000, lr = 0.0077697
I0112 16:57:11.724349 15972 solver.cpp:218] Iteration 4100 (951.667 iter/s, 0.105079s/100 iters), loss = 0.0309141
I0112 16:57:11.724376 15972 solver.cpp:237] Train net output #0: loss = 0.030914 (* 1 = 0.030914 loss)
I0112 16:57:11.724381 15972 sgd_solver.cpp:105] Iteration 4100, lr = 0.00772833
I0112 16:57:11.831650 15972 solver.cpp:218] Iteration 4200 (932.261 iter/s, 0.107266s/100 iters), loss = 0.0153108
I0112 16:57:11.831676 15972 solver.cpp:237] Train net output #0: loss = 0.0153108 (* 1 = 0.0153108 loss)
I0112 16:57:11.831679 15972 sgd_solver.cpp:105] Iteration 4200, lr = 0.00768748
I0112 16:57:11.937731 15972 solver.cpp:218] Iteration 4300 (942.974 iter/s, 0.106047s/100 iters), loss = 0.042013
I0112 16:57:11.937757 15972 solver.cpp:237] Train net output #0: loss = 0.0420129 (* 1 = 0.0420129 loss)
I0112 16:57:11.937762 15972 sgd_solver.cpp:105] Iteration 4300, lr = 0.00764712
I0112 16:57:12.044248 15972 solver.cpp:218] Iteration 4400 (939.132 iter/s, 0.106481s/100 iters), loss = 0.0216015
I0112 16:57:12.044273 15972 solver.cpp:237] Train net output #0: loss = 0.0216014 (* 1 = 0.0216014 loss)
I0112 16:57:12.044278 15972 sgd_solver.cpp:105] Iteration 4400, lr = 0.00760726
I0112 16:57:12.148998 15972 solver.cpp:330] Iteration 4500, Testing net (#0)
I0112 16:57:12.194150 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:12.195402 15972 solver.cpp:397] Test net output #0: accuracy = 0.9886
I0112 16:57:12.195430 15972 solver.cpp:397] Test net output #1: loss = 0.0372087 (* 1 = 0.0372087 loss)
I0112 16:57:12.196346 15972 solver.cpp:218] Iteration 4500 (657.618 iter/s, 0.152064s/100 iters), loss = 0.00623277
I0112 16:57:12.196362 15972 solver.cpp:237] Train net output #0: loss = 0.00623274 (* 1 = 0.00623274 loss)
I0112 16:57:12.196368 15972 sgd_solver.cpp:105] Iteration 4500, lr = 0.00756788
I0112 16:57:12.302517 15972 solver.cpp:218] Iteration 4600 (942.096 iter/s, 0.106146s/100 iters), loss = 0.0101835
I0112 16:57:12.302541 15972 solver.cpp:237] Train net output #0: loss = 0.0101834 (* 1 = 0.0101834 loss)
I0112 16:57:12.302546 15972 sgd_solver.cpp:105] Iteration 4600, lr = 0.00752897
I0112 16:57:12.391556 15985 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:12.409374 15972 solver.cpp:218] Iteration 4700 (936.122 iter/s, 0.106824s/100 iters), loss = 0.00535819
I0112 16:57:12.409400 15972 solver.cpp:237] Train net output #0: loss = 0.00535815 (* 1 = 0.00535815 loss)
I0112 16:57:12.409405 15972 sgd_solver.cpp:105] Iteration 4700, lr = 0.00749052
I0112 16:57:12.515069 15972 solver.cpp:218] Iteration 4800 (946.43 iter/s, 0.10566s/100 iters), loss = 0.0143103
I0112 16:57:12.515115 15972 solver.cpp:237] Train net output #0: loss = 0.0143102 (* 1 = 0.0143102 loss)
I0112 16:57:12.515118 15972 sgd_solver.cpp:105] Iteration 4800, lr = 0.00745253
I0112 16:57:12.620286 15972 solver.cpp:218] Iteration 4900 (950.898 iter/s, 0.105164s/100 iters), loss = 0.0104277
I0112 16:57:12.620312 15972 solver.cpp:237] Train net output #0: loss = 0.0104277 (* 1 = 0.0104277 loss)
I0112 16:57:12.620316 15972 sgd_solver.cpp:105] Iteration 4900, lr = 0.00741498
I0112 16:57:12.723434 15972 solver.cpp:447] Snapshotting to binary proto file examples/mnist/lenet_iter_5000.caffemodel
I0112 16:57:12.727984 15972 sgd_solver.cpp:273] Snapshotting solver state to binary proto file examples/mnist/lenet_iter_5000.solverstate
I0112 16:57:12.729532 15972 solver.cpp:330] Iteration 5000, Testing net (#0)
I0112 16:57:12.774241 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:12.775404 15972 solver.cpp:397] Test net output #0: accuracy = 0.9904
I0112 16:57:12.775423 15972 solver.cpp:397] Test net output #1: loss = 0.0291668 (* 1 = 0.0291668 loss)
I0112 16:57:12.776384 15972 solver.cpp:218] Iteration 5000 (640.755 iter/s, 0.156066s/100 iters), loss = 0.0374402
I0112 16:57:12.776398 15972 solver.cpp:237] Train net output #0: loss = 0.0374401 (* 1 = 0.0374401 loss)
I0112 16:57:12.776403 15972 sgd_solver.cpp:105] Iteration 5000, lr = 0.00737788
I0112 16:57:12.880753 15972 solver.cpp:218] Iteration 5100 (958.346 iter/s, 0.104346s/100 iters), loss = 0.0187042
I0112 16:57:12.880779 15972 solver.cpp:237] Train net output #0: loss = 0.0187041 (* 1 = 0.0187041 loss)
I0112 16:57:12.880784 15972 sgd_solver.cpp:105] Iteration 5100, lr = 0.0073412
I0112 16:57:12.987210 15972 solver.cpp:218] Iteration 5200 (939.653 iter/s, 0.106422s/100 iters), loss = 0.00544662
I0112 16:57:12.987236 15972 solver.cpp:237] Train net output #0: loss = 0.00544658 (* 1 = 0.00544658 loss)
I0112 16:57:12.987241 15972 sgd_solver.cpp:105] Iteration 5200, lr = 0.00730495
I0112 16:57:13.092567 15972 solver.cpp:218] Iteration 5300 (949.468 iter/s, 0.105322s/100 iters), loss = 0.00226478
I0112 16:57:13.092592 15972 solver.cpp:237] Train net output #0: loss = 0.00226473 (* 1 = 0.00226473 loss)
I0112 16:57:13.092597 15972 sgd_solver.cpp:105] Iteration 5300, lr = 0.00726911
I0112 16:57:13.199589 15972 solver.cpp:218] Iteration 5400 (934.688 iter/s, 0.106988s/100 iters), loss = 0.00857045
I0112 16:57:13.199614 15972 solver.cpp:237] Train net output #0: loss = 0.0085704 (* 1 = 0.0085704 loss)
I0112 16:57:13.199618 15972 sgd_solver.cpp:105] Iteration 5400, lr = 0.00723368
I0112 16:57:13.307129 15972 solver.cpp:330] Iteration 5500, Testing net (#0)
I0112 16:57:13.353451 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:13.354149 15972 solver.cpp:397] Test net output #0: accuracy = 0.9896
I0112 16:57:13.354166 15972 solver.cpp:397] Test net output #1: loss = 0.0317738 (* 1 = 0.0317738 loss)
I0112 16:57:13.355186 15972 solver.cpp:218] Iteration 5500 (642.814 iter/s, 0.155566s/100 iters), loss = 0.00679892
I0112 16:57:13.355202 15972 solver.cpp:237] Train net output #0: loss = 0.00679889 (* 1 = 0.00679889 loss)
I0112 16:57:13.355207 15972 sgd_solver.cpp:105] Iteration 5500, lr = 0.00719865
I0112 16:57:13.461220 15972 solver.cpp:218] Iteration 5600 (943.32 iter/s, 0.106009s/100 iters), loss = 0.000815537
I0112 16:57:13.461246 15972 solver.cpp:237] Train net output #0: loss = 0.000815498 (* 1 = 0.000815498 loss)
I0112 16:57:13.461251 15972 sgd_solver.cpp:105] Iteration 5600, lr = 0.00716402
I0112 16:57:13.482585 15985 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:13.568003 15972 solver.cpp:218] Iteration 5700 (936.775 iter/s, 0.106749s/100 iters), loss = 0.00257927
I0112 16:57:13.568029 15972 solver.cpp:237] Train net output #0: loss = 0.00257923 (* 1 = 0.00257923 loss)
I0112 16:57:13.568033 15972 sgd_solver.cpp:105] Iteration 5700, lr = 0.00712977
I0112 16:57:13.675142 15972 solver.cpp:218] Iteration 5800 (933.683 iter/s, 0.107103s/100 iters), loss = 0.0324859
I0112 16:57:13.675184 15972 solver.cpp:237] Train net output #0: loss = 0.0324858 (* 1 = 0.0324858 loss)
I0112 16:57:13.675189 15972 sgd_solver.cpp:105] Iteration 5800, lr = 0.0070959
I0112 16:57:13.779691 15972 solver.cpp:218] Iteration 5900 (956.936 iter/s, 0.1045s/100 iters), loss = 0.00432003
I0112 16:57:13.779717 15972 solver.cpp:237] Train net output #0: loss = 0.00431998 (* 1 = 0.00431998 loss)
I0112 16:57:13.779721 15972 sgd_solver.cpp:105] Iteration 5900, lr = 0.0070624
I0112 16:57:13.884342 15972 solver.cpp:330] Iteration 6000, Testing net (#0)
I0112 16:57:13.929298 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:13.930799 15972 solver.cpp:397] Test net output #0: accuracy = 0.991
I0112 16:57:13.930824 15972 solver.cpp:397] Test net output #1: loss = 0.0275324 (* 1 = 0.0275324 loss)
I0112 16:57:13.931828 15972 solver.cpp:218] Iteration 6000 (657.445 iter/s, 0.152104s/100 iters), loss = 0.00648839
I0112 16:57:13.931843 15972 solver.cpp:237] Train net output #0: loss = 0.00648833 (* 1 = 0.00648833 loss)
I0112 16:57:13.931849 15972 sgd_solver.cpp:105] Iteration 6000, lr = 0.00702927
I0112 16:57:14.035997 15972 solver.cpp:218] Iteration 6100 (960.191 iter/s, 0.104146s/100 iters), loss = 0.00184724
I0112 16:57:14.036022 15972 solver.cpp:237] Train net output #0: loss = 0.00184717 (* 1 = 0.00184717 loss)
I0112 16:57:14.036026 15972 sgd_solver.cpp:105] Iteration 6100, lr = 0.0069965
I0112 16:57:14.141840 15972 solver.cpp:218] Iteration 6200 (945.1 iter/s, 0.105809s/100 iters), loss = 0.0105831
I0112 16:57:14.141866 15972 solver.cpp:237] Train net output #0: loss = 0.010583 (* 1 = 0.010583 loss)
I0112 16:57:14.141870 15972 sgd_solver.cpp:105] Iteration 6200, lr = 0.00696408
I0112 16:57:14.247936 15972 solver.cpp:218] Iteration 6300 (942.851 iter/s, 0.106061s/100 iters), loss = 0.00713655
I0112 16:57:14.247961 15972 solver.cpp:237] Train net output #0: loss = 0.00713647 (* 1 = 0.00713647 loss)
I0112 16:57:14.247967 15972 sgd_solver.cpp:105] Iteration 6300, lr = 0.00693201
I0112 16:57:14.353080 15972 solver.cpp:218] Iteration 6400 (951.381 iter/s, 0.10511s/100 iters), loss = 0.00495782
I0112 16:57:14.353106 15972 solver.cpp:237] Train net output #0: loss = 0.00495776 (* 1 = 0.00495776 loss)
I0112 16:57:14.353109 15972 sgd_solver.cpp:105] Iteration 6400, lr = 0.00690029
I0112 16:57:14.456463 15972 solver.cpp:330] Iteration 6500, Testing net (#0)
I0112 16:57:14.501313 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:14.502967 15972 solver.cpp:397] Test net output #0: accuracy = 0.9896
I0112 16:57:14.502984 15972 solver.cpp:397] Test net output #1: loss = 0.032202 (* 1 = 0.032202 loss)
I0112 16:57:14.503904 15972 solver.cpp:218] Iteration 6500 (663.16 iter/s, 0.150793s/100 iters), loss = 0.0113013
I0112 16:57:14.503917 15972 solver.cpp:237] Train net output #0: loss = 0.0113013 (* 1 = 0.0113013 loss)
I0112 16:57:14.503922 15972 sgd_solver.cpp:105] Iteration 6500, lr = 0.0068689
I0112 16:57:14.566067 15985 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:14.610998 15972 solver.cpp:218] Iteration 6600 (933.955 iter/s, 0.107071s/100 iters), loss = 0.0261829
I0112 16:57:14.611023 15972 solver.cpp:237] Train net output #0: loss = 0.0261828 (* 1 = 0.0261828 loss)
I0112 16:57:14.611028 15972 sgd_solver.cpp:105] Iteration 6600, lr = 0.00683784
I0112 16:57:14.715735 15972 solver.cpp:218] Iteration 6700 (955.083 iter/s, 0.104703s/100 iters), loss = 0.00955832
I0112 16:57:14.715759 15972 solver.cpp:237] Train net output #0: loss = 0.00955829 (* 1 = 0.00955829 loss)
I0112 16:57:14.715764 15972 sgd_solver.cpp:105] Iteration 6700, lr = 0.00680711
I0112 16:57:14.823729 15972 solver.cpp:218] Iteration 6800 (926.258 iter/s, 0.107961s/100 iters), loss = 0.00218818
I0112 16:57:14.823757 15972 solver.cpp:237] Train net output #0: loss = 0.00218814 (* 1 = 0.00218814 loss)
I0112 16:57:14.823763 15972 sgd_solver.cpp:105] Iteration 6800, lr = 0.0067767
I0112 16:57:14.932546 15972 solver.cpp:218] Iteration 6900 (919.266 iter/s, 0.108782s/100 iters), loss = 0.00640281
I0112 16:57:14.932574 15972 solver.cpp:237] Train net output #0: loss = 0.00640276 (* 1 = 0.00640276 loss)
I0112 16:57:14.932579 15972 sgd_solver.cpp:105] Iteration 6900, lr = 0.0067466
I0112 16:57:15.038229 15972 solver.cpp:330] Iteration 7000, Testing net (#0)
I0112 16:57:15.083498 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:15.084985 15972 solver.cpp:397] Test net output #0: accuracy = 0.9909
I0112 16:57:15.085001 15972 solver.cpp:397] Test net output #1: loss = 0.0287419 (* 1 = 0.0287419 loss)
I0112 16:57:15.085930 15972 solver.cpp:218] Iteration 7000 (652.098 iter/s, 0.153351s/100 iters), loss = 0.00745059
I0112 16:57:15.085944 15972 solver.cpp:237] Train net output #0: loss = 0.00745054 (* 1 = 0.00745054 loss)
I0112 16:57:15.085950 15972 sgd_solver.cpp:105] Iteration 7000, lr = 0.00671681
I0112 16:57:15.192251 15972 solver.cpp:218] Iteration 7100 (940.75 iter/s, 0.106298s/100 iters), loss = 0.0092454
I0112 16:57:15.192276 15972 solver.cpp:237] Train net output #0: loss = 0.00924535 (* 1 = 0.00924535 loss)
I0112 16:57:15.192282 15972 sgd_solver.cpp:105] Iteration 7100, lr = 0.00668733
I0112 16:57:15.298920 15972 solver.cpp:218] Iteration 7200 (937.778 iter/s, 0.106635s/100 iters), loss = 0.00755884
I0112 16:57:15.298946 15972 solver.cpp:237] Train net output #0: loss = 0.00755879 (* 1 = 0.00755879 loss)
I0112 16:57:15.298951 15972 sgd_solver.cpp:105] Iteration 7200, lr = 0.00665815
I0112 16:57:15.405158 15972 solver.cpp:218] Iteration 7300 (941.592 iter/s, 0.106203s/100 iters), loss = 0.0164236
I0112 16:57:15.405182 15972 solver.cpp:237] Train net output #0: loss = 0.0164236 (* 1 = 0.0164236 loss)
I0112 16:57:15.405187 15972 sgd_solver.cpp:105] Iteration 7300, lr = 0.00662927
I0112 16:57:15.511374 15972 solver.cpp:218] Iteration 7400 (941.775 iter/s, 0.106182s/100 iters), loss = 0.00699133
I0112 16:57:15.511404 15972 solver.cpp:237] Train net output #0: loss = 0.00699128 (* 1 = 0.00699128 loss)
I0112 16:57:15.511409 15972 sgd_solver.cpp:105] Iteration 7400, lr = 0.00660067
I0112 16:57:15.612582 15985 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:15.616096 15972 solver.cpp:330] Iteration 7500, Testing net (#0)
I0112 16:57:15.661656 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:15.662813 15972 solver.cpp:397] Test net output #0: accuracy = 0.9888
I0112 16:57:15.662830 15972 solver.cpp:397] Test net output #1: loss = 0.0320767 (* 1 = 0.0320767 loss)
I0112 16:57:15.663769 15972 solver.cpp:218] Iteration 7500 (656.337 iter/s, 0.152361s/100 iters), loss = 0.00364991
I0112 16:57:15.663782 15972 solver.cpp:237] Train net output #0: loss = 0.00364984 (* 1 = 0.00364984 loss)
I0112 16:57:15.663789 15972 sgd_solver.cpp:105] Iteration 7500, lr = 0.00657236
I0112 16:57:15.769531 15972 solver.cpp:218] Iteration 7600 (945.732 iter/s, 0.105738s/100 iters), loss = 0.00753101
I0112 16:57:15.769559 15972 solver.cpp:237] Train net output #0: loss = 0.00753096 (* 1 = 0.00753096 loss)
I0112 16:57:15.769563 15972 sgd_solver.cpp:105] Iteration 7600, lr = 0.00654433
I0112 16:57:15.874474 15972 solver.cpp:218] Iteration 7700 (953.237 iter/s, 0.104906s/100 iters), loss = 0.0289207
I0112 16:57:15.874498 15972 solver.cpp:237] Train net output #0: loss = 0.0289206 (* 1 = 0.0289206 loss)
I0112 16:57:15.874502 15972 sgd_solver.cpp:105] Iteration 7700, lr = 0.00651658
I0112 16:57:15.980185 15972 solver.cpp:218] Iteration 7800 (946.291 iter/s, 0.105676s/100 iters), loss = 0.00428822
I0112 16:57:15.980212 15972 solver.cpp:237] Train net output #0: loss = 0.00428816 (* 1 = 0.00428816 loss)
I0112 16:57:15.980216 15972 sgd_solver.cpp:105] Iteration 7800, lr = 0.00648911
I0112 16:57:16.085966 15972 solver.cpp:218] Iteration 7900 (945.668 iter/s, 0.105745s/100 iters), loss = 0.00351561
I0112 16:57:16.085991 15972 solver.cpp:237] Train net output #0: loss = 0.00351555 (* 1 = 0.00351555 loss)
I0112 16:57:16.085995 15972 sgd_solver.cpp:105] Iteration 7900, lr = 0.0064619
I0112 16:57:16.190408 15972 solver.cpp:330] Iteration 8000, Testing net (#0)
I0112 16:57:16.235270 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:16.236681 15972 solver.cpp:397] Test net output #0: accuracy = 0.9908
I0112 16:57:16.236699 15972 solver.cpp:397] Test net output #1: loss = 0.0286602 (* 1 = 0.0286602 loss)
I0112 16:57:16.237660 15972 solver.cpp:218] Iteration 8000 (659.523 iter/s, 0.151625s/100 iters), loss = 0.00676719
I0112 16:57:16.237674 15972 solver.cpp:237] Train net output #0: loss = 0.00676713 (* 1 = 0.00676713 loss)
I0112 16:57:16.237679 15972 sgd_solver.cpp:105] Iteration 8000, lr = 0.00643496
I0112 16:57:16.347872 15972 solver.cpp:218] Iteration 8100 (907.546 iter/s, 0.110187s/100 iters), loss = 0.00852934
I0112 16:57:16.347895 15972 solver.cpp:237] Train net output #0: loss = 0.00852928 (* 1 = 0.00852928 loss)
I0112 16:57:16.347898 15972 sgd_solver.cpp:105] Iteration 8100, lr = 0.00640827
I0112 16:57:16.454094 15972 solver.cpp:218] Iteration 8200 (941.706 iter/s, 0.10619s/100 iters), loss = 0.0115561
I0112 16:57:16.454119 15972 solver.cpp:237] Train net output #0: loss = 0.011556 (* 1 = 0.011556 loss)
I0112 16:57:16.454124 15972 sgd_solver.cpp:105] Iteration 8200, lr = 0.00638185
I0112 16:57:16.559404 15972 solver.cpp:218] Iteration 8300 (949.887 iter/s, 0.105276s/100 iters), loss = 0.034071
I0112 16:57:16.559429 15972 solver.cpp:237] Train net output #0: loss = 0.0340709 (* 1 = 0.0340709 loss)
I0112 16:57:16.559433 15972 sgd_solver.cpp:105] Iteration 8300, lr = 0.00635567
I0112 16:57:16.664906 15972 solver.cpp:218] Iteration 8400 (948.156 iter/s, 0.105468s/100 iters), loss = 0.00643312
I0112 16:57:16.664930 15972 solver.cpp:237] Train net output #0: loss = 0.00643307 (* 1 = 0.00643307 loss)
I0112 16:57:16.664934 15972 sgd_solver.cpp:105] Iteration 8400, lr = 0.00632975
I0112 16:57:16.701421 15985 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:16.770987 15972 solver.cpp:330] Iteration 8500, Testing net (#0)
I0112 16:57:16.815822 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:16.816979 15972 solver.cpp:397] Test net output #0: accuracy = 0.9902
I0112 16:57:16.816998 15972 solver.cpp:397] Test net output #1: loss = 0.0293806 (* 1 = 0.0293806 loss)
I0112 16:57:16.817926 15972 solver.cpp:218] Iteration 8500 (653.64 iter/s, 0.152989s/100 iters), loss = 0.00642806
I0112 16:57:16.817939 15972 solver.cpp:237] Train net output #0: loss = 0.00642801 (* 1 = 0.00642801 loss)
I0112 16:57:16.817945 15972 sgd_solver.cpp:105] Iteration 8500, lr = 0.00630407
I0112 16:57:16.922857 15972 solver.cpp:218] Iteration 8600 (953.21 iter/s, 0.104909s/100 iters), loss = 0.000773309
I0112 16:57:16.922883 15972 solver.cpp:237] Train net output #0: loss = 0.000773261 (* 1 = 0.000773261 loss)
I0112 16:57:16.922886 15972 sgd_solver.cpp:105] Iteration 8600, lr = 0.00627864
I0112 16:57:17.028585 15972 solver.cpp:218] Iteration 8700 (946.13 iter/s, 0.105694s/100 iters), loss = 0.00240101
I0112 16:57:17.028609 15972 solver.cpp:237] Train net output #0: loss = 0.00240097 (* 1 = 0.00240097 loss)
I0112 16:57:17.028614 15972 sgd_solver.cpp:105] Iteration 8700, lr = 0.00625344
I0112 16:57:17.134737 15972 solver.cpp:218] Iteration 8800 (942.338 iter/s, 0.106119s/100 iters), loss = 0.000822896
I0112 16:57:17.134763 15972 solver.cpp:237] Train net output #0: loss = 0.000822852 (* 1 = 0.000822852 loss)
I0112 16:57:17.134766 15972 sgd_solver.cpp:105] Iteration 8800, lr = 0.00622847
I0112 16:57:17.240761 15972 solver.cpp:218] Iteration 8900 (943.52 iter/s, 0.105986s/100 iters), loss = 0.000357424
I0112 16:57:17.240788 15972 solver.cpp:237] Train net output #0: loss = 0.00035738 (* 1 = 0.00035738 loss)
I0112 16:57:17.240794 15972 sgd_solver.cpp:105] Iteration 8900, lr = 0.00620374
I0112 16:57:17.345778 15972 solver.cpp:330] Iteration 9000, Testing net (#0)
I0112 16:57:17.390568 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:17.392069 15972 solver.cpp:397] Test net output #0: accuracy = 0.9901
I0112 16:57:17.392086 15972 solver.cpp:397] Test net output #1: loss = 0.0306536 (* 1 = 0.0306536 loss)
I0112 16:57:17.393012 15972 solver.cpp:218] Iteration 9000 (656.95 iter/s, 0.152219s/100 iters), loss = 0.0103844
I0112 16:57:17.393026 15972 solver.cpp:237] Train net output #0: loss = 0.0103843 (* 1 = 0.0103843 loss)
I0112 16:57:17.393031 15972 sgd_solver.cpp:105] Iteration 9000, lr = 0.00617924
I0112 16:57:17.498286 15972 solver.cpp:218] Iteration 9100 (950.113 iter/s, 0.105251s/100 iters), loss = 0.00593015
I0112 16:57:17.498311 15972 solver.cpp:237] Train net output #0: loss = 0.00593011 (* 1 = 0.00593011 loss)
I0112 16:57:17.498317 15972 sgd_solver.cpp:105] Iteration 9100, lr = 0.00615496
I0112 16:57:17.604336 15972 solver.cpp:218] Iteration 9200 (943.254 iter/s, 0.106016s/100 iters), loss = 0.00277393
I0112 16:57:17.604362 15972 solver.cpp:237] Train net output #0: loss = 0.00277389 (* 1 = 0.00277389 loss)
I0112 16:57:17.604365 15972 sgd_solver.cpp:105] Iteration 9200, lr = 0.0061309
I0112 16:57:17.710184 15972 solver.cpp:218] Iteration 9300 (945.038 iter/s, 0.105816s/100 iters), loss = 0.0077666
I0112 16:57:17.710209 15972 solver.cpp:237] Train net output #0: loss = 0.00776655 (* 1 = 0.00776655 loss)
I0112 16:57:17.710213 15972 sgd_solver.cpp:105] Iteration 9300, lr = 0.00610706
I0112 16:57:17.786052 15985 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:17.818255 15972 solver.cpp:218] Iteration 9400 (925.6 iter/s, 0.108038s/100 iters), loss = 0.0196706
I0112 16:57:17.818280 15972 solver.cpp:237] Train net output #0: loss = 0.0196705 (* 1 = 0.0196705 loss)
I0112 16:57:17.818284 15972 sgd_solver.cpp:105] Iteration 9400, lr = 0.00608343
I0112 16:57:17.922022 15972 solver.cpp:330] Iteration 9500, Testing net (#0)
I0112 16:57:17.967216 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:17.968677 15972 solver.cpp:397] Test net output #0: accuracy = 0.9898
I0112 16:57:17.968694 15972 solver.cpp:397] Test net output #1: loss = 0.0338548 (* 1 = 0.0338548 loss)
I0112 16:57:17.969615 15972 solver.cpp:218] Iteration 9500 (660.818 iter/s, 0.151328s/100 iters), loss = 0.00473422
I0112 16:57:17.969630 15972 solver.cpp:237] Train net output #0: loss = 0.00473418 (* 1 = 0.00473418 loss)
I0112 16:57:17.969635 15972 sgd_solver.cpp:105] Iteration 9500, lr = 0.00606002
I0112 16:57:18.075714 15972 solver.cpp:218] Iteration 9600 (942.724 iter/s, 0.106076s/100 iters), loss = 0.00438592
I0112 16:57:18.075738 15972 solver.cpp:237] Train net output #0: loss = 0.00438588 (* 1 = 0.00438588 loss)
I0112 16:57:18.075743 15972 sgd_solver.cpp:105] Iteration 9600, lr = 0.00603682
I0112 16:57:18.180905 15972 solver.cpp:218] Iteration 9700 (950.944 iter/s, 0.105159s/100 iters), loss = 0.00262575
I0112 16:57:18.180933 15972 solver.cpp:237] Train net output #0: loss = 0.00262571 (* 1 = 0.00262571 loss)
I0112 16:57:18.180938 15972 sgd_solver.cpp:105] Iteration 9700, lr = 0.00601382
I0112 16:57:18.287052 15972 solver.cpp:218] Iteration 9800 (942.419 iter/s, 0.10611s/100 iters), loss = 0.0164966
I0112 16:57:18.287078 15972 solver.cpp:237] Train net output #0: loss = 0.0164966 (* 1 = 0.0164966 loss)
I0112 16:57:18.287084 15972 sgd_solver.cpp:105] Iteration 9800, lr = 0.00599102
I0112 16:57:18.393187 15972 solver.cpp:218] Iteration 9900 (942.501 iter/s, 0.106101s/100 iters), loss = 0.00642607
I0112 16:57:18.393215 15972 solver.cpp:237] Train net output #0: loss = 0.00642603 (* 1 = 0.00642603 loss)
I0112 16:57:18.393221 15972 sgd_solver.cpp:105] Iteration 9900, lr = 0.00596843
I0112 16:57:18.497715 15972 solver.cpp:447] Snapshotting to binary proto file examples/mnist/lenet_iter_10000.caffemodel
I0112 16:57:18.501235 15972 sgd_solver.cpp:273] Snapshotting solver state to binary proto file examples/mnist/lenet_iter_10000.solverstate
I0112 16:57:18.503170 15972 solver.cpp:310] Iteration 10000, loss = 0.00217584
I0112 16:57:18.503185 15972 solver.cpp:330] Iteration 10000, Testing net (#0)
I0112 16:57:18.548034 15986 data_layer.cpp:73] Restarting data prefetching from start.
I0112 16:57:18.549484 15972 solver.cpp:397] Test net output #0: accuracy = 0.991
I0112 16:57:18.549500 15972 solver.cpp:397] Test net output #1: loss = 0.0278947 (* 1 = 0.0278947 loss)
I0112 16:57:18.549504 15972 solver.cpp:315] Optimization Done.
I0112 16:57:18.549506 15972 caffe.cpp:259] Optimization Done.

It obviously learns nothing at all.

Steps to reproduce

Makefile.config

# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers
# USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
# You should not set this flag if you will be reading LMDBs with any
# possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3
OPENCV_VERSION := 3

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20
-gencode arch=compute_20,code=sm_21
-gencode arch=compute_30,code=sm_30
-gencode arch=compute_35,code=sm_35
-gencode arch=compute_50,code=sm_50
-gencode arch=compute_52,code=sm_52
-gencode arch=compute_60,code=sm_60
-gencode arch=compute_61,code=sm_61
-gencode arch=compute_61,code=compute_61

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
#BLAS_INCLUDE := /opt/OpenBLAS/include
#BLAS_LIB := /opt/OpenBLAS/lib

# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python2.7
/usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include
# $(ANACONDA_HOME)/include/python2.7
# $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include

# Uncomment to use Python 3 (default is Python 2)
# PYTHON_LIBRARIES := boost_python-py35 python3.5m
# PYTHON_INCLUDE := /usr/include/python3.5m
# /usr/lib/python3.5/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib
# PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.file)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)
WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib

# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1

# Uncomment to use pkg-config to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on make clean
BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to #171
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @

Your system configuration

Operating system: Ubuntu 16.04
Compiler:GCC
CUDA version (if applicable):8.0
CUDNN version (if applicable):8.0
BLAS:ATLAS
Python or MATLAB version (for pycaffe and matcaffe respectively):pycaffe
@Vilour Vilour closed this as completed Jan 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Vilour