Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ResourceExhaustedError while runing the program on VGG16 #29

Open
Pan-zhaoyu opened this issue Nov 22, 2017 · 1 comment
Open

ResourceExhaustedError while runing the program on VGG16 #29

Pan-zhaoyu opened this issue Nov 22, 2017 · 1 comment

Comments

@Pan-zhaoyu
Copy link

Caused by op u'block5_fc6/truncated_normal/TruncatedNormal', defined at:
File "/home/robotics/keras-fcn/FCN-16.py", line 2, in
fcn_vgg16 = FCN(input_shape=(500, 500, 3), classes=21, weights='imagenet', trainable_encoder=True)
File "/home/robotics/keras-fcn/keras_fcn/models.py", line 29, in FCN
return FCN_VGG16(*args, **kwargs)
File "/home/robotics/keras-fcn/keras_fcn/models.py", line 54, in FCN_VGG16
weights=weights, trainable=trainable_encoder)
File "/home/robotics/keras-fcn/keras_fcn/encoders.py", line 150, in init
trainable=trainable)
File "/home/robotics/keras-fcn/keras_fcn/encoders.py", line 124, in init
weights=weights, trainable=trainable)
File "/home/robotics/keras-fcn/keras_fcn/encoders.py", line 61, in init
y = fc_block(x)
File "/home/robotics/keras-fcn/keras_fcn/blocks.py", line 64, in f
name='{}_fc6'.format(block_name))(x)
File "/home/robotics/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 575, in call
self.build(input_shapes[0])
File "/home/robotics/anaconda2/lib/python2.7/site-packages/keras/layers/convolutional.py", line 134, in build
constraint=self.kernel_constraint)
File "/home/robotics/anaconda2/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
return func(*args, **kwargs)
File "/home/robotics/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 396, in add_weight
weight = K.variable(initializer(shape),
File "/home/robotics/anaconda2/lib/python2.7/site-packages/keras/initializers.py", line 208, in call
dtype=dtype, seed=self.seed)
File "/home/robotics/anaconda2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 3590, in truncated_normal
return tf.truncated_normal(shape, mean, stddev, dtype=dtype, seed=seed)
File "/home/robotics/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/random_ops.py", line 172, in truncated_normal
shape_tensor, dtype, seed=seed1, seed2=seed2)
File "/home/robotics/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_random_ops.py", line 316, in _truncated_normal
seed=seed, seed2=seed2, name=name)
File "/home/robotics/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/home/robotics/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/robotics/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[7,7,512,4096]
[[Node: block5_fc6/truncated_normal/TruncatedNormal = TruncatedNormalT=DT_INT32, dtype=DT_FLOAT, seed=87654321, seed2=8705206, _device="/job:localhost/replica:0/task:0/gpu:0"]]
Here are my errors when I run the program on VGG16.
My PC is:
name: GeForce GTX 1080 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.645
pciBusID 0000:01:00.0
Total memory: 10.90GiB
Free memory: 341.00MiB
I think the problem is about the GPU, but I don't know how to figure out it. BTW, I am the beginner of this method.
I will appreciate it if you could give me some advice. Thanks~

@Pan-zhaoyu
Copy link
Author

Pan-zhaoyu commented Nov 22, 2017

I ran the same program again, and I found another error as following:
Traceback (most recent call last):
File "/home/robotics/keras-fcn/FCN-16.py", line 4, in
fcn_vgg16.fit(X_train, y_train, batch_size=1)
NameError: name 'X_train' is not defined

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant