Tensor slice is too large to serialize

[jinhou]

Hi,
I met a issue said that "Tensor slice is too large to serialize (conservative estimate: 2268204567 bytes)". And i am wondered that why there is a limitation to prevents the TensorSliceWriter from attempting to serialize variables that are larger than 2GB. How can I to skip this limitation if I want to use a variable large than 2GB?

Thanks
Jinlong

[aselle]

There is a hard limit of 2GB for serializing individual tensors because of the 32bit signed size in protobuf.

See #3766, #2648, etc.
http://stackoverflow.com/questions/34128872/google-protobuf-maximum-size

You need to find a way to break things up into smaller tensors/variables.

[jinhou]

@aselle Thanks for your quick response. Are you planning to provide some operations to break big tensors/variables into smaller ones, since it is very easy to exceed 2 GB limitation for some big models.

[aselle]

@concretevitamin can comment further if need be, but the plan is to make tensors be checkpointed in a format that does not use protobufs, sidestepping this limitation.

[concretevitamin]

Agreed with @aselle. @jinhou, a quick workaround is to use, for variables, tf.get_variable(..., partitioner=..), and for non-ref tensors, tf.split().

[formidable255]

@concretevitamin Are you able to provide an example on how to use tf.split(). Currently my code is something like:
dataset = tf.data.Dataset.from_tensor_slices((x,y)).map(parse)
dataset = dataset.shuffle(buffer_size).repeat().batch(batch_size)
return dataset.make_one_shot_iterator().get_next()

[paulzhangh]

finally, how to solve that problem?