Functional Preprocessing and Augmentation API

[ahundt]

Edit: A new example proposal + details is in #6891 (comment).

Discussion about a dense prediction API such as image segmentation in #6538 brought up the possibility of a functional preprocessing API, which could move preprocessing steps into the Keras backend APIs and generalize preprocessing to more network designs. Dropout provides a precedent for such augmentation layers.

If the layers could be designed much like dropout, and I would expect them to be configured so that the augmentation operations could be applied identically to one or more image inputs as well as one or more image label data, useful for dense prediction tasks.

This could have the advantages of being easy to use, easily applied consistently for arbitrary data inputs, and make it possible to use the TF backend image augmentation APIs thus improving performance.

What would be the pros/cons and barriers to a Functional Preprocessing API?

Example usage, label augmentation optional for dense prediction tasks:

L = InputLabel(...)
input = Input(...)

# augmentation
x,L = Flip(axis=0)([input,L])
x,L = Flip(axis=1)([x,L])
x,L = Zoom(range=[0.5,2])([x,L])
x,L = Transform(matrix=[1,0,0,0,1,0,0,0,1], interpolation='bilinear')([x,L])
x,L = RandomCrop(fill_mode='constant', cval=0.)([x,L])

# network definition
x = Conv2D(inter_channel, (1, 1), kernel_initializer='he_uniform', padding='same', use_bias=False,
                   kernel_regularizer=l2(weight_decay))(x)

# ...snip...
model = Model(inputs=input, outputs=x, labels=L)
model.compile(Adam(), loss='categorical_crossentropy',metrics=['acc'])
# ...snip...
model.fit()

[Dref360]

Is there anything like the tf preprocessing for theano? Otherwise, I guess it's a deal-breaker for theano users.

[ahundt]

@Dref360 I don't know, perhaps plain python APIs could be run on the theano side? They already have the sequential api you mentioned so adaptation may not be too hard, where would a feature request for theano be submitted?

[ahundt]

Edit: A new example proposal + details is in #6891 (comment).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[ahundt]

If you're using the TensorFlow backend, you can just use input tensors by following the mnist_tfrecord.py example, and before passing your tensors to the Input layer apply TensorFlow preprocessing ops such as tf.random_crop.

Since there is a way to accomplish this, I'm closing the issue.