Cross validation support

[dan-ryan]

Looking at the docs I couldn't see anything about cross-validation built into brain.js. A way to input a test size out of your input data and set test error would be nice to have.

May I suggest adding options similar to Neataptic.

[robertleeplummerjr]

There is no cross validator that exists currently in brain, but I actually am needing one as well. I think the proposal is simple and to the point, it'd be to simply add something like:

network.train(trainingSet, {
  crossValidate :
    {
      testSize: 0.4,
      testError: 0.02
    }
});

[robertleeplummerjr]

I'm actually dead wrong! https://github.com/BrainJS/brain.js/blob/11324b7dead480577a41420da8cbd32c3cd06684/src/cross-validate.js It does exist. @mubaidr what do you think about evaluating unit tests on it and ensuring it gets properly documented?

[mubaidr]

Sure, will do.

[robertleeplummerjr]

As it exists, we currently have this as the api:

brain.crossValidate(brain.NeuralNetwork, trainingData, opts, trainingOpts);

Because of the 1. amount of arguments, 2. that this isn't a documented feature, and 3. the existing api, what if we 1. shortened arguments to a single one, and 2. provided a means of passing in either a constructor or a instance of the neural network. Proposal:

brain.crossValidate({
  constructor: brain.NeuralNetwork, // use this
  instance: new brain.NerualNetwork() // or use this, but not both
  data,
  trainingOpts,
  opts,
  folds
});

[robertleeplummerjr]

I think we want to have the winning network as a result. Perhaps something like keyNetwork?

[robertleeplummerjr]

I ended up finding a much more dynamic means by using a class and instance rather than using raw json and passing the classifier around. It is up for code review, and is as well documented therein.

[robertleeplummerjr]

Documentation here, we may make a couple tweaks to it in the next few days, but it is now supported and part of the family.

[robertleeplummerjr]

To clarify, this is the existing api as it has landed:

const crossValidate = new brain.CrossValidate(brain.NeuralNetwork);
const stats = crossValidate.train(data, networkOptions, trainingOptions);
const net = crossValidate.toNetwork();


// optionally later
const json = crossValidate.toJSON();
const net = crossValidate.fromJSON(json);

I think one minor thing we want to do with it is this (proposal):

const crossValidate = new brain.CrossValidate(brain.NeuralNetwork, networkOptions);
const stats = crossValidate.train(data, trainingOptions);
const net = crossValidate.toNetwork();

The reason is so we potentially could use cross validate with something like TrainStream (proposal):

const crossValidate = new brain.CrossValidate(brain.NeuralNetwork, networkOptions);
const stream = new brain.TrainStream({ neuralNetwork: crossValidate ...});

This way the options for the net are attached to the instance, can be reused, and now we can handle REAL BIG DATA in the train stream using cross validate.

Also, somewhat related, very little is now preventing us from using any other network in TrainStream, so it follows the flow of using tiny peaces to do amazing things without much work.

[mubaidr]

Looks good. Optiins are passed with theor relative user.

[amadeus-torwell]

Dear all,

when trying to run the example code on branch v1.x I am constantly getting this error:

(node:2888) UnhandledPromiseRejectionWarning: ReferenceError: options is not defined at CrossValidate.train (/server/node_modules/brain.js/dist/cross-validate.js:141:41)

It happens when I try to run crossValidate.train(..):

const crossValidate = new brain.CrossValidate(brain.NeuralNetwork, {});
const stats = crossValidate.train(probes, {});

[mubaidr]

@deusama A fix is on the way! Expect new release soon: #280

[amadeus-torwell]

The fix works for me, but I get another error when trying to read 'trainingData' from the cross validation example code:

(node:3000) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'input' of undefined at NeuralNetwork._formatData (/server/node_modules/brain.js/dist/neural-network.js:846:27) at NeuralNetwork._prepTraining (/server/node_modules/brain.js/dist/neural-network.js:513:19) at NeuralNetwork.train (/server/node_modules/brain.js/dist/neural-network.js:545:33) at CrossValidate.testPartition (/server/node_modules/brain.js/dist/cross-validate.js:40:38) at CrossValidate.train (/server/node_modules/brain.js/dist/cross-validate.js:141:27)

It seems there is something wrong with the data format expected by train().

[robertleeplummerjr]

Looking into this.

[robertleeplummerjr]

@deusama can you put a small jsfiddle or similar together demoing the problem?

[amadeus-torwell]

Sure, https://repl.it/repls/SnoopyDeeppinkDigits

[mubaidr]

@robertleeplummerjr data is an object but an array method is being called on it:

brain.js/src/cross-validate.js

Line 116 in 5797b87

let dclone = data.slice(0);

from here:

brain.js/src/cross-validate.js

Line 88 in 5797b87

let newData = {};

[mubaidr]

Added & Fixed in: eebe6c8 & #281