diff --git a/README.md b/README.md index 9dad706e..a4b05394 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,6 @@ Visit each link for its content 5. [Data modeling and feature extraction](saul-core/doc/DATAMODELING.md) 6. [Learners and constraints](saul-core/doc/SAULLANGUAGE.md) 7. [Model configurations](saul-core/doc/MODELS.md) - 8. [Saul library](saul-core/doc/LBJLIBRARY.md) The api docs are included [here](http://cogcomp.cs.illinois.edu/software/doc/saul/). diff --git a/saul-core/doc/MODELS.md b/saul-core/doc/MODELS.md index cfdacadf..10df04df 100644 --- a/saul-core/doc/MODELS.md +++ b/saul-core/doc/MODELS.md @@ -1,6 +1,141 @@ -* Designing flexible learning models including various configurations such as: - * Local models i.e. single classifiers. (Learning only models (LO)). - * Constrained conditional models (CCM)[1] for training independent classifiers and using them jointly for global decision making in prediction time. (Learning+Inference (L+I)). - * Global models for joint training and joint inference (Inference-Based-Training (IBT)). - * Pipeline models for complex problems where the output of each layer is used as the input of the next layer. + +#Learning Paradigms + +/*Documented by Parisa Kordjamshidi*/ + +Saul facilitates the flexible design of complex learning models with various configurations. +By complex models we mean the models that aim at prediction of more than one output variable where these outputs might have relationships to each other. +Such models can be designed using the following paradigms, + + * [Local models](#local) trains single classifiers (Learning only models (LO)) each of which learns and predicts a single variable in the output independently. + * [Constrained conditional models (CCM)](#L+I) for training independent classifiers and using them jointly for global decision making in prediction time. (Learning+Inference (L+I)). + * [Global models](#IBT) for joint training and joint inference (Inference-Based-Training (IBT)). + * [Pipeline models](#pipeline) for complex problems where the output of each model is used as the input of the next model (these models are different layers in a pipeline). + +The above mentioned paradigms can be tested using this simple badge classifier example, [here](saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BagesApp.scala). + + +##Local models +These models are a set of single classifiers. Each classifier is defined with the `Learnable` construct and is trained and makes prediction independent from other classifiers. + The `Learnable` construct requires specifying a single output variable, that is, a label which is itself a property in the data model, and the features which is also a + comma separated list of properties. + + ```scala + object ClassifierName extends Learnable (node) { + def label = property1 + def feature = using(property2,property3,...) + //a comma separated list of properties + } + ``` + + For the details about the `Learnable` construct see [here](SAULLANGUAGE.md). + + +##Learning+Inference models +These models are useful for when we need to consider the global relations between the outputs of a bunch of classifiers during the +prediction time. Each classifier is defined with the same `Learnable` construct as a local model. In addition to the Learnable definitions, the programmer +has the possibility to define a number of logical constraints between the output of the Learnables (classifiers). +Having the constraint definitions in place (see [here](SAULLANGUAGE.md) for syntax details), the programmer is able to define +new constrained classifiers that use the Learnables and constraints. + +```scala +object ConstrainedClassifierName extends ConstrainedClassifier[local_node_type,global_node_type](LocalClassifier) + { + def subjectTo = constraintExpression + // Logical constraint expression comes here, it defines the relations between the + // LocalClassifier and other Learnables defined before + } + ``` +When we use the above `ConstrainedClassifierName` to call test or make predictions, the `LocalClassifier` is used +but the predictions are made in way that `constraintExpression` is hold. There is no limitation for the type of local classifiers. +They can be SVMs, decision trees or any other learning models available in Saul, [here](lbjava/blob/master/lbjava/doc/ALGORITHMS.md) + and [here](saul-core/src/main/java/edu/illinois/cs/cogcomp/saul/learn/SaulWekaWrapper.md). + + +##Inference-based Learning +For the inference based models the basic definitions are exactly similar to the L+I models. In other words, the programmer +just needs to define the `Learnables` and `ConstrainedClassifiers`. However, to train the ConstrainedClassifiers jointly, instead of +training local classifiers independently, there are a couple of joint training functions that can be called. +These functions receive the list of ConstrainedClassifiers as input and train their parameters jointly. In contrast to +L+I models here the local classifiers can be defined as `SparsePerceptron`s or `SparseNetworkLearner`s only. This is because the +joint training should have its own strategy for the wight updates of the involved variables (those variables come down to be the outputs of the local classifiers here). +For the two cases the programmer can use + +```scala + JointTrainSparseNetwork.train(param1,param2,...) /* a list of parameters go here*/ + JointTrainSparsePerceptron.train(param1,param2,...) /*a list of parameters here*/ +``` + +For example, + + ```scala + JointTrainSparseNetwork.train(badge, cls, 5, init = true, lossAugmented = true) + ``` + +The list of parameters are the following: + +- param1: The name of a global node in the data model that itself or the connected nodes to it are used by the involved `Learnable`s. + +- param2: The collection of ```ConstainedClassifier```s + +- param3: The number of iterations over the training data. + +- param4: If the local classifiers should be cleaned from the possibly pre-trained values and initialized by zero weights, this parameter should be true. + +- param5: If the approach uses the loss augmented objective for making inference, see below for description. + +###Basic approach + +The basic approach for training the models jointly is to do a global prediction at each step of the training and if the +predictions are wrong update the weights of the related variables. + +###Loss augmented + +The loss-augmented approach adds the loss of the prediction explicitly to the objective of the training and finds the most violated output per each training example; +it updates the weights of the model according to the errors made in the prediction of the most violated output. +This approach minimizes a convex upper bound of the loss function and has been used in structured SVMs and Structured Perceptrons. + However, considering an arbitrary loss in the objective will make complexities in the optimization, therefore in the implemented version, here, we assume the loss is decomposed similar to +feature function. That is, the loss is a hamming loss defined per classifier. The loss of the whole structured output is computed by the weighted sum of + the loss of its components. For exmaple if there are two variables `c1` and `c2` in the output with corresponding predictions `cp1` and `cp2` then the loss will be + `[hamming(c1,cp1)/2+hamming(c2,cp2)/2]`. The weight of each component's loss is `1/numberOfOutputVariables` by default. + + In Saul, the programmer can indicate if he/she needs to consider this global hamming loss in the objective or not. And this can be done by passing + the above mentioned `param5` as true in the `JointTrainingSparseNetwork` algorithm. + An example of this usage can be seen [here](saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BagesApp.scala#L64). + + +##Pipelines + +Building pipelines is naturally granted in Saul. The programmer can simply define properties that are the predictions of +the classifiers and use those outputs as the input of other classifiers by mentioning them in the list of the properties in the below construct when defining the +pipeline classifiers, + +```scala + def feature = using(/*list of properties including the prediction of other classifiers.*/) +``` + +Here is a more complete example which passes the output of the `ClassifierLayer1` to the input of the `ClassifierLayer2`: + + ```scala + object ClassifierLayer1 extends Learnable (node) { + def label = labelProperty1 + def feature = using(property2, property3,...) + } + object ClassifierLayer2 extends Learnable (node) { + def label = labelProperty2 + def feature = using(classifier1Labels, ,...) // using the prediction of the classifier in the previous layer + } + ``` + +This will be defined in data-model object: + + ```scala + val classifier1Labels = new Property(node){ x: Type => ClassifierLayer1(x) } + ``` + +See [here](saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeClassifiers.scala#L43) for a working example. + + + + diff --git a/saul-core/doc/SAULLANGUAGE.md b/saul-core/doc/SAULLANGUAGE.md index ec022301..f4cab89a 100644 --- a/saul-core/doc/SAULLANGUAGE.md +++ b/saul-core/doc/SAULLANGUAGE.md @@ -35,8 +35,8 @@ OrgClassifier.test() ### Availale algorithms Here is a list of available algorithms in Saul: - - [LBJava learning algorithms](https://github.com/IllinoisCogComp/lbjava/blob/master/lbjava/doc/ALGORITHMS.md) - - [Weka learning algorithms](https://github.com/IllinoisCogComp/saul/blob/master/saul-core/src/main/java/edu/illinois/cs/cogcomp/saul/learn/SaulWekaWrapper.md) + - [LBJava learning algorithms](https://github.com/IllinoisCogComp/lbjava/blob/master/lbjava/doc/ALGORITHMS.md) + - [Weka learning algorithms](saul-core/src/main/java/edu/illinois/cs/cogcomp/saul/learn/SaulWekaWrapper.md) ### Saving and loading classifiers diff --git a/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparseNetwork.scala b/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparseNetwork.scala index 4ac33f82..e114fadf 100644 --- a/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparseNetwork.scala +++ b/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparseNetwork.scala @@ -9,7 +9,7 @@ package edu.illinois.cs.cogcomp.saul.classifier import edu.illinois.cs.cogcomp.lbjava.learn.{ LinearThresholdUnit, SparseNetworkLearner } import edu.illinois.cs.cogcomp.saul.datamodel.node.Node import org.slf4j.{ Logger, LoggerFactory } - +import Predef._ import scala.reflect.ClassTag /** Created by Parisa on 5/22/15. @@ -18,16 +18,16 @@ object JointTrainSparseNetwork { val logger: Logger = LoggerFactory.getLogger(this.getClass) var difference = 0 - def apply[HEAD <: AnyRef](node: Node[HEAD], cls: List[ConstrainedClassifier[_, HEAD]], init: Boolean)(implicit headTag: ClassTag[HEAD]) = { - train[HEAD](node, cls, 1, init) + def apply[HEAD <: AnyRef](node: Node[HEAD], cls: List[ConstrainedClassifier[_, HEAD]], init: Boolean, lossAugmented: Boolean)(implicit headTag: ClassTag[HEAD]) = { + train[HEAD](node, cls, 1, init, lossAugmented) } - def apply[HEAD <: AnyRef](node: Node[HEAD], cls: List[ConstrainedClassifier[_, HEAD]], it: Int, init: Boolean)(implicit headTag: ClassTag[HEAD]) = { - train[HEAD](node, cls, it, init) + def apply[HEAD <: AnyRef](node: Node[HEAD], cls: List[ConstrainedClassifier[_, HEAD]], it: Int, init: Boolean, lossAugmented: Boolean = false)(implicit headTag: ClassTag[HEAD]) = { + train[HEAD](node, cls, it, init, lossAugmented) } @scala.annotation.tailrec - def train[HEAD <: AnyRef](node: Node[HEAD], cls: List[ConstrainedClassifier[_, HEAD]], it: Int, init: Boolean)(implicit headTag: ClassTag[HEAD]): Unit = { + def train[HEAD <: AnyRef](node: Node[HEAD], cls: List[ConstrainedClassifier[_, HEAD]], it: Int, init: Boolean, lossAugmented: Boolean = false)(implicit headTag: ClassTag[HEAD]): Unit = { // forall members in collection of the head (dm.t) do logger.info("Training iteration: " + it) if (init) ClassifierUtils.InitializeClassifiers(node, cls: _*) @@ -43,19 +43,25 @@ object JointTrainSparseNetwork { if (idx % 5000 == 0) logger.info(s"Training: $idx examples inferred.") - cls.foreach { - case classifier: ConstrainedClassifier[_, HEAD] => - val typedClassifier = classifier.asInstanceOf[ConstrainedClassifier[_, HEAD]] - val oracle = typedClassifier.onClassifier.getLabeler + if (lossAugmented) + cls.foreach { cls_i => + cls_i.onClassifier.classifier.setLossFlag() + cls_i.onClassifier.classifier.setCandidates(cls_i.getCandidates(h).size * cls.size) + } - typedClassifier.getCandidates(h) foreach { + cls.foreach { + currentClassifier: ConstrainedClassifier[_, HEAD] => + assert(currentClassifier.onClassifier.classifier.getClass.getName.contains("SparseNetworkLearner"), "The classifier should be of type SparseNetworkLearner!") + val oracle = currentClassifier.onClassifier.getLabeler + val baseClassifier = currentClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner] + currentClassifier.getCandidates(h) foreach { candidate => { def trainOnce() = { - val result = typedClassifier.classifier.discreteValue(candidate) + + val result = currentClassifier.classifier.discreteValue(candidate) val trueLabel = oracle.discreteValue(candidate) - val ilearner = typedClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner] - val lLexicon = typedClassifier.onClassifier.getLabelLexicon + val lLexicon = currentClassifier.onClassifier.getLabelLexicon var LTU_actual: Int = 0 var LTU_predicted: Int = 0 for (i <- 0 until lLexicon.size()) { @@ -69,26 +75,25 @@ object JointTrainSparseNetwork { // and the LTU of the predicted class should be demoted. if (!result.equals(trueLabel)) //equals("true") && trueLabel.equals("false") ) { - val a = typedClassifier.onClassifier.getExampleArray(candidate) + val a = currentClassifier.onClassifier.getExampleArray(candidate) val a0 = a(0).asInstanceOf[Array[Int]] //exampleFeatures val a1 = a(1).asInstanceOf[Array[Double]] // exampleValues val exampleLabels = a(2).asInstanceOf[Array[Int]] val label = exampleLabels(0) - var N = ilearner.getNetwork.size + val N = baseClassifier.getNetwork.size - if (label >= N || ilearner.getNetwork.get(label) == null) { - val conjugateLabels = ilearner.isUsingConjunctiveLabels | ilearner.getLabelLexicon.lookupKey(label).isConjunctive - ilearner.setConjunctiveLabels(conjugateLabels) + if (label >= N || baseClassifier.getNetwork.get(label) == null) { + val conjugateLabels = baseClassifier.isUsingConjunctiveLabels | baseClassifier.getLabelLexicon.lookupKey(label).isConjunctive + baseClassifier.setConjunctiveLabels(conjugateLabels) - val ltu: LinearThresholdUnit = ilearner.getBaseLTU - ltu.initialize(ilearner.getNumExamples, ilearner.getNumFeatures) - ilearner.getNetwork.set(label, ltu) - N = label + 1 + val ltu: LinearThresholdUnit = baseClassifier.getBaseLTU.clone().asInstanceOf[LinearThresholdUnit] + ltu.initialize(baseClassifier.getNumExamples, baseClassifier.getNumFeatures) + baseClassifier.getNetwork.set(label, ltu) } // test push - val ltu_actual = ilearner.getLTU(LTU_actual).asInstanceOf[LinearThresholdUnit] - val ltu_predicted = ilearner.getLTU(LTU_predicted).asInstanceOf[LinearThresholdUnit] + val ltu_actual = baseClassifier.getLTU(LTU_actual).asInstanceOf[LinearThresholdUnit] + val ltu_predicted = baseClassifier.getLTU(LTU_predicted).asInstanceOf[LinearThresholdUnit] if (ltu_actual != null) ltu_actual.promote(a0, a1, 0.1) @@ -100,8 +105,13 @@ object JointTrainSparseNetwork { trainOnce() } } + } } + if (lossAugmented) + cls.foreach { cls_i => + cls_i.onClassifier.classifier.unsetLossFlag() + } } train(node, cls, it - 1, false) } diff --git a/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrain.scala b/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparsePerceptron.scala similarity index 99% rename from saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrain.scala rename to saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparsePerceptron.scala index 2f046380..4b70d89e 100644 --- a/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrain.scala +++ b/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparsePerceptron.scala @@ -14,7 +14,7 @@ import scala.reflect.ClassTag /** Created by parisakordjamshidi on 29/01/15. */ -object JointTrain { +object JointTrainSparsePerceptron { def testClassifiers(cls: Classifier, oracle: Classifier, ds: List[AnyRef]): Unit = { val results = ds.map({ diff --git a/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/infer/InitSparseNetwork.scala b/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/infer/InitSparseNetwork.scala index a9df6c4c..d1f738fe 100644 --- a/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/infer/InitSparseNetwork.scala +++ b/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/infer/InitSparseNetwork.scala @@ -18,7 +18,10 @@ object InitSparseNetwork { //this means we are not reading any model into the SparseNetworks // but we forget all the models and go over the data to build the right // size for the lexicon and the right number of the ltu s + cClassifier.onClassifier.classifier.forget() + assert(cClassifier.onClassifier.classifier.getClass.getName.contains("SparseNetworkLearner"), "The classifier should be of type SparseNetworkLearner!") + val iLearner = cClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner] allHeads.foreach { head => @@ -33,7 +36,7 @@ object InitSparseNetwork { if (label >= N || iLearner.getNetwork.get(label) == null) { val isConjunctiveLabels = iLearner.isUsingConjunctiveLabels | iLearner.getLabelLexicon.lookupKey(label).isConjunctive iLearner.setConjunctiveLabels(isConjunctiveLabels) - val ltu: LinearThresholdUnit = iLearner.getBaseLTU + val ltu: LinearThresholdUnit = iLearner.getBaseLTU.clone().asInstanceOf[LinearThresholdUnit] ltu.initialize(iLearner.getNumExamples, iLearner.getNumFeatures) iLearner.getNetwork.set(label, ltu) } diff --git a/saul-core/src/test/scala/edu/illinois/cs/cogcomp/saul/classifier/JoinTrainingTests/IntializeSparseNetwork.scala b/saul-core/src/test/scala/edu/illinois/cs/cogcomp/saul/classifier/JoinTrainingTests/IntializeSparseNetwork.scala index 605d03ba..cc284619 100644 --- a/saul-core/src/test/scala/edu/illinois/cs/cogcomp/saul/classifier/JoinTrainingTests/IntializeSparseNetwork.scala +++ b/saul-core/src/test/scala/edu/illinois/cs/cogcomp/saul/classifier/JoinTrainingTests/IntializeSparseNetwork.scala @@ -1,3 +1,9 @@ +/** This software is released under the University of Illinois/Research and Academic Use License. See + * the LICENSE file in the root folder for details. Copyright (c) 2016 + * + * Developed by: The Cognitive Computations Group, University of Illinois at Urbana-Champaign + * http://cogcomp.cs.illinois.edu/ + */ package edu.illinois.cs.cogcomp.saul.classifier.JoinTrainingTests import edu.illinois.cs.cogcomp.infer.ilp.OJalgoHook @@ -71,7 +77,7 @@ class InitializeSparseNetwork extends FlatSpec with Matchers { val wv1After = clNet1.getNetwork.get(0).asInstanceOf[LinearThresholdUnit].getWeightVector val wv2After = clNet2.getNetwork.get(0).asInstanceOf[LinearThresholdUnit].getWeightVector - wv1After.size() should be(5) + wv1After.size() should be(6) wv2After.size() should be(12) } diff --git a/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeReader.java b/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeReader.java new file mode 100644 index 00000000..456f5e6f --- /dev/null +++ b/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeReader.java @@ -0,0 +1,32 @@ +/** This software is released under the University of Illinois/Research and Academic Use License. See + * the LICENSE file in the root folder for details. Copyright (c) 2016 + * + * Developed by: The Cognitive Computations Group, University of Illinois at Urbana-Champaign + * http://cogcomp.cs.illinois.edu/ + */ +package edu.illinois.cs.cogcomp.saulexamples.Badge; + +import java.io.BufferedReader; +import java.io.FileInputStream; +import java.io.InputStreamReader; +import java.util.ArrayList; +import java.util.List; + +public class BadgeReader { + public List badges; + + public BadgeReader(String dataFile) { + badges = new ArrayList(); + + try { + BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(dataFile))); + + String str; + while ((str = br.readLine()) != null) { + badges.add(str); + } + + br.close(); + }catch (Exception e) {} + } +} \ No newline at end of file diff --git a/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLConfigurator.java b/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLConfigurator.java index f006e1af..a1ad3b03 100644 --- a/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLConfigurator.java +++ b/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLConfigurator.java @@ -21,7 +21,6 @@ public class SRLConfigurator extends Configurator { public static final Property TREEBANK_HOME = new Property("treebankHome", "../saul-examples/src/test/resources/SRLToy/treebank"); public static final Property PROPBANK_HOME = new Property("propbankHome","../saul-examples/src/test/resources/SRLToy/propbank"); - public static final Property TEST_SECTION = new Property("testSection","00"); public static final Property MODELS_DIR = new Property("modelsDir", "../models"); @@ -30,7 +29,7 @@ public class SRLConfigurator extends Configurator { // The running mode of the program. Can be "true" for only testing, or "false" for training public static final Property RUN_MODE = new Property("runMode", Configurator.TRUE); - // The training mode for the examples. Can be "pipeline", "joint", or "other" + // The training mode for the examples. Can be "pipeline", "joint", "jointLoss" or "other" public static final Property TRAINING_MODE = new Property("trainingMode", "joint"); /*********** SRL PROPERTIES ***********/ diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeClassifiers.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeClassifiers.scala new file mode 100644 index 00000000..d9febc2b --- /dev/null +++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeClassifiers.scala @@ -0,0 +1,48 @@ +/** This software is released under the University of Illinois/Research and Academic Use License. See + * the LICENSE file in the root folder for details. Copyright (c) 2016 + * + * Developed by: The Cognitive Computations Group, University of Illinois at Urbana-Champaign + * http://cogcomp.cs.illinois.edu/ + */ +package edu.illinois.cs.cogcomp.saulexamples.Badge + +import edu.illinois.cs.cogcomp.lbjava.learn.{ SparseNetworkLearner, SparsePerceptron } +import edu.illinois.cs.cogcomp.saul.classifier.Learnable +import edu.illinois.cs.cogcomp.saulexamples.Badge.BadgeDataModel._ +/** Created by Parisa on 9/13/16. + */ + +object BadgeClassifiers { + + /*a binary classifier that predicts the badge label*/ + object BadgeClassifier extends Learnable[String](badge) { + def label = BadgeLabel + override lazy val classifier = new SparsePerceptron() + override def feature = using(BadgeFeature1) + } + /*a binary classifier that predicts a lable that is exactly the opposite of the badge label of the BadgeClassifier*/ + object BadgeOppositClassifier extends Learnable[String](badge) { + def label = BadgeOppositLabel + override lazy val classifier = new SparsePerceptron() + override def feature = using(BadgeFeature1) + } + /*This is a multi-class classifier version of the above binary BadgeClassifier, +it uses SparseNetworks instead of SparsePerceptrons*/ + object BadgeClassifierMulti extends Learnable[String](badge) { + def label = BadgeLabel + override lazy val classifier = new SparseNetworkLearner() + override def feature = using(BadgeFeature1) + } + /*This is the opposite multi-class classifier of the BadgeClassifierMulti */ + object BadgeOppositClassifierMulti extends Learnable[String](badge) { + def label = BadgeOppositLabel + override lazy val classifier = new SparseNetworkLearner() + override def feature = using(BadgeFeature1) + } + /* */ + object BadgeOppositPipeline extends Learnable[String](badge) { + def label = BadgeOppositLabel + override lazy val classifier = new SparsePerceptron() + override def feature = using(BadgePrediction) + } +} \ No newline at end of file diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeConstrainedClassifiers.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeConstrainedClassifiers.scala new file mode 100644 index 00000000..b5587d9d --- /dev/null +++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeConstrainedClassifiers.scala @@ -0,0 +1,47 @@ +/** This software is released under the University of Illinois/Research and Academic Use License. See + * the LICENSE file in the root folder for details. Copyright (c) 2016 + * + * Developed by: The Cognitive Computations Group, University of Illinois at Urbana-Champaign + * http://cogcomp.cs.illinois.edu/ + */ +package edu.illinois.cs.cogcomp.saulexamples.Badge + +import edu.illinois.cs.cogcomp.infer.ilp.OJalgoHook +import edu.illinois.cs.cogcomp.saul.classifier.ConstrainedClassifier +import edu.illinois.cs.cogcomp.saul.constraint.ConstraintTypeConversion._ +import edu.illinois.cs.cogcomp.saulexamples.Badge.BadgeClassifiers.{ BadgeOppositClassifierMulti, BadgeClassifierMulti, BadgeClassifier, BadgeOppositClassifier } + +/** Created by Parisa on 11/1/16. + */ +object BadgeConstrainedClassifiers { + + val binaryConstraint = ConstrainedClassifier.constraint[String] { + x: String => + (BadgeClassifier on x is "negative") ==> (BadgeOppositClassifier on x is "positive") + } + + val binaryConstraintOverMultiClassifiers = ConstrainedClassifier.constraint[String] { + x: String => + (BadgeClassifierMulti on x is "negative") ==> (BadgeOppositClassifierMulti on x is "positive") + } + object badgeConstrainedClassifier extends ConstrainedClassifier[String, String](BadgeClassifier) { + def subjectTo = binaryConstraint + override val solver = new OJalgoHook + } + + object oppositBadgeConstrainedClassifier extends ConstrainedClassifier[String, String](BadgeOppositClassifier) { + def subjectTo = binaryConstraint + override val solver = new OJalgoHook + } + + object badgeConstrainedClassifierMulti extends ConstrainedClassifier[String, String](BadgeClassifierMulti) { + def subjectTo = binaryConstraintOverMultiClassifiers + override val solver = new OJalgoHook + } + + object oppositBadgeConstrainedClassifierMulti extends ConstrainedClassifier[String, String](BadgeOppositClassifierMulti) { + def subjectTo = binaryConstraintOverMultiClassifiers + override val solver = new OJalgoHook + } + +} diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeDataModel.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeDataModel.scala new file mode 100644 index 00000000..3d1e9366 --- /dev/null +++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeDataModel.scala @@ -0,0 +1,49 @@ +/** This software is released under the University of Illinois/Research and Academic Use License. See + * the LICENSE file in the root folder for details. Copyright (c) 2016 + * + * Developed by: The Cognitive Computations Group, University of Illinois at Urbana-Champaign + * http://cogcomp.cs.illinois.edu/ + */ +package edu.illinois.cs.cogcomp.saulexamples.Badge + +import edu.illinois.cs.cogcomp.saul.datamodel.DataModel +import edu.illinois.cs.cogcomp.saulexamples.Badge.BadgeClassifiers.BadgeClassifier + +/** Created by Parisa on 9/13/16. + */ +object BadgeDataModel extends DataModel { + + val badge = node[String] + + val BadgeFeature1 = property(badge) { + x: String => + { + val tokens = x.split(" ") + tokens(1).charAt(1).toString + } + } + + val BadgeLabel = property(badge)("true", "false") { + x: String => + { + val tokens = x.split(" ") + if (tokens(0).equals("+")) + "true" + else + "false" + } + } + + val BadgeOppositLabel = property(badge)("true", "false") { + x: String => + { + val tokens = x.split(" ") + if (tokens(0).equals("+")) + "false" + else + "true" + } + } + + val BadgePrediction = property(badge)("true", "false") { x: String => BadgeClassifier(x) } +} diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgesApp.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgesApp.scala new file mode 100644 index 00000000..87c93b72 --- /dev/null +++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgesApp.scala @@ -0,0 +1,82 @@ +/** This software is released under the University of Illinois/Research and Academic Use License. See + * the LICENSE file in the root folder for details. Copyright (c) 2016 + * + * Developed by: The Cognitive Computations Group, University of Illinois at Urbana-Champaign + * http://cogcomp.cs.illinois.edu/ + */ +package edu.illinois.cs.cogcomp.saulexamples.Badge + +/** Created by Parisa on 9/13/16. + */ + +import edu.illinois.cs.cogcomp.saul.classifier.{ JointTrainSparseNetwork, JointTrainSparsePerceptron } +import edu.illinois.cs.cogcomp.saulexamples.Badge.BadgeClassifiers._ +import edu.illinois.cs.cogcomp.saulexamples.Badge.BadgeConstrainedClassifiers.{ badgeConstrainedClassifier, badgeConstrainedClassifierMulti, oppositBadgeConstrainedClassifier, oppositBadgeConstrainedClassifierMulti } +import edu.illinois.cs.cogcomp.saulexamples.Badge.BadgeDataModel._ + +import scala.collection.JavaConversions._ +object BadgesApp { + + val allNamesTrain = new BadgeReader("data/badges/badges.train").badges + val allNamesTest = new BadgeReader("data/badges/badges.test").badges + + badge.populate(allNamesTrain) + badge.populate(allNamesTest, false) + + val cls = List(badgeConstrainedClassifierMulti, oppositBadgeConstrainedClassifierMulti) + + object BadgeExperimentType extends Enumeration { + val JoinTrainSparsePerceptron, JointTrainSparseNetwork, JointTrainSparseNetworkLossAugmented, Pipeline = Value + } + + def main(args: Array[String]): Unit = { + + /** Choose the experiment you're interested in by changing the following line */ + val testType = BadgeExperimentType.Pipeline + + testType match { + case BadgeExperimentType.JoinTrainSparsePerceptron => JoinTrainSparsePerceptron() + case BadgeExperimentType.JointTrainSparseNetwork => JoinTrainSparseNetwork() + case BadgeExperimentType.JointTrainSparseNetworkLossAugmented => LossAugmentedJoinTrainSparseNetwork() + case BadgeExperimentType.Pipeline => Pipeline() + } + } + + /*Test the join training with SparsePerceptron*/ + def JoinTrainSparsePerceptron(): Unit = { + BadgeClassifier.test() + BadgeOppositClassifier.test() + JointTrainSparsePerceptron.train(BadgeDataModel.badge, List(badgeConstrainedClassifier, oppositBadgeConstrainedClassifier), 5) + oppositBadgeConstrainedClassifier.test() + badgeConstrainedClassifier.test() + BadgeClassifier.test() + } + + /*Test the joinTraining with SparseNetwork*/ + def JoinTrainSparseNetwork(): Unit = { + + JointTrainSparseNetwork.train(badge, cls, 5, init = true) + + badgeConstrainedClassifierMulti.test() + oppositBadgeConstrainedClassifierMulti.test() + } + + /*Test the joinTraining with SparseNetwork and doing loss augmented inference*/ + def LossAugmentedJoinTrainSparseNetwork(): Unit = { + + JointTrainSparseNetwork.train(badge, cls, 5, init = true, lossAugmented = true) + + badgeConstrainedClassifierMulti.test() + oppositBadgeConstrainedClassifierMulti.test() + } + + /* This model trains the BadgeClassifier and then it takes the prediction of the BadgeClassifier as the only input + feature and trains a pipeline function to predict the opposite label*/ + def Pipeline(): Unit = { + BadgeClassifier.learn(5) + BadgeClassifier.test() + BadgeOppositPipeline.learn(5) + BadgeOppositPipeline.test() + } + +} \ No newline at end of file diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/README.md b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/README.md new file mode 100644 index 00000000..71a0d5be --- /dev/null +++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/README.md @@ -0,0 +1,9 @@ + +This example is a simple model, that receives names of people and assigns a label either positive or negative to them. +It uses the second character of the first names as an input feature. This is a gold feature that can distinguish 100% between +the positive and negative class. +We show the usage of binary classifiers as well as multi-class classifiers in [here](BadgeClassifiers.scala). +In this file you could see we define two type of the classifiers that they take opposite labels. The goal is to simply show +how a simple constraint can impose the predictions of these two classifiers to be opposite. +The constrained versions of all these classifiers can be found [here](BadgeConstraintClassifiers.scala). +Using a simple constraint we train various joint models that train the two opposite classifiers jointly in [here](BadgesApp.scala). \ No newline at end of file diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/DrugResponse/myApp.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/DrugResponse/myApp.scala index 5c5ca870..5b93d1c8 100644 --- a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/DrugResponse/myApp.scala +++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/DrugResponse/myApp.scala @@ -75,6 +75,7 @@ object myApp extends Logging { //dResponseClassifier.testContinuos(patient_drug_data) //DrugResponseRegressor.learn(1) + //DrugResponseRegressor.testContinuos(patientDrug.getTrainingInstances) logger.info("finished!") } diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/PopulateSRLDataModel.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/PopulateSRLDataModel.scala index 6ed49e04..9796653b 100644 --- a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/PopulateSRLDataModel.scala +++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/PopulateSRLDataModel.scala @@ -19,10 +19,10 @@ import edu.illinois.cs.cogcomp.nlp.common.PipelineConfigurator._ import edu.illinois.cs.cogcomp.nlp.utilities.ParseUtils import edu.illinois.cs.cogcomp.saul.util.Logging import edu.illinois.cs.cogcomp.saulexamples.data.{ SRLDataReader, SRLFrameManager } -import edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.SRLSensors._ import edu.illinois.cs.cogcomp.saulexamples.nlp.CommonSensors._ +import edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.SRLSensors._ import edu.illinois.cs.cogcomp.saulexamples.nlp.TextAnnotationFactory - +import edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.SRLscalaConfigurator._ import scala.collection.JavaConversions._ /** Created by Parisa on 1/17/16. @@ -34,7 +34,7 @@ object PopulateSRLDataModel extends Logging { useGoldArgBoundaries: Boolean = false, rm: ResourceManager = new SRLConfigurator().getDefaultConfig ): SRLMultiGraphDataModel = { - val frameManager: SRLFrameManager = new SRLFrameManager(rm.getString(SRLConfigurator.PROPBANK_HOME.key)) + val frameManager: SRLFrameManager = new SRLFrameManager(PROPBANK_HOME) val useCurator = rm.getBoolean(SRLConfigurator.USE_CURATOR) val parseViewName = rm.getString(SRLConfigurator.SRL_PARSE_VIEW) val graphs = new SRLMultiGraphDataModel(parseViewName, frameManager) @@ -86,16 +86,11 @@ object PopulateSRLDataModel extends Logging { logger.debug(s"Number of $readerType data arguments: $numArguments") } - val trainingFromSection = 2 - val trainingToSection = 2 var gr: SRLMultiGraphDataModel = null if (!testOnly) { - logger.info(s"Reading training data from sections $trainingFromSection to $trainingToSection") - val trainReader = new SRLDataReader( - rm.getString(SRLConfigurator.TREEBANK_HOME.key), - rm.getString(SRLConfigurator.PROPBANK_HOME.key), - trainingFromSection, trainingToSection - ) + logger.info(s"Reading training data from sections $TRAIN_SECTION_S to $TRAIN_SECTION_E") + val trainReader = new SRLDataReader(TREEBANK_HOME, PROPBANK_HOME, + TRAIN_SECTION_S, TRAIN_SECTION_E) trainReader.readData() logger.info(s"Annotating ${trainReader.textAnnotations.size} training sentences") val filteredTa = addViewAndFilter(trainReader.textAnnotations.toList) @@ -124,13 +119,9 @@ object PopulateSRLDataModel extends Logging { if (graphs.sentences().size % 1000 == 0) logger.info("loaded graphs in memory:" + graphs.sentences().size) } } - val testSection = rm.getInt(SRLConfigurator.TEST_SECTION) - val testReader = new SRLDataReader( - rm.getString(SRLConfigurator.TREEBANK_HOME.key), - rm.getString(SRLConfigurator.PROPBANK_HOME.key), - testSection, testSection - ) - logger.info(s"Reading test data from section $testSection") + + val testReader = new SRLDataReader(TREEBANK_HOME, PROPBANK_HOME, TEST_SECTION, TEST_SECTION) + logger.info(s"Reading test data from section $TEST_SECTION") testReader.readData() logger.info(s"Annotating ${testReader.textAnnotations.size} test sentences") diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/README.md b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/README.md index d27abf74..43fafeaf 100644 --- a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/README.md +++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/README.md @@ -5,13 +5,13 @@ This task is to annotate natural language sentences with semantic roles. To run the main app with default properties: ``` -sbt "project saulExamples" "run-main edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.srlApp" +sbt "project saulExamples" "run-main edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.RunningApps" ``` -To use a custom configuration file (containing the property keys of `ExamplesConfigurator`): +To use a custom configuration file (containing the property keys of `ExamplesConfigurator`), also extending memory to 4G: ``` - sbt "project saulExamples" "run-main edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.srlApp config/saul-srl.properties" + sbt -mem 4000 "project saulExamples" "run-main edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.RunningApps config/saul-srl.properties" ``` ## Example @@ -29,6 +29,7 @@ Similar to other applications in Saul, here also we have a datamodel in file `SR classifier definitions in file `SRLClassifires`, a bunch of constraints to be used by global models during either training or test in file `SRLConstraints`, a bunch of constrained classifiers in file `SRLConstrainedClassifiers` and the running configurations that are all placed in one file called `SRLApp`. + For using the reader and populating data there is a program in file `PopulateSRLDataModel`. In contrast to other Saul applications this data model has been defined as a class instead of as an object. The reason is the efficiency of the population of the data model, we skip the details of this implementation choice. @@ -40,7 +41,7 @@ the properties which use the FrameNet frames can receive it as a parameter. We refer the reader to see an example of defining such parametrized properties along with `Learnable` classes and contrast it with the `Learnable` objects in `DrugResponse` example of `KnowEng` data model. -There are various machine learning configurations to solve this including pipelines, learning only models (LO), +There are various machine learning configurations to solve SRL including pipelines, learning only models (LO), learning plus inference models (L+I) and Joint Learning models (IBT). The test units and SRLApp are runnable on a sample toy dataset located in `resources/SRLToy` folder. Accessing the full dataset needs proper licenses. If you have access to the PropBank data, you could @@ -61,13 +62,12 @@ In the following lines `Pred.` stands for "Predicate" and `Cand.` stands for "C | Predicate | Argument | Model | Name | | -----------| -------------------- | ------------------------ | ---- | -| Gold Pred. | Gold Boundaries | Argument Type Classifier | aTr | +| Gold Pred. | Gold Boundaries | Argument Type Classifier | aTr,jTr,lTr | | Gold Pred. | XuPalmer Candidates | Argument identifier | bTr | | Gold Pred. | XuPalmer Candidates | Argument Type Classifier | cTr | | Predicted Cand. | NA | Predicate Classifier | dTr | | Predicted Cand. | XuPalmer Candidates | Argument identifier | eTr | | Predicted Cand. | XuPalmer Candidates | Argument Type Classifier | fTr | -| Gold Pred. | Gold Boundries | Argument Type Classifier | jTr | | Gold Pred. | Argument Identifier | Argument Type Classifier | pTr | @@ -577,6 +577,122 @@ In the following lines `Pred.` stands for "Predicate" and `Cand.` stands for "C * Add constraints gradually and test. #### Third phase: training joint models + - [x] **[lTr]** Train aTr jointly using hamming loss; test using constraints + + + Average evaluation time: 0.04143100828729282 seconds + + Label Precision Recall F1 LCount PCount + ---------------------------------------------- + A0 96.585 91.699 94.079 3578 3397 + A1 90.352 93.517 91.907 4967 5141 + A2 78.791 64.711 71.060 1108 910 + A3 61.176 60.465 60.819 172 170 + A4 28.525 85.294 42.752 102 305 + A5 57.143 80.000 66.667 5 7 + AA 0.000 0.000 0.000 0 2 + AM-ADV 75.987 45.652 57.037 506 304 + AM-CAU 73.333 43.421 54.545 76 45 + AM-DIR 42.308 51.765 46.561 85 104 + AM-DIS 75.833 85.313 80.294 320 360 + AM-EXT 50.000 50.000 50.000 32 32 + AM-LOC 59.244 38.525 46.689 366 238 + AM-MNR 34.126 70.115 45.908 348 715 + AM-MOD 97.513 99.637 98.564 551 563 + AM-NEG 97.854 99.130 98.488 230 233 + AM-PNC 57.333 37.391 45.263 115 75 + AM-PRD 0.000 0.000 0.000 5 11 + AM-REC 0.000 0.000 0.000 2 0 + AM-TMP 75.690 76.097 75.893 1117 1123 + C-A0 83.333 27.778 41.667 18 6 + C-A1 68.465 72.052 70.213 229 241 + C-A2 14.286 40.000 21.053 5 14 + C-A3 0.000 0.000 0.000 3 11 + C-A4 0.000 0.000 0.000 0 7 + C-A5 0.000 0.000 0.000 0 1 + C-AM-ADV 0.000 0.000 0.000 0 1 + C-AM-DIR 0.000 0.000 0.000 0 3 + C-AM-EXT 0.000 0.000 0.000 0 1 + C-AM-LOC 0.000 0.000 0.000 0 1 + C-AM-MNR 0.000 0.000 0.000 0 7 + C-AM-NEG 0.000 0.000 0.000 0 3 + C-AM-PNC 0.000 0.000 0.000 0 1 + C-V 0.000 0.000 0.000 141 31 + R-A0 85.388 88.208 86.775 212 219 + R-A1 68.919 77.863 73.118 131 148 + R-A2 62.500 38.462 47.619 13 8 + R-A3 0.000 0.000 0.000 1 0 + R-A4 0.000 0.000 0.000 1 7 + R-AM-ADV 0.000 0.000 0.000 2 1 + R-AM-CAU 0.000 0.000 0.000 1 3 + R-AM-EXT 0.000 0.000 0.000 1 3 + R-AM-LOC 69.231 56.250 62.069 16 13 + R-AM-MNR 0.000 0.000 0.000 2 5 + R-AM-PNC 0.000 0.000 0.000 0 3 + ---------------------------------------------- + R-AM-TMP 16.667 5.556 8.333 18 6 + ---------------------------------------------- + Overall 82.644 82.644 82.644 14479 14479 + Accuracy 82.644 - - - 14479 + + Total time: 244961 s, completed Dec 2, 2016 11:08:22 AM + + - [x] **[jTr]** Train aTr jointly without considering the loss explicitly; test using constraints + + Label Precision Recall F1 LCount PCount + ---------------------------------------------- + A0 96.425 93.488 94.934 3578 3469 + A1 91.531 94.000 92.749 4967 5101 + A2 67.846 76.173 71.769 1108 1244 + A3 78.652 40.698 53.640 172 89 + A4 83.951 66.667 74.317 102 81 + A5 57.143 80.000 66.667 5 7 + AA 0.000 0.000 0.000 0 2 + AM-ADV 68.623 60.079 64.067 506 443 + AM-CAU 80.952 44.737 57.627 76 42 + AM-DIR 43.220 60.000 50.246 85 118 + AM-DIS 81.250 81.250 81.250 320 320 + AM-EXT 33.333 68.750 44.898 32 66 + AM-LOC 83.969 30.055 44.266 366 131 + AM-MNR 51.940 50.000 50.952 348 335 + AM-MOD 97.340 99.637 98.475 551 564 + AM-NEG 96.624 99.565 98.073 230 237 + AM-PNC 22.118 81.739 34.815 115 425 + AM-PRD 16.667 20.000 18.182 5 6 + AM-REC 0.000 0.000 0.000 2 6 + AM-TMP 84.706 70.904 77.193 1117 935 + C-A0 21.154 61.111 31.429 18 52 + C-A1 60.067 78.166 67.932 229 298 + C-A2 0.000 0.000 0.000 5 11 + C-A3 0.000 0.000 0.000 3 4 + C-A4 0.000 0.000 0.000 0 3 + C-A5 0.000 0.000 0.000 0 1 + C-AM-DIR 0.000 0.000 0.000 0 3 + C-AM-EXT 0.000 0.000 0.000 0 3 + C-AM-LOC 0.000 0.000 0.000 0 1 + C-AM-MNR 0.000 0.000 0.000 0 2 + C-AM-NEG 0.000 0.000 0.000 0 8 + C-AM-PNC 0.000 0.000 0.000 0 6 + C-AM-TMP 0.000 0.000 0.000 0 1 + C-V 0.000 0.000 0.000 141 31 + R-A0 82.427 92.925 87.361 212 239 + R-A1 79.612 62.595 70.085 131 103 + R-A2 21.875 53.846 31.111 13 32 + R-A3 0.000 0.000 0.000 1 0 + R-A4 0.000 0.000 0.000 1 1 + R-AM-ADV 0.000 0.000 0.000 2 1 + R-AM-CAU 0.000 0.000 0.000 1 2 + R-AM-EXT 0.000 0.000 0.000 1 7 + R-AM-LOC 78.571 68.750 73.333 16 14 + R-AM-MNR 0.000 0.000 0.000 2 1 + R-AM-PNC 0.000 0.000 0.000 0 6 + ---------------------------------------------- + R-AM-TMP 28.571 44.444 34.783 18 28 + ---------------------------------------------- + Overall 83.673 83.673 83.673 14479 14479 + Accuracy 83.673 - - - 14479 + Total time: 214836 s, completed Nov 20, 2016 9:15:22 AM + - [ ] **[aTrJ]** Train **dTr, eTr, fTr** jointly * Add constraints gradually and train various models considering subsets of constraints @@ -586,11 +702,7 @@ In the following lines `Pred.` stands for "Predicate" and `Cand.` stands for "C - [ ] **[aTsJ]** Test the **cTs** of the second phase for joint models. -The defaul configuration when running the sprlApp will run only the test for pretrained cTr model while it uses srl global constraints during prediction. -You can run it from command line by: - -```scala - -sbt -mem 4000 "project saulExamples" "run-main edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.SRLApps" - -``` +The training results of independent models are after 100 iterations of training, however, since joinnTraining is computationally much more +complex, we stopped the training after 30 iterations. +The overall results were similar to running independent models after 30 iterations, no improvment observed. +However, there was a per class variation in results and the results for some of the lables improved. \ No newline at end of file diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLApps.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLApps.scala index b6c81203..9970b93b 100644 --- a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLApps.scala +++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLApps.scala @@ -8,56 +8,84 @@ package edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling import java.io.File -import edu.illinois.cs.cogcomp.core.utilities.configuration.ResourceManager +import edu.illinois.cs.cogcomp.core.datastructures.ViewNames import edu.illinois.cs.cogcomp.saul.classifier.{ ClassifierUtils, JointTrainSparseNetwork } import edu.illinois.cs.cogcomp.saul.util.Logging import edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.SRLClassifiers._ import edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.SRLConstrainedClassifiers.argTypeConstraintClassifier -object SRLApps extends Logging { - import SRLConfigurator._ - - val properties: ResourceManager = { - // Load the default properties if the user hasn't entered a file as an argument - //if (args.length == 0) { - logger.info("Loading default configuration parameters") - new SRLConfigurator().getDefaultConfig - //} else { - // logger.info("Loading parameters from {}", args(0)) - //new SRLConfigurator().getConfig(new ResourceManager(args(0))) - // } - } - val modelDir = properties.getString(MODELS_DIR) + - File.separator + properties.getString(SRLConfigurator.SRL_MODEL_DIR) + File.separator - val srlPredictionsFile = properties.getString(SRLConfigurator.SRL_OUTPUT_FILE) - val runningMode = properties.getBoolean(SRLConfigurator.RUN_MODE) - val trainingMode = properties.getString(SRLConfigurator.TRAINING_MODE) +object SRLscalaConfigurator { + + val TREEBANK_HOME = "../saul-examples/src/test/resources/SRLToy/treebank" + val PROPBANK_HOME = "../saul-examples/src/test/resources/SRLToy/propbank" + + val TEST_SECTION = 0 + val TRAIN_SECTION_S = 2 + val TRAIN_SECTION_E = 21 + + val MODELS_DIR = "../models" + val USE_CURATOR = false + + // The running mode of the program. Can be "true" for only testing, or "false" for training + val TEST_MODE: Boolean = true + + // The training mode for the examples. Can be "pipeline", "joint", "jointLoss" or "other" + val TRAINING_MODE = "joint" + + /*********** SRL PROPERTIES ***********/ + // The (sub)directory to store and retrieve the trained SRL models (to be used with MODELS_DIR) + val SRL_MODEL_DIR = "srl" + val SRL_JAR_MODEL_PATH = "models" + + // This is used to determine the parse view in SRL experiments (can be ViewNames.GOLD or ViewNames.STANFORD) + // For replicating the published experiments this needs to be GOLD + val SRL_PARSE_VIEW = ViewNames.PARSE_GOLD + + // A file to store the predictions of the SRL classifier (for argument types only) + val SRL_OUTPUT_FILE = "srl-predictions.txt" + + // Whether to use gold predicates (if FALSE, predicateClassifier will be used instead) + val SRL_GOLD_PREDICATES = true + + // Whether to use gold argument boundaries (if FALSE, argumentXuIdentifierGivenApredicate will be used instead) + val SRL_GOLD_ARG_BOUNDARIES = true - // Training parameters - val trainPredicates = properties.getBoolean(SRLConfigurator.SRL_TRAIN_PREDICATES) - val trainArgIdentifier = properties.getBoolean(SRLConfigurator.SRL_TRAIN_ARG_IDENTIFIERS) - val trainArgType = properties.getBoolean(SRLConfigurator.SRL_TRAIN_ARG_TYPE) + /*Testing parameters*/ - // Testing parameters - val testWithConstraints = properties.getBoolean(SRLConfigurator.SRL_TEST_CONSTRAINTS) - val testWithPipeline = properties.getBoolean(SRLConfigurator.SRL_TEST_PIPELINE) + // Should we use the pipeline during testing + val SRL_TEST_PIPELINE = false + // Should we use constraints during testing + val SRL_TEST_CONSTRAINTS = false - val useGoldPredicate = properties.getBoolean(SRLConfigurator.SRL_GOLD_PREDICATES) - val useGoldBoundaries = properties.getBoolean(SRLConfigurator.SRL_GOLD_ARG_BOUNDARIES) + /*Training parameters*/ - val modelJars = properties.getString(SRLConfigurator.SRL_JAR_MODEL_PATH) + // Should we train a predicate classifier given predicate candidates + val SRL_TRAIN_PREDICATES = false + // Should we train an argument identifier given the XuPalmer argument candidates + val SRL_TRAIN_ARG_IDENTIFIERS = false + // Should we train an argument type classifier + val SRL_TRAIN_ARG_TYPE = true + +} + +object SRLApps extends Logging { + + import SRLscalaConfigurator._ + + val modelDir = MODELS_DIR + File.separator + SRL_MODEL_DIR + File.separator val expName: String = { - if (trainingMode.equals("other")) - if (trainArgType && useGoldBoundaries && useGoldPredicate && trainingMode.equals("other")) "aTr" - else if (trainArgIdentifier && useGoldPredicate && useGoldPredicate) "bTr" - else if (trainArgType && useGoldPredicate && !useGoldBoundaries) "cTr" - else if (trainPredicates && useGoldPredicate) "dTr" - else if (trainArgIdentifier && !useGoldPredicate) "eTr" - else if (trainArgType && !useGoldPredicate) "fTr" + if (TRAINING_MODE.equals("other")) + if (SRL_TRAIN_ARG_TYPE && SRL_GOLD_ARG_BOUNDARIES && SRL_GOLD_PREDICATES && TRAINING_MODE.equals("other")) "aTr" + else if (SRL_TRAIN_ARG_IDENTIFIERS && SRL_GOLD_PREDICATES && SRL_GOLD_PREDICATES) "bTr" + else if (SRL_TRAIN_ARG_TYPE && SRL_GOLD_PREDICATES && !SRL_GOLD_ARG_BOUNDARIES) "cTr" + else if (SRL_TRAIN_PREDICATES && SRL_GOLD_PREDICATES) "dTr" + else if (SRL_TRAIN_ARG_IDENTIFIERS && !SRL_GOLD_PREDICATES) "eTr" + else if (SRL_TRAIN_ARG_TYPE && !SRL_GOLD_PREDICATES) "fTr" else "" - else if (trainingMode.equals("pipeline")) "pTr" - else if (trainingMode.equals("joint")) "jTr" + else if (TRAINING_MODE.equals("pipeline")) "pTr" + else if (TRAINING_MODE.equals("joint")) "jTr" + else if (TRAINING_MODE.equals("jointLoss")) "lTr" else "" } @@ -65,7 +93,7 @@ object SRLApps extends Logging { logger.info("population starts.") // Here, the data is loaded into the graph - val srlDataModelObject = PopulateSRLDataModel(testOnly = runningMode, useGoldPredicate, useGoldBoundaries) + val srlDataModelObject = PopulateSRLDataModel(testOnly = TEST_MODE, SRL_GOLD_PREDICATES, SRL_GOLD_ARG_BOUNDARIES) import srlDataModelObject._ @@ -79,13 +107,14 @@ object SRLApps extends Logging { object RunningApps extends App with Logging { import SRLApps._ import SRLApps.srlDataModelObject._ + import SRLscalaConfigurator._ // TRAINING - if (!runningMode) { + if (!TEST_MODE) { expName match { case "aTr" => argumentTypeLearner.modelDir = modelDir + expName - argumentTypeLearner.learn(100, relations.getTrainingInstances) + argumentTypeLearner.learn(30, relations.getTrainingInstances) argumentTypeLearner.test() argumentTypeLearner.save() @@ -146,17 +175,26 @@ object RunningApps extends App with Logging { case "jTr" => argumentTypeLearner.modelDir = modelDir + expName - val outputFile = modelDir + srlPredictionsFile + val outputFile = modelDir + SRL_OUTPUT_FILE logger.info("Global training... ") - JointTrainSparseNetwork(sentences, argTypeConstraintClassifier :: Nil, 100, true) + JointTrainSparseNetwork(sentences, argTypeConstraintClassifier :: Nil, 30, init = true) + argumentTypeLearner.save() + argTypeConstraintClassifier.test(relations.getTestingInstances, outputFile, 200, exclude = "candidate") + + case "lTr" => + argumentTypeLearner.modelDir = modelDir + expName + val outputFile = modelDir + SRL_OUTPUT_FILE + logger.info("Global training using loss augmented inference... ") + JointTrainSparseNetwork(sentences, argTypeConstraintClassifier :: Nil, 30, init = true, lossAugmented = true) argumentTypeLearner.save() argTypeConstraintClassifier.test(relations.getTestingInstances, outputFile, 200, exclude = "candidate") } + } // TESTING - if (runningMode) { - (testWithPipeline, testWithConstraints) match { + if (TEST_MODE) { + (SRL_TEST_PIPELINE, SRL_TEST_CONSTRAINTS) match { case (true, true) => ClassifierUtils.LoadClassifier(SRLConfigurator.SRL_JAR_MODEL_PATH.value + "/models_bTr/", argumentXuIdentifierGivenApredicate) diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SpatialRoleLabeling/SpRLDataModel.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SpatialRoleLabeling/SpRLDataModel.scala index 3167cccc..b84ce932 100644 --- a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SpatialRoleLabeling/SpRLDataModel.scala +++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SpatialRoleLabeling/SpRLDataModel.scala @@ -10,11 +10,10 @@ import edu.illinois.cs.cogcomp.core.datastructures.ViewNames import edu.illinois.cs.cogcomp.edison.features.factory.WordNetFeatureExtractor import edu.illinois.cs.cogcomp.saul.datamodel.DataModel import edu.illinois.cs.cogcomp.saulexamples.nlp.CommonSensors._ -import edu.illinois.cs.cogcomp.saulexamples.nlp.SpatialRoleLabeling.Triplet.{ SpRelation, SpRoleTypes } import edu.illinois.cs.cogcomp.saulexamples.nlp.SpatialRoleLabeling.SpRLSensors._ +import edu.illinois.cs.cogcomp.saulexamples.nlp.SpatialRoleLabeling.Triplet.{ SpRelation, SpRoleTypes } import scala.collection.JavaConverters._ -import scala.collection.immutable.HashSet import scala.collection.mutable.ListBuffer /** Created by taher on 8/10/16. @@ -60,13 +59,13 @@ object SpRLDataModel extends DataModel { val BF6 = property(relations) { x: SpRelation => { - val t = x.getTrajector.getFirstConstituent.getStartSpan + val trStart = x.getTrajector.getFirstConstituent.getStartSpan val spStart = x.getSpatialIndicator.getFirstConstituent.getStartSpan val spEnd = x.getSpatialIndicator.getLastConstituent.getStartSpan - if (t < spStart) - getDependencyPath(x.getTextAnnotation, t, spStart) + if (trStart < spStart) + getDependencyPath(x.getTextAnnotation, trStart, spStart) else - getDependencyPath(x.getTextAnnotation, t, spEnd) + getDependencyPath(x.getTextAnnotation, trStart, spEnd) } } diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/TextAnnotationFactory.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/TextAnnotationFactory.scala index 1602d462..2620b50a 100644 --- a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/TextAnnotationFactory.scala +++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/TextAnnotationFactory.scala @@ -9,12 +9,9 @@ package edu.illinois.cs.cogcomp.saulexamples.nlp import java.util.Properties import edu.illinois.cs.cogcomp.annotation.AnnotatorService -import edu.illinois.cs.cogcomp.core.datastructures.ViewNames import edu.illinois.cs.cogcomp.core.datastructures.textannotation.{ TextAnnotation, TokenLabelView } import edu.illinois.cs.cogcomp.core.utilities.configuration.{ Configurator, Property, ResourceManager } import edu.illinois.cs.cogcomp.curator.{ CuratorConfigurator, CuratorFactory } -import edu.illinois.cs.cogcomp.curator.CuratorConfigurator._ -import edu.illinois.cs.cogcomp.nlp.common.PipelineConfigurator._ import edu.illinois.cs.cogcomp.nlp.pipeline.IllinoisPipelineFactory /** Created by taher on 7/30/16. diff --git a/saul-examples/src/test/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/EntityRelation/EntityRelationTests.scala b/saul-examples/src/test/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/EntityRelation/EntityRelationTests.scala index 298fe7eb..3ce4f58a 100644 --- a/saul-examples/src/test/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/EntityRelation/EntityRelationTests.scala +++ b/saul-examples/src/test/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/EntityRelation/EntityRelationTests.scala @@ -92,14 +92,12 @@ class EntityRelationTests extends FlatSpec with Matchers { ClassifierUtils.TrainClassifiers(1, cls_base) - PerConstrainedClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner].getNetwork.get(0).asInstanceOf[LinearThresholdUnit].getWeightVector.size() should be(1660) + PerConstrainedClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner].getNetwork.get(0).asInstanceOf[LinearThresholdUnit].getWeightVector.size() should be(1654) - val jointTrainIteration = 1 - JointTrainSparseNetwork.train[ConllRelation]( - pairs, cls, jointTrainIteration, init = true - ) + val jointTrainIteration = 2 + JointTrainSparseNetwork.train[ConllRelation](pairs, cls, jointTrainIteration, init = true) - PerConstrainedClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner].getNetwork.get(0).asInstanceOf[LinearThresholdUnit].getWeightVector.size() should be(50) + PerConstrainedClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner].getNetwork.get(0).asInstanceOf[LinearThresholdUnit].getWeightVector.size() should be(81) } } \ No newline at end of file