diff --git a/README.md b/README.md
index 9dad706e..a4b05394 100644
--- a/README.md
+++ b/README.md
@@ -31,7 +31,6 @@ Visit each link for its content
5. [Data modeling and feature extraction](saul-core/doc/DATAMODELING.md)
6. [Learners and constraints](saul-core/doc/SAULLANGUAGE.md)
7. [Model configurations](saul-core/doc/MODELS.md)
- 8. [Saul library](saul-core/doc/LBJLIBRARY.md)
The api docs are included [here](http://cogcomp.cs.illinois.edu/software/doc/saul/).
diff --git a/saul-core/doc/MODELS.md b/saul-core/doc/MODELS.md
index cfdacadf..10df04df 100644
--- a/saul-core/doc/MODELS.md
+++ b/saul-core/doc/MODELS.md
@@ -1,6 +1,141 @@
-* Designing flexible learning models including various configurations such as:
- * Local models i.e. single classifiers. (Learning only models (LO)).
- * Constrained conditional models (CCM)[1] for training independent classifiers and using them jointly for global decision making in prediction time. (Learning+Inference (L+I)).
- * Global models for joint training and joint inference (Inference-Based-Training (IBT)).
- * Pipeline models for complex problems where the output of each layer is used as the input of the next layer.
+
+#Learning Paradigms
+
+/*Documented by Parisa Kordjamshidi*/
+
+Saul facilitates the flexible design of complex learning models with various configurations.
+By complex models we mean the models that aim at prediction of more than one output variable where these outputs might have relationships to each other.
+Such models can be designed using the following paradigms,
+
+ * [Local models](#local) trains single classifiers (Learning only models (LO)) each of which learns and predicts a single variable in the output independently.
+ * [Constrained conditional models (CCM)](#L+I) for training independent classifiers and using them jointly for global decision making in prediction time. (Learning+Inference (L+I)).
+ * [Global models](#IBT) for joint training and joint inference (Inference-Based-Training (IBT)).
+ * [Pipeline models](#pipeline) for complex problems where the output of each model is used as the input of the next model (these models are different layers in a pipeline).
+
+The above mentioned paradigms can be tested using this simple badge classifier example, [here](saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BagesApp.scala).
+
+
+##Local models
+These models are a set of single classifiers. Each classifier is defined with the `Learnable` construct and is trained and makes prediction independent from other classifiers.
+ The `Learnable` construct requires specifying a single output variable, that is, a label which is itself a property in the data model, and the features which is also a
+ comma separated list of properties.
+
+ ```scala
+ object ClassifierName extends Learnable (node) {
+ def label = property1
+ def feature = using(property2,property3,...)
+ //a comma separated list of properties
+ }
+ ```
+
+ For the details about the `Learnable` construct see [here](SAULLANGUAGE.md).
+
+
+##Learning+Inference models
+These models are useful for when we need to consider the global relations between the outputs of a bunch of classifiers during the
+prediction time. Each classifier is defined with the same `Learnable` construct as a local model. In addition to the Learnable definitions, the programmer
+has the possibility to define a number of logical constraints between the output of the Learnables (classifiers).
+Having the constraint definitions in place (see [here](SAULLANGUAGE.md) for syntax details), the programmer is able to define
+new constrained classifiers that use the Learnables and constraints.
+
+```scala
+object ConstrainedClassifierName extends ConstrainedClassifier[local_node_type,global_node_type](LocalClassifier)
+ {
+ def subjectTo = constraintExpression
+ // Logical constraint expression comes here, it defines the relations between the
+ // LocalClassifier and other Learnables defined before
+ }
+ ```
+When we use the above `ConstrainedClassifierName` to call test or make predictions, the `LocalClassifier` is used
+but the predictions are made in way that `constraintExpression` is hold. There is no limitation for the type of local classifiers.
+They can be SVMs, decision trees or any other learning models available in Saul, [here](lbjava/blob/master/lbjava/doc/ALGORITHMS.md)
+ and [here](saul-core/src/main/java/edu/illinois/cs/cogcomp/saul/learn/SaulWekaWrapper.md).
+
+
+##Inference-based Learning
+For the inference based models the basic definitions are exactly similar to the L+I models. In other words, the programmer
+just needs to define the `Learnables` and `ConstrainedClassifiers`. However, to train the ConstrainedClassifiers jointly, instead of
+training local classifiers independently, there are a couple of joint training functions that can be called.
+These functions receive the list of ConstrainedClassifiers as input and train their parameters jointly. In contrast to
+L+I models here the local classifiers can be defined as `SparsePerceptron`s or `SparseNetworkLearner`s only. This is because the
+joint training should have its own strategy for the wight updates of the involved variables (those variables come down to be the outputs of the local classifiers here).
+For the two cases the programmer can use
+
+```scala
+ JointTrainSparseNetwork.train(param1,param2,...) /* a list of parameters go here*/
+ JointTrainSparsePerceptron.train(param1,param2,...) /*a list of parameters here*/
+```
+
+For example,
+
+ ```scala
+ JointTrainSparseNetwork.train(badge, cls, 5, init = true, lossAugmented = true)
+ ```
+
+The list of parameters are the following:
+
+- param1: The name of a global node in the data model that itself or the connected nodes to it are used by the involved `Learnable`s.
+
+- param2: The collection of ```ConstainedClassifier```s
+
+- param3: The number of iterations over the training data.
+
+- param4: If the local classifiers should be cleaned from the possibly pre-trained values and initialized by zero weights, this parameter should be true.
+
+- param5: If the approach uses the loss augmented objective for making inference, see below for description.
+
+###Basic approach
+
+The basic approach for training the models jointly is to do a global prediction at each step of the training and if the
+predictions are wrong update the weights of the related variables.
+
+###Loss augmented
+
+The loss-augmented approach adds the loss of the prediction explicitly to the objective of the training and finds the most violated output per each training example;
+it updates the weights of the model according to the errors made in the prediction of the most violated output.
+This approach minimizes a convex upper bound of the loss function and has been used in structured SVMs and Structured Perceptrons.
+ However, considering an arbitrary loss in the objective will make complexities in the optimization, therefore in the implemented version, here, we assume the loss is decomposed similar to
+feature function. That is, the loss is a hamming loss defined per classifier. The loss of the whole structured output is computed by the weighted sum of
+ the loss of its components. For exmaple if there are two variables `c1` and `c2` in the output with corresponding predictions `cp1` and `cp2` then the loss will be
+ `[hamming(c1,cp1)/2+hamming(c2,cp2)/2]`. The weight of each component's loss is `1/numberOfOutputVariables` by default.
+
+ In Saul, the programmer can indicate if he/she needs to consider this global hamming loss in the objective or not. And this can be done by passing
+ the above mentioned `param5` as true in the `JointTrainingSparseNetwork` algorithm.
+ An example of this usage can be seen [here](saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BagesApp.scala#L64).
+
+
+##Pipelines
+
+Building pipelines is naturally granted in Saul. The programmer can simply define properties that are the predictions of
+the classifiers and use those outputs as the input of other classifiers by mentioning them in the list of the properties in the below construct when defining the
+pipeline classifiers,
+
+```scala
+ def feature = using(/*list of properties including the prediction of other classifiers.*/)
+```
+
+Here is a more complete example which passes the output of the `ClassifierLayer1` to the input of the `ClassifierLayer2`:
+
+ ```scala
+ object ClassifierLayer1 extends Learnable (node) {
+ def label = labelProperty1
+ def feature = using(property2, property3,...)
+ }
+ object ClassifierLayer2 extends Learnable (node) {
+ def label = labelProperty2
+ def feature = using(classifier1Labels, ,...) // using the prediction of the classifier in the previous layer
+ }
+ ```
+
+This will be defined in data-model object:
+
+ ```scala
+ val classifier1Labels = new Property(node){ x: Type => ClassifierLayer1(x) }
+ ```
+
+See [here](saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeClassifiers.scala#L43) for a working example.
+
+
+
+
diff --git a/saul-core/doc/SAULLANGUAGE.md b/saul-core/doc/SAULLANGUAGE.md
index ec022301..f4cab89a 100644
--- a/saul-core/doc/SAULLANGUAGE.md
+++ b/saul-core/doc/SAULLANGUAGE.md
@@ -35,8 +35,8 @@ OrgClassifier.test()
### Availale algorithms
Here is a list of available algorithms in Saul:
- - [LBJava learning algorithms](https://github.com/IllinoisCogComp/lbjava/blob/master/lbjava/doc/ALGORITHMS.md)
- - [Weka learning algorithms](https://github.com/IllinoisCogComp/saul/blob/master/saul-core/src/main/java/edu/illinois/cs/cogcomp/saul/learn/SaulWekaWrapper.md)
+ - [LBJava learning algorithms](https://github.com/IllinoisCogComp/lbjava/blob/master/lbjava/doc/ALGORITHMS.md)
+ - [Weka learning algorithms](saul-core/src/main/java/edu/illinois/cs/cogcomp/saul/learn/SaulWekaWrapper.md)
### Saving and loading classifiers
diff --git a/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparseNetwork.scala b/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparseNetwork.scala
index 4ac33f82..e114fadf 100644
--- a/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparseNetwork.scala
+++ b/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparseNetwork.scala
@@ -9,7 +9,7 @@ package edu.illinois.cs.cogcomp.saul.classifier
import edu.illinois.cs.cogcomp.lbjava.learn.{ LinearThresholdUnit, SparseNetworkLearner }
import edu.illinois.cs.cogcomp.saul.datamodel.node.Node
import org.slf4j.{ Logger, LoggerFactory }
-
+import Predef._
import scala.reflect.ClassTag
/** Created by Parisa on 5/22/15.
@@ -18,16 +18,16 @@ object JointTrainSparseNetwork {
val logger: Logger = LoggerFactory.getLogger(this.getClass)
var difference = 0
- def apply[HEAD <: AnyRef](node: Node[HEAD], cls: List[ConstrainedClassifier[_, HEAD]], init: Boolean)(implicit headTag: ClassTag[HEAD]) = {
- train[HEAD](node, cls, 1, init)
+ def apply[HEAD <: AnyRef](node: Node[HEAD], cls: List[ConstrainedClassifier[_, HEAD]], init: Boolean, lossAugmented: Boolean)(implicit headTag: ClassTag[HEAD]) = {
+ train[HEAD](node, cls, 1, init, lossAugmented)
}
- def apply[HEAD <: AnyRef](node: Node[HEAD], cls: List[ConstrainedClassifier[_, HEAD]], it: Int, init: Boolean)(implicit headTag: ClassTag[HEAD]) = {
- train[HEAD](node, cls, it, init)
+ def apply[HEAD <: AnyRef](node: Node[HEAD], cls: List[ConstrainedClassifier[_, HEAD]], it: Int, init: Boolean, lossAugmented: Boolean = false)(implicit headTag: ClassTag[HEAD]) = {
+ train[HEAD](node, cls, it, init, lossAugmented)
}
@scala.annotation.tailrec
- def train[HEAD <: AnyRef](node: Node[HEAD], cls: List[ConstrainedClassifier[_, HEAD]], it: Int, init: Boolean)(implicit headTag: ClassTag[HEAD]): Unit = {
+ def train[HEAD <: AnyRef](node: Node[HEAD], cls: List[ConstrainedClassifier[_, HEAD]], it: Int, init: Boolean, lossAugmented: Boolean = false)(implicit headTag: ClassTag[HEAD]): Unit = {
// forall members in collection of the head (dm.t) do
logger.info("Training iteration: " + it)
if (init) ClassifierUtils.InitializeClassifiers(node, cls: _*)
@@ -43,19 +43,25 @@ object JointTrainSparseNetwork {
if (idx % 5000 == 0)
logger.info(s"Training: $idx examples inferred.")
- cls.foreach {
- case classifier: ConstrainedClassifier[_, HEAD] =>
- val typedClassifier = classifier.asInstanceOf[ConstrainedClassifier[_, HEAD]]
- val oracle = typedClassifier.onClassifier.getLabeler
+ if (lossAugmented)
+ cls.foreach { cls_i =>
+ cls_i.onClassifier.classifier.setLossFlag()
+ cls_i.onClassifier.classifier.setCandidates(cls_i.getCandidates(h).size * cls.size)
+ }
- typedClassifier.getCandidates(h) foreach {
+ cls.foreach {
+ currentClassifier: ConstrainedClassifier[_, HEAD] =>
+ assert(currentClassifier.onClassifier.classifier.getClass.getName.contains("SparseNetworkLearner"), "The classifier should be of type SparseNetworkLearner!")
+ val oracle = currentClassifier.onClassifier.getLabeler
+ val baseClassifier = currentClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner]
+ currentClassifier.getCandidates(h) foreach {
candidate =>
{
def trainOnce() = {
- val result = typedClassifier.classifier.discreteValue(candidate)
+
+ val result = currentClassifier.classifier.discreteValue(candidate)
val trueLabel = oracle.discreteValue(candidate)
- val ilearner = typedClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner]
- val lLexicon = typedClassifier.onClassifier.getLabelLexicon
+ val lLexicon = currentClassifier.onClassifier.getLabelLexicon
var LTU_actual: Int = 0
var LTU_predicted: Int = 0
for (i <- 0 until lLexicon.size()) {
@@ -69,26 +75,25 @@ object JointTrainSparseNetwork {
// and the LTU of the predicted class should be demoted.
if (!result.equals(trueLabel)) //equals("true") && trueLabel.equals("false") )
{
- val a = typedClassifier.onClassifier.getExampleArray(candidate)
+ val a = currentClassifier.onClassifier.getExampleArray(candidate)
val a0 = a(0).asInstanceOf[Array[Int]] //exampleFeatures
val a1 = a(1).asInstanceOf[Array[Double]] // exampleValues
val exampleLabels = a(2).asInstanceOf[Array[Int]]
val label = exampleLabels(0)
- var N = ilearner.getNetwork.size
+ val N = baseClassifier.getNetwork.size
- if (label >= N || ilearner.getNetwork.get(label) == null) {
- val conjugateLabels = ilearner.isUsingConjunctiveLabels | ilearner.getLabelLexicon.lookupKey(label).isConjunctive
- ilearner.setConjunctiveLabels(conjugateLabels)
+ if (label >= N || baseClassifier.getNetwork.get(label) == null) {
+ val conjugateLabels = baseClassifier.isUsingConjunctiveLabels | baseClassifier.getLabelLexicon.lookupKey(label).isConjunctive
+ baseClassifier.setConjunctiveLabels(conjugateLabels)
- val ltu: LinearThresholdUnit = ilearner.getBaseLTU
- ltu.initialize(ilearner.getNumExamples, ilearner.getNumFeatures)
- ilearner.getNetwork.set(label, ltu)
- N = label + 1
+ val ltu: LinearThresholdUnit = baseClassifier.getBaseLTU.clone().asInstanceOf[LinearThresholdUnit]
+ ltu.initialize(baseClassifier.getNumExamples, baseClassifier.getNumFeatures)
+ baseClassifier.getNetwork.set(label, ltu)
}
// test push
- val ltu_actual = ilearner.getLTU(LTU_actual).asInstanceOf[LinearThresholdUnit]
- val ltu_predicted = ilearner.getLTU(LTU_predicted).asInstanceOf[LinearThresholdUnit]
+ val ltu_actual = baseClassifier.getLTU(LTU_actual).asInstanceOf[LinearThresholdUnit]
+ val ltu_predicted = baseClassifier.getLTU(LTU_predicted).asInstanceOf[LinearThresholdUnit]
if (ltu_actual != null)
ltu_actual.promote(a0, a1, 0.1)
@@ -100,8 +105,13 @@ object JointTrainSparseNetwork {
trainOnce()
}
}
+
}
}
+ if (lossAugmented)
+ cls.foreach { cls_i =>
+ cls_i.onClassifier.classifier.unsetLossFlag()
+ }
}
train(node, cls, it - 1, false)
}
diff --git a/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrain.scala b/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparsePerceptron.scala
similarity index 99%
rename from saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrain.scala
rename to saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparsePerceptron.scala
index 2f046380..4b70d89e 100644
--- a/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrain.scala
+++ b/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/JointTrainSparsePerceptron.scala
@@ -14,7 +14,7 @@ import scala.reflect.ClassTag
/** Created by parisakordjamshidi on 29/01/15.
*/
-object JointTrain {
+object JointTrainSparsePerceptron {
def testClassifiers(cls: Classifier, oracle: Classifier, ds: List[AnyRef]): Unit = {
val results = ds.map({
diff --git a/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/infer/InitSparseNetwork.scala b/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/infer/InitSparseNetwork.scala
index a9df6c4c..d1f738fe 100644
--- a/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/infer/InitSparseNetwork.scala
+++ b/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/classifier/infer/InitSparseNetwork.scala
@@ -18,7 +18,10 @@ object InitSparseNetwork {
//this means we are not reading any model into the SparseNetworks
// but we forget all the models and go over the data to build the right
// size for the lexicon and the right number of the ltu s
+
cClassifier.onClassifier.classifier.forget()
+ assert(cClassifier.onClassifier.classifier.getClass.getName.contains("SparseNetworkLearner"), "The classifier should be of type SparseNetworkLearner!")
+
val iLearner = cClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner]
allHeads.foreach {
head =>
@@ -33,7 +36,7 @@ object InitSparseNetwork {
if (label >= N || iLearner.getNetwork.get(label) == null) {
val isConjunctiveLabels = iLearner.isUsingConjunctiveLabels | iLearner.getLabelLexicon.lookupKey(label).isConjunctive
iLearner.setConjunctiveLabels(isConjunctiveLabels)
- val ltu: LinearThresholdUnit = iLearner.getBaseLTU
+ val ltu: LinearThresholdUnit = iLearner.getBaseLTU.clone().asInstanceOf[LinearThresholdUnit]
ltu.initialize(iLearner.getNumExamples, iLearner.getNumFeatures)
iLearner.getNetwork.set(label, ltu)
}
diff --git a/saul-core/src/test/scala/edu/illinois/cs/cogcomp/saul/classifier/JoinTrainingTests/IntializeSparseNetwork.scala b/saul-core/src/test/scala/edu/illinois/cs/cogcomp/saul/classifier/JoinTrainingTests/IntializeSparseNetwork.scala
index 605d03ba..cc284619 100644
--- a/saul-core/src/test/scala/edu/illinois/cs/cogcomp/saul/classifier/JoinTrainingTests/IntializeSparseNetwork.scala
+++ b/saul-core/src/test/scala/edu/illinois/cs/cogcomp/saul/classifier/JoinTrainingTests/IntializeSparseNetwork.scala
@@ -1,3 +1,9 @@
+/** This software is released under the University of Illinois/Research and Academic Use License. See
+ * the LICENSE file in the root folder for details. Copyright (c) 2016
+ *
+ * Developed by: The Cognitive Computations Group, University of Illinois at Urbana-Champaign
+ * http://cogcomp.cs.illinois.edu/
+ */
package edu.illinois.cs.cogcomp.saul.classifier.JoinTrainingTests
import edu.illinois.cs.cogcomp.infer.ilp.OJalgoHook
@@ -71,7 +77,7 @@ class InitializeSparseNetwork extends FlatSpec with Matchers {
val wv1After = clNet1.getNetwork.get(0).asInstanceOf[LinearThresholdUnit].getWeightVector
val wv2After = clNet2.getNetwork.get(0).asInstanceOf[LinearThresholdUnit].getWeightVector
- wv1After.size() should be(5)
+ wv1After.size() should be(6)
wv2After.size() should be(12)
}
diff --git a/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeReader.java b/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeReader.java
new file mode 100644
index 00000000..456f5e6f
--- /dev/null
+++ b/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeReader.java
@@ -0,0 +1,32 @@
+/** This software is released under the University of Illinois/Research and Academic Use License. See
+ * the LICENSE file in the root folder for details. Copyright (c) 2016
+ *
+ * Developed by: The Cognitive Computations Group, University of Illinois at Urbana-Champaign
+ * http://cogcomp.cs.illinois.edu/
+ */
+package edu.illinois.cs.cogcomp.saulexamples.Badge;
+
+import java.io.BufferedReader;
+import java.io.FileInputStream;
+import java.io.InputStreamReader;
+import java.util.ArrayList;
+import java.util.List;
+
+public class BadgeReader {
+ public List badges;
+
+ public BadgeReader(String dataFile) {
+ badges = new ArrayList();
+
+ try {
+ BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(dataFile)));
+
+ String str;
+ while ((str = br.readLine()) != null) {
+ badges.add(str);
+ }
+
+ br.close();
+ }catch (Exception e) {}
+ }
+}
\ No newline at end of file
diff --git a/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLConfigurator.java b/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLConfigurator.java
index f006e1af..a1ad3b03 100644
--- a/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLConfigurator.java
+++ b/saul-examples/src/main/java/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLConfigurator.java
@@ -21,7 +21,6 @@ public class SRLConfigurator extends Configurator {
public static final Property TREEBANK_HOME = new Property("treebankHome", "../saul-examples/src/test/resources/SRLToy/treebank");
public static final Property PROPBANK_HOME = new Property("propbankHome","../saul-examples/src/test/resources/SRLToy/propbank");
-
public static final Property TEST_SECTION = new Property("testSection","00");
public static final Property MODELS_DIR = new Property("modelsDir", "../models");
@@ -30,7 +29,7 @@ public class SRLConfigurator extends Configurator {
// The running mode of the program. Can be "true" for only testing, or "false" for training
public static final Property RUN_MODE = new Property("runMode", Configurator.TRUE);
- // The training mode for the examples. Can be "pipeline", "joint", or "other"
+ // The training mode for the examples. Can be "pipeline", "joint", "jointLoss" or "other"
public static final Property TRAINING_MODE = new Property("trainingMode", "joint");
/*********** SRL PROPERTIES ***********/
diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeClassifiers.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeClassifiers.scala
new file mode 100644
index 00000000..d9febc2b
--- /dev/null
+++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeClassifiers.scala
@@ -0,0 +1,48 @@
+/** This software is released under the University of Illinois/Research and Academic Use License. See
+ * the LICENSE file in the root folder for details. Copyright (c) 2016
+ *
+ * Developed by: The Cognitive Computations Group, University of Illinois at Urbana-Champaign
+ * http://cogcomp.cs.illinois.edu/
+ */
+package edu.illinois.cs.cogcomp.saulexamples.Badge
+
+import edu.illinois.cs.cogcomp.lbjava.learn.{ SparseNetworkLearner, SparsePerceptron }
+import edu.illinois.cs.cogcomp.saul.classifier.Learnable
+import edu.illinois.cs.cogcomp.saulexamples.Badge.BadgeDataModel._
+/** Created by Parisa on 9/13/16.
+ */
+
+object BadgeClassifiers {
+
+ /*a binary classifier that predicts the badge label*/
+ object BadgeClassifier extends Learnable[String](badge) {
+ def label = BadgeLabel
+ override lazy val classifier = new SparsePerceptron()
+ override def feature = using(BadgeFeature1)
+ }
+ /*a binary classifier that predicts a lable that is exactly the opposite of the badge label of the BadgeClassifier*/
+ object BadgeOppositClassifier extends Learnable[String](badge) {
+ def label = BadgeOppositLabel
+ override lazy val classifier = new SparsePerceptron()
+ override def feature = using(BadgeFeature1)
+ }
+ /*This is a multi-class classifier version of the above binary BadgeClassifier,
+it uses SparseNetworks instead of SparsePerceptrons*/
+ object BadgeClassifierMulti extends Learnable[String](badge) {
+ def label = BadgeLabel
+ override lazy val classifier = new SparseNetworkLearner()
+ override def feature = using(BadgeFeature1)
+ }
+ /*This is the opposite multi-class classifier of the BadgeClassifierMulti */
+ object BadgeOppositClassifierMulti extends Learnable[String](badge) {
+ def label = BadgeOppositLabel
+ override lazy val classifier = new SparseNetworkLearner()
+ override def feature = using(BadgeFeature1)
+ }
+ /* */
+ object BadgeOppositPipeline extends Learnable[String](badge) {
+ def label = BadgeOppositLabel
+ override lazy val classifier = new SparsePerceptron()
+ override def feature = using(BadgePrediction)
+ }
+}
\ No newline at end of file
diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeConstrainedClassifiers.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeConstrainedClassifiers.scala
new file mode 100644
index 00000000..b5587d9d
--- /dev/null
+++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeConstrainedClassifiers.scala
@@ -0,0 +1,47 @@
+/** This software is released under the University of Illinois/Research and Academic Use License. See
+ * the LICENSE file in the root folder for details. Copyright (c) 2016
+ *
+ * Developed by: The Cognitive Computations Group, University of Illinois at Urbana-Champaign
+ * http://cogcomp.cs.illinois.edu/
+ */
+package edu.illinois.cs.cogcomp.saulexamples.Badge
+
+import edu.illinois.cs.cogcomp.infer.ilp.OJalgoHook
+import edu.illinois.cs.cogcomp.saul.classifier.ConstrainedClassifier
+import edu.illinois.cs.cogcomp.saul.constraint.ConstraintTypeConversion._
+import edu.illinois.cs.cogcomp.saulexamples.Badge.BadgeClassifiers.{ BadgeOppositClassifierMulti, BadgeClassifierMulti, BadgeClassifier, BadgeOppositClassifier }
+
+/** Created by Parisa on 11/1/16.
+ */
+object BadgeConstrainedClassifiers {
+
+ val binaryConstraint = ConstrainedClassifier.constraint[String] {
+ x: String =>
+ (BadgeClassifier on x is "negative") ==> (BadgeOppositClassifier on x is "positive")
+ }
+
+ val binaryConstraintOverMultiClassifiers = ConstrainedClassifier.constraint[String] {
+ x: String =>
+ (BadgeClassifierMulti on x is "negative") ==> (BadgeOppositClassifierMulti on x is "positive")
+ }
+ object badgeConstrainedClassifier extends ConstrainedClassifier[String, String](BadgeClassifier) {
+ def subjectTo = binaryConstraint
+ override val solver = new OJalgoHook
+ }
+
+ object oppositBadgeConstrainedClassifier extends ConstrainedClassifier[String, String](BadgeOppositClassifier) {
+ def subjectTo = binaryConstraint
+ override val solver = new OJalgoHook
+ }
+
+ object badgeConstrainedClassifierMulti extends ConstrainedClassifier[String, String](BadgeClassifierMulti) {
+ def subjectTo = binaryConstraintOverMultiClassifiers
+ override val solver = new OJalgoHook
+ }
+
+ object oppositBadgeConstrainedClassifierMulti extends ConstrainedClassifier[String, String](BadgeOppositClassifierMulti) {
+ def subjectTo = binaryConstraintOverMultiClassifiers
+ override val solver = new OJalgoHook
+ }
+
+}
diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeDataModel.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeDataModel.scala
new file mode 100644
index 00000000..3d1e9366
--- /dev/null
+++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgeDataModel.scala
@@ -0,0 +1,49 @@
+/** This software is released under the University of Illinois/Research and Academic Use License. See
+ * the LICENSE file in the root folder for details. Copyright (c) 2016
+ *
+ * Developed by: The Cognitive Computations Group, University of Illinois at Urbana-Champaign
+ * http://cogcomp.cs.illinois.edu/
+ */
+package edu.illinois.cs.cogcomp.saulexamples.Badge
+
+import edu.illinois.cs.cogcomp.saul.datamodel.DataModel
+import edu.illinois.cs.cogcomp.saulexamples.Badge.BadgeClassifiers.BadgeClassifier
+
+/** Created by Parisa on 9/13/16.
+ */
+object BadgeDataModel extends DataModel {
+
+ val badge = node[String]
+
+ val BadgeFeature1 = property(badge) {
+ x: String =>
+ {
+ val tokens = x.split(" ")
+ tokens(1).charAt(1).toString
+ }
+ }
+
+ val BadgeLabel = property(badge)("true", "false") {
+ x: String =>
+ {
+ val tokens = x.split(" ")
+ if (tokens(0).equals("+"))
+ "true"
+ else
+ "false"
+ }
+ }
+
+ val BadgeOppositLabel = property(badge)("true", "false") {
+ x: String =>
+ {
+ val tokens = x.split(" ")
+ if (tokens(0).equals("+"))
+ "false"
+ else
+ "true"
+ }
+ }
+
+ val BadgePrediction = property(badge)("true", "false") { x: String => BadgeClassifier(x) }
+}
diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgesApp.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgesApp.scala
new file mode 100644
index 00000000..87c93b72
--- /dev/null
+++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/BadgesApp.scala
@@ -0,0 +1,82 @@
+/** This software is released under the University of Illinois/Research and Academic Use License. See
+ * the LICENSE file in the root folder for details. Copyright (c) 2016
+ *
+ * Developed by: The Cognitive Computations Group, University of Illinois at Urbana-Champaign
+ * http://cogcomp.cs.illinois.edu/
+ */
+package edu.illinois.cs.cogcomp.saulexamples.Badge
+
+/** Created by Parisa on 9/13/16.
+ */
+
+import edu.illinois.cs.cogcomp.saul.classifier.{ JointTrainSparseNetwork, JointTrainSparsePerceptron }
+import edu.illinois.cs.cogcomp.saulexamples.Badge.BadgeClassifiers._
+import edu.illinois.cs.cogcomp.saulexamples.Badge.BadgeConstrainedClassifiers.{ badgeConstrainedClassifier, badgeConstrainedClassifierMulti, oppositBadgeConstrainedClassifier, oppositBadgeConstrainedClassifierMulti }
+import edu.illinois.cs.cogcomp.saulexamples.Badge.BadgeDataModel._
+
+import scala.collection.JavaConversions._
+object BadgesApp {
+
+ val allNamesTrain = new BadgeReader("data/badges/badges.train").badges
+ val allNamesTest = new BadgeReader("data/badges/badges.test").badges
+
+ badge.populate(allNamesTrain)
+ badge.populate(allNamesTest, false)
+
+ val cls = List(badgeConstrainedClassifierMulti, oppositBadgeConstrainedClassifierMulti)
+
+ object BadgeExperimentType extends Enumeration {
+ val JoinTrainSparsePerceptron, JointTrainSparseNetwork, JointTrainSparseNetworkLossAugmented, Pipeline = Value
+ }
+
+ def main(args: Array[String]): Unit = {
+
+ /** Choose the experiment you're interested in by changing the following line */
+ val testType = BadgeExperimentType.Pipeline
+
+ testType match {
+ case BadgeExperimentType.JoinTrainSparsePerceptron => JoinTrainSparsePerceptron()
+ case BadgeExperimentType.JointTrainSparseNetwork => JoinTrainSparseNetwork()
+ case BadgeExperimentType.JointTrainSparseNetworkLossAugmented => LossAugmentedJoinTrainSparseNetwork()
+ case BadgeExperimentType.Pipeline => Pipeline()
+ }
+ }
+
+ /*Test the join training with SparsePerceptron*/
+ def JoinTrainSparsePerceptron(): Unit = {
+ BadgeClassifier.test()
+ BadgeOppositClassifier.test()
+ JointTrainSparsePerceptron.train(BadgeDataModel.badge, List(badgeConstrainedClassifier, oppositBadgeConstrainedClassifier), 5)
+ oppositBadgeConstrainedClassifier.test()
+ badgeConstrainedClassifier.test()
+ BadgeClassifier.test()
+ }
+
+ /*Test the joinTraining with SparseNetwork*/
+ def JoinTrainSparseNetwork(): Unit = {
+
+ JointTrainSparseNetwork.train(badge, cls, 5, init = true)
+
+ badgeConstrainedClassifierMulti.test()
+ oppositBadgeConstrainedClassifierMulti.test()
+ }
+
+ /*Test the joinTraining with SparseNetwork and doing loss augmented inference*/
+ def LossAugmentedJoinTrainSparseNetwork(): Unit = {
+
+ JointTrainSparseNetwork.train(badge, cls, 5, init = true, lossAugmented = true)
+
+ badgeConstrainedClassifierMulti.test()
+ oppositBadgeConstrainedClassifierMulti.test()
+ }
+
+ /* This model trains the BadgeClassifier and then it takes the prediction of the BadgeClassifier as the only input
+ feature and trains a pipeline function to predict the opposite label*/
+ def Pipeline(): Unit = {
+ BadgeClassifier.learn(5)
+ BadgeClassifier.test()
+ BadgeOppositPipeline.learn(5)
+ BadgeOppositPipeline.test()
+ }
+
+}
\ No newline at end of file
diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/README.md b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/README.md
new file mode 100644
index 00000000..71a0d5be
--- /dev/null
+++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/Badge/README.md
@@ -0,0 +1,9 @@
+
+This example is a simple model, that receives names of people and assigns a label either positive or negative to them.
+It uses the second character of the first names as an input feature. This is a gold feature that can distinguish 100% between
+the positive and negative class.
+We show the usage of binary classifiers as well as multi-class classifiers in [here](BadgeClassifiers.scala).
+In this file you could see we define two type of the classifiers that they take opposite labels. The goal is to simply show
+how a simple constraint can impose the predictions of these two classifiers to be opposite.
+The constrained versions of all these classifiers can be found [here](BadgeConstraintClassifiers.scala).
+Using a simple constraint we train various joint models that train the two opposite classifiers jointly in [here](BadgesApp.scala).
\ No newline at end of file
diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/DrugResponse/myApp.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/DrugResponse/myApp.scala
index 5c5ca870..5b93d1c8 100644
--- a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/DrugResponse/myApp.scala
+++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/DrugResponse/myApp.scala
@@ -75,6 +75,7 @@ object myApp extends Logging {
//dResponseClassifier.testContinuos(patient_drug_data)
//DrugResponseRegressor.learn(1)
+
//DrugResponseRegressor.testContinuos(patientDrug.getTrainingInstances)
logger.info("finished!")
}
diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/PopulateSRLDataModel.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/PopulateSRLDataModel.scala
index 6ed49e04..9796653b 100644
--- a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/PopulateSRLDataModel.scala
+++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/PopulateSRLDataModel.scala
@@ -19,10 +19,10 @@ import edu.illinois.cs.cogcomp.nlp.common.PipelineConfigurator._
import edu.illinois.cs.cogcomp.nlp.utilities.ParseUtils
import edu.illinois.cs.cogcomp.saul.util.Logging
import edu.illinois.cs.cogcomp.saulexamples.data.{ SRLDataReader, SRLFrameManager }
-import edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.SRLSensors._
import edu.illinois.cs.cogcomp.saulexamples.nlp.CommonSensors._
+import edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.SRLSensors._
import edu.illinois.cs.cogcomp.saulexamples.nlp.TextAnnotationFactory
-
+import edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.SRLscalaConfigurator._
import scala.collection.JavaConversions._
/** Created by Parisa on 1/17/16.
@@ -34,7 +34,7 @@ object PopulateSRLDataModel extends Logging {
useGoldArgBoundaries: Boolean = false,
rm: ResourceManager = new SRLConfigurator().getDefaultConfig
): SRLMultiGraphDataModel = {
- val frameManager: SRLFrameManager = new SRLFrameManager(rm.getString(SRLConfigurator.PROPBANK_HOME.key))
+ val frameManager: SRLFrameManager = new SRLFrameManager(PROPBANK_HOME)
val useCurator = rm.getBoolean(SRLConfigurator.USE_CURATOR)
val parseViewName = rm.getString(SRLConfigurator.SRL_PARSE_VIEW)
val graphs = new SRLMultiGraphDataModel(parseViewName, frameManager)
@@ -86,16 +86,11 @@ object PopulateSRLDataModel extends Logging {
logger.debug(s"Number of $readerType data arguments: $numArguments")
}
- val trainingFromSection = 2
- val trainingToSection = 2
var gr: SRLMultiGraphDataModel = null
if (!testOnly) {
- logger.info(s"Reading training data from sections $trainingFromSection to $trainingToSection")
- val trainReader = new SRLDataReader(
- rm.getString(SRLConfigurator.TREEBANK_HOME.key),
- rm.getString(SRLConfigurator.PROPBANK_HOME.key),
- trainingFromSection, trainingToSection
- )
+ logger.info(s"Reading training data from sections $TRAIN_SECTION_S to $TRAIN_SECTION_E")
+ val trainReader = new SRLDataReader(TREEBANK_HOME, PROPBANK_HOME,
+ TRAIN_SECTION_S, TRAIN_SECTION_E)
trainReader.readData()
logger.info(s"Annotating ${trainReader.textAnnotations.size} training sentences")
val filteredTa = addViewAndFilter(trainReader.textAnnotations.toList)
@@ -124,13 +119,9 @@ object PopulateSRLDataModel extends Logging {
if (graphs.sentences().size % 1000 == 0) logger.info("loaded graphs in memory:" + graphs.sentences().size)
}
}
- val testSection = rm.getInt(SRLConfigurator.TEST_SECTION)
- val testReader = new SRLDataReader(
- rm.getString(SRLConfigurator.TREEBANK_HOME.key),
- rm.getString(SRLConfigurator.PROPBANK_HOME.key),
- testSection, testSection
- )
- logger.info(s"Reading test data from section $testSection")
+
+ val testReader = new SRLDataReader(TREEBANK_HOME, PROPBANK_HOME, TEST_SECTION, TEST_SECTION)
+ logger.info(s"Reading test data from section $TEST_SECTION")
testReader.readData()
logger.info(s"Annotating ${testReader.textAnnotations.size} test sentences")
diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/README.md b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/README.md
index d27abf74..43fafeaf 100644
--- a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/README.md
+++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/README.md
@@ -5,13 +5,13 @@ This task is to annotate natural language sentences with semantic roles.
To run the main app with default properties:
```
-sbt "project saulExamples" "run-main edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.srlApp"
+sbt "project saulExamples" "run-main edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.RunningApps"
```
-To use a custom configuration file (containing the property keys of `ExamplesConfigurator`):
+To use a custom configuration file (containing the property keys of `ExamplesConfigurator`), also extending memory to 4G:
```
- sbt "project saulExamples" "run-main edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.srlApp config/saul-srl.properties"
+ sbt -mem 4000 "project saulExamples" "run-main edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.RunningApps config/saul-srl.properties"
```
## Example
@@ -29,6 +29,7 @@ Similar to other applications in Saul, here also we have a datamodel in file `SR
classifier definitions in file `SRLClassifires`, a bunch of constraints to be used by global models during either training
or test in file `SRLConstraints`, a bunch of constrained classifiers in file `SRLConstrainedClassifiers` and the running
configurations that are all placed in one file called `SRLApp`.
+
For using the reader and populating data there is a program in file `PopulateSRLDataModel`.
In contrast to other Saul applications this data model has been defined as a class instead of as an object. The reason
is the efficiency of the population of the data model, we skip the details of this implementation choice.
@@ -40,7 +41,7 @@ the properties which use the FrameNet frames can receive it as a parameter.
We refer the reader to see an example of defining such parametrized properties along with `Learnable` classes and
contrast it with the `Learnable` objects in `DrugResponse` example of `KnowEng` data model.
-There are various machine learning configurations to solve this including pipelines, learning only models (LO),
+There are various machine learning configurations to solve SRL including pipelines, learning only models (LO),
learning plus inference models (L+I) and Joint Learning models (IBT).
The test units and SRLApp are runnable on a sample toy dataset located in `resources/SRLToy` folder.
Accessing the full dataset needs proper licenses. If you have access to the PropBank data, you could
@@ -61,13 +62,12 @@ In the following lines `Pred.` stands for "Predicate" and `Cand.` stands for "C
| Predicate | Argument | Model | Name |
| -----------| -------------------- | ------------------------ | ---- |
-| Gold Pred. | Gold Boundaries | Argument Type Classifier | aTr |
+| Gold Pred. | Gold Boundaries | Argument Type Classifier | aTr,jTr,lTr |
| Gold Pred. | XuPalmer Candidates | Argument identifier | bTr |
| Gold Pred. | XuPalmer Candidates | Argument Type Classifier | cTr |
| Predicted Cand. | NA | Predicate Classifier | dTr |
| Predicted Cand. | XuPalmer Candidates | Argument identifier | eTr |
| Predicted Cand. | XuPalmer Candidates | Argument Type Classifier | fTr |
-| Gold Pred. | Gold Boundries | Argument Type Classifier | jTr |
| Gold Pred. | Argument Identifier | Argument Type Classifier | pTr |
@@ -577,6 +577,122 @@ In the following lines `Pred.` stands for "Predicate" and `Cand.` stands for "C
* Add constraints gradually and test.
#### Third phase: training joint models
+ - [x] **[lTr]** Train aTr jointly using hamming loss; test using constraints
+
+
+ Average evaluation time: 0.04143100828729282 seconds
+
+ Label Precision Recall F1 LCount PCount
+ ----------------------------------------------
+ A0 96.585 91.699 94.079 3578 3397
+ A1 90.352 93.517 91.907 4967 5141
+ A2 78.791 64.711 71.060 1108 910
+ A3 61.176 60.465 60.819 172 170
+ A4 28.525 85.294 42.752 102 305
+ A5 57.143 80.000 66.667 5 7
+ AA 0.000 0.000 0.000 0 2
+ AM-ADV 75.987 45.652 57.037 506 304
+ AM-CAU 73.333 43.421 54.545 76 45
+ AM-DIR 42.308 51.765 46.561 85 104
+ AM-DIS 75.833 85.313 80.294 320 360
+ AM-EXT 50.000 50.000 50.000 32 32
+ AM-LOC 59.244 38.525 46.689 366 238
+ AM-MNR 34.126 70.115 45.908 348 715
+ AM-MOD 97.513 99.637 98.564 551 563
+ AM-NEG 97.854 99.130 98.488 230 233
+ AM-PNC 57.333 37.391 45.263 115 75
+ AM-PRD 0.000 0.000 0.000 5 11
+ AM-REC 0.000 0.000 0.000 2 0
+ AM-TMP 75.690 76.097 75.893 1117 1123
+ C-A0 83.333 27.778 41.667 18 6
+ C-A1 68.465 72.052 70.213 229 241
+ C-A2 14.286 40.000 21.053 5 14
+ C-A3 0.000 0.000 0.000 3 11
+ C-A4 0.000 0.000 0.000 0 7
+ C-A5 0.000 0.000 0.000 0 1
+ C-AM-ADV 0.000 0.000 0.000 0 1
+ C-AM-DIR 0.000 0.000 0.000 0 3
+ C-AM-EXT 0.000 0.000 0.000 0 1
+ C-AM-LOC 0.000 0.000 0.000 0 1
+ C-AM-MNR 0.000 0.000 0.000 0 7
+ C-AM-NEG 0.000 0.000 0.000 0 3
+ C-AM-PNC 0.000 0.000 0.000 0 1
+ C-V 0.000 0.000 0.000 141 31
+ R-A0 85.388 88.208 86.775 212 219
+ R-A1 68.919 77.863 73.118 131 148
+ R-A2 62.500 38.462 47.619 13 8
+ R-A3 0.000 0.000 0.000 1 0
+ R-A4 0.000 0.000 0.000 1 7
+ R-AM-ADV 0.000 0.000 0.000 2 1
+ R-AM-CAU 0.000 0.000 0.000 1 3
+ R-AM-EXT 0.000 0.000 0.000 1 3
+ R-AM-LOC 69.231 56.250 62.069 16 13
+ R-AM-MNR 0.000 0.000 0.000 2 5
+ R-AM-PNC 0.000 0.000 0.000 0 3
+ ----------------------------------------------
+ R-AM-TMP 16.667 5.556 8.333 18 6
+ ----------------------------------------------
+ Overall 82.644 82.644 82.644 14479 14479
+ Accuracy 82.644 - - - 14479
+
+ Total time: 244961 s, completed Dec 2, 2016 11:08:22 AM
+
+ - [x] **[jTr]** Train aTr jointly without considering the loss explicitly; test using constraints
+
+ Label Precision Recall F1 LCount PCount
+ ----------------------------------------------
+ A0 96.425 93.488 94.934 3578 3469
+ A1 91.531 94.000 92.749 4967 5101
+ A2 67.846 76.173 71.769 1108 1244
+ A3 78.652 40.698 53.640 172 89
+ A4 83.951 66.667 74.317 102 81
+ A5 57.143 80.000 66.667 5 7
+ AA 0.000 0.000 0.000 0 2
+ AM-ADV 68.623 60.079 64.067 506 443
+ AM-CAU 80.952 44.737 57.627 76 42
+ AM-DIR 43.220 60.000 50.246 85 118
+ AM-DIS 81.250 81.250 81.250 320 320
+ AM-EXT 33.333 68.750 44.898 32 66
+ AM-LOC 83.969 30.055 44.266 366 131
+ AM-MNR 51.940 50.000 50.952 348 335
+ AM-MOD 97.340 99.637 98.475 551 564
+ AM-NEG 96.624 99.565 98.073 230 237
+ AM-PNC 22.118 81.739 34.815 115 425
+ AM-PRD 16.667 20.000 18.182 5 6
+ AM-REC 0.000 0.000 0.000 2 6
+ AM-TMP 84.706 70.904 77.193 1117 935
+ C-A0 21.154 61.111 31.429 18 52
+ C-A1 60.067 78.166 67.932 229 298
+ C-A2 0.000 0.000 0.000 5 11
+ C-A3 0.000 0.000 0.000 3 4
+ C-A4 0.000 0.000 0.000 0 3
+ C-A5 0.000 0.000 0.000 0 1
+ C-AM-DIR 0.000 0.000 0.000 0 3
+ C-AM-EXT 0.000 0.000 0.000 0 3
+ C-AM-LOC 0.000 0.000 0.000 0 1
+ C-AM-MNR 0.000 0.000 0.000 0 2
+ C-AM-NEG 0.000 0.000 0.000 0 8
+ C-AM-PNC 0.000 0.000 0.000 0 6
+ C-AM-TMP 0.000 0.000 0.000 0 1
+ C-V 0.000 0.000 0.000 141 31
+ R-A0 82.427 92.925 87.361 212 239
+ R-A1 79.612 62.595 70.085 131 103
+ R-A2 21.875 53.846 31.111 13 32
+ R-A3 0.000 0.000 0.000 1 0
+ R-A4 0.000 0.000 0.000 1 1
+ R-AM-ADV 0.000 0.000 0.000 2 1
+ R-AM-CAU 0.000 0.000 0.000 1 2
+ R-AM-EXT 0.000 0.000 0.000 1 7
+ R-AM-LOC 78.571 68.750 73.333 16 14
+ R-AM-MNR 0.000 0.000 0.000 2 1
+ R-AM-PNC 0.000 0.000 0.000 0 6
+ ----------------------------------------------
+ R-AM-TMP 28.571 44.444 34.783 18 28
+ ----------------------------------------------
+ Overall 83.673 83.673 83.673 14479 14479
+ Accuracy 83.673 - - - 14479
+ Total time: 214836 s, completed Nov 20, 2016 9:15:22 AM
+
- [ ] **[aTrJ]** Train **dTr, eTr, fTr** jointly
* Add constraints gradually and train various models considering subsets of constraints
@@ -586,11 +702,7 @@ In the following lines `Pred.` stands for "Predicate" and `Cand.` stands for "C
- [ ] **[aTsJ]** Test the **cTs** of the second phase for joint models.
-The defaul configuration when running the sprlApp will run only the test for pretrained cTr model while it uses srl global constraints during prediction.
-You can run it from command line by:
-
-```scala
-
-sbt -mem 4000 "project saulExamples" "run-main edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.SRLApps"
-
-```
+The training results of independent models are after 100 iterations of training, however, since joinnTraining is computationally much more
+complex, we stopped the training after 30 iterations.
+The overall results were similar to running independent models after 30 iterations, no improvment observed.
+However, there was a per class variation in results and the results for some of the lables improved.
\ No newline at end of file
diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLApps.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLApps.scala
index b6c81203..9970b93b 100644
--- a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLApps.scala
+++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SemanticRoleLabeling/SRLApps.scala
@@ -8,56 +8,84 @@ package edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling
import java.io.File
-import edu.illinois.cs.cogcomp.core.utilities.configuration.ResourceManager
+import edu.illinois.cs.cogcomp.core.datastructures.ViewNames
import edu.illinois.cs.cogcomp.saul.classifier.{ ClassifierUtils, JointTrainSparseNetwork }
import edu.illinois.cs.cogcomp.saul.util.Logging
import edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.SRLClassifiers._
import edu.illinois.cs.cogcomp.saulexamples.nlp.SemanticRoleLabeling.SRLConstrainedClassifiers.argTypeConstraintClassifier
-object SRLApps extends Logging {
- import SRLConfigurator._
-
- val properties: ResourceManager = {
- // Load the default properties if the user hasn't entered a file as an argument
- //if (args.length == 0) {
- logger.info("Loading default configuration parameters")
- new SRLConfigurator().getDefaultConfig
- //} else {
- // logger.info("Loading parameters from {}", args(0))
- //new SRLConfigurator().getConfig(new ResourceManager(args(0)))
- // }
- }
- val modelDir = properties.getString(MODELS_DIR) +
- File.separator + properties.getString(SRLConfigurator.SRL_MODEL_DIR) + File.separator
- val srlPredictionsFile = properties.getString(SRLConfigurator.SRL_OUTPUT_FILE)
- val runningMode = properties.getBoolean(SRLConfigurator.RUN_MODE)
- val trainingMode = properties.getString(SRLConfigurator.TRAINING_MODE)
+object SRLscalaConfigurator {
+
+ val TREEBANK_HOME = "../saul-examples/src/test/resources/SRLToy/treebank"
+ val PROPBANK_HOME = "../saul-examples/src/test/resources/SRLToy/propbank"
+
+ val TEST_SECTION = 0
+ val TRAIN_SECTION_S = 2
+ val TRAIN_SECTION_E = 21
+
+ val MODELS_DIR = "../models"
+ val USE_CURATOR = false
+
+ // The running mode of the program. Can be "true" for only testing, or "false" for training
+ val TEST_MODE: Boolean = true
+
+ // The training mode for the examples. Can be "pipeline", "joint", "jointLoss" or "other"
+ val TRAINING_MODE = "joint"
+
+ /*********** SRL PROPERTIES ***********/
+ // The (sub)directory to store and retrieve the trained SRL models (to be used with MODELS_DIR)
+ val SRL_MODEL_DIR = "srl"
+ val SRL_JAR_MODEL_PATH = "models"
+
+ // This is used to determine the parse view in SRL experiments (can be ViewNames.GOLD or ViewNames.STANFORD)
+ // For replicating the published experiments this needs to be GOLD
+ val SRL_PARSE_VIEW = ViewNames.PARSE_GOLD
+
+ // A file to store the predictions of the SRL classifier (for argument types only)
+ val SRL_OUTPUT_FILE = "srl-predictions.txt"
+
+ // Whether to use gold predicates (if FALSE, predicateClassifier will be used instead)
+ val SRL_GOLD_PREDICATES = true
+
+ // Whether to use gold argument boundaries (if FALSE, argumentXuIdentifierGivenApredicate will be used instead)
+ val SRL_GOLD_ARG_BOUNDARIES = true
- // Training parameters
- val trainPredicates = properties.getBoolean(SRLConfigurator.SRL_TRAIN_PREDICATES)
- val trainArgIdentifier = properties.getBoolean(SRLConfigurator.SRL_TRAIN_ARG_IDENTIFIERS)
- val trainArgType = properties.getBoolean(SRLConfigurator.SRL_TRAIN_ARG_TYPE)
+ /*Testing parameters*/
- // Testing parameters
- val testWithConstraints = properties.getBoolean(SRLConfigurator.SRL_TEST_CONSTRAINTS)
- val testWithPipeline = properties.getBoolean(SRLConfigurator.SRL_TEST_PIPELINE)
+ // Should we use the pipeline during testing
+ val SRL_TEST_PIPELINE = false
+ // Should we use constraints during testing
+ val SRL_TEST_CONSTRAINTS = false
- val useGoldPredicate = properties.getBoolean(SRLConfigurator.SRL_GOLD_PREDICATES)
- val useGoldBoundaries = properties.getBoolean(SRLConfigurator.SRL_GOLD_ARG_BOUNDARIES)
+ /*Training parameters*/
- val modelJars = properties.getString(SRLConfigurator.SRL_JAR_MODEL_PATH)
+ // Should we train a predicate classifier given predicate candidates
+ val SRL_TRAIN_PREDICATES = false
+ // Should we train an argument identifier given the XuPalmer argument candidates
+ val SRL_TRAIN_ARG_IDENTIFIERS = false
+ // Should we train an argument type classifier
+ val SRL_TRAIN_ARG_TYPE = true
+
+}
+
+object SRLApps extends Logging {
+
+ import SRLscalaConfigurator._
+
+ val modelDir = MODELS_DIR + File.separator + SRL_MODEL_DIR + File.separator
val expName: String = {
- if (trainingMode.equals("other"))
- if (trainArgType && useGoldBoundaries && useGoldPredicate && trainingMode.equals("other")) "aTr"
- else if (trainArgIdentifier && useGoldPredicate && useGoldPredicate) "bTr"
- else if (trainArgType && useGoldPredicate && !useGoldBoundaries) "cTr"
- else if (trainPredicates && useGoldPredicate) "dTr"
- else if (trainArgIdentifier && !useGoldPredicate) "eTr"
- else if (trainArgType && !useGoldPredicate) "fTr"
+ if (TRAINING_MODE.equals("other"))
+ if (SRL_TRAIN_ARG_TYPE && SRL_GOLD_ARG_BOUNDARIES && SRL_GOLD_PREDICATES && TRAINING_MODE.equals("other")) "aTr"
+ else if (SRL_TRAIN_ARG_IDENTIFIERS && SRL_GOLD_PREDICATES && SRL_GOLD_PREDICATES) "bTr"
+ else if (SRL_TRAIN_ARG_TYPE && SRL_GOLD_PREDICATES && !SRL_GOLD_ARG_BOUNDARIES) "cTr"
+ else if (SRL_TRAIN_PREDICATES && SRL_GOLD_PREDICATES) "dTr"
+ else if (SRL_TRAIN_ARG_IDENTIFIERS && !SRL_GOLD_PREDICATES) "eTr"
+ else if (SRL_TRAIN_ARG_TYPE && !SRL_GOLD_PREDICATES) "fTr"
else ""
- else if (trainingMode.equals("pipeline")) "pTr"
- else if (trainingMode.equals("joint")) "jTr"
+ else if (TRAINING_MODE.equals("pipeline")) "pTr"
+ else if (TRAINING_MODE.equals("joint")) "jTr"
+ else if (TRAINING_MODE.equals("jointLoss")) "lTr"
else ""
}
@@ -65,7 +93,7 @@ object SRLApps extends Logging {
logger.info("population starts.")
// Here, the data is loaded into the graph
- val srlDataModelObject = PopulateSRLDataModel(testOnly = runningMode, useGoldPredicate, useGoldBoundaries)
+ val srlDataModelObject = PopulateSRLDataModel(testOnly = TEST_MODE, SRL_GOLD_PREDICATES, SRL_GOLD_ARG_BOUNDARIES)
import srlDataModelObject._
@@ -79,13 +107,14 @@ object SRLApps extends Logging {
object RunningApps extends App with Logging {
import SRLApps._
import SRLApps.srlDataModelObject._
+ import SRLscalaConfigurator._
// TRAINING
- if (!runningMode) {
+ if (!TEST_MODE) {
expName match {
case "aTr" =>
argumentTypeLearner.modelDir = modelDir + expName
- argumentTypeLearner.learn(100, relations.getTrainingInstances)
+ argumentTypeLearner.learn(30, relations.getTrainingInstances)
argumentTypeLearner.test()
argumentTypeLearner.save()
@@ -146,17 +175,26 @@ object RunningApps extends App with Logging {
case "jTr" =>
argumentTypeLearner.modelDir = modelDir + expName
- val outputFile = modelDir + srlPredictionsFile
+ val outputFile = modelDir + SRL_OUTPUT_FILE
logger.info("Global training... ")
- JointTrainSparseNetwork(sentences, argTypeConstraintClassifier :: Nil, 100, true)
+ JointTrainSparseNetwork(sentences, argTypeConstraintClassifier :: Nil, 30, init = true)
+ argumentTypeLearner.save()
+ argTypeConstraintClassifier.test(relations.getTestingInstances, outputFile, 200, exclude = "candidate")
+
+ case "lTr" =>
+ argumentTypeLearner.modelDir = modelDir + expName
+ val outputFile = modelDir + SRL_OUTPUT_FILE
+ logger.info("Global training using loss augmented inference... ")
+ JointTrainSparseNetwork(sentences, argTypeConstraintClassifier :: Nil, 30, init = true, lossAugmented = true)
argumentTypeLearner.save()
argTypeConstraintClassifier.test(relations.getTestingInstances, outputFile, 200, exclude = "candidate")
}
+
}
// TESTING
- if (runningMode) {
- (testWithPipeline, testWithConstraints) match {
+ if (TEST_MODE) {
+ (SRL_TEST_PIPELINE, SRL_TEST_CONSTRAINTS) match {
case (true, true) =>
ClassifierUtils.LoadClassifier(SRLConfigurator.SRL_JAR_MODEL_PATH.value + "/models_bTr/", argumentXuIdentifierGivenApredicate)
diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SpatialRoleLabeling/SpRLDataModel.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SpatialRoleLabeling/SpRLDataModel.scala
index 3167cccc..b84ce932 100644
--- a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SpatialRoleLabeling/SpRLDataModel.scala
+++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/SpatialRoleLabeling/SpRLDataModel.scala
@@ -10,11 +10,10 @@ import edu.illinois.cs.cogcomp.core.datastructures.ViewNames
import edu.illinois.cs.cogcomp.edison.features.factory.WordNetFeatureExtractor
import edu.illinois.cs.cogcomp.saul.datamodel.DataModel
import edu.illinois.cs.cogcomp.saulexamples.nlp.CommonSensors._
-import edu.illinois.cs.cogcomp.saulexamples.nlp.SpatialRoleLabeling.Triplet.{ SpRelation, SpRoleTypes }
import edu.illinois.cs.cogcomp.saulexamples.nlp.SpatialRoleLabeling.SpRLSensors._
+import edu.illinois.cs.cogcomp.saulexamples.nlp.SpatialRoleLabeling.Triplet.{ SpRelation, SpRoleTypes }
import scala.collection.JavaConverters._
-import scala.collection.immutable.HashSet
import scala.collection.mutable.ListBuffer
/** Created by taher on 8/10/16.
@@ -60,13 +59,13 @@ object SpRLDataModel extends DataModel {
val BF6 = property(relations) {
x: SpRelation =>
{
- val t = x.getTrajector.getFirstConstituent.getStartSpan
+ val trStart = x.getTrajector.getFirstConstituent.getStartSpan
val spStart = x.getSpatialIndicator.getFirstConstituent.getStartSpan
val spEnd = x.getSpatialIndicator.getLastConstituent.getStartSpan
- if (t < spStart)
- getDependencyPath(x.getTextAnnotation, t, spStart)
+ if (trStart < spStart)
+ getDependencyPath(x.getTextAnnotation, trStart, spStart)
else
- getDependencyPath(x.getTextAnnotation, t, spEnd)
+ getDependencyPath(x.getTextAnnotation, trStart, spEnd)
}
}
diff --git a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/TextAnnotationFactory.scala b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/TextAnnotationFactory.scala
index 1602d462..2620b50a 100644
--- a/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/TextAnnotationFactory.scala
+++ b/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/TextAnnotationFactory.scala
@@ -9,12 +9,9 @@ package edu.illinois.cs.cogcomp.saulexamples.nlp
import java.util.Properties
import edu.illinois.cs.cogcomp.annotation.AnnotatorService
-import edu.illinois.cs.cogcomp.core.datastructures.ViewNames
import edu.illinois.cs.cogcomp.core.datastructures.textannotation.{ TextAnnotation, TokenLabelView }
import edu.illinois.cs.cogcomp.core.utilities.configuration.{ Configurator, Property, ResourceManager }
import edu.illinois.cs.cogcomp.curator.{ CuratorConfigurator, CuratorFactory }
-import edu.illinois.cs.cogcomp.curator.CuratorConfigurator._
-import edu.illinois.cs.cogcomp.nlp.common.PipelineConfigurator._
import edu.illinois.cs.cogcomp.nlp.pipeline.IllinoisPipelineFactory
/** Created by taher on 7/30/16.
diff --git a/saul-examples/src/test/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/EntityRelation/EntityRelationTests.scala b/saul-examples/src/test/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/EntityRelation/EntityRelationTests.scala
index 298fe7eb..3ce4f58a 100644
--- a/saul-examples/src/test/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/EntityRelation/EntityRelationTests.scala
+++ b/saul-examples/src/test/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/EntityRelation/EntityRelationTests.scala
@@ -92,14 +92,12 @@ class EntityRelationTests extends FlatSpec with Matchers {
ClassifierUtils.TrainClassifiers(1, cls_base)
- PerConstrainedClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner].getNetwork.get(0).asInstanceOf[LinearThresholdUnit].getWeightVector.size() should be(1660)
+ PerConstrainedClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner].getNetwork.get(0).asInstanceOf[LinearThresholdUnit].getWeightVector.size() should be(1654)
- val jointTrainIteration = 1
- JointTrainSparseNetwork.train[ConllRelation](
- pairs, cls, jointTrainIteration, init = true
- )
+ val jointTrainIteration = 2
+ JointTrainSparseNetwork.train[ConllRelation](pairs, cls, jointTrainIteration, init = true)
- PerConstrainedClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner].getNetwork.get(0).asInstanceOf[LinearThresholdUnit].getWeightVector.size() should be(50)
+ PerConstrainedClassifier.onClassifier.classifier.asInstanceOf[SparseNetworkLearner].getNetwork.get(0).asInstanceOf[LinearThresholdUnit].getWeightVector.size() should be(81)
}
}
\ No newline at end of file