StatFSMPaperSite

This page explains how we obtained the data presented in our StatFSM paper and how to replicate our experiments.

Random Automata

The random automata we used in our evaluations can be found here. Each folder in the archive has the form 'x_y' where 'x' corresponds to the 'l' parameter mentioned in the paper and '0.y' corresponds to the gamma parameter. There are five automata in every folder, written in the normal Alto format for tree automata. We computed the evaluation stats on each of them, but the general trends in the data were the same.

The shell script and configuration files for generating the automata can be found here. The configuration files have the following fields:

• folder - where to put the random automata once they have been generated
• fileNamePrefix - what to name the files (for the final file the number of the generated automaton, plus the file ending .auto will be added, files are always just overwritten)
• size - the l parameter from the paper
• toGenerate - how many automata the program should generate
• seed - the random number seed that should be used
• alpha - the gamma parameter from the paper

Results should be the same whenever the program is run with the same input parameters, but there might be variations in the way that the random number generator works depending on the plattform.

The original automata were build with the jar with dependencies from this version of the alto code. The jar is put in the same folder as the script and the config files, then the script is executed. The java main for generating random automata is in de.up.ling.irtg.script.CreateRandomAutomata, but that code really only reads the config file and then calls the methods in de.up.ling.irtg.random_automata.CreateRandomAutomata which are documented to some extent in the code.

Convergence Experiments

The data for the convergence analysis in the paper can be found here. The folder contains Veusz files used to generate the plots from the paper. The Veusz files plot the data for the first automaton (the one with the subscript _0) except for the experiments with l = 30, there it plots the second automaton (the one with the subscript _1). This must have been an error I made when importing the data. I re-checked the plots for the _0 files and at least I cannot see a substantial difference. This error exists in the workshop version of the paper. I have corrected it for the revised version. The workshop version also uses the graph for l=20, y=0.5 with 500 samples twice instead of using the graph with 2000 samples. This is also corrected in the revised version.

The folders of measurements start their name with x_y where x is the l parameter and 0.y is the gamma parameter from the paper. If there is no additional _z number after the y, then the data was generated with 500 samples per round. Otherwise we used 2000 samples.

The measurements are in a files of semicolon-separated values. Each column contains all the measurements made for a single round of adaption over all repetitions of the experiment and each row contains all the measurements made for one repetition over all its adaption rounds.

The shell script and configuration files for generating the measurements data can be found here. The configuration files for each experiment contain the following parameters:

• automataFolder - where to find the random automata on which evaluation will be done
• baseLearningRate - the base learning rate for the Adagrad learning rate scheme
• resultFolder - the folder in which to store the experimental results
• resultPrefix - a prefix for the files in which the experiment results will be stored; The files will have the name PREFIX_AUTOMATONNAME.stats and any existing files will be overwritten
• normalizationExponent - in the paper we use the euclidean norm, which corresponds to exponent 2; But you can use 1 to get l_1 normalization, or any other integer greater than 2 to get closer to the maximum norm
• normalizationDivisor - the lambda parameter from the paper
• populationSize - how many samples are drawn in each adaption round
• rounds - how many adaption rounds will be done
• seed - the random number seed
• repetitions - how many times the experiment should be repeated

Note that the results should be the same for two experiments if all input values are the same, but there may be variations in the implementation of the random number generator between platforms.

The java class that executes the experiments is de.up.ling.irtg.script.EvaluateSamplingFromRules which mostly just reads ín parameters, runs a configuration for the adaptive sampler and then evaluates the quality of the approximation of the uniform weighting function.