Rater Scoring Modeling Tool

Introduction

Automated scoring of written and spoken test responses is a growing field in educational natural language processing. Automated scoring engines employ machine learning models to predict scores for such responses based on features extracted from the text/audio of these responses. Examples of automated scoring engines include Project Essay Grade for written responses and SpeechRater for spoken responses.

Rater Scoring Modeling Tool (RSMTool) is a python package which automates and combines in a single pipeline multiple analyses that are commonly conducted when building and evaluating such scoring models. The output of RSMTool is a comprehensive, customizable HTML statistical report that contains the output of these multiple analyses. While RSMTool does make it really simple to run a set of standard analyses using a single command, it is also fully customizable and allows users to easily exclude unneeded analyses, modify the default analyses, and even include custom analyses in the report.

We expect the primary users of RSMTool to be researchers working on developing new automated scoring engines or on improving existing ones. Note that RSMTool is not a scoring engine by itself but rather a tool for building and evaluating machine learning models that may be used in such engines.

RSMTool is driven by a configuration file that users have to supply. Given the large number of available options, this can get complicated especially for new users. That's why RSMTool can help users generate configuration files interactively via guided prompts! The video below demonstrates this feature.

Getting Started

To get started with RSMTool, please see the extensive official documentation. If you use the Dash app on macOS, you can also download the complete RSMTool documentation for offline use. Go to the Dash preferences, click on "Downloads", then "User Contributed", and search for "RSMTool".

Requirements

• ipython
• jupyter
• notebook
• numpy<2
• openpyxl
• pandas
• python>=3.8
• scikit-learn
• scipy
• seaborn
• shap
• skll==5.0.1
• statsmodels
• wandb
• xlrd
• xlwt

Contributing

Contributions to RSMTool are very welcome. Please refer to the documentation for how to get started on developing new features or functionality for RSMTool.

Citing

If you are using RSMTool in your work, you can cite it as follows:

MLA

Madnani, Nitin and Loukina, Anastassia. "RSMTool: A Collection of Tools for Building and Evaluating Automated Scoring Models". Journal of Open Source Software 1(3), 2016.

BibTex

@article{MadnaniLoukina2016,
  doi = {10.21105/joss.00033},
  url = {http://dx.doi.org/10.21105/joss.00033},
  year  = {2016},
  month = {jul},
  publisher = {The Open Journal},
  volume = {1},
  number = {3},
  author = {Nitin Madnani and Anastassia Loukina},
  title = {{RSMTool}: A Collection of Tools for Building and Evaluating Automated Scoring Models},
  journal = {{Journal of Open Source Software}}
}

Changelog

See GitHub Releases.