Cleaning data

Simple examples about cleaning text data

See source code, output and comments on the scala files:

TextCleanExample.scala

• Word freq
• Special chars, identify and clean
• Word stemmer

StanfordNLPExample.scala

• Natural Language Processing

## notebook/regexs.ipynb, regexs.pdf

• Regexs

RemoveStopWordsExample.scala

• Stop words

PatentNumbersPatterns.scala

• Find patterns in tokens

EPFLPatentsProject.scala

• querying Patstat outside the SQL relational model

Next time

• comparing text, text distance, alignment, disambiguation, google refine…
• regex vs CFG, web scraping, table stats, validation, data curation workflow

Requirements

• Java JDK 8
• SBT >=0.13.12 (build tool for Scala)
• IntelliJ with the Scala plugin (IDE, optional)

How to run a scala example

$ sbt "runMain application.TextCleanExample"
$ sbt "runMain application.StanfordNLPExample"
$ export dbUrl="jdbc:mysql://example.com/patstat_2015a?user=__USER__&password=__PASSWORD__&useSSL=false"
$ sbt "runMain application.RemoveStopWordsExample $dbUrl"
$ sbt "runMain application.PatentNumbersPatterns $dbUrl"
$ sbt "runMain application.EPFLPatentsProject $dbUrl"

How to run jupyther with the regexs example

docker run -it --rm -p 8888:8888 -v $PWD/notebook:/home/jovyan/work jupyter/all-spark-notebook start-notebook.sh

Do you have other use cases or questions?

Contact me at [email protected]