H2O is an in-memory platform for distributed, scalable machine learning. H2O uses familiar interfaces like R, Python, Scala, Java, JSON and the Flow notebook/web interface, and works seamlessly with big data technologies like Hadoop and Spark. H2O provides implementations of many popular algorithms such as GBM, Random Forest, Deep Neural Networks, Word2Vec and Stacked Ensembles. H2O is extensible so that developers can add data transformations and custom algorithms of their choice and access them through all of those clients.
Data collection is easy. Decision making is hard. H2O makes it fast and easy to derive insights from your data through faster and better predictive modeling. H2O allows online scoring and modeling in a single platform.
H2O-3 (this repository) is the third incarnation of H2O, and the successor to H2O-2.
- Downloading H2O-3
- Open Source Resources
- Using H2O-3 Code Artifacts (libraries)
- Building H2O-3
- Launching H2O after Building
- Building H2O on Hadoop
- Sparkling Water
- Documentation
- Citing H2O
- Community / Advisors / Investors
While most of this README is written for developers who do their own builds, most H2O users just download and use a pre-built version. If you are a Python or R user, the easiest way to install H2O is via PyPI or Anaconda (for Python) or CRAN (for R):
pip install h2oinstall.packages("h2o")For the latest stable, nightly, Hadoop (or Spark / Sparkling Water) releases, or the stand-alone H2O jar, please visit: https://h2o.ai/download
More info on downloading & installing H2O is available in the H2O User Guide.
Most people interact with three or four primary open source resources: GitHub (which you've already found), JIRA (for bug reports and issue tracking), Stack Overflow for H2O code/software-specific questions, and h2ostream (a Google Group / email discussion forum) for questions not suitable for Stack Overflow. There is also a Gitter H2O developer chat group, however for archival purposes & to maximize accessibility, we'd prefer that standard H2O Q&A be conducted on Stack Overflow.
(Note: There is only one issue tracking system for the project. GitHub issues are not enabled; you must use JIRA.)
You can browse and create new issues in our open source JIRA: http://jira.h2o.ai
- You can browse and search for issues without logging in to JIRA:
- Click the
Issuesmenu - Click
Search for issues
- Click the
- To create an issue (either a bug or a feature request), please create yourself an account first:
- Click the
Log Inbutton on the top right of the screen - Click
Create an acccountnear the bottom of the login box - Once you have created an account and logged in, use the
Createbutton on the menu to create an issue - Create H2O-3 issues in the PUBDEV project. (Note: Sparkling Water questions should be filed under the SW project.)
- Click the
- You can also vote for feature requests and/or other issues. Voting can help H2O prioritize the features that are included in each release.
- Go to the H2O JIRA page.
- Click Log In to either log in or create an account if you do not already have one.
- Search for the feature that you want to prioritize, or create a new feature.
- Click on the Vote for this issue link. This is located on the right side of the issue under the People section.
-
GitHub
-
JIRA -- file bug reports / track issues here
- The PUBDEV project contains issues for the current H2O-3 project)
-
Stack Overflow -- ask all code/software questions here
-
Cross Validated (Stack Exchange) -- ask algorithm/theory questions here
-
h2ostream Google Group -- ask non-code related questions here
- Web: https://groups.google.com/d/forum/h2ostream
- Mail to: [email protected]
-
Gitter H2O Developer Chat
-
Documentation
- H2O User Guide (main docs): http://docs.h2o.ai/h2o/latest-stable/h2o-docs/index.html
- All H2O documenation links: http://docs.h2o.ai
- Nightly build page (nightly docs linked in page): https://s3.amazonaws.com/h2o-release/h2o/master/latest.html
-
Download (pre-built packages)
-
Jenkins (H2O build and test system)
-
Website
-
Twitter -- follow us for updates and H2O news!
-
Awesome H2O -- share your H2O-powered creations with us
Every nightly build publishes R, Python, Java, and Scala artifacts to a build-specific repository. In particular, you can find Java artifacts in the maven/repo directory.
Here is an example snippet of a gradle build file using h2o-3 as a dependency. Replace x, y, z, and nnnn with valid numbers.
// h2o-3 dependency information
def h2oBranch = 'master'
def h2oBuildNumber = 'nnnn'
def h2oProjectVersion = "x.y.z.${h2oBuildNumber}"
repositories {
// h2o-3 dependencies
maven {
url "https://s3.amazonaws.com/h2o-release/h2o-3/${h2oBranch}/${h2oBuildNumber}/maven/repo/"
}
}
dependencies {
compile "ai.h2o:h2o-core:${h2oProjectVersion}"
compile "ai.h2o:h2o-algos:${h2oProjectVersion}"
compile "ai.h2o:h2o-web:${h2oProjectVersion}"
compile "ai.h2o:h2o-app:${h2oProjectVersion}"
}
Refer to the latest H2O-3 bleeding edge nightly build page for information about installing nightly build artifacts.
Refer to the h2o-droplets GitHub repository for a working example of how to use Java artifacts with gradle.
Note: Stable H2O-3 artifacts are periodically published to Maven Central (click here to search) but may substantially lag behind H2O-3 Bleeding Edge nightly builds.
Getting started with H2O development requires JDK 1.7, Node.js, and Gradle. We use the Gradle wrapper (called gradlew) to ensure up-to-date local versions of Gradle and other dependencies are installed in your development directory.
To build H2O from the repository, perform the following steps.
# Build H2O
git clone https://github.com/h2oai/h2o-3.git
cd h2o-3
./gradlew build -x test
You may encounter problems: e.g. npm missing. Install it:
brew install npm
# Start H2O
java -jar build/h2o.jar
# Point browser to http://localhost:54321
git clone https://github.com/h2oai/h2o-3.git
cd h2o-3
./gradlew syncSmalldata
./gradlew syncRPackages
./gradlew build
Notes:
- Running tests starts five test JVMs that form an H2O cluster and requires at least 8GB of RAM (preferably 16GB of RAM).
- Running
./gradlew syncRPackagesis supported on Windows, OS X, and Linux, and is strongly recommended but not required../gradlew syncRPackagesensures a complete and consistent environment with pre-approved versions of the packages required for tests and builds. The packages can be installed manually, but we recommend setting an ENV variable and using./gradlew syncRPackages. To set the ENV variable, use the following format (where `${WORKSPACE} can be any path):mkdir -p ${WORKSPACE}/Rlibrary export R_LIBS_USER=${WORKSPACE}/Rlibrary
git pull
./gradlew syncSmalldata
./gradlew syncRPackages
./gradlew clean
./gradlew build
-
We recommend using
./gradlew cleanafter eachgit pull. -
Skip tests by adding
-x testat the end the gradle build command line. Tests typically run for 7-10 minutes on a Macbook Pro laptop with 4 CPUs (8 hyperthreads) and 16 GB of RAM. -
Syncing smalldata is not required after each pull, but if tests fail due to missing data files, then try
./gradlew syncSmalldataas the first troubleshooting step. Syncing smalldata downloads data files from AWS S3 to the smalldata directory in your workspace. The sync is incremental. Do not check in these files. The smalldata directory is in .gitignore. If you do not run any tests, you do not need the smalldata directory. -
Running
./gradlew syncRPackagesis supported on Windows, OS X, and Linux, and is strongly recommended but not required../gradlew syncRPackagesensures a complete and consistent environment with pre-approved versions of the packages required for tests and builds. The packages can be installed manually, but we recommend setting an ENV variable and using./gradlew syncRPackages. To set the ENV variable, use the following format (where `${WORKSPACE} can be any path):mkdir -p ${WORKSPACE}/Rlibrary export R_LIBS_USER=${WORKSPACE}/Rlibrary
./gradlew clean && ./gradlew build -x test && (export DO_FAST=1; ./gradlew dist)
open target/docs-website/h2o-docs/index.html
Note: The following instructions assume you have installed the latest version of pip, which is installed with the latest version of Python.
pip install grip
pip install tabulate
pip install wheel
pip install scikit-learn
Python tests require:
pip install scikit-learn
pip install numpy
pip install scipy
pip install pandas
pip install statsmodels
pip install patsy
pip install future
Step 1: Download and install WinPython.
From the command line, validate python is using the newly installed package by using which python (or sudo which python). Update the Environment variable with the WinPython path.
pip install grip
pip install tabulate
pip install wheel
Install Java 1.7 and add the appropriate directory C:\Program Files\Java\jdk1.7.0_65\bin with java.exe to PATH in Environment Variables. To make sure the command prompt is detecting the correct Java version, run:
javac -version
The CLASSPATH variable also needs to be set to the lib subfolder of the JDK:
CLASSPATH=/<path>/<to>/<jdk>/lib
Install Node.js and add the installed directory C:\Program Files\nodejs, which must include node.exe and npm.cmd to PATH if not already prepended.
Install R and add the bin directory to your PATH if not already included.
Install the following R packages:
To install these packages from within an R session:
pkgs <- c("RCurl", "jsonlite", "statmod", "devtools", "roxygen2", "testthat")
for (pkg in pkgs) {
if (! (pkg %in% rownames(installed.packages()))) install.packages(pkg)
}Note that libcurl is required for installation of the RCurl R package.
Finally, install Rtools, which is a collection of command line tools to facilitate R development on Windows.
NOTE: During Rtools installation, do not install Cygwin.dll.
Step 6. Install Cygwin
NOTE: During installation of Cygwin, deselect the Python packages to avoid a conflict with the Python.org package.
If Cygwin is already installed, remove the Python packages or ensure that Native Python is before Cygwin in the PATH variable.
Step 8. Git Clone h2o-3
If you don't already have a Git client, please install one. The default one can be found here http://git-scm.com/downloads. Make sure that command prompt support is enabled before the installation.
Download and update h2o-3 source codes:
git clone https://github.com/h2oai/h2o-3
cd h2o-3
./gradlew.bat build
If you encounter errors run again with
--stacktracefor more instructions on missing dependencies.
If you don't have Homebrew, we recommend installing it. It makes package management for OS X easy.
Install Java 1.7. To make sure the command prompt is detecting the correct Java version, run:
javac -version
Using Homebrew:
brew install node
Otherwise, install from the NodeJS website.
Install R and add the bin directory to your PATH if not already included.
Install the following R packages:
To install these packages from within an R session:
pkgs <- c("RCurl", "jsonlite", "statmod", "devtools", "roxygen2", "testthat")
for (pkg in pkgs) {
if (! (pkg %in% rownames(installed.packages()))) install.packages(pkg)
}Note that libcurl is required for installation of the RCurl R package.
Step 4. Git Clone h2o-3
OS X should already have Git installed. To download and update h2o-3 source codes:
git clone https://github.com/h2oai/h2o-3
cd h2o-3
./gradlew build
If you encounter errors run again with
--stacktracefor more instructions on missing dependencies.
curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -
sudo apt-get install -y nodejs
Install Java 1.7. Installation instructions can be found here JDK installation. To make sure the command prompt is detecting the correct Java version, run:
javac -version
Installation instructions can be found here R installation. Click “Download R for Linux”. Click “ubuntu”. Follow the given instructions.
To install the required packages, follow the same instructions as for OS X above.
Note: If the process fails to install RStudio Server on Linux, run one of the following:
sudo apt-get install libcurl4-openssl-devor
sudo apt-get install libcurl4-gnutls-dev
Step 4. Git Clone h2o-3
If you don't already have a Git client:
sudo apt-get install git
Download and update h2o-3 source codes:
git clone https://github.com/h2oai/h2o-3
cd h2o-3
./gradlew build
If you encounter errors, run again using
--stacktracefor more instructions on missing dependencies.
Make sure that you are not running as root, since
bowerwill reject such a run.
curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -
sudo apt-get install -y nodejs
For users of Intellij's IDEA, generate project files with:
./gradlew idea
For users of Eclipse, generate project files with:
./gradlew eclipse
cd /opt
sudo wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.tar.gz"
sudo tar xzf jdk-7u79-linux-x64.tar.gz
cd jdk1.7.0_79
sudo alternatives --install /usr/bin/java java /opt/jdk1.7.0_79/bin/java 2
sudo alternatives --install /usr/bin/jar jar /opt/jdk1.7.0_79/bin/jar 2
sudo alternatives --install /usr/bin/javac javac /opt/jdk1.7.0_79/bin/javac 2
sudo alternatives --set jar /opt/jdk1.7.0_79/bin/jar
sudo alternatives --set javac /opt/jdk1.7.0_79/bin/javac
cd /opt
sudo wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
sudo rpm -ivh epel-release-7-5.noarch.rpm
sudo echo "multilib_policy=best" >> /etc/yum.conf
sudo yum -y update
sudo yum -y install R R-devel git python-pip openssl-devel libxml2-devel libcurl-devel gcc gcc-c++ make openssl-devel kernel-devel texlive texinfo texlive-latex-fonts libX11-devel mesa-libGL-devel mesa-libGL nodejs npm python-devel numpy scipy python-pandas
sudo pip install scikit-learn grip tabulate statsmodels wheel
mkdir ~/Rlibrary
export JAVA_HOME=/opt/jdk1.7.0_79
export JRE_HOME=/opt/jdk1.7.0_79/jre
export PATH=$PATH:/opt/jdk1.7.0_79/bin:/opt/jdk1.7.0_79/jre/bin
export R_LIBS_USER=~/Rlibrary
# install local R packages
R -e 'install.packages(c("RCurl","jsonlite","statmod","devtools","roxygen2","testthat"), dependencies=TRUE, repos="http://cran.rstudio.com/")'
cd
git clone https://github.com/h2oai/h2o-3.git
cd h2o-3
# Build H2O
./gradlew syncSmalldata
./gradlew syncRPackages
./gradlew build -x test
To start the H2O cluster locally, execute the following on the command line:
java -jar build/h2o.jar
A list of available start-up JVM and H2O options (e.g. -Xmx, -nthreads, -ip), is available in the H2O User Guide.
Pre-built H2O-on-Hadoop zip files are available on the download page. Each Hadoop distribution version has a separate zip file in h2o-3.
To build H2O with Hadoop support yourself, first install sphinx for python: pip install sphinx
Then start the build by entering the following from the top-level h2o-3 directory:
(export BUILD_HADOOP=1; ./gradlew build -x test)
./gradlew dist
This will create a directory called 'target' and generate zip files there. Note that BUILD_HADOOP is the default behavior when the username is jenkins (refer to settings.gradle); otherwise you have to request it, as shown above.
In the h2o-hadoop directory, each Hadoop version has a build directory for the driver and an assembly directory for the fatjar.
You need to:
- Add a new driver directory and assembly directory (each with a
build.gradlefile) inh2o-hadoop - Add these new projects to
h2o-3/settings.gradle - Add the new Hadoop version to
HADOOP_VERSIONSinmake-dist.sh - Add the new Hadoop version to the list in
h2o-dist/buildinfo.json
These are the required steps to debug HDFS in IDEA as a standalone H2O process.
Debugging H2O on Hadoop as a hadoop jar hadoop mapreduce job is a difficult thing to do. However, what you can do relatively easily is tweak the gradle settings for the project so that H2OApp has HDFS as a dependency. Here are the steps:
- Make the following changes to gradle build files below
- Change the
hadoop-clientversion inh2o-persist-hdfsto the desired version - Add
h2o-persist-hdfsas a dependency toh2o-app
- Change the
- Close IDEA
./gradlew cleanIdea./gradlew idea- Re-open IDEA
- Run or debug H2OApp, and you will now be able to read from HDFS inside the IDE debugger
h2o-persist-hdfs is normally only a dependency of the assembly modules, since those are not used by any downstream modules. We want the final module to define its own version of HDFS if any is desired.
Note this example is for MapR 4, which requires the additional org.json dependency to work properly.
$ git diff
diff --git a/h2o-app/build.gradle b/h2o-app/build.gradle
index af3b929..097af85 100644
--- a/h2o-app/build.gradle
+++ b/h2o-app/build.gradle
@@ -8,5 +8,6 @@ dependencies {
compile project(":h2o-algos")
compile project(":h2o-core")
compile project(":h2o-genmodel")
+ compile project(":h2o-persist-hdfs")
}
diff --git a/h2o-persist-hdfs/build.gradle b/h2o-persist-hdfs/build.gradle
index 41b96b2..6368ea9 100644
--- a/h2o-persist-hdfs/build.gradle
+++ b/h2o-persist-hdfs/build.gradle
@@ -2,5 +2,6 @@ description = "H2O Persist HDFS"
dependencies {
compile project(":h2o-core")
- compile("org.apache.hadoop:hadoop-client:2.0.0-cdh4.3.0")
+ compile("org.apache.hadoop:hadoop-client:2.4.1-mapr-1408")
+ compile("org.json:org.json:chargebee-1.0")
}
Sparkling Water combines two open-source technologies: Apache Spark and the H2O Machine Learning platform. It makes H2O’s library of advanced algorithms, including Deep Learning, GLM, GBM, K-Means, and Distributed Random Forest, accessible from Spark workflows. Spark users can select the best features from either platform to meet their Machine Learning needs. Users can combine Spark's RDD API and Spark MLLib with H2O’s machine learning algorithms, or use H2O independently of Spark for the model building process and post-process the results in Spark.
Sparkling Water Resources:
- Download page for pre-built packages
- Sparkling Water GitHub repository
- README
- Developer documentation
The main H2O documentation is the H2O User Guide. Visit http://docs.h2o.ai for the top-level introduction to documentation on H2O projects.
To generate the REST API documentation, use the following commands:
cd ~/h2o-3
cd py
python ./generate_rest_api_docs.py # to generate Markdown only
python ./generate_rest_api_docs.py --generate_html --github_user GITHUB_USER --github_password GITHUB_PASSWORD # to generate Markdown and HTML
The default location for the generated documentation is build/docs/REST.
If the build fails, try gradlew clean, then git clean -f.
Documentation for each bleeding edge nightly build is available on the nightly build page.
If you use H2O as part of your workflow in a publication, please cite your H2O resource(s) using the following BibTex entry:
@Manual{h2o_package_or_module,
title = {package_or_module_title},
author = {H2O.ai},
year = {year},
month = {month},
note = {version_information},
url = {resource_url},
}
Formatted H2O Software citation examples:
- H2O.ai (Oct. 2016). Python Interface for H2O, Python module version 3.10.0.8. https://github.com/h2oai/h2o-3.
- H2O.ai (Oct. 2016). R Interface for H2O, R package version 3.10.0.8. https://github.com/h2oai/h2o-3.
- H2O.ai (Oct. 2016). H2O, H2O version 3.10.0.8. https://github.com/h2oai/h2o-3.
H2O algorithm booklets are available at the Documentation Homepage.
@Manual{h2o_booklet_name,
title = {booklet_title},
author = {list_of_authors},
year = {year},
month = {month},
url = {link_url},
}
Formatted booklet citation examples:
Arora, A., Candel, A., Lanford, J., LeDell, E., and Parmar, V. (Oct. 2016). Deep Learning with H2O. http://docs.h2o.ai/h2o/latest-stable/h2o-docs/booklets/DeepLearningBooklet.pdf.
Click, C., Lanford, J., Malohlava, M., Parmar, V., and Roark, H. (Oct. 2016). Gradient Boosted Models with H2O. http://docs.h2o.ai/h2o/latest-stable/h2o-docs/booklets/GBMBooklet.pdf.
H2O has been built by a great many number of contributors over the years both within H2O.ai (the company) and the greater open source community. You can begin to contribute to H2O by answering Stack Overflow questions or filing bug reports. Please join us!
SriSatish Ambati
Cliff Click
Tom Kraljevic
Tomas Nykodym
Michal Malohlava
Kevin Normoyle
Spencer Aiello
Anqi Fu
Nidhi Mehta
Arno Candel
Josephine Wang
Amy Wang
Max Schloemer
Ray Peck
Prithvi Prabhu
Brandon Hill
Jeff Gambera
Ariel Rao
Viraj Parmar
Kendall Harris
Anand Avati
Jessica Lanford
Alex Tellez
Allison Washburn
Amy Wang
Erik Eckstrand
Neeraja Madabhushi
Sebastian Vidrio
Ben Sabrin
Matt Dowle
Mark Landry
Erin LeDell
Oleg Rogynskyy
Nick Martin
Nancy Jordan
Nishant Kalonia
Nadine Hussami
Jeff Cramer
Stacie Spreitzer
Vinod Iyengar
Charlene Windom
Parag Sanghavi
Navdeep Gill
Lauren DiPerna
Anmol Bal
Mark Chan
Nick Karpov
Avni Wadhwa
Ashrith Barthur
Karen Hayrapetyan
Jo-fai Chow
Dmitry Larko
Branden Murray
Jakub Hava
Wen Phan
Magnus Stensmo
Pasha Stetsenko
Angela Bartz
Mateusz Dymczyk
Micah Stubbs
Ivy Wang
Terone Ward
Leland Wilkinson
Wendy Wong
Scientific Advisory Council
Stephen Boyd
Rob Tibshirani
Trevor Hastie
Systems, Data, FileSystems and Hadoop
Doug Lea
Chris Pouliot
Dhruba Borthakur
Jishnu Bhattacharjee, Nexus Venture Partners
Anand Babu Periasamy
Anand Rajaraman
Ash Bhardwaj
Rakesh Mathur
Michael Marks
Egbert Bierman
Rajesh Ambati