RedStorm Gem v0.4.x Documentation

Dependencies

Tested on OSX 10.6.8 and Linux 10.04 using Storm 0.6.2 and JRuby 1.6.6

Installation

$ gem install redstorm

Usage overview

• create a new empty project directory.
• install the RedStorm gem.
• create a subdirectory which will contain your sources.
• perform the initial setup as described below to install the dependencies in the target/ subdir of your project directory.
• run your topology in local mode and/or on a production cluster as described below.

Initial setup

• install RedStom dependencies; from your project root directory execute:

$ redstorm install

The install command will install all Java jars dependencies using [ruby-maven][ruby-maven] in target/dependency and generate & compile the Java bindings in target/classes

DON'T PANIC it's Maven. The first time you run $ redstorm install Maven will take a few minutes resolving dependencies and in the end will download and install the dependency jar files.

• create a topology class. The underscore topology_class_file_name.rb MUST correspond to its CamelCase class name.

Gems

Until this is better integrated, you can use gems in local mode and on a production cluster:

• local mode: simply install your gems the usual way, they will be picked up when run in local mode.

• production cluster: install your gem in the target/gems folder using:

gem install <the gem> --install-dir target/gems/ --no-ri --no-rdoc

Run in local mode

$ redstorm local <path/to/topology_class_file_name.rb>

See examples below to run examples in local mode or on a production cluster.

Run on production cluster

• generate target/cluster-topology.jar. This jar file will include your sources directory plus the required dependencies from the target/ directory:

$ redstorm jar <sources_directory>

• submit the cluster topology jar file to the cluster. Assuming you have the Storm distribution installed and the Storm bin/ directory in your path:

storm jar ./target/cluster-topology.jar redstorm.TopologyLauncher cluster <path/to/topology_class_file_name.rb>

Basically you must follow the Storm instructions to setup a production cluster and submit your topology to the cluster.

Examples

Install the example files in your project. The examples/ dir will be created in your project root dir.

$ redstorm examples

All examples using the simple DSL are located in examples/simple. Examples using the standard Java interface are in examples/native.

Local mode

$ redstorm local examples/simple/exclamation_topology.rb
$ redstorm local examples/simple/exclamation_topology2.rb
$ redstorm local examples/simple/word_count_topology.rb

This next example requires the use of the Redis Gem and a [Redis][redis] server running on localhost:6379

$ redstorm local examples/simple/redis_word_count_topology.rb

Using redis-cli, push words into the test list and watch Storm pick them up

Production cluster

All examples using the simple DSL can also run on a productions cluster. The only native example compatible with a production cluster is the ClusterWordCountTopology

• genererate the target/cluster-topology.jar and include the examples/ directory.

$ redstorm jar examples

• submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm bin/ directory in your path:

$ storm jar ./target/cluster-topology.jar redstorm.TopologyLauncher cluster examples/simple/word_count_topology.rb

• to run examples/simple/redis_word_count_topology.rb you need a [Redis][redis] server running on localhost:6379 and the Redis gem in target/gems using:

gem install redis --install-dir target/gems/ --no-ri --no-rdoc

• generate jar and submit:

$ redstorm jar examples
$ storm jar ./target/cluster-topology.jar redstorm.TopologyLauncher cluster examples/simple/redis_word_count_topology.rb

• using redis-cli, push words into the test list and watch Storm pick them up

Basically you must follow the Storm instructions to setup a production cluster and submit your topology to the cluster.

DSL

Ruby DSL Documentation

Development

Requirements

• JRuby 1.6.6
• rake gem ~> 0.9.2.2
• ruby-maven gem ~> 3.0.3.0.28.5
• rspec gem ~> 2.8.0

Workflow

• fork project
• create branch
• install dependencies in target/dependencies

$ rake deps

• generate and build Java source into target/classes

$ rake build

• run topology in local dev cluster

$ bin/redstorm local path/to/topology_class.rb

• generate remote cluster topology jar into target/cluster-topology.jar, including the examples/ directory.

$ rake jar['examples']