Retrieve AWS metadata for multi-box environment

[quintona]

Hi, I have an older vagrant file that I used to setup a small cluster environment. I would like to try get this working on AWS now. Here is a snippet of the previous config:

boxes = [
  { :name => :nimbus, :ip => '192.168.33.100', :cpus =>2, :memory => 512 },
  { :name => :supervisor1, :ip => '192.168.33.101', :cpus =>4, :memory => 1024 },
  { :name => :supervisor2, :ip => '192.168.33.102', :cpus =>4, :memory => 1024 },
  { :name => :zookeeper1, :ip => '192.168.33.201', :cpus =>1, :memory => 1024 },
]

Vagrant::Config.run do |config|

  boxes.each do |opts|
    config.vm.define opts[:name] do |config|
...
...

Now I know the providers doesn't support any of the networking properties, however it would be good if you could extract the host names from the meta data and pass them to the provisioners, this way the provisioning scripts could configure the cluster setup correctly. This also implies some level of ordering. In the case above, zookeeper would need to come first, as its hostname would be important for the remaining nodes.

[miguno]

Seems like you want to setup a Storm cluster. :-)

I am currently working on auto-deploying Storm clusters with Vagrant and Puppet -- and I hope I can open source it soon. The tool can already deploy locally (VirtualBox) as well as to AWS.

Here are some pointers to get you started:

• Control the hostnames and FQDNs of machines via e.g. an /etc/rc.local script (runs "hostname", updates /etc/sysconfig/network and /etc/resolv.comf). This is requires so you can set those hostnames in your Puppet manifests (e.g. "zookeeper1" is the first ZK server).
• I opted to use Amazon Route 53 to control DNS so that the machines' hostnames correctly resolve to the seemingly random private IP addresses that EC2 assigns instances after boot. After successful provisioning it takes around 3-4mins until the cluster's DNS entries are available to all machines in my experience. Other approaches include: a) manage your own internal DNS server; b) use Puppet exported resources -- with a Puppet master or PuppetDB -- to dynamically construct /etc/hosts; c) add AWS networking support to vagrant-aws (doh!).
• I am injecting Puppet facts (FACTER_* env variables in /etc/environment via a cloud-config in aws.user_data as well as Vagrant's puppet.facter) to tell Puppet what to do with a machine. That is, I am running a masterless and nodeless (i.e. no use of Puppet's "node" type) Puppet setup. I only care about the hostnames for writing the config files -- they do not influence Puppet's behavior.
• I am using my workaround describe in Failing with EC2 Amazon Linux Images? #72 to ensure that Vagrant can reliably provision on EC2. Hence my aws.user_data is a mime multipart.

Note that you do not need the ZK servers to start before the Storm nodes. Storm is clever enough to retry until it works. One less thing to worry about :-)

Hope that helps!

[quintona]

Hi,

I just opened sourced my implementation of same. Works on virtual box and aws vpc. We should maybe trade notes and create something even better?

Check out my bitbucket account, qanderson.

Sent from my iPhone

On 04/07/2013, at 6:35, "Michael G. Noll" [email protected] wrote:

Seems like you want to setup a Storm cluster. :-)

I am currently working on auto-deploying Storm clusters with Vagrant and Puppet -- and I hope I can open source it soon. The tool can already deploy locally (VirtualBox) as well as to AWS.

Here are some pointers to get you started:

Control the hostnames and FQDNs of machines via e.g. an /etc/rc.local script (runs "hostname", updates /etc/sysconfig/network and /etc/resolv.comf). This is requires so you can set those hostnames in your Puppet manifests (e.g. "zookeeper1" is the first ZK server).

I opted to use Amazon Route 53 to control DNS (or /etc/hosts) so that the machines' hostnames correctly resolve to the seemingly random private IP addresses that EC2 assigns instances after boot. After successful provisioning it takes around 3-4mins until the cluster's DNS entries are available to all machines in my experience. Other approaches are: a) manage your own internal DNS server; b) use Puppet exported resources -- with a Puppet master or PuppetDB -- to dynamically construct /etc/hosts.

I am injecting Puppet facts (FACTER_* env variables in /etc/environment via a cloud-config in aws.user_data as well as Vagrant's puppet.facter) to tell Puppet what to do with a machine. That is, I am running a masterless and nodeless (i.e. no use of Puppet's "node" type) Puppet setup. I only care about the hostnames for writing the config files -- they do not influence Puppet's behavior.

I am using my workaround describe in #72 to ensure that Vagrant can reliably provision on EC2. Hence my aws.user_data is a mime multipart.

Note that you do not need the ZK servers to start before the Storm nodes. Storm is clever enough to retry until it works. One less thing to worry about :-)

Hope that helps!

—
Reply to this email directly or view it on GitHub.

[miguno]

I'll take a look! I noticed you also used fpm to package Storm & Co. And I found already some similarities (and a few differences -- e.g. I am running a nodeless Puppet setup). More than happy to share/collaborate.

[y4roslav]

@miguno I'm looking into your repositories to see how you made multi-machine provisioning at AWS. Can you point me to the right repo?

[miguno]

@Y4Rvik I think your starting point for multi-machine AWS provisioning would be https://github.com/miguno/wirbelsturm.