Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Odd shard distribution in Elasticsearch 1.4.2 #9023

Closed
vichargrave opened this issue Dec 22, 2014 · 46 comments · Fixed by #9149
Closed

Odd shard distribution in Elasticsearch 1.4.2 #9023

vichargrave opened this issue Dec 22, 2014 · 46 comments · Fixed by #9149

Comments

@vichargrave
Copy link

I'm using a 4 node Elasticsearch cluster to ingest tweets from one of the public Twitter streams. After upgrading my from 1.3.2 to 1.4.2 I'm noticing an odd behavior with respect to how shards are distributed across the nodes. Whereas with 1.3.2 the 5 primary shards and 1 replica set of shards were evenly distributed across all 4 nodes, now I'm seeing this behavior:

  1. All the primary shards are saved to one node
  2. The replicas are split between two other nodes
  3. The fourth node receives no shards at all

I'm using default values for these settings:

cluster.routing.allocation.balance.shard
cluster.routing.allocation.balance.index
cluster.routing.allocation.balance.primary
cluster.routing.allocation.balance.threshold

However I've set cluster.routing.allocation.balance.shard to 0.6f and cluster.routing.allocation.balance.primary to 0.06f in the hope that these settings will coerce Elasticsearch to distribute the shards more evenly.

Is this issue a bug?

@clintongormley
Copy link
Contributor

Hi @vichargrave

What other settings do you have?

@vichargrave
Copy link
Author

Hi Clinton.

I've enclosed the 3 config files I'm using. Note that I observed the
problem with the default values of the following fields:

cluster.routing.allocation.balance.shard: 1.0f
cluster.routing.allocation.balance.primary: 0.50f

However, the problem persists after applying these settings.

On Mon, Dec 22, 2014 at 5:10 AM, Clinton Gormley [email protected]
wrote:

Hi @vichargrave https://github.com/vichargrave

What other settings do you have?


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@clintongormley
Copy link
Contributor

Hi @vichargrave

I assume that you attached them to a reply email? They don't show up on github. While you're about it, please can you also provide:

GET /_cluster/settings
GET /_settings

thanks

@vichargrave
Copy link
Author

Sorry about that. OK here are the contents of my 3 configuration files and the results of the commands you requested:

==== /etc/elasticsearch/elasticsearch.yml
######################### Elasticsearch Configuration #########################
# This file contains an overview of various configuration settings,
# targeted at operations staff. Application developers should
# consult the guide at .
#
# The installation procedure is covered at
# .
#
# ElasticSearch comes with reasonable defaults for most settings,
# so you can try it out without bothering with configuration.
#
# Most of the time, these defaults are just fine for running a production
# cluster. If you're fine-tuning your cluster, or wondering about the
# effect of certain configuration option, please _do ask_ on the
# mailing list or IRC channel [http://elasticsearch.org/community].
# Any element in the configuration can be replaced with environment variables
# by placing them in ${...} notation. For example:
#
# node.rack: ${RACK_ENV_VAR}
# For information on supported formats and syntax for the config file, see
# 
################################### Cluster ###################################
# Cluster name identifies your cluster for auto-discovery. If you're running
# multiple clusters on the same network, make sure you're using unique names.
#
cluster.name: twitter_filter
#################################### Node #####################################
# Node names are generated dynamically on startup, so you're relieved
# from configuring them manually. You can tie this node to a specific name:
#
node.name: "r5-9-37"
# Every node can be configured to allow or deny being eligible as the master,
# and to allow or deny to store the data.
#
# Allow this node to be eligible as a master node (enabled by default):
#
# node.master: true
#
# Allow this node to store data (enabled by default):
#
# node.data: true
# You can exploit these settings to design advanced cluster topologies.
#
# 1. You want this node to never become a master node, only to hold data.
#    This will be the "workhorse" of your cluster.
#
# node.master: false
# node.data: true
#
# 2. You want this node to only serve as a master: to not store any data and
#    to have free resources. This will be the "coordinator" of your cluster.
#
# node.master: true
# node.data: false
#
# 3. You want this node to be neither master nor data node, but
#    to act as a "search load balancer" (fetching data from nodes,
#    aggregating results, etc.)
#
# node.master: false
# node.data: false
# Use the Cluster Health API [http://localhost:9200/_cluster/health], the
# Node Info API [http://localhost:9200/_cluster/nodes] or GUI tools
# such as  and
#  to inspect the cluster state.
# A node can have generic attributes associated with it, which can later be used
# for customized shard allocation filtering, or allocation awareness. An attribute
# is a simple key value pair, similar to node.key: value, here is an example:
#
# node.rack: rack314
# By default, multiple nodes are allowed to start from the same installation location
# to disable it, set the following:
# node.max_local_storage_nodes: 1
#################################### Index ####################################
# You can set a number of options (such as shard/replica options, mapping
# or analyzer definitions, translog settings, ...) for indices globally,
# in this file.
#
# Note, that it makes more sense to configure index settings specifically for
# a certain index, either when creating it or by using the index templates API.
#
# See  and
# 
# for more information.
# Set the number of shards (splits) of an index (5 by default):
#
index.number_of_shards: 5
# Set the number of replicas (additional copies) of an index (1 by default):
#
index.number_of_replicas: 1
# Note, that for development on a local machine, with small indices, it usually
# makes sense to "disable" the distributed features:
#
# index.number_of_shards: 1
# index.number_of_replicas: 0
# These settings directly affect the performance of index and search operations
# in your cluster. Assuming you have enough machines to hold shards and
# replicas, the rule of thumb is:
#
# 1. Having more *shards* enhances the _indexing_ performance and allows to
#    _distribute_ a big index across machines.
# 2. Having more *replicas* enhances the _search_ performance and improves the
#    cluster _availability_.
#
# The "number_of_shards" is a one-time setting for an index.
#
# The "number_of_replicas" can be increased or decreased anytime,
# by using the Index Update Settings API.
#
# ElasticSearch takes care about load balancing, relocating, gathering the
# results from nodes, etc. Experiment with different settings to fine-tune
# your setup.
# Use the Index Status API () to inspect
# the index status.
#################################### Paths ####################################
# Path to directory containing configuration (this file and logging.yml):
#
# path.conf: /path/to/conf
# Path to directory where to store index data allocated for this node.
#
path.data: /data/0,/data/1
#
# Can optionally include more than one location, causing data to be striped across
# the locations (a la RAID 0) on a file level, favouring locations with most free
# space on creation. For example:
#
# path.data: /path/to/data1,/path/to/data2
# Path to temporary files:
#
# path.work: /path/to/work
# Path to log files:
#
# path.logs: /path/to/logs
# Path to where plugins are installed:
#
# path.plugins: /path/to/plugins
#################################### Plugin ###################################
# If a plugin listed here is not installed for current node, the node will not start.
#
# plugin.mandatory: mapper-attachments,lang-groovy
################################### Memory ####################################
# ElasticSearch performs poorly when JVM starts swapping: you should ensure that
# it _never_ swaps.
#
# Set this property to true to lock the memory:
#
bootstrap.mlockall: true
# Make sure that the ES_MIN_MEM and ES_MAX_MEM environment variables are set
# to the same value, and that the machine has enough memory to allocate
# for ElasticSearch, leaving enough memory for the operating system itself.
#
# You should also make sure that the ElasticSearch process is allowed to lock
# the memory, eg. by using `ulimit -l unlimited`.
############################## Network And HTTP ###############################
# ElasticSearch, by default, binds itself to the 0.0.0.0 address, and listens
# on port [9200-9300] for HTTP traffic and on port [9300-9400] for node-to-node
# communication. (the range means that if the port is busy, it will automatically
# try the next port).
# Set the bind address specifically (IPv4 or IPv6):
#
# network.bind_host: 192.168.0.1
# Set the address other nodes will use to communicate with this node. If not
# set, it is automatically derived. It must point to an actual IP address.
#
# network.publish_host: 192.168.0.1
# Set both 'bind_host' and 'publish_host':
#
# network.host: 192.168.0.1
# Set a custom port for the node to node communication (9300 by default):
#
# transport.tcp.port: 9300
# Enable compression for all communication between nodes (disabled by default):
#
# transport.tcp.compress: true
# Set a custom port to listen for HTTP traffic:
#
# http.port: 9200
# Set a custom allowed content length:
#
# http.max_content_length: 100mb
# Disable HTTP completely:
#
# http.enabled: false
################################### Gateway ###################################
# The gateway allows for persisting the cluster state between full cluster
# restarts. Every change to the state (such as adding an index) will be stored
# in the gateway, and when the cluster starts up for the first time,
# it will read its state from the gateway.
# There are several types of gateway implementations. For more information, see
# .
# The default gateway type is the "local" gateway (recommended):
#
# gateway.type: local
# Settings below control how and when to start the initial recovery process on
# a full cluster restart (to reuse as much local data as possible when using shared
# gateway).
# Allow recovery process after N nodes in a cluster are up:
#
# gateway.recover_after_nodes: 1
# Set the timeout to initiate the recovery process, once the N nodes
# from previous setting are up (accepts time value):
#
# gateway.recover_after_time: 5m
# Set how many nodes are expected in this cluster. Once these N nodes
# are up (and recover_after_nodes is met), begin recovery process immediately
# (without waiting for recover_after_time to expire):
#
# gateway.expected_nodes: 2
############################# Recovery Throttling #############################
# These settings allow to control the process of shards allocation between
# nodes during initial recovery, replica allocation, rebalancing,
# or when adding and removing nodes.
# Set the number of concurrent recoveries happening on a node:
#
# 1. During the initial recovery
#
# cluster.routing.allocation.node_initial_primaries_recoveries: 4
#
# 2. During adding/removing nodes, rebalancing, etc
#
# cluster.routing.allocation.node_concurrent_recoveries: 2
# Set to throttle throughput when recovering (eg. 100mb, by default 20mb):
#
# indices.recovery.max_bytes_per_sec: 20mb
# Set to limit the number of open concurrent streams when
# recovering a shard from a peer:
#
# indices.recovery.concurrent_streams: 5
################################## Discovery ##################################
# Discovery infrastructure ensures nodes can be found within a cluster
# and master node is elected. Multicast discovery is the default.
# Set to ensure a node sees N other master eligible nodes to be considered
# operational within the cluster. Its recommended to set it to a higher value
# than 1 when running more than 2 nodes in the cluster.
#
discovery.zen.minimum_master_nodes: 3
# Set the time to wait for ping responses from other nodes when discovering.
# Set this option to a higher value on a slow or congested network
# to minimize discovery failures:
#
discovery.zen.ping.timeout: 15s
# For more information, see
# 
# Unicast discovery allows to explicitly control which nodes will be used
# to discover the cluster. It can be used when multicast is not present,
# or to restrict the cluster communication-wise.
#
# 1. Disable multicast discovery (enabled by default):
#
discovery.zen.ping.multicast.enabled: false
#
# 2. Configure an initial list of master nodes in the cluster
#    to perform discovery when new nodes (master or data) are started:
#
# Note these are not the real IP addresses just placeholders
discovery.zen.ping.unicast.hosts: ["10.0.0.1","10.0.0.2","10.0.0.3","10.0.0.4"]
# EC2 discovery allows to use AWS EC2 API in order to perform discovery.
#
# You have to install the cloud-aws plugin for enabling the EC2 discovery.
#
# For more information, see
# 
#
# See 
# for a step-by-step tutorial.
################################## Slow Log ##################################
# Shard level query and fetch threshold logging.
#index.search.slowlog.threshold.query.warn: 10s
#index.search.slowlog.threshold.query.info: 5s
#index.search.slowlog.threshold.query.debug: 2s
#index.search.slowlog.threshold.query.trace: 500ms
#index.search.slowlog.threshold.fetch.warn: 1s
#index.search.slowlog.threshold.fetch.info: 800ms
#index.search.slowlog.threshold.fetch.debug: 500ms
#index.search.slowlog.threshold.fetch.trace: 200ms
#index.indexing.slowlog.threshold.index.warn: 10s
#index.indexing.slowlog.threshold.index.info: 5s
#index.indexing.slowlog.threshold.index.debug: 2s
#index.indexing.slowlog.threshold.index.trace: 500ms
################################## GC Logging ################################
#monitor.jvm.gc.ParNew.warn: 1000ms
#monitor.jvm.gc.ParNew.info: 700ms
#monitor.jvm.gc.ParNew.debug: 400ms
#monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s
#monitor.jvm.gc.ConcurrentMarkSweep.info: 5s
#monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s
############################ Lhotsky optimizations ##########################
## Threadpool Settings ##
# Search pool
threadpool.search.type: fixed
threadpool.search.size: 20
threadpool.search.queue_size: 100
# Bulk pool
threadpool.bulk.type: fixed
threadpool.bulk.size: 60
threadpool.bulk.queue_size: 300
# Index pool
threadpool.index.type: fixed
threadpool.index.size: 20
threadpool.index.queue_size: 100
## Index Settings ##
# Indices settings
indices.memory.min_shard_index_buffer_size: 12mb
indices.memory.min_index_buffer_size: 96mb
# Cache Sizes
indices.fielddata.cache.size: 15%
indices.cache.filter.size: 15%
# Indexing Settings for Writes
index.refresh_interval: 30s
index.translog.flush_threshold_ops: 50000
############################ Other optimizations ##########################
## See: http://www.elasticsearch.org/blog/performance-considerations-elasticsearch-indexing/
# Thread settings
index.merge.scheduler.max_thread_count: 1
# Indices settings
indices.memory.index_buffer_size: 15%
# Indexing Settings for Writes
index.translog.flush_threshold_size: 1gb
## See: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-update-settings.html
# Cluster balancing #
cluster.routing.allocation.balance.shard: 1.0f
cluster.routing.allocation.balance.primary: 0.50f
==== /etc/sysconfig/elasticsearch
# Directory where the Elasticsearch binary distribution resides
ES_HOME=/usr/share/elasticsearch
# Heap Size (defaults to 256m min, 1g max)
ES_HEAP_SIZE=16g
# Heap new generation
#ES_HEAP_NEWSIZE=
# max direct memory
#ES_DIRECT_SIZE=
# Additional Java OPTS
#ES_JAVA_OPTS=
# Maximum number of open files
MAX_OPEN_FILES=131072
# Maximum amount of locked memory
#MAX_LOCKED_MEMORY=
# Maximum number of VMA (Virtual Memory Areas) a process can own
MAX_MAP_COUNT=262144
# Elasticsearch log directory
LOG_DIR=/var/log/elasticsearch
# Elasticsearch data directory
DATA_DIR=/var/lib/elasticsearch
# Elasticsearch work directory
WORK_DIR=/tmp/elasticsearch
# Elasticsearch conf directory
CONF_DIR=/etc/elasticsearch
# Elasticsearch configuration file (elasticsearch.yml)
CONF_FILE=/etc/elasticsearch/elasticsearch.yml
# User to run as, change this to a specific elasticsearch user if possible
# Also make sure, this user can write into the log directories in case you change them
# This setting only works for the init script, but has to be configured separately for systemd startup
ES_USER=elasticsearch
# Configure restart on package upgrade (true, every other setting will lead to not restarting)
#RESTART_ON_UPGRADE=true
==== /etc/security/limits.conf
# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#            
#
#Where:
# can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
# can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
# can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open files
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#
#                 
#
#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4
# Ensure ElasticSearch can open files and lock memory!
elasticsearch   soft    nofile          131072
elasticsearch   hard    nofile          131072
elasticsearch   -       memlock         unlimited
==== curl localhost:9200/_cluster/settings
{
  "persistent" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "all",
          "balance" : {
            "primary" : "0.50f",
            "shard" : "1.0f"
          }
        }
      }
    }
  },
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "all"
        }
      }
    }
  }
}
==== curl localhost:9200/_settings
{
  "tweets-2014-12-13:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "UYsabdN4T865kErAdx1AJg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-13:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "XmbQJ9acTvade6S_8qUwDA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "xP38TU1_SiGKxs86uUaiqw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "JJcHeHVfSL-JIRC7cstKnQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "qyZiPYIFRbiu3CffO-0uzA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "bIwh5NEPToGFIMPFR1Vnug",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "x0-PqYayRButRuoY28t_8Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "IbPsjCm2SSeZ4c4GvuZ9Pg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Hbw7FrzITKCzqYj6Nvq-9g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "-gHV9htqStOQSi4Sz8LvtA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "L0NCAj_nQAWyHtVBJIqCbw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "IKay9tNNTGuSa_6N4Bqu_g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "qp0MvPMKTrikwSjJJA--uQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "njWl7thuRlC4lyME-hES2A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "jrLaWTD1S7OJ7Y2MWNtNjA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "9Xabn995RVqfiiAItxBR1Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "PdR6LW1DSg2lOypj_q05_Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "t4lYWrKRRI6NsRiIyAkmag",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-13:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "znjWieCZQpCqFH-B5EWsdQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "2uuhVBKnQI6fep2fPIlM9Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "MYhfJiCAQGmQD8UY9_EvQA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "srpRd9DkRJq7ountfue2bw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "1xDyyYbJQUWbLCFyOiSeaQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "p1Z543ULRfySf2xHBiuqRg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "PCt35UWXRhuWYej-wR80HQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "htTlSifaSkCr2-5deD_m3A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "H5Rxi0ACQF6BB0BOIV4i4g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "j1kbSwEMS3qKRSjg01FNKQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "-MDxgP0UTF-d-4AmwQT78g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "gv1-kUxGQWSXKhsdz6Co5g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "EcrqVojxSGy50NT0bFbHpQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "VeyDvV6gQQyOB_RXdK2Ocw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "-594qw5sTcqBubG8aI7sWg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "GHvNtTY7Qr-_srxRpCOK4g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "WywItVurR6CZhjI0mVB23Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "DhKJz-kuSCejFRxYEJko3Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "T7Qoq1ZWSg6lqrGIDW709Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "dl2RTgkSTHWzxqNJ_CR-SA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "AouHHvMtQyegMou2IXTF0g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "lljTY-mOS7Km0u7F0hoXNA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "OpEj0jQfTdm_Jw8LRNWnyA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "2QFiH-iMRUCPyXZ8xLrHng",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Ccl5agM9QZu84doN6hrc5g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "DUMj-e3eSTaocM8yplzd4A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "EQ_OEMIYTZahqUFV-g3wVQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "PAEgY97DT12Ajsqn6Kjeyw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "SURp1kWpRzORNW8QA4XoGQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "FeSh0SZcQmGyI_HSUqeeJw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "GFThutg0QLqvFkB-fzBhdA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "0pRe5fK_QoGapxZMfg658Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-13:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "htDrRm5STX6xjymV9zwavA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "5FugA0RhQa-g6j8h2gyfIw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "aXvYB__eRyOAbOyL2VIsYQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "DAc3LFbdTpaZKDg5ALkiLw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "LYYiTU6FQFm7mf5e4zLksw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "kV6W4bQKRzObTt1nQBRz-g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "ugMLX-PORgGn2vvJDVc8Ew",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "E4XHaKt1TY6U2aM9Jy9BPA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "4oRje_rWTAKgsSk7EaE1XA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "G9hJ0NPmRciMWorHhwLdLQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "81VNgAmHT8W_HLlKzKEV0w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "bmTOmptnSIqfHu3wuw_x6Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "6RS8UV4LRDWX_pS0K6U5cA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "15qozk_tTqK1bTt0hbp0eQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "lnML8R-xT0abTTm5ssE3sA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "xO0_3gRTRbuTDDPav1HrLA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "WxkiU7JCSDimulaKFdRaGQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "tD6Q2J5qQS6AhcvVcsG4UQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "DKp30gl3Sxe03-fOOP3nQw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "bmgqQIXQRdGaVfKDo3X9Uw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "unVDIeoSSvmoSqSNoQ1x-g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "kjGdTH66TVuIVOLKhLQl2g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "pJcG-qJsQmigApM7UMjBdQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Ky_lY_GNQsyrXx-WKSzOMw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "_22oYqPkSXqfJnXqC6QN5g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "EUpEVkVpT6GeWOyKxMidIw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "nQy507JMQJ-nsOIT69nPsA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "20FRd6IvSxK2gVv8JVFrqA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "1abDESSdRvm1AkGwqoqKaw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "-CowMUqYToqosKyqEiE_ag",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "REi8lL3wQwiWsRLLgMq1RQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "L-Z67pJlRi-VuKCR05crvg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "caz67Lb8T9GmH1OiqRV2_A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "tm0f6GOaQqO4jyOKWo-JyA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Sy12Gb9HR62gFNasbvJGkQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Bk178p4AR4KHwd6LfadX-g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "ccUrleOFRE-hlCJGOVXlPA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "lsUggPfQSqmnWv4z7befLw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "whois_cache" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Cblh3K-0ScmiYDqJei4mmw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "F5NwMz9uTdavapwAQI4VCQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "j-TAq6ZwTPa3TRjrd7jP8g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-13:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "9uWEZXIoQ6K-9H9qtByrEA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Y7n82EPrRNebX0Aaxl-rgQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "kh9OCxVGT2yS2q_GNwy2Kg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "4C5IMnHKS2eNDBkk4pPrSQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "e2EWlpiZQBC4GTHlN4CfmQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "beUtunNwS225UgEAoNWb0g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "dn_cache" : {
    "settings" : {
      "index" : {
        "uuid" : "N-Vd613DRAqEpTlb5JZrtg",
        "analysis" : {
          "analyzer" : {
            "word" : {
              "type" : "pattern",
              "pattern" : "\\W"
            }
          }
        },
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "0LREWi0eTOGhyQtwumPIKw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "u3mCgpjZQ3m10cMKewnTfA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "cYWaMe9wTXe24Y4QbRB6Sw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "if1NIF_6QU6hLNlMi6IH_A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "u0hcm5-VS6ylnp8x5DIVDQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "DZB1AJfaQfeCp8wCH58f5w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "dazxQtOjRK68OXsVBhHV5A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "oLs5vaveQXK09fZHjeaDMw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "5WhVijvgRu2V9eMWcgmEMQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Yh9NsYqeRt--LOLpkliN9w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "CYhV1IVuQDWJOQE72F0r1g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "_2n4EiTlTPGdbbEwkudYlg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "CiJm4UcBSbum5EYud6wumg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "j5HRoYoKTMGzmTB6JSs97w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Bmp69fHHRpC_1DrzsaBwwQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "LfyvRDFKSOaQyMJr81HrgQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Nn5P67hoSwOGxt5VQ04JeQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "kHDcYTRJSqanNALamvOnYQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "WAMTMWqmSVGDyBa9aRR-3w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "BJLUGQjtSDS-T6juhFZ8Tg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-22" : {
    "settings" : {
      "index" : {
        "creation_date" : "1419206402879",
        "uuid" : "XmLTPkgSTOmgOcX266Ouyw",
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "version" : {
          "created" : "1040299"
        }
      }
    }
  },
  "tweets-2014-12-19:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "4T41ZTexRwuSSnNrjhZ8ow",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "yNV6Mnw6QSOFwI9oiAGGJg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "K8KNoc40Q6WBTxLiL4o-wA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "vQttEkAHSai3zUaiq_B9hg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "vo9ZyHf-SnORAFQ1W6dmvw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "wDanhf76QzyoDZ-gZTR4QQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "HiNJVmMFSqyoTskEJF_Lug",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "RKJXz8hDQCKPIy3BIcbP_g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "0TrgCoRERY6nFfxXr6J81A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "HPpiPNdjS_m7azPHTw0ceg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "8PFxJJ8NQDePsEMKdlzWTw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "tc8-h160T0q9xLMmK6Jx6g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "mcmm18QoQui_eZ7KsZlfdg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "N4V6PVPcSGWOsBw8ccmJZQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "0ns6gR-0S3itlmS5j5HSrA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Du8Mjc-oRXCS2pjtUNmSEg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "iRbjNDYMRre34tJs3vAVUA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "xiyM884jSxaQWVn0LECN_A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "PfTFS76ZQlGZcdvwXDnrTg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "vAWtOjNZTmq0j0-tXcotHw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "tchFhPX1TXKdHRkaWo9WuA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "3vdrqSF3Rv6_7JA8YppVlQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "OHrDK8nVT1KyXQNketGrAQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "xaj5-9ZIQPO_B7_D69YXZg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "jecmdjbSQ0yKNU0BZzDIeQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "fL4PDKH0SGiV35zvrqw6qA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "mCVEHYH_Sb2ZvjBzmhUXXw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "c5EI5RmdSBOgIdrJSgoYEQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "FezK9CJxSv6zS7_1jGvIQg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "A6DG-JZ4R8S3bQa2NItseQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "XAQyfj4jRG21Ldp7RQSRjg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-19:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "-PY0fawtTQu8cDc3GZmaWA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "bhdUiXXaQGmDwASnoG7EEA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "o1yS3uHHTc2kGPgMjan_xg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "enlwmtzyQreDYm9zLFP3uA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "TgoszQiBRhmFmxjjOdaPdA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "24RKPXUKRE6wO_SLDprIGQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "YZptP8AZTFa3jya7NlWdnA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "h0T37_HBRRezV5t4gKWcfQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "OeULRArjROmzOD_2Sd1bPA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "cKtDwZBgSwewdYdSZRsvHg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "hIQURcdLT9yJp6IbN9ocGw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "FliGWQcKSdqH027uwr-k3Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "YQyKINoYQbKioKxxqejMGA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "NvzrYw-GTxuHNLNPOyNESw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Fc2NJaAlSUKU3g4_8wrTrg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "QaDAFWNqQ5unS8RIsm-XNg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "tuVJ1PM2SfKVt5xo-xU2mA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "hXFe9ebUT2CuRsGNWZweOw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "VMpOMGiwSJqHPY18_FRGBA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "FX4G6PksRkmhmTLJyg6scg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "PFsYFQxZSym5vXrkg0fY0g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "e-C4XPVIQeqKRV1cyS7WYA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "EljN37_1T2aeEvrxGmVWIA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "yQMhNAGBQsivFaNrRWZI9A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "aYNnjxOhRciFoXJQLbhFUA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "EDDpOPM7QwSFE5UtNe9jqw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "QZmzAkKqSgipw-Cd9OyzUA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "gchidFbpS7eSSCpu2RZpRA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "DxzwBm06RAiAng1XDhSGow",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Up2GCPkWSHa_DoPFggtZGQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "LF-3OVhuSmC9zqQkdkFxiA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "_lHrpHARSIGUGRYxTEZqrA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "JAvCK8tmQo6xNgKQUVCMEQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "RQ8H3o6fSrelQ8M_0Jn5zw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "lDHflm0mS6SO3Q25ipMhjA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "bLKrjsm_Rz6gdNc67yEe9Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "8mg7AV2nRimJvG7xBGdGMA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "acM0ydCqTtOH0BqaS-ojhQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "whqAKmASR-6NSLLpKr2tKA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "mNs12_ZoT7u7SyLNFlAwJQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "G63BOecERWKc-5igVaoKIA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "jqaHo3_BQ166veyPs5LyhA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-13:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "dFgCYiRWQIeNnidp-l2RdQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "cyjaNYsuTGyOkPOP6FiQbQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "UvL16K6lS-Og6MD2uvNKPA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "S2ZzBMHSQ2Wm_jN_Hmkrig",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "qIBTeXtxTDqK3fuMh2cklw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "l_seD5ZfQCSXpjpGuazQeQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "YaRs4oEoQZqYj4BcpfGRgw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "NIIDCLOYT-meTqFRwLrA1g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "QIExRezATSWz0lDjwcsuWw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "7okuvEFpRqyjc4qaWG4dmQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "cq1qj4Y6QCyeeft3YPck1g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "K3liiY1xQiiLkHS0JtDfHw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "hmeE9dttRIa4IcmlVjA7EA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "bMciGxpMT2yJslDKxVS92w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "zRvlYW_uSTeh5yqCkktidA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "uP0-Yq5hRP2A6jCAAncGjQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-13:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "pjdxU5MKSZeKCCNA0yYMOQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "8VvCyFBVQVaCdV4-gc3FcQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "XJ4ldf04RA-N_FRCFE44lA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "z-qqkw1UQgKPOiJskF9Eng",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "37crw3NTSkqyAPRI5JNtCw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "eQh_7P3WRmSlv7SZZWVuxw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "9laYPeJGQaW4wVHI704Rtg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-13:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "MheQTW_ZQlCTP6U3dmZrEQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "9W6WdTnLQUyJdYLmUwM_Sg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "X9gB7xeCQUeyvcNo1XLy5A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "MxR_TeKRTG665A5gB_pn0g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "cyOi5WmqRuuKBKzQLhiUCQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "pUSKoEBGQlKrOUlbI4Rdbw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-20" : {
    "settings" : {
      "index" : {
        "creation_date" : "1419041578845",
        "uuid" : "K94ciIyDQo6sa8hHdDjT1A",
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "version" : {
          "created" : "1040299"
        }
      }
    }
  },
  "tweets-2014-12-06:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "dIz_C3ICS1msigjeJfHCkQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "pnYQF7Y0Q-uqtrF1MXydyA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "SJkiLoOYQySnWrSN9XBb9A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "W4rbJpt4Q9O6rcuXyD3oUg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "jmiBuktJRT6Cozz11hBUGQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "XMlFA9xtR0S7vTeN-DzFcQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "_lpa_zJERQyAmVyl3ES_Pw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "lX4xSmkAT7iM80nFD8OFug",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "LPdIHchnQgqAZi8B9NFruQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "PC8xo1FURre9AEXNGUQPjA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "0CmYrZTtQouHXBnnSt7cmQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "tN2RbsMZQ3-8QVF3AwJC4A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "73mr3gomS82zVr9-0yH33g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "-9MXDStzS5O7RkOsiCM3aQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "ycAcvFOMQYWjJctYOwxGlQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "TQm6F5-ySUODbNAN2euTyA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "iJxNrLTdScS2o_4BzIcQJw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "BgZOsa_4SYawfCDPoltLgQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "hVCshHisTdG8RnH4-LLabQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "xomv8WTeTZOzgii1__H4UQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "48BWgRB_TFaLVz3_03FCdg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "7fo6fg0IR9-bo9bl1xEkhg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "bHGDokKBQH2acvsiR8bPyA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "GpJkLu-rQqyrcEpjlfib6g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "jc7MIoBXTm6UP87ZMN8ABw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "c316-bXrRqOhvGzWjmiovg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "W4nnsfm2QcCyklOMd-DP9Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "7bklvc3TT8GhaSyXwr2KSQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-13:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "MY1uLKB-QDSQNHXLHqPvCg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "ZOy8Aw8vQGWFNwl1mkcA6A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "dO369ikoTMOEdoL6NTYw7Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "0UqS94o8TPWoA3xgGayIzw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "uSsC4DGdTj68QvUuh2o_Nw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "_XIeKXcES8OP8onkcZdn_w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Sa9XhtwYTEKnjizXfyKrLw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "6cigMTHdRO26nfR6y-6y5Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "LJjmjqxDTmKob66dqijFKg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "JERYE_fkSKmRjk42kWf7hQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "9Cu4DuibQWK8QTyqZB1cQw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "JM1TafaWSt6tlFJJRsN1ag",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "zt4gTYMlTOCKd1j9f19TeQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "i6bJbb5CQu6_CP7XyTTVgw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "O6XU0gy8Sgi4bqHHJ42bFw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "BT56x8CHRrGXX-Y0l3lw7g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "cB75lTLuQ0azct6mW1ZH2g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "4Wvi-lOZTZCbRn2p86Tr0g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "1a_qrZL0TZqTlw8NCmOSDQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "JT1ZFwIqRg6LW8JTMCS5yg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "cgXtIanTSTm422khRWlaKw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "BAjRTjHTQWSJXmTIep_FEQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "PV01lfRgSz-PfPZH3S82aQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "BSCVy3LTRJiWxiee9t89CQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "1WBbSb6cR_ieBjsweQ7gJA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "ErA7BKvcSdmpwpcSmIyB5w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "_pPUeA94QJmjaJW-QC087w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "ipSEOBNFSTeAA3be9WcXCg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "CC7YSTWKTJuVZeyspXqZFA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "1xQSVOZVT0ypNHdX17qVGg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "vNNMRTRrQGyE8K7jcMmAjA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "4tk90W9ZRBGBgvJi112Ilw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "ejfKRxEgQNO_Rp1N7iboaQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "svGCFi-_T5CD1MVLiSt2EA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "GszhXL0KTyGguZszKd9b1g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "OlGw410ZTcqZHcX9rsIWPA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "TFTTaXhoStGCSWu5eZH6Fw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "pACFgfq7TSq_6niyTfOdYA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "HNLJZFPwSW235tIVpg9Pbg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "J4swhhOqR4uHXJeyZCG6Vw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "UhIa66-lSHCjzEnX0jCAMw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "gcxWa8PMSKy6Xnbqo37x-g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "GKJ17bpXRV6JBYX1EQCI2w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "zFZodalkTvumwQBbvAm2FQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "6LMa0qlKQhKCnzc-KiqM1w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "7SsMATtEQFO_LdpGzhqzUA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "S36f6kXYQqOaG2DKHGDKjw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "5MQiTt7ZRtS6LaF4HiB10w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "CFOxuFwsSEKSiv-IMenVaQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Qs_YpWBrTJSL2VGWRXXR_Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "NFaoDbxWReKHs0ixAAgiww",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "rK8lYH6bQgiBzgvnqmcFgw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "jWnUs3veRuGBxYgFHyOMbw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "7GxGtTruRcydKO94KbMAxw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "5dqwfhpvQKap8u6p-cRcIg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "iPPjvdisS-e5zB1muioTdA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "jpMZCFn6RByzjZljRpXySQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "_XSiFYe_RsejBfwzlfnD8w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "vavuYz_YSHiFgXoHcUktmA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "IgvvBNaWT3SQPTa0m5Vuow",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "-opggS70SWWQo3dTcAQ2sg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "ePygOAqsTzi2fgE-Ow_Hmg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Z5tSiuEhQX2E_9dmFTzR-g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "k4edXd7cQb2obUA553gG4g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "V8yuRWy6Sj6kTU2nBgf-qA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "x1VfZH8_SiCN1geaQy3Eng",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Y2U6mc6TQBanl3ve3SJyhA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "PyHM3h1LS126m2OP9FkFjw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "7ni3rrZRQAOqWXgNLkvbZQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "F1c3PFnTSr-8n8vaJJcnwg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Ty8TtCoASLSpUSVkeWKoaw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "mRMZyut_TESWcjTxtNE3nA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "kMCYtgugTTmtruxczrgw2Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "r3_-8adtR_qqBpwVkbrp6g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "8K6rrRRyQueaJQJjYzYWyQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "nRlcYKXsQXuzAZGSL3_X5A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "i8fIFwTjSri2h9xYmr8ZDg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "7-HHZZiGS6KSnHtbmQjzAw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "tAZG6CeIRy6WlQ_y3avmZQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "mT8jkrTSTWmS-7RP2g-QYQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "8bv6GdmWSsG_17k8vabFcQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "5ZwYeg3AQKCpgWHY0LXGzQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "w6gydbZJTPSEv9stFTo_1A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "u5mvY6EjQ4ifnj8Yed3Wjg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "7oxilByPT-O1sKe7auDy2A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "BKZ8htPYSkOqz8Oq66hLTg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "QrnP2ELhRoqsZl8qRzyAEQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "HO9Y2QD5QkSvxt6vYfxePQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "VY32w_t3TT-_t5UQzf2G4Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "keNiCAUBSEKDJbhM2jAD3Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "bLdTuIDPR9Gj7r_qw-V8bg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "mai1tOA4QtiTXF1e6kA0VQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "En3LuDS6TtOX1FN1AY8kwg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "MgRkB8j1SMeC0lArdiDHzA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Mzlk0tAlSY-WkhyI55tcsA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "rOxGWx5MSU2MsKU8jNJjdQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "1A1lwXNdQc-LzDCjY2V99g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "e-xbJg3NTWuGMNhs0p_lYg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "1EJbaSBnTBiUCRWXSh6RHA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "gfsuUdkOTaa0ukwiS0GO3w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "SG4QSfB4RUSPDJK9YBS7cw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "sgUwjEEJTrCDXLjgMNoQdQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "0mI54mTgST2zJOF3UT0K6g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "1gf3BNMWTyCG_PXUCyTcbA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "md3vBTwhSgu9ksUnTHXFtg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "oDIWSfUNQnCOpwlpPVGK8w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "jthO1U1MSCurnbJY6jMd3w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:14" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "TV1XBlmITzuyxdUUv5s9Dg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "oWCxk0r2RROeoddAKakeGw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "DZebIVfXTqmTe3_GxeXtDg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "bpCrV9FcT4aJvBwYRg2zog",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "OuSA6_5MTy2VyrAuXGS4Ug",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "RCa6O7KFRt6i4KLmsVew2w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "j_ReKubOQvKnbTM71dK3hQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:12" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "BfuhEAdLSMe_nhwB-kUbmg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "VbNtx7ZoQi6yUNPwu1WGsQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "3EpcUt5WScunctSVA83j4g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "zoKce1rjRemUmbSjiOYhgA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "sUIp_klyRt-lB94F37m5uA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:04" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "HVaqK1i2TJODZVMUvNA4HQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "U4EzHkTkRiyTMsfmRGZkRQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "6N3G9UY8Rc-EFpMRZGzQGw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:20" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "0tUR2gNnTfqTq3gKyy-pXw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "eJW4a5ZGTkOR5uKiS_XbkA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "q4T3sDO2SBi4fKTIu5ZqDg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "b09P_exwTiWGPTEokVTCPg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "2U7DlZ_wQ4mdYgRXIboEZg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-07:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "oxWe_wA6QFSmuocKt5xoSw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "yLZG3u2VQ6mQe4KgrcCF5A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "x7vkXJXvTq63ATiwMi68CQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "WuwchPdxSy-Me6szB4sXYQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "q3tiMMGHRN-2L4Ctr9DoBg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "_s3vMTGLT2K1I6cmNVPx6Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "VFVwNmlhQ-GmoitPQFpMIg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-03:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "s1HXuMZ5TcCcUdoedX0fEQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "IMIcUcnwSculLXSFjZJHZg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "lqZ2Ze5qSRqhqcthuE0Dtw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "zQUmfQWjSyKvdD9cNIxKOQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "EH-21G15TQS3NTJznFMa-w",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "FMwigrSBTBeDDzdPgVpdZA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "LSyGtxM9THu3CIe4LIwJVw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "fn407gr2Tf2e_2WiBJ-3_A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:17" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "GUaRuI9jRU-kMWrrb9NvcQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "B2RSTFyrQO6E4DDp6lir0Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-13:21" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "TVGWYUNkS2K0bqGxnlFs5A",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:16" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "UdOPND9dS0O_WnOwULbdzg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:03" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "kkwxBPwLQLGuV3Ci9du7IA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "4XBQpNDRTUGDBtLLmbEARA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "W8TV3KUbSiOtqK_9wickoA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-11:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "lyFEoB6TRJSWkKbly16k5g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "jspOBzuJSGaowpWCZDLXeg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:01" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "QmpxHcEYSm-cbt6UTH17kQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "h-fiHXYbR6SSZFFJWZ-gZQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-14:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Hr7DN0GLQ1WrMlJy6h0U1g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:10" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "VbI06arxQq6K6ucQoxzY4g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-08:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "lDNDRq7ZThikvl4CoLPwwQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-15:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "8rC2jLpHQtKO-xJQkzcALw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:18" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "S2PC0xpXQiSRgMP4CdKdTA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-09:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "WUdz1emRQIuJZsQkA-XvqQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-05:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "NSioQga6Tk6DNszCBPUgDA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-02:08" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "ar80Ou48RUOt4KYJnQz4cQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:02" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "qSFuoxmiQoym4H2IbsfiqQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:23" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "luksgGIcQWSDJXhjE3fxsw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-12:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "RvvbLCqcRkSDv9ZEvwJyTg",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:13" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "74rNVrGMTNWlMAlTYiXpgw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:00" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Z8DSs8HlRp6dLSJfNvEt6g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-01:05" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "VzLSIPUgScKMGIhhFM-FNA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-04:09" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "Cd-fI8kWSuSM3JrxiEJWaA",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-18:19" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "b5xupp2fRPGmYhmG3bruvw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-10:07" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "2AmIloROSAeL8pQ3h5Ns8g",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-21" : {
    "settings" : {
      "index" : {
        "creation_date" : "1419120003834",
        "uuid" : "AUEJtbO7SUaYc19yd4VG4A",
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "version" : {
          "created" : "1040299"
        }
      }
    }
  },
  "tweets-2014-12-14:06" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "xCHssjvCRN-MONIpy4lUYQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-06:15" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "jun_i_ExQamrv-uhPlxlDw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-17:11" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "zc7CCzITTc-SmoruPMw8Iw",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "tweets-2014-12-16:22" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "JPqZyT9WQl-9KiIBBVS28Q",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  },
  "kibana-int" : {
    "settings" : {
      "index" : {
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "uuid" : "xU72wrkLQVGCemOFq6UlYQ",
        "version" : {
          "created" : "1030299"
        }
      }
    }
  }
}

@clintongormley
Copy link
Contributor

Hi @vichargrave

OK, so we're not talking about a single index here. It's one index in the midst of many others, all of which have to be taken into account.

Primaries and replicas are essentially identical, they do the same work (with the exception of updates, see #8369), so it doesn't matter if all the primaries are on a single node or not. The first thing I'd say is: change the allocation settings back to their defaults and don't touch them unless you are seeing an actual problem.

Then, if you ARE seeing a problem, choose one setting only and adjust it by small amounts (eg 0.05) until you see something happen, ie when it crosses the threshold.

The risk that you run with changes these allocations settings is that you end up shuffling shards around your cluster for no good reason, causing a lot of extra I/O.

Btw, looking at your settings, I'd recommend deleting this:

index.translog.flush_threshold_ops: 50000

It is better handled by the flush_threshold_size setting.

Also, your threadpool settings are crazy, unless you actually have 60 processors that you can devote to bulk indexing. The default is one bulk thread per processor. More than that and you're just causing a lot of context switching.

@vichargrave
Copy link
Author

OK thanks for suggestions. I can see what you are saying with regard to
all the primaries residing on a single node. However in my case, when I
upgraded to ES 1.4.2 I saw that one of my four nodes did not get any
shards, primary or replica. That situation is not good from a disk
utilization standpoint. Elasticsearch should be distributing shards across
all nodes to take advantage of the extra disk space that the entire cluster
has to offer. If some nodes are ignored then the effective disk space is
reduced.

On Tue, Dec 23, 2014 at 4:38 AM, Clinton Gormley [email protected]
wrote:

Hi @vichargrave https://github.com/vichargrave

OK, so we're not talking about a single index here. It's one index in the
midst of many others, all of which have to be taken into account.

Primaries and replicas are essentially identical, they do the same work
(with the exception of updates, see #8369
#8369), so it
doesn't matter if all the primaries are on a single node or not. The first
thing I'd say is: change the allocation settings back to their defaults and
don't touch them unless you are seeing an actual problem.

Then, if you ARE seeing a problem, choose one setting only and adjust it
by small amounts (eg 0.05) until you see something happen, ie when it
crosses the threshold.

The risk that you run with changes these allocations settings is that you
end up shuffling shards around your cluster for no good reason, causing a
lot of extra I/O.

Btw, looking at your settings, I'd recommend deleting this:

index.translog.flush_threshold_ops: 50000

It is better handled by the flush_threshold_size setting.

Also, your threadpool settings are crazy, unless you actually have 60
processors that you can devote to bulk indexing. The default is one bulk
thread per processor. More than that and you're just causing a lot of
context switching.


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@clintongormley
Copy link
Contributor

@vichargrave Actually, in 1.4 there were a number of commits around the disk allocation decider, which includes automatically rerouting shards when your disks start to fill up. See #8206, #8659, #8382, #8270, #8803

So (in theory at least) this should be better in 1.4.2 than before.

Disk space is just one of the many factors taken into account. Presumably it wasn't the most important factor at the time you created the index, which is why you ended up with the current layout. We prefer not to move shards unless really necessary otherwise we generate masses of I/O for no good purpose.

Unfortunately, understanding the decisions made by the allocator is really quite hard. While you can turn on debug logging, it produces way too much info to be useful in production (which reflects the complexity of the decision making process).

@clintongormley
Copy link
Contributor

It sounds like this ticket can be closed?

@vichargrave
Copy link
Author

It appears that these settings get primaries and replicas distributed across all 4 nodes:

cluster.routing.allocation.balance.shard: 1.0f
cluster.routing.allocation.balance.primary: 1.0f

Still I think the default shard distribution behavior for 1.4.2 is weird and that I have to make these adjustments even weirder. Yes we can close this issue, but it will be raised by other users.

Thanks again for your help. It is always appreciated.

@vichargrave
Copy link
Author

Since this issue hasn't been closed yet, I'll continue. I have changed the shard distribution settings to this:

cluster.routing.allocation.balance.shard: 1.5f
cluster.routing.allocation.balance.primary: 2.0f

And yet the shard distribution still looks like the following (note - the bright green squares are primary shards and the dim green squares are replica shards):

shard_distribution

Notice all the primary shards on the second node and no shards of any kind on the first node even though I have increased these settings.

Now I have a 4 node Elasticsearch 1.4.2 cluster where 25% of the disk space is unavailable to store data. This situation has to be a bug of some sort OR I have to increase the balancing settings to values much larger than I have tried so far.

@vichargrave
Copy link
Author

I think this should be logged as a bug. This is not a desirable behavior and it tempts me to roll back to 1.3.2.

@s1monw
Copy link
Contributor

s1monw commented Jan 2, 2015

why are you setting cluster.routing.allocation.balance.primary: 2.0f you should not modify these settings. Did you have them set before? I don't necessarily see this as a bug. We might have some disbalance due to a rolling upgrade, how did you upgrade and did you disable allocation? is the rest of your cluster well balanced?

@vichargrave
Copy link
Author

I'm trying to get the shards evenly distributed between the nodes in my cluster as they were in version 1.3.2. I tried a rolling upgrade from 1.3.2 to 1.4.2, which is when this issues appeared. Now, as I stated before I have a node with no data on it do 25% of my cluster's disk capacity is unused. That can't be right if that is what Elasticsearch is doing, so yes it is a bug as far s I'm concerned.

Please review the previous posts which tell the entire story.

@s1monw
Copy link
Contributor

s1monw commented Jan 2, 2015

I did read the post and the story, there are still some things I don't understand. Bare with me and help me to understand:

  • you have only one index in your cluster and one node has no shards of this index so 25% of your cluster is unused assumig 4 nodes, is this correct?
  • if you have more indices are all of these indices well balanced or are there some indices that are not fully balanced?
  • if you look at your cluster, is it balanced based on the number of shards per node cluster wide not per index?
  • you set cluster.routing.allocation.balance.primary and cluster.routing.allocation.balance.shard did you also set cluster.routing.allocation.balance.index? if you didn't I think what you are seeing is expected or rather possible. if you want to have well balanced indices you should give more weight to cluster.routing.allocation.balance.index and cluster.routing.allocation.balance.primary should really only be a tie-breaker. It's pretty hard to get back to full balance if you have many indices that are not well balanced since the shard allocator was build to be conservative with respect to relocations.

I'd start with restoring defaults for these values:

cluster.routing.allocation.balance.primary: 0.05
cluster.routing.allocation.balance.shard: 0.45
cluster.routing.allocation.balance.index: 0.5

and calling _reroute

@s1monw
Copy link
Contributor

s1monw commented Jan 2, 2015

FYI I just fixed the wrong default in my last comment from: cluster.routing.allocation.balance.primary: 0.5 to cluster.routing.allocation.balance.primary: 0.05 sorry for the glitch

@vichargrave
Copy link
Author

No I have hundreds of indices. Look at the previous settings out put I
posted when Clinton was answering my questions. The graphic I induced was
just the result of a recent index. I see what you mean about the
cluster.routing.allocation.balance.index field. I have it set to the
default. I experimented with the cluster.routing.allocation.balance.primary
and cluster.routing.allocation.balance.shard settings according to
Clinton's suggestions. There seems to be some confusion on this balancing
issue.

I'll set the fields you indicate back to their defaults. I'm not sure what
you mean about then call _reroute. I know how to omove shards around, but
what just to make sure we are talking about the same thing, what is the
_reroute call I should use?

On Fri, Jan 2, 2015 at 1:09 PM, Simon Willnauer [email protected]
wrote:

I did read the post and the story, there are still some things I don't
understand. Bare with me and help me to understand:

  • you have only one index in your cluster and one node has no shards
    of this index so 25% of your cluster is unused assumig 4 nodes, is this
    correct?
    • if you have more indices are all of these indices well balanced
      or are there some indices that are not fully balanced?
    • if you look at your cluster, is it balanced based on the number
      of shards per node cluster wide not per index?
    • you set cluster.routing.allocation.balance.primary and
      cluster.routing.allocation.balance.shard did you also set
      cluster.routing.allocation.balance.index? if you didn't I think
      what you are seeing is expected or rather possible. if you want to have
      well balanced indices you should give more weight to
      cluster.routing.allocation.balance.index and
      cluster.routing.allocation.balance.primary should really only be a
      tie-breaker. It's pretty hard to get back to full balance if you have many
      indices that are not well balanced since the shard allocator was build to
      be conservative with respect to relocations.

I'd start with restoring defaults for these values:

cluster.routing.allocation.balance.primary: 0.5
cluster.routing.allocation.balance.shard: 0.45
cluster.routing.allocation.balance.index: 0.5

and calling _reroute


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@s1monw
Copy link
Contributor

s1monw commented Jan 2, 2015

ok so if you have that many indices I can totally see how this can happen with the settings. Did you have any of these settings changed before this issue happened?

I'll set the fields you indicate back to their defaults. I'm not sure what
you mean about then call _reroute. I know how to omove shards around, but
what just to make sure we are talking about the same thing, what is the
_reroute call I should use?

I am referring to http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-reroute.html if you call

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{}`

we do a rebalance run so you changes should kick in after that call. The pure update call won't trigger a reroute.

@vichargrave
Copy link
Author

No I did not change any of the default settings. That came later when I
noticed the problem. Let me restore the default settings, then I'll give
the reroute call a try. I'll keep you posted.

Many thanks.

On Fri, Jan 2, 2015 at 1:42 PM, Simon Willnauer [email protected]
wrote:

ok so if you have that many indices I can totally see how this can happen
with the settings. Did you have any of these settings changed before this
issue happened?

I'll set the fields you indicate back to their defaults. I'm not sure what
you mean about then call _reroute. I know how to omove shards around, but
what just to make sure we are talking about the same thing, what is the
_reroute call I should use?

I am referring to
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-reroute.html
if you call

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{}`

we do a rebalance run so you changes should kick in after that call. The
pure update call won't trigger a reroute.


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@s1monw
Copy link
Contributor

s1monw commented Jan 2, 2015

I am curious, would it be possible to get an output of the CAT API for shards:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cat-shards.html
I wanna try to simulate the situation you are in that might give me a better insight into the shard allocation. It might be that the allocator has numerical problems with lots of shards and low number of nodes?

@vichargrave
Copy link
Author

So I need to do this on each node, correct?

On Fri, Jan 2, 2015 at 2:12 PM, Simon Willnauer [email protected]
wrote:

I am curious, would it be possible to get an output of the CAT API for
shards:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cat-shards.html
I wanna try to simulate the situation you are in that might give me a
better insight into the shard allocation. It might be that the allocator
has numerical problems with lots of shards and low number of nodes?


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@s1monw
Copy link
Contributor

s1monw commented Jan 2, 2015

So I need to do this on each node, correct?

no, any node will do.

@vichargrave
Copy link
Author

This a really long file. Can I send it to you directly via email?

@s1monw
Copy link
Contributor

s1monw commented Jan 2, 2015

sure simon at elasticsearch DOT com

s1monw added a commit that referenced this issue Jan 6, 2015
In some situations the shard balanceing weight delta becomes negative. Yet,
a negative delta is always treated as `well balanced` which is wrong. I wasn't
able to reproduce the issue in any way other than useing the real world data
from issue #9023. This commit adds a fix for absolute deltas as well as a base
test class that allows to build tests or simulations from the cat API output.

Closes #9023
s1monw added a commit that referenced this issue Jan 6, 2015
In some situations the shard balanceing weight delta becomes negative. Yet,
a negative delta is always treated as `well balanced` which is wrong. I wasn't
able to reproduce the issue in any way other than useing the real world data
from issue #9023. This commit adds a fix for absolute deltas as well as a base
test class that allows to build tests or simulations from the cat API output.

Closes #9023
@s1monw
Copy link
Contributor

s1monw commented Jan 6, 2015

@vichargrave I pushed the fix to 1.4 and 1.3 as well. Do you have a chance to test this? Note, if you are are upgrading to a snapshot build you might not be able to revert back to 1.4.2! Please consider this, I don't know what purpose your cluster has or if you are running in production. I can totally understand if you say no thanks... if you can deal with it I'd be happy if you can try it out.
You also need to be aware that if you move to this new build you might see quite some relocations happening, in my tests I saw about 5% of your shards got relocated to get back balance in the cluster.

if you wanna simualte it I have a tests checked in :)

@vichargrave
Copy link
Author

OK , so by I might not be able to revert back to 1.4.2, what does that
mean. Would I be able to revert back to anything, 1.3.2, etc? Also, if
the upgrade to the new code is successful, will all the shards on my
cluster be redistributed automaltically or do I still have to do that by
calling _reroute?

On Tue, Jan 6, 2015 at 1:40 PM, Simon Willnauer [email protected]
wrote:

@vichargrave https://github.com/vichargrave I pushed the fix to 1.4 and
1.3 as well. Do you have a chance to test this? Note, if you are are
upgrading to a snapshot build you might not be able to revert back to
1.4.2! Please consider this, I don't know what purpose your cluster has or
if you are running in production. I can totally understand if you say no
thanks... if you can deal with it I'd be happy if you can try it out.
You also need to be aware that if you move to this new build you might see
quite some relocations happening, in my tests I saw about 5% of your shards
got relocated to get back balance in the cluster.

if you wanna simualte it I have a tests checked in :)


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@s1monw
Copy link
Contributor

s1monw commented Jan 7, 2015

oh ok so for 1.4.3-SNAPSHOT you can downgrade to 1.4.2 since they have the same lucene version. You can't go back to 1.3.2 you will likely index too new exceptions. If you are doing a rolling restart I suggestion you to upgrade the non-master nodes first. once they all come up everything should rebalance automactially. Be aware it might take a while...

@vichargrave
Copy link
Author

I checked out the code from github, but I don't see a 1.4.3-SNAPSHOT.
Perhaps I would be better off rolling back to back to 1.3.2 until the fix
goes into the mainstream code. Also I'm more comfortable upgrading from
RPMs.

On Wed, Jan 7, 2015 at 1:31 PM, Simon Willnauer [email protected]
wrote:

oh ok so for 1.4.3-SNAPSHOT you can downgrade to 1.4.2 since they have
the same lucene version. You can't go back to 1.3.2 you will likely index
too new exceptions. If you are doing a rolling restart I suggestion you to
upgrade the non-master nodes first. once they all come up everything should
rebalance automactially. Be aware it might take a while...


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@vichargrave
Copy link
Author

OK I decided to go ahead an build the 1.4.3-SNAPSHOT. So I just
replace elasticsearch-1.4.2.jar
with 1.4.3-SNAPSHOT.jar correct? Before I do that I'd like to review the
steps for an upgrade. Here are the steps from your website:

  1. If possible, stop indexing new data. This is not always possible, but
    will help speed up recovery time
    2.

    Disable shard allocation. This prevents Elasticsearch from re-balancing
    missing shards until you tell it otherwise. If you know the maintenance
    will be short, this is a good idea. You can disable allocation with:

    PUT /_cluster/settings
    {
    "transient" : {
    "cluster.routing.allocation.enable" : "none"
    }
    }

    Shutdown a single node, preferably using the Shutdown API on that
    particular machine:

    POST /_cluster/nodes/_local/_shutdown

  2. Perform maintenance/upgrade

  3. Restart node, confirm that it joins the cluster

  4. Repeat 3-5 for the rest of your nodes
    7.

    Re-enable shard allocation using:

    PUT /_cluster/settings
    {
    "transient" : {
    "cluster.routing.allocation.enable" : "all"
    }
    }

In step 2, shouldn't the "transient" be changed to "persistent" because
when the node is restarted it will start tor rebalance, correct?

Please advise.

On Thu, Jan 8, 2015 at 10:58 AM, Vic Hargrave [email protected] wrote:

I checked out the code from github, but I don't see a 1.4.3-SNAPSHOT.
Perhaps I would be better off rolling back to back to 1.3.2 until the fix
goes into the mainstream code. Also I'm more comfortable upgrading from
RPMs.

On Wed, Jan 7, 2015 at 1:31 PM, Simon Willnauer [email protected]
wrote:

oh ok so for 1.4.3-SNAPSHOT you can downgrade to 1.4.2 since they have
the same lucene version. You can't go back to 1.3.2 you will likely
index too new exceptions. If you are doing a rolling restart I suggestion
you to upgrade the non-master nodes first. once they all come up everything
should rebalance automactially. Be aware it might take a while...


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@s1monw
Copy link
Contributor

s1monw commented Jan 11, 2015

sorry for the long delay I would not replace the jar file. I'd download a real package from https://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch/1.4.3-SNAPSHOT/ you can get a .deb, .zip, or.tar.gz and do a real install on each node.

In step 2, shouldn't the "transient" be changed to "persistent" because
when the node is restarted it will start tor rebalance, correct?

since you are doing a rolling restart (cluster is always up) you can use transient.

Note: you will see balancing going on once the master is upgraded and your cluster has allocated all shards that where there before ie. all indices are active.

@s1monw
Copy link
Contributor

s1monw commented Jan 29, 2015

any updates on this? @vichargrave

@vichargrave
Copy link
Author

Not quite yet. I'll be pulling the trigger in a couple of days.

On Thu, Jan 29, 2015 at 9:41 AM, Simon Willnauer [email protected]
wrote:

any updates on this? @vichargrave https://github.com/vichargrave


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@vichargrave
Copy link
Author

Hi Simon.

Something weird happened. I have not yet tried the ES 1.4.3 snapshot you
provided, but when I deleted about half of the data we ingested, my cluster
spontaneously redistributed the shards evenly across the cluster. We are
not talking about a lot of data - it's about 750 GB. Do you know what
might have caused this?

So I'm not sure what to do now. I'm disinclined to apply the patched build
since the cluster is in state I wanted all along.

Let me know your thoughts when you get a chance.

-- vic

On Thu, Jan 29, 2015 at 9:41 AM, Simon Willnauer [email protected]
wrote:

any updates on this? @vichargrave https://github.com/vichargrave


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@vichargrave
Copy link
Author

OK it looks like 1.4.2 is still not putting shards one of my 4 nodes.
After the data deletion though, it is another node that is avoided. So I'm
going to try the patch soon.

-- vic

On Fri, Jan 30, 2015 at 6:53 PM, Vic Hargrave [email protected] wrote:

Hi Simon.

Something weird happened. I have not yet tried the ES 1.4.3 snapshot you
provided, but when I deleted about half of the data we ingested, my cluster
spontaneously redistributed the shards evenly across the cluster. We are
not talking about a lot of data - it's about 750 GB. Do you know what
might have caused this?

So I'm not sure what to do now. I'm disinclined to apply the patched
build since the cluster is in state I wanted all along.

Let me know your thoughts when you get a chance.

-- vic

On Thu, Jan 29, 2015 at 9:41 AM, Simon Willnauer <[email protected]

wrote:

any updates on this? @vichargrave https://github.com/vichargrave


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@orenorgad
Copy link

Hello,
I'm also having a similar issue. Perhaps it's related.
Using version 1.4.1, using 2 nodes,total of 90 indexes ,2 shards for each, loading with 0 replicas and then updating to 1 replica after loading completes.

It appears that after a node restart/recovery most of the primary shards are located on the same node and the other node has most of the replicas respectively. Therefore, when creating a new index with 0 replicas, it puts all shards (which are obviously primary) on the same node, the one with more replicas, causing the cluster not to use the other node for the loading process.

Tried fixing it by setting the below setting but that didn't work:
cluster.routing.allocation.balance.index: 0.9f
cluster.routing.allocation.balance.shard: 0.0f

  • to make sure it only cares about shard distribution within an index.

Since cluster.routing.allocation.balance.primary was deprecated since 1.3.8, I can't think of anyway to rebalance the primary shards.
My only solution at this point is to index with 1 replica from the beginning...

Any thought?

@vichargrave
Copy link
Author

Just wanted to follow p to let you kmow that I upgraded to 1.4.3. After
re-starting the system the shards were all rerouted so they are now evenly
distributed across my four nodes. All is well again.

On Mon, Feb 2, 2015 at 9:50 PM, Vic Hargrave [email protected] wrote:

OK it looks like 1.4.2 is still not putting shards one of my 4 nodes.
After the data deletion though, it is another node that is avoided. So I'm
going to try the patch soon.

-- vic

On Fri, Jan 30, 2015 at 6:53 PM, Vic Hargrave [email protected]
wrote:

Hi Simon.

Something weird happened. I have not yet tried the ES 1.4.3 snapshot
you provided, but when I deleted about half of the data we ingested, my
cluster spontaneously redistributed the shards evenly across the cluster.
We are not talking about a lot of data - it's about 750 GB. Do you know
what might have caused this?

So I'm not sure what to do now. I'm disinclined to apply the patched
build since the cluster is in state I wanted all along.

Let me know your thoughts when you get a chance.

-- vic

On Thu, Jan 29, 2015 at 9:41 AM, Simon Willnauer <
[email protected]> wrote:

any updates on this? @vichargrave https://github.com/vichargrave


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@tnine
Copy link

tnine commented Feb 18, 2015

Should this be fixed in 1.4.3? I assume so from the issue tag, but we're on the 1.4.3 release and also experiencing this. I've added the following config to our yaml to ensure we're correct on the nodes.

cluster.routing.allocation.balance.primary: 0.05
cluster.routing.allocation.balance.shard: 0.45
cluster.routing.allocation.balance.index: 0.5

Then hit this endpoint

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{}'

Does this need to be run on the master node or will any node suffice? I have 87 indexes in our cluster, each one has 6 shards, and 1 replica. We have 6 nodes in our cluster. When I perform the rebalance, I'm still getting some nodes that have 4 primary shards. This is causing very uneven load, and some of our applications are suffering significant performance degradation as a result. Note that we applied the setting and restarted after disabling allocation, then re-enabling.

@vichargrave
Copy link
Author

Hi Todd.

I reported this as a bug to Elasticsearch when I first upgraded to 1.4.2
two months ago. Elasticsearch verified it was indeed a bug and fixed it in
1.4.3. I verified this over the weekend when I upgraded to 1.4.3.

Good news is, when you upgrade to 1.4.3 it will automatically reroute your
shards so they are evenly distributed across your cluster.

-- vic

On Tue, Feb 17, 2015 at 5:03 PM, Todd Nine [email protected] wrote:

Should this be fixed in 1.4.3? We're also experiencing this issue. I've
added the following config to our yaml to ensure we're correct.

cluster.routing.allocation.balance.primary: 0.05
cluster.routing.allocation.balance.shard: 0.45
cluster.routing.allocation.balance.index: 0.5

Then hit this endpoint

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{}'

Does this need to be run on the master node or will any node suffice? I
have 87 indexes in our cluster, each one has 6 shards, and 1 replica. We
have 6 nodes in our cluster. When I perform the rebalance, I'm still
getting some nodes that have 4 primary shards. This is causing very uneven
load, and some of our applications are suffering significant performance
degradation as a result. Note that we applied the setting and restarted
after disabling allocation.

$ curl -XPUT 'http://localhost:9200/_cluster/settings' -d '{
"transient" : {
"cluster.routing.allocation.disable_allocation": "true"
}
}'


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@tnine
Copy link

tnine commented Feb 18, 2015

Hey @vichargrave

Thanks for the response. I'm able to rebalance shards manually via move, but running the rebalance command has no effect. I have 12 shards as a test in a 6 node environment. With everything balanced evenly, I would expect 2 primaries per node. Instead, I have 4 primaries in 1 node, and 3 in 2 others, causing some strange load profiles. Is there another setting that could possibly be blocking shard-reallocation? I receive no errors in the logs, and the HTTP call returns successfully. @s1monw Is there something I can look at to gather more information?

@vichargrave
Copy link
Author

You won't necessarily get 2 primaries per node. You'll get a mixture of
primary and replica shards on any given node in your cluster. If you see
shards on all your nodes for any given index then you are OK. However, if
you see that a node consistently gets no shards, then you have the problem
I saw.

On Tue, Feb 17, 2015 at 5:59 PM, Todd Nine [email protected] wrote:

Hey @vichargrave https://github.com/vichargrave

Thanks for the response. I'm able to rebalance shards manually via move,
but running the rebalance command has no effect. I have 12 shards as a test
in a 6 node environment. Where everything balanced evenly, I would expect 2
primaries per node. Instead, I have 4 primaries in 1 node, and 3 in 2
others, causing some strange load profiles. Is there another setting that
could possibly be blocking shard-reallocation? I receive no errors in the
logs, and the HTTP call returns successfully.


Reply to this email directly or view it on GitHub
#9023 (comment)
.

@tnine
Copy link

tnine commented Feb 18, 2015

Thanks @vichargrave. It appears we don't have the same issue, I'll move this to the mailing list.

@s1monw
Copy link
Contributor

s1monw commented Feb 18, 2015

@tnine this is a common misunderstanding that the primaries are balanced across the cluster. The primary has to do the same amount of work as any other shard but it's the first one doing it. Balancing based on this is not necessary. we used it as a tie-breaker but removed the primary factor due to this misunderstanding. Are you using upsert / update api by any chance?

@francoisforster
Copy link

I found that the node that is the primary for all shards for my index is using more CPU, has slower search response times and uses more memory. I would benefit from having primaries balanced.

@tnine
Copy link

tnine commented Feb 24, 2015

@s1monw Correct, however, what happens when there are 0 replicas? This case, the primary shard is the only shard doing work. If there are no replicas, I would expect the primary shards to be evenly distributed across the cluster per index. Otherwise, nodes with more than 1 primary shard are going to receive increased traffic when there are no replicas to replicate data to. We're seeing this in production now.

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
In some situations the shard balanceing weight delta becomes negative. Yet,
a negative delta is always treated as `well balanced` which is wrong. I wasn't
able to reproduce the issue in any way other than useing the real world data
from issue elastic#9023. This commit adds a fix for absolute deltas as well as a base
test class that allows to build tests or simulations from the cat API output.

Closes elastic#9023
mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
In some situations the shard balanceing weight delta becomes negative. Yet,
a negative delta is always treated as `well balanced` which is wrong. I wasn't
able to reproduce the issue in any way other than useing the real world data
from issue elastic#9023. This commit adds a fix for absolute deltas as well as a base
test class that allows to build tests or simulations from the cat API output.

Closes elastic#9023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants