Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow configuring Elasticsearch cluster name #1478

Merged
merged 11 commits into from
Jun 13, 2022
Merged

Allow configuring Elasticsearch cluster name #1478

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
07ca698
Allow configuring Elasticsearch cluster name
dnhatn Apr 17, 2022
198618e
Fix Docker cluster name
dnhatn Apr 20, 2022
30d4e50
Fix tests
dnhatn Apr 20, 2022
daa41b9
Lint
dnhatn Apr 20, 2022
7d3d35f
Add parameter to `race` sub command
dnhatn May 2, 2022
bfad99a
fix docs
dnhatn May 2, 2022
5127ed3
Merge remote-tracking branch 'elastic/master' into cluster_name
dnhatn Jun 13, 2022
0f84fb9
Add cluster-name opts to install and race parsers
dnhatn Jun 13, 2022
b630a50
Keep old docs
dnhatn Jun 13, 2022
6946dd0
Add an example with two clusters
dnhatn Jun 13, 2022
538418d
typo
dnhatn Jun 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/cluster_management.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ In this section we will setup a single Elasticsearch node locally, run a benchma

First we need to install Elasticearch::

esrally install --quiet --distribution-version=7.4.2 --node-name="rally-node-0" --network-host="127.0.0.1" --http-port=39200 --master-nodes="rally-node-0" --seed-hosts="127.0.0.1:39300"
esrally install --quiet --distribution-version=8.1.0 --node-name="rally-node-0" --cluster-name=rally-benchmark --network-host="127.0.0.1" --http-port=39200 --master-nodes="rally-node-0" --seed-hosts="127.0.0.1:39300"
dnhatn marked this conversation as resolved.
Show resolved Hide resolved

The parameter ``--network-host`` defines the network interface this node will bind to and ``--http-port`` defines which port will be exposed for HTTP traffic. Rally will automatically choose the transport port range as 100 above (39300). The parameters ``--master-nodes`` and ``--seed-hosts`` are necessary for the discovery process. Please see the respective Elasticsearch documentation on `discovery <https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>`_ for more details.

Expand Down Expand Up @@ -67,11 +67,11 @@ Levelling Up: Benchmarking a Cluster
This approach of being able to manage individual cluster nodes shows its power when we want to setup a cluster consisting of multiple nodes. At the moment Rally only supports a uniform cluster architecture but with this approach we can also setup arbitrarily complex clusters. The following examples shows how to setup a uniform three node cluster on three machines with the IPs ``192.168.14.77``, ``192.168.14.78`` and ``192.168.14.79``. On each machine we will issue the following command (pick the right one per machine)::

# on 192.168.14.77
export INSTALLATION_ID=$(esrally install --quiet --distribution-version=7.4.2 --node-name="rally-node-0" --network-host="192.168.14.77" --http-port=39200 --master-nodes="rally-node-0,rally-node-1,rally-node-2" --seed-hosts="192.168.14.77:39300,192.168.14.78:39300,192.168.14.79:39300" | jq --raw-output '.["installation-id"]')
export INSTALLATION_ID=$(esrally install --quiet --distribution-version=8.1.0 --node-name="rally-node-0" --cluster-name=rally-benchmark --network-host="192.168.14.77" --http-port=39200 --master-nodes="rally-node-0,rally-node-1,rally-node-2" --seed-hosts="192.168.14.77:39300,192.168.14.78:39300,192.168.14.79:39300" | jq --raw-output '.["installation-id"]')
# on 192.168.14.78
export INSTALLATION_ID=$(esrally install --quiet --distribution-version=7.4.2 --node-name="rally-node-1" --network-host="192.168.14.78" --http-port=39200 --master-nodes="rally-node-0,rally-node-1,rally-node-2" --seed-hosts="192.168.14.77:39300,192.168.14.78:39300,192.168.14.79:39300" | jq --raw-output '.["installation-id"]')
export INSTALLATION_ID=$(esrally install --quiet --distribution-version=8.1.0 --node-name="rally-node-1" --cluster-name=rally-benchmark --network-host="192.168.14.78" --http-port=39200 --master-nodes="rally-node-0,rally-node-1,rally-node-2" --seed-hosts="192.168.14.77:39300,192.168.14.78:39300,192.168.14.79:39300" | jq --raw-output '.["installation-id"]')
# on 192.168.14.79
export INSTALLATION_ID=$(esrally install --quiet --distribution-version=7.4.2 --node-name="rally-node-2" --network-host="192.168.14.79" --http-port=39200 --master-nodes="rally-node-0,rally-node-1,rally-node-2" --seed-hosts="192.168.14.77:39300,192.168.14.78:39300,192.168.14.79:39300" | jq --raw-output '.["installation-id"]')
export INSTALLATION_ID=$(esrally install --quiet --distribution-version=8.1.0 --node-name="rally-node-2" --cluster-name=rally-benchmark --network-host="192.168.14.79" --http-port=39200 --master-nodes="rally-node-0,rally-node-1,rally-node-2" --seed-hosts="192.168.14.77:39300,192.168.14.78:39300,192.168.14.79:39300" | jq --raw-output '.["installation-id"]')
dnhatn marked this conversation as resolved.
Show resolved Hide resolved

Then we pick a random race id, e.g. ``fb38013d-5d06-4b81-b81a-b61c8c10f6e5`` and set it on each machine (including the machine where will generate load)::

Expand Down
11 changes: 10 additions & 1 deletion docs/command_line_reference.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ This will show the path to the default distribution::

This subcommand can be used to install a single Elasticsearch node. Example::

esrally install --quiet --distribution-version=7.4.2 --node-name="rally-node-0" --network-host="127.0.0.1" --http-port=39200 --master-nodes="rally-node-0" --seed-hosts="127.0.0.1:39300"
esrally install --quiet --distribution-version=8.1.0 --node-name="rally-node-0" --cluster-name=rally-benchmark --network-host="127.0.0.1" --http-port=39200 --master-nodes="rally-node-0" --seed-hosts="127.0.0.1:39300"
dnhatn marked this conversation as resolved.
Show resolved Hide resolved

This will output the id of this installation::

Expand Down Expand Up @@ -272,6 +272,15 @@ A unique identifier for an installation. This id is automatically generated by R

Used to specify the current node's name in the cluster when it is setup via the ``install`` subcommand.

``cluster-name``
~~~~~~~~~~~~~~~~

.. warning::

This command line parameter is experimental. Expect the functionality and the command line interface to change significantly even in patch releases.
dnhatn marked this conversation as resolved.
Show resolved Hide resolved

Used to specify the cluster name of the current node when it is setup via the ``install`` subcommand.
dnhatn marked this conversation as resolved.
Show resolved Hide resolved

``network-host``
~~~~~~~~~~~~~~~~

Expand Down
4 changes: 2 additions & 2 deletions esrally/mechanic/provisioner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@

def local(cfg, car, plugins, ip, http_port, all_node_ips, all_node_names, target_root, node_name):
distribution_version = cfg.opts("mechanic", "distribution.version", mandatory=False)
cluster_name = cfg.opts("mechanic", "cluster.name", default_value="rally=benchmark", mandatory=False)
cluster_name = cfg.opts("mechanic", "cluster.name")

node_root_dir = os.path.join(target_root, node_name)

Expand All @@ -49,7 +49,7 @@ def local(cfg, car, plugins, ip, http_port, all_node_ips, all_node_names, target

def docker(cfg, car, ip, http_port, target_root, node_name):
distribution_version = cfg.opts("mechanic", "distribution.version", mandatory=False)
cluster_name = cfg.opts("mechanic", "cluster.name", default_value="rally=benchmark", mandatory=False)
cluster_name = cfg.opts("mechanic", "cluster.name")
rally_root = cfg.opts("node", "rally.root")

node_root_dir = os.path.join(target_root, node_name)
Expand Down
6 changes: 6 additions & 0 deletions esrally/rally.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -535,6 +535,11 @@ def add_track_source(subparser):
help="Define a comma-separated list of key:value pairs that are injected verbatim to all plugins as variables.",
default="",
)
race_parser.add_argument(
"--cluster-name",
help="The name of this Elasticsearch cluster",
default="rally-benchmark",
)
dnhatn marked this conversation as resolved.
Show resolved Hide resolved
race_parser.add_argument(
"--target-hosts",
help="Define a comma-separated list of host:port pairs which should be targeted if using the pipeline 'benchmark-only' "
Expand Down Expand Up @@ -976,6 +981,7 @@ def dispatch_sub_command(arg_parser, args, cfg):
cfg.add(config.Scope.applicationOverride, "mechanic", "plugin.params", opts.to_dict(args.plugin_params))
cfg.add(config.Scope.applicationOverride, "mechanic", "preserve.install", convert.to_bool(args.preserve_install))
cfg.add(config.Scope.applicationOverride, "mechanic", "skip.rest.api.check", convert.to_bool(args.skip_rest_api_check))
cfg.add(config.Scope.applicationOverride, "mechanic", "cluster.name", args.cluster_name)

configure_reporting_params(args, cfg)
race(cfg, args.kill_running_processes)
Expand Down
3 changes: 2 additions & 1 deletion it/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ def install(self, distribution_version, node_name, car, http_port):
output = process.run_subprocess_with_output(
"esrally install --configuration-name={cfg} --quiet --distribution-version={dist} --build-type=tar "
"--http-port={http_port} --node={node_name} --master-nodes={node_name} --car={car} "
'--seed-hosts="127.0.0.1:{transport_port}"'.format(
'--seed-hosts="127.0.0.1:{transport_port}" --cluster-name={cfg}'.format(
cfg=self.cfg,
dist=distribution_version,
http_port=http_port,
Expand All @@ -178,6 +178,7 @@ def start(self, race_id):
raise AssertionError("Failed to start Elasticsearch test cluster.")
es = client.EsClientFactory(hosts=[{"host": "127.0.0.1", "port": self.http_port}], client_options={}).create()
client.wait_for_rest_layer(es)
assert es.info()["cluster_name"] == self.cfg

def stop(self):
if self.installation_id:
Expand Down