Skip to content

Commit

Permalink
Elasticsearch keystore passphrase for startup scripts (elastic#44775)
Browse files Browse the repository at this point in the history
This commit allows a user to provide a keystore password on Elasticsearch
startup, but only prompts when the keystore exists and is encrypted.

The entrypoint in Java code is standard input. When the Bootstrap class is
checking for secure keystore settings, it checks whether or not the keystore
is encrypted. If so, we read one line from standard input and use this as the
password. For simplicity's sake, we allow a maximum passphrase length of 128
characters. (This is an arbitrary limit and could be increased or eliminated.
It is also enforced in the keystore tools, so that a user can't create a
password that's too long to enter at startup.)

In order to provide a password on standard input, we have to account for four
different ways of starting Elasticsearch: the bash startup script, the Windows
batch startup script, systemd startup, and docker startup. We use wrapper
scripts to reduce systemd and docker to the bash case: in both cases, a
wrapper script can read a passphrase from the filesystem and pass it to the
bash script.

In order to simplify testing the need for a passphrase, I have added a
has-passwd command to the keystore tool. This command can run silently, and
exit with status 0 when the keystore has a password. It exits with status 1 if
the keystore doesn't exist or exists and is unencrypted.

A good deal of the code-change in this commit has to do with refactoring
packaging tests to cleanly use the same tests for both the "archive" and the
"package" cases. This required not only moving tests around, but also adding
some convenience methods for an abstraction layer over distribution-specific
commands.

I will write some user-facing documentation for these changes in a follow-up
commit.
  • Loading branch information
williamrandolph committed Nov 18, 2019
1 parent 6bb0e45 commit fafaead
Show file tree
Hide file tree
Showing 31 changed files with 1,125 additions and 175 deletions.
1 change: 1 addition & 0 deletions Vagrantfile
Original file line number Diff line number Diff line change
Expand Up @@ -475,6 +475,7 @@ JAVA
ensure curl
ensure unzip
ensure rsync
ensure expect
installed bats || {
# Bats lives in a git repository....
Expand Down
16 changes: 13 additions & 3 deletions distribution/docker/src/docker/bin/docker-entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -118,8 +118,18 @@ if [[ -f bin/elasticsearch-users ]]; then
# honor the variable if it's present.
if [[ -n "$ELASTIC_PASSWORD" ]]; then
[[ -f /usr/share/elasticsearch/config/elasticsearch.keystore ]] || (run_as_other_user_if_needed elasticsearch-keystore create)
if ! (run_as_other_user_if_needed elasticsearch-keystore list | grep -q '^bootstrap.password$'); then
(run_as_other_user_if_needed echo "$ELASTIC_PASSWORD" | elasticsearch-keystore add -x 'bootstrap.password')
if ! (run_as_other_user_if_needed elasticsearch-keystore has-passwd --silent) ; then
# keystore is unencrypted
if ! (run_as_other_user_if_needed elasticsearch-keystore list | grep -q '^bootstrap.password$'); then
(run_as_other_user_if_needed echo "$ELASTIC_PASSWORD" | elasticsearch-keystore add -x 'bootstrap.password')
fi
else
# keystore requires password
if ! (run_as_other_user_if_needed echo "$KEYSTORE_PASSWORD" \
| elasticsearch-keystore list | grep -q '^bootstrap.password$') ; then
COMMANDS="$(printf "%s\n%s" "$KEYSTORE_PASSWORD" "$ELASTIC_PASSWORD")"
(run_as_other_user_if_needed echo "$COMMANDS" | elasticsearch-keystore add -x 'bootstrap.password')
fi
fi
fi
fi
Expand All @@ -131,4 +141,4 @@ if [[ "$(id -u)" == "0" ]]; then
fi
fi

run_as_other_user_if_needed /usr/share/elasticsearch/bin/elasticsearch "${es_opts[@]}"
run_as_other_user_if_needed /usr/share/elasticsearch/bin/elasticsearch "${es_opts[@]}" <<<"$KEYSTORE_PASSWORD"
4 changes: 4 additions & 0 deletions distribution/packages/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,10 @@ Closure commonPackageConfig(String type, boolean oss, boolean jdk) {
from "${packagingFiles}/systemd/sysctl/elasticsearch.conf"
fileMode 0644
}
into('/usr/share/elasticsearch/bin') {
from "${packagingFiles}/systemd/systemd-entrypoint"
fileMode 0755
}

// ========= sysV init =========
configurationFile '/etc/init.d/elasticsearch'
Expand Down
7 changes: 6 additions & 1 deletion distribution/packages/src/common/scripts/posttrans
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,12 @@ if [ ! -f /etc/elasticsearch/elasticsearch.keystore ]; then
chmod 660 /etc/elasticsearch/elasticsearch.keystore
md5sum /etc/elasticsearch/elasticsearch.keystore > /etc/elasticsearch/.elasticsearch.keystore.initial_md5sum
else
/usr/share/elasticsearch/bin/elasticsearch-keystore upgrade
if /usr/share/elasticsearch/bin/elasticsearch-keystore has-passwd --silent ; then
echo "### Warning: unable to upgrade encrypted keystore" 1>&2
echo " Please run elasticsearch-keystore upgrade and enter password" 1>&2
else
/usr/share/elasticsearch/bin/elasticsearch-keystore upgrade
fi
fi

${scripts.footer}
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ WorkingDirectory=/usr/share/elasticsearch
User=elasticsearch
Group=elasticsearch

ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet
ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet

# StandardOutput is configured to redirect to journalctl since
# some error messages may be logged in standard output before
Expand Down
10 changes: 10 additions & 0 deletions distribution/packages/src/common/systemd/systemd-entrypoint
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/bin/sh

# This wrapper script allows SystemD to feed a file containing a passphrase into
# the main Elasticsearch startup script

if [ -n "$ES_KEYSTORE_PASSPHRASE_FILE" ] ; then
exec /usr/share/elasticsearch/bin/elasticsearch "$@" < "$ES_KEYSTORE_PASSPHRASE_FILE"
else
exec /usr/share/elasticsearch/bin/elasticsearch "$@"
fi
17 changes: 15 additions & 2 deletions distribution/src/bin/elasticsearch
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,19 @@ if [ -z "$ES_TMPDIR" ]; then
ES_TMPDIR=`"$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.TempDirectory`
fi

# get keystore password before setting java options to avoid
# conflicting GC configurations for the keystore tools
unset KEYSTORE_PASSWORD
KEYSTORE_PASSWORD=
if ! echo $* | grep -E -q '(^-h |-h$| -h |--help$|--help |^-V |-V$| -V |--version$|--version )' \
&& "`dirname "$0"`"/elasticsearch-keystore has-passwd --silent
then
if ! read -s -r -p "Elasticsearch keystore password: " KEYSTORE_PASSWORD ; then
echo "Failed to read keystore password on console" 1>&2
exit 1
fi
fi

ES_JVM_OPTIONS="$ES_PATH_CONF"/jvm.options
ES_JAVA_OPTS=`export ES_TMPDIR; "$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JvmOptionsParser "$ES_JVM_OPTIONS"`

Expand All @@ -35,7 +48,7 @@ if ! echo $* | grep -E '(^-d |-d$| -d |--daemonize$|--daemonize )' > /dev/null;
-Des.bundled_jdk="$ES_BUNDLED_JDK" \
-cp "$ES_CLASSPATH" \
org.elasticsearch.bootstrap.Elasticsearch \
"$@"
"$@" <<<"$KEYSTORE_PASSWORD"
else
exec \
"$JAVA" \
Expand All @@ -48,7 +61,7 @@ else
-cp "$ES_CLASSPATH" \
org.elasticsearch.bootstrap.Elasticsearch \
"$@" \
<&- &
<<<"$KEYSTORE_PASSWORD" &
retval=$?
pid=$!
[ $retval -eq 0 ] || exit $retval
Expand Down
2 changes: 1 addition & 1 deletion distribution/src/bin/elasticsearch-cli.bat
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,5 @@ set ES_JAVA_OPTS=-Xms4m -Xmx64m -XX:+UseSerialGC %ES_JAVA_OPTS%
-cp "%ES_CLASSPATH%" ^
"%ES_MAIN_CLASS%" ^
%*

exit /b %ERRORLEVEL%
42 changes: 41 additions & 1 deletion distribution/src/bin/elasticsearch.bat
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ setlocal enabledelayedexpansion
setlocal enableextensions

SET params='%*'
SET checkpassword=Y

:loop
FOR /F "usebackq tokens=1* delims= " %%A IN (!params!) DO (
Expand All @@ -18,6 +19,20 @@ FOR /F "usebackq tokens=1* delims= " %%A IN (!params!) DO (
SET silent=Y
)

IF "!current!" == "-h" (
SET checkpassword=N
)
IF "!current!" == "--help" (
SET checkpassword=N
)

IF "!current!" == "-V" (
SET checkpassword=N
)
IF "!current!" == "--version" (
SET checkpassword=N
)

IF "!silent!" == "Y" (
SET nopauseonerror=Y
) ELSE (
Expand All @@ -41,6 +56,18 @@ IF ERRORLEVEL 1 (
EXIT /B %ERRORLEVEL%
)

SET KEYSTORE_PASSWORD=
IF "%checkpassword%"=="Y" (
CALL "%~dp0elasticsearch-keystore.bat" has-passwd --silent
IF !ERRORLEVEL! EQU 0 (
SET /P KEYSTORE_PASSWORD=Elasticsearch keystore password:
IF !ERRORLEVEL! NEQ 0 (
ECHO Failed to read keystore password on standard input
EXIT /B !ERRORLEVEL!
)
)
)

if not defined ES_TMPDIR (
for /f "tokens=* usebackq" %%a in (`CALL %JAVA% -cp "!ES_CLASSPATH!" "org.elasticsearch.tools.launchers.TempDirectory"`) do set ES_TMPDIR=%%a
)
Expand All @@ -54,7 +81,20 @@ if "%MAYBE_JVM_OPTIONS_PARSER_FAILED%" == "jvm_options_parser_failed" (
exit /b 1
)

%JAVA% %ES_JAVA_OPTS% -Delasticsearch -Des.path.home="%ES_HOME%" -Des.path.conf="%ES_PATH_CONF%" -Des.distribution.flavor="%ES_DISTRIBUTION_FLAVOR%" -Des.distribution.type="%ES_DISTRIBUTION_TYPE%" -Des.bundled_jdk="%ES_BUNDLED_JDK%" -cp "%ES_CLASSPATH%" "org.elasticsearch.bootstrap.Elasticsearch" !newparams!
rem windows batch pipe will choke on special characters in strings
SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^^=^^^^!
SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^&=^^^&!
SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^|=^^^|!
SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^<=^^^<!
SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^>=^^^>!
SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^\=^^^\!

ECHO.!KEYSTORE_PASSWORD!| %JAVA% %ES_JAVA_OPTS% -Delasticsearch ^
-Des.path.home="%ES_HOME%" -Des.path.conf="%ES_PATH_CONF%" ^
-Des.distribution.flavor="%ES_DISTRIBUTION_FLAVOR%" ^
-Des.distribution.type="%ES_DISTRIBUTION_TYPE%" ^
-Des.bundled_jdk="%ES_BUNDLED_JDK%" ^
-cp "%ES_CLASSPATH%" "org.elasticsearch.bootstrap.Elasticsearch" !newparams!

endlocal
endlocal
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.elasticsearch.common.settings;

import joptsimple.OptionSet;
import org.elasticsearch.cli.KeyStoreAwareCommand;
import org.elasticsearch.cli.Terminal;
import org.elasticsearch.cli.UserException;
import org.elasticsearch.env.Environment;

import java.nio.file.Path;

public class HasPasswordKeyStoreCommand extends KeyStoreAwareCommand {

static final int NO_PASSWORD_EXIT_CODE = 1;

HasPasswordKeyStoreCommand() {
super("Succeeds if the keystore exists and is password-protected, " +
"fails with exit code " + NO_PASSWORD_EXIT_CODE + " otherwise.");
}

@Override
protected void execute(Terminal terminal, OptionSet options, Environment env) throws Exception {
final Path configFile = env.configFile();
final KeyStoreWrapper keyStore = KeyStoreWrapper.load(configFile);

// We handle error printing here so we can respect the "--silent" flag
// We have to throw an exception to get a nonzero exit code
if (keyStore == null) {
terminal.errorPrintln(Terminal.Verbosity.NORMAL, "ERROR: Elasticsearch keystore not found");
throw new UserException(NO_PASSWORD_EXIT_CODE, null);
}
if (keyStore.hasPassword() == false) {
terminal.errorPrintln(Terminal.Verbosity.NORMAL, "ERROR: Keystore is not password-protected");
throw new UserException(NO_PASSWORD_EXIT_CODE, null);
}

terminal.println(Terminal.Verbosity.NORMAL, "Keystore is password-protected");
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ private KeyStoreCli() {
subcommands.put("remove", new RemoveSettingKeyStoreCommand());
subcommands.put("upgrade", new UpgradeKeyStoreCommand());
subcommands.put("passwd", new ChangeKeyStorePasswordCommand());
subcommands.put("has-passwd", new HasPasswordKeyStoreCommand());
}

public static void main(String[] args) throws Exception {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,17 +29,24 @@
import org.junit.After;
import org.junit.Before;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.nio.file.FileSystem;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;

import static org.hamcrest.Matchers.equalTo;

public class BootstrapTests extends ESTestCase {
Environment env;
List<FileSystem> fileSystems = new ArrayList<>();

private static final int MAX_PASSPHRASE_LENGTH = 10;

@After
public void closeMockFileSystems() throws IOException {
IOUtils.close(fileSystems);
Expand All @@ -66,4 +73,43 @@ public void testLoadSecureSettings() throws Exception {
assertTrue(Files.exists(configPath.resolve("elasticsearch.keystore")));
}
}

public void testReadCharsFromStdin() throws Exception {
assertPassphraseRead("hello", "hello");
assertPassphraseRead("hello\n", "hello");
assertPassphraseRead("hello\r\n", "hello");

assertPassphraseRead("hellohello", "hellohello");
assertPassphraseRead("hellohello\n", "hellohello");
assertPassphraseRead("hellohello\r\n", "hellohello");

assertPassphraseRead("hello\nhi\n", "hello");
assertPassphraseRead("hello\r\nhi\r\n", "hello");
}

public void testPassphraseTooLong() throws Exception {
byte[] source = "hellohello!\n".getBytes(StandardCharsets.UTF_8);
try (InputStream stream = new ByteArrayInputStream(source)) {
expectThrows(RuntimeException.class, "Password exceeded maximum length of 10",
() -> Bootstrap.readPassphrase(stream, MAX_PASSPHRASE_LENGTH));
}
}

public void testNoPassPhraseProvided() throws Exception {
byte[] source = "\r\n".getBytes(StandardCharsets.UTF_8);
try (InputStream stream = new ByteArrayInputStream(source)) {
expectThrows(RuntimeException.class, "Keystore passphrase required but none provided.",
() -> Bootstrap.readPassphrase(stream, MAX_PASSPHRASE_LENGTH));
}
}

private void assertPassphraseRead(String source, String expected) {
try (InputStream stream = new ByteArrayInputStream(source.getBytes(StandardCharsets.UTF_8))) {
SecureString result = Bootstrap.readPassphrase(stream, MAX_PASSPHRASE_LENGTH);
assertThat(result, equalTo(expected));
} catch (IOException e) {
throw new RuntimeException(e);
}
}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.elasticsearch.common.settings;

import org.elasticsearch.cli.Command;
import org.elasticsearch.cli.UserException;
import org.elasticsearch.env.Environment;

import java.util.Map;

import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.emptyString;
import static org.hamcrest.Matchers.nullValue;

public class HasPasswordKeyStoreCommandTests extends KeyStoreCommandTestCase {
@Override
protected Command newCommand() {
return new HasPasswordKeyStoreCommand() {
@Override
protected Environment createEnv(Map<String, String> settings) throws UserException {
return env;
}
};
}

public void testFailsWithNoKeystore() throws Exception {
UserException e = expectThrows(UserException.class, this::execute);
assertEquals("Exit code should be 1", HasPasswordKeyStoreCommand.NO_PASSWORD_EXIT_CODE, e.exitCode);
assertThat("Exception should have null message", e.getMessage(), is(nullValue()));
}

public void testFailsWhenKeystoreLacksPassword() throws Exception {
createKeystore("");
UserException e = expectThrows(UserException.class, this::execute);
assertEquals("Exit code should be 1", HasPasswordKeyStoreCommand.NO_PASSWORD_EXIT_CODE, e.exitCode);
assertThat("Exception should have null message", e.getMessage(), is(nullValue()));
}

public void testSucceedsWhenKeystoreHasPassword() throws Exception {
createKeystore("password");
String output = execute();
assertThat(output, containsString("Keystore is password-protected"));
}

public void testSilentSucceedsWhenKeystoreHasPassword() throws Exception {
createKeystore("password");
String output = execute("--silent");
assertThat(output, is(emptyString()));
}
}
Loading

0 comments on commit fafaead

Please sign in to comment.