"could not find java; set JAVA_HOME" from Debian package install script breaks cloud-init, breaks automation

[minfrin]

Elasticsearch version (bin/elasticsearch --version): 6.8.5

Plugins installed: [elasticsearch-analysis-icu]

JVM version (java -version): openjdk version "11.0.4" 2019-07-16

OS version (uname -a if on a Unix-like system): Linux x-jm-unstable-black-els01 4.15.0-1056-aws #58-Ubuntu SMP Tue Nov 26 15:14:34 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

When an attempt is made to install elasticsearch v6.8.5 from a Debian package, the installation fails because JAVA_HOME has not been set.

This creates a chicken and egg problem. Automation tooling expects the package to be installed first, and then configured after, Elasticsearch expects itself to be configured first, then installed after.

As a result, automated installations via cloud-init break.

Steps to reproduce:

1. Tell cloud-init to install the "elasticsearch" package.
2. Instantiate the machine.
3. Cloud-init fails with the following fatal error, and all box configuration is halted:

Selecting previously unselected package elasticsearch.
Preparing to unpack .../080-elasticsearch_6.8.5_all.deb ...
could not find java; set JAVA_HOME
dpkg: error processing archive /tmp/apt-dpkg-install-R5VwK1/080-elasticsearch_6.8.5_all.deb (--unpack):
new elasticsearch package pre-installation script subprocess returned error exit status 1

To fix this, the debian package install scripts must not fail with a fatal error if the system is not yet sane. The system will become sane later when configuration is completed.

Provide logs (if relevant):

[minfrin]

After the installation has failed, this puts the apt packaging system in a state where no further packages can be installed.

The box appears to be unrecoverable.

root@x-jm-unstable-black-els01:~# apt --fix-broken install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  elasticsearch
The following NEW packages will be installed:
  elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
241 not fully installed or removed.
Need to get 0 B/149 MB of archives.
After this operation, 240 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
(Reading database ... 106485 files and directories currently installed.)
Preparing to unpack .../elasticsearch_6.8.5_all.deb ...
could not find java; set JAVA_HOME
dpkg: error processing archive /var/cache/apt/archives/elasticsearch_6.8.5_all.deb (--unpack):
 new elasticsearch package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/elasticsearch_6.8.5_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

[minfrin]

It looks like the regression was added here: #31343

We were previously using elasticsearch v6.2.2, and this worked.

What seems to be happening is that if you try and install openjdk and elasticsearch together in the same apt-get install, which is how cloud-init works, the preinstall script for elasticsearch appears to run before the postinstall script for openjdk which sets up /etc/alternatives and /usr/bin/java.

The installation then fails. The failure in elasticsearch preinst causes the openjdk postinst to be skipped, and as a result the system remains permanently broken.

The root cause looks like it's because elasticsearch doesn't declare itself dependent on a jdk of any kind:

root@x-jm-unstable-black-els01:~# apt-cache depends elasticsearch
elasticsearch
Depends: bash
Depends: libc6
Depends: adduser
Depends: coreutils
Conflicts:

As a result, apt-get doesn't know that openjdk needs to go before elasticsearch, and chaos ensues.

It looks like the fix is for elasticsearch to declare it needs a JDK.

[jasontedor]

We are not going to modify 6.8 to require a JDK. In 7.x we have solved this by bundling a JDK with Elasticsearch, that’s our long-term fix for this problem.

[minfrin]

Unfortunately our vendor does not support elasticsearch 7.x at this time.

How do you propose we solve this problem?

[minfrin]

I've added #49983 to properly track the problem, with the solution as per debian packaging guidelines.