Systemd startup timeouts after upgrade to 7.4

[PhaedrusTheGreek]

It has been observed/reported in a few separate environments where after an upgrade from Elasticsearch 7.3 to 7.4, systemd will kill the elasticsearch process before it finishes starting up.

The end of the log looks like this:

[2019-11-25T07:16:46,095][DEBUG][o.e.a.ActionModule ] [node1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-11-25T07:17:22,306][INFO ][o.e.x.m.p.NativeController] [node1] Native controller process has stopped - no new native processes can be started

Trace inspection of the log reveals that the ES node is busy upgrading.

Systemd requires that processes signal after successful startup with sd_notify(READY) before TimeoutStartSec, which in ES doesn't happen until metadata upgrades are complete. It seems that we need to update SystemdPlugin to support EXTEND_TIMEOUT_USEC in the event of any expected startup delays such as upgrade.

[PhaedrusTheGreek]

Also curious here is why this hasn't been observed prior to 7.4. Has something changed in this area?

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Packaging)

[jasontedor]

Also curious here is why this hasn't been observed prior to 7.4. Has something changed in this area?

Yes, the sd_notify functionality was not used until 7.4.0: #44673

[ppf2]

(Please ignore previous notification. I was reading 7.4 as 7.14 😄 ). This was added back in 7.4. Though some users may not hit this until later on nodes that are slow to start :)

[jasontedor]

Since 7.5.1/7.6 (#49784), nodes that are slow to start should not present an issue.

[ppf2]

Thanks Jason! It looks like we have user run into this on 7.14. I will collect the details and file an issue accordingly for investigation.