[Self-managed]: Unable to install fleet-server on 8.4.2

[ghost]

Kibana version: 8.4.2 Kibana self-managed environment

Host OS and Browser version: Windows

Build details:
VERSION: 8.4.2 Snapshot self-managed
BUILD: 55523
COMMIT: d34da2c98a97aac80c2b9e8ab197c76cc149574e

Preconditions:

• 8.4.2 Kibana self-managed environment should be available.

Steps to reproduce:

1. On fresh Kibana setup navigate to Fleet tab.
2. Create Fleet Server Policy.
3. Select 'Quick start'
4. Add Fleet-server host say https//10.10.6.10:8220
5. Copy the command in the cli.
6. Observe fleet-server is not installed.

Expected Result:
Fleet-server should be installed using Quick start

Screenshots:

Related issue: #1825

[amolnater-qasource]

Secondary review for this issue is Done.

FYI @joshdover @jlind23

[jlind23]

@amolnater-qasource seems to be a certificate issue. @michel-laterman @narph can one of you take a look at it please?

[narph]

@amolnater-qasource , @deepikakeshav-qasource I've tried reproducing this on Windows 11 using:

https://snapshots.elastic.co/8.4.2-36096067/downloads/kibana/kibana-8.4.2-SNAPSHOT-windows-x86_64.zip with the commit you mentioned
https://snapshots.elastic.co/8.4.2-36096067/downloads/elastic-agent/elastic-agent-8.4.2-SNAPSHOT-windows-x86_64.zip
https://snapshots.elastic.co/8.4.2-36096067/downloads/elasticsearch/elasticsearch-8.4.2-SNAPSHOT-windows-x86_64.zip

installed es and kibana, followed the steps you mentioned above and then ran:

PS C:\Users\...\Desktop\test\test\842\elastic-agent-8.4.2-SNAPSHOT-windows-x86_64> .\elastic-agent.exe install `
>>   --fleet-server-es=http://localhost:9200 `
>>   --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjMyMzkzMDQ4ODQ6Qi1OUlFXX3RRS09XNVFyVUJjY3daUQ `
>>   --fleet-server-policy=fleet-server-policy
Elastic Agent will be installed at C:\Program Files\Elastic\Agent and will run as a service. Do you want to continue? [Y/n]:y
{"log.level":"info","@timestamp":"2022-09-15T12:56:32.744+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":403},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-15T12:56:34.144+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":742},"message":"Waiting for Elastic Agent to start","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-15T12:56:36.148+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-15T12:56:40.160+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":773},"message":"Fleet Server - Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-15T12:56:41.108+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":471},"message":"Starting enrollment to URL: https://DESKTOP-K76UDQL:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-15T12:56:42.273+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":273},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.
PS C:\Users\...\Desktop\test\test\842\elastic-agent-8.4.2-SNAPSHOT-windows-x86_64>

Is there any step I might have missed during the test? Are those the right artifacts you have been using, can you still reproduce the issue?

[narph]

I had a look at the issue you linked and I there the communication is encrypted, will test further with certificates but it's worth mentioning that the ticket description.

[amolnater-qasource]

Hi @narph
We have revalidated this issue on 8.5 Build Candidate Kibana self-managed environment and found this issue still reproducible:

• We are still not able to install Fleet server and getting x509 certificate error.

Further as per our observation for your comment #1869 (comment)

--fleet-server-es=http://localhost:9200

We have setup our kibana using Security on by default feature at link https://github.com/elastic/obs-infraobs-team/issues/565
So our elasticsearch is setup on https.

Screenshot:

Build details:
8.4.2 BC1
BUILD: 55523
COMMIT: d34da2c98a97aac80c2b9e8ab197c76cc149574e

Please let us know if anything else is required from our end.
Thanks

[joshdover]

@amolnater-qasource Are we sure this isn't expected behavior? I'd expect that Fleet Server would refuse to connect to an ES with a self-signed cert unless one of the following CLI flags are passed: --insecure or --fleet-server-es-ca-fingerprint

[narph]

@amolnater-qasource , @joshdover took some time but I wanted to test this further.

I've configured tls/ssl for elasticsearch/kibana/fleet server and then ran the following:

 ./elastic-agent install --url=https://127.0.0.1:8220  --fleet-server-es=https://127.0.0.1:9200  --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjM1OTQ2NjYyOTg6N1J0X2RvX2lTWXVpSEhpS0hxcWg1dw  --fleet-server-es-ca=C:\Users\...\Desktop\test\test\842\elasticsearch-8.4.2-SNAPSHOT\ca\ca.crt --certificate-authorities=C:\Users\...\Desktop\test\test\842\elasticsearch-8.4.2-SNAPSHOT\ca\ca.crt  --fleet-server-cert=C:\Users\...\Desktop\test\test\842\elasticsearch-8.4.2-SNAPSHOT\config\fleet-server.crt --fleet-server-cert-key=C:\Users\...\Desktop\test\test\842\elasticsearch-8.4.2-SNAPSHOT\config\fleet-server.key   --fleet-server-policy=fleet-server-policy
Elastic Agent will be installed at C:\Program Files\Elastic\Agent and will run as a service. Do you want to continue? [Y/n]:y
....
{"log.level":"info","@timestamp":"2022-09-19T15:54:04.749+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":273},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.

which means tls enabled should work.

then ran:

./elastic-agent install --url=https://127.0.01:8220  --fleet-server-es=https://127.0.0.1:9200    --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjM1OTQ2NjYyOTg6N1J0X2RvX2lTWXVpSEhpS0hxcWg1dw  --fleet-server-policy=fleet-server-policy --insecure

Elastic Agent will be installed at C:\Program Files\Elastic\Agent and will run as a service. Do you want to continue? [Y/n]:
{"log.level":"info","@timestamp":"2022-09-19T16:01:12.342+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":403},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-19T16:01:13.827+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":759},"message":"Waiting for Elastic Agent to start Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-19T16:01:15.838+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-19T16:01:19.849+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Error - x509: certificate signed by unknown authority","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-19T16:01:43.874+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Restarting","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html

and got the same error as in the ticket description.

then ran:

 ./elastic-agent install --url=https://127.0.01:8220  --fleet-server-es=https://127.0.0.1:9200    --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjM1OTQ2NjYyOTg6N1J0X2RvX2lTWXVpSEhpS0hxcWg1dw  --fleet-server-policy=fleet-server-policy  --fleet-server-es-insecure

Elastic Agent will be installed at C:\Program Files\Elastic\Agent and will run as a service. Do you want to continue? [Y/n]:
{"log.level":"info","@timestamp":"2022-09-19T16:03:41.755+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":403},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
...
{"log.level":"info","@timestamp":"2022-09-19T16:03:50.764+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":273},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.

There are 2 separate flags here: --insecure and --fleet-server-es-insecure

--insecure Allow insecure connection to fleet-server
--fleet-server-es-insecure Disables validation of certificates

When it comes bypassing es certificates the --fleet-server-es-insecure can be used. can you give it a try?

[amolnater-qasource]

Hi @narph @joshdover
Thank you for looking into this.

We have revalidated this issue by running Fleet server installation command using --fleet-server-es-insecure and found it working fine.

• We are successfully able to install fleet server using --fleet-server-es-insecure flag.

Screenshot:

Further we have observed that fingerprint flag is missing from elasticsearch output and is also not available under Fleet Server install command. As per our understanding fingerprint flag should be available as per security on by default feature.

Screenshot:

We have revalidated the guide for Fleet Server and observed that Fingerprint flag is available in installation command.

Please let us know if separate issue is required to be logged for this.

Thanks!

[narph]

@amolnater-qasource can you create a separate issue regarding the fingerprint flag and the steps you are taking to reproduce it? it will be easier to follow on a separate thread

[amolnater-qasource]

Thank you for the feedback @narph
We have logged a separate issue fingerprint flag under elastic/kibana#141046

Please let us know if anything else is required from our end.
Thanks!

[amolnater-qasource]

Hi @narph

We have attempted to install fleet server using --fleet-server-es-insecure flag on 8.5 BC2 Kibana self-managed environment and observed that fleet server gets stuck in updating state after running install command.

Build details:
VERSION: 8.5.0 BC2
BUILD: 56806
COMMIT: dc769f45a5a6dafb0a8c8f0c0cabcced4df45e11

Screenshot:

Thanks

[narph]

@amolnater-qasource this only happens when using the --fleet-server-es-insecure flag? does it work when entering the certificates configuration?

[amolnater-qasource]

Hi @narph

We are unable to install fleet server without using --fleet-server-es-insecure flag and actual reported issue is reproducible to us.
Further using --fleet-server-es-insecure it gets stuck in updating state as shared #1869 (comment) .
Please let us know if anything else is required from our end.

Thanks

[joshdover]

@nchaulet do you know what may be going on here?

[nathanatimu]

Hi,
I'm facing also this issue. detail are provided here:
https://discuss.elastic.co/t/elastic-agent-error-dialing-x509-certificate-signed-by-unknown-authority/315675/3

in short: command:
./elastic-agent install --url=https://ip-adress-of-other-server:8220 --fleet-server-es=https://machine-host-name:9200 --fleet-server-service-token=token --fleet-server-policy=fleet-server-policy --certificate-authorities=/etc/pki/elasticsearch/ca.crt --fleet-server-cert=/etc/pki/elasticsearch/fleet-server.crt --fleet-server-cert-key=/etc/pki/elasticsearch/fleet-server.key --fleet-server-es-insecure

gives a a x509 error. while not expecting it here.

[cmacknz]

Also seems to be reported in #1866

[nathanatimu]

@cmacknz yeah, only difference is that the --fleet-server-es-insecure flag doesn't provide a solution

[nchaulet]

@nchaulet do you know what may be going on here?

The fact that the trusted fingerprint is missing is related to that bug elastic/kibana#142109

Using --fleet-server-es-insecure seems to be the correct workaround, The fact that the agent is stuck in updating means the agent is not able to checkin, they should probably check the fleet server url here I saw this in the discussion that seems incorrect to me url=https://127.0.01:8220

[amolnater-qasource]

Hi Team
We have attempted to install fleet-server using production mode with certs and fleet server stuck in Updating state.

• We have run below command:

 .\elastic-agent.exe install --url=https://3.85.118.153:8220 `
   --fleet-server-es=https://172.31.20.150:9200 `
   --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjQ4Njg2MjIyMDg6UWNDMGFnZnZTZTZVcXdmVy1yY3A5dw `
   --fleet-server-policy=fleet-server-policy `
   --certificate-authorities=C:\elk\elasticsearch\ca\ca.crt `
   --fleet-server-es-ca=C:\elk\elasticsearch\ca\ca.crt `
   --fleet-server-cert=C:\elk\elasticsearch\fleet-server\fleet-server.crt `
   --fleet-server-cert-key=C:\elk\elasticsearch\fleet-server\fleet-server.key --fleet-server-es-insecure

Build details:

BUILD: 56806
COMMIT: dc769f45a5a6dafb0a8c8f0c0cabcced4df45e11

Screenshots:

Logs:
elastic-agent-diagnostics-2022-10-04T08-59-35Z-00.zip

Please let us know if we are missing anything.
Thanks

[narph]

@amolnater-qasource looking at the errors in the logs you linked:

{"log.level":"info","@timestamp":"2022-10-04T08:59:15.451Z","log.origin":{"file.name":"operation/operator.go","file.line":387},"message":"operation 'operation-install' skipped for filebeat.8.5.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-10-04T08:59:16.030Z","log.origin":{"file.name":"status/reporter.go","file.line":326},"message":"Elastic Agent status changed to \"error\": \"app filebeat--8.5.0--36643631373035623733363936343635-dbac99b8: \\\"filebeat_monitoring\\\" failed to prepare monitor for \\\"Filebeat\\\": failed to create a directory \\\"\\\": mkdir : The system cannot find the path specified.\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-04T08:59:16.036Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-10-04T08:59:16Z - message: Application: filebeat--8.5.0--36643631373035623733363936343635[944cd610-db45-42be-a49b-9681d0c614f5]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}

I see they match the issue #1934 and there is a fix PR elastic/elastic-agent#1371. I believe all should be good now. Let me know if you can still reproduce it after the fix

[amolnater-qasource]

Hi Team
We have revalidated this on 8.5 BC4 kibana self-managed environment and found it fixed now.

Observations:

• Fleet-server is installed Healthy using Quickstart mode.

Build details:
BUILD: 57008
COMMIT: b2b4d5c5b4742fcfe5699dbcbffb9a98a5f06b5f

Screenshots:

Hence we are closing this issue and are marking this issue as QA:Validated.
Thanks

[ghost]

Bug Conversion

• Test-Case not required as this particular checkpoint is already covered in the following testcase:
  • https://elastic.testrail.io/index.php?/cases/view/147843&group_by=cases:section_id&group_order=asc&display_deleted_cases=0&group_id=30364

Thanks!