[Heartbeat] Build elastic agent with synthetics support / offline mode #27052

andrewvc · 2021-07-27T03:10:02Z

Creates a new offline tagged docker image that includes the dependencies required for synthetics. Building elastic-agent docker images will now create additional -offline tagged images with these extras. We may need to do additional work to ensure that these additional images are incorporated into the release process and onto the public website.

These changes also set the ELASTIC_SYNTHETICS_CAPABLE env flag, as done in heartbeat, to enable synthetics features in the docker environment. This dovetails with the work @dominiqueclarke is doing in elastic/integrations#1064 and elsewhere

Fixes #22932

CC @ruflin @v1v @mostlyjason @drewpost @vigneshshanmugam @dominiqueclarke

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~- [ ] I have made corresponding change to the default configuration files~~
~~- [ ] I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

TODO: Test this with the upcoming synthetics support in integrations (see elastic/integrations#1064)

elasticmachine · 2021-07-27T03:10:04Z

Pinging @elastic/uptime (Team:Uptime)

elasticmachine · 2021-07-27T04:23:37Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-10T20:09:47.259+0000
Duration: 214 min 21 sec
Commit: 3fa0117

Test stats 🧪

Test	Results
Failed	0
Passed	52978
Skipped	5318
Total	58296

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	52978
Skipped	5318
Total	58296

paulb-elastic · 2021-07-27T09:03:49Z

Linking to #26398

mostlyjason · 2021-07-27T15:10:08Z

I see a checkbox for documentation. We documented the docker container here elastic/observability-docs#622. Can you add content for the offline mode or at least create an issue to track adding the docs?

ruflin · 2021-07-28T08:26:40Z

As a first step I think the most important part is that it does not affect the current build we have. Would be good to validate this with a package build. I did not test it locally or go into the detail about overall LGTM.

@andrewkroh @blakerouse Could you take a closer look at this as you are more familiar with the agent docker configs and the build system overall and might spot some side effects of this?

ruflin · 2021-07-28T08:27:51Z

dev-tools/mage/dockerbuilder.go

@@ -58,6 +59,11 @@ func newDockerBuilder(spec PackageSpec) (*dockerBuilder, error) {
 }

 func (b *dockerBuilder) Build() error {
+	variants := []string{""}
+	if os.Getenv("OFFLINE_DOCKER_IMAGE") == "true" || b.ServiceName == "elastic-agent" {


We should document this somewhere in the Beats repo?

Is there a place we already document similar things? It's my understanding that we don't have such a place. I'd be glad to document it, my only question is where?

I started to put things into elastic agent readme: https://github.com/elastic/beats/tree/master/x-pack/elastic-agent Lets put it there so we have it and can figure out a better place later.

I've since removed this and replaced it in the package spec, where it has an explanatory comment. I've also added a note to the README you mentioned about the behavior of the package spec.

paulb-elastic · 2021-07-30T11:10:01Z

I see a checkbox for documentation. We documented the docker container here elastic/observability-docs#622. Can you add content for the offline mode or at least create an issue to track adding the docs?

@mostlyjason we have this captured in elastic/observability-docs#663 (I've added a reference to the existing docs page you mentioned)

andrewkroh

If there is an "offline" image available I would expect it to include other artifacts that are not currently bundled with Agent like osquerybeat. Do others have this expectation? Should we add that as well?

andrewkroh · 2021-08-03T15:38:03Z

dev-tools/mage/dockerbuilder.go

@@ -58,6 +59,11 @@ func newDockerBuilder(spec PackageSpec) (*dockerBuilder, error) {
 }

 func (b *dockerBuilder) Build() error {


This doesn't seem like the right place for any project specific build options. IIRC this code path doesn't have knowledge of elastic-agent at all.

If there are new artifacts that the Elastic Agent build should produce those should be defined in a package spec YAML (elastic-agent appears to be adding artifacts to the https://github.com/elastic/beats/blob/master/dev-tools/packaging/packages.yml#L938-L939).

dev-tools/mage/dockerbuilder.go

ruflin · 2021-08-04T10:40:58Z

@andrewkroh ++ on your proposal that the "offline" image contains "all" of it.

Co-authored-by: Manuel de la Peña <[email protected]>

andrewvc · 2021-08-10T16:35:33Z

My latest push removes the assigning of ExtraVars and fixes env var assignment, which was incorrectly before a docker FROM statement.

andrewkroh

LGTM 👍

andrewkroh · 2021-08-10T19:17:41Z

dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl

@@ -6,6 +6,10 @@
 # the final image because of permission changes.
 FROM {{ .buildFrom }} AS home

+# Contains the elastic agent image variant, an empty string for the standard variant
+# or "offline" for the offline one.
+ENV ELASTIC_AGENT_IMAGE_VARIANT={{.Variant}}


What do you think about omitting this from the environment when .Variant is empty?

I'm neutral on it. I do kind of like having it empty because it makes it maybe more explicit that this is the "" variant.

IIUC, this particular variable does nothing but reporting the docker image is a variant? If so, maybe the docker metadata attributes could be a better approach?

dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl

ruflin

Code change LGTM. Only one part I'm not sure how it is related.

I assume you did a full package run to see if things still work as expected? Lets make sure as soon as we get this in we keep an eye on the builds to make sure these still pass.

For the publishing of the images, I assume some additional work in the release manager will be needed?

ruflin · 2021-08-11T09:25:16Z

x-pack/heartbeat/monitors/browser/synthexec/enrich.go

@@ -122,9 +122,9 @@ func (je *journeyEnricher) enrichSynthEvent(event *beat.Event, se *SynthEvent) e
 	case "step/screenshot_ref":
 		fallthrough
 	case "screenshot/block":
-		add_data_stream_index.SetEventDataset(event, "browser_screenshot")
+		add_data_stream_index.SetEventDataset(event, "browser.screenshot")


How is this related?

The main purpose of this PR, really, is to enable the integration, which breaks without this. I agree it's a bit unrelated, but I think it makes sense to include because this PR isn't useful without it.

vigneshshanmugam

LGTM

vigneshshanmugam · 2021-08-11T14:18:54Z

dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl

+{{- if (and (eq .Variant "offline") (not (contains .from "ubi-minimal")))  }}
+# Setup synthetics env vars
+ENV ELASTIC_SYNTHETICS_CAPABLE=true
+ENV SUITES_DIR={{ $beatHome }}/suites


Are we using a dedicated folder instead of /tmp for copying suites? I forgot what wat was the real use of this flag.

This can probably be nixed. It's not really used for anything.

vigneshshanmugam · 2021-08-11T14:35:21Z

x-pack/heartbeat/monitors/browser/synthexec/enrich.go

 	case "journey/network_info":
-		add_data_stream_index.SetEventDataset(event, "browser_network")
+		add_data_stream_index.SetEventDataset(event, "browser.network")


wouldn't it break for existing users?

No, existing users don't use datasets, only people using the integration do. For existing users it's all under heartbeat-*

Gotcha 👍🏽

andrewvc · 2021-08-11T19:43:24Z

@Mergifyio backport 7.x

@dominiqueclarke

#27052) Creates a new offline tagged docker image that includes the dependencies required for synthetics. Building elastic-agent docker images will now create additional -offline tagged images with these extras. We may need to do additional work to ensure that these additional images are incorporated into the release process and onto the public website. These changes also set the ELASTIC_SYNTHETICS_CAPABLE env flag, as done in heartbeat, to enable synthetics features in the docker environment. This dovetails with the work @dominiqueclarke is doing in elastic/integrations#1064 and elsewhere Fixes #22932 (cherry picked from commit 4727470)

mergify · 2021-08-11T19:44:45Z

Command backport 7.x: success

Backports have been created

#27324 [Heartbeat] Build elastic agent with synthetics support / offline mode (backport #27052) has been created for branch 7.x

@dominiqueclarke

#27052) (#27324) Creates a new offline tagged docker image that includes the dependencies required for synthetics. Building elastic-agent docker images will now create additional -offline tagged images with these extras. We may need to do additional work to ensure that these additional images are incorporated into the release process and onto the public website. These changes also set the ELASTIC_SYNTHETICS_CAPABLE env flag, as done in heartbeat, to enable synthetics features in the docker environment. This dovetails with the work @dominiqueclarke is doing in elastic/integrations#1064 and elsewhere Fixes #22932 (cherry picked from commit 4727470) Co-authored-by: Andrew Cholakian <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

andrewvc · 2021-08-18T13:36:36Z

@Mergifyio backport 7.15

@dominiqueclarke

#27052) Creates a new offline tagged docker image that includes the dependencies required for synthetics. Building elastic-agent docker images will now create additional -offline tagged images with these extras. We may need to do additional work to ensure that these additional images are incorporated into the release process and onto the public website. These changes also set the ELASTIC_SYNTHETICS_CAPABLE env flag, as done in heartbeat, to enable synthetics features in the docker environment. This dovetails with the work @dominiqueclarke is doing in elastic/integrations#1064 and elsewhere Fixes #22932 (cherry picked from commit 4727470) # Conflicts: # dev-tools/mage/dockerbuilder.go

mergify · 2021-08-18T13:37:41Z

Command backport 7.15: success

Backports have been created

#27467 [Heartbeat] Build elastic agent with synthetics support / offline mode (backport #27052) has been created for branch 7.15

@dominiqueclarke

#27052) (#27324) Creates a new offline tagged docker image that includes the dependencies required for synthetics. Building elastic-agent docker images will now create additional -offline tagged images with these extras. We may need to do additional work to ensure that these additional images are incorporated into the release process and onto the public website. These changes also set the ELASTIC_SYNTHETICS_CAPABLE env flag, as done in heartbeat, to enable synthetics features in the docker environment. This dovetails with the work @dominiqueclarke is doing in elastic/integrations#1064 and elsewhere Fixes #22932 (cherry picked from commit 4727470) Co-authored-by: Andrew Cholakian <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> (cherry picked from commit f9b2610) # Conflicts: # dev-tools/mage/dockerbuilder.go

…e (backport #27052) (backport #27324) (#27468)

mergify · 2021-10-19T11:34:29Z

This pull request does not have a backport label. Could you fix it @andrewvc? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

Build elasttic agent with synthetics support

70f13ef

andrewvc added enhancement Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team v7.15.0 labels Jul 27, 2021

andrewvc requested review from v1v, vigneshshanmugam and dominiqueclarke July 27, 2021 03:10

andrewvc self-assigned this Jul 27, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 27, 2021

andrewvc marked this pull request as draft July 27, 2021 03:10

andrewvc changed the title ~~[Heartbeat] Build elasttic agent with synthetics support / offline mode~~ [Heartbeat] Build elastic agent with synthetics support / offline mode Jul 27, 2021

Omit ubi8 images from offline build

4bab4c0

ruflin requested review from blakerouse and andrewkroh July 28, 2021 08:24

ruflin reviewed Jul 28, 2021

View reviewed changes

paulb-elastic mentioned this pull request Jul 30, 2021

[Synthetics] Document offline synthetics use case elastic/observability-docs#663

Closed

andrewkroh requested changes Aug 3, 2021

View reviewed changes

andrewvc added 4 commits August 5, 2021 12:05

Use dots for datasets

f3046ab

Switch to package spec

46ca076

Fix docker build delay

169d591

Mage fmt

d25860c

andrewvc and others added 2 commits August 10, 2021 07:20

Update dev-tools/packaging/packages.yml

324b419

Co-authored-by: Manuel de la Peña <[email protected]>

Fix env vars in dockerfile, cleanup extravars

420816c

andrewkroh approved these changes Aug 10, 2021

View reviewed changes

Merge branch 'envoffline' of github.com:andrewvc/beats into envoffline

45d7946

andrewvc requested a review from a team as a code owner August 10, 2021 20:05

Fix heartbeat.yml

3fa0117

ruflin reviewed Aug 11, 2021

View reviewed changes

vigneshshanmugam approved these changes Aug 11, 2021

View reviewed changes

andrewvc merged commit 4727470 into elastic:master Aug 11, 2021

andrewvc deleted the envoffline branch August 11, 2021 19:43

mergify bot mentioned this pull request Aug 11, 2021

[Heartbeat] Build elastic agent with synthetics support / offline mode (backport #27052) #27324

Merged

andrewvc mentioned this pull request Aug 17, 2021

[Heartbeat] 'complete' Agent image (synthetics deps included) #26398

Closed

cachedout mentioned this pull request Aug 17, 2021

Add the -complete image for elastic-agent #27427

Merged

6 tasks

mergify bot mentioned this pull request Aug 18, 2021

[Heartbeat] Build elastic agent with synthetics support / offline mode (backport #27052) #27467

Closed

andrewvc pushed a commit that referenced this pull request Aug 18, 2021

[Heartbeat] Build elastic agent with synthetics support / offline mod…

c4f3907

…e (backport #27052) (backport #27324) (#27468)

This was referenced Aug 26, 2021

Complete variant for the elastic agent is not generating the right binaries #27608

Closed

fix: generate complete variant binaries properly #27609

Closed

andrewvc mentioned this pull request Aug 31, 2021

[Heartbeat][Agent] Synthetics deps and Elastic Agent #22932

Closed

mdelapenya mentioned this pull request Oct 19, 2021

fix: use declarative style for complete variant of the elastic-agent #28526

Merged

6 tasks

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 19, 2021

		@@ -58,6 +59,11 @@ func newDockerBuilder(spec PackageSpec) (*dockerBuilder, error) {
		}

		func (b *dockerBuilder) Build() error {

[Heartbeat] Build elastic agent with synthetics support / offline mode #27052

[Heartbeat] Build elastic agent with synthetics support / offline mode #27052

Conversation

andrewvc commented Jul 27, 2021 • edited Loading

Checklist

How to test this PR locally

elasticmachine commented Jul 27, 2021

elasticmachine commented Jul 27, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

paulb-elastic commented Jul 27, 2021

mostlyjason commented Jul 27, 2021

ruflin commented Jul 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulb-elastic commented Jul 30, 2021

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Aug 4, 2021

andrewvc commented Aug 10, 2021

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vigneshshanmugam left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewvc commented Aug 11, 2021

mergify bot commented Aug 11, 2021

andrewvc commented Aug 18, 2021

mergify bot commented Aug 18, 2021

mergify bot commented Oct 19, 2021

andrewvc commented Jul 27, 2021 •

edited

Loading

elasticmachine commented Jul 27, 2021 •

edited by jenkins-beats-ci bot

Loading