Merge branch '7.9' of github.com:elastic/beats into 7.9

.ci/packaging.groovy

@@ -229,11 +229,11 @@ def withMacOSEnv(Closure body){
 
 def publishPackages(baseDir){
   def bucketUri = "gs://${JOB_GCS_BUCKET}/snapshots"
-  if (env.CHANGE_ID?.trim()) {
+  if (isPR()) {
     bucketUri = "gs://${JOB_GCS_BUCKET}/pull-requests/pr-${env.CHANGE_ID}"
   }
-
-  googleStorageUpload(bucket: "${bucketUri}",
+  def beatsFolderName = getBeatsName(baseDir)
+  googleStorageUpload(bucket: "${bucketUri}/${beatsFolderName}",
     credentialsId: "${JOB_GCS_CREDENTIALS}",
     pathPrefix: "${baseDir}/build/distributions/",
     pattern: "${baseDir}/build/distributions/**/*",
@@ -242,6 +242,18 @@ def publishPackages(baseDir){
   )
 }
 
+/**
+* There is a specific folder structure in https://staging.elastic.co/ and https://artifacts.elastic.co/downloads/
+* therefore the storage bucket in GCP should follow the same folder structure.
+* This is required by https://github.com/elastic/beats-tester
+* e.g.
+* baseDir=name -> return name
+* baseDir=name1/name2/name3-> return name2
+*/
+def getBeatsName(baseDir) {
+  return baseDir.replace('x-pack/', '')
+}
+
 def withBeatsEnv(Closure body) {
   withMageEnv(){
     withEnv([

CHANGELOG-developer.next.asciidoc

@@ -92,3 +92,4 @@ The list below covers the major changes between 7.0.0-rc2 and master only.
 - Added SQL helper that can be used from any Metricbeat module {pull}18955[18955]
 - Update Go version to 1.14.4. {pull}19753[19753]
 - Update Go version to 1.14.7. {pull}20508[20508]
+- Make the mage binary used by the build process in the docker container to be statically compiled. {pull}20827[20827]

CHANGELOG.asciidoc

@@ -3,6 +3,52 @@
 :issue: https://github.com/elastic/beats/issues/
 :pull: https://github.com/elastic/beats/pull/
 
+[[release-notes-7.9.1]]
+=== Beats version 7.9.1
+https://github.com/elastic/beats/compare/v7.9.0...v7.9.1[View commits]
+
+==== Breaking changes
+
+*Affecting all Beats*
+
+- Removed experimental modules `citrix`, `kaspersky`, `rapid7` and `tenable`. {pull}20706[20706]
+
+==== Bugfixes
+
+*Affecting all Beats*
+
+- Update replicaset group to apps/v1 {pull}15854[15854]
+- Rename cloud.provider `az` value to `azure` inside the add_cloud_metadata processor. {pull}20689[20689]
+- Add missing country_name geo field in `add_host_metadata` and `add_observer_metadata` processors. {issue}20796[20796] {pull}20811[20811]
+
+*Filebeat*
+
+- Fix long registry migration times. {pull}20717[20717] {issue}20705[20705]
+- Fix event types and categories in auditd module to comply with ECS {pull}20652[20652]
+- Update documentation in the azure module filebeat. {pull}20815[20815]
+
+*Heartbeat*
+
+- Stop rescheduling tasks of stopped monitors. {pull}20570[20570]
+
+*Metricbeat*
+
+- Updates vm_compute metricset with more info on guest metrics. {pull}20448[20448]
+- Add fallback for PdhExpandWildCardPathW failing in perfmon metricset. {issue}20139[20139] {pull}20630[20630]
+- Fix resource tags in aws cloudwatch metricset {issue}20326[20326]  {pull}20385[20385]
+- Fill cloud.account.name with accountID if account alias doesn't exist. {pull}20736[20736]
+
+*Winlogbeat*
+
+- Fix duplicated field error when exporting index-pattern with migration.6_to_7.enabled. {issue}20521[20521] {pull}20540[20540]
+- Fix `event.outcome` in the security module for non-English languages. {issue}20079[20079] {pull}20564[20564]
+
+==== Added
+
+*Affecting all Beats*
+
+- Added support for more message types for Cisco ASA and FTD. {pull}20565[20565]
+
 [[release-notes-7.9.0]]
 === Beats version 7.9.0
 https://github.com/elastic/beats/compare/v7.8.1...v7.9.0[View commits]
@@ -229,6 +275,12 @@ field. You can revert this change by configuring tags for the module and omittin
 - Deprecate tags config parameter in cloudwatch metricset. {pull}16733[16733]
 - Deprecate tags.resource_type_filter config parameter and replace with resource_type. {pull}19688[19688]
 
+==== Known Issues
+
+*Functionbeat*
+
+- Functionbeat cannot be deployed on Google Cloud Platform. {issue}20830[20830]
+
 [[release-notes-7.8.1]]
 === Beats version 7.8.1
 https://github.com/elastic/beats/compare/v7.8.0...v7.8.1[View commits]

CHANGELOG.next.asciidoc

@@ -12,14 +12,14 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 - Update to Golang 1.12.1. {pull}11330[11330]
 - Disable Alibaba Cloud and Tencent Cloud metadata providers by default. {pull}13812[12812]
+- Autodiscover doesn't generate any configuration when a variable is missing. Previously it generated an incomplete configuration. {pull}20898[20898]
 
 *Auditbeat*
 
 
 *Filebeat*
 
 - Fix parsing of Elasticsearch node name by `elasticsearch/slowlog` fileset. {pull}14547[14547]
-- Removed experimental modules `citrix`, `kaspersky`, `rapid7` and `tenable`. {pull}20706[20706]
 
 *Heartbeat*
 
@@ -46,12 +46,12 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 - Fix a race condition with the Kafka pipeline client, it is possible that `Close()` get called before `Connect()` . {issue}11945[11945]
 - Allow users to configure only `cluster_uuid` setting under `monitoring` namespace. {pull}14338[14338]
-- Update replicaset group to apps/v1 {pull}15854[15854]
 - Fix missing output in dockerlogbeat {pull}15719[15719]
 - Do not rotate log files on startup when interval is configured and rotateonstartup is disabled. {pull}17613[17613]
 - Fix `setup.dashboards.index` setting not working. {pull}17749[17749]
 - Fix Elasticsearch license endpoint URL referenced in error message. {issue}17880[17880] {pull}18030[18030]
 - Change `decode_json_fields` processor, to merge parsed json objects with existing objects in the event instead of fully replacing them. {pull}17958[17958]
+- Explicitly detect missing variables in autodiscover configuration, log them at the debug level. {issue}20568[20568] {pull}20898[20898]
 
 *Auditbeat*
 
@@ -74,7 +74,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 *Heartbeat*
 
-- Stop rescheduling tasks of stopped monitors. {pull}20570[20570]
 
 *Journalbeat*
 
@@ -98,19 +97,13 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix storage metricset to allow config without region/zone. {issue}17623[17623] {pull}17624[17624]
 - Fix overflow on Prometheus rates when new buckets are added on the go. {pull}17753[17753]
 - Add a switch to the driver definition on SQL module to use pretty names {pull}17378[17378]
-- Modify doc for app_insights metricset to contain example of config. {pull}20185[20185]
-- Add required option for `metrics` in app_insights. {pull}20406[20406]
-- Groups same timestamp metric values to one event in the app_insights metricset. {pull}20403[20403]
-- Updates vm_compute metricset with more info on guest metrics. {pull}20448[20448]
-- Fix resource tags in aws cloudwatch metricset {issue}20326[20326]  {pull}20385[20385]
+- The `elasticsearch/index` metricset only requests wildcard expansion for hidden indices if the monitored Elasticsearch cluster supports it. {pull}20938[20938]
 
 *Packetbeat*
 
 
 *Winlogbeat*
 
-- Fix duplicated field error when exporting index-pattern with migration.6_to_7.enabled. {issue}20521[20521] {pull}20540[20540]
-- Fix `event.outcome` in the security module for non-English languages. {issue}20079[20079] {pull}20564[20564]
 
 *Functionbeat*
 
@@ -125,6 +118,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Update documentation for system.process.memory fields to include clarification on Windows os's. {pull}17268[17268]
 - When using the `decode_json_fields` processor, decoded fields are now deep-merged into existing event. {pull}17958[17958]
 - Add keystore support for autodiscover static configurations. {pull]16306[16306]
+- Add container ECS fields in kubernetes metadata. {pull}20984[20984]
 
 *Auditbeat*
 

deploy/kubernetes/auditbeat-kubernetes.yaml

@@ -109,7 +109,7 @@ spec:
       dnsPolicy: ClusterFirstWithHostNet
       containers:
       - name: auditbeat
-        image: docker.elastic.co/beats/auditbeat:7.9.0
+        image: docker.elastic.co/beats/auditbeat:7.9.2
         args: [
           "-c", "/etc/auditbeat.yml",
           "-e",

deploy/kubernetes/filebeat-kubernetes.yaml

@@ -64,7 +64,7 @@ spec:
       dnsPolicy: ClusterFirstWithHostNet
       containers:
       - name: filebeat
-        image: docker.elastic.co/beats/filebeat:7.9.0
+        image: docker.elastic.co/beats/filebeat:7.9.2
         args: [
           "-c", "/etc/filebeat.yml",
           "-e",

deploy/kubernetes/heartbeat-kubernetes.yaml

@@ -74,7 +74,7 @@ spec:
       dnsPolicy: ClusterFirstWithHostNet
       containers:
       - name: heartbeat
-        image: docker.elastic.co/beats/heartbeat:7.9.0
+        image: docker.elastic.co/beats/heartbeat:7.9.2
         args: [
           "-c", "/etc/heartbeat.yml",
           "-e",

deploy/kubernetes/metricbeat-kubernetes.yaml

@@ -114,7 +114,7 @@ spec:
       dnsPolicy: ClusterFirstWithHostNet
       containers:
       - name: metricbeat
-        image: docker.elastic.co/beats/metricbeat:7.9.0
+        image: docker.elastic.co/beats/metricbeat:7.9.2
         args: [
           "-c", "/etc/metricbeat.yml",
           "-e",
@@ -270,7 +270,7 @@ spec:
       dnsPolicy: ClusterFirstWithHostNet
       containers:
       - name: metricbeat
-        image: docker.elastic.co/beats/metricbeat:7.9.0
+        image: docker.elastic.co/beats/metricbeat:7.9.2
         args: [
           "-c", "/etc/metricbeat.yml",
           "-e",

dev-tools/mage/crossbuild.go

@@ -173,7 +173,7 @@ func CrossBuildXPack(options ...CrossBuildOption) error {
 // values for Docker. It has the benefit of speeding up the build because the
 // mage -compile is done only once rather than in each Docker container.
 func buildMage() error {
-	return sh.Run("mage", "-f", "-goos=linux", "-goarch=amd64",
+	return sh.RunWith(map[string]string{"CGO_ENABLED": "0"}, "mage", "-f", "-goos=linux", "-goarch=amd64",
 		"-compile", CreateDir(filepath.Join("build", "mage-linux-amd64")))
 }
 

docs/devguide/index.asciidoc

@@ -27,6 +27,8 @@ include::./fields-yml.asciidoc[]
 
 include::./event-conventions.asciidoc[]
 
+include::./python.asciidoc[]
+
 include::./newdashboards.asciidoc[]
 
 include::./new_protocol.asciidoc[]

docs/devguide/python.asciidoc

@@ -1,5 +1,5 @@
 [[python-beats]]
-== Python in Beats
+=== Python in Beats
 
 Python is used for Beats development, it is the language used to implement
 system tests and some other tools. Python dependencies are managed by the use of
@@ -9,7 +9,7 @@ https://docs.python.org/3/library/venv.html[venv].
 Beats development requires Python >= {python}.
 
 [[installing-python]]
-=== Installing Python and venv
+==== Installing Python and venv
 
 Python uses to be installed in many operating systems. If it is not installed in
 your system you can follow the instructions available in https://www.python.org/downloads/
@@ -32,7 +32,7 @@ sudo apt-get install python3.7 python3.7-venv
 It is recommended to use Python >= {python}.
 
 [[python-virtual-environments]]
-=== Working with virtual environments
+==== Working with virtual environments
 
 All `make` and `mage` targets manage their own virtual environments in a transparent
 way, so for the most common operations required when contributing to beats,
@@ -65,7 +65,7 @@ To recreate a virtual environment, remove its directory. All virtual
 environments are also removed with `make clean`.
 
 [[python-older-versions]]
-=== Working with older versions
+==== Working with older versions
 
 Older versions of Beats were not compatible with Python 3, if you need to
 temporary work on one of these versions of Beats, and you don't want to remove

filebeat/docs/autodiscover-hints.asciidoc

@@ -112,7 +112,7 @@ filebeat.autodiscover:
       hints.default_config:
         type: container
         paths:
-          - /var/log/container/*-${container.id}.log  # CRI path
+          - /var/log/containers/*-${data.container.id}.log  # CRI path
 -----
 
 You can also disable default settings entirely, so only Pods annotated like `co.elastic.logs/enabled: true`
@@ -215,7 +215,7 @@ filebeat.autodiscover:
       hints.default_config:
         type: container
         paths:
-          - /var/log/container/*-${container.id}.log  # CRI path
+          - /var/log/containers/*-${data.container.id}.log  # CRI path
 -----
 
 You can also disable default settings entirely, so only containers labeled with `co.elastic.logs/enabled: true`