Merge branch 'master' of github.com:elastic/kibana into rule-schema-u…

…pdates-2

.gitignore

@@ -31,6 +31,7 @@ disabledPlugins
 webpackstats.json
 /config/*
 !/config/kibana.yml
+!/config/node.options
 coverage
 selenium
 .babel_register_cache.json

NOTICE.txt

@@ -147,6 +147,70 @@ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
 
+---
+Detection Rules
+Copyright 2020 Elasticsearch B.V.
+
+---
+This product bundles rules based on https://github.com/BlueTeamLabs/sentinel-attack
+which is available under a "MIT" license. The files based on this license are:
+
+- defense_evasion_via_filter_manager
+- discovery_process_discovery_via_tasklist_command
+- persistence_priv_escalation_via_accessibility_features
+- persistence_via_application_shimming
+- defense_evasion_execution_via_trusted_developer_utilities
+
+MIT License
+
+Copyright (c) 2019 Edoardo Gerosa, Olaf Hartong
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+---
+This product bundles rules based on https://github.com/FSecureLABS/leonidas
+which is available under a "MIT" license. The files based on this license are:
+
+- credential_access_secretsmanager_getsecretvalue.toml
+
+MIT License
+
+Copyright (c) 2020 F-Secure LABS
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
 ---
 This product bundles [email protected] which is available under a
 "MIT" license.
@@ -220,38 +284,6 @@ WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 
----
-This product bundles rules based on https://github.com/BlueTeamLabs/sentinel-attack
-which is available under a "MIT" license. The files based on this license are:
-
-- windows_defense_evasion_via_filter_manager.json
-- windows_process_discovery_via_tasklist_command.json
-- windows_priv_escalation_via_accessibility_features.json
-- windows_persistence_via_application_shimming.json
-- windows_execution_via_trusted_developer_utilities.json
-
-MIT License
-
-Copyright (c) 2019 Edoardo Gerosa, Olaf Hartong
-
-Permission is hereby granted, free of charge, to any person obtaining a copy of
-this software and associated documentation files (the "Software"), to deal in
-the Software without restriction, including without limitation the rights to
-use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
-of the Software, and to permit persons to whom the Software is furnished to do
-so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
-
 ---
 This product includes code that is adapted from mapbox-gl-js, which is
 available under a "BSD-3-Clause" license.

config/node.options

@@ -0,0 +1,6 @@
+## Node command line options
+## See `node --help` and `node --v8-options` for available options
+## Please note you should specify one option per line
+
+## max size of old space in megabytes
+#--max-old-space-size=4096

docs/developer/advanced/development-basepath.asciidoc

@@ -0,0 +1,18 @@
+[[development-basepath]]
+=== Considerations for basepath
+
+In dev mode, {kib} by default runs behind a proxy which adds a random path component to its URL.
+
+You can set this explicitly using `server.basePath` in <<settings>>. 
+
+Use the server.rewriteBasePath setting to tell {kib} if it should remove the basePath from requests it receives, and to prevent a deprecation warning at startup. This setting cannot end in a slash (/).
+
+If you want to turn off the basepath when in development mode, start {kib} with the `--no-basepath` flag
+
+[source,bash]
+----
+yarn start --no-basepath
+----
+
+
+

docs/developer/core/development-es-snapshots.asciidoc → docs/developer/advanced/development-es-snapshots.asciidoc

@@ -1,21 +1,21 @@
 [[development-es-snapshots]]
 === Daily Elasticsearch Snapshots
 
-For local development and CI, Kibana, by default, uses Elasticsearch snapshots that are built daily when running tasks that require Elasticsearch (e.g. functional tests).
+For local development and CI, {kib}, by default, uses Elasticsearch snapshots that are built daily when running tasks that require Elasticsearch (e.g. functional tests).
 
 A snapshot is just a group of tarballs, one for each supported distribution/architecture/os of Elasticsearch, and a JSON-based manifest file containing metadata about the distributions.
 
 https://ci.kibana.dev/es-snapshots[A dashboard] is available that shows the current status and compatibility of the latest Elasticsearch snapshots.
 
 ==== Process Overview
 
-1. Elasticsearch snapshots are built for each current tracked branch of Kibana.
+1. Elasticsearch snapshots are built for each current tracked branch of {kib}.
 2. Each snapshot is uploaded to a public Google Cloud Storage bucket, `kibana-ci-es-snapshots-daily`.
 ** At this point, the snapshot is not automatically used in CI or local development. It needs to be tested/verified first.
-3. Each snapshot is tested with the latest commit of the corresponding Kibana branch, using the full CI suite.
+3. Each snapshot is tested with the latest commit of the corresponding {kib} branch, using the full CI suite.
 4. After CI
 ** If the snapshot passes, it is promoted and automatically used in CI and local development.
-** If the snapshot fails, the issue must be investigated and resolved. A new incompatibility may exist between Elasticsearch and Kibana.
+** If the snapshot fails, the issue must be investigated and resolved. A new incompatibility may exist between Elasticsearch and {kib}.
 
 ==== Using the latest snapshot
 
@@ -39,7 +39,7 @@ KBN_ES_SNAPSHOT_USE_UNVERIFIED=true node scripts/functional_tests_server
 
 Currently, there is not a way to run your pull request with the latest unverified snapshot without a code change. You can, however, do it with a small code change.
 
-1. Edit `Jenkinsfile` in the root of the Kibana repo
+1. Edit `Jenkinsfile` in the root of the {kib} repo
 2. Add `env.KBN_ES_SNAPSHOT_USE_UNVERIFIED = 'true'` at the top of the file.
 3. Commit the change
 
@@ -75,13 +75,13 @@ The file structure for this bucket looks like this:
 
 ==== How snapshots are built, tested, and promoted
 
-Each day, a https://kibana-ci.elastic.co/job/elasticsearch+snapshots+trigger/[Jenkins job] runs that triggers Elasticsearch builds for each currently tracked branch/version. This job is automatically updated with the correct branches whenever we release new versions of Kibana.
+Each day, a https://kibana-ci.elastic.co/job/elasticsearch+snapshots+trigger/[Jenkins job] runs that triggers Elasticsearch builds for each currently tracked branch/version. This job is automatically updated with the correct branches whenever we release new versions of {kib}.
 
 ===== Build
 
 https://kibana-ci.elastic.co/job/elasticsearch+snapshots+build/[This Jenkins job] builds the Elasticsearch snapshots and uploads them to GCS.
 
-The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_build_es[in the kibana repo].
+The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_build_es[in the {kib} repo].
 
 1. Checkout Elasticsearch repo for the given branch/version.
 2. Run `./gradlew -p distribution/archives assemble --parallel` to create all of the Elasticsearch distributions.
@@ -91,15 +91,15 @@ The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/ma
 ** e.g. `<version>/archives/<unique id>`
 6. Replace `<version>/manifest-latest.json` in GCS with this newest manifest.
 ** This allows the `KBN_ES_SNAPSHOT_USE_UNVERIFIED` flag to work.
-7. Trigger the verification job, to run the full Kibana CI test suite with this snapshot.
+7. Trigger the verification job, to run the full {kib} CI test suite with this snapshot.
 
 ===== Verification and Promotion
 
-https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/[This Jenkins job] tests the latest Elasticsearch snapshot with the full Kibana CI pipeline, and promotes if it there are no test failures.
+https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/[This Jenkins job] tests the latest Elasticsearch snapshot with the full {kib} CI pipeline, and promotes if it there are no test failures.
 
-The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_verify_es[in the kibana repo].
+The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_verify_es[in the {kib} repo].
 
-1. Checkout Kibana and set up CI environment as normal.
+1. Checkout {kib} and set up CI environment as normal.
 2. Set the `ES_SNAPSHOT_MANIFEST` env var to point to the latest snapshot manifest.
 3. Run CI (functional tests, integration tests, etc).
 4. After CI

docs/developer/advanced/index.asciidoc

@@ -0,0 +1,12 @@
+[[advanced]]
+== Advanced
+
+* <<running-elasticsearch>>
+* <<development-es-snapshots>>
+* <<development-basepath>>
+
+include::development-es-snapshots.asciidoc[]
+
+include::running-elasticsearch.asciidoc[]
+
+include::development-basepath.asciidoc[]

docs/developer/advanced/running-elasticsearch.asciidoc

@@ -0,0 +1,118 @@
+[[running-elasticsearch]]
+=== Running elasticsearch during development
+
+There are many ways to run Elasticsearch while you are developing.
+
+[float]
+
+==== By snapshot
+
+This will run a snapshot of elasticsearch that is usually built nightly. Read more about <<development-es-snapshots>>.
+
+[source,bash]
+----
+yarn es snapshot
+----
+
+See all available options, like how to specify a specific license, with the `--help` flag.
+
+[source,bash]
+----
+yarn es snapshot --help
+----
+
+`trial` will give you access to all capabilities.
+
+**Keeping data between snapshots**
+
+If you want to keep the data inside your Elasticsearch between usages of this command, you should use the following command, to keep your data folder outside the downloaded snapshot folder:
+
+[source,bash]
+----
+yarn es snapshot -E path.data=../data
+----
+
+==== By source
+
+If you have the Elasticsearch repo checked out locally and wish to run against that, use `source`. By default, it will reference an elasticsearch checkout which is a sibling to the {kib} directory named elasticsearch. If you wish to use a checkout in another location you can provide that by supplying --source-path 
+
+[source,bash]
+----
+yarn es source
+----
+
+==== From an archive
+
+Use this if you already have a distributable. For released versions, one can be obtained on the Elasticsearch downloads page.
+
+[source,bash]
+----
+yarn es archive <full_path_to_archive>
+----
+
+Each of these will run Elasticsearch with a basic license. Additional options are available, pass --help for more information.
+
+==== From a remote host
+
+You can save some system resources, and the effort of generating sample data, if you have a remote Elasticsearch cluster to connect to. (Elasticians: you do! Check with your team about where to find credentials)
+
+You'll need to create a kibana.dev.yml (<<customize-kibana-yml>>) and add the following to it:
+
+[source,bash]
+----
+elasticsearch.hosts:
+  - {{ url }}
+elasticsearch.username: {{ username }}
+elasticsearch.password: {{ password }}
+elasticsearch.ssl.verificationMode: none
+----
+
+If many other users will be interacting with your remote cluster, you'll want to add the following to avoid causing conflicts:
+
+[source,bash]
+----
+kibana.index: '.{YourGitHubHandle}-kibana'
+xpack.task_manager.index: '.{YourGitHubHandle}-task-manager-kibana'
+----
+
+===== Running remote clusters
+
+Setup remote clusters for cross cluster search (CCS) and cross cluster replication (CCR).
+
+Start your primary cluster by running:
+
+[source,bash]
+----
+yarn es snapshot -E path.data=../data_prod1
+----
+
+Start your remote cluster by running:
+
+[source,bash]
+----
+yarn es snapshot -E transport.port=9500 -E http.port=9201 -E path.data=../data_prod2
+----
+
+Once both clusters are running, start {kib}. {kib} will connect to the primary cluster.
+
+Setup the remote cluster in {kib} from either Management -> Elasticsearch -> Remote Clusters UI or by running the following script in Console.
+
+[source,bash]
+----
+PUT _cluster/settings
+{
+  "persistent": {
+    "cluster": {
+      "remote": {
+        "cluster_one": {
+          "seeds": [
+            "localhost:9500"
+          ]
+        }
+      }
+    }
+  }
+}
+----
+
+Follow the cross-cluster search instructions for setting up index patterns to search across clusters (<<management-cross-cluster-search>>).