Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Autodiscover provider for Nomad #14954

Merged
merged 91 commits into from
Jan 7, 2021
Merged

Autodiscover provider for Nomad #14954

merged 91 commits into from
Jan 7, 2021

Commits on Aug 1, 2019

  1. Metadata extractor from Nomad

    @jorgelbg
    jorgelbg committed Aug 1, 2019
    Configuration menu
    Copy the full SHA
    86996ff View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2019

  1. Add hint generation from the nomad event's meta 

    Add a test for the basic hint features
    @jorgelbg
    jorgelbg committed Sep 4, 2019
    Configuration menu
    Copy the full SHA
    2d4d434 View commit details
    Browse the repository at this point in the history

  2. Autodetect the hostname to filter Nomad events

    @jorgelbg
    jorgelbg committed Sep 4, 2019
    Configuration menu
    Copy the full SHA
    164bf0e View commit details
    Browse the repository at this point in the history

  3. Use GCP region as datacenter

    @jorgelbg
    jorgelbg committed Sep 4, 2019
    Configuration menu
    Copy the full SHA
    a03f588 View commit details
    Browse the repository at this point in the history

  4. Remove duplicated task and job keys from the event. Use the proper fl… 

    …ag (event type)
    @jorgelbg
    jorgelbg committed Sep 4, 2019
    Configuration menu
    Copy the full SHA
    023765c View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2019

  1. Add host from the allocation to the event. 

    Add tests for the emition of events
    @jorgelbg
    jorgelbg committed Sep 5, 2019
    Configuration menu
    Copy the full SHA
    fa7df7e View commit details
    Browse the repository at this point in the history

  2. Add host from the allocation to the event. 

    Add tests for the emition of events
    @jorgelbg
    jorgelbg committed Sep 5, 2019
    Configuration menu
    Copy the full SHA
    83884c7 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'autodiscovery-nomad' of github.com:jorgelbg/beats into … 

    …autodiscovery-nomad
    @jorgelbg
    jorgelbg committed Sep 5, 2019
    Configuration menu
    Copy the full SHA
    6a5e46c View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2019

  1. Fire the proper events depending on the status & update time of the a… 

    …llocations

Add local constants to track the possible status of the allocations
    @jorgelbg
    jorgelbg committed Sep 6, 2019
    Configuration menu
    Copy the full SHA
    281a384 View commit details
    Browse the repository at this point in the history

  2. Rename Flag on the test to a closer name to match the nomad nomenclature

    @jorgelbg
    jorgelbg committed Sep 6, 2019
    Configuration menu
    Copy the full SHA
    60c65ef View commit details
    Browse the repository at this point in the history

  3. Add support for older nomad versions 

    On older nomad versions (0.8.4) the `NodeName` attribute of the allocation is empty. This means that sometimes we cannot assign a proper `host` to the event. As a workaround we use the `NodeID` to get the the name from the actual client node.
    @jorgelbg
    jorgelbg committed Sep 6, 2019
    Configuration menu
    Copy the full SHA
    145a92f View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2019

  1. Use the internal watcher logger

    @jorgelbg
    jorgelbg committed Sep 9, 2019
    Configuration menu
    Copy the full SHA
    37b509e View commit details
    Browse the repository at this point in the history

  2. WIP pass the client to the metadata generator 

    This is a workaround for Nomad v0.8 that doesn't provide the NodeName directly in the allocation object. We use the NodeID to fetch it from the API.
    @jorgelbg
    jorgelbg committed Sep 9, 2019
    Configuration menu
    Copy the full SHA
    372db05 View commit details
    Browse the repository at this point in the history

  3. Add cache for the NodeName data

    @jorgelbg
    jorgelbg committed Sep 9, 2019
    Configuration menu
    Copy the full SHA
    afccc97 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2019

  1. Patch the alloc.NodeName before emitting allocation events 

    Remove debug statement
    @jorgelbg
    jorgelbg committed Sep 10, 2019
    Configuration menu
    Copy the full SHA
    a30057f View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2019

  1. Add filebeat allocation ID extractor from the log path

    @jorgelbg
    jorgelbg committed Sep 25, 2019
    Configuration menu
    Copy the full SHA
    97ea28f View commit details
    Browse the repository at this point in the history

  2. Fix usage of the of the WaitIndex on the allocation watcher

    @jorgelbg
    jorgelbg committed Sep 25, 2019
    Configuration menu
    Copy the full SHA
    05f7e3f View commit details
    Browse the repository at this point in the history

  3. Add add_nomad_metadata processor to libbeat

    @jorgelbg
    jorgelbg committed Sep 25, 2019
    Configuration menu
    Copy the full SHA
    af3dfdc View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2019

  1. Add the nomad API package to the vendors

    @jorgelbg
    jorgelbg committed Oct 9, 2019
    Configuration menu
    Copy the full SHA
    bf59c3a View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2019

  1. Glue the add_nomad_metadata processor to the rest of filebeat 

    - WIP emit only one task metadata event.
- Rename the matchers/indexers of the add_nomad_metadata processor to match the Nomad lingo.
    @jorgelbg
    jorgelbg committed Oct 14, 2019
    Configuration menu
    Copy the full SHA
    9ec5f00 View commit details
    Browse the repository at this point in the history

  2. Log only the object IDs in the INFO level 

    - Rename `meta.meta` to `meta.tasks` and fix the tests.
- Add the main import to the nomad provider in the cmd tool.
    @jorgelbg
    jorgelbg committed Oct 14, 2019
    Configuration menu
    Copy the full SHA
    0b6e9d4 View commit details
    Browse the repository at this point in the history

  3. Update NOTICE.txt

    @jorgelbg
    jorgelbg committed Oct 14, 2019
    Configuration menu
    Copy the full SHA
    f1b6283 View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2019

  1. Fix issue with detecting updated allocations. 

    WIP patch for the unchanged allocations and avoids triggering new harvesters for those allocations that were previously discovered.
    @jorgelbg
    jorgelbg committed Nov 11, 2019
    Configuration menu
    Copy the full SHA
    caae329 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2019

  1. Split ResourceMetadata and GroupMetadata into two different functions 

    - Rename `uuid` field to `alloc_id`.
- Use WatchOptions.RefreshInterval (SyncInterval on the config) for the sync interval of the watcher
    @jorgelbg
    jorgelbg committed Nov 14, 2019
    Configuration menu
    Copy the full SHA
    c8b7da7 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2019

  1. Rename some kubernetes ocurrences from the nomad autodiscover beat

    @jorgelbg
    jorgelbg committed Nov 18, 2019
    Configuration menu
    Copy the full SHA
    a647a0a View commit details
    Browse the repository at this point in the history

Commits on Nov 19, 2019

  1. Add AllowStale option to the Nomad configuration

    @jorgelbg
    jorgelbg committed Nov 19, 2019
    Configuration menu
    Copy the full SHA
    f398d51 View commit details
    Browse the repository at this point in the history

  2. Remove config.Prefix fields from the final event

    @jorgelbg
    jorgelbg committed Nov 19, 2019
    Configuration menu
    Copy the full SHA
    148ba9f View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2019

  1. Add status field to the allocation metadata

    @jorgelbg
    jorgelbg committed Nov 21, 2019
    Configuration menu
    Copy the full SHA
    4cca342 View commit details
    Browse the repository at this point in the history

  2. Allow stale responses from the Nomad client by default

    @jorgelbg
    jorgelbg committed Nov 21, 2019
    Configuration menu
    Copy the full SHA
    f0c63e0 View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2019

  1. Fix the value of the status field

    @jorgelbg
    jorgelbg committed Nov 29, 2019
    Configuration menu
    Copy the full SHA
    1d7cc78 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2019

  1. Merge branch 'master' into autodiscovery-nomad

    @jorgelbg
    jorgelbg committed Dec 5, 2019
    Configuration menu
    Copy the full SHA
    1da443b View commit details
    Browse the repository at this point in the history

  2. Add nomad API dependency using govendor. Move import of the new packa… 

    …ges to imports_docker
    @jorgelbg
    jorgelbg committed Dec 5, 2019
    Configuration menu
    Copy the full SHA
    a03006c View commit details
    Browse the repository at this point in the history

  3. Add status field to the allocation metadata. Fix metadata tests

    @jorgelbg
    jorgelbg committed Dec 5, 2019
    Configuration menu
    Copy the full SHA
    60ee670 View commit details
    Browse the repository at this point in the history

  4. Formatting with mage

    @jorgelbg
    jorgelbg committed Dec 5, 2019
    Configuration menu
    Copy the full SHA
    446e296 View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2019

  1. Vendor github.com/hashicorp/nomad/helper

    @jorgelbg
    jorgelbg committed Dec 9, 2019
    Configuration menu
    Copy the full SHA
    67ffef8 View commit details
    Browse the repository at this point in the history

Commits on Dec 17, 2019

  1. Move the nomad autodiscover module and processors into x-pack

    @jorgelbg
    jorgelbg committed Dec 17, 2019
    Configuration menu
    Copy the full SHA
    42e3c17 View commit details
    Browse the repository at this point in the history

  2. Update NOTICE.txt

    @jorgelbg
    jorgelbg committed Dec 17, 2019
    Configuration menu
    Copy the full SHA
    1bcee44 View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2020

  1. Remove unused configuration options from the Nomad provider

    @jorgelbg
    jorgelbg committed Jan 9, 2020
    Configuration menu
    Copy the full SHA
    141be1d View commit details
    Browse the repository at this point in the history

Commits on Jan 20, 2020

  1. Add a first draft of the documentation for the Nomad autodiscover pro… 

    …vider
    @jorgelbg
    jorgelbg committed Jan 20, 2020
    Configuration menu
    Copy the full SHA
    f8364c7 View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2020

  1. Add CleanupTimeout and check for goroutines leak on the main provider… 

    … test
    @jorgelbg
    jorgelbg committed Jan 24, 2020
    Configuration menu
    Copy the full SHA
    e1a8ea2 View commit details
    Browse the repository at this point in the history

  2. Remove redundant temporary variable when processing the job metadata

    @jorgelbg
    jorgelbg committed Jan 24, 2020
    Configuration menu
    Copy the full SHA
    348fd81 View commit details
    Browse the repository at this point in the history

  3. Refactor the allocation watcher and add some tests

    @jorgelbg
    jorgelbg committed Jan 24, 2020
    Configuration menu
    Copy the full SHA
    de1b09e View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'upstream/master' into autodiscovery-nomad

    @jorgelbg
    jorgelbg committed Jan 24, 2020
    Configuration menu
    Copy the full SHA
    f96964e View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2020

  1. Improve handling of Nomad events 

    * Add tests for the watcher to test ignoring old allocations
    @jorgelbg
    jorgelbg committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    b48fb54 View commit details
    Browse the repository at this point in the history

Commits on Feb 10, 2020

  1. Handle in place allocations update on the Watcher

    @jorgelbg
    jorgelbg committed Feb 10, 2020
    Configuration menu
    Copy the full SHA
    6523942 View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2020

  1. Use the local AllocClientStatus constant

    @jorgelbg
    jorgelbg committed Mar 3, 2020
    Configuration menu
    Copy the full SHA
    8140448 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/master' into autodiscovery-nomad

    @jorgelbg
    jorgelbg committed Mar 3, 2020
    Configuration menu
    Copy the full SHA
    834c0be View commit details
    Browse the repository at this point in the history

  3. Use go modules for handling the dependencies

    @jorgelbg
    jorgelbg committed Mar 3, 2020
    Configuration menu
    Copy the full SHA
    4d62dfe View commit details
    Browse the repository at this point in the history

  4. Add some changes that were missing from the commit

    @jorgelbg
    jorgelbg committed Mar 3, 2020
    Configuration menu
    Copy the full SHA
    79f6444 View commit details
    Browse the repository at this point in the history

  5. Fix formatting of the Elastic License header

    @jorgelbg
    jorgelbg committed Mar 3, 2020
    Configuration menu
    Copy the full SHA
    5f3451f View commit details
    Browse the repository at this point in the history

  6. Remove some internal Nomad dependencies. Adjusted tests 

    Some Nomad dependencies are difficult to vendor because means pulling a lot of different dependencies (sometimes specific forks). We now only depend on the public Nomad API.
    @jorgelbg
    jorgelbg committed Mar 3, 2020
    Configuration menu
    Copy the full SHA
    341ce22 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2020

  1. Fix formatting issue

    @jorgelbg
    jorgelbg committed Mar 4, 2020
    Configuration menu
    Copy the full SHA
    96d4ab6 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/master' into autodiscovery-nomad

    @jorgelbg
    jorgelbg committed Mar 4, 2020
    Configuration menu
    Copy the full SHA
    f098bc3 View commit details
    Browse the repository at this point in the history

  3. Remove elastic/beats dependency

    @jorgelbg
    jorgelbg committed Mar 4, 2020
    Configuration menu
    Copy the full SHA
    6666499 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2020

  1. Merge remote-tracking branch 'upstream/master' into autodiscovery-nomad

    @jorgelbg
    jorgelbg committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    1135c86 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2020

  1. Fix some remarks from code review

    @jorgelbg
    jorgelbg committed May 8, 2020
    Configuration menu
    Copy the full SHA
    6acebf8 View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2020

  1. Merge remote-tracking branch 'upstream/master' into autodiscovery-nomad

    @jorgelbg
    jorgelbg committed Jun 15, 2020
    Configuration menu
    Copy the full SHA
    66334ed View commit details
    Browse the repository at this point in the history

  2. Update vendor dependencies

    @jorgelbg
    jorgelbg committed Jun 15, 2020
    Configuration menu
    Copy the full SHA
    f74f075 View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2020

  1. Fix a bug when an allocation was updated without being reallocated 

    Adapt the nomad provider to comply with the new changes to the interface
from the keystore availability.
    @jorgelbg
    jorgelbg committed Jul 20, 2020
    Configuration menu
    Copy the full SHA
    fd7547b View commit details
    Browse the repository at this point in the history

  2. Fix some issues from the code review

    @jorgelbg
    jorgelbg committed Jul 20, 2020
    Configuration menu
    Copy the full SHA
    ae84e97 View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2020

  1. Merge remote-tracking branch 'upstream/master' into autodiscovery-nomad

    @jorgelbg
    jorgelbg committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    5199248 View commit details
    Browse the repository at this point in the history

  2. Fix typo in license

    @jorgelbg
    jorgelbg committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    91d86a6 View commit details
    Browse the repository at this point in the history

  3. Remove vendor directory 

    PR elastic#18655 removed the vendor folder.
    @jorgelbg
    jorgelbg committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    c772a41 View commit details
    Browse the repository at this point in the history

  4. Fix typo on the autodiscover documentation

    @jorgelbg
    jorgelbg committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    8b8fe79 View commit details
    Browse the repository at this point in the history

  5. Fix unreachable code statement

    @jorgelbg
    jorgelbg committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    6702c69 View commit details
    Browse the repository at this point in the history

  6. Update NOTICE.txt (via mage)

    @jorgelbg
    jorgelbg committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    68b8886 View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2020

  1. Avoid skipping over stop events

    @jorgelbg
    jorgelbg committed Jul 27, 2020
    Configuration menu
    Copy the full SHA
    118427b View commit details
    Browse the repository at this point in the history

  2. Refactor the watcher tests into a table test

    @jorgelbg
    jorgelbg committed Jul 27, 2020
    Configuration menu
    Copy the full SHA
    fc671d4 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2020

  1. Make fields in hints-based autodiscover nomad specific 

    * Use `data.nomad` instead of `data.meta` for the hints-based autodiscover
* Update documentation to reflect the new path
* Add missing comments to several components
    @jorgelbg
    jorgelbg committed Nov 16, 2020
    Configuration menu
    Copy the full SHA
    8e59cda View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/master' into autodiscovery-nomad

    @jorgelbg
    jorgelbg committed Nov 16, 2020
    Configuration menu
    Copy the full SHA
    847cefe View commit details
    Browse the repository at this point in the history

  3. Fix signatuer of the AutodiscoverBuilder function 

    After PR elastic#20281 was merged, the signature of the `autodiscover.ProviderBuilder` changed to accept an additional name (string) parameter.
    @jorgelbg
    jorgelbg committed Nov 16, 2020
    Configuration menu
    Copy the full SHA
    5e456ed View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2020

  1. Update packetbeat configuration files (via mage update) 

    After catching up with the latest changes in master these files changed and CI was complaining about the difference.
    @jorgelbg
    jorgelbg committed Nov 17, 2020
    Configuration menu
    Copy the full SHA
    b971ec8 View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2021

  1. Use client status for the allocation status

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    61a3a56 View commit details
    Browse the repository at this point in the history

  2. Add add_nomad_metadata to the list of included processors

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    4016974 View commit details
    Browse the repository at this point in the history

  3. Small refactors on configuration 

    * Rename host to node for filtering, following nomad terminology.
* Pass address, namespace, region and secret ids to clients.
    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    a53aeef View commit details
    Browse the repository at this point in the history

  4. Add scope to nomad config

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    758d72f View commit details
    Browse the repository at this point in the history

  5. Refactor meta collection from tasks

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    1787d54 View commit details
    Browse the repository at this point in the history

  6. Update docs

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    f5b23b2 View commit details
    Browse the repository at this point in the history

  7. Fix cluster scope

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    8bf00a0 View commit details
    Browse the repository at this point in the history

  8. Merge remote-tracking branch 'origin/master' into autodiscover-nomad

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    601ebd9 View commit details
    Browse the repository at this point in the history

  9. Fix tests

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    434b133 View commit details
    Browse the repository at this point in the history

  10. Fix docs

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    5e40743 View commit details
    Browse the repository at this point in the history

  11. Add changelog entry

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    4a141e3 View commit details
    Browse the repository at this point in the history

  12. Use singular in datacenters field name

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    2115077 View commit details
    Browse the repository at this point in the history

  13. Reorganize allocation and job fields

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    6077aed View commit details
    Browse the repository at this point in the history

  14. Add fields definition

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    bf2c03d View commit details
    Browse the repository at this point in the history

  15. Fix test in Windows

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    a1dc3f1 View commit details
    Browse the repository at this point in the history

  16. Merge remote-tracking branch 'origin/master' into autodiscover-nomad

    @jsoriano @jorgelbg
    jsoriano authored and jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    e182038 View commit details
    Browse the repository at this point in the history

  17. Merge remote-tracking branch 'upstream/master' into autodiscovery-nomad

    @jorgelbg
    jorgelbg committed Jan 5, 2021
    Configuration menu
    Copy the full SHA
    221622b View commit details
    Browse the repository at this point in the history

  18. Remove duplicated entry in changelog

    @jsoriano
    jsoriano authored Jan 5, 2021
    Configuration menu
    Copy the full SHA
    a18e6bf View commit details
    Browse the repository at this point in the history

Commits on Jan 6, 2021

  1. Merge remote-tracking branch 'upstream/master' into autodiscovery-nomad

    @jorgelbg
    jorgelbg committed Jan 6, 2021
    Configuration menu
    Copy the full SHA
    ad49e39 View commit details
    Browse the repository at this point in the history