Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add config option to select a different azure cloud env in the azure-eventhub input and azure module #17659

Merged
merged 7 commits into from
Apr 21, 2020
Merged

Add config option to select a different azure cloud env in the azure-eventhub input and azure module #17659

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2d3aa03
add option to select a different azure env
narph Apr 10, 2020
f1dc7da
changelog
narph Apr 10, 2020
33dd8f3
rename
narph Apr 20, 2020
2fafce9
add test
narph Apr 20, 2020
4bbd313
Merge branch 'master' into add-other-env
narph Apr 20, 2020
c244182
mage fmt update
narph Apr 20, 2020
7f0414a
Merge branch 'master' into add-other-env
narph Apr 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add dashboard for Google Cloud Audit and AWS CloudTrail. {pull}17379[17379]
- Improve ECS categorization field mappings for mysql module. {issue}16172[16172] {pull}17491[17491]
- Release Google Cloud module as GA. {pull}17511[17511]
- Add config option to select a different azure cloud env in the azure-eventhub input and azure module. {issue}17649[17649] {pull}17659[17659]

*Heartbeat*

Expand Down
7 changes: 7 additions & 0 deletions filebeat/docs/modules/azure.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ Will retrieve azure Active Directory audit logs. The audit logs provide traceabi
connection_string: ""
storage_account: ""
storage_account_key: ""
override_resource_manager_endpoint: ""

auditlogs:
enabled: false
Expand All @@ -52,6 +53,7 @@ Will retrieve azure Active Directory audit logs. The audit logs provide traceabi
connection_string: ""
storage_account: ""
storage_account_key: ""
override_resource_manager_endpoint: ""

signinlogs:
enabled: false
Expand All @@ -61,6 +63,7 @@ Will retrieve azure Active Directory audit logs. The audit logs provide traceabi
connection_string: ""
storage_account: ""
storage_account_key: ""
override_resource_manager_endpoint: ""

```

Expand Down Expand Up @@ -90,6 +93,10 @@ The name of the storage account the state/offsets will be stored and updated.
_string_
The storage account key, this key will be used to authorize access to data in your storage account.

`override_resource_manager_endpoint` ::
_string_
Optional, by default we are using the azure public environment, to override, users can provide a specific resource manager endpoint in order to use a different azure environment.

include::../include/what-happens.asciidoc[]

include::../include/gs-link.asciidoc[]
Expand Down
11 changes: 8 additions & 3 deletions x-pack/filebeat/docs/inputs/input-azure-eventhub.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ Example configuration:
storage_account: "azureeph"
storage_account_key: "....."
storage_account_container: ""
override_resource_manager_endpoint: ""

----

==== Configuration options
Expand All @@ -36,7 +38,7 @@ The `azure-eventhub` input supports the following configuration:

==== `eventhub`

The name of the eventhub users would like to read from.
The name of the eventhub users would like to read from, field required.

==== `consumer_group`

Expand All @@ -50,14 +52,17 @@ A Blob Storage account is required in order to store/retrieve/update the offset

==== `storage_account`

The name of the storage account.
The name of the storage account. Required.

==== `storage_account_key`

The storage account key, this key will be used to authorize access to data in your storage account.
The storage account key, this key will be used to authorize access to data in your storage account, option is required.

==== `storage_account_container`

Optional, the name of the storage account container you would like to store the offset information in.

==== `override_resource_manager_endpoint`

Optional, by default we are using the azure public environment, to override, users can provide a specific resource manager endpoint in order to use a different azure environment.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with the comments by Blake and Kaiyan, don't consider this an override, just a normal setting. Add also something about possible values. And possibly something about hybrid clouds support.

Suggested change
==== `override_resource_manager_endpoint`
Optional, by default we are using the azure public environment, to override, users can provide a specific resource manager endpoint in order to use a different azure environment.
==== `resource_manager_endpoint`
Optional, resource manager endpoint to use, defaults to the public environment, possible values are... . If a URL is provided, the environment is obtained from the URL, this is useful in Hybrid Cloud deployments.

Maybe we could add an explicit option for environment URLs, instead of accepting names and urls in the same option. And maybe use something like environment instead of resource_manager_endpoint, in case we use the same settings for features that need access to other endpoints.

Suggested change
==== `override_resource_manager_endpoint`
Optional, by default we are using the azure public environment, to override, users can provide a specific resource manager endpoint in order to use a different azure environment.
==== `environment.name`
Optional, environment to use, defaults to the public environment, possible values are... .
==== `environment.url`
Optional, loads the environment from a URL, this is useful in the Hybrid Cloud model. If this setting is used, `environment.name` is ignored.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano , the reason I went for the override- is because I do not want users to necessary specify this setting, this should be only used for other private env. To look for the azure resource manager endpoint might take some effort for the users and I want to avoid that in most of the cases where the are using the public cloud.
One workaround I would go for is to rename this option to resource_manager_endpoint but not add it in the config files so users will not be tempted to enter it if unnecessary.
Not sure what you mean by " instead of accepting names and urls in the same option" , we only accept azure resource manager endpoints, the azure-eventhub sdk can retrieve the specific environment based on that.
Also adding environment.name and environment.url might cause more confusion as users will have a hard time figuring those out, also there is no environment.url , we can talk about management_portal_URL or resource_manager_endpoint or service_endpoint but with the resource_manager_endpoint the SDK can help us identify the right env.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano , the reason I went for the override- is because I do not want users to necessary specify this setting, this should be only used for other private env.

Other optional settings behave the same way, and don't have an override prefix, would it be different to other optional settings?

To look for the azure resource manager endpoint might take some effort for the users and I want to avoid that in most of the cases where the are using the public cloud. One workaround I would go for is to rename this option to resource_manager_endpoint but not add it in the config files so users will not be tempted to enter it if unnecessary.

Yes, I find perfectly fine to have this option in the reference docs, but not in the config files.

Not sure what you mean by " instead of accepting names and urls in the same option" , we only accept azure resource manager endpoints, the azure-eventhub sdk can retrieve the specific environment based on that.

Oh, ok, my mistake then, I saw that we accept URLs, and I thought we also accepted some kind of identifier there. One more reason to add some example values in the docs 🙂
I have still a question about that, the URLs accepted by EnvironmentFromURL, are of the same kind as the ones of the resource manager endpoints?

Also adding environment.name and environment.url might cause more confusion as users will have a hard time figuring those out, also there is no environment.url , we can talk about management_portal_URL or resource_manager_endpoint or service_endpoint but with the resource_manager_endpoint the SDK can help us identify the right env.

I was suggesting that to make configuration and its documentation easier. A user wanting to use the US Gov cloud would use environment.name: usgov, or something similar, accepted values could be in the documentation, so they don't need to look for specific endpoints in other places. A user that is using a hybrid cloud would use environment.url: http://.... with an URL that can be accepted by EnvironmentFromURL.
Something like this would be specially helpful if we need to use at some point other endpoints, e.g: the Service Manager Endpoint, so we don't have to add a setting like service_manager_endpoint.

But I don't know much about these endpoints and URLs, so as you prefer 🙂

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I answer myself after reading EnvironmentFromURL source. It seems that the URL accepted by this method is actually the resource manager endpoints and from that it obtains the other endpoints, so this solves my concerns about having different kind of values for hybrid clouds.

So I think that the only pending things to consider would be:

  • Rename of the setting to resource_manager_endpoint.
  • Document possible values for this setting, and how it works for hybrid clouds.


12 changes: 8 additions & 4 deletions x-pack/filebeat/filebeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,16 +288,18 @@ filebeat.modules:
activitylogs:
enabled: true
var:
# Eventhub name containing the activity logs, overwrite he default value if the logs are exported in a different eventhub
# eventhub name containing the activity logs, overwrite he default value if the logs are exported in a different eventhub
eventhub: "insights-operational-logs"
# Consumer group name that has access to the event hub, we advise creating a dedicated consumer group for the azure module
# consumer group name that has access to the event hub, we advise creating a dedicated consumer group for the azure module
consumer_group: "$Default"
# the connection string required to communicate with Event Hubs, steps to generate one here https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string
connection_string: ""
# the name of the storage account the state/offsets will be stored and updated.
# the name of the storage account the state/offsets will be stored and updated
storage_account: ""
#The storage account key, this key will be used to authorize access to data in your storage account.
# the storage account key, this key will be used to authorize access to data in your storage account
storage_account_key: ""
# by default the azure public environment is used, to override, users can provide a specific resource manager endpoint
override_resource_manager_endpoint: ""

auditlogs:
enabled: false
Expand All @@ -307,6 +309,7 @@ filebeat.modules:
# connection_string: ""
# storage_account: ""
# storage_account_key: ""
# override_resource_manager_endpoint: ""
signinlogs:
enabled: false
# var:
Expand All @@ -315,6 +318,7 @@ filebeat.modules:
# connection_string: ""
# storage_account: ""
# storage_account_key: ""
# override_resource_manager_endpoint: ""

#--------------------------------- CEF Module ---------------------------------
- module: cef
Expand Down
2 changes: 2 additions & 0 deletions x-pack/filebeat/input/azureeventhub/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ type azureInputConfig struct {
SAName string `config:"storage_account"`
SAKey string `config:"storage_account_key"`
SAContainer string `config:"storage_account_container"`
// by default the azure public environment is used, to override, users can provide a specific resource manager endpoint
OverrideEnvironment string `config:"override_resource_manager_endpoint"`
}

const ephContainerName = "filebeat"
Expand Down
26 changes: 25 additions & 1 deletion x-pack/filebeat/input/azureeventhub/eph.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,26 @@ import (
"github.com/Azure/go-autorest/autorest/azure"
)

// users can select from one of the already defined azure cloud envs
var environments = map[string]azure.Environment{
azure.ChinaCloud.ResourceManagerEndpoint: azure.ChinaCloud,
azure.GermanCloud.ResourceManagerEndpoint: azure.GermanCloud,
azure.PublicCloud.ResourceManagerEndpoint: azure.PublicCloud,
azure.USGovernmentCloud.ResourceManagerEndpoint: azure.USGovernmentCloud,
}

// runWithEPH will consume ingested events using the Event Processor Host (EPH) https://github.com/Azure/azure-event-hubs-go#event-processor-host, https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-event-processor-host
func (a *azureInput) runWithEPH() error {
// create a new Azure Storage Leaser / Checkpointer
cred, err := azblob.NewSharedKeyCredential(a.config.SAName, a.config.SAKey)
if err != nil {
return err
}
leaserCheckpointer, err := storage.NewStorageLeaserCheckpointer(cred, a.config.SAName, a.config.SAContainer, azure.PublicCloud)
env, err := getAzureEnvironment(a.config.OverrideEnvironment)
if err != nil {
return err
}
leaserCheckpointer, err := storage.NewStorageLeaserCheckpointer(cred, a.config.SAName, a.config.SAContainer, env)
if err != nil {
return err
}
Expand Down Expand Up @@ -74,3 +86,15 @@ func (a *azureInput) runWithEPH() error {
}
return nil
}

func getAzureEnvironment(overrideResManager string) (azure.Environment, error) {
narph marked this conversation as resolved.
Show resolved Hide resolved
// if no overrride is set then the azure public cloud is used
if overrideResManager == "" {
return azure.PublicCloud, nil
}
if env, ok := environments[overrideResManager]; ok {
return env, nil
}
Copy link
Member

@jsoriano jsoriano Apr 20, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. Should we try without the / path? So it works both for https://management.azure.com/ and https://management.azure.com.

// can retrieve hybrid env from the resource manager endpoint
return azure.EnvironmentFromURL(overrideResManager)
narph marked this conversation as resolved.
Show resolved Hide resolved
}
12 changes: 8 additions & 4 deletions x-pack/filebeat/module/azure/_meta/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,18 @@
activitylogs:
enabled: true
var:
# Eventhub name containing the activity logs, overwrite he default value if the logs are exported in a different eventhub
# eventhub name containing the activity logs, overwrite he default value if the logs are exported in a different eventhub
eventhub: "insights-operational-logs"
# Consumer group name that has access to the event hub, we advise creating a dedicated consumer group for the azure module
# consumer group name that has access to the event hub, we advise creating a dedicated consumer group for the azure module
consumer_group: "$Default"
# the connection string required to communicate with Event Hubs, steps to generate one here https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string
connection_string: ""
# the name of the storage account the state/offsets will be stored and updated.
# the name of the storage account the state/offsets will be stored and updated
storage_account: ""
#The storage account key, this key will be used to authorize access to data in your storage account.
# the storage account key, this key will be used to authorize access to data in your storage account
storage_account_key: ""
# by default the azure public environment is used, to override, users can provide a specific resource manager endpoint
override_resource_manager_endpoint: ""

auditlogs:
enabled: false
Expand All @@ -22,6 +24,7 @@
# connection_string: ""
# storage_account: ""
# storage_account_key: ""
# override_resource_manager_endpoint: ""
signinlogs:
enabled: false
# var:
Expand All @@ -30,3 +33,4 @@
# connection_string: ""
# storage_account: ""
# storage_account_key: ""
# override_resource_manager_endpoint: ""
7 changes: 7 additions & 0 deletions x-pack/filebeat/module/azure/_meta/docs.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ Will retrieve azure Active Directory audit logs. The audit logs provide traceabi
connection_string: ""
storage_account: ""
storage_account_key: ""
override_resource_manager_endpoint: ""

auditlogs:
enabled: false
Expand All @@ -47,6 +48,7 @@ Will retrieve azure Active Directory audit logs. The audit logs provide traceabi
connection_string: ""
storage_account: ""
storage_account_key: ""
override_resource_manager_endpoint: ""

signinlogs:
enabled: false
Expand All @@ -56,6 +58,7 @@ Will retrieve azure Active Directory audit logs. The audit logs provide traceabi
connection_string: ""
storage_account: ""
storage_account_key: ""
override_resource_manager_endpoint: ""

```

Expand Down Expand Up @@ -85,6 +88,10 @@ The name of the storage account the state/offsets will be stored and updated.
_string_
The storage account key, this key will be used to authorize access to data in your storage account.

`override_resource_manager_endpoint` ::
_string_
Optional, by default we are using the azure public environment, to override, users can provide a specific resource manager endpoint in order to use a different azure environment.

include::../include/what-happens.asciidoc[]

include::../include/gs-link.asciidoc[]
Expand Down
1 change: 1 addition & 0 deletions x-pack/filebeat/module/azure/activitylogs/config/azure-eventhub.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ eventhub: {{ .eventhub }}
consumer_group: {{ .consumer_group }}
storage_account: {{ .storage_account }}
storage_account_key: {{ .storage_account_key }}
override_resource_manager_endpoint: {{ .override_resource_manager_endpoint }}
1 change: 1 addition & 0 deletions x-pack/filebeat/module/azure/activitylogs/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ var:
- name: connection_string
- name: storage_account
- name: storage_account_key
- name: override_resource_manager_endpoint

ingest_pipeline:
- ingest/pipeline.json
Expand Down
1 change: 1 addition & 0 deletions x-pack/filebeat/module/azure/auditlogs/config/azure-eventhub.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ eventhub: {{ .eventhub }}
consumer_group: {{ .consumer_group }}
storage_account: {{ .storage_account }}
storage_account_key: {{ .storage_account_key }}
override_resource_manager_endpoint: {{ .override_resource_manager_endpoint }}

1 change: 1 addition & 0 deletions x-pack/filebeat/module/azure/auditlogs/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ var:
- name: connection_string
- name: storage_account
- name: storage_account_key
- name: override_resource_manager_endpoint

ingest_pipeline:
- ingest/pipeline.json
Expand Down
1 change: 1 addition & 0 deletions x-pack/filebeat/module/azure/signinlogs/config/azure-eventhub.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ eventhub: {{ .eventhub }}
consumer_group: {{ .consumer_group }}
storage_account: {{ .storage_account }}
storage_account_key: {{ .storage_account_key }}
override_resource_manager_endpoint: {{ .override_resource_manager_endpoint }}
1 change: 1 addition & 0 deletions x-pack/filebeat/module/azure/signinlogs/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ var:
- name: connection_string
- name: storage_account
- name: storage_account_key
- name: override_resource_manager_endpoint

ingest_pipeline:
- ingest/pipeline.json
Expand Down
12 changes: 8 additions & 4 deletions x-pack/filebeat/modules.d/azure.yml.disabled
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,18 @@
activitylogs:
enabled: true
var:
# Eventhub name containing the activity logs, overwrite he default value if the logs are exported in a different eventhub
# eventhub name containing the activity logs, overwrite he default value if the logs are exported in a different eventhub
eventhub: "insights-operational-logs"
# Consumer group name that has access to the event hub, we advise creating a dedicated consumer group for the azure module
# consumer group name that has access to the event hub, we advise creating a dedicated consumer group for the azure module
consumer_group: "$Default"
# the connection string required to communicate with Event Hubs, steps to generate one here https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string
connection_string: ""
# the name of the storage account the state/offsets will be stored and updated.
# the name of the storage account the state/offsets will be stored and updated
storage_account: ""
#The storage account key, this key will be used to authorize access to data in your storage account.
# the storage account key, this key will be used to authorize access to data in your storage account
storage_account_key: ""
# by default the azure public environment is used, to override, users can provide a specific resource manager endpoint
override_resource_manager_endpoint: ""

auditlogs:
enabled: false
Expand All @@ -25,6 +27,7 @@
# connection_string: ""
# storage_account: ""
# storage_account_key: ""
# override_resource_manager_endpoint: ""
signinlogs:
enabled: false
# var:
Expand All @@ -33,3 +36,4 @@
# connection_string: ""
# storage_account: ""
# storage_account_key: ""
# override_resource_manager_endpoint: ""