-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support remote logging in elasticsearch with filebeat 7 #14625
Conversation
I think this is a very internal details and the user should not configure it explicitly. If possible, we should check which version of the service we are connecting to and act accordingly. When this is not possible, we recommend introducing a more descriptive configuration option, e.g. use_filebeat_7_headers or something similar. Also, we should update the documentation and describe these configuration options. |
If anything I think we may want to go the other way and make it more configurable as filebeats is just one way to get logs into elasticsearch. I feel like we should try and be agnostic especially since Airflow isn't directly involved with getting the logs there in the first place. Thanks for the feedback though @mik-laj, I'll chew on this a bit more. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code LGTM -- I think given there are so many ways of getting logs in to ES (versions of filebeat, logstash, graylog, vector.dev etc) that we should improve the docs and say what we required, and leave it flexible.
Code LGTM too, seems like a nice minimal way to get exactly what's needed for filebeat 7 |
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
Filebeat 7 renamed some fields (offset->log.offset and host->host.name), so allow the field names Airflow uses to be configured. Airflow isn't directly involved with getting the logs _to_ elasticsearch, so we should allow easy configuration to accomodate whatever tools are used in that process.
Co-authored-by: Ash Berlin-Taylor <[email protected]>
Filebeat 7 renamed some fields (offset->log.offset and host->host.name), so allow the field names Airflow uses to be configured. Airflow isn't directly involved with getting the logs _to_ elasticsearch, so we should allow easy configuration to accomodate whatever tools are used in that process. (cherry picked from commit 5cd0bf7)
closes: #13755
This adds support to choose different fields for
host
andoffset
. Filebeat 7 changed the defaults, and while one could probably move them back, making it configurable makes more sense.https://www.elastic.co/guide/en/beats/libbeat/current/breaking-changes-7.0.html
offset
->log.offset
host.name
->host.hostname
(And based on #13755, something also uses a strhost
, not sure what though)This still needs test coverage.