Convert Filebeat iis.access to ECS #9084

webmat · 2018-11-14T21:33:15Z

Caveats

user agent is encoded in the log. Decoding to get better UA parsing results.
- It's not quoted, so spaces are replaced with + signs. Parens and slashes are not replaced, though.
- Using urldecode on it makes it more palatable to user_agent parser.

Renames

iis.access.server_ip => destination.ip
iis.access.remote_ip => source.ip
iis.access.method => http.request.method
iis.access.url => url.path
iis.access.query_string => url.query
iis.access.port => destination.port
iis.access.user_name => user.name
iis.access.referrer => http.request.referrer
iis.access.response_code => http.response.status_code
iis.access.hostname => destination.domain
iis.access.user_agent.original => user_agent.original
iis.access.geoip => source.geo

TODO

Convert status code, port, byte counts, request times to int
Update ECS-migration.yml file
Changelog
Create field aliases
Output user_agent to ECS field names
- Next step: figure out what to do with fully broken down OS version numbers, when ECS supports one version string :-)
List decoding of user agent in breaking changes

ruflin

Add changelog and migration file.

filebeat/module/iis/access/test/test.log-expected.json

webmat · 2018-11-20T13:46:43Z

@ruflin Ready for review.

ruflin

For user_agent problem, see other issue.

ruflin · 2018-11-20T13:50:24Z

filebeat/module/iis/access/ingest/default.json

-        "%{TIMESTAMP_ISO8601:iis.access.time} %{IPORHOST:iis.access.server_ip} %{WORD:iis.access.method} %{URIPATH:iis.access.url} %{NOTSPACE:iis.access.query_string} %{NUMBER:iis.access.port} %{NOTSPACE:iis.access.user_name} %{IPORHOST:iis.access.remote_ip} %{NOTSPACE:iis.access.agent} %{NOTSPACE:iis.access.referrer} %{NUMBER:iis.access.response_code} %{NUMBER:iis.access.sub_status} %{NUMBER:iis.access.win32_status} %{NUMBER:iis.access.request_time_ms}",
-        "%{TIMESTAMP_ISO8601:iis.access.time} %{NOTSPACE:iis.access.site_name} %{WORD:iis.access.method} %{URIPATH:iis.access.url} %{NOTSPACE:iis.access.query_string} %{NUMBER:iis.access.port} %{NOTSPACE:iis.access.user_name} %{IPORHOST:iis.access.remote_ip} %{NOTSPACE:iis.access.agent} %{NOTSPACE:iis.access.cookie} %{NOTSPACE:iis.access.referrer} %{NOTSPACE:iis.access.hostname} %{NUMBER:iis.access.response_code} %{NUMBER:iis.access.sub_status} %{NUMBER:iis.access.win32_status} %{NUMBER:iis.access.body_sent.bytes} %{NUMBER:iis.access.body_received.bytes} %{NUMBER:iis.access.request_time_ms}",
-        "%{TIMESTAMP_ISO8601:iis.access.time} %{NOTSPACE:iis.access.site_name} %{NOTSPACE:iis.access.server_name} %{IPORHOST:iis.access.server_ip} %{WORD:iis.access.method} %{URIPATH:iis.access.url} %{NOTSPACE:iis.access.query_string} %{NUMBER:iis.access.port} %{NOTSPACE:iis.access.user_name} %{IPORHOST:iis.access.remote_ip} HTTP/%{NUMBER:iis.access.http_version} %{NOTSPACE:iis.access.agent} %{NOTSPACE:iis.access.cookie} %{NOTSPACE:iis.access.referrer} %{NOTSPACE:iis.access.hostname} %{NUMBER:iis.access.response_code} %{NUMBER:iis.access.sub_status} %{NUMBER:iis.access.win32_status} %{NUMBER:iis.access.body_sent.bytes} %{NUMBER:iis.access.body_received.bytes} %{NUMBER:iis.access.request_time_ms}"
+        "%{TIMESTAMP_ISO8601:iis.access.time} %{IPORHOST:destination.ip} %{WORD:http.request.method} %{URIPATH:url.path} %{NOTSPACE:url.query} %{NUMBER:destination.port:int} %{NOTSPACE:user.name} %{IPORHOST:source.ip} %{NOTSPACE:iis.access.agent} %{NOTSPACE:http.request.referrer} %{NUMBER:http.response.status_code:int} %{NUMBER:iis.access.sub_status:int} %{NUMBER:iis.access.win32_status:int} %{NUMBER:iis.access.request_time_ms:int}",


IPORHOST here seems also one of these fields where we should do a follow up PR to make it only IP as otherwise it could break on ingest time.

filebeat/module/iis/access/ingest/default.json

webmat · 2018-11-21T22:16:32Z

@ruflin I've worked a bit on the user_agent parsing. You were right about the mapping issue. So this is resolved. My work has revealed a few things:

I noticed that in IIS logs, the UA is not quoted, and they therefore replace spaces with the +. I'm adding a urldecode step to get more precise UA parsing. (introduced in 6.1)
I added a different UA in the test log, which does get it's OS major and minor version parsed successfully. This revealed a problem, outlined at next point.
UA parser gives us fully broken down version fields (major, minor, patch), but no whole "version" field, whereas in ECS we only define "version".
- This is showing up as a problem for user_agent.os.version. For now I haven't fixed this issue yet, I only created fields temporarily, so the tests don't break for this.

I think there's many ways we can go about this.

ECS should likely add support for version breakdown fields
Tackling the user_agent in this PR is informative. But we may want to close the other PRs without the UA fix, and do a subsequent PR only about getting UA right in all access logs.

- iis.access.server_ip => destination.ip - iis.access.remote_ip => source.ip - iis.access.method => http.request.method - iis.access.url => url.path - iis.access.query_string => url.query - iis.access.port => destination.port - iis.access.user_name => user.name - iis.access.referrer => http.request.referrer - iis.access.response_code => http.response.status_code - iis.access.hostname => destination.domain - iis.access.user_agent.original => user_agent.original - iis.access.geoip => source.geo

…veals a bug

…his works.

webmat · 2018-11-22T19:09:11Z

This is ready for a final review.

Please make sure to check out the caveat above. I've made sure to list this "breaking" change improvement separately from the ECS transition.

ruflin · 2018-11-22T20:33:31Z

For the version: I prefer the full version string. We had it broken down in an early version of ECS but I didn't see the benefits. Elasticsearch offers prefix queries which means queries for major.minor as an example are possible. The only thing not possible is for example > major but I would rather see ES have a type version which would support all these type of queries.

ruflin

LGTM. Ready for merge.

For the user_agent decoding: Do we need this only for IIS or also in other modules?

ruflin · 2018-11-22T20:37:33Z

filebeat/module/iis/access/_meta/fields.yml

-          description: >
-            The major version of the operating system.
+          type: alias
+          path: user_agent.os.major


We could probably use script processor to concatenate the version fields. Initially hoped the the join processor would do this but seems it has a different purpose.

Yeah, but all of this refining needs to go in a shared pipeline. We're not going to duplicate this work everywhere, on the eve of starting to share pipelines ;-)

webmat · 2018-11-22T21:22:05Z

For the user_agent decoding: Do we need this only for IIS or also in other modules?

No, in all of the open source web servers modules I've been working on, the user agent was always a quoted string. None of them needed to encode all the spaces by replacing them with a +. IIS is the only one where I've seen that.

webmat · 2018-11-22T21:36:18Z

Sorry, had missed this comment #9084 (comment) before reading your review.

Reconstructing the version string is out of scope for this PR, IMO. The user_agent IN processor doesn't give it to us, at this time. I'm down with requesting that change to the IN team, or reconstructing it, but I don't think it should block this mapping to ECS.

I've added this to #9208 so we don't lose track of this.

…ccess logs (#9955) - Introduce IPv6 zone workaround iis.access log as well, resolving #9836. - Update the IPv6 zone fix (#9869) for iis.error to use the ECS `.address` field instead of a transient field. - Convert many fields under `iis.error.*` to ECS. Previous field names are field aliases towards the new corresponding ECS field: - iis.error.remote_ip => source.address - iis.error.remote_port => source.port - iis.error.server_ip => destination.address - iis.error.server_port => destination.port - iis.error.http_version => http.version - iis.error.method => http.request.method - iis.error.url => url.original - iis.error.response_code => http.response.status_code - iis.error.geoip.* => source.geo.* - read_timestamp => event.created (not aliased, still used elsewhere) - Update field aliases introduced in #9084 to point to `.address` instead of `.ip`, since this value can be ambiguous. The IP field is populated with the cleaned up IP without the zone. This is also true for the `.ip` fields under populated by the error logs. - iis.access.remote_ip => source.address - iis.access.server_ip => destination.address - Coerce to long: source.port, destination.port and http.response.status_code in the iis.error fileset

ruflin mentioned this pull request Nov 14, 2018

Transition Beats to ECS #8655

Closed

webmat self-assigned this Nov 14, 2018

webmat requested a review from ruflin November 14, 2018 21:33

webmat added in progress Pull request is currently in progress. module Filebeat Filebeat ecs labels Nov 14, 2018

ruflin reviewed Nov 19, 2018

View reviewed changes

filebeat/module/iis/access/test/test.log-expected.json Show resolved Hide resolved

webmat force-pushed the ecs-iis-access branch from 5afd5c5 to 1d16d14 Compare November 20, 2018 03:53

webmat removed the in progress Pull request is currently in progress. label Nov 20, 2018

ruflin added the review label Nov 20, 2018

ruflin changed the title ~~[WIP] Convert Filebeat iis.access to ECS~~ Convert Filebeat iis.access to ECS Nov 20, 2018

ruflin reviewed Nov 21, 2018

View reviewed changes

webmat force-pushed the ecs-iis-access branch from 2de8a2a to a8e1881 Compare November 21, 2018 22:08

Mathieu Martin added 12 commits November 22, 2018 14:00

Convert ints to ints

a78d499

Shove all renames so far in ecs-migration.yml

7e4ab6e

Changelog

befbc8b

Add http.request.referrer to this PR as well

edb2ae0

Output successfully to user_agent

d203691

URL decode the user agent string, to get more detailed UA analysis

ac46086

Introduce user agent whose version gets parsed successfully, which re…

eac7bb6

…veals a bug

Add temporary field defs to go past this failure for now

8bd9fcf

Rename so the version breakdown looks like ua.os.major/minor/patch. T…

108a02a

…his works.

Replace previous definitions with aliases for all normalized fields

db4191c

Move non-normalized field defs at the top, to promote sanity.

c36be2e

webmat force-pushed the ecs-iis-access branch from a8e1881 to c36be2e Compare November 22, 2018 19:08

Add breaking change about decoding the user agent.

2f98beb

ruflin approved these changes Nov 22, 2018

View reviewed changes

webmat merged commit 04c951d into elastic:master Nov 22, 2018

webmat deleted the ecs-iis-access branch November 22, 2018 21:44

webmat mentioned this pull request Dec 6, 2018

Re-introduce a simplified version of user_agent. elastic/ecs#240

Merged

webmat mentioned this pull request Dec 17, 2018

Fix recently translated Filebeat modules where integer coercions were added #9598

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert Filebeat iis.access to ECS #9084

Convert Filebeat iis.access to ECS #9084

webmat commented Nov 14, 2018 •

edited

Loading

ruflin left a comment

webmat commented Nov 20, 2018

ruflin left a comment

ruflin Nov 20, 2018

webmat commented Nov 21, 2018

webmat commented Nov 22, 2018

ruflin commented Nov 22, 2018

ruflin left a comment

ruflin Nov 22, 2018

webmat Nov 22, 2018

webmat commented Nov 22, 2018

webmat commented Nov 22, 2018

Convert Filebeat iis.access to ECS #9084

Convert Filebeat iis.access to ECS #9084

Conversation

webmat commented Nov 14, 2018 • edited Loading

ruflin left a comment

Choose a reason for hiding this comment

webmat commented Nov 20, 2018

ruflin left a comment

Choose a reason for hiding this comment

ruflin Nov 20, 2018

Choose a reason for hiding this comment

webmat commented Nov 21, 2018

webmat commented Nov 22, 2018

ruflin commented Nov 22, 2018

ruflin left a comment

Choose a reason for hiding this comment

ruflin Nov 22, 2018

Choose a reason for hiding this comment

webmat Nov 22, 2018

Choose a reason for hiding this comment

webmat commented Nov 22, 2018

webmat commented Nov 22, 2018

webmat commented Nov 14, 2018 •

edited

Loading