Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fields.yml doesn't generate correct properties for elasticsearch template for nested field types #23178

Closed
dcode opened this issue Dec 16, 2020 · 1 comment · Fixed by #23183
Labels
needs_team Indicates that the issue/PR needs a Team:* label

Comments

@dcode
Copy link
Contributor

dcode commented Dec 16, 2020

Version: x-pack filebeat, master branch, commit 87ff5c086
Operating System: Mac OS 10.15.7
Steps to Reproduce:

For my PR #21815, some data is modeled as nested data in order to maintain the proper semantics. Given the example fields.yml entry below, I expect that the fields would be implemented in the Elasticsearch template as properties the same as if the type was marked as group, similar to what is found in the Elasticsearch docs.

The current behavior yields a mapping that relies on dynamic mapping. Unfortunately, in this particular example, sometimes the chi2 property is parsed as an integer, sometimes it is parsed as a float, causing a mapping conflict during ingest of documents.

Example: fields.yml

- name: file.pe.overlay
  type: nested
  fields:
    - name: chi2
      description: >
        Chi2 information of the PE file.
      type: float
    - name: entropy
      description: >
        Entropy information of the PE file.
      type: float
    - name: filetype
      description: >
        Filetype of the PE file.
      type: keyword
    - name: md5
      description: >
        Overlay MD5 hash of the PE file.
      type: keyword
    - name: offset
      description: >
        Offset of the overlay information of the PE file.
      type: keyword
    - name: size
      description: >
        Size of the PE file.
      format: bytes
      type: long

Actual resulting mapping (trimmed to only include field definition)

              "overlay": {
                "type": "nested"
              },

Expected resulting mapping

              "overlay": {
                "type": "nested",
                "properties": {
                  "chi2": {
                    "type": "float"
                  },
                  "filetype": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "entropy": {
                    "type": "float"
                  },
                  "offset": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "size": {
                    "type": "long"
                  },
                  "md5": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              },
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 16, 2020
@botelastic
Copy link

botelastic bot commented Dec 16, 2020

This issue doesn't have a Team:<team> label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant