[RFC] Wildcard - stage 1 proposal

[ebeahan]

Summary

Revisions to the stage 0 wildcard adoption for consideration to be accepted as a stage 1 proposal.

Criteria for consideration

• Opened pull request revising the existing strawperson
• Identified "sponsor" at Elastic who will participate in RFC process and take ownership of the change after the process completes
• Types of fields or changes outlined
• High-level description of examples of usage
• High-level description of example sources of data
• Identified potential concerns and implementation challenges/complexity
• Subject matter experts identified and weighed in on the high level utility of these changes in the pull request
• ECS team weighed in on appropriateness of these changes in the pull request

Markdown preview of this RFC

[ebeahan]

Seeking feedback from folks on the wildcard adoption RFC. I've noted a few areas needing discussion before advancing as stage one:

• Any concerns with the outlined approach or any critical missing use cases that should be captured?
• Any concerns or implementation challenges overlooked?
• Any field sets not mentioned that should be represented? (Any specifics on particular fields is also welcomed.)

@webmat @jamiehynds @epixa @rw-access @randomuserid @leehinman @jonathan-buttner

[webmat]

Fantastic work, this is looking great!

I have a lot of comments below.

On the criteria checklist, I will check the ECS team weighing in checkbox. There's a few adjustments needed still for stage 1, but we're almost there I think. The approving PR review will be the actual gate, here. 🙂

On the other two criteria, let's leave until beginning of next week for folks to chime in, perhaps? But I think this stage 1 PR is solid enough to be merged soon, and we can tackle the nitty gritty details in the stage 2 PR.

The stage 2 PR is (among others) when I expect us to start listing exhaustively each field we will consider moving to wildcard. This is the moment where other SMEs participating actively will be a blocker for acceptance at stage 2.

[ebeahan]

Thanks @webmat for all the great, detailed feedback! I've made all the suggested changes except for the review comment discussing hostname, email, and username. I'm leaning towards removing all three from that section. WDYT?

[webmat]

the review comment discussing hostname, email, and username. I'm leaning towards removing all three from that section.

Let's leave it in for now. In stage 2, when we start doing a comprehensive list of all fields we consider migrating, we can come to a decision on this, and remove that section if appropriate.

[ebeahan]

@cyrille-leclerc @leehinman following up from our discussion last week, would you be able to briefly review and weigh in on the utility of these changes and any concerns or complexity that isn't captured?

[webmat]

I think we have a good coverage of the potential concerns. Let's check that box :-)

For my part, I'm good with this stage 1 proposal as it is 🚢

We're only waiting for a quick thumbs up / thumbs down from observability and security

@leehinman and @cyrille-leclerc (or representatives), what we're looking for at stage 1 is simply whether or not we've missed a big concern from your point of view, and whether or not this is heading in the right direction. Stage 2 is when we will hammer out the precise list of fields to migrate, and that's therefore when we will need more thorough reviews. For now I think a cursory review is enough.

[leehinman]

Another important item outside the scope of this proposal is backwards compatibility. Producers will need to consider the case where the producer has adopted wildcard, but the Elasticsearch instance they are sending data to doesn't support wildcard (example ES 6.8).

[leehinman]

+1 for moving to Stage 2

[roncohen]

generally LGTM 👍