Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: earliest/latest_by_offset should accept nulls #5729

Merged
merged 21 commits into from
Sep 11, 2020

Conversation

big-andy-coates
Copy link
Contributor

@big-andy-coates big-andy-coates commented Jun 30, 2020

Description

fixes: #5727

Adds additional, optional, ignoreNulls parameter to the EARLIEST_BY_OFFSET and LATEST_BY_OFFSET UDAFs, that controls null handling. The default is to ignore nulls, which is inline with previous behaviour. If the user passes ignoreNulls as false then the methods will treat null as a valid value and use it as the first/last value.

To achieve this, I've removed the limit on UDAFs that said you could only have one constant parameter. UDAFs now support any number of constant parameters, (a.k.a. literals). Though they can still only have one non-constant param.

Testing done

Usual. Including adding tests around session window merging.

Reviewer checklist

  • Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
  • Ensure relevant issues are linked (description should include text like "Fixes #")

fixes: confluentinc#5727

The `EARLIEST_BY_OFFSET` and `LATEST_BY_OFFSET` should treat a `NULL` value the same as any other value.

If the first value seen is `null`, then `EARLIEST_BY_OFFSET` should return `null`.
If the last value seen is `null`, then `LATEST_BY_OFFSET` should return `null`.
@big-andy-coates big-andy-coates requested a review from a team as a code owner June 30, 2020 15:08
@big-andy-coates big-andy-coates changed the base branch from master to 6.0.x June 30, 2020 15:08
@big-andy-coates big-andy-coates added this to the 0.10.1 milestone Jun 30, 2020
@apurvam apurvam modified the milestones: 0.10.1, 0.11.0 Jul 1, 2020
@big-andy-coates
Copy link
Contributor Author

big-andy-coates commented Jul 6, 2020

Update: Blocked during discussions of best way to proceed on this - see #5727, and other PR in same area of code: #6014

@big-andy-coates big-andy-coates changed the base branch from 6.0.x to master September 9, 2020 15:14
@big-andy-coates
Copy link
Contributor Author

Ready for review

Copy link
Member

@JimGalasyn JimGalasyn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, with some copy edits.

Copy link
Member

@spena spena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

big-andy-coates and others added 18 commits September 9, 2020 22:22
@big-andy-coates big-andy-coates merged commit 6eb5a41 into confluentinc:master Sep 11, 2020
@big-andy-coates big-andy-coates deleted the nulls_by_offset branch September 11, 2020 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

earliest/latest_by_offset should not ignore null values
4 participants