[Bug]: If you choose a date format mask when entering a value for the filter transform, your value will not be used for compare corrrectly.

[usbrandon]

Apache Hop version?

2.5-GA

Java version?

11

Operating system

Windows

What happened?

When you are setting up your Filter Transform and entering a value, if you say choose
2023-02-01 00:00:00, and use the proper date mask matching that and Ok through it, you'll see a different date mask representation of the date value in the filter on the right side. The filter will now fail to compare the right way giving you output like this:

The step will work if you happen to enter your date values exactly as the default date mask suggests, then when you run the pipeline the filter will be applied correctly.

The attached can reproduce the issue.
sample_filter.zip

Issue Priority

Priority: 3

Issue Component

Component: Transforms

[mattcasters]

The mask is indeed saved and used but it is incorrect for the milliseconds specified.

[mattcasters]

TODO: Allow editing of the mask on previously entered values, despite the error thrown: see issue #3196

[usbrandon]

Remember, a key behavior is that if you were to put yyyy-MM-dd and a value like 2023-12-06, and then click Ok. When you go back into the filter step, it will forget your manually entered mask and try to put its own in there not matching it and permanently fail.

[adbrown101]

Same issue noted here:

hop version 2.7.0 (2023-11-17 12.19.09)
java.specification.version=11
java.version=11.0.20
java.vm.vendor=Microsoft

Then if I go and try and edit the condition I get the following error.

HopValueException: 
constant String : couldn't convert string [2023/03/31 00:00:00.000] to a date using format [yyyy-MM-dd] on offset location 4
2023/03/31 00:00:00.000

Root cause: ParseException: 2023/03/31 00:00:00.000

If I put the date in yyyy/MM/dd hh:MM:ss format the comparison works, but I do prefer to use yyyy-MM-dd for all of my date masks.

[sramazzina]

@usbrandon one of the dates' mask definition in your sample was wrong. That was the reason why the filter transform was raising an exception. Try my sample
sample_filter1.zip

The only issue I foresee is related to code hardening by putting a check over the mask and see that is aligned to the expectations of the date you entered. I will move on that way for the moment as a fix for this transform. Let me know what you think. My sample works correctly and returns the row you are expecting

[sramazzina]

@adbrown101, as I was writing in my previous message, the date's format mask MUST be aligned to the format of the date you entered. Otherwise it cannot work.

[usbrandon]

Reproduction video

https://www.youtube.com/watch?v=HeC9oQKygQU

[usbrandon]

The only workaround currently is to format your dates as the transform demands with milliseconds at the end and according to your local java date format with slashes instead of dashes etc. If you customize the date mask to match your stream in any way, then the transform hammers in its mask over what you had but records what you had, just doesn't use it.