Both columns for rolling and non-equi joins

[mattdowle]

I can't push to remote branch sritchie73:non-equi-key so copied it into main from PR #2706.
Looks like commit history from Scott is retained which is good, and retained the branch name.
The extended discussion history will just be in the previous PR #2706 though. Can't think of a way round this. Scott is member of project so can create branches in main going forward.

Closes #1615
Closes #1700
Closes #2006
Closes #2569
Other related issues already closed (e.g. dups) : #1469, #1807, #2307, #2595, #2602

• test.data.table() fails with 28 errors when datatable.bothkeycols is set to TRUE (caused by typo datatable.bothkeycol vs datatable.bothkeycols)
• bothkeycols= renamed to old.nonequi= (explanation in comments below)
• Place extra roll= column just after the first one, not at the end of the result.
• Add short examples to the news item

[mattdowle]

@sritchie73 Hence new strategy moving roll= into nomatch=. Better?

[sritchie73]

The new behavior also impacts non-equi joins, and has the potential to cause nasty surprises for those who expect the old column naming behavior, e.g. see the date column in this example:

> DT = data.table(id=c("foo","foo","bar"), date=c(1L,3L,5L), val=6:8, key="id,date")
> DT2 <- data.table(id = "foo", date.V2 = 2)
> DT[DT2, .(id, date, i.date.V2, x.date), on = .(id, date < date.V2), old.nonequi = TRUE] 
    id date i.date.V2 x.date
1: foo    2         2      1
> DT[DT2, .(id, date, i.date.V2, x.date),  on = .(id, date < date.V2)]
    id date i.date.V2 x.date
1: foo    1         2      1

I also think we do want to return both join columns when performing a rolling join, e.g. as @MarkusBonsch demonstrated in this comment, although I suspect you might be able to use non-equi joins in conjunction with mult= to achieve this.

[mattdowle]

Did you see I wrote :

We'd still have the issue of managing breaking changes in non-equi on=.

[sritchie73]

Sorry, must have missed it.

In any case its useful to have the above example demonstrating the breaking change for others coming to this PR so they don't need to dig through the old one.

What are your thoughts on the following if we move roll= into nomatch=?:

I also think we do want to return both join columns when performing a rolling join, e.g. as @MarkusBonsch demonstrated in this comment, although I suspect you might be able to use non-equi joins in conjunction with mult= to achieve this.

[mattdowle]

Again, I'm unsure why you're pointing to it in a way that makes me think you think I disagree or missed it. Did you see I wrote :

The new rolling join behaviour of this PR (i.e. returning both columns and with the different contents of the column with the same name) would be in nomatch= as per #857.

Are you reading on a phone or via email, perhaps? It's as if you are either not seeing all comments or not reading closely.

[sritchie73]

I'm not sure I follow. My understanding is that #857 proposes to remove roll= entirely, and move the functionality to nomatch=. A rolling join with nomatch= would "fill in" the values of the (single) join column where there is no direct match. The example I link to shows are case where you want the functionality of a rolling join, but where it makes sense to preserve both columns in the result. Unless I'm missing something, it would be impossible to do this with nomatch=?

[mattdowle]

Currently a rolling join ignores nomatch=. But a rolling join is what to do when there is no match, which is the idea of #857 to do it in nomatch=.
At the top of #857 it suggests a symbol nomatch=locf. Then later Arun suggests a function nomatch=LOCF(...) which I'm currently liking since a function can accept the limit and end control and avoids a new symbol: #857 (comment)

[MichaelChirico]

I believe this Q I was struggling a bit to answer cleanly on SO yesterday is related:

https://stackoverflow.com/q/56916899/3576984

[jangorecki]

@sritchie73 could you please merge to recent master and work out code coverage?

[tdhock]

hi any update/plan to merge this soon? I would love to use non-equi join more often in research and teaching and this PR would make it so much easier to use/explain.