Fix #459: In CI, polars error: col timestamp_right already exists

[trentmc]

No description provided.

[idiom-bytes]

I'm assuming no timestamp because now it's the pd index

[trentmc]

Yes

[idiom-bytes]

I believe the reason why we are getting _right columns, is because of the outer join. Generally, I don't like using outer joins as they can introduce unexpected data.

As long as we use left joins on expected timestamps, this should not happen. This can be done in two ways:

1. With a dataset you trust (i.e. kaiko that may have guaranteed timestamps)
2. With a dataset you build (i.e. just populate 1 column w/ expect values)

example

# Create a range of timestamps with 5-minute intervals
timestamps = pd.date_range(start='2023-01-01', periods=100, freq='5T')
timestamps = timestamps.astype(int) // 10**9  # Convert to Unix timestamps in seconds

# Create a Polars DataFrame
df = pl.DataFrame({
    "timestamp": timestamps
})

merged_df.extend(df)
merged_df = merged_df.join(newraw_df, on="timestamp", how="left")

This avoids:

• situations where you introduce bad records into the data structure due to outer join
• situations where you get _right/_left columns due to outer join
• the need to run merge_cols() to drop _left/_right

This also enforces strict timestamp records in your data structure that you're expecting. I.E. It either joins w/ expected timestamps, or it's dropped.

[trentmc]

Fyi I was building off the outer join code that you had introduced.

We can't just use "data we trust". There's always going to be missing values etc and we need to account for it. Eg most people using pdr-backend will not be buying Kaiko data.

The example improvement you gave us to a good idea. If you want to make it happen, I recommend to make a new github issue.