Epoch values should be considered valid timestamps

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/262

Originally created by @stkbailey on 2021-11-02 00:33:39

---

Summary

From what I can tell, the SDK cannot currently parse epoch-encoded timestamps and instead pushes them through the handle_invalid_timestamp logic. This is because the timestamp parsing is done using the dateutil.parser.parse functionality. Example below:

from dateutil import parser

ts = 1634606164.000200
parser.parse(ts)

>>> ERROR: dateutil.parser._parser.ParserError: year 1634606164 is out of range: 1634606164.000200

However, if we use datetime.datetime.fromtimestamp, the value is parsed successfully.

from datetime import datetime
datetime.fromtimestamp(ts)

>>> datetime.datetime(2021, 10, 18, 21, 16, 4, 200)

In this scenario, I would like to be able to use ts as a date-time formatted replication key column, but the strategy for doing so is not entirely clear to me.

Proposed benefits

Enabling epoch parsing would allow more timestamp data to be treated in a "standard" way.

Proposal details

I would like to see the SDK handle this case easily, to reduce friction when developing against this type of data.

Best reasons not to build

It is not entirely clear to me whether handling the data as a timestamp is the best way to approach this problem. Should we add a new "format" -- date_epoch? The major issue I could see is that, if an epoch timestamp is used as the replication key (e.g.), it would need to remain unconverted when being pushed to the external system. (The SDK shouldn't convert epoch -> date_iso8601, and then send that timestamp in the request.)

[MeltyBot]

View 3 previous comments from the original issue on GitLab

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

Still relevant.

Related:

• bug: _parse_timestamps_in_record throws exception for key not present in schema #1836
• [Bug]: Targets seem very slow #1181

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.