Conversion from python datetime to C++ time_point

[AlexanderZvyagin]

According to the documentation:

datetime.datetime → std::chrono::system_clock::time_point
Date/time objects are converted into system clock timepoints. Any timezone information is ignored and the type is treated as a naive object.

But this is what I observe:
# PYTHON
d=datetime.datetime(2018,9,3)
d.utctimetuple() # ...tm_year=2018, tm_mon=9, tm_mday=3 .... tm_isdst=0
// C++
#include <date/date.h>
void f(const std::chrono::system_clock::time_point &d1) {
using date::operator<<;
date::year_month_day d2 = date::floor<date::days>(d1);
std::cout << d1 << " " << d2 << "\n"; // 2018-09-02 22:00:00.000000000 2018-09-02
}

And because of the two hours shift, I have a wrong date in my C++ code.

I am confused, how to convert a date from python to C++ with the library?!

[YannickJadoul]

Should be fixed by #1844 and #1848 introducing casters for datetime.date?

[jeandet]

Hello @YannickJadoul, this issue has been closed but I'm not sure this is actually solved.

with the following C++ code:

m.def("datetime_loop",
        [](std::chrono::time_point<std::chrono::system_clock,std::chrono::nanoseconds> tp)
        {
            std::cout << tp << "\n";
            std::cout << tp.time_since_epoch() << "\n";
            return tp;
        });

I get from Python:

>>> from datetime import datetime, timezone
>>> import cpp_mod
>>> cpp_mod.datetime_loop(datetime(2022,1,1,20))
2022-01-01 19:00:00.000000000
1641063600000000000ns
datetime.datetime(2022, 1, 1, 20, 0)
>>> cpp_mod.datetime_loop(datetime(2022,1,1,20, tzinfo=timezone.utc))
2022-01-01 19:00:00.000000000
1641063600000000000ns
datetime.datetime(2022, 1, 1, 20, 0)
>>> import os
>>> os.environ['TZ'] = 'UTC'
>>> cpp_mod.datetime_loop(datetime(2022,1,1,20))
2022-01-01 20:00:00.000000000
1641067200000000000ns
datetime.datetime(2022, 1, 1, 20, 0)
>>> os.environ['TZ'] = 'Africa/Addis_Ababa'
>>> cpp_mod.datetime_loop(datetime(2022,1,1,20))
2022-01-01 17:00:00.000000000
1641056400000000000ns
datetime.datetime(2022, 1, 1, 20, 0)

I'm not sure this should behave like this, we should be able to have the same date from python and C++ and TZ should be either preserved or always ignored. Having a quick look into Pybind11 source code, I'm not sure where this come from.

[kcajf]

I can reproduce what @jeandet observed - could this issue be re-opened please? It seems clear that the conversion behaviour of a timezone-naive datetime.datetime should not depend on the system timezone.

[agoose77]

@henryiii this issue relates to the problems that I was having with Python→ C++ timezone handling.

The docs say:

datetime.datetime or datetime.date or datetime.time → std::chrono::system_clock::time_point
Date/time objects are converted into system clock timepoints. Any timezone information is ignored and the type is treated as a naive object.

I think this last part is not quite right; the current conversion passes tm_isdst=-1 and calls mktime. This converts from local time into UTC. This is a lossy conversion (1 AM BST and 2 AM BST are the same time: 1AM UTC), and depends upon the system localtime configuration (e.g. the TZ environment variable).

In my case (UK), I'm interested in representing the datetime as "time since epoch". Because pybind11 (via mktime) is assuming I'm giving it datetime objects according to my localtime, it dutifully accounts for daylight savings, thinking I'm giving it a BST-adjusted time. However, I don't want that to happen; I want mktime to treat my input as UTC. However, there's no interface for doing this. Were it not lossless, users could post-process the times given by the conversion routines to detect whether a DST correction has been applied (tm_isdst == 1), and then determine the DST offset¹.

I think we actually want the other way around; to treat the Python datetime as timezone-less, we need to work in (and, in the case of this issue, accept datetimes as) UTC where there are no offsets (or DST changes).

There is timegm to build the time_t value from a UTC tm struct, but this is non-standard. There's also the date library which looks promising.

Footnotes

1. set tm_isdst=0, call mktime again (maketime will realise the given time is during the DST period and "fix" your time), take difference from original time. ↩