Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for datetimes with Arrow #21866

Closed
joshuataylor opened this issue Mar 18, 2021 · 1 comment
Closed

Support for datetimes with Arrow #21866

joshuataylor opened this issue Mar 18, 2021 · 1 comment
Assignees
@Avogar
Labels
comp-formats Input / output formats unfinished code

Comments

@joshuataylor
Copy link
Contributor

Describe the unexpected behaviour
Trying to select data with a timestamp doesn't work as expected:

select * from file('/var/lib/clickhouse/user_files/data', ArrowStream, '\"FOOBAR\" Nullable(String), \"SOMETHING\" Nullable(DateTime)')


Code: 70, e.displayText() = DB::Exception: The type "struct" of an input column "SOMETHING" is not supported for conversion from a Arrow data format: While executing File (version 21.3.2.5 (official build))
 

2021.03.18 20:13:17.691548 [ 23230 ] {b4fa9c9d-1143-44ad-80e5-0faa222ab437} <Debug> executeQuery: (from 127.0.0.1:60822, using production parser) select * from file('/var/lib/clickhouse/user_files/data', ArrowStream, '\"FOOBAR\" Nullable(String), \"SOMETHING\" Nullable(DateTime)') FORMAT TabSeparatedWithNamesAndTypes;
2021.03.18 20:13:17.691631 [ 23230 ] {b4fa9c9d-1143-44ad-80e5-0faa222ab437} <Trace> ContextAccess (default): Access granted: CREATE TEMPORARY TABLE, FILE ON *.*
2021.03.18 20:13:17.691772 [ 23230 ] {b4fa9c9d-1143-44ad-80e5-0faa222ab437} <Trace> ContextAccess (default): Access granted: CREATE TEMPORARY TABLE, FILE ON *.*
2021.03.18 20:13:17.692087 [ 23230 ] {b4fa9c9d-1143-44ad-80e5-0faa222ab437} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
2021.03.18 20:13:17.692940 [ 23230 ] {b4fa9c9d-1143-44ad-80e5-0faa222ab437} <Error> executeQuery: Code: 70, e.displayText() = DB::Exception: The type "struct" of an input column "SOMETHING" is not supported for conversion from a Arrow data format: While executing File (version 21.3.2.5 (official build)) (from 127.0.0.1:60822) (in query: select * from file('/var/lib/clickhouse/user_files/data', ArrowStream, '\"FOOBAR\" Nullable(String), \"SOMETHING\" Nullable(DateTime)') FORMAT TabSeparatedWithNamesAndTypes;), Stack trace (when copying this message, always include the lines below):

0. DB::ArrowColumnToCHColumn::arrowTableToCHChunk(DB::Chunk&, std::__1::shared_ptr<arrow::Table>&, DB::Block const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) @ 0xf94b007 in /usr/bin/clickhouse
1. DB::ArrowBlockInputFormat::generate() @ 0xf941c06 in /usr/bin/clickhouse
2. DB::ISource::tryGenerate() @ 0xf8f7df5 in /usr/bin/clickhouse
3. DB::ISource::work() @ 0xf8f79ea in /usr/bin/clickhouse
4. DB::InputStreamFromInputFormat::readImpl() @ 0xd000f2f in /usr/bin/clickhouse
5. DB::IBlockInputStream::read() @ 0xe8ac565 in /usr/bin/clickhouse
6. DB::StorageFileSource::generate() @ 0xf318ea9 in /usr/bin/clickhouse
7. DB::ISource::tryGenerate() @ 0xf8f7df5 in /usr/bin/clickhouse
8. DB::ISource::work() @ 0xf8f79ea in /usr/bin/clickhouse
9. DB::SourceWithProgress::work() @ 0xfaaa4ba in /usr/bin/clickhouse
10. ? @ 0xf931ddd in /usr/bin/clickhouse
11. DB::PipelineExecutor::executeStepImpl(unsigned long, unsigned long, std::__1::atomic<bool>*) @ 0xf92ea01 in /usr/bin/clickhouse
12. DB::PipelineExecutor::executeImpl(unsigned long) @ 0xf92c956 in /usr/bin/clickhouse
13. DB::PipelineExecutor::execute(unsigned long) @ 0xf92c6ad in /usr/bin/clickhouse
14. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>) @ 0xf1316a6 in /usr/bin/clickhouse
15. DB::HTTPHandler::processQuery(DB::Context&, DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::__1::optional<DB::CurrentThread::QueryScope>&) @ 0xf84675a in /usr/bin/clickhouse
16. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0xf84a8ee in /usr/bin/clickhouse
17. DB::HTTPServerConnection::run() @ 0xf8d35ff in /usr/bin/clickhouse
18. Poco::Net::TCPServerConnection::start() @ 0x11f7ccbf in /usr/bin/clickhouse
19. Poco::Net::TCPServerDispatcher::run() @ 0x11f7e6d1 in /usr/bin/clickhouse
20. Poco::PooledThread::run() @ 0x120b4df9 in /usr/bin/clickhouse
21. Poco::ThreadImpl::runnableEntry(void*) @ 0x120b0c5a in /usr/bin/clickhouse
22. start_thread @ 0x9590 in /usr/lib/x86_64-linux-gnu/libpthread-2.32.so
23. __clone @ 0x119223 in /usr/lib/x86_64-linux-gnu/libc-2.32.so

2021.03.18 20:13:17.693109 [ 23230 ] {b4fa9c9d-1143-44ad-80e5-0faa222ab437} <Error> DynamicQueryHandler: Code: 70, e.displayText() = DB::Exception: The type "struct" of an input column "SOMETHING" is not supported for conversion from a Arrow data format: While executing File, Stack trace (when copying this message, always include the lines below):

0. DB::ArrowColumnToCHColumn::arrowTableToCHChunk(DB::Chunk&, std::__1::shared_ptr<arrow::Table>&, DB::Block const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) @ 0xf94b007 in /usr/bin/clickhouse
1. DB::ArrowBlockInputFormat::generate() @ 0xf941c06 in /usr/bin/clickhouse
2. DB::ISource::tryGenerate() @ 0xf8f7df5 in /usr/bin/clickhouse
3. DB::ISource::work() @ 0xf8f79ea in /usr/bin/clickhouse
4. DB::InputStreamFromInputFormat::readImpl() @ 0xd000f2f in /usr/bin/clickhouse
5. DB::IBlockInputStream::read() @ 0xe8ac565 in /usr/bin/clickhouse
6. DB::StorageFileSource::generate() @ 0xf318ea9 in /usr/bin/clickhouse
7. DB::ISource::tryGenerate() @ 0xf8f7df5 in /usr/bin/clickhouse
8. DB::ISource::work() @ 0xf8f79ea in /usr/bin/clickhouse
9. DB::SourceWithProgress::work() @ 0xfaaa4ba in /usr/bin/clickhouse
10. ? @ 0xf931ddd in /usr/bin/clickhouse
11. DB::PipelineExecutor::executeStepImpl(unsigned long, unsigned long, std::__1::atomic<bool>*) @ 0xf92ea01 in /usr/bin/clickhouse
12. DB::PipelineExecutor::executeImpl(unsigned long) @ 0xf92c956 in /usr/bin/clickhouse
13. DB::PipelineExecutor::execute(unsigned long) @ 0xf92c6ad in /usr/bin/clickhouse
14. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>) @ 0xf1316a6 in /usr/bin/clickhouse
15. DB::HTTPHandler::processQuery(DB::Context&, DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::__1::optional<DB::CurrentThread::QueryScope>&) @ 0xf84675a in /usr/bin/clickhouse
16. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0xf84a8ee in /usr/bin/clickhouse
17. DB::HTTPServerConnection::run() @ 0xf8d35ff in /usr/bin/clickhouse
18. Poco::Net::TCPServerConnection::start() @ 0x11f7ccbf in /usr/bin/clickhouse
19. Poco::Net::TCPServerDispatcher::run() @ 0x11f7e6d1 in /usr/bin/clickhouse
20. Poco::PooledThread::run() @ 0x120b4df9 in /usr/bin/clickhouse
21. Poco::ThreadImpl::runnableEntry(void*) @ 0x120b0c5a in /usr/bin/clickhouse
22. start_thread @ 0x9590 in /usr/lib/x86_64-linux-gnu/libpthread-2.32.so
23. __clone @ 0x119223 in /usr/lib/x86_64-linux-gnu/libc-2.32.so
 (version 21.3.2.5 (official build))

==> clickhouse-server.err.log <==
2021.03.18 20:13:17.692940 [ 23230 ] {b4fa9c9d-1143-44ad-80e5-0faa222ab437} <Error> executeQuery: Code: 70, e.displayText() = DB::Exception: The type "struct" of an input column "SOMETHING" is not supported for conversion from a Arrow data format: While executing File (version 21.3.2.5 (official build)) (from 127.0.0.1:60822) (in query: select * from file('/var/lib/clickhouse/user_files/data', ArrowStream, '\"FOOBAR\" Nullable(String), \"SOMETHING\" Nullable(DateTime)') FORMAT TabSeparatedWithNamesAndTypes;), Stack trace (when copying this message, always include the lines below):

0. DB::ArrowColumnToCHColumn::arrowTableToCHChunk(DB::Chunk&, std::__1::shared_ptr<arrow::Table>&, DB::Block const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) @ 0xf94b007 in /usr/bin/clickhouse
1. DB::ArrowBlockInputFormat::generate() @ 0xf941c06 in /usr/bin/clickhouse
2. DB::ISource::tryGenerate() @ 0xf8f7df5 in /usr/bin/clickhouse
3. DB::ISource::work() @ 0xf8f79ea in /usr/bin/clickhouse
4. DB::InputStreamFromInputFormat::readImpl() @ 0xd000f2f in /usr/bin/clickhouse
5. DB::IBlockInputStream::read() @ 0xe8ac565 in /usr/bin/clickhouse
6. DB::StorageFileSource::generate() @ 0xf318ea9 in /usr/bin/clickhouse
7. DB::ISource::tryGenerate() @ 0xf8f7df5 in /usr/bin/clickhouse
8. DB::ISource::work() @ 0xf8f79ea in /usr/bin/clickhouse
9. DB::SourceWithProgress::work() @ 0xfaaa4ba in /usr/bin/clickhouse
10. ? @ 0xf931ddd in /usr/bin/clickhouse
11. DB::PipelineExecutor::executeStepImpl(unsigned long, unsigned long, std::__1::atomic<bool>*) @ 0xf92ea01 in /usr/bin/clickhouse
12. DB::PipelineExecutor::executeImpl(unsigned long) @ 0xf92c956 in /usr/bin/clickhouse
13. DB::PipelineExecutor::execute(unsigned long) @ 0xf92c6ad in /usr/bin/clickhouse
14. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>) @ 0xf1316a6 in /usr/bin/clickhouse
15. DB::HTTPHandler::processQuery(DB::Context&, DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::__1::optional<DB::CurrentThread::QueryScope>&) @ 0xf84675a in /usr/bin/clickhouse
16. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0xf84a8ee in /usr/bin/clickhouse
17. DB::HTTPServerConnection::run() @ 0xf8d35ff in /usr/bin/clickhouse
18. Poco::Net::TCPServerConnection::start() @ 0x11f7ccbf in /usr/bin/clickhouse
19. Poco::Net::TCPServerDispatcher::run() @ 0x11f7e6d1 in /usr/bin/clickhouse
20. Poco::PooledThread::run() @ 0x120b4df9 in /usr/bin/clickhouse
21. Poco::ThreadImpl::runnableEntry(void*) @ 0x120b0c5a in /usr/bin/clickhouse
22. start_thread @ 0x9590 in /usr/lib/x86_64-linux-gnu/libpthread-2.32.so
23. __clone @ 0x119223 in /usr/lib/x86_64-linux-gnu/libc-2.32.so

2021.03.18 20:13:17.693109 [ 23230 ] {b4fa9c9d-1143-44ad-80e5-0faa222ab437} <Error> DynamicQueryHandler: Code: 70, e.displayText() = DB::Exception: The type "struct" of an input column "SOMETHING" is not supported for conversion from a Arrow data format: While executing File, Stack trace (when copying this message, always include the lines below):

0. DB::ArrowColumnToCHColumn::arrowTableToCHChunk(DB::Chunk&, std::__1::shared_ptr<arrow::Table>&, DB::Block const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) @ 0xf94b007 in /usr/bin/clickhouse
1. DB::ArrowBlockInputFormat::generate() @ 0xf941c06 in /usr/bin/clickhouse
2. DB::ISource::tryGenerate() @ 0xf8f7df5 in /usr/bin/clickhouse
3. DB::ISource::work() @ 0xf8f79ea in /usr/bin/clickhouse
4. DB::InputStreamFromInputFormat::readImpl() @ 0xd000f2f in /usr/bin/clickhouse
5. DB::IBlockInputStream::read() @ 0xe8ac565 in /usr/bin/clickhouse
6. DB::StorageFileSource::generate() @ 0xf318ea9 in /usr/bin/clickhouse
7. DB::ISource::tryGenerate() @ 0xf8f7df5 in /usr/bin/clickhouse
8. DB::ISource::work() @ 0xf8f79ea in /usr/bin/clickhouse
9. DB::SourceWithProgress::work() @ 0xfaaa4ba in /usr/bin/clickhouse
10. ? @ 0xf931ddd in /usr/bin/clickhouse
11. DB::PipelineExecutor::executeStepImpl(unsigned long, unsigned long, std::__1::atomic<bool>*) @ 0xf92ea01 in /usr/bin/clickhouse
12. DB::PipelineExecutor::executeImpl(unsigned long) @ 0xf92c956 in /usr/bin/clickhouse
13. DB::PipelineExecutor::execute(unsigned long) @ 0xf92c6ad in /usr/bin/clickhouse
14. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>) @ 0xf1316a6 in /usr/bin/clickhouse
15. DB::HTTPHandler::processQuery(DB::Context&, DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::__1::optional<DB::CurrentThread::QueryScope>&) @ 0xf84675a in /usr/bin/clickhouse
16. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0xf84a8ee in /usr/bin/clickhouse
17. DB::HTTPServerConnection::run() @ 0xf8d35ff in /usr/bin/clickhouse
18. Poco::Net::TCPServerConnection::start() @ 0x11f7ccbf in /usr/bin/clickhouse
19. Poco::Net::TCPServerDispatcher::run() @ 0x11f7e6d1 in /usr/bin/clickhouse
20. Poco::PooledThread::run() @ 0x120b4df9 in /usr/bin/clickhouse
21. Poco::ThreadImpl::runnableEntry(void*) @ 0x120b0c5a in /usr/bin/clickhouse
22. start_thread @ 0x9590 in /usr/lib/x86_64-linux-gnu/libpthread-2.32.so
23. __clone @ 0x119223 in /usr/lib/x86_64-linux-gnu/libc-2.32.so
 (version 21.3.2.5 (official build))

==> clickhouse-server.log <==
2021.03.18 20:13:17.693257 [ 23230 ] {b4fa9c9d-1143-44ad-80e5-0faa222ab437} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.

How to reproduce

  • Version: ClickHouse server version 21.3.2.5 (official build).
    Data attached
    data.zip

Expected behavior
It works

Additional context
https://arrow.apache.org/docs/python/timestamps.html

Arrow timestamps are stored as a 64-bit integer with column metadata to associate a time unit (e.g. milliseconds, microseconds, or nanoseconds), and an optional time zone.
@filimonov filimonov added the comp-formats Input / output formats label Mar 18, 2021
@Avogar Avogar self-assigned this May 18, 2021
@Avogar Avogar mentioned this issue May 25, 2021
Merged
@Avogar Avogar closed this as completed Jun 22, 2021
@Avogar
Copy link
Member

Avogar commented Jun 22, 2021

Should be fixed in #24341

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Avogar Avogar

Labels
comp-formats Input / output formats unfinished code
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@joshuataylor @filimonov @Avogar