[CI][C++] CPP valgrind job is failing on Azure tests

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

test-conda-cpp-valgrind is failing on TestAzureFileSystenOnAllEnv and TestAzureFileSystemOnAllScenarios test failures:

[  FAILED  ] 5 tests, listed below:
[  FAILED  ] TestAzureFileSystemOnAllEnvs/0.CreateDirOnRoot, where TypeParam = arrow::fs::TestingScenario<arrow::fs::AzuriteEnv, false>
[  FAILED  ] TestAzureFileSystemOnAllScenarios/0.CreateDirOnMissingContainer, where TypeParam = arrow::fs::TestingScenario<arrow::fs::AzuriteEnv, true>
[  FAILED  ] TestAzureFileSystemOnAllScenarios/0.OpenOutputStreamWithMissingContainer, where TypeParam = arrow::fs::TestingScenario<arrow::fs::AzuriteEnv, true>
[  FAILED  ] TestAzureFileSystemOnAllScenarios/1.CreateDirOnMissingContainer, where TypeParam = arrow::fs::TestingScenario<arrow::fs::AzuriteEnv, false>
[  FAILED  ] TestAzureFileSystemOnAllScenarios/1.OpenOutputStreamWithMissingContainer, where TypeParam = arrow::fs::TestingScenario<arrow::fs::AzuriteEnv, false>

 5 FAILED TESTS
==16391== 
==16391== HEAP SUMMARY:
==16391==     in use at exit: 11,298 bytes in 159 blocks
==16391==   total heap usage: 3,136,856 allocs, 3,136,697 frees, 5,688,532,534 bytes allocated
==16391== 
==16391== 1,424 bytes in 2 blocks are definitely lost in loss record 19 of 20
==16391==    at 0x485094F: calloc (vg_replace_malloc.c:1595)
==16391==    by 0x7EA454F: xmlGetThreadLocalStorage (in /opt/conda/envs/arrow/lib/libxml2.so.2.12.6)
==16391==    by 0x7EA52DD: __xmlDefaultBufferSize (in /opt/conda/envs/arrow/lib/libxml2.so.2.12.6)
==16391==    by 0x7EE149A: xmlAllocParserInputBuffer (in /opt/conda/envs/arrow/lib/libxml2.so.2.12.6)
==16391==    by 0x7EE1ACA: xmlParserInputBufferCreateStatic (in /opt/conda/envs/arrow/lib/libxml2.so.2.12.6)
==16391==    by 0x7EE1B6D: xmlParserInputBufferCreateMem (in /opt/conda/envs/arrow/lib/libxml2.so.2.12.6)
==16391==    by 0x7F12AF8: xmlReaderForMemory (in /opt/conda/envs/arrow/lib/libxml2.so.2.12.6)
==16391==    by 0x6BB3363: Azure::Storage::_internal::XmlReader::XmlReader(char const*, unsigned long) (in /opt/conda/envs/arrow/lib/libazure-storage-common.so)
==16391==    by 0x4B8B690: Azure::Storage::Blobs::_detail::BlobContainerClient::ListBlobsByHierarchy(Azure::Core::Http::_internal::HttpPipeline&, Azure::Core::Url const&, Azure::Storage::Blobs::_detail::BlobContainerClient::ListBlobContainerBlobsByHierarchyOptions const&, Azure::Core::Context const&) (in /opt/conda/envs/arrow/lib/libazure-storage-blobs.so)
==16391==    by 0x4AD5185: Azure::Storage::Blobs::BlobContainerClient::ListBlobsByHierarchy(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Azure::Storage::Blobs::ListBlobsOptions const&, Azure::Core::Context const&) const (in /opt/conda/envs/arrow/lib/libazure-storage-blobs.so)
==16391==    by 0x5CE0878: arrow::fs::AzureFileSystem::Impl::GetFileInfoWithSelectorFromContainer(Azure::Storage::Blobs::BlobContainerClient const&, Azure::Core::Context const&, Azure::Nullable<int>, arrow::fs::FileSelector const&, std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> >*) (azurefs.cc:1680)
==16391==    by 0x5CE0E6A: arrow::fs::AzureFileSystem::Impl::GetFileInfoWithSelector(Azure::Core::Context const&, Azure::Nullable<int>, arrow::fs::FileSelector const&, std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> >*) (azurefs.cc:1778)
==16391==    by 0x5CE0F73: arrow::fs::AzureFileSystem::GetFileInfo(arrow::fs::FileSelector const&) (azurefs.cc:2863)
==16391==    by 0x5C8C2B0: operator() (filesystem.cc:182)
==16391==    by 0x5C8C2B0: std::enable_if<((!std::is_void<arrow::Result<std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> > > >::value)&&(!arrow::detail::is_future<arrow::Result<std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> > > >::value))&&((!arrow::Future<std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> > >::is_empty)||std::is_same<arrow::Result<std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> > >, arrow::Status>::value), void>::type arrow::detail::ContinueFuture::operator()<arrow::fs::FileSystem::GetFileInfoGenerator(arrow::fs::FileSelector const&)::{lambda(std::shared_ptr<arrow::fs::FileSystem>)#1}&, std::shared_ptr<arrow::fs::FileSystem>&, arrow::Result<std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> > >, arrow::Future<std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> > > >(arrow::Future<std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> > >, arrow::fs::FileSystem::GetFileInfoGenerator(arrow::fs::FileSelector const&)::{lambda(std::shared_ptr<arrow::fs::FileSystem>)#1}&, std::shared_ptr<arrow::fs::FileSystem>&) const (future.h:150)
==16391==    by 0x5C8C369: void std::__invoke_impl<void, arrow::detail::ContinueFuture&, arrow::Future<std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> > >&, arrow::fs::FileSystem::GetFileInfoGenerator(arrow::fs::FileSelector const&)::{lambda(std::shared_ptr<arrow::fs::FileSystem>)#1}&, std::shared_ptr<arrow::fs::FileSystem>&>(std::__invoke_other, arrow::detail::ContinueFuture&, arrow::Future<std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> > >&, arrow::fs::FileSystem::GetFileInfoGenerator(arrow::fs::FileSelector const&)::{lambda(std::shared_ptr<arrow::fs::FileSystem>)#1}&, std::shared_ptr<arrow::fs::FileSystem>&) (invoke.h:61)
==16391==    by 0x5C8C3C8: __invoke<arrow::detail::ContinueFuture&, arrow::Future<std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> > >&, arrow::fs::FileSystem::GetFileInfoGenerator(const arrow::fs::FileSelector&)::<lambda(std::shared_ptr<arrow::fs::FileSystem>)>&, std::shared_ptr<arrow::fs::FileSystem>&> (invoke.h:96)
==16391==    by 0x5C8C3C8: __call<void, 0, 1, 2> (functional:495)
==16391==    by 0x5C8C3C8: void std::_Bind<arrow::detail::ContinueFuture (arrow::Future<std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> > >, arrow::fs::FileSystem::GetFileInfoGenerator(arrow::fs::FileSelector const&)::{lambda(std::shared_ptr<arrow::fs::FileSystem>)#1}, std::shared_ptr<arrow::fs::FileSystem>)>::operator()<, void>() (functional:580)
==16391==    by 0x5C8C3DA: arrow::internal::FnOnce<void ()>::FnImpl<std::_Bind<arrow::detail::ContinueFuture (arrow::Future<std::vector<arrow::fs::FileInfo, std::allocator<arrow::fs::FileInfo> > >, arrow::fs::FileSystem::GetFileInfoGenerator(arrow::fs::FileSelector const&)::{lambda(std::shared_ptr<arrow::fs::FileSystem>)#1}, std::shared_ptr<arrow::fs::FileSystem>)> >::invoke() (functional.h:152)
==16391==    by 0x5ECA64B: arrow::internal::FnOnce<void ()>::operator()() && (functional.h:140)
==16391==    by 0x5ECCEC0: arrow::internal::WorkerLoop(std::shared_ptr<arrow::internal::ThreadPool::State>, std::_List_iterator<std::thread>) (thread_pool.cc:457)
==16391==    by 0x5ECD199: arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::{lambda()#1}::operator()() const (thread_pool.cc:618)
==16391==    by 0x5ECD1EE: __invoke_impl<void, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::<lambda()> > (invoke.h:61)
==16391==    by 0x5ECD1EE: __invoke<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::<lambda()> > (invoke.h:96)
==16391==    by 0x5ECD1EE: _M_invoke<0> (std_thread.h:279)
==16391==    by 0x5ECD1EE: operator() (std_thread.h:286)
==16391==    by 0x5ECD1EE: std::thread::_State_impl<std::thread::_Invoker<std::tuple<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::{lambda()#1}> > >::_M_run() (std_thread.h:231)
==16391==    by 0x6731E94: execute_native_thread_routine (thread.cc:104)
==16391== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==16391== 
==16391== For lists of detected and suppressed errors, rerun with: -s
==16391== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Component(s)

C++, Continuous Integration

[raulcd]

cc @felipecrv

[kou]

The Valgrind may be unrelated. Invalid response from Azurite may be related instead:

https://dev.azure.com/ursacomputing/crossbow/_build/results?buildId=62773&view=logs&j=50a69d0a-7972-5459-cdae-135ee6ebe312&t=13df7b5c-76db-5c26-6592-75581a9ed64a&l=8681

[ RUN      ] TestAzureFileSystemOnAllEnvs/0.CreateDirOnRoot
unknown file: Failure
C++ exception with description "Connection closed before getting full response or response is less than expected. Expected response length = 254. Read until now = 236" thrown in the test body.

BTW, the Valgrind report may be related to https://github.com/apache/arrow/pull/40567/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R398-R412 .
GetFileInfoGenerator() may report the report.

[felipecrv]

@kou but how to proceed here, then? Should we add a try-catch in the code to guard against azurite behavior or disable the test case?

[kou]

How about trying this on local as the first step?
I think that we need to know why full response isn't returned.

[pitrou]

This is probably not a regression, so not a blocker for 16.0.0?

[kou]

Issue resolved by pull request 41163
#41163

[kou]

@raulcd You can include this to 16.0.0 or move this to 17.0.0.