Skip to content
This repository has been archived by the owner on Dec 8, 2021. It is now read-only.

[Flake] Windows build failed with segfault #1034

Closed
tmatsuo opened this issue Nov 8, 2019 · 19 comments
Closed

[Flake] Windows build failed with segfault #1034

tmatsuo opened this issue Nov 8, 2019 · 19 comments
Labels
api: spanner Issues related to the googleapis/google-cloud-cpp-spanner API. priority: p2 Lowest priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@tmatsuo
Copy link
Contributor

tmatsuo commented Nov 8, 2019

https://source.cloud.google.com/results/invocations/ec1ef760-9a66-4f4d-825e-96cbbcefb146/targets

The sample encountered segfault.
@tmatsuo tmatsuo added priority: p2 Lowest priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Nov 8, 2019
@tmatsuo
Copy link
Contributor Author

tmatsuo commented Nov 8, 2019

Oh I see segfault two times in a row
https://source.cloud.google.com/results/invocations/f4ccf4ea-0dfb-4c4d-8316-cfe7778d2fa4/targets

@tmatsuo
Copy link
Contributor Author

tmatsuo commented Nov 8, 2019

Three times in a row:
https://source.cloud.google.com/results/invocations/66705657-f488-4e3f-8bcd-d97779e43b9d/targets

@devbww
Copy link
Contributor

devbww commented Dec 5, 2019

Something similar is currently a problem.

 9/11 Test #57: multiple_rows_cpu_benchmark ..............***Exception: SegFault 91.94 sec

https://source.cloud.google.com/results/invocations/b11cc291-acdc-4006-a768-4b802714261c/targets/cloud-devrel%2Fclient-libraries%2Fcpp%2Fgoogle-cloud-cpp-spanner%2Fwindows%2Fcmake/log

@devjgm devjgm added priority: p1 Medium priority. Will be fixed prior to next release. and removed priority: p2 Lowest priority. Fix may not be included in next release. labels Dec 13, 2019
@devjgm
Copy link
Contributor

devjgm commented Dec 13, 2019

This happens a lot. Seems like every PR.

Note: That once we have a dep on Abseil, there are some flags that we can set that will enable stack traces:

To quote from @derekmauro

I added stacktrace support to the opensource version of GoogleTest when it is build with Abseil. We (Abseil) are using it ourselves in our tests on Linux, but don't have it enabled on Windows because of a Bazel bug (bazelbuild/bazel#5640). This bug shouldn't affect you, but it does mean these directions are untested on Windows.

With Bazel + GoogleTest, all you should need to do is add '--define="absl=1" --test_env="GTEST_INSTALL_FAILURE_SIGNAL_HANDLER=1"' to the "bazel test" command.

If you are using CMake (or some other more traditional build system), it looks like the '--define="absl=1"' part turns causes '-DGTEST_HAS_ABSL=1' to be passed to the compiler. If you run the tests with GTEST_INSTALL_FAILURE_SIGNAL_HANDLER=1 set in the environment, I think this will get you everything you need.

As far as I know, MSVC binaries require pdb files for debug info as well. You may need to research how to enable these if they aren't enabled already. Bazel seems to build them when you pass "-c dbg" to the build command.

@mr-salty
Copy link
Contributor

6 tries (at 20 minutes a pop, plus the extra delay of having to notice and push the button) to get this to pass... regardless of not getting stack traces, seems like we should temporarily disable multiple_rows_cpu_benchmark

https://fusion.corp.google.com/projectanalysis/summary/KOKORO/prod:cloud-devrel%2Fclient-libraries%2Fcpp%2Fgoogle-cloud-cpp-spanner%2Fwindows%2Fcmake-presubmit

@coryan
Copy link
Contributor

coryan commented Dec 16, 2019

For what it is worth, I am running the integration tests in a Windows VM. It is not just multiple_rows_cpu_benchmark that fails, I get failures in many other tests (see below). And they are all related to the CreateFile() call to determine if the machine is a GCE instance. You may recall that I submitted fixes for that about a month ago, the 1.26.0 (pre-)release has those fixes.

==================== Test output for //google/cloud/spanner/integration_tests:spanner_client_rpc_failure_threshold_integration_test (run 9 of 10):
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RpcFailureThresholdTest
[ RUN      ] RpcFailureThresholdTest.ExecuteDmlDeleteErrors
E1216 22:06:58.838000000  5932 external/com_github_grpc_grpc/src/core/lib/security/credentials/alts/check_gcp_environment_windows.cc:72] CreateFile failed (3).
Creating database [db-0icx6hf1wx2dg94lm9capvs43qg] and table ................... DONE
Running test ..........E1216 22:07:20.406000000  6472 external/com_github_grpc_grpc/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_ev_driver_windows.cc:382] assertion failed: GRPC_SLICE_LENGTH(write_buf_) == 0
================================================================================
INFO: From Testing //google/cloud/spanner/integration_tests:spanner_client_rpc_failure_threshold_integration_test (run 5 of 10):
==================== Test output for //google/cloud/spanner/integration_tests:spanner_client_rpc_failure_threshold_integration_test (run 5 of 10):
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RpcFailureThresholdTest
[ RUN      ] RpcFailureThresholdTest.ExecuteDmlDeleteErrors
E1216 22:08:14.249000000  3188 external/com_github_grpc_grpc/src/core/lib/security/credentials/alts/check_gcp_environment_windows.cc:72] CreateFile failed (3).
Creating database [db-hps6hpu4savovp9our_rfn4649s] and table unknown file: error: SEH exception with code 0xc0000005 thrown in SetUp().
Dropping database db-hps6hpu4savovp9our_rfn4649s================================================================================

@mr-salty
Copy link
Contributor

That's a different issue though (#923), and seems to happen much less frequently. The ones I (and others) have been seeing lhere ook like:

9/11 Test #59: multiple_rows_cpu_benchmark ..............***Exception: SegFault 65.78 sec

this does happen in other tests (takashi's original report is 9/9 Test #56: samples ..................................***Exception: SegFault107.22 sec) but lately it's mostly been the multiple_rows_cpu_benchmark from what I've seen (I looked at other runs and not just my own).

in this case, there's no sign of the SEH exception with code 0xc0000005 or similar errors from the other bug. of course, it's possible it's happening and just not getting output in this case, but we just don't get any other debug info.

@coryan
Copy link
Contributor

coryan commented Dec 18, 2019

@mr-salty You are right of course sorry, I was confused.

I ran that program under a debugger (stack trace at the bottom). The last call in our code is:

google-cloud-cpp-spanner/google/cloud/spanner/internal/spanner_stub.cc

Line 109 in c872116

grpc::Status grpc_status =

started from this code in the benchmark:

google-cloud-cpp-spanner/google/cloud/spanner/benchmarks/multiple_rows_cpu_benchmark.cc

Line 705 in c872116

auto rows = client.Read(table_name_, key, column_names);

I will run it a couple more times to see if it always crashes in the same spot.

multiple_rows_cpu_benchmark.exe!grpc_core::GrpcPolledFdWindows::gotten_into_driver_list() Line 671 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver_windows.cc:671)
multiple_rows_cpu_benchmark.exe!grpc_core::SockToPolledFdMap::CloseSocket(unsigned __int64 s, void * user_data) Line 821 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver_windows.cc:821)
multiple_rows_cpu_benchmark.exe!ares__socket_close(ares_channeldata * channel, unsigned __int64 s) Line 1470 (c:\Users\coryan\vcpkg\buildtrees\c-ares\src\c4d45be459-24e25a63ee\ares_process.c:1470)
multiple_rows_cpu_benchmark.exe!ares__close_sockets(ares_channeldata * channel, server_state * server) Line 59 (c:\Users\coryan\vcpkg\buildtrees\c-ares\src\c4d45be459-24e25a63ee\ares__close_sockets.c:59)
multiple_rows_cpu_benchmark.exe!ares__destroy_servers_state(ares_channeldata * channel) Line 107 (c:\Users\coryan\vcpkg\buildtrees\c-ares\src\c4d45be459-24e25a63ee\ares_destroy.c:107)
multiple_rows_cpu_benchmark.exe!ares_destroy(ares_channeldata * channel) Line 78 (c:\Users\coryan\vcpkg\buildtrees\c-ares\src\c4d45be459-24e25a63ee\ares_destroy.c:78)
multiple_rows_cpu_benchmark.exe!grpc_ares_ev_driver_unref(grpc_ares_ev_driver * ev_driver) Line 112 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver.cc:112)
multiple_rows_cpu_benchmark.exe!on_ares_backup_poll_alarm_locked(void * arg, grpc_error * error) Line 289 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver.cc:289)
multiple_rows_cpu_benchmark.exe!grpc_combiner_continue_exec_ctx() Line 269 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\iomgr\combiner.cc:269)
multiple_rows_cpu_benchmark.exe!grpc_core::ExecCtx::Flush() Line 151 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\iomgr\exec_ctx.cc:151)
multiple_rows_cpu_benchmark.exe!pollset_work(grpc_pollset * pollset, grpc_pollset_worker * * worker_hdl, __int64 deadline) Line 132 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\iomgr\pollset_windows.cc:132)
multiple_rows_cpu_benchmark.exe!grpc_pollset_work(grpc_pollset * pollset, grpc_pollset_worker * * worker, __int64 deadline) Line 49 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\iomgr\pollset.cc:49)
multiple_rows_cpu_benchmark.exe!cq_pluck(grpc_completion_queue * cq, void * tag, gpr_timespec deadline, void * reserved) Line 1284 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\surface\completion_queue.cc:1284)
multiple_rows_cpu_benchmark.exe!grpc_completion_queue_pluck(grpc_completion_queue * cq, void * tag, gpr_timespec deadline, void * reserved) Line 1312 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\surface\completion_queue.cc:1312)
multiple_rows_cpu_benchmark.exe!grpc::CoreCodegen::grpc_completion_queue_pluck(grpc_completion_queue * cq, void * tag, gpr_timespec deadline, void * reserved) Line 74 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\cpp\common\core_codegen.cc:74)
multiple_rows_cpu_benchmark.exe!grpc_impl::CompletionQueue::Pluck(grpc::internal::CompletionQueueTag * tag) Line 316 (c:\Users\coryan\vcpkg\installed\x64-windows-static\include\grpcpp\impl\codegen\completion_queue_impl.h:316)
multiple_rows_cpu_benchmark.exe!grpc::internal::BlockingUnaryCallImpl<google::spanner::v1::BatchCreateSessionsRequest,google::spanner::v1::BatchCreateSessionsResponse>::BlockingUnaryCallImpl<google::spanner::v1::BatchCreateSessionsRequest,google::spanner::v1::BatchCreateSessionsResponse>(grpc::ChannelInterface * channel, const grpc::internal::RpcMethod & method, grpc_impl::ClientContext * context, const google::spanner::v1::BatchCreateSessionsRequest & request, google::spanner::v1::BatchCreateSessionsResponse * result) Line 80 (c:\Users\coryan\vcpkg\installed\x64-windows-static\include\grpcpp\impl\codegen\client_unary_call.h:80)
multiple_rows_cpu_benchmark.exe!grpc::internal::BlockingUnaryCall<google::spanner::v1::BatchCreateSessionsRequest,google::spanner::v1::BatchCreateSessionsResponse>(grpc::ChannelInterface * channel, const grpc::internal::RpcMethod & method, grpc_impl::ClientContext * context, const google::spanner::v1::BatchCreateSessionsRequest & request, google::spanner::v1::BatchCreateSessionsResponse * result) Line 41 (c:\Users\coryan\vcpkg\installed\x64-windows-static\include\grpcpp\impl\codegen\client_unary_call.h:41)
multiple_rows_cpu_benchmark.exe!google::spanner::v1::Spanner::Stub::BatchCreateSessions(grpc_impl::ClientContext * context, const google::spanner::v1::BatchCreateSessionsRequest & request, google::spanner::v1::BatchCreateSessionsResponse * response) Line 94 (c:\Users\coryan\vcpkg\buildtrees\googleapis\x64-windows-static-dbg\google\spanner\v1\spanner.grpc.pb.cc:94)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::`anonymous namespace'::DefaultSpannerStub::BatchCreateSessions(grpc_impl::ClientContext & client_context, const google::spanner::v1::BatchCreateSessionsRequest & request) Line 109 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\spanner_stub.cc:109)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::MetadataSpannerStub::BatchCreateSessions(grpc_impl::ClientContext & client_context, const google::spanner::v1::BatchCreateSessionsRequest & request) Line 43 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\metadata_spanner_stub.cc:43)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::SessionPool::CreateSessions::__l2::<lambda>(grpc_impl::ClientContext & context, const google::spanner::v1::BatchCreateSessionsRequest & request) Line 198 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\session_pool.cc:198)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::RetryLoopImpl<google::cloud::v0::StatusOr<google::spanner::v1::BatchCreateSessionsResponse> <lambda>(grpc_impl::ClientContext &, const google::spanner::v1::BatchCreateSessionsRequest &),google::spanner::v1::BatchCreateSessionsRequest,void <lambda>(std::chrono::duration<__int64,std::ratio<1,1000> >),0>(std::unique_ptr<google::cloud::v0::internal::RetryPolicy<google::cloud::v0::Status,google::cloud::spanner::v0::internal::SafeGrpcRetry>,std::default_delete<google::cloud::v0::internal::RetryPolicy<google::cloud::v0::Status,google::cloud::spanner::v0::internal::SafeGrpcRetry> > > retry_policy, std::unique_ptr<google::cloud::v0::internal::BackoffPolicy,std::default_delete<google::cloud::v0::internal::BackoffPolicy> > backoff_policy, bool is_idempotent, google::cloud::spanner::v0::internal::SessionPool::CreateSessions::__l2::google::cloud::v0::StatusOr<google::spanner::v1::BatchCreateSessionsResponse> <lambda>(grpc_impl::ClientContext &, const google::spanner::v1::BatchCreateSessionsRequest &) && functor, const google::spanner::v1::BatchCreateSessionsRequest & request, const char * location, google::cloud::spanner::v0::internal::RetryLoop::__l2::void <lambda>(std::chrono::duration<__int64,std::ratio<1,1000> >) sleeper) Line 83 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\retry_loop.h:83)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::RetryLoop<google::cloud::v0::StatusOr<google::spanner::v1::BatchCreateSessionsResponse> <lambda>(grpc_impl::ClientContext &, const google::spanner::v1::BatchCreateSessionsRequest &),google::spanner::v1::BatchCreateSessionsRequest,0>(std::unique_ptr<google::cloud::v0::internal::RetryPolicy<google::cloud::v0::Status,google::cloud::spanner::v0::internal::SafeGrpcRetry>,std::default_delete<google::cloud::v0::internal::RetryPolicy<google::cloud::v0::Status,google::cloud::spanner::v0::internal::SafeGrpcRetry> > > retry_policy, std::unique_ptr<google::cloud::v0::internal::BackoffPolicy,std::default_delete<google::cloud::v0::internal::BackoffPolicy> > backoff_policy, bool is_idempotent, google::cloud::spanner::v0::internal::SessionPool::CreateSessions::__l2::google::cloud::v0::StatusOr<google::spanner::v1::BatchCreateSessionsResponse> <lambda>(grpc_impl::ClientContext &, const google::spanner::v1::BatchCreateSessionsRequest &) && functor, const google::spanner::v1::BatchCreateSessionsRequest & request, const char * location) Line 120 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\retry_loop.h:120)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::SessionPool::CreateSessions(std::unique_lock<std::mutex> & lk, google::cloud::spanner::v0::internal::SessionPool::ChannelInfo & channel, const std::map<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,std::basic_string<char,std::char_traits<char>,std::allocator<char> > > > > & labels, int num_sessions) Line 193 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\session_pool.cc:193)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::SessionPool::Allocate(bool dissociate_from_pool) Line 137 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\session_pool.cc:137)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::ConnectionImpl::PrepareSession(std::shared_ptr<google::cloud::spanner::v0::internal::Session> & session, bool dissociate_from_pool) Line 287 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\connection_impl.cc:287)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::ConnectionImpl::ReadImpl(std::shared_ptr<google::cloud::spanner::v0::internal::Session> & session, google::spanner::v1::TransactionSelector & s, google::cloud::spanner::v0::Connection::ReadParams params) Line 299 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\connection_impl.cc:299)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::ConnectionImpl::Read::__l2::<lambda>(std::shared_ptr<google::cloud::spanner::v0::internal::Session> & session, google::spanner::v1::TransactionSelector & s, __int64 __formal) Line 108 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\connection_impl.cc:108)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::TransactionImpl::Visit<google::cloud::spanner::v0::RowStream <lambda>(std::shared_ptr<google::cloud::spanner::v0::internal::Session> &, google::spanner::v1::TransactionSelector &, __int64) >(google::cloud::spanner::v0::internal::ConnectionImpl::Read::__l2::google::cloud::spanner::v0::RowStream <lambda>(std::shared_ptr<google::cloud::spanner::v0::internal::Session> &, google::spanner::v1::TransactionSelector &, __int64) && f) Line 86 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\transaction_impl.h:86)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::Visit<google::cloud::spanner::v0::RowStream <lambda>(std::shared_ptr<google::cloud::spanner::v0::internal::Session> &, google::spanner::v1::TransactionSelector &, __int64) >(google::cloud::spanner::v0::Transaction txn, google::cloud::spanner::v0::internal::ConnectionImpl::Read::__l2::google::cloud::spanner::v0::RowStream <lambda>(std::shared_ptr<google::cloud::spanner::v0::internal::Session> &, google::spanner::v1::TransactionSelector &, __int64) && f) Line 222 (c:\Users\coryan\span-develop\google\cloud\spanner\transaction.h:222)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::ConnectionImpl::Read(google::cloud::spanner::v0::Connection::ReadParams params) Line 104 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\connection_impl.cc:104)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::Client::Read(std::basic_string<char,std::char_traits<char>,std::allocator<char> > table, google::cloud::spanner::v0::KeySet keys, std::vector<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::allocator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > > columns, google::cloud::spanner::v0::ReadOptions read_options) Line 35 (c:\Users\coryan\span-develop\google\cloud\spanner\client.cc:35)
multiple_rows_cpu_benchmark.exe!`anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::ReadRowsViaClients(const google::cloud::spanner_benchmarks::v0::Config & config, int thread_count, int client_count, google::cloud::spanner::v0::Client client) Line 705 (c:\Users\coryan\span-develop\google\cloud\spanner\benchmarks\multiple_rows_cpu_benchmark.cc:705)
multiple_rows_cpu_benchmark.exe!std::_Invoker_pmf_pointer::_Call<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Client>(std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> >(`anonymous-namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(const google::cloud::spanner_benchmarks::v0::Config &, int, int, google::cloud::spanner::v0::Client) _Pmf, `anonymous-namespace'::ReadExperiment<`anonymous namespace'::StringTraits> * && _Arg1, google::cloud::spanner_benchmarks::v0::Config && <_Args2_0>, int && <_Args2_1>, int && <_Args2_2>, google::cloud::spanner::v0::Client && <_Args2_3>) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::invoke<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Client>(std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> >(`anonymous-namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(const google::cloud::spanner_benchmarks::v0::Config &, int, int, google::cloud::spanner::v0::Client) && _Obj, `anonymous-namespace'::ReadExperiment<`anonymous namespace'::StringTraits> * && <_Args_0>, google::cloud::spanner_benchmarks::v0::Config && <_Args_1>, int && <_Args_2>, int && <_Args_3>, google::cloud::spanner::v0::Client && <_Args_4>) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::_Invoke_stored_explicit<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Client,0,1,2,3,4,5>(std::tuple<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Client> && _Tuple, std::integer_sequence<unsigned __int64,0,1,2,3,4,5> __formal) Line 520 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\mutex:520)
multiple_rows_cpu_benchmark.exe!std::_Invoke_stored<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Client>(std::tuple<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Client> && _Tuple) Line 527 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\mutex:527)
multiple_rows_cpu_benchmark.exe!std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Client &>::operator()() Line 1561 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\future:1561)
multiple_rows_cpu_benchmark.exe!std::_Invoker_functor::_Call<std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Client &> &>(std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Client &> & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::invoke<std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Client &> &>(std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Client &> & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::_Invoker_ret<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> >,0>::_Call<std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Client &> &>(std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Client &> & <_Vals_0>) Line 1597 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1597)
multiple_rows_cpu_benchmark.exe!std::_Func_impl_no_alloc<std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::ReadExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Client),A0x776927bd::ReadExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Client &>,std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::_Do_call() Line 964 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:964)
multiple_rows_cpu_benchmark.exe!std::_Func_class<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::operator()() Line 1007 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:1007)
multiple_rows_cpu_benchmark.exe!std::_Packaged_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > __cdecl(void)>::_Call_immediate() Line 509 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\future:509)
multiple_rows_cpu_benchmark.exe!std::_Task_async_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::{ctor}::__l2::<lambda>() Line 714 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\future:714)
multiple_rows_cpu_benchmark.exe!std::_Invoker_functor::_Call<void <lambda>(void) &>(std::_Task_async_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::{ctor}::__l2::void <lambda>(void) & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::invoke<void <lambda>(void) &>(std::_Task_async_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::{ctor}::__l2::void <lambda>(void) & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::_Invoker_ret<void,1>::_Call<void <lambda>(void) &>(std::_Task_async_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::{ctor}::__l2::void <lambda>(void) & <_Vals_0>) Line 1590 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1590)
multiple_rows_cpu_benchmark.exe!std::_Func_impl_no_alloc<void <lambda>(void),void>::_Do_call() Line 965 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:965)
multiple_rows_cpu_benchmark.exe!std::_Func_class<void>::operator()() Line 1008 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:1008)
multiple_rows_cpu_benchmark.exe!Concurrency::details::_MakeVoidToUnitFunc::__l2::<lambda>() Line 2348 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\ppltasks.h:2348)
multiple_rows_cpu_benchmark.exe!std::_Invoker_functor::_Call<unsigned char <lambda>(void) &>(Concurrency::details::_MakeVoidToUnitFunc::__l2::unsigned char <lambda>(void) & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::invoke<unsigned char <lambda>(void) &>(Concurrency::details::_MakeVoidToUnitFunc::__l2::unsigned char <lambda>(void) & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::_Invoker_ret<unsigned char,0>::_Call<unsigned char <lambda>(void) &>(Concurrency::details::_MakeVoidToUnitFunc::__l2::unsigned char <lambda>(void) & <_Vals_0>) Line 1598 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1598)
multiple_rows_cpu_benchmark.exe!std::_Func_impl_no_alloc<unsigned char <lambda>(void),unsigned char>::_Do_call() Line 965 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:965)
multiple_rows_cpu_benchmark.exe!std::_Func_class<unsigned char>::operator()() Line 1008 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:1008)
multiple_rows_cpu_benchmark.exe!Concurrency::task<unsigned char>::_InitialTaskHandle<void,void <lambda>(void),Concurrency::details::_TypeSelectorNoAsync>::_LogWorkItemAndInvokeUserLambda<std::function<unsigned char __cdecl(void)> >(std::function<unsigned char __cdecl(void)> _func) Line 3513 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\ppltasks.h:3513)
multiple_rows_cpu_benchmark.exe!Concurrency::task<unsigned char>::_InitialTaskHandle<void,void <lambda>(void),Concurrency::details::_TypeSelectorNoAsync>::_Init(Concurrency::details::_TypeSelectorNoAsync __formal) Line 3533 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\ppltasks.h:3533)
multiple_rows_cpu_benchmark.exe!Concurrency::task<unsigned char>::_InitialTaskHandle<void,void <lambda>(void),Concurrency::details::_TypeSelectorNoAsync>::_Perform() Line 3519 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\ppltasks.h:3519)

@coryan coryan self-assigned this Dec 18, 2019
@coryan
Copy link
Contributor

coryan commented Dec 18, 2019

Second run, crashes in the same point, but very different stack trace:

multiple_rows_cpu_benchmark.exe!grpc_core::GrpcPolledFdWindows::gotten_into_driver_list() Line 671 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver_windows.cc:671)
multiple_rows_cpu_benchmark.exe!grpc_core::SockToPolledFdMap::CloseSocket(unsigned __int64 s, void * user_data) Line 821 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver_windows.cc:821)
multiple_rows_cpu_benchmark.exe!ares__socket_close(ares_channeldata * channel, unsigned __int64 s) Line 1470 (c:\Users\coryan\vcpkg\buildtrees\c-ares\src\c4d45be459-24e25a63ee\ares_process.c:1470)
multiple_rows_cpu_benchmark.exe!ares__close_sockets(ares_channeldata * channel, server_state * server) Line 59 (c:\Users\coryan\vcpkg\buildtrees\c-ares\src\c4d45be459-24e25a63ee\ares__close_sockets.c:59)
multiple_rows_cpu_benchmark.exe!ares__destroy_servers_state(ares_channeldata * channel) Line 107 (c:\Users\coryan\vcpkg\buildtrees\c-ares\src\c4d45be459-24e25a63ee\ares_destroy.c:107)
multiple_rows_cpu_benchmark.exe!ares_destroy(ares_channeldata * channel) Line 78 (c:\Users\coryan\vcpkg\buildtrees\c-ares\src\c4d45be459-24e25a63ee\ares_destroy.c:78)
multiple_rows_cpu_benchmark.exe!grpc_ares_ev_driver_unref(grpc_ares_ev_driver * ev_driver) Line 112 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver.cc:112)
multiple_rows_cpu_benchmark.exe!on_ares_backup_poll_alarm_locked(void * arg, grpc_error * error) Line 289 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver.cc:289)
multiple_rows_cpu_benchmark.exe!grpc_combiner_continue_exec_ctx() Line 269 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\iomgr\combiner.cc:269)
multiple_rows_cpu_benchmark.exe!grpc_core::ExecCtx::Flush() Line 151 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\iomgr\exec_ctx.cc:151)
multiple_rows_cpu_benchmark.exe!pollset_work(grpc_pollset * pollset, grpc_pollset_worker * * worker_hdl, __int64 deadline) Line 132 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\iomgr\pollset_windows.cc:132)
multiple_rows_cpu_benchmark.exe!grpc_pollset_work(grpc_pollset * pollset, grpc_pollset_worker * * worker, __int64 deadline) Line 49 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\iomgr\pollset.cc:49)
multiple_rows_cpu_benchmark.exe!cq_pluck(grpc_completion_queue * cq, void * tag, gpr_timespec deadline, void * reserved) Line 1284 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\surface\completion_queue.cc:1284)
multiple_rows_cpu_benchmark.exe!grpc_completion_queue_pluck(grpc_completion_queue * cq, void * tag, gpr_timespec deadline, void * reserved) Line 1312 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\surface\completion_queue.cc:1312)
multiple_rows_cpu_benchmark.exe!grpc::CoreCodegen::grpc_completion_queue_pluck(grpc_completion_queue * cq, void * tag, gpr_timespec deadline, void * reserved) Line 74 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\cpp\common\core_codegen.cc:74)
multiple_rows_cpu_benchmark.exe!grpc_impl::CompletionQueue::Pluck(grpc::internal::CompletionQueueTag * tag) Line 316 (c:\Users\coryan\vcpkg\installed\x64-windows-static\include\grpcpp\impl\codegen\completion_queue_impl.h:316)
multiple_rows_cpu_benchmark.exe!grpc::internal::BlockingUnaryCallImpl<google::spanner::v1::CreateSessionRequest,google::spanner::v1::Session>::BlockingUnaryCallImpl<google::spanner::v1::CreateSessionRequest,google::spanner::v1::Session>(grpc::ChannelInterface * channel, const grpc::internal::RpcMethod & method, grpc_impl::ClientContext * context, const google::spanner::v1::CreateSessionRequest & request, google::spanner::v1::Session * result) Line 80 (c:\Users\coryan\vcpkg\installed\x64-windows-static\include\grpcpp\impl\codegen\client_unary_call.h:80)
multiple_rows_cpu_benchmark.exe!grpc::internal::BlockingUnaryCall<google::spanner::v1::CreateSessionRequest,google::spanner::v1::Session>(grpc::ChannelInterface * channel, const grpc::internal::RpcMethod & method, grpc_impl::ClientContext * context, const google::spanner::v1::CreateSessionRequest & request, google::spanner::v1::Session * result) Line 41 (c:\Users\coryan\vcpkg\installed\x64-windows-static\include\grpcpp\impl\codegen\client_unary_call.h:41)
multiple_rows_cpu_benchmark.exe!google::spanner::v1::Spanner::Stub::CreateSession(grpc_impl::ClientContext * context, const google::spanner::v1::CreateSessionRequest & request, google::spanner::v1::Session * response) Line 66 (c:\Users\coryan\vcpkg\buildtrees\googleapis\x64-windows-static-dbg\google\spanner\v1\spanner.grpc.pb.cc:66)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::`anonymous namespace'::DefaultSpannerStub::CreateSession(grpc_impl::ClientContext & client_context, const google::spanner::v1::CreateSessionRequest & request) Line 96 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\spanner_stub.cc:96)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::MetadataSpannerStub::CreateSession(grpc_impl::ClientContext & client_context, const google::spanner::v1::CreateSessionRequest & request) Line 35 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\metadata_spanner_stub.cc:35)
multiple_rows_cpu_benchmark.exe!`anonymous-namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::InsertRowsViaStub::__l2::<lambda>() Line 1315 (c:\Users\coryan\span-develop\google\cloud\spanner\benchmarks\multiple_rows_cpu_benchmark.cc:1315)
multiple_rows_cpu_benchmark.exe!`anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::InsertRowsViaStub(const google::cloud::spanner_benchmarks::v0::Config & config, int thread_count, int client_count, const google::cloud::spanner::v0::Database & database, const std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> & stub) Line 1309 (c:\Users\coryan\span-develop\google\cloud\spanner\benchmarks\multiple_rows_cpu_benchmark.cc:1309)
multiple_rows_cpu_benchmark.exe!std::_Invoker_pmf_pointer::_Call<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> >(std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> >(`anonymous-namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(const google::cloud::spanner_benchmarks::v0::Config &, int, int, const google::cloud::spanner::v0::Database &, const std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &) _Pmf, `anonymous-namespace'::MutationExperiment<`anonymous namespace'::BytesTraits> * && _Arg1, google::cloud::spanner_benchmarks::v0::Config && <_Args2_0>, int && <_Args2_1>, int && <_Args2_2>, google::cloud::spanner::v0::Database && <_Args2_3>, std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> && <_Args2_4>) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::invoke<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> >(std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> >(`anonymous-namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(const google::cloud::spanner_benchmarks::v0::Config &, int, int, const google::cloud::spanner::v0::Database &, const std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &) && _Obj, `anonymous-namespace'::MutationExperiment<`anonymous namespace'::BytesTraits> * && <_Args_0>, google::cloud::spanner_benchmarks::v0::Config && <_Args_1>, int && <_Args_2>, int && <_Args_3>, google::cloud::spanner::v0::Database && <_Args_4>, std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> && <_Args_5>) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::_Invoke_stored_explicit<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub>,0,1,2,3,4,5,6>(std::tuple<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> > && _Tuple, std::integer_sequence<unsigned __int64,0,1,2,3,4,5,6> __formal) Line 520 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\mutex:520)
multiple_rows_cpu_benchmark.exe!std::_Invoke_stored<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> >(std::tuple<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> > && _Tuple) Line 527 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\mutex:527)
multiple_rows_cpu_benchmark.exe!std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &>::operator()() Line 1561 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\future:1561)
multiple_rows_cpu_benchmark.exe!std::_Invoker_functor::_Call<std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &> &>(std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &> & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::invoke<std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &> &>(std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &> & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::_Invoker_ret<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> >,0>::_Call<std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &> &>(std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &> & <_Vals_0>) Line 1597 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1597)
multiple_rows_cpu_benchmark.exe!std::_Func_impl_no_alloc<std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::BytesTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::BytesTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &>,std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::_Do_call() Line 964 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:964)
multiple_rows_cpu_benchmark.exe!std::_Func_class<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::operator()() Line 1007 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:1007)
multiple_rows_cpu_benchmark.exe!std::_Packaged_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > __cdecl(void)>::_Call_immediate() Line 509 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\future:509)
multiple_rows_cpu_benchmark.exe!std::_Task_async_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::{ctor}::__l2::<lambda>() Line 714 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\future:714)
multiple_rows_cpu_benchmark.exe!std::_Invoker_functor::_Call<void <lambda>(void) &>(std::_Task_async_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::{ctor}::__l2::void <lambda>(void) & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::invoke<void <lambda>(void) &>(std::_Task_async_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::{ctor}::__l2::void <lambda>(void) & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::_Invoker_ret<void,1>::_Call<void <lambda>(void) &>(std::_Task_async_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::{ctor}::__l2::void <lambda>(void) & <_Vals_0>) Line 1590 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1590)
multiple_rows_cpu_benchmark.exe!std::_Func_impl_no_alloc<void <lambda>(void),void>::_Do_call() Line 965 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:965)
multiple_rows_cpu_benchmark.exe!std::_Func_class<void>::operator()() Line 1008 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:1008)

@coryan
Copy link
Contributor

coryan commented Dec 18, 2019

Third run, similar stack trace:

multiple_rows_cpu_benchmark.exe!grpc_core::GrpcPolledFdWindows::gotten_into_driver_list() Line 671 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver_windows.cc:671)
multiple_rows_cpu_benchmark.exe!grpc_core::SockToPolledFdMap::CloseSocket(unsigned __int64 s, void * user_data) Line 821 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver_windows.cc:821)
multiple_rows_cpu_benchmark.exe!ares__socket_close(ares_channeldata * channel, unsigned __int64 s) Line 1470 (c:\Users\coryan\vcpkg\buildtrees\c-ares\src\c4d45be459-24e25a63ee\ares_process.c:1470)
multiple_rows_cpu_benchmark.exe!ares__close_sockets(ares_channeldata * channel, server_state * server) Line 59 (c:\Users\coryan\vcpkg\buildtrees\c-ares\src\c4d45be459-24e25a63ee\ares__close_sockets.c:59)
multiple_rows_cpu_benchmark.exe!ares__destroy_servers_state(ares_channeldata * channel) Line 107 (c:\Users\coryan\vcpkg\buildtrees\c-ares\src\c4d45be459-24e25a63ee\ares_destroy.c:107)
multiple_rows_cpu_benchmark.exe!ares_destroy(ares_channeldata * channel) Line 78 (c:\Users\coryan\vcpkg\buildtrees\c-ares\src\c4d45be459-24e25a63ee\ares_destroy.c:78)
multiple_rows_cpu_benchmark.exe!grpc_ares_ev_driver_unref(grpc_ares_ev_driver * ev_driver) Line 112 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver.cc:112)
multiple_rows_cpu_benchmark.exe!on_ares_backup_poll_alarm_locked(void * arg, grpc_error * error) Line 289 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver.cc:289)
multiple_rows_cpu_benchmark.exe!grpc_combiner_continue_exec_ctx() Line 269 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\iomgr\combiner.cc:269)
multiple_rows_cpu_benchmark.exe!grpc_core::ExecCtx::Flush() Line 151 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\iomgr\exec_ctx.cc:151)
multiple_rows_cpu_benchmark.exe!pollset_work(grpc_pollset * pollset, grpc_pollset_worker * * worker_hdl, __int64 deadline) Line 132 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\iomgr\pollset_windows.cc:132)
multiple_rows_cpu_benchmark.exe!grpc_pollset_work(grpc_pollset * pollset, grpc_pollset_worker * * worker, __int64 deadline) Line 49 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\iomgr\pollset.cc:49)
multiple_rows_cpu_benchmark.exe!cq_pluck(grpc_completion_queue * cq, void * tag, gpr_timespec deadline, void * reserved) Line 1284 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\surface\completion_queue.cc:1284)
multiple_rows_cpu_benchmark.exe!grpc_completion_queue_pluck(grpc_completion_queue * cq, void * tag, gpr_timespec deadline, void * reserved) Line 1312 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\core\lib\surface\completion_queue.cc:1312)
multiple_rows_cpu_benchmark.exe!grpc::CoreCodegen::grpc_completion_queue_pluck(grpc_completion_queue * cq, void * tag, gpr_timespec deadline, void * reserved) Line 74 (c:\Users\coryan\vcpkg\buildtrees\grpc\src\v1.23.1-ebfd5c51df\src\cpp\common\core_codegen.cc:74)
multiple_rows_cpu_benchmark.exe!grpc_impl::CompletionQueue::Pluck(grpc::internal::CompletionQueueTag * tag) Line 316 (c:\Users\coryan\vcpkg\installed\x64-windows-static\include\grpcpp\impl\codegen\completion_queue_impl.h:316)
multiple_rows_cpu_benchmark.exe!grpc::internal::BlockingUnaryCallImpl<google::spanner::v1::CreateSessionRequest,google::spanner::v1::Session>::BlockingUnaryCallImpl<google::spanner::v1::CreateSessionRequest,google::spanner::v1::Session>(grpc::ChannelInterface * channel, const grpc::internal::RpcMethod & method, grpc_impl::ClientContext * context, const google::spanner::v1::CreateSessionRequest & request, google::spanner::v1::Session * result) Line 80 (c:\Users\coryan\vcpkg\installed\x64-windows-static\include\grpcpp\impl\codegen\client_unary_call.h:80)
multiple_rows_cpu_benchmark.exe!grpc::internal::BlockingUnaryCall<google::spanner::v1::CreateSessionRequest,google::spanner::v1::Session>(grpc::ChannelInterface * channel, const grpc::internal::RpcMethod & method, grpc_impl::ClientContext * context, const google::spanner::v1::CreateSessionRequest & request, google::spanner::v1::Session * result) Line 41 (c:\Users\coryan\vcpkg\installed\x64-windows-static\include\grpcpp\impl\codegen\client_unary_call.h:41)
multiple_rows_cpu_benchmark.exe!google::spanner::v1::Spanner::Stub::CreateSession(grpc_impl::ClientContext * context, const google::spanner::v1::CreateSessionRequest & request, google::spanner::v1::Session * response) Line 66 (c:\Users\coryan\vcpkg\buildtrees\googleapis\x64-windows-static-dbg\google\spanner\v1\spanner.grpc.pb.cc:66)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::`anonymous namespace'::DefaultSpannerStub::CreateSession(grpc_impl::ClientContext & client_context, const google::spanner::v1::CreateSessionRequest & request) Line 96 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\spanner_stub.cc:96)
multiple_rows_cpu_benchmark.exe!google::cloud::spanner::v0::internal::MetadataSpannerStub::CreateSession(grpc_impl::ClientContext & client_context, const google::spanner::v1::CreateSessionRequest & request) Line 35 (c:\Users\coryan\span-develop\google\cloud\spanner\internal\metadata_spanner_stub.cc:35)
multiple_rows_cpu_benchmark.exe!`anonymous-namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::InsertRowsViaStub::__l2::<lambda>() Line 1315 (c:\Users\coryan\span-develop\google\cloud\spanner\benchmarks\multiple_rows_cpu_benchmark.cc:1315)
multiple_rows_cpu_benchmark.exe!`anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::InsertRowsViaStub(const google::cloud::spanner_benchmarks::v0::Config & config, int thread_count, int client_count, const google::cloud::spanner::v0::Database & database, const std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> & stub) Line 1309 (c:\Users\coryan\span-develop\google\cloud\spanner\benchmarks\multiple_rows_cpu_benchmark.cc:1309)
multiple_rows_cpu_benchmark.exe!std::_Invoker_pmf_pointer::_Call<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> >(std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> >(`anonymous-namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(const google::cloud::spanner_benchmarks::v0::Config &, int, int, const google::cloud::spanner::v0::Database &, const std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &) _Pmf, `anonymous-namespace'::MutationExperiment<`anonymous namespace'::StringTraits> * && _Arg1, google::cloud::spanner_benchmarks::v0::Config && <_Args2_0>, int && <_Args2_1>, int && <_Args2_2>, google::cloud::spanner::v0::Database && <_Args2_3>, std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> && <_Args2_4>) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::invoke<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> >(std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> >(`anonymous-namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(const google::cloud::spanner_benchmarks::v0::Config &, int, int, const google::cloud::spanner::v0::Database &, const std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &) && _Obj, `anonymous-namespace'::MutationExperiment<`anonymous namespace'::StringTraits> * && <_Args_0>, google::cloud::spanner_benchmarks::v0::Config && <_Args_1>, int && <_Args_2>, int && <_Args_3>, google::cloud::spanner::v0::Database && <_Args_4>, std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> && <_Args_5>) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::_Invoke_stored_explicit<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub>,0,1,2,3,4,5,6>(std::tuple<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> > && _Tuple, std::integer_sequence<unsigned __int64,0,1,2,3,4,5,6> __formal) Line 520 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\mutex:520)
multiple_rows_cpu_benchmark.exe!std::_Invoke_stored<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> >(std::tuple<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config,int,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> > && _Tuple) Line 527 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\mutex:527)
multiple_rows_cpu_benchmark.exe!std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &>::operator()() Line 1561 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\future:1561)
multiple_rows_cpu_benchmark.exe!std::_Invoker_functor::_Call<std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &> &>(std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &> & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::invoke<std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &> &>(std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &> & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::_Invoker_ret<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> >,0>::_Call<std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &> &>(std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &> & <_Vals_0>) Line 1597 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1597)
multiple_rows_cpu_benchmark.exe!std::_Func_impl_no_alloc<std::_Fake_no_copy_callable_adapter<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > (__cdecl `anonymous namespace'::MutationExperiment<`anonymous namespace'::StringTraits>::*)(google::cloud::spanner_benchmarks::v0::Config const &,int,int,google::cloud::spanner::v0::Database const &,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> const &),A0x776927bd::MutationExperiment<`anonymous namespace'::StringTraits> *,google::cloud::spanner_benchmarks::v0::Config const &,int &,int,google::cloud::spanner::v0::Database,std::shared_ptr<google::cloud::spanner::v0::internal::SpannerStub> &>,std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::_Do_call() Line 964 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:964)
multiple_rows_cpu_benchmark.exe!std::_Func_class<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::operator()() Line 1007 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:1007)
multiple_rows_cpu_benchmark.exe!std::_Packaged_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > __cdecl(void)>::_Call_immediate() Line 509 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\future:509)
multiple_rows_cpu_benchmark.exe!std::_Task_async_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::{ctor}::__l2::<lambda>() Line 714 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\future:714)
multiple_rows_cpu_benchmark.exe!std::_Invoker_functor::_Call<void <lambda>(void) &>(std::_Task_async_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::{ctor}::__l2::void <lambda>(void) & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::invoke<void <lambda>(void) &>(std::_Task_async_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::{ctor}::__l2::void <lambda>(void) & _Obj) Line 1571 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1571)
multiple_rows_cpu_benchmark.exe!std::_Invoker_ret<void,1>::_Call<void <lambda>(void) &>(std::_Task_async_state<std::vector<`anonymous namespace'::RowCpuSample,std::allocator<`anonymous namespace'::RowCpuSample> > >::{ctor}::__l2::void <lambda>(void) & <_Vals_0>) Line 1590 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\type_traits:1590)
multiple_rows_cpu_benchmark.exe!std::_Func_impl_no_alloc<void <lambda>(void),void>::_Do_call() Line 965 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:965)
multiple_rows_cpu_benchmark.exe!std::_Func_class<void>::operator()() Line 1008 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include\functional:1008)

@coryan
Copy link
Contributor

coryan commented Dec 18, 2019

It might be related to grpc/grpc#18461, that is fixed in 1.25.x, and we are currently using 1.23.x in the CMake+Windows builds.

@coryan
Copy link
Contributor

coryan commented Dec 18, 2019

The workaround suggested in grpc/grpc#18461 (comment) seems to work for now.

@coryan coryan mentioned this issue Dec 18, 2019
Merged
@yoshi-automation yoshi-automation added the 🚨 This issue needs some love. label Dec 20, 2019
@coryan coryan added priority: p2 Lowest priority. Fix may not be included in next release. and removed 🚨 This issue needs some love. priority: p1 Medium priority. Will be fixed prior to next release. labels Dec 20, 2019
@coryan
Copy link
Contributor

coryan commented Dec 20, 2019

How that this is under control, and we know what the fix is, I think P2 is a more reasonable priority.

@coryan
Copy link
Contributor

coryan commented Jan 11, 2020

FWIW, the Windows CI builds are upgraded to gRPC-1.26.0. We may be able to disable the workaround in #1158

@coryan
Copy link
Contributor

coryan commented Jan 16, 2020

Actually, only the CMake builds for Windows are updated to gRPC-1.26.0. The upgrade for Bazel is in progress (see #1167).

@coryan
Copy link
Contributor

coryan commented Jan 16, 2020

The workaround may not be working:

https://source.cloud.google.com/results/invocations/f95cc664-fff5-4b0b-bf7d-a06c35a19900/targets/cloud-devrel%2Fclient-libraries%2Fcpp%2Fgoogle-cloud-cpp-spanner%2Fwindows%2Fbazel%2Fpresubmit/log

@google-cloud-label-sync google-cloud-label-sync bot added the api: spanner Issues related to the googleapis/google-cloud-cpp-spanner API. label Jan 29, 2020
@coryan
Copy link
Contributor

coryan commented Jan 31, 2020

https://source.cloud.google.com/results/invocations/105db660-9a8d-42b3-a020-aaa25e825fe2

@coryan
Copy link
Contributor

coryan commented Mar 9, 2020

I am not actively working on this, we need to wait to see if it repeats. Close by 2020-05-01 if it does not repeat.

@coryan coryan removed their assignment Mar 9, 2020
@yoshi-automation yoshi-automation added the 🚨 This issue needs some love. label May 6, 2020
@coryan
Copy link
Contributor

coryan commented May 7, 2020

Has not repeated for 3 months, closing.

@coryan coryan closed this as completed May 7, 2020
@coryan coryan removed the 🚨 This issue needs some love. label May 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
api: spanner Issues related to the googleapis/google-cloud-cpp-spanner API. priority: p2 Lowest priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@tmatsuo @coryan @mr-salty @devjgm @devbww @yoshi-automation