tiflash crash when query

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、set placement rule
2、query tiflash

2. What did you expect to see? (Required)

no crash

3. What did you see instead (Required)

tiflash crash
{"namespace":"pdms-sql-related-tps-7549516-1-774","pod":"tc-tiflash-0","container":"errorlog","time":"2024-07-29T03:37:48.040411009Z","stream":"stdout","log":"[2024/07/29 11:37:47.639 +08:00] [FATAL] [Exception.cpp:106] ["Code: 81, e.displayText() = DB::Exception: Database db_170 doesn't exist, e.what() = DB::Exception, Stack trace:\n\n\n 0x56247118e553\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, int) [tiflash+32740691]\n \tdbms/src/Common/StackTrace.cpp:23\n 0x5624769c286a\tDB::Context::assertDatabaseExists(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, bool) const [tiflash+125229162]\n \tdbms/src/Interpreters/Context.cpp:865\n 0x5624769c255e\tDB::Context::getDatabase(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&) const [tiflash+125228382]\n \tdbms/src/Interpreters/Context.cpp:507\n 0x562476c1cd8f\tDB::InterpreterRenameQuery::execute() [tiflash+127696271]\n \tdbms/src/Interpreters/InterpreterRenameQuery.cpp:171\n 0x5624778a886c\tDB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyRenamePhysicalTable(long, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, TiDB::TableInfo const&, std::__1::shared_ptrDB::IManageableStorage const&) [tiflash+140851308]\n \tdbms/src/TiDB/Schema/SchemaBuilder.cpp:749\n 0x56247789afe2\tDB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyRenameTable(long, long) [tiflash+140795874]\n \tdbms/src/TiDB/Schema/SchemaBuilder.cpp:678\n 0x5624778962cb\tDB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyDiff(DB::SchemaDiff const&) [tiflash+140776139]\n \tdbms/src/TiDB/Schema/SchemaBuilder.cpp:323\n 0x5624710cac31\tDB::TiDBSchemaSyncer<false, false>::syncSchemasByGetter(DB::Context&, DB::SchemaGetter&) [tiflash+31939633]\n \tdbms/src/TiDB/Schema/TiDBSchemaSyncer.cpp:194\n 0x5624710cbd0b\tDB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+31943947]\n \tdbms/src/TiDB/Schema/TiDBSchemaSyncer.cpp:277\n 0x562477f7615e\tstd::__1::optionalDB::DM::RaftWriteResult DB::writeRegionDataToStorage<std::__1::vector<DB::RegionDataReadInfo, std::__1::allocatorDB::RegionDataReadInfo>>(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<DB::RegionDataReadInfo, std::__1::allocatorDB::RegionDataReadInfo>&, std::__1::shared_ptrDB::Logger const&) [tiflash+147984734]\n \tdbms/src/TiDB/Schema/TiDBSchemaManager.h:41\n 0x562477f75b09\tDB::RegionTable::writeCommittedByRegion(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<DB::RegionDataReadInfo, std::__1::allocatorDB::RegionDataReadInfo>&, std::__1::shared_ptrDB::Logger const&, bool) [tiflash+147983113]\n \tdbms/src/Storages/KVStore/Decode/PartitionStreams.cpp:456\n 0x562477eee851\tDB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&) [tiflash+147429457]\n \tdbms/src/Storages/KVStore/MultiRaft/RaftCommands.cpp:481\n 0x562477ef58d7\tDB::KVStore::handleWriteRaftCmdInner(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&, std::1::optionalDB::DM::RaftWriteResult&) [tiflash+147458263]\n \tdbms/src/Storages/KVStore/MultiRaft/RaftCommandsKVS.cpp:63\n 0x562477ef5744\tDB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&) [tiflash+147457860]\n \tdbms/src/Storages/KVStore/MultiRaft/RaftCommandsKVS.cpp:42\n 0x562477e87765\tHandleWriteRaftCmd [tiflash+147007333]\n \tdbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:98\n 0x7f73992bfea9\t$LT$engine_store_ffi..observer..TiFlashObserver$LT$T$C$ER$GT$$u20$as$u20$raftstore..coprocessor..QueryObserver$GT$::post_exec_query::h240c2c3d84d2ee6f [libtiflash_proxy.so+28831401]\n \tcontrib/tiflash-proxy/proxy_components/engine_store_ffi/src/observer.rs:155\n 0x7f739a54336c\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::apply_raft_cmd::hb8d12067eb11c51c [libtiflash_proxy.so+48243564]\n \tcontrib/tiflash-proxy/components/raftstore/src/store/fsm/apply.rs:1530\n 0x7f739a5557fe\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::process_raft_cmd::hf829479500040f95 [libtiflash_proxy.so+48318462]\n \tcontrib/tiflash-proxy/components/raftstore/src/store/fsm/apply.rs:1399\n 0x7f739a5580e9\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::handle_raft_committed_entries::hee24047b52360b75 [libtiflash_proxy.so+48328937]\n \tcontrib/tiflash-proxy/components/raftstore/src/store/fsm/apply.rs:1145\n 0x7f739a53373d\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_apply::hca3398a59a64f6ce [libtiflash_proxy.so+48179005]\n \tcontrib/tiflash-proxy/components/raftstore/src/store/fsm/apply.rs:4068\n 0x7f739a537a01\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_tasks::h17829962adcd7513 [libtiflash_proxy.so+48196097]\n \tcontrib/tiflash-proxy/components/raftstore/src/store/fsm/apply.rs:4423\n 0x7f739942ee55\t$LT$raftstore..store..fsm..apply..ApplyPoller$LT$EK$GT$$u20$as$u20$batch_system..batch..PollHandler$LT$raftstore..store..fsm..apply..ApplyFsm$LT$EK$GT$$C$raftstore..store..fsm..apply..ControlFsm$GT$$GT$::handle_normal::hd7705b57f19da499 [libtiflash_proxy.so+30334549]\n \tcontrib/tiflash-proxy/components/raftstore/src/store/fsm/apply.rs:4712\n 0x7f73993515d6\tbatch_system::batch::Poller$LT$N$C$C$C$Handler$GT$::poll::h24e1fe1cc3f17aa1 [libtiflash_proxy.so+29427158]\n \tcontrib/tiflash-proxy/components/batch-system/src/batch.rs:380\n 0x7f739948ffbf\tstd::sys_common::backtrace::__rust_begin_short_backtrace::h0d3b967cdfc91423 [libtiflash_proxy.so+30732223]\n \t/rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:155\n 0x7f73994ee4f9\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h544bde9cf8936717 [libtiflash_proxy.so+31118585]\n \t/rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/ops/function.rs:250\n 0x7f739ae55186\tstd::sys::unix::thread::Thread::new::thread_start::hd01330a228cdd35d [libtiflash_proxy.so+57753990]\n \t/rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys/unix/thread.rs:108\n 0x7f7397411802\tstart_thread [libc.so.6+653314]\n 0x7f73973b1314\tclone [libc.so.6+258836]"] [source="EngineStoreApplyRes DB::HandleWriteRaftCmd(const EngineStoreServerWrap *, WriteCmdsView, RaftCmdHeader)"] [thread_id=39]\n"}

4. What is your TiFlash version? (Required)

v8.3.0 master

[Lily2025]

/assign Lloyd-Pottiger

[Lily2025]

/type bug
/severity critical

[JaySon-Huang]

The database has been dropped in TiDB before TiFlash sync the schema diff of "create database". And the following rename table to that database (database_id=170) throw exception.

TiDB	TiFlash
create table db_2.t_166;
	sync create table db_2.t_166 and success
create database db_170;
rename table db_2.t_166 to db_170.t_166;
drop table db_170.t_166;
drop database db_170;
	raft-log comes after database has been dropped, tiflash begin its schema sync
	begin to sync create db_170 but failed sliently
	begin to sync rename db_2.t_166 to db_170.t_166 but exception thrown (crash here)

The related loggings

[2024/07/29 11:16:07.402 +08:00] [INFO] [SchemaBuilder.cpp:1130] ["Create table db_2.t1 begin, database_id=2, table_id=166 action=CreateTable"] [source="keyspace=4294967295"] [thread_id=33]
[2024/07/29 11:16:07.403 +08:00] [INFO] [SchemaBuilder.cpp:1210] ["Creat table db_2.t1 end, database_id=2 table_id=166 action=CreateTable"] [source="keyspace=4294967295"] [thread_id=33]
[2024/07/29 11:16:07.403 +08:00] [INFO] [SchemaBuilder.cpp:916] ["Create database is ignored because database is not exist in TiKV, database_id=170"] [source="keyspace=4294967295"] [thread_id=33]
[2024/07/29 11:16:07.405 +08:00] [INFO] [SchemaBuilder.cpp:736] ["Rename table db_2.t_166 (display name: t1) to db_170.t2 begin, database_id=170 table_id=166"] [source="keyspace=4294967295"] [thread_id=33]
[2024/07/29 11:16:11.472 +08:00] [ERROR] [SchemaSyncService.cpp:112] ["Sync schemas, keyspace=4294967295 failed by DB::Exception: Database db_170 doesn't exist 
 stack : 
  0x56247118e553    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, int) [tiflash+32740691]
                    dbms/src/Common/StackTrace.cpp:23
  0x5624769c286a    DB::Context::assertDatabaseExists(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, bool) const [tiflash+125229162]
                    dbms/src/Interpreters/Context.cpp:865
  0x5624769c255e    DB::Context::getDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) const [tiflash+125228382]
                    dbms/src/Interpreters/Context.cpp:507
  0x562476c1cd8f    DB::InterpreterRenameQuery::execute() [tiflash+127696271]
                    dbms/src/Interpreters/InterpreterRenameQuery.cpp:171
  0x5624778a886c    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyRenamePhysicalTable(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, TiDB::TableInfo const&, std::__1::shared_ptr<DB::IManageableStorage> const&) [tiflash+140851308]
                    dbms/src/TiDB/Schema/SchemaBuilder.cpp:749
  0x56247789afe2    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyRenameTable(long, long) [tiflash+140795874]
                    dbms/src/TiDB/Schema/SchemaBuilder.cpp:678
  0x5624778962cb    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyDiff(DB::SchemaDiff const&) [tiflash+140776139]
                    dbms/src/TiDB/Schema/SchemaBuilder.cpp:323
  0x5624710cac31    DB::TiDBSchemaSyncer<false, false>::syncSchemasByGetter(DB::Context&, DB::SchemaGetter&) [tiflash+31939633]
                    dbms/src/TiDB/Schema/TiDBSchemaSyncer.cpp:194
  0x5624710ca529    DB::TiDBSchemaSyncer<false, false>::syncSchemas(DB::Context&) [tiflash+31937833]
                    dbms/src/TiDB/Schema/TiDBSchemaSyncer.cpp:52
  0x562477869489    std::__1::__function::__func<DB::SchemaSyncService::addKeyspaceGCTasks()::$_0, std::__1::allocator<DB::SchemaSyncService::addKeyspaceGCTasks()::$_0>, bool ()>::operator()() (.dd66e44688ab4f550ab008c20c3eeab0) [tiflash+140592265]
                    dbms/src/TiDB/Schema/TiDBSchemaManager.h:35
  0x5624772c6cf3    void* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>)::$_1>>(void*) [tiflash+134683891]
                    /usr/local/bin/../include/c++/v1/__functional/function.h:517
  0x7f7397411802    start_thread [libc.so.6+653314]
  0x7f73973b1314    clone [libc.so.6+258836]"] [source="keyspace=4294967295"] [thread_id=33]

[JaySon-Huang]

/assign @JaySon-Huang

[JaySon-Huang]

release-7.1/6.5/6.1 or previous versions are also affected. The stack is as follow

[2024/07/31 21:54:21.914 +08:00] [WARN] [TiDBSchemaSyncer.h:279] ["apply diff meets exception : DB::TiFlashException: miss database: 115 
 stack is 
  0x55741499cb0e    StackTrace::StackTrace() [tiflash+66059022]
                    dbms/src/Common/StackTrace.cpp:23
  0x55741498c7d2    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, int) [tiflash+65992658]
                    dbms/src/Common/Exception.h:46
  0x5574149d8f47    DB::TiFlashException::TiFlashException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::TiFlashError const&) [tiflash+66305863]
                    dbms/src/Common/TiFlashException.h:250
  0x55741e30a072    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyDiff(DB::SchemaDiff const&) [tiflash+226938994]
                    dbms/src/TiDB/Schema/SchemaBuilder.cpp:506
  0x55741e22062b    DB::TiDBSchemaSyncer<false, false>::tryLoadSchemaDiffs(DB::SchemaGetter&, long, long, DB::Context&, std::__1::shared_ptr<DB::Logger> const&) [tiflash+225981995]
                    dbms/src/TiDB/Schema/TiDBSchemaSyncer.h:270
  0x55741e21cf3f    DB::TiDBSchemaSyncer<false, false>::syncSchemas(DB::Context&, unsigned int) [tiflash+225967935]
                    dbms/src/TiDB/Schema/TiDBSchemaSyncer.h:170
...