DB::TiFlashException: Column index out of bound: 4, should in [0,3) #9188

JinheLin · 2024-07-03T14:36:50Z

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

 drop table if exists test.t;
 create table if not exists test.t(a int);
 alter table test.t add column b int as (a+1) virtual;
 alter table test.t add column c int;
 alter table test.t add column d int as (c+1) virtual;
 alter table test.t add column t time(6);


 insert into test.t(a, c, t) values(1, 2, '000:10:10.123456'), (3, 4, '001:10:10.123456'), (5, 6, '002:10:10.123456');
 insert into test.t(a, c, t) select a, c, t from test.t;
 insert into test.t(a, c, t) select a, c, t from test.t;
 insert into test.t(a, c, t) select a, c, t from test.t;
 insert into test.t(a, c, t) select a, c, t from test.t;
 insert into test.t(a, c, t) select a, c, t from test.t;
 insert into test.t(a, c, t) select a, c, t from test.t;
 insert into test.t(a, c, t) select a, c, t from test.t;
 insert into test.t(a, c, t) select a, c, t from test.t;
 insert into test.t(a, c, t) select a, c, t from test.t;
 insert into test.t(a, c, t) select a, c, t from test.t;
 insert into test.t(a, c, t) select a, c, t from test.t;
 insert into test.t(a, c, t) select a, c, t from test.t;
 insert into test.t(a, c, t) select a, c, t from test.t;
 

 alter table test.t set tiflash replica 1;

select a, b, c, d, hour(t) from test.t where t = '000:10:10.123456';

2. What did you expect to see? (Required)

+------+------+------+------+---------+
| a    | b    | c    | d    | hour(t) |
+------+------+------+------+---------+
|    1 |    2 |    2 |    3 |       0 |
|    1 |    2 |    2 |    3 |       0 |
|    1 |    2 |    2 |    3 |       0 |
|    1 |    2 |    2 |    3 |       0 |
|    1 |    2 |    2 |    3 |       0 |
|    1 |    2 |    2 |    3 |       0 |
|    1 |    2 |    2 |    3 |       0 |
|    1 |    2 |    2 |    3 |       0 |
|    1 |    2 |    2 |    3 |       0 |
|    1 |    2 |    2 |    3 |       0 |
|    1 |    2 |    2 |    3 |       0 |
|    1 |    2 |    2 |    3 |       0 |
|    1 |    2 |    2 |    3 |       0 |
|    1 |    2 |    2 |    3 |       0 |
...
...
|    1 |    2 |    2 |    3 |       0 |
+------+------+------+------+---------+
8192 rows in set (0.04 sec)

3. What did you see instead (Required)

ERROR 1105 (HY000): other error for mpp stream: Code: 0, e.displayText() = DB::TiFlashException: Column index out of bound: 4, should in [0,3): (while creating read sources from storage `db_2`.`t_534`, keyspace_id=4294967295 table_id=534), e.what() = DB::TiFlashException,

4. What is your TiFlash version? (Required)

 ./tiflash --version
TiFlash
Release Version: v8.2.0-alpha-79-g20d46162a
Edition:         Community
Git Commit Hash: 20d46162a6a426991cd9ad2f755072caa8408e9f
Git Branch:      HEAD
UTC Build Time:  2024-07-03 04:13:41
Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto
Profile:         RELWITHDEBINFO
Compiler:        clang++ 17.0.6

Raft Proxy
Git Commit Hash:   b8c00d3953fc847a4958cec69b69ad5aa45c8a6d
Git Commit Branch: HEAD
UTC Build Time:    2024-07-03 04:15:32
Rust Version:      rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Storage Engine:    tiflash
Prometheus Prefix: tiflash_proxy_
Profile:           release
Enable Features:   external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored

The text was updated successfully, but these errors were encountered:

JinheLin · 2024-07-04T02:46:15Z

[2024/07/03 22:13:05.170 +08:00] [ERROR] [MPPTask.cpp:644] ["task running meets error: Code: 0, e.displayText() = DB::TiFlashException: Column index out of bound: 4, should in [0,3): (while creating read sources from storage `db_2`.`t_534`, keyspace_id=4294967295 table_id=534), e.what() = DB::TiFlashException, Stack trace:


  0x55d805e05111\tDB::TiFlashException::TiFlashException<long&, unsigned long>(DB::TiFlashError const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, long&, unsigned long&&) [tiflash+122278161]
                \tdbms/src/Common/StackTrace.cpp:23
  0x55d805e04f1b\tDB::DM::cop::getColumnDefineForColumnExpr(tipb::Expr const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine>> const&) [tiflash+122277659]
                \tdbms/src/Storages/DeltaMerge/FilterParser/FilterParser.cpp:88
  0x55d805e060c9\tDB::DM::cop::parseTiCompareExpr(tipb::Expr const&, DB::DM::FilterParser::RSFilterType, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine>> const&, std::__1::function<DB::DM::Attr (long)> const&, DB::TimezoneInfo const&) [tiflash+122282185]
                \tdbms/src/Storages/DeltaMerge/FilterParser/FilterParser.cpp:166
  0x55d805e055f0\tDB::DM::cop::parseTiExpr(tipb::Expr const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine>> const&, std::__1::function<DB::DM::Attr (long)> const&, DB::TimezoneInfo const&, std::__1::shared_ptr<DB::Logger> const&) [tiflash+122279408]
                \tdbms/src/Storages/DeltaMerge/FilterParser/FilterParser.cpp:310
  0x55d805e07ac3\tDB::DM::FilterParser::parseDAGQuery(DB::DAGQueryInfo const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine>> const&, std::__1::function<DB::DM::Attr (long)>&&, std::__1::shared_ptr<DB::Logger> const&) [tiflash+122288835]
                \tdbms/src/Storages/DeltaMerge/FilterParser/FilterParser.cpp:374
  0x55d805dfb96e\tDB::DM::RSOperator::build(std::__1::unique_ptr<DB::DAGQueryInfo, std::__1::default_delete<DB::DAGQueryInfo>> const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine>> const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine>> const&, bool, std::__1::shared_ptr<DB::Logger> const&) [tiflash+122239342]
                \tdbms/src/Storages/DeltaMerge/Filter/RSOperator.cpp:79
  0x55d80699f74e\tDB::DM::PushDownFilter::build(DB::SelectQueryInfo const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine>> const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine>> const&, DB::Context const&, std::__1::shared_ptr<DB::Logger> const&) [tiflash+134444878]
                \tdbms/src/Storages/DeltaMerge/Filter/PushDownFilter.cpp:166
  0x55d806984dce\tDB::StorageDeltaMerge::read(DB::PipelineExecutorContext&, DB::PipelineExecGroupBuilder&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, DB::SelectQueryInfo const&, DB::Context const&, unsigned long, unsigned int) [tiflash+134335950]
                \tdbms/src/Storages/StorageDeltaMerge.cpp:881
  0x55d8073c4144\tDB::DAGStorageInterpreter::executeImpl(DB::PipelineExecutorContext&, DB::PipelineExecGroupBuilder&) [tiflash+145080644]
                \tdbms/src/Flash/Coprocessor/DAGStorageInterpreter.cpp:1124
  0x55d8073b981c\tDB::PhysicalTableScan::buildPipeline(DB::PipelineBuilder&, DB::Context&, DB::PipelineExecutorContext&) [tiflash+145037340]
                \tdbms/src/Flash/Coprocessor/DAGStorageInterpreter.cpp:327
  0x55d80725f7f2\tDB::PhysicalPlanNode::buildPipeline(DB::PipelineBuilder&, DB::Context&, DB::PipelineExecutorContext&) [tiflash+143620082]
                \tdbms/src/Flash/Planner/PhysicalPlanNode.cpp:142
  0x55d80725f7f2\tDB::PhysicalPlanNode::buildPipeline(DB::PipelineBuilder&, DB::Context&, DB::PipelineExecutorContext&) [tiflash+143620082]
                \tdbms/src/Flash/Planner/PhysicalPlanNode.cpp:142
  0x55d80724f565\tDB::PhysicalPlan::toPipeline(DB::PipelineExecutorContext&, DB::Context&) [tiflash+143553893]
                \tdbms/src/Flash/Planner/PhysicalPlan.cpp:326
  0x55d8071e88ab\tDB::PipelineExecutor::PipelineExecutor(std::__1::shared_ptr<MemoryTracker> const&, DB::AutoSpillTrigger*, std::__1::function<void (std::__1::shared_ptr<DB::OperatorSpillContext> const&)> const&, DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [tiflash+143132843]
                \tdbms/src/Flash/Executor/PipelineExecutor.cpp:45
  0x55d8070ec0bf\tDB::queryExecute(DB::Context&, bool) [tiflash+142098623]
                \t/usr/local/bin/../include/c++/v1/__memory/unique_ptr.h:689
  0x55d80718b30c\tDB::MPPTask::runImpl() [tiflash+142750476]
                \tdbms/src/Flash/Mpp/MPPTask.cpp:482
  0x55d80623b845\tauto DB::wrapInvocable<std::__1::function<void ()>>(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+126695493]
                \t/usr/local/bin/../include/c++/v1/__functional/function.h:517
  0x55d800968c86\tDB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask>>&) [tiflash+33557638]
                \tdbms/src/Common/DynamicThreadPool.cpp:124
  0x55d8009686cb\tDB::DynamicThreadPool::fixedWork(unsigned long) [tiflash+33556171]
                \tdbms/src/Common/DynamicThreadPool.cpp:138
  0x55d80096a2ef\tvoid* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...), DB::DynamicThreadPool*, unsigned long>>(void*) [tiflash+33563375]
                \t/usr/local/bin/../include/c++/v1/__type_traits/invoke.h:308
  0x7fd200b66802\tstart_thread [libc.so.6+653314]
  0x7fd200b06450\tclone3 [libc.so.6+259152]"] [source="MPP<gather_id:1, query_ts:1720015981735969134, local_query_id:15, server_id:1028, start_ts:450891869515677697,task_id:1>"] [thread_id=768]

JinheLin · 2024-07-04T02:47:09Z

https://github.com/pingcap/tiflash/blob/master/dbms/src/Storages/DeltaMerge/FilterParser/FilterParser.cpp#L88

ColumnDefine getColumnDefineForColumnExpr(const tipb::Expr & expr, const ColumnDefines & columns_to_read)
{
    assert(isColumnExpr(expr));
    auto column_index = decodeDAGInt64(expr.val());
    if (column_index < 0 || column_index >= static_cast<Int64>(columns_to_read.size()))
    {
        throw TiFlashException(
            Errors::Coprocessor::BadRequest,
            "Column index out of bound: {}, should in [0,{})",
            column_index,
            columns_to_read.size());
    }
    return columns_to_read[column_index];
}

JinheLin · 2024-07-04T06:27:55Z

Another case of reproducing the issue with incorrect query results.

1. Create table and insert data.

 drop table if exists test.t;
 create table if not exists test.t(a int);
 alter table test.t add column b int as (a+1) virtual;
 alter table test.t add column c int;
 alter table test.t add column d int as (c+1) virtual;
 alter table test.t add column e int;


insert into test.t(a, c, e) values(1, 10, 100), (2, 20, 200), (3, 30, 300), (4, 40, 400), (5, 50, 500), (6, 60, 600), (7, 70, 700), (8, 80, 800), (9, 90, 900);

alter table test.t set tiflash replica 1;

2. Query result with TiFlash.

mysql> set tidb_isolation_read_engines='tiflash';
Query OK, 0 rows affected (0.00 sec)

mysql> select a, b, c, d, e from test.t where c = 10;
Empty set (0.10 sec)

3. Query result with TiKV.

mysql> set tidb_isolation_read_engines='tikv';
Query OK, 0 rows affected (0.00 sec)

mysql> select a, b, c, d, e from test.t where c = 10;
+------+------+------+------+------+
| a    | b    | c    | d    | e    |
+------+------+------+------+------+
|    1 |    2 |   10 |   11 |  100 |
+------+------+------+------+------+
1 row in set (0.00 sec)

…irtual columns in the query (#9189) close #9188 1. Use the original columns in `query_info.dag_query` instead of `columns_to_read` when building `RSOperator`. 2. For runtime filters, creating the `DM::Attr` object in `StorageDeltaMerge::read`, so it doesn't need to rely on `column_to_read`. Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

…irtual columns in the query (#9189) (#9206) close #9188 1. Use the original columns in `query_info.dag_query` instead of `columns_to_read` when building `RSOperator`. 2. For runtime filters, creating the `DM::Attr` object in `StorageDeltaMerge::read`, so it doesn't need to rely on `column_to_read`. Signed-off-by: ti-chi-bot <[email protected]> Co-authored-by: jinhelin <[email protected]>

…irtual columns in the query (release-8.1) (#9265) close #9188 1. Use the original columns in `query_info.dag_query` instead of `columns_to_read` when building `RSOperator`. 2. For runtime filters, creating the `DM::Attr` object in `StorageDeltaMerge::read`, so it doesn't need to rely on `column_to_read`. Signed-off-by: ti-chi-bot <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: jinhelin <[email protected]>

…irtual columns in the query (release-7.1) (#9267) close #9188 Use the original columns in `query_info.dag_query` instead of `columns_to_read` when building `RSOperator`. Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: Lloyd-Pottiger <[email protected]> Co-authored-by: Lloyd-Pottiger <[email protected]>

JinheLin added type/bug The issue is confirmed as a bug. severity/moderate labels Jul 3, 2024

JaySon-Huang added severity/major component/storage and removed severity/moderate labels Jul 4, 2024

ti-chi-bot bot added may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 labels Jul 4, 2024

JaySon-Huang added affects-7.1 affects-7.5 affects-8.1 and removed may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 labels Jul 4, 2024

JinheLin mentioned this issue Jul 5, 2024

Storages: Fix obtaining incorrect column information when there are virtual columns in the query #9189

Merged

12 tasks

ti-chi-bot bot closed this as completed in #9189 Jul 9, 2024

JinheLin mentioned this issue Jul 29, 2024

Storages: Fix obtaining incorrect column information when there are virtual columns in the query (release-8.1) #9265

Merged

12 tasks

JinheLin mentioned this issue Jul 30, 2024

Storages: Fix obtaining incorrect column information when there are virtual columns in the query (release-7.1) #9267

Merged

12 tasks

jebter added the impact/panic label Aug 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DB::TiFlashException: Column index out of bound: 4, should in [0,3) #9188

DB::TiFlashException: Column index out of bound: 4, should in [0,3) #9188

JinheLin commented Jul 3, 2024

JinheLin commented Jul 4, 2024

JinheLin commented Jul 4, 2024 •

edited

Loading

JinheLin commented Jul 4, 2024

DB::TiFlashException: Column index out of bound: 4, should in [0,3) #9188

DB::TiFlashException: Column index out of bound: 4, should in [0,3) #9188

Comments

JinheLin commented Jul 3, 2024

Bug Report

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

JinheLin commented Jul 4, 2024

JinheLin commented Jul 4, 2024 • edited Loading

JinheLin commented Jul 4, 2024

Another case of reproducing the issue with incorrect query results.

1. Create table and insert data.

2. Query result with TiFlash.

3. Query result with TiKV.

JinheLin commented Jul 4, 2024 •

edited

Loading