[TTL] Full compaction does not delete expired data

[shanlai]

Please check the FAQ documentation before raising an issue

Describe the bug (required)
as title

Your Environments (required)
3.5.0 rc 40c4bc7

How To Reproduce(required)

1. create schema and insert test data

CREATE SPACE `nba1` (partition_num = 10, replica_factor = 1, charset = utf8, collate = utf8_bin, vid_type = FIXED_STRING(32));

USE nba1;
CREATE TAG player(name string, age int, update_time timestamp DEFAULT now()) TTL_DURATION= 3, TTL_COL = "update_time";

INSERT VERTEX player(name, age) VALUES "player100":("Tim Duncan", 42);

2. After the data is expired, submit job:compact and stats in order, then use show stats to check, and find that the statistics contains expired data

(root@nebula) [nba1]> match (v) return v;
+---+
| v |
+---+
+---+
Empty set (time spent 844µs/14.927584ms)

Fri, 21 Apr 2023 12:34:26 CST

(root@nebula) [nba1]> submit job compact
+------------+
| New Job Id |
+------------+
| 730        |
+------------+
Got 1 rows (time spent 1.175ms/23.192875ms)

Fri, 21 Apr 2023 12:34:33 CST

(root@nebula) [nba1]> submit job stats
+------------+
| New Job Id |
+------------+
| 731        |
+------------+
Got 1 rows (time spent 930µs/17.026792ms)

Fri, 21 Apr 2023 12:34:38 CST

(root@nebula) [nba1]> show stats
+---------+------------+-------+
| Type    | Name       | Count |
+---------+------------+-------+
| "Tag"   | "player"   | 1     |
| "Space" | "vertices" | 1     |
| "Space" | "edges"    | 0     |
+---------+------------+-------+
Got 3 rows (time spent 469µs/14.285ms)

Fri, 21 Apr 2023 12:34:42 CST

Expected behavior
Full compaction should to delete expired data

[luyade]

set min_level_for_custom_filter to be 0 will be fine

[HarrisChu]

#5447

[HarrisChu]

root cause:

if the tag is TTL expired, we would always calculate the stats unless compaction.
maybe we could fix it in StatsTask

[shanlai]

set min_level_for_custom_filter to be 0 will be fine

test passed after setting min_level_for_custom_filter to 0, It is recommended to add a description of this parameter in the documentation

[Sophie-Xie]

@luyade @critical27 discussed it with @MuYiYong offline, can we keep the same behavior as in the past, default value is 0?

[critical27]

root cause:

if the tag is TTL expired, we would always calculate the stats unless compaction. maybe we could fix it in StatsTask

Agree, the bug has no relation with the new gflag min_level_for_custom_filter.
Change it to 0 is a bypass, you can decide whether to change it or not. @MuYiYong

[Sophie-Xie]

root cause:
if the tag is TTL expired, we would always calculate the stats unless compaction. maybe we could fix it in StatsTask

Agree, the bug has no relation with the new gflag min_level_for_custom_filter. Change it to 0 is a bypass, you can decide whether to change it or not. @MuYiYong

discuss offline with @MuYiYong : set min_level_for_custom_filter to 0 by default.

[Sophie-Xie]

@luyade can you fix it? Thanks~~

[luyade]

@luyade can you fix it? Thanks~~

No pro. Already added the PR.