Query that uses skip index fails if index is not materialized

[CurtizJ]

CREATE TABLE t ( a UInt32) ENGINE = MergeTree ORDER BY tuple();

INSERT INTO t VALUES (1);

ALTER TABLE t ADD INDEX ind (a) TYPE set(1) GRANULARITY 1;

SELECT count() FROM t WHERE a = 1;

Received exception from server (version 21.11.5):
Code: 1001. DB::Exception: Received from localhost:9000. DB::Exception: std::__1::__fs::filesystem::filesystem_error: filesystem error: in file_size: No such file or directory [./store/402/402d2274-1310-43f7-802d-22741310f3f7/all_1_1_0_2/skp_idx_ind.mrk3]. (STD_EXCEPTION)

Does it reproduce on recent release?

Reproduces on 21.11 and master.

Expected behavior

Index should be skipped if it doesn't exist in part.

Add any other context about the problem here.

Most likely introduced in #27250.