[Ansor][AutoTVM v2.0] Phase 2: Basic GPU Sketch Search Policy

[jcf94]

For full upstream plan, see Ansor RFC.

In this PR, we bring the basic Sketch search policy which is proposed in our paper on GPU ops/subgraphs.
The complete search policy still waits for the feature extraction & cost model support, so this will be a basic search policy using random cost model.

We'll continue to work on the upstreaming process to bring the complete auto schedule support for various workloads on CPU/GPU with some reproducible performance results, a custom sketch rule support to work like the AutoTVM, as well as the tutorials of how to play with AutoScheduler.

Review guide

Main changes:

• Add PreloadMeasuredStates
• Split the Python API of search_policy to a separate file search_policy.py
• Bug fix on auto_scheduler.auto_schedule returning empty state
• Bug fix for measure record. The former implementation used the default std::vector<int> handler, which output multiple lines when the array size is larger than 10
• A brief tutorial for CPU that used RandomModel, the performance is not guaranteed since currently our cost model support has not finished
• Use macro definition to simplify the sketch_policy_rules.h
• The most important part: Add extra rules for GPU policy

cc @merrymercy @FrozenGene @minminsun @tqchen @junrushao1994

[merrymercy]

Please remove the tutorial. It is better to make the tutorials as a single PR, so people can review it easily and give more suggestions on the API design. It is not good to leave "todo" in the tutorial.

I think the current API still has room for improvement and I will work on it later.

[jcf94]

Please remove the tutorial. It is better to make the tutorials as a single PR, so people can review it easily and give more suggestions on the API design. It is not good to leave "todo" in the tutorial.

I think the current API still has room for improvement and I will work on it later.

Fine, these modifications are removed.

[CyanFi]

I think there is a typo in the comments here
line 287 should be "Skip the rest rules" instead of "Skip the reset rules"

[jcf94]

I think there is a typo in the comments here
line 287 should be "Skip the rest rules" instead of "Skip the reset rules"

Thanks, you're right.
You can write your review comments on the code directly. : )

[merrymercy]

cc @comaniac @Laurawly

[comaniac]

Just minor comments. Also I think #6310 should be based on this PR as they share some utilities and structures. I'll rebase it before marking it ready for review after this PR is merged.

[comaniac]

I'd like to raise a discussion about using macro here. Although we can of course largely reduce the code size with this macro, it makes code tracing more difficult. For example, most IDE navigation cannot locate class RuleSkipStage anymore. If we want to avoid redundant lines of

std::vector<std::pair<State, int>> Apply(const SketchPolicyNode& policy, const State& state,	
                                           int stage_id) const final;

maybe we can make it not a virtual function so that we can simply override it in the cc file without declaration in .h file?

[jcf94]

I don't have clear idea on this part, but I think it's fine to try your suggestion.
Simply merged this PR first, so it will be more convenient for you to try different class structures in your PR.

[FrozenGene]

Thanks @jcf94 @merrymercy @comaniac merged