Document Top-N pushdown

[findinpath]

Resolves: #7786

[lhofhansl]

Do we also want to document partial topN replacement by partial Limit for pre-sorted inputs?

[hashhar]

This depends on the connector though. The string comes from how the connector implemented ConnectorTableHandle#toString.

The actual thing which lets you know pushdown happenned is the fact that there's no TopNNode or LimitNode in the plan - only a TableScan.

@mosabua can point to some other pushdown docs which can be consulted for the best way to word it. Maybe https://github.com/trinodb/trino/pull/8463/files#diff-3a35320d12c22f7d413cda275bad87d6467ea3482f35eda3f89c51af38c99e85R163-R165?

[findinpath]

Do we also want to document partial topN replacement by partial Limit for pre-sorted inputs?

@lhofhansl yes, sure. Can you please point me to a concrete sample of a partial TOP N query?

[lhofhansl]

@findinpath That feature was implemented in #6634. Do you need an example query?

[findinpath]

@lhofhansl yes, please do share a sample query for partial top N

[hashhar]

@findinpath Partial TopN example -

trino/plugin/trino-base-jdbc/src/test/java/io/trino/plugin/jdbc/BaseJdbcConnectorTest.java

Lines 884 to 897 in 338425d

	// We expect PARTIAL TopN on the LEFT side of join to be pushed down.
	assertThat(query(
	// Explicitly disable join pushdown. The purpose of the this test is to verify partial TopN gets pushed down when Join is not.
	Session.builder(getSession())
	.setCatalogSessionProperty(getSession().getCatalog().orElseThrow(), JOIN_PUSHDOWN_ENABLED, "false")
	.build(),
	"SELECT * FROM nation n LEFT JOIN region r ON n.regionkey = r.regionkey " +
	"ORDER BY n.nationkey LIMIT 3"))
	.ordered()
	.isNotFullyPushedDown(
	node(TopNNode.class, // FINAL TopN
	anyTree(node(JoinNode.class,
	node(ExchangeNode.class, node(ProjectNode.class, node(TableScanNode.class))), // no PARTIAL TopN
	anyTree(node(TableScanNode.class))))));

See

trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveConnectorTest.java

Lines 7616 to 7640 in 5b785cf

	public void testUseSortedProperties()
	{
	String tableName = "test_propagate_table_scan_sorting_properties";
	@Language("SQL") String createTableSql = format("" +
	"CREATE TABLE %s " +
	"WITH (" +
	" bucket_count = 8," +
	" bucketed_by = ARRAY['custkey']," +
	" sorted_by = ARRAY['custkey']" +
	") AS " +
	"SELECT * FROM tpch.tiny.customer",
	tableName);
	assertUpdate(createTableSql, 1500L);
	@Language("SQL") String expected = "SELECT custkey FROM customer ORDER BY 1 NULLS FIRST LIMIT 100";
	@Language("SQL") String actual = format("SELECT custkey FROM %s ORDER BY 1 NULLS FIRST LIMIT 100", tableName);
	Session session = getSession();
	assertQuery(session, actual, expected, assertPartialLimitWithPreSortedInputsCount(session, 0));
	session = Session.builder(getSession())
	.setCatalogSessionProperty("hive", "propagate_table_scan_sorting_properties", "true")
	.build();
	assertQuery(session, actual, expected, assertPartialLimitWithPreSortedInputsCount(session, 1));
	assertUpdate("DROP TABLE " + tableName);
	}

and

trino/plugin/trino-phoenix5/src/test/java/io/trino/plugin/phoenix5/TestPhoenixConnectorTest.java

Lines 514 to 528 in 5b785cf

	public void testUseSortedPropertiesForPartialTopNElimination()
	{
	String tableName = "test_propagate_table_scan_sorting_properties";
	// salting ensures multiple splits
	String createTableSql = format("" +
	"CREATE TABLE %s WITH (salt_buckets = 5) AS " +
	"SELECT * FROM tpch.tiny.customer",
	tableName);
	assertUpdate(createTableSql, 1500L);
	String expected = "SELECT custkey FROM customer ORDER BY 1 NULLS FIRST LIMIT 100";
	String actual = format("SELECT custkey FROM %s ORDER BY 1 NULLS FIRST LIMIT 100", tableName);
	assertQuery(getSession(), actual, expected, assertPartialLimitWithPreSortedInputsCount(getSession(), 1));
	assertUpdate("DROP TABLE " + tableName);
	}

for the TopN -> Limit optimization for pre-sorted inputs.

I don't think you can use an just an example query, you'd need to include CREATE TABLE (since that's what will let Trino know that the table has pre-sorted data).

[colebow]

👋 @findinpath - this PR is inactive and doesn't seem to be under development. If you'd like to continue work on this at any point in the future, feel free to re-open.

cc @mosabua as well, we could potentially pick this up instead.

[mosabua]

We should continue this with @findinpath .. can you review @colebow ?

[hashhar]

Some comments. Technically accurate.

[hashhar]

LGTM % nits.

Thanks a lot for adding this.

[hashhar]

Maybe @mosabua can suggest some better opening statement?

[mosabua]

For example, you can examine at two queries that implement a Top-N behavior in the following section.

[hashhar]

WDYT? For example, you can find two queries to learn how to identify Top-N pushdown behavior in the following section.

[colebow]

@hashhar can we get this merged? It looks good to go.

[hashhar]

Sorry, this slipped through cracks. LGTM