[fix #8904] Improved output of datafuson-cli with empty results

[Tangruilin]

Which issue does this PR close?

Closes #8904 .

Rationale for this change

What changes are included in this PR?

Improved output of datafuson-cli with empty results

Are these changes tested?

There is alreay tests

Are there any user-facing changes?

[Tangruilin]

[Tangruilin]

@alamb maybe you can look at it.

[alamb]

Thank you @Tangruilin -- this is a good start

I am worried about this approach as it seems to require that a plan produce an empty RecordBatch when it makes now rows, but I don't think they all do so (some will continue to not make any)

What would you think about changing the display logic in datafusion-cli to detect the case when there have been no record batches printed and in that case add / print an empty batch?

Perhaps changing this code:

https://github.com/apache/arrow-datafusion/blob/795c71f3e8249847593a58dafe578ffcbcd3e012/datafusion-cli/src/exec.rs#L203-L259

[Tangruilin]

Thank you @Tangruilin -- this is a good start

I am worried about this approach as it seems to require that a plan produce an empty RecordBatch when it makes now rows, but I don't think they all do so (some will continue to not make any)

What would you think about changing the display logic in datafusion-cli to detect the case when there have been no record batches printed and in that case add / print an empty batch?

Perhaps changing this code:

https://github.com/apache/arrow-datafusion/blob/795c71f3e8249847593a58dafe578ffcbcd3e012/datafusion-cli/src/exec.rs#L203-L259

but how can i get the rows that is selected if I change exec_and_print . Do you have some suggestion?

[berkaysynnada]

I also believe we should not overfit when propagating empty batches during execution. In print_batches, there is a counter for printed rows. Perhaps we can make use of it.

[Tangruilin]

I also believe we should not overfit when propagating empty batches during execution. In print_batches, there is a counter for printed rows. Perhaps we can make use of it.

could you show me the code.
It seems that I can not get counter while printing rows

[berkaysynnada]

I also believe we should not overfit when propagating empty batches during execution. In print_batches, there is a counter for printed rows. Perhaps we can make use of it.

could you show me the code. It seems that I can not get counter while printing rows

https://github.com/apache/arrow-datafusion/blob/827668ab3d4659329ff30e6672f398c934b803be/datafusion-cli/src/print_options.rs#L108

We could perhaps insert some logic here, such that if no row has been printed, an empty batch is printed.

[Tangruilin]

I also believe we should not overfit when propagating empty batches during execution. In print_batches, there is a counter for printed rows. Perhaps we can make use of it.

could you show me the code. It seems that I can not get counter while printing rows

https://github.com/apache/arrow-datafusion/blob/827668ab3d4659329ff30e6672f398c934b803be/datafusion-cli/src/print_options.rs#L108

We could perhaps insert some logic here, such that if no row has been printed, an empty batch is printed.

This is a problem that I can not get column data here.

From example:
when select * from (values (1)) where column1 = 0, I can not know the column name is column1 here.

[berkaysynnada]

This is a problem that I can not get column data here.

From example: when select * from (values (1)) where column1 = 0, I can not know the column name is column1 here.

I understand the problem now. When we don't have a record batch, we cannot infer the header columns. Nothing comes to my mind yet, but I believe we can find a better way than propagating empty batches.

[alamb]

I understand the problem now. When we don't have a record batch,

What about using execute_stream instead of collect? The stream that is returned has the schema even if there are no record batches

You could the call https://docs.rs/datafusion/latest/datafusion/physical_plan/common/fn.collect.html if you still needed a Vec of batches

However, it would be great if we could figure out how to print record batches for every plan (even ones that are not "streaming") as they arrive rather than neeting to collect them all together - this would take a bigger refactoring (and need some finagling for the table output

[jonahgao]

This is a problem that I can not get column data here.

From example: when select * from (values (1)) where column1 = 0, I can not know the column name is column1 here.

Perhaps we can retrieve the column names from the schema of the physical plan.
https://github.com/apache/arrow-datafusion/blob/31b9b48b08592b7d293f46e75707aad7dadd7cbc/datafusion-cli/src/exec.rs#L240
And then pass the schema to print_batches, using it when batches are empty.

let schema = physical_plan.schema();
print_options.print_batches(&results, &schema, now)?;

[Tangruilin]

This is a problem that I can not get column data here.
From example: when select * from (values (1)) where column1 = 0, I can not know the column name is column1 here.

Perhaps we can retrieve the column names from the schema of the physical plan.

https://github.com/apache/arrow-datafusion/blob/31b9b48b08592b7d293f46e75707aad7dadd7cbc/datafusion-cli/src/exec.rs#L240

And then pass the schema to print_batches, using it when batches are empty.

let schema = physical_plan.schema();
print_options.print_batches(&results, &schema, now)?;

I may try the way this week

[alamb]

Marking as draft to note this PR is not waiting on feedback anymore -- please mark it ready for review when next ready.

Thanks again @Tangruilin

[alamb]

How is this PR coming along?

[Tangruilin]

How is this PR coming along?

Yes. I will submit a PR on Sunday~~~

[Tangruilin]

@alamb These days are Chinese New Year.

I will finish my issues after it.

[alamb]

@alamb These days are Chinese New Year.

I will finish my issues after it.

Thank you for the update and I hope your new year is everything you hope it will be 🎉

[github-actions]

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.