An exception in the renderer when adding a single document summary to the result twice

[ddashenkov]

Describe the bug
I run a complex query with several grouping requests. With this one particular grouping request, I first ouput all the hits of each group to get their IDs and then also output a single hit per group with a larger summary to get a particular field.

However, when I do so, the default renderer fails with a NPE:

Exception caught during response rendering.\nexception=\njava.lang.NullPointerException: Cannot invoke "com.yahoo.collections.ListenableArrayList.iterator()" because "this.hits" is null
at com.yahoo.search.result.HitGroup.getFilled(HitGroup.java:895)
at com.yahoo.search.result.HitGroup.getSummaryNamesNextFilledHit(HitGroup.java:926)
at com.yahoo.search.result.HitGroup.getFilled(HitGroup.java:896)
at com.yahoo.search.result.HitGroup.getSummaryNamesNextFilledHit(HitGroup.java:926)
at com.yahoo.search.result.HitGroup.getFilled(HitGroup.java:896)
at com.yahoo.search.result.HitGroup.likelyHitsHaveCorrectValueForSortFields(HitGroup.java:613)
at com.yahoo.search.result.HitGroup.ensureSorted(HitGroup.java:651)
at com.yahoo.search.result.HitGroup.asList(HitGroup.java:558)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.renderDataList(AsynchronousSectionedRenderer.java:409)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.renderData(AsynchronousSectionedRenderer.java:392)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.render(AsynchronousSectionedRenderer.java:351)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$RendererListener.run(AsynchronousSectionedRenderer.java:526)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.childCompleted(AsynchronousSectionedRenderer.java:374)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.endListLevel(AsynchronousSectionedRenderer.java:364)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.render(AsynchronousSectionedRenderer.java:354)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$RendererListener.run(AsynchronousSectionedRenderer.java:526)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.childCompleted(AsynchronousSectionedRenderer.java:374)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.endListLevel(AsynchronousSectionedRenderer.java:364)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.render(AsynchronousSectionedRenderer.java:354)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$RendererListener.run(AsynchronousSectionedRenderer.java:526)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.childCompleted(AsynchronousSectionedRenderer.java:374)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.endListLevel(AsynchronousSectionedRenderer.java:364)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.render(AsynchronousSectionedRenderer.java:354)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$RendererListener.run(AsynchronousSectionedRenderer.java:526)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.childCompleted(AsynchronousSectionedRenderer.java:374)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.endListLevel(AsynchronousSectionedRenderer.java:364)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.render(AsynchronousSectionedRenderer.java:354)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$RendererListener.run(AsynchronousSectionedRenderer.java:526)
at com.yahoo.processing.rendering.AsynchronousSectionedRenderer$DataListListener.lambda$listenTo$0(AsynchronousSectionedRenderer.java:439)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)\n

It seems that the check at AsynchronousSectionedRenderer.java:409 does not work as expected:

if (list.asList() == null) {
    logger.log(Level.WARNING, "DataList.asList() returned null, indicating it is closed. " +
        "This is likely caused by adding the same list multiple " +
        "times in the response.");
    return;
}

The main issue is that I don't even get an error. Instead, the response JSON just cuts off like this:

    ...
                                        "children": [
                                            {
                                                "id": "index:product/0/7c6a33a429318644d37d2192",
                                                "relevance": 0.4049755388852222,
                                                "source": "product",
                                                "fields": {
                                                    "sddocname": "product",
                                                    "id": "4159831",
                                                    "summaryfeatures":

To Reproduce

Here is the grouping request (formatted for better readability):

all(
  group(array.at(my_array, 1000)) 
  each(max(10) 
    each(output(summary(ony_ids_summary)))) as(only_ids)
    each(max(1) each(output(summary(rich_output_summary)))) as(rich_output)
)

Expected behavior
The output is rendered normally. Or, if this is the intended behavior, an error is rendered.

Environment (please complete the following information):

• OS: Linux
• Infrastructure: Kubernetes, self-hosted

Vespa version
8.360.19 (I am not able to check if the behavior is different with a newer version at this time).

Additional context
Since the input was cut off at the summaryfeatures, I added omit-summary-features to both summary classes. This did not seem to help.

At this moment, I am not horribly affected by the issue because I process the group request results in a custom searcher and use a custom renderer.

[arnej27959]

I can't easily reproduce this, but adding some more null checks might help.
It looks like this problem requires sorting, do you have a sorting or 'order' also?
also, what does the "my_array" field look like in the schema?

[arnej27959]

The NPE is fixed by #32507 but if you have any more details that could help me reproduce this I would be interested in digging a bit deeper.

[ddashenkov]

@arnej27959, thanks for the update. I'll try to come up with a working example within a few days, one which, hopefully, I'll be able to test on a newer version of Vespa to check if the NPE is gone for me.