Presentation of complex data types in Brim app

[philrz]

This issue was originally written in the context of some specific examples of complex security data that were not well-presented in the app. Given the new focus toward being able to "shape" arbitrary data, we've recognized we should up-level this to a more holistic/ambitious design effort about how we'll present any complex data types in the app, such as in the context of shaping.

The original examples are preserved below for completeness.

---

Some of the newest ZQL functionality reveals limitations of how Brim currently visualizes data. The following repro is with Brim commit 4a1c2bd which talks to zqd commit 0759cc3. The test data is https://www.malware-traffic-analysis.net/2020/09/02/2020-09-01-Emotet-epoch-3-infection-with-Trickbot-gtag-mor119.pcap.zip (password: infected).

Consider the following aggregation:

Zooming in on one of the rows in Log Detail:

What's not evident from this output is easier to see if we look at the TZNG representation of the same.

$ zq -t '_path=http | count() by status_code,host | counts=collect(cut(status_code,count)) by host' all.zng 
#0:record[host:bstring,counts:array[record[status_code:uint64,count:uint64]]]
0:[45.230.228.26;[[200;2;]]]
0:[-;[[200;9;]]]
0:[118.110.236.121;[[200;3;]]]
0:[195.123.242.119;[[200;4;]]]
0:[t-privat.de;[[200;1;]]]

As you can see, counts is actually an array of records, but this is not evident from the output. (Incidentally, this topic of visual cues for the complex ZNG types is also touched on in #1182 where we lost the brackets we once had around sets & arrays.)

Perhaps also related, we've known for some time that we could perhaps stand to reveal the hierarchical nature of records, such as the Zeek id fields that we currently always flatten.

[philrz]

We recently had a discussion with a community user who described a desire to read JSON objects such as those returned from REST API calls. These could be arbitrarily complex ("up to 15 depth" was one way they were described) and hence may not lend themselves to the structured row/column output formats we largely use today. I have an action item to get a sanitized version of their JSON and more use case info. I'm just adding this comment as a reminder to perhaps share drafts of designs with that user to see if it fits how they want query-time presentation to look for this data that started life as such JSON.

[philrz]

Another example we'll want to be sure to address is visible in the verification notes at #1576 (comment) Notice how the name of the object is not shown in the Log Detail.

[philrz]

To add one more to the pile, as noted in brimdata/super#2989 (comment), map values are presented pretty crudely at the moment. In Brim 121f2ea, here's an example map value that's returned from the URL parsing function:

[philrz]

This was addressed some time ago in #1981. As verification, I'll revisit the examples cited in this issue and to confirm they're all well-handled by the inspector. Verifications are done with Brim commit 55d1fdc.

For the example from the Emotet pcap that kicked off the issue, the Inspector shows the hierarchical structure and values. While confirming this, I noticed that the Table view still handles this improperly, so I've opened #2453 to track that.

For the community user with the JSON objects "up to 15 depth", I never did hear back from them with specific test data. However, in the time since, we've now got the zq tutorial that walks through operations with the prs.json test data that's similar in nature. The attached video shows the successful navigation of this data in the Inspector view.

Verify.mp4

For the empty values mentioned #1576, here's that example test data along with some similar ones:

$ cat empty.ndjson 
{"empty_object": {}}
{"null_value": null}
{"empty_array": []}

And here they are rendered successfully in Inspector, though #2454 has been opened to track a new problem this reveals:

Here's the rendering of the map values shown in brimdata/super#2989:

Thanks @jameskerr!