Zed JS #1603

jameskerr · 2021-05-03T21:00:18Z

The full zed type system in JavaScript and all the implications of the new apis in the app.

The Zed type system is located in zealot/zed. There are classes for each type and for each value. The context classes handles decoding and encoding into and from zjson.
Each process in the app will use a singleton ZealotContext to decode and encode any Zed data it sees.
A "responses" system has been created to make it easy to re-generate zqd responses if things change again in the future. See test/responses/index.ts
The log cell renderer has been updated and simplified to render complex fields like Arrays and Sets. See app/viewer/value.tsx and app/viewer/cell.tsx.
The rightClick menu build prop drilling is gone, in favor of a simple dispatched thunk 🎉. (Some dangling code probably needs to be deleted)
The contextMenu integration test has been refactored and updated.
The createCell object has been removed and replaced with other simpler functions.
Record serialization now looks like ZealotContext.encodeRecord(r) and ZealotContext.decodeRecord(zjson)
Encoding a single field is ZealotContext.encodeField(f) and ZealotContext.decodeField(f)

Happy to answer any other questions as the come up!

mccanne

I don't understand all the react etc pieces, but the zealot/zed stuff looks good to me. At some point, we'll want more comprehensive treatment of vaulues (e.g., 64-bit. precision etc)

mccanne · 2021-05-04T18:18:03Z

app/core/formatters/format-zed.ts

+  if (zed.isInt(data)) return withCommas(data.toString())
+  if (zed.isTime(data)) return brim.time(data.toDate()).format()
+  if (zed.isStringy(data)) {
+    // only whitespace


This isn't doing anything...

mccanne · 2021-05-04T18:29:39Z

ppl/detail/flows/get-correlation-query.test.ts

-    null
-  ]
-  const record = new zng.Record(type, value)
+  const record = createRecord({


Did createRecord() already exist? I think the zed library would have this as a helper method, e.g., CreateRecordFromObject() takes an javascript object, normalizes the keys into an deterministic column order, and looks up the zed.TypeRecorcd, etc.

That's a good idea.

mccanne · 2021-05-04T18:38:03Z

test/factories/zed-factory.ts

+
+export function createRecord(object): zed.Record {
+  let fields: zed.Field[] = []
+  for (let name in object) {


Should you normalize the column order or does javascript semantics these days do the right thing? The main thing is to look up the same zng type no matter what the column order is.

Most javascript engines keep the order, but it's not a guarantee. I'd like to normalize the order as you suggest eventually. When we move it into the zed package, that would be a good time. This is just a helper function used in unit tests.

mccanne · 2021-05-04T18:40:16Z

test/fixtures/zjson-types.ts

+
+export const STRING = {kind: "primitive", name: "string"} as PrimitiveType
+export const TIME = {kind: "primitive", name: "time"} as PrimitiveType
+export const COUNT = {kind: "primitive", name: "count"} as PrimitiveType


There is no primitive in Zed called "count". With zeek import, the count type is turned to uint64.

Thanks, this whole file was left over from old times. I'll delete.

mccanne · 2021-05-04T18:40:58Z

test/fixtures/zjson-types.ts

+export const STRING = {kind: "primitive", name: "string"} as PrimitiveType
+export const TIME = {kind: "primitive", name: "time"} as PrimitiveType
+export const COUNT = {kind: "primitive", name: "count"} as PrimitiveType
+export const INTERVAL = {kind: "primitive", name: "interval"} as PrimitiveType


Same... no "interval" primtiive.

mccanne · 2021-05-04T18:54:18Z

zealot/zed/context.ts

+import {Field} from "./values/field"
+
+export class ZedContext {
+  private id = 23


I didn't think you needed 23 since it doesn't appear anywhere in the ZJSON protocol, unless I messed up?

This is arbitrary. It was convenient to have the same starting id as the backend for testing.

mccanne · 2021-05-04T18:56:01Z

zealot/zed/context.ts

+    //                 table, and return it. If it is not, it is an alias. Look it up in the
+    //                 context's typeAliases table, and return it.
+
+    switch (obj.kind) {


This looks great! How'd you figure out how to structure this? :-)

mccanne · 2021-05-04T18:56:33Z

zealot/zed/primitive.ts

@@ -0,0 +1,54 @@
+// export class zed.Primitive {


Why is this here all commented out?

Forgot to clean that up. It was a reference to the old code. Deleted it.

mccanne · 2021-05-04T18:57:27Z

zealot/zed/types/type-def.ts

+
+  create(value) {
+    console.log(value)
+    throw new Error("Im here!")


This was not used, TypeAlias is the class is actually used for this. Deleted

mccanne · 2021-05-04T18:59:21Z

zealot/zed/values/int64.ts

+
+  toInt() {
+    if (isNull(this.value)) return null
+    return parseInt(this.value)


This doesn't provide 64-bit resolution

I can use BigInt(this.value) to handle the big ints. I tried just now, but it breaks the histogram somehow when trying to sum numbers to gather. I'll follow this PR up with a fix for the 64 bits soon.

mccanne · 2021-05-04T19:01:33Z

I'm deferring to @mason-fish for the approval button.

mason-fish

This lgtm! Left a few comments but am mostly curious about dropping the timestamp format in brimcap. Awesome work wrangling all this together!!

mason-fish · 2021-05-04T19:09:17Z

app/viewer/value.tsx

+`
+const pad = (bool) => (bool ? <Space /> : null)
+
+export default function Value(props: ValueProps) {


mason-fish · 2021-05-04T19:17:57Z

plugins/brimcap/brimcap-plugin.ts

+    //   "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSS'Z'"
+    // )
+
+    const tsString = ts.toString()


Will this still preserve nano second precision? I threw in this js-joda stuff here to help with the formatting since brimcap explicitly expects RFC3999nano

Yes, the time values from zjson are now represent as RFC39999nano strings. I moved the js-joda code in the zed.Time class because I need to parse them there.

mason-fish · 2021-05-04T19:19:01Z

plugins/brimcap/brimcap-plugin.ts

+    const dur = log.get("duration") as zed.Duration
+    const dest = join(
+      this.api.getTempDir(),
+      `packets-${ts.toString()}.pcap`.replaceAll(":", "_")


Is this the fix for Windows you mentioned?

mason-fish · 2021-05-04T19:22:50Z

ppl/menus/detailFieldContextMenu.ts

-  ): MenuItemConstructorOptions[] {
-    const isTime = field.data.getType() === "time"
+export default function detailFieldContextMenu({field, record, value}) {
+  return (_, getState) => {


context menus as thunks ftw 🔥

mason-fish · 2021-05-04T19:23:45Z

rollup.config.js

@@ -1,5 +1,6 @@
 import typescript from "@rollup/plugin-typescript"
 import commonjs from "@rollup/plugin-commonjs"
+import nodeResolve from "@rollup/plugin-node-resolve"


I noticed this in the dependency changes too, what is it for?

This is to pull in the js-joda library into the zealot bundle. We previously had no external dependencies.

mason-fish · 2021-05-04T19:26:13Z

scripts/rename.js

@@ -0,0 +1,22 @@
+const {Project} = require("ts-morph")


Did this sneak in as part of your file renaming PR?

Will delete.

mason-fish · 2021-05-04T19:27:31Z

src/js/components/LogRow.tsx

-          log={log}
-          style={{width}}
-        />
+        <Cell width={width} key={key} name={field.name}>


mason-fish · 2021-05-04T19:31:19Z

src/pkg/brimcap/index.ts

@@ -0,0 +1,5 @@
+function load() {}


Is this just some scaffolding from our convos about the brimcap loaders? 🤔

You are correct. Will delete.

mason-fish · 2021-05-04T19:32:00Z

test/data/count-by-typeof.zson

@@ -0,0 +1,2 @@
+{ first_name: "James" } (=person_schema)
+{ score: 100 }


nwt · 2021-05-04T19:47:37Z

app/search/flows/next-page-viewer-search.ts

-      const ts = logs[index].get("ts") as zng.Primitive
-      if (ts.getType() == "time") {
+      const ts = logs[index].try<zed.Time>("ts")
+      if (ts && ts instanceof zed.Time) {


Would

if (ts instanceof zed.Time) {`

suffice here or is the ts && necessary?

You are right, it is not needed. Sweet.

philrz · 2021-05-04T20:09:19Z

I happened to just try testing a pcap import on this branch (so, commit 6d3f120 at the moment) and it looks like the colored tiles for the different Zeek/Suricata record types have stopped working. @jameskerr is this expected?

This reverts commit 1df4971.

jameskerr · 2021-05-04T21:22:59Z

Thanks for pointing that out. I've put the colors back in.

jameskerr · 2021-05-04T21:28:11Z

Thanks so much for the reviews everyone. Glad to have this merged. When tests pass, I will merge. But will follow it right up with a proper "duration" parsing function.

philrz · 2022-08-30T18:37:21Z

src/js/brim/primitiveCell.ts

-    display() {
-      if (value === "(empty)") {
-        return ""


I'm just going back in time to recognize that this looks like what fixed #832.

jameskerr added 29 commits April 14, 2021 16:06

WIP

444d841

Updated to most recent branch

3022764

Work in progress

1aeaebf

Zed is done

d2b3edf

No typescript errors

a7b447e

Tests green

d144e80

Removed most of cell.ts only need to guess width

95db2db

Started zed types implementation

918dc13

got a test passing

b4de42d

Snapshot

afacc2d

20 errors left

f362eda

Zero errors

26a9d42

Fixed viewer-search test

ebd916d

Need to rebase

7c6b089

Merge branch 'main' into zed-js-draft

587e59d

TS Errors Fixed

0378309

All tests pass

8658ad1

Fixed api tests

068f037

Guessing widths again

6d8d196

Working on itests

ba73876

Fixed ingest itest

0642a34

Fixed query test

d4b768c

Refactor context menu integration tests

2170253

Fix pcap integration tests

59ce8ff

Fixed unit tests

3234691

Fix lint

58b7248

Added array renderer

172a691

Merge branch 'main' into zed-js

486fdf3

Update zed branch

23e3c62

jameskerr requested a review from mccanne May 4, 2021 00:06

jameskerr requested a review from mason-fish May 4, 2021 00:06

jameskerr added 4 commits May 3, 2021 17:13

Clean up straggler files

f9c49fb

No colons in pcap names

2fc56db

Update zed to master

19545f9

Merge remote-tracking branch 'origin/main' into zed-js

6d3f120

mccanne reviewed May 4, 2021

View reviewed changes

mason-fish approved these changes May 4, 2021

View reviewed changes

nwt reviewed May 4, 2021

View reviewed changes

jameskerr added 11 commits May 4, 2021 13:13

Cleanup commented out code

c370cba

Remove unused test file

b5f8166

Remove old reference files

49f45cc

Remove unused file

23274d2

Handle 64 bit integers

1df4971

Delete old script

788b991

Remove old scaffold file

ed67242

Remove extra null check

5845d88

Revert "Handle 64 bit integers"

badbe8b

This reverts commit 1df4971.

Removed unused file

dc21f59

Put the zeek and suricata colors back in

914e47c

jameskerr merged commit f7dabb8 into main May 4, 2021

jameskerr deleted the zed-js branch May 4, 2021 21:50

philrz mentioned this pull request Jun 25, 2021

Result of typeof(this) is presented as "[object Object]" #1725

Closed

philrz mentioned this pull request Sep 16, 2021

Braces/brackets no longer around sets & arrays #1182

Closed

philrz reviewed Aug 30, 2022

View reviewed changes

philrz mentioned this pull request Aug 30, 2022

Brim hides the string value "(empty)" #832

Closed

		@@ -0,0 +1,2 @@
		{ first_name: "James" } (=person_schema)
		{ score: 100 }

Zed JS #1603

Zed JS #1603

Conversation

jameskerr commented May 3, 2021 • edited by philrz Loading

mccanne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mccanne commented May 4, 2021

mason-fish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philrz commented May 4, 2021

jameskerr commented May 4, 2021

jameskerr commented May 4, 2021

Choose a reason for hiding this comment

jameskerr commented May 3, 2021 •

edited by philrz

Loading