How to get the path for objects filtered using Pick in streamValues?

[dhwaneetbhatt]

Hello, I have the following object:

{
  "id": "11937b58-4561-466b-8d48-ea2a519747f9",
  "name": "Person XYZ",
  "optionalKey1": "value1",
  "optionalKey2": "value2",
  "optionalKey3": {
    "key": "val"
  },
  "bigArray": []
}

My use case is to stream this JSON and retain everything except bigArray because, for large objects,bigArray is usually 99% of the object size.

I tried the following way:

const obj = {};

const pipeline = StreamChain.chain([
  fs.createReadStream('sample.json'),
  Pick.withParser({
    filter: (stack) => {
      return stack.length > 1 && stack[0] !== 'bigArray';
    },
  }),
  StreamValues.streamValues(),
]);

pipeline.on('data', (data) => (obj[data.key] = data.value));

The data emitted does not retain the actual path of the value in the original JSON, something like optionalKey3.key when val is emitted. Due to this, I'm not able to re-create the original JSON.

Since the JSON can have optional keys too, it is not possible for me to map back a fixed set of values using the order of data emitted.

I tried using streamObject without using the Pick because the resulting Assembler has the key, but it seems like using objectFilter is not as efficient as using a filter.

// this works, but may not be as efficient as using filter?
const pipeline = StreamChain.chain([
  fs.createReadStream('sample.json'),
  StreamObject.withParser({
    objectFilter: (asm) => {
      return asm.key !== 'bigArray';
    }
  }),
]);

I tried both approaches and here are the results. The file I'm trying this on is a 375 MB file.

// Approach 1 (filter + streamValues)
time: 38.656s
{
  rss: '176.77 MB -> Resident Set Size - total memory allocated for the process execution',
  heapTotal: '124.04 MB -> total size of the allocated heap',
  heapUsed: '95.25 MB -> actual memory used during the execution',
  external: '1.12 MB -> V8 external memory'
}

// Approach 2 (streamObject with objectFilter)
time: 42.733s
{
  rss: '3789.88 MB -> Resident Set Size - total memory allocated for the process execution',
  heapTotal: '3723.37 MB -> total size of the allocated heap',
  heapUsed: '3634.03 MB -> actual memory used during the execution',
  external: '1.12 MB -> V8 external memory'
}

So it definitely makes sense that using filter is efficient, but not being able to get the path of the value forces me to use streamObject. Is there any way to get the key/path of the value when using Pick? Or am I understanding things wrong?

[uhop]

My understanding is that you want to preserve the overall shape of your objects. Pick picks subobjects, so the parent shape is lost obviously. To edit a token stream preserving the shape of original objects we have three tools:

1. Filter — white-lists what properties should be preserved. The rest will be removed.
2. Ignore — black-lists what properties should be removed. The rest will be passed through.
3. Replace — replaces properties with different (default) values. It is rarely used but it is a foundation for Ignore.

Did you try to ignore the field in question?

[dhwaneetbhatt]

@uhop Thanks a bunch, ignore worked for me. Guess I should've read the entire wiki before troubling the community here with questions.