Goofys memory optimizations

[romilbhardwaj]

This PR changes the the goofys version to a different branch that cuts goofys memory utilization by half. Also adds a stress test for goofys.

Goofys memory utilization after ls-ing 1M files:
kahing/goofys/latests: 5.3GB, OOM after 8.9M files
romilbhardwaj/goofys/upstream: 2.1GB, OOM after 16.1M files

This somewhat helps with #706, but still OOMs.

Tested:

• tests/stress/mountedstorage/mount_stress.yaml

[concretevitamin]

Are there correctness concerns (losing or corrupting data?) with the new branch? Suppose it’s ok if it passes their test suite.

[romilbhardwaj]

Yep, it passed the tests in their Makefile

[Michaelvll]

Wow, we can build our own binary. Then maybe we can build our own binary for macOS as well to solve the installation problem we met before?

[Michaelvll]

For reference, I tried this branch with the following codes adapted from the code provided by @romilbhardwaj from #622 doc.

It seems read-only mode has no effect to memory usage.

Benchmarks on an m4.2xlarge:

wandb panel	File listing speed	Mem Increase (GB) / M files
bbe678b (romil’s branch)	90 K /s	~3
Read-only option (-o ro)	90 K /s	~3

import os
import psutil
import wandb

wandb.init(project='goofys-profile')

path = '/data_ro'

file_cnt = 0
for root, dirs, files in os.walk(path):
    for file in files:
        fp = os.path.join(root, file)
        if file_cnt % 100000 == 0:
            wandb.log({'num_files': file_cnt, 'mem_used_gb': psutil.virtual_memory().used / 2**30})
        file_cnt += 1

[Michaelvll]

Thank you for adding this! It mitigates the OOM problem. LGTM. A small nit: the latest master branch added a -o allow_others option to the goofys mounting cmd. (Updated the evaluation with the master merged, and -o allow_others does not affect the performance).

[romilbhardwaj]

That's an awesome benchmark! Thanks @Michaelvll - I'll add benchmark and script to the issue (#706) so we can refer to it in the future.