Tuning for large compile task

[myc724]

We’ve deployed JuiceFS (version 0.11.0-37 2021-03-19 6127863) on the top of two Redis 6.2.2 (one for metadata and the one for object-storage) using “--io-threads 16" and “-io-threads-do-reads yes” but found that, for the following application, JuiceFS lagging behind a standard NFS (tier2) storage. After some tuning exercises, all knob tweaking (eg. smaller block-size, larger cache-size) seems only result incremental speedup. Our next step is to try Redis cluster (with at least 3 Sentinals to guarantee the atomicity of metadata operations) or TiKV. Is there any obvious thing that we can try before deep dive into distributed database solution? Thanks!

The application is to compile 140GB source code (1.2M files - avg. 113K per file) using 1250 CPUs and wrote 70GB object files (115K files - avg. 608KB per obj).

• NFS: 80 minutes.
• JuiceFS: 235 minutes.
• Two JuiceFS each has own servers (one for reading source code the other one for writing object): 200 minutes.

By switching source-code and object to various partitions and exhausting all combinations, we noticed that reading the 140GB source code is the bottleneck. Also, Redis config file seems implies that multiple threading might NOT help reading so much, so we’re looking into distributed database solutions. Please let us know if any obvious tuning that we can try. Thanks!

[xiaogaozi]

Hi @myc724, thanks for your feedback.

First of all, the v0.11.0 of JuiceFS is too old (latest version is v0.15.2), could you try the latest version?

we noticed that reading the 140GB source code is the bottleneck

You mean read the metadata or data? JuiceFS has a profile subcommand could help you profiling your case, please refer to document for more details.

Our next step is to try Redis cluster (with at least 3 Sentinals to guarantee the atomicity of metadata operations) or TiKV

If my understanding is correct, the "Redis cluster" you mentioned is this one. But JuiceFS relies on Redis transaction to guarantee atomicity, Sentinel seems helpless in this case?

TiKV metadata engine is ready now in main branch, you could try it immediately, please refer to document for more details.

[davies]

Is 10.15.24.254:1225 the address of your Redis instance?

It seems there is a performance issue on the Redis who stores data. How much memory it has? Is swap enabled in that machine?

[myc724]

Yes, it was the Redis server that I used for the object-store. The server has swap enabled. The memory consumption was around 70GB. But, the host machine has 3TB RAM (and 244 cores) and the memory usage was not near the threshold at all. Also, I ran Redis with 32 IO threads (read-write enabled), but "top" DID NOT show the system started thrashing at all. I guess ... if IO error came from 5 minutes timeout, I can only boost server performance. We're looking into TiKV now.

[davies]

Maybe Redis (for meta) + TiKV (for data) could be good choice.

[myc724]

Does "write inode:152655 error: input/output error" indicate the metadata server was also stressed (since recording inode should be handled by metadata server)?

[davies]

no, that means it failed to write data into Redis