Extreme resource usage & hanging

[NightRa]

Using IPFS master, d48cd56
Context: I have about 7 GB of pinned files, about ~100kb each. (The ipfs blocks directory is 18GB in size), migrated from IPFS 0.3 to 0.4.

ipfs cat of a short text file I added just before doesn't seem to terminate after a few minutes of continuously reading ~400MB/s from an SSD where the IPFS blocks are stored.

I also noticed, that with smaller sized local ipfs block dirs (7 GB), freshly created on 0.4, ipfs cat does succeed, but after about ~5 seconds of full CPU usage. (it succeeds possibly because all the just added 7GB of data is cached in memory)

Seems like either a serious bug or a performance problem.

Question:
Does IPFS use a trie for accessing/indexing hashes?
If not, it definitely should.

[Kubuxu]

Looks like I have the same problem with #2405.

I can confirm that ipfs cat [some hash] also hangs using massive amount of disk bandwidth.

[whyrusleeping]

This is caused by is checking to see if we should run a gc before each cat command. We should remove that call for now until we rethink how it should work.

[alphaCTzo7G]

i was running the last example here: https://github.com/ipfs/examples/tree/master/examples/pinning

even though the file size is very small and has been garbage collected, ipfs cat just hangs for ever.. now its been ~30 mins..

[Stebalien]

Closing due to age.