Seeking leaks memory

[maxnoe]

Seeking on a zstandard stream reader causes the full file to be read into ram.

Minimal example:

$ head -c 500M /dev/urandom > test.dat
$ zstd test.dat

import zstandard as zstd
import os
import psutil

p = psutil.Process(os.getpid())
path = 'test.dat.zst'

reader = zstd.ZstdDecompressor().stream_reader(open(path, 'rb'))

chunk = reader.read(1)
while chunk:
    # seek 100 MB forward
    reader.seek(100 * 1024**2, 1)
    print(p.memory_info()[0] / 1024**2)
    chunk = reader.read(1)

Result:

114.1171875
214.1484375
314.1796875
414.2109375
514.2421875

If one only reads, only the currently read data is in RAM:

import zstandard as zstd
import os
import psutil

p = psutil.Process(os.getpid())
path = 'test.dat.zst'

reader = zstd.ZstdDecompressor().stream_reader(open(path, 'rb'))

chunk = reader.read(1)
while chunk:
    # seek one MB forward
    reader.read(100 * 1024**2)
    print(p.memory_info()[0] / 1024**2)
    chunk = reader.read(1)

Result

14.28515625
14.2890625
14.2890625
14.2890625
14.2890625

[indygreg]

This is definitely a bug. Thank you for reporting it.

[maxnoe]

Fixed in #83