Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chunk ByteBuffer to soon recycled #158

Closed
infeo opened this issue Mar 14, 2023 · 1 comment
Closed

Chunk ByteBuffer to soon recycled #158

infeo opened this issue Mar 14, 2023 · 1 comment
Assignees
Labels
Milestone

Comments

@infeo
Copy link
Member

infeo commented Mar 14, 2023

In PR #125, the class BufferPool was introduced to reduce buffer allocations and reuse already allocated buffers for de- and encryption. The affected buffers are part of "Chunk" records, which represent a contigous memory area in a file.

A buffer is recylced, when:

  • a chunk is saved to the ciphertextfile
  • a chunk is loaded from the ciphertextfile and
  • a chunk is evicted from the ChunkCache:
    private void removeChunk(RemovalNotification<Long, Chunk> removal) {
    try {
    chunkSaver.save(removal.getKey(), removal.getValue());
    bufferPool.recycle(removal.getValue().data());
    } catch (IOException e) {
    throw new UncheckedIOException(e);
    }
    }

The third point (recycle on eviction) causes a severe bug. Imagine the following scenario:

  • Small size of chunk cache (e.g., 5)
  • Great number of concurrent reading threads (e.g., 100)
  • File spanning over several chunks
  • Random reading pattern leading to a low chunk cache hit rate

Low hit rate -> High number of cache evictions -> High recylce rate. But only because a chunk is evicted does not imply, the thread reading on the chunk has finished its operation. If the recycled buffer is selected to be filled again, the (still not finished) thread is reading wrong content.

@infeo infeo added the bug label Mar 14, 2023
@infeo infeo added this to the 2.6.2 milestone Mar 14, 2023
@infeo infeo self-assigned this Mar 14, 2023
@infeo
Copy link
Member Author

infeo commented Mar 15, 2023

To recycle the data buffer of Chunks, a synchronization mechanism is needed which ensures:

  1. Once the chunk is evicted from cache, no new accesses to data can be made
  2. the chunk keeps track of accesses
  3. Only if the last access is "closed", the buffer is recycled

This would add another layer of complexity, hence, for now we decided to keep it simple and just synchronize threads on methods in CleartextFileChannel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant