FT-677: Allow fractal tree to run on filesystems with sector sizes other than 512 #385
Conversation
|
yes, gets rather invasive quickly. Probably why no-one has done this yet. https://github.com/grooverdan/PerconaFT/commit/918858efa82380d274907de92992c036fba33b92 (still WIP). Also vague in the direct IO read/write is if the block size is what memory needs to be aligned with or just the chunk size. |
|
@grooverdan |
|
As the alignment is a global constant, the simple way is that we add an api set_blocksize to the buildheader/db-4.6.19.h or setting in ydb.cc/env_open function. |
…her than 512 This uses the block size from the fstat call to determine the filesystem block size. SPDX-License-Identifier: BSD-2-Clause Signed-off-by: Daniel Black <[email protected]>
SPDX-License-Identifier: BSD-2-Clause Signed-off-by: Daniel Black <[email protected]>
SPDX-License-Identifier: BSD-2-Clause
… == file_size Seems to frequently hit this assertion now that the blocksize bas based on filesystem. SPDX-License-Identifier: BSD-2-Clause
…BLOCK_ALLOCATOR_HEADER_RESERVE Appeared consistent with other assertions. Worked around an unexpected failure on a 4k filesystem.
|
@BohuTANG thanks for the tips. Eventually got back to this. I've populated the cachefile structure with the blocksize of the filesystem from the fstat and added this blocksize to the API calls that need it. I haven't adjusted the test suite. I used a loopback filesystem created as follows: sudo mkfs.xfs -b size=4k -s size=2k -i size=1024 -f /dev/loop0
meta-data=/dev/loop0 isize=1024 agcount=4, agsize=131072 blks
= sectsz=2048 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0
data = bsize=4096 blocks=524287, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=2048 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Using gdb I determined 4k was being used as the blocksize. This is currently the upperlimit due to the assertion in toku_serialize_ft_to (<= BlockAllocator::BLOCK_ALLOCATOR_HEADER_RESERVE). I applied this set of patches to mariadb-10.3 branch and ran the tokudb suite: $ mysql-test/mtr --port-base=8000 --vardir=/mnt --parallel=4 --force --max-test-fail=40 --suite=tokudb ... tokudb.change_column_char_descriptor w2 [ pass ] 13779 tokudb.change_column_char_key w2 [ pass ] 13734 tokudb.change_column_char_null w2 [ pass ] 6613 tokudb.type_ranges w4 [ pass ] 154472 tokudb.type_uint w4 [ pass ] 4878 tokudb.change_column_char_rename w2 [ pass ] 181524 tokudb.change_column_delete_change_int_5674 w2 [ pass ] 4330 tokudb.change_column_delete_change_varchar_5674 w2 [ pass ] 3955 tokudb.type_varchar w4 [ pass ] 183003 tokudb.change_column_delete_change_char_5674 w4 [ pass ] 6573 tokudb.change_column_int_default w4 [ pass ] 48002 tokudb.change_column_int_descriptor w4 [ pass ] 16861 .... Some aborts and segfaults still to go through. |
This uses the block size from the fstat call to determine the filesystem block size.
Tested by running on XFS with 4096 BS.
I hope using SPDX-License-Identifier is acceptable as used in the Linux Kernel. Its a bit briefer. I can change if required however.