-
Notifications
You must be signed in to change notification settings - Fork 37
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[YDBTest#501] [DEBUG-ONLY] ydb_test_4g_db_blks env var allows creatio…
…n of databases with > 4Gib blocks Background ---------- * GT.M V7.0-000 added support for up to 16Gi blocks in a region (see YDBTest#501 for details). * Once the block number goes more than 4Gi, it becomes an 8-byte value. So GT.M had to change all usages of 4-byte block numbers in the code base (using `block_id` type) to 8-byte values. * YDBTest#501 had to test this change. But testing it was not straightforward as more than 4GiB blocks meant creating a 2 terabyte file even with the smallest block size of 512 bytes. With a default block size of 4KiB, it meant creating an 8 terabyte file. Although such huge files are sparse, the database structure implied a bitmap block every 512 blocks which meant the database file in the file system will be a huge file where only 1 in every 512 block is allocated. But even such a file would end up taking a lot of space and take a lot of time to create making it very difficult to run the entire suite of YDBTest tests with such huge database files. Solution -------- * Below is pasted from https://gitlab.com/YottaDB/DB/YDBTest/-/issues/501#note_1468633286. For the record. This is an idea to test block numbers greater than 4GiB (i.e. where the higher order 4-byte is non-zero) without the overhead of test runtime due to creating bitmaps (which happen every 512 blocks) in the database file. A debug-build only env var `ydb_test_4g_db_blks` will be honored by YottaDB. If this var is undefined or set to 0, then there is no change. But if this is set to a non-zero value, it indicates the size of the HOLE in the database starting from local bitmap number `1` (i.e. database block number `512`). Any database block (including bitmap blocks) in the range from block 512 to the bitmap block number specified by the env var will not be allocated/used by the database logic. This way we force the database logic to allocate blocks past the HOLE effectively getting into 8-byte block number territory without actually allocating HUGE files in the file system. The created database file will be a giant sparse file. * If this env var is set to the value `8388608`, the size of the database file with a default block size of 4KiB would be 16TiB (16 terabytes) and the database would support 4GiB blocks. Implementation -------------- * The value of the env var `ydb_test_4g_db_blks` is stored in the debug-only global variable `ydb_skip_bml_num` (defined in `sr_port/gbldefs.c`). This happens in `sr_port/gtm_env_init.c`. Since this is a new env var, there is a new line in `sr_port/ydb_logicals_tab.h`. * The YDBTest test system framework will be changed separately to run all existing tests with this env var set to the value `8388608`. * Various files had to be modified in this commit to ensure the database code does not use any block in the range from local bitmap block `512` upto (but not including) the local bitmap block number specified by `$ydb_test_4g_db_blks * 512` thus creating a big HOLE in the database file after block `511`. * Below is a high level list of the changes that happened. - MUPIP CREATE and MUPIP EXTEND create the HOLE as appropriate (mucregini.c, mu_cre_file.c, mu_int_maps.c, gdsfilext.c and gdsfilext_nojnl.c). They accurately maintains `total_blks` and `free_blks` even when there is a HOLE. Blocks in the HOLE are counted towards both these counters. - `bm_getfree.c` ensures any newly allocated blocks are not in the HOLE. - The following commands skip/honor/ignore the HOLE. - MUPIP REORG UPGRADE/DOWNGRADE (`sr_port/mu_reorg_upgrd_dwngrd.c`) - MUPIP REORG (in `mu_swap_blk.c` and `sr_unix/mu_swap_root.c`) - MUPIP REORG -TRUNCATE (`sr_unix/mu_truncate.c`) - MUPIP INTEG (in `sr_unix/mu_int_init.c`, `sr_port/mu_int_maps.c` and `sr_port/mupip_integ.c`) - MUPIP RESTORE (in `sr_unix/mupip_restore.c`) - DSE MAPS (in `sr_port/dse_maps.c`) - MUPIP BACKUP -INCREMENTAL (in `sr_unix/mubinccpy.c`) - DSE RANGE (in `sr_port/dse_range.c`) - DB_LSEEKREAD (in `sr_unix/gtmio.h`) and DB_LSEEKWRITE (in `sr_port/anticipatory_freeze.h`) macros now have asserts to ensure we never read/write any block in the HOLE. One exception was that the function `db_write_eof_block()` writes data at the end of the database file which could end up being in the HOLE in case the database has blocks 0 to 511 allocated presently. Therefore the asserts were avoided in that case by temporarily resetting `ydb_skip_bml_num` to 0. - `dsk_read.c` needed to handle read of a block in the HOLE region (possible due to concurrency issues). - `warn_db_sz.c` needed to not issue a LOWSPC warning when huge db files are in use. Or else one would see test failures due to these extra messages whenever the env var is randomly enabled by the tests. Bugs identified --------------- * Below was a pre-existing GT.M V7.0-000 code issue I noticed after the changes in this commit. The `mu_int_blks_to_upgrd` global variable counted the number of blocks but was typed as `int4`. It should have been changed to `block_id`. This got missed out in GT.M V7.0-000. And is fixed in this commit. Interestingly, the same fix happened on the GT.M side in V7.1-000. * Additionally, I noticed a few missing 4-byte to 8-byte conversions thanks to the changes in the above bullets. They are the following and are fixed in this commit to use `block_id` instead of `int4`/`uint4`. - `freeblks` local variable in `sr_port/mur_process_intrpt_recov.c` - `i` and `fcnt` local variables in `sr_port/mur_blocks_free.c` - Return value of `mur_blocks_free()` (which returns the number of free blocks and hence should be 8-byte)
- Loading branch information
Showing
28 changed files
with
349 additions
and
28 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.