Option to reduce IO impact of MUPIP REORG #233
Comments
nars1
added a commit
to nars1/YottaDB
that referenced
this issue
May 3, 2018
…SEC qualifier * A new field "reorg_sleep_nsec" is introduced in the database file header (sgmnt_data). Any MUPIP REORG process will check this value and if non-zero will sleep for the specified # of nanoseconds before processing every block in the database. This field can be changed with a MUPIP SET -REORG_SLEEP_NSEC= command which can be run while the reorg process is still running (i.e. the command does not require standalone access to the database file). This lets the user try out different sleeps in the same reorg process in case it runs for a long time. * DSE DUMP -FILE displays the current setting of reorg_sleep_nsec. * Because of the new field in the file header, the database minor version needed change. * And GTMDefinedTypesInit* files needed to be regenerated (to ensure PEEKBYNAME works with the new field). * Changed MICROSEC_IN_SEC -> MICROSECS_IN_SEC to be consistent with MILLISECS_IN_SEC and NANOSECS_IN_SEC. * Moved an assert about range of the input to the NANOSLEEP macro into the macro itself so it is centralized instead of duplicated in callers of this macro. * Simplified sleep.h by removing code corresponding to unsupported platforms.
nars1
added a commit
that referenced
this issue
May 4, 2018
…lifier * [#233] Reduce IO impact of MUPIP REORG with new -REORG_SLEEP_NSEC qualifier * A new field "reorg_sleep_nsec" is introduced in the database file header (sgmnt_data). Any MUPIP REORG process will check this value and if non-zero will sleep for the specified # of nanoseconds before processing every block in the database. This field can be changed with a MUPIP SET -REORG_SLEEP_NSEC= command which can be run while the reorg process is still running (i.e. the command does not require standalone access to the database file). This lets the user try out different sleeps in the same reorg process in case it runs for a long time. * DSE DUMP -FILE displays the current setting of reorg_sleep_nsec. * Because of the new field in the file header, the database minor version needed change. * And GTMDefinedTypesInit* files needed to be regenerated (to ensure PEEKBYNAME works with the new field). * Changed MICROSEC_IN_SEC -> MICROSECS_IN_SEC to be consistent with MILLISECS_IN_SEC and NANOSECS_IN_SEC. * Moved an assert about range of the input to the NANOSLEEP macro into the macro itself so it is centralized instead of duplicated in callers of this macro. * Simplified sleep.h by removing code corresponding to unsupported platforms.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Final Release Note
MUPIP -REORG_SLEEP_NSEC= specifies the number of nanoseconds that a MUPIP REORG process operating between blocks that it processes, with default value of 0 and a maximum of 999999999 (i.e. 999,999,999, or 1 nanosecond less than 1 second). Using non-zero values reduces the IO impact of MUPIP REORG, at the cost of increasing the duration of the operation. Note that the existing environment variable ydb_poollimit / gtm_poollimit is the appropriate technique to limit the impact of MUPIP REORG on global buffers; the -reorg_sleep_nsec can be used to limit the impact on the IO subsystem. (#233)
Description
MUPIP REORG can improve database layout and in turn performance. But it can slow down other concurrent users of the database (e.g. restarts) due to its heavyweight use of the database (reads and updates almost every block in the database file). Hence the desire to let the user control the speed of MUPIP REORG.
Draft Release Note
MUPIP SET -REORG_SLEEP_NSEC= on a database file specifies the number of nanoseconds that any MUPIP REORG process operating on that database sleeps before processing every block. The default value is 0. The maximum value of supported is 999,999,999 (i.e. 1 nanosecond less than 1 second).
The text was updated successfully, but these errors were encountered: