default reduced to half of all logical CPUs

NEWS.md

@@ -8,6 +8,8 @@
 
 2. `foverlaps` supports `type = "equal"` now. Closes [#3416](https://github.com/Rdatatable/data.table/issues/3416). Also addresses part of [#3002](https://github.com/Rdatatable/data.table/issues/3002).
 
+3. The number of logical CPUs used by default has been reduced from 100% to 50%. The previous 100% default was reported to cause significant slow downs when other non-trivial processes were also running: [#3395](https://github.com/Rdatatable/data.table/issues/3395), [#3298](https://github.com/Rdatatable/data.table/issues/3298). Two new optional environment variables (`R_DATATABLE_NUM_PROCS_PERCENT` & `R_DATATABLE_NUM_THREADS`) control this default. \code(setDTthreads()) gains \code{percent=} and \code{?setDTthreads} has been significantly revised. \code{getDTthreads(verbose=TRUE)} has been expanded. The environment variable `OMP_THREAD_LIMIT` is now respected ([#3300](https://github.com/Rdatatable/data.table/issues/3300)) in addition to `OMP_NUM_THREADS` as before.
+
 #### BUG FIXES
 
 1. `rbindlist()` of a malformed factor missing levels attribute is now a helpful error rather than a cryptic error about `STRING_ELT`, [#3315](https://github.com/Rdatatable/data.table/issues/3315). Thanks to Michael Chirico for reporting.
@@ -44,15 +46,9 @@
 
 6. v1.12.0 did not compile on Solaris 10 using Oracle Developer Studio 12.6, [#3285](https://github.com/Rdatatable/data.table/issues/3285). Many thanks to Prof Ripley for providing and testing a patch. For future reference and other package developers, a `const` variable should not be passed to OpenMP's `num_threads()` directive otherwise `left operand must be modifiable lvalue` occurs.
 
-7. `getDTthreads()` respected the `OMP_NUM_THREADS` environment variable but not `OMP_THREAD_LIMIT`, [#3300](https://github.com/Rdatatable/data.table/issues/3300). There are two very similar OpenMP functions: `omp_get_max_threads()` and `omp_get_thread_limit()`. It now calls both and chooses the minimum. Note that these environment variables should be set before the R session starts. Using the R command `Sys.setenv()` to set them is too late because the OpenMP runtime is already running by then; use `setDTthreads()` instead.
-
-8. `foverlaps` provides clearer error message when interval columns are of type factor. Closes [#2645](https://github.com/Rdatatable/data.table/issues/2645). Thanks to @sritchie73 for the report.
-
-9. `foverlaps` provides clearer error messages when interval columns have NA in them. Closes [#3007](https://github.com/Rdatatable/data.table/issues/3007). Thanks to @msummersgill for the report and code to integrate into foverlaps.R.
-
-10. `foverlaps` provides better error messages when one interval is POSIXct and other isn't. Also, it warns when POSIXct interval columns are not all of same timezone. Closes [#1143](https://github.com/Rdatatable/data.table/issues/1143). Thanks to @DavidArenburg for the report.
+7. `foverlaps` provides clearer error messages w.r.t. factor and POSIXct interval columns, [#2645](https://github.com/Rdatatable/data.table/issues/2645) [#3007](https://github.com/Rdatatable/data.table/issues/3007) [#1143](https://github.com/Rdatatable/data.table/issues/1143). Thanks to @sritchie73, @msummersgill and @DavidArenburg for the reports.
 
-11. From v1.12.0, `unique(DT)` checks up-front the types of all the columns and will fail if any column is type `list`. Use `unique(DT, by=...)` to specify columns that are not type `list`. v1.11.8 and before would also correctly fail with the same error, but not when uniqueness had been established in prior columns (it would stop early and not look at the `list` column). Checking up-front was necessary for some internal optimizations and it's probably best to be explicit anyway. Thanks to James Lamb for reporting, [#3332](https://github.com/Rdatatable/data.table/issues/3332). The error message has been embellished :
+8. From v1.12.0, `unique(DT)` checks up-front the types of all the columns and will fail if any column is type `list`. Use `unique(DT, by=...)` to specify columns that are not type `list`. v1.11.8 and before would also correctly fail with the same error, but not when uniqueness had been established in prior columns (it would stop early and not look at the `list` column). Checking up-front was necessary for some internal optimizations and it's probably best to be explicit anyway. Thanks to James Lamb for reporting, [#3332](https://github.com/Rdatatable/data.table/issues/3332). The error message has been embellished :
     ```
     Column 2 of by= (2) is type 'list', not yet supported. Please use the by= argument to specify columns with types that are supported.
     ```

R/onAttach.R

@@ -17,7 +17,8 @@
       g = ""
   }
   dev = as.integer(v[1L, 3L]) %% 2L == 1L  # version number odd => dev
-  packageStartupMessage("data.table ", v, if(dev)paste0(" IN DEVELOPMENT built ",d,g), "  Latest news: r-datatable.com")
+  packageStartupMessage("data.table ", v, if(dev)paste0(" IN DEVELOPMENT built ",d,g),
+                        " using ", getDTthreads(verbose=FALSE), " threads (see ?getDTthreads).  Latest news: r-datatable.com")
   if (dev && (Sys.Date() - as.Date(d))>28)
     packageStartupMessage("**********\nThis development version of data.table was built more than 4 weeks ago. Please update: data.table::update.dev.pkg()\n**********")
   if (!.Call(ChasOpenMP))

R/openmp-utils.R

@@ -1,8 +1,16 @@
-setDTthreads <- function(threads=0, restore_after_fork=NULL) {
-  invisible(.Call(CsetDTthreads, as.integer(threads), restore_after_fork))
+setDTthreads <- function(threads=NULL, restore_after_fork=NULL, percent=NULL) {
+  if (!missing(percent)) {
+    if (!missing(threads)) stop("Provide either threads= or percent= but not both")
+    if (length(percent)!=1) stop("percent= is provided but is length ", length(percent))
+    percent=as.integer(percent)
+    if (is.na(percent) || percent<2L || percent>100L) stop("percent==",percent," but should be a number between 2 and 100")
+    invisible(.Call(CsetDTthreads, percent, restore_after_fork, TRUE))
+  } else {
+    invisible(.Call(CsetDTthreads, threads, restore_after_fork, FALSE))
+  }
 }
 
-getDTthreads <- function(verbose=getOption("datatable.verbose", FALSE)) {
+getDTthreads <- function(verbose=getOption("datatable.verbose")) {
   .Call(CgetDTthreads, verbose)
 }
 

R/test.data.table.R

@@ -59,7 +59,6 @@ test.data.table <- function(verbose=FALSE, pkg="pkg", silent=FALSE, with.other.p
   assign("nfail", 0L, envir=env)
   assign("ntest", 0L, envir=env)
   assign("whichfail", NULL, envir=env)
-  setDTthreads(2) # explicitly limit to 2 so as not to breach CRAN policy (but tests are small so should not use more than 2 anyway)
   assign("started.at", proc.time(), envir=env)
   assign("lasttime", proc.time()[3L], envir=env)  # used by test() to attribute time inbetween tests to the next test
   assign("timings", data.table( ID = seq_len(9999L), time=0.0, nTest=0L ), envir=env)   # test timings aggregated to integer id
@@ -75,7 +74,6 @@ test.data.table <- function(verbose=FALSE, pkg="pkg", silent=FALSE, with.other.p
   options(oldverbose)
   options(oldenc)
   # Sys.setlocale("LC_CTYPE", oldlocale)
-  setDTthreads(0)
   ans = env$nfail==0
 
   if (is.na(orig__R_CHECK_LENGTH_1_LOGIC2_)) {
@@ -95,12 +93,14 @@ test.data.table <- function(verbose=FALSE, pkg="pkg", silent=FALSE, with.other.p
   whichfail = get("whichfail", envir=env)
 
   # Summary. This code originally in tests.Rraw and moved up here in #3307
+  # One big long line because CRAN checks output last 13 lines. One long line counts as one out of 13.
   plat = paste0("endian==", .Platform$endian,
                 ", sizeof(long double)==", .Machine$sizeof.longdouble,
                 ", sizeof(pointer)==", .Machine$sizeof.pointer,
                 ", TZ=", suppressWarnings(Sys.timezone()),
                 ", locale='", Sys.getlocale(), "'",
-                ", l10n_info()='", paste0(names(l10n_info()), "=", l10n_info(), collapse="; "), "'")
+                ", l10n_info()='", paste0(names(l10n_info()), "=", l10n_info(), collapse="; "), "'",
+                ", getDTthreads()='", paste0(capture.output(invisible(getDTthreads(verbose=TRUE))), collapse="; "), "'")
   DT = head(timings[-1L][order(-time)],10)   # exclude id 1 as in dev that includes JIT
   if ((x<-sum(timings[["nTest"]])) != ntest) {
     warning("Timings count mismatch:",x,"vs",ntest)  # nocov

inst/tests/tests.Rraw

@@ -11968,7 +11968,7 @@ test(1894.14, DT[, sum(y)*..z], error="Variable 'z' is not found in calling scop
 ..z = 4L
 test(1894.15, DT[, sum(y)*..z], 60L)  # 'manual' prefix (we have recommended in the past) still works, for now.
 
-test(1895, getDTthreads(verbose=TRUE), output="omp_get_max_threads.*omp_get_thread_limit.*DTthreads")
+test(1895, getDTthreads(verbose=TRUE), output="num_procs.*R_DATATABLE.*thread_limit.*RestoreAfterFork")
 
 # Non ascii missing protects on ENC2UTF8; issue #2674
 utf8_strings = c("\u00e7ile", "fa\u00e7ile", "El. pa\u00c5\u00a1tas", "\u00a1tas", "\u00de")
@@ -13583,6 +13583,32 @@ qcall = quote(b := .(.Primitive("sum")(1, 2))) # such calls can be issued by dca
 d = data.table(a=1L)
 test(1996.2, d[, eval(qcall)], data.table(a=1L, b=3))
 
+# setDTthreads; #3435
+test(1997.01, setDTthreads(NULL, percent=75), error="Provide either threads= or percent= but not both")
+test(1997.02, setDTthreads(1L, percent=75), error="Provide either threads= or percent= but not both")
+test(1997.03, setDTthreads(-1L), error="must be either NULL or a single integer >= 0")
+test(1997.04, setDTthreads(percent=101), error="should be a number between 2 and 100")
+test(1997.05, setDTthreads(percent=1), error="should be a number between 2 and 100")
+test(1997.06, setDTthreads(percent=NULL), error="but is length 0")
+test(1997.07, setDTthreads(percent=1:2), error="but is length 2")
+test(1997.08, setDTthreads(restore_after_fork=21), error="must be TRUE, FALSE, or NULL")
+old = getDTthreads()
+oldenv = Sys.getenv("R_DATATABLE_NUM_PROCS_PERCENT")
+Sys.setenv(R_DATATABLE_NUM_PROCS_PERCENT="3.0")
+test(1997.09, setDTthreads(), old, warning="Ignoring invalid.*Please remove any.*not a digit")
+test(1997.10, getDTthreads(), old)
+Sys.setenv(R_DATATABLE_NUM_PROCS_PERCENT="1")
+test(1997.11, setDTthreads(), old, warning="Ignoring invalid.*integer between 2 and 100")
+test(1997.12, getDTthreads(), old)
+Sys.setenv(R_DATATABLE_NUM_PROCS_PERCENT="75")
+test(1997.13, setDTthreads(), old)
+new = getDTthreads()
+setDTthreads(percent=75)
+test(1997.14, getDTthreads(), new)
+Sys.setenv(R_DATATABLE_NUM_PROCS_PERCENT=oldenv)
+test(1997.15, setDTthreads(old), new)
+test(1997.16, getDTthreads(), old)
+
 
 ###################################
 #  Add new tests above this line  #

man/openmp-utils.Rd

@@ -3,24 +3,31 @@
 \alias{getDTthreads}
 \title{ Set or get number of threads that data.table should use }
 \description{
-Set and get number of threads to be used in \code{data.table} functions that are parallelized with OpenMP. Default value 0 means to utilize all CPU available with an appropriate number of threads calculated by OpenMP. \code{getDTthreads()} returns the number of threads that will be used. This affects \code{data.table} only and does not change R itself or other packages using OpenMP.  We have followed the advice of section 1.2.1.1 in the R-exts manual: "\ldots or, better, for the regions in your code as part of their specification\ldots num_threads(nthreads)\ldots That way you only control your own code and not that of other OpenMP users."  All the parallel region in data.table contain this directive. This is mandated by a \code{grep} in data.table's quality control CRAN release procedure script.
+  Set and get number of threads to be used in \code{data.table} functions that are parallelized with OpenMP.
 }
 \usage{
-setDTthreads(threads = 0, restore_after_fork = NULL)
-getDTthreads(verbose = getOption("datatable.verbose", FALSE))
+  setDTthreads(threads = NULL, restore_after_fork = NULL, percent = NULL)
+  getDTthreads(verbose = getOption("datatable.verbose"))
 }
 \arguments{
-  \item{threads}{ An integer >= 0. Default 0 means use all CPU available and leave the operating system to multi task. }
+  \item{threads}{ NULL (default) rereads environment variables. 0 means to use all logical CPUs available. Otherwise a number >= 1 }
   \item{restore_after_fork}{ Should data.table be multi-threaded after a fork has completed? NULL leaves the current setting unchanged which by default is TRUE. See details below. }
-  \item{verbose}{ Display the value of some OpenMP settings, including the restore_after_fork internal option. }
+  \item{percent}{ If provided it should be a number betwen 2 and 100; the percentage of logical CPUs to use. } By default on startup this is data.table uses 50%. } 
+  \item{verbose}{ Display the value of relevant OpenMP settings plus the \code{restore_after_fork} internal option. }
 }
 \value{
-A length 1 \code{integer}. The old value is returned by \code{setDTthreads} so you can store that prior value and pass it to \code{setDTthreads()} again after the section of your code where you control the number of threads.
+  A length 1 \code{integer}. The old value is returned by \code{setDTthreads} so you can store that prior value and pass it to \code{setDTthreads()} again after the section of your code where you control the number of threads.
 }
 \details{
-  \code{data.table} automatically switches to single threaded mode upon fork (the mechanism used by \code{\link[parallel]{mclapply}} and the foreach package). Otherwise, nested parallelism would very likely overload your CPUs and result in much slower execution. As \code{data.table} becomes more parallel internally, we expect explicit user parallelism to be needed less often. The \code{restore_after_fork} option controls what happens after the explicit fork parallelism completes. It needs to be at C level so it is not a regular R option using \code{options()}. By default \code{data.table} will be multi-threaded again; restoring the prior setting of \code{getDTthreads()}. But problems have been reported in the past on Mac and Intel OpenMP libraries, whereas success has been reported on Linux. If you experience problems after fork, start a new R session and change the default behaviour by calling \code{setDTthreads(restore_after_fork=FALSE)} before retrying. Please raise issues on the data.table GitHub issues page.
+  \code{data.table} automatically switches to single threaded mode upon fork (the mechanism used by \code{\link[parallel]{mclapply}} and the foreach package). Otherwise, nested parallelism would very likely overload your CPUs and result in much slower execution. As \code{data.table} becomes more parallel internally, we expect explicit user parallelism to be needed less often. The \code{restore_after_fork} option controls what happens after the explicit fork parallelism completes. It needs to be at C level so it is not a regular R option using \code{options()}. By default \code{data.table} will be multi-threaded again; restoring the prior setting of \code{getDTthreads()}. But problems have been reported in the past on Mac with Intel OpenMP libraries whereas success has been reported on Linux. If you experience problems after fork, start a new R session and change the default behaviour by calling \code{setDTthreads(restore_after_fork=FALSE)} before retrying. Please raise issues on the data.table GitHub issues page.
 
-  Attempting to \code{setDTthreads()} to more than the number of logical CPUs is intended to be ineffective; i.e., \code{getDTthreads()} will still return the number of logical CPUs in that case. Further, there is a hard coded limit of 1024 threads (with warning when imposed) to prevent accidentally picking up the value \code{INT_MAX} (2 billion; i.e. unlimited) which we have seen returned by OpenMP's \code{omp_get_thread_limit()} in some cases.
+  The number of logical CPUs is determined by the OpenMP function \code{omp_get_num_procs()} whose meaning may vary across platforms and OpenMP implementations. \code{setDTthreads()} will not allow more than this limit. Neither will it allow more than \code{omp_get_thread_limit()} nor the current value of \code{Sys.getenv("OMP_THREAD_LIMIT")}. Note that CRAN sets \code{OMP_THREAD_LIMIT} to 2 and should always be respected; i.e., never change \code{OMP_THREAD_LIMIT} in a package to a value greater than 2. Some hardware allows CPUs to be removed and/or replaced while the server is running. If this happens, our understanding is that \code{omp_get_num_procs()} will reflect the new number of processors available. If a processor has been physically removed, or the logical processors reconfigured since data.table started, \code{setDTthreads(...)} will need to be called again by you before data.table will reflect the change. If you have such hardware, please let us know your experience via GitHub issues / feature requests.
+
+  Use \code{getDTthreads(verbose=TRUE)} to see the relevant environment variables, their values and the current number of threads data.table is using. For example, the environment variable \code{R_DATATABLE_NUM_PROCS_PERCENT} can be used to change the default number of logical CPUs from 50% to another value between 2 and 100. If you change these environment variables using `Sys.setenv()` after data.table and/or OpenMP has initialized then you will need to call \code{setDTthreads(threads=NULL)} to reread their current values. \code{getDTthreads()} merely retrieves the internal value that was set by the last call to \code{setDTthreads()}. \code{setDTthreads(threads=NULL)} is called when data.table is first loaded and is not called again unless you call it.
+
+  \code{setDTthreads()} affects \code{data.table} only and does not change R itself or other packages using OpenMP. We have followed the advice of section 1.2.1.1 in the R-exts manual: "\ldots or, better, for the regions in your code as part of their specification\ldots num_threads(nthreads)\ldots That way you only control your own code and not that of other OpenMP users." Every parallel regions in data.table contain a \code{num_threads(getDTthreads())} directive. This is mandated by a \code{grep} in data.table's quality control CRAN release procedure script.
+
+  \code{setDTthreads(0)} is the same as \code{setDTthreads(percent=100)}; i.e. use all logical CPUs, subject to \code{Sys.getenv("OMP_THREAD_LIMIT")}. Please note again that CRAN sets OMP_THREAD_LIMIT to 2, so never change \code{OMP_THREAD_LIMIT} in a CRAN package to a value greater than 2.
 }
 \keyword{ data }
 

src/data.table.h

@@ -151,6 +151,7 @@ double iquickselect(int *x, int n, int k);
 double wallclock();
 
 // openmp-utils.c
+void initDTthreads();
 int getDTthreads();
 void avoid_openmp_hang_within_fork();
 

src/fread.c

@@ -2001,9 +2001,10 @@ int freadMain(freadMainArgs _args) {
     {
       nth = omp_get_num_threads();
       if (me!=0) {
-        // should never happen
-        snprintf(internalErr, internalErrSize, "Internal error: Master thread is not thread 0 but thread %d.\n", me); // # nocov
+        // # nocov start
+        snprintf(internalErr, internalErrSize, "Internal error: Master thread is not thread 0 but thread %d.\n", me);
         stopTeam = true;
+        // # nocov end
       }
       myShowProgress = args.showProgress;
     }
@@ -2254,11 +2255,13 @@ int freadMain(freadMainArgs _args) {
       #pragma omp ordered
       {
         if (stopTeam) {             // A previous thread stopped while I was waiting my turn to enter ordered
-          myNrow = 0;               // discard my buffer
+          myNrow = 0;               // # nocov; discard my buffer
         }
         else if (headPos!=thisJumpStart) {
-          snprintf(internalErr, internalErrSize, "Internal error: invalid head position. jump=%d, headPos=%p, thisJumpStart=%p, sof=%p", jump, (void*)headPos, (void*)thisJumpStart, (void*)sof); // # nocov
+           // # nocov start
+          snprintf(internalErr, internalErrSize, "Internal error: invalid head position. jump=%d, headPos=%p, thisJumpStart=%p, sof=%p", jump, (void*)headPos, (void*)thisJumpStart, (void*)sof);
           stopTeam = true;
+          // # nocov end
         }
         else {
           ctx.DTi = DTi;  // fetch shared DTi (where to write my results to the answer). The previous thread just told me.

src/init.c

@@ -268,6 +268,7 @@ void attribute_visible R_init_datatable(DllInfo *info)
   sym_BY      = install(".BY");
   sym_maxgrpn = install("maxgrpn");
 
+  initDTthreads();
   avoid_openmp_hang_within_fork();
 }