mirai 1.3.0

mirai 1.3.0

New Features

• daemons(dispatcher = "thread") implements threaded dispatcher (experimental), a faster and more efficient alternative to running dispatcher in a separate process.
• mirai_map() adds [.progress_cli] as an alternative progress indicator, using the cli package to show % complete and ETA.
• daemons() gains argument 'force' to control whether further calls reset previous settings for the same compute profile.
• daemon() gains argument 'asyncdial' to allow control of connection behaviour independently of what happens when the daemon exits.

Behavioural Changes

• For daemons():
  • Argument 'dispatcher' now takes the character options 'process', 'thread' and 'none'. Previous values of TRUE/FALSE continue to be accepted (thanks @hadley #157).
  • Return value is now always an integer - either the number of daemons set if using dispatcher, or the number of daemons launched locally (zero if using a remote launcher).
  • Invalid type of ... arguments are now dropped instead of throwing an error. This allows ... containing unused arguments to be more easily passed from other functions.
• For mirai_map():
  • Now only performs multiple map over the rows of matrices and dataframes (thanks @andrewGhazi, #147).
  • Combining collection options is now easier, in the fashion of: x[.stop, .progress].
  • Collection options now work even if mirai is not on the search path e.g. mirai::mirai_map(1:4, Sys.sleep)[.progress].
• dispatcher() drops argument 'asyncdial' as it is rarely useful to set this here.
• everywhere() now errors if the specified compute profile is not yet set up, rather than fail silently.
• launch_local() and launch_remote() now strictly require daemons to be set, and will error otherwise.
• serial_config() now validates the arguments provided and returns them as a list. This means any saved configurations from previous package versions must be re-generated.

Updates

• Fixes daemons() to correctly handle a vector of URLs passed to 'url' again.
• Fixes flatmap with mirai_map()[.flat] assigning a variable 'typ' to the calling environment.
• Performance enhancements for mirai(), mirai_map() and the promises method.
• Requires nanonext >= 1.3.0.
• The package has a shiny new hex logo.

mirai 1.2.0

• everywhere() adds argument '.serial' to accept serialization configurations created by serial_config(). These allow normally non-exportable reference objects such as Arrow Tables or torch tensors to be used seamlessly across parallel processes without additional marshalling steps. Configurations apply on a per compute profile basis.
• serialization() is now deprecated in favour of the above usage of everywhere(), and will be removed in a future version.
• mirai_map() enhanced to perform multiple map over 2D lists/vectors, allowing advanced patterns such as mapping over the rows of a dataframe or matrix.
• 'mirai_map' [] method gains the option [.flat] to collect and flatten results, avoiding coercion.
• Collecting a 'mirai_map' no longer spuriously introduces empty names where none were present originally.
• Faster local daemons(dispatcher = FALSE) and make_cluster() by using asynchronous launches (thanks @mtmorgan #123).
• Local dispatcher daemons now synchronize with host, the same as non-dispatcher daemons (prevents use before all have connected).
• Fixes rare cases of everywhere() not reaching all daemons when using dispatcher.
• More efficient dispatcher startup by only loading the base package, in addition to not reading startup configurations (thanks @krlmlr).
• Removes hard dependency on stats and utils base packages.
• Requires nanonext >= 1.2.0.

mirai 1.1.1

• serialization() function signature and return value slightly modified for clarity. Successful registration / cancellation messages are no longer printed to the console.
• dispatcher() argument 'retry' now defaults to FALSE for consistency with non-dispatcher behaviour.
• remote_config() gains argument 'quote' to control whether or not to quote the daemon launch command, and now works with Slurm (thanks @michaelmayer2 #119).
• Ephemeral daemons now exit as soon as permissible, eliminating the 2s linger period.
• Requires nanonext >= 1.1.1.

mirai 1.1.0

• Adds mirai_map() for asynchronous parallel/distributed map using mirai, with promises integration. Allows recovery from partial failure or else early stopping, together with optional progress reporting.
  • x[] collects the results of a mirai_map x, waiting for all asynchronous operations to complete.
  • x[.progress] collects the results whilst showing a text progress bar.
  • x[.stop] collects the results applying early-stopping, which stops at the first error, and aborts remaining in-progress operations.
• Adds the 'mirai' method x[] as a more efficient equivalent of the interruptible call_mirai_(x)$data.
• Adds collect_mirai() as a more efficient equivalent of the non-interruptible call_mirai(x)$data.
• unresolved(), call_mirai(), collect_mirai() and stop_mirai() now accept a list of 'mirai' such as that returned by mirai_map().
• Improved mirai print method indicates whether a mirai has resolved.
• Calling daemons() with new settings when the compute profile is already set now implicitly resets daemons before applying the new settings instead of silently doing nothing.
• Argument 'resilience' retired at daemons() as automatic re-tries are no longer performed for non-dispatcher daemons.
• New argument 'retry' at dispatcher() governs whether to auto-retry in the dispatcher case.
• Fixes promises method for potential crashes when launching improbably short-lived mirai.
• Fixes bug that could cause a hang or crash when launching additional non-dispatcher daemons.
• Requires nanonext >= 1.1.0.

mirai 1.0.0

• Implements completely event-driven (non-polling) promises (thanks @jcheng5 for prototyping).
  • This is an innovation which allows higher responsiveness and massive scalability for 'mirai' promises.
• Behavioural changes to mirai() and everywhere():
  • (breaking change) no longer permits an unnamed list to be supplied to '.args'.
  • allows an environment e.g. environment() to be supplied to '.args' or as the only element of '...'.
  • allows evaluation of a symbol in the 'mirai' environment, e.g. mirai(x, x = 1).
• ssh_config() improvements:
  • new argument 'host' allows specifying the localhost URL and port to create a standalone configuration object.
  • order of arguments 'tunnel' and 'timeout' reversed.
• stop_mirai() now resolves to an 'errorValue' 20 (operation canceled) in the case the asynchronous task was still ongoing (thanks @jcheng5 #110).
• Rejected promises now show the complete error code and message in the case of an 'errorValue'.
• A 'miraiError' reverts to not including a trailing line break (as prior to mirai 0.13.2).
• Non-dispatcher local daemons now synchronize with host in all cases (prevents use before all have connected).
• [ method for 'miraiCluster' no longer produces a 'miraiCluster' object (thanks @HenrikBengtsson #83).
• Faster startup time as the parallel package is now only loaded when first used.
• Requires nanonext >= 1.0.0.

mirai 0.13.2

• mirai() and everywhere() behaviour changed such that '...' args are now assigned to the global environment of the daemon process.
• Adds with() method for mirai daemons, allowing for example: with(daemons(4), {expr}), where the daemons last for the duration of 'expr'.
• Adds register_cluster() for registering 'miraiCluster' as a parallel Cluster type (requires R >= 4.4).
• Adds is.promising() method for 'mirai' for the promises package.
• A 'miraiError' now includes the full call stack, which may be accessed at $stack.trace, and includes the trailing line break for consistency with 'as.character.error()'.
• mirai promises now preserve deep stacks when a 'miraiError' occurs within a Shiny app (thanks @jcheng5 #104).
• Simplified registration for 'parallel' and 'promises' methods (thanks @jcheng5 #103).
• Fixes to promises error handling and Shiny vignette (thanks @jcheng5 #98 #99).
• Requires R >= 3.6.

mirai 0.13.1

• Fixes regression in mirai 0.12.1, which introduced the potential for unintentional low level errors to emerge when querying dispatcher (thanks @dsweber2 for reporting in downstream {targets}).

mirai 0.13.0

• serialization adds arguments 'class' and 'vec' for custom serialisation of all reference object types.
• Requires nanonext >= 0.13.3.

CRAN release 0.12.1

• Dispatcher initial sync timeout widened to 10s to allow for launching large numbers of daemons.
• Default for ssh_config() argument 'timeout' widened to 10 (seconds).
• Fixes daemons() specifying 'output = FALSE' registering as TRUE instead.
• Fixes use of everywhere() specifying '.args' as an unnamed list or '.expr' as a language object.
• Ensures compatibility with nanonext >= 0.13.0.
• Internal performance enhancements.

CRAN release 0.12.0

• More minimal print methods for 'mirai' and 'miraiCluster'.
• Adds local_url() helper to construct a random inter-process communications URL for local daemons (thanks @noamross #90).
• daemon() argument 'autoexit' now accepts a signal value such as tools::SIGINT in order to raise it upon exit.
• daemon() now records the state of initial global environment objects (e.g. those created in .Rprofile) for cleanup purposes (thanks @noamross #91).
• Slightly more optimal as.promise() method for 'mirai'.
• Eliminates potential memory leaks along certain error paths.
• Requires nanonext >= 0.12.0.