Release/v 1.0.0

.wci.yml

@@ -2,8 +2,7 @@ name: libEnsemble
 icon: https://raw.githubusercontent.com/Libensemble/libensemble/main/docs/images/libE_logo.png
 headline: Tool for running dynamic ensembles.
 description: |
-  libEnsemble is a Python toolkit for coordinating workflows of asynchronous
-  and dynamic ensembles of calculations.
+  libEnsemble is a Python toolkit for running dynamic ensembles of calculations.
 
   Users write generator and simulator functions to express their ensembles.
   A library of example functions is available which can be modified as needed.
@@ -17,8 +16,8 @@ description: |
 language: Python
 
 release:
-  version: 0.10.2
-  date: 2023-07-24
+  version: 1.0.0
+  date: 2023-09-22
 
 documentation:
   general: https://libensemble.readthedocs.io

CHANGELOG.rst

@@ -8,6 +8,71 @@ GitHub issues are referenced, and can be viewed with hyperlinks on the `github r
 
 .. _`github releases page`: https://github.com/Libensemble/libensemble/releases
 
+Release 1.0.0
+--------------
+
+:Date: September 22, 2023
+
+New capabilities:
+
+* *libE_specs* option `final_gen_send` returns last results to the generator (replaces `final_fields`). #1086
+* *libE_specs* option `reuse_output_dir` allows reuse of workflow and ensemble directories. #1028 #1041
+* *libE_specs* option `calc_dir_id_width` no. of digits for calc ID in output sim/gen dirs. #1052 / #1066
+* Added `gen_num_procs` and `gen_num_gpus` *libE_specs* (and *persis_info*) options for resourcing a generator. #1068
+* Added `gpu_env_fallback` option to platform fields - specifies a GPU environment variable (for non-MPI usage). #1050
+* New MPIExecutor `submit()` argument `mpi_runner_type` specifies an MPI runner for current call only. #1054
+* Allow oversubscription when using the `num_procs` *gen_specs["out"]* option. #1058
+* sim/gen_specs can use `outputs` in place of `out` to be consistent with `inputs`. #1075
+* Executor can be obtained from `libE_info` (4th parameter) in user functions. #1078
+
+Breaking changes:
+
+* *libE_specs* option `final_fields` is removed in favor of `final_gen_send`. #1086
+* *libE_specs* option `kill_canceled_sims` now defaults to **False**. #1062
+* *parse_args* is not run automatically by `Ensemble` constructor.
+
+Updates to **Object Oriented** ensemble interface:
+
+* Added `parse_args` as option to `Ensemble` constructor. #1065
+* The *executor* can be passed as an option to the `Ensemble` constructor. #1078
+* Better handling of `Ensemble.add_random_streams` and `ensemble.persis_info`. #1074
+
+* Output changes:
+  * The worker ID suffix is removed from sim/gen output directories. #1041
+  * Separate *ensemble.log* and *libE_stats.txt* for diff workflows directories. #1027 #1041
+  * Defaults to four digits for sim/gen ID in output dirs (adds digits on overflow). #1052 / #1066
+
+Bug fixes:
+* Resolved PETSc/OpenMPI issue (when using the Executor). #1064
+* Prevent `mpi4py` validation running during local comms (when using OO interface). #1065
+
+Performance changes:
+* Optimize `kill_cancelled_sims` function.  #1043 / #1063
+* *safe_mode* defaults to **False** (for performance). #1053
+
+Updates to example functions:
+
+* Multiple regression tests and examples ported to use OO ensemble interface. #1014
+
+* Update forces examples:
+  * Make persistent generator the default for both simple and GPU examples (inc. updated tutorials).
+  * Update to object oriented interface.
+  * Added separate variable resources example for forces GPU.
+  * Rename `multi_task` example to `multi_app`.
+
+Documentation:
+
+* General overhaul and simplification of documentation. #992
+
+:Note:
+
+* Tested platforms include Linux, MacOS, Windows, and major systems such as Frontier (OLCF), Polaris, and Perlmutter (NERSC). The major system tests ran heterogeneous workflows.
+* Tested Python versions: (Cpython) 3.7, 3.8, 3.9, 3.10, 3.11.
+
+:Known issues:
+
+* See known issues section in the documentation.
+
 Release 0.10.2
 --------------
 

docs/FAQ.rst

@@ -4,7 +4,16 @@ Frequently Asked Questions
 
 If you have any additional questions, feel free to contact us through Support_.
 
-.. _Support: https://github.com/Libensemble/libensemble#resources
+.. _Support: https://libensemble.readthedocs.io/en/main/introduction.html#resources
+
+Debugging
+---------
+
+We recommend using the following options to help debug workflows::
+
+    from libensemble import logger
+    logger.set_level("DEBUG")
+    libE_specs["safe_mode"] = True
 
 Common Errors
 -------------
@@ -29,7 +38,7 @@ Common Errors
     one worker, leaving none to run simulation functions.
 
   - An error in the allocation function. For example, perhaps the allocation
-    waiting for all requested evaluations to be returned (e.g, before starting a
+    waiting for all requested evaluations to be returned (e.g., before starting a
     new generator), but this condition
     is not returning True even though all scheduled evaluations have returned. This
     can be due to incorrect implementation (e.g., it has not considered points that
@@ -126,7 +135,7 @@ HPC Errors and Questions
 
   This has been observed with the OFA fabric when using mpi4py and usually
   indicates MPI messages aren't being received correctly. The solution
-  is to either switch fabric or turn off matching probes. See the answer for "Why
+  is to either switch fabric or turn off matching probes. See the answer to "Why
   does libEnsemble hang on certain systems when running with MPI?"
 
   For more information see https://bitbucket.org/mpi4py/mpi4py/issues/102/unpicklingerror-on-commrecv-after-iprobe.
@@ -251,7 +260,7 @@ macOS and Windows Errors
   **"RuntimeError: An attempt has been made to start a new process... this probably means that you are not using fork...
   " if __name__ == "__main__": freeze_support() ...**
 
-  You need to place your main entrypoint code underneath an ``if __name__ == "__main__":`` block.
+  You need to place your main entry point code underneath an ``if __name__ == "__main__":`` block.
 
   Explanation: Python chooses one of three methods to start new processes when using multiprocessing
   (``--comms local`` with libEnsemble). These are ``"fork"``, ``"spawn"``, and ``"forkserver"``. ``"fork"``

docs/advanced_installation.rst

@@ -13,7 +13,7 @@ automatically installed alongside libEnsemble:
 * pyyaml_       ``>= v6.0``
 * tomli_        ``>= 1.2.1``
 
-In view of libEnsemble's compiled dependencies, the following installation
+Given libEnsemble's compiled dependencies, the following installation
 methods each offer a trade-off between convenience and the ability
 to customize builds, including platform-specific optimizations.
 
@@ -103,8 +103,8 @@ Further recommendations for selected HPC systems are given in the
             spack install py-libensemble
 
         The above command will install the latest release of libEnsemble with
-        the required dependencies only. There are other optional
-        dependencies that can be specified through variants. The following
+        the required dependencies only. Other optional
+        dependencies can be specified through variants. The following
         line installs libEnsemble version 0.7.2 with some common variants
         (e.g., using :doc:`APOSMM<../examples/aposmm>`):
 
@@ -122,12 +122,12 @@ Further recommendations for selected HPC systems are given in the
 
             spack install py-libensemble +scipy +mpmath +petsc4py ^py-petsc4py~mpi ^petsc~mpi~hdf5~hypre~superlu-dist
 
-        The install will create modules for libEnsemble and the dependent
+        The installation will create modules for libEnsemble and the dependent
         packages. These can be loaded by running::
 
             spack load -r py-libensemble
 
-        Any Python packages will be added to the PYTHONPATH, when the modules are loaded. If you do not have
+        Any Python packages will be added to the PYTHONPATH when the modules are loaded. If you do not have
         modules on your system you may need to install ``lmod`` (also available in Spack)::
 
             spack install lmod
@@ -181,7 +181,7 @@ The following packages may be installed separately to enable additional features
 .. _Balsam: https://balsam.readthedocs.io/en/latest/
 .. _conda-forge: https://conda-forge.org/
 .. _Conda: https://docs.conda.io/en/latest/
-.. _Globus Compute: https://funcx.readthedocs.io/en/latest/
+.. _Globus Compute: https://www.globus.org/compute
 .. _GitHub: https://github.com/Libensemble/libensemble
 .. _MPICH: https://www.mpich.org/
 .. _NumPy: http://www.numpy.org

docs/data_structures/libE_specs.rst

@@ -25,13 +25,13 @@ libEnsemble is primarily customized by setting options within a ``LibeSpecs`` cl
         .. tab-item:: General
 
                 "comms" [str] = ``"mpi"``:
-                    Manager/Worker communications mode. ``'mpi'``, ``'local'``, or ``'tcp'``
+                    Manager/Worker communications mode: ``'mpi'``, ``'local'``, or ``'tcp'``.
                 "nworkers" [int]:
                     Number of worker processes in ``"local"`` or ``"tcp"``.
                 "mpi_comm" [MPI communicator] = ``MPI.COMM_WORLD``:
                     libEnsemble MPI communicator.
                 "dry_run" [bool] = ``False``:
-                    Whether libEnsemble should immediately exit after validating all inputs
+                    Whether libEnsemble should immediately exit after validating all inputs.
                 "abort_on_exception" [bool] = ``True``:
                     In MPI mode, whether to call ``MPI_ABORT`` on an exception.
                     If ``False``, an exception will be raised by the manager.
@@ -45,7 +45,7 @@ libEnsemble is primarily customized by setting options within a ``LibeSpecs`` cl
                     On libEnsemble shutdown, number of seconds after which workers considered timed out,
                     then terminated.
                 "kill_canceled_sims" [bool] = ``False``:
-                    Try to kill sims with ``"cancel_requested"`` set ``True``.
+                    Try to kill sims with ``"cancel_requested"`` set to ``True``.
                     If ``False``, the manager avoids this moderate overhead.
                 "disable_log_files" [bool] = ``False``:
                     Disable ``ensemble.log`` and ``libE_stats.txt`` log files.
@@ -74,7 +74,7 @@ libEnsemble is primarily customized by setting options within a ``LibeSpecs`` cl
 
                     "ensemble_copy_back" [bool] = ``False``:
                         Whether to copy back contents of ``ensemble_dir_path`` to launch
-                        location. Useful if ``ensemble_dir_path`` located on node-local storage.
+                        location. Useful if ``ensemble_dir_path`` is located on node-local storage.
 
                     "reuse_output_dir" [bool] = ``False``:
                         Whether to allow overwrites and access to previous ensemble and workflow directories in subsequent runs.

docs/data_structures/persis_info.rst

@@ -13,7 +13,7 @@ and from the corresponding workers. These are received in the ``persis_info``
 argument of user functions, and returned as the optional second return value.
 
 A typical example is a random number generator stream to be used in consecutive
-calls to a generator (see function
+calls to a generator (see
 :meth:`add_unique_random_streams()<tools.add_unique_random_streams>`)
 
 All other entries persist on the manager and can be updated in the calling script
@@ -55,7 +55,7 @@ Examples:
     .. literalinclude:: ../../libensemble/alloc_funcs/start_only_persistent.py
        :linenos:
        :start-at:    if gen_count < persis_info.get("num_gens_started", 0):
-       :end-before:    # Give evaluated results back to a running persistent gen
+       :end-before:    # Give evaluated results back to the persistent gen
        :emphasize-lines: 1
        :caption: libensemble/alloc_funcs/start_only_persistent.py
 

docs/data_structures/platform_specs.rst

@@ -5,7 +5,7 @@ Platform Specs
 
 libEnsemble detects platform specifications including MPI runners and resources.
 Usually this will result in the correct settings. However, users can configure
-platform specification via the `platform_specs`_ option or indicate a known
+platform specifications via the `platform_specs`_ option or indicate a known
 platform via the `platform`_ option.
 
 platform_specs
@@ -47,7 +47,7 @@ To define a platform (in calling script):
                "scheduler_match_slots": False,
            }
 
-The list of platform fields are given below. Any fields not given, will be
+The list of platform fields is given below. Any fields not given will be
 auto-detected by libEnsemble.
 
 .. _platform-fields:
@@ -76,15 +76,15 @@ See :ref:`known platforms<known-platforms>`.
 platform
 --------
 
-A string giving the name of a known platform  defined in the platforms module.
+A string giving the name of a known platform defined in the platforms module.
 
 .. code-block:: python
 
     libE_specs["platform"] = "perlmutter_g"
 
 Note: the environment variable LIBE_PLATFORM is an alternative way of setting.
 
-E.g., on command line or batch submission script:
+E.g., in the command line or batch submission script:
 
 .. code-block:: shell
 

docs/executor/executor.rst

@@ -43,11 +43,11 @@ See the Executor APIs for optional arguments.
 
     :task.process: (process obj) The process object used by the underlying process
                   manager (e.g., return value of subprocess.Popen).
-    :task.errcode: (int) The errorcode/return code used by the underlying process manager.
+    :task.errcode: (int) The error code (or return code) used by the underlying process manager.
     :task.finished: (boolean) True means task has finished running - not whether it was successful.
-    :task.success: (boolean) Did task complete successfully (e.g., the returncode is zero)?
+    :task.success: (boolean) Did task complete successfully (e.g., the return code is zero)?
     :task.runtime: (int) Time in seconds that task has been running.
-    :task.submit_time: (int) Time since epoch that task was submitted
+    :task.submit_time: (int) Time since epoch that task was submitted.
     :task.total_time: (int) Total time from task submission to completion (only available when task is finished).
 
     Run configuration attributes - some will be autogenerated:

docs/executor/overview.rst

@@ -90,8 +90,8 @@ Example use-cases:
 See the :doc:`Executor<executor>` or :doc:`MPIExecutor<mpi_executor>` interface
 for the complete API.
 
-See :doc:`Running on HPC Systems<../platforms/platforms_index>` to see, with
-diagrams, how common options such as ``libE_specs["dedicated_mode"]`` affect the
+See :doc:`Running on HPC Systems<../platforms/platforms_index>` for illustrations
+of how common options such as ``libE_specs["dedicated_mode"]`` affect the
 run configuration on clusters and supercomputers.
 
 Advanced Features
@@ -105,6 +105,7 @@ In simulation function (sim_f).
 
     import time
 
+
     def sim_func(H, persis_info, sim_specs, libE_info):
 
         input_param = str(int(H["x"][0][0]))
@@ -121,7 +122,7 @@ In simulation function (sim_f).
         timeout_sec = 600
         poll_delay_sec = 1
 
-        while(not task.finished):
+        while not task.finished:
 
             # Has manager sent a finish signal
             if exctr.manager_kill_received():
@@ -153,7 +154,7 @@ In simulation function (sim_f).
 ..     ...
 
 Users who wish to poll only for manager kill signals and timeouts don't necessarily
-need to construct a polling loop like above, but can instead use an the ``Executor``
+need to construct a polling loop like above, but can instead use the ``Executor``
 built-in ``polling_loop()`` method. An alternative to the above simulation function
 may resemble:
 

docs/function_guides/allocator.rst

@@ -11,7 +11,7 @@ The ``alloc_f`` is unique since it is called by libEnsemble's manager instead of
 For allocation functions, as with the other user functions, the level of complexity can
 vary widely. We encourage experimenting with:
 
-    1.  Prioritization of simulations,
+    1.  Prioritization of simulations
     2.  Sending results immediately or in batch
     3.  Assigning varying resources to evaluations
 
@@ -26,16 +26,16 @@ Most ``alloc_f`` function definitions written by users resemble::
 
 where:
 
-    * :ref:`W<funcguides-workerarray>` is an array containing worker state info,
+    * :ref:`W<funcguides-workerarray>` is an array containing worker state info
     * :ref:`H<funcguides-history>` is the *trimmed* History array, containing rows from the generator
     * :ref:`libE_info<libE_info_alloc>` is a set of statistics to determine the progress of work or exit conditions
 
-Most users first check that its appropriate to allocate work::
+Most users first check that it is appropriate to allocate work::
 
         if libE_info["sim_max_given"] or not libE_info["any_idle_workers"]:
             return {}, persis_info
 
-If allocation is to continue, a support class is instantiated and a
+If the allocation is to continue, a support class is instantiated and a
 :ref:`Work dictionary<funcguides-workdict>` is initialized::
 
         manage_resources = "resource_sets" in H.dtype.names or libE_info["use_resource_sets"]
@@ -76,10 +76,10 @@ dictionary values to give to those workers:
 
 This Work dictionary instructs each worker to call the ``sim_f`` (``tag: 1``)
 with data from ``"x"`` and a given ``"H_row"`` from the
-History array. A worker specific ``persis_info`` is also given.
+History array. A worker-specific ``persis_info`` is also given.
 
 Constructing these arrays and determining which workers are available
-for receiving data is simplified by use of the ``AllocSupport`` class
+for receiving data is simplified by the ``AllocSupport`` class
 available within the ``libensemble.tools.alloc_support`` module:
 
 .. dropdown:: AllocSupport
@@ -92,7 +92,7 @@ available within the ``libensemble.tools.alloc_support`` module:
         .. automethod:: __init__
 
 The Work dictionary is returned to the manager alongside ``persis_info``. If ``1``
-is returned as third value, this instructs the ensemble to stop.
+is returned as the third value, this instructs the ensemble to stop.
 
 .. note:: An error occurs when the ``alloc_f`` returns nothing while
           all workers are idle
@@ -124,7 +124,7 @@ allocation function and detect impending timeouts, then pack up cleanup work req
 or mark points for cancellation.
 
 The remaining values above are useful for efficient filtering of H values
-(e.g., ``sim_ended_count``), saves a filtering an entire column of H.
+(e.g., ``sim_ended_count`` saves filtering by an entire column of H.)
 
 Descriptions of included allocation functions can be found :doc:`here<../examples/alloc_funcs>`.
 The default allocation function is

docs/function_guides/function_guide_index.rst

@@ -2,8 +2,8 @@
 Writing User Functions
 ======================
 
-User functions typically only require some familiarity with NumPy_, but if they conform to
-the :ref:`user function APIs<user_api>`, they can incorporate any other machine-learning,
+User functions typically require only some familiarity with NumPy_, but if they conform to
+the :ref:`user function APIs<user_api>`, they can incorporate methods from machine-learning,
 mathematics, resource management, or other libraries/applications.
 
 These guides describe common development patterns and optional components: