Feature/synced collection/optimization #453
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Although this behavior is undesirable for a validator, we allow it because it is purely a backwards compatibility layer for signac 1.x that will be removed in signac 2.0 and the performance hit from calling this function every time the JSON backend saves is too large to justify.
…o handle JSONCollections transparently.
vyasr
requested review from
bdice and
Tobias-Dwyer
and removed request for
a team
Codecov Report
@@ Coverage Diff @@
## feature/synced_collections #453 +/- ##
==============================================================
+ Coverage 77.81% 77.85% +0.04%
==============================================================
Files 55 56 +1
Lines 6517 6575 +58
Branches 1226 1232 +6
==============================================================
+ Hits 5071 5119 +48
- Misses 1147 1155 +8
- Partials 299 301 +2
Continue to review full report at Codecov.
|
bdice
reviewed
Dec 30, 2020
There was a problem hiding this comment.
This is nice, @vyasr! The design of the abstract type resolver is unusual (because it's handling an unusual problem) but I think it will be fine to use in the long term. For now, I'm just going to leave a few small comments from a quick read-through.
12 tasks
vyasr
added a commit
that referenced
this pull request
Feb 20, 2021
* Improve Sync Data Structures (#336) Adds a generic framework for synced collections to signac. The abstract SyncedCollection defines the interface for any Collection-like object (as defined by Python's collections.abc.Collection) that is synced with some form of persistent storage. This concept generalizes the existing _SyncedDict and SyncedList classes by implementing a common framework for these. Critically, this framework decouples the act of saving to and loading from the backend from the recursive conversion between a SyncedCollection and its base data type (e.g. a SyncedDict to a dict and vice versa), the latter of which is necessary to ensure proper synchronization of nested collections. This decoupling makes it much easier to enable different back ends for a given data structure without having to modify the synchronization logic. * Add alternative backends for synced_collection (#364) Adds Redis, Zarr, and MongoDB backends. * Added validation layer to SyncedCollection (#378) SyncedCollections can now arbitrary validators to input data to ensure that data conforms to a desired specification. This layer can be employed to ensure, for instance, that all inputs are valid JSON. * Added buffering to SyncedCollection (#363) Adds the ability to buffer I/O in SyncedCollections using an in-memory cache to store data. * Feature/synced collection/reorg (#437) Reorganizes synced collections into their own subpackage and move all corresponding tests into a subdirectory for easier identification. * Feature/synced collection/reorg tests (#438) Cleans up and standardizes the testing architecture so that all backends can easily apply the exact same set of tets using a few minor extra pieces of information. Prevents duplicate execution of tests upon import, while also adding a large number of additional tests for backends that were not previously tested. * Fix incomplete merge * Remove lingering old file. * Feature/synced collection/cleanup (#445) Rename various methods and remove unnecessary ones from the public API. Standardize internal APIs and simplify the implementation of new backends by automating more subclass behaviors. Improve constructors to enable positional arguments. Improve interfaces for various backends by making it easier for the user to specify and access the precise storage mechanisms. * Merge master, apply all relevant formatting tools, and add documentation (#446) Makes the SyncedCollection framework adhere to black, isort, flake8, pydocstyle, and mypy requirements while adding sufficiently detailed documentation to all classes. * Feature/synced collection/cleanup2 (#447) Simplifies and standardizes file buffering implementation. Adds extra tests for SyncedAttrDict.update and simplifies its implementation to use _update. * Feature/synced collection/optimization (#453) Optimize various aspects of the SyncedCollection framework, including validators, abstract base class type lookups, and the core data structures. * Remove unnecessary backend str from tests. * Feature/synced collection/test mongodb redis (#464) MongoDB and Redis will no longer be silently skipped on CI, so any changes that break support for those will be immediately discovered. * Make SyncedCollections thread-safe (#463) Operations that modify SyncedCollections (and their subclasses) now acquire object-specific mutexes prior to modification in order to guarantee consistency between threads. Thread safety can be enabled or disabled by the user. * Implements an in-memory buffer for SyncedCollections (#462) The new buffering mode is a variant on the old one that avoids unnecessary encoding, decoding, and in-memory updating by directly sharing memory between various collections accessing the same file in the buffer. This direct sharing allows all changes to be automatically persisted, avoiding any cache inconsistencies without the high overhead of JSON serialization for every single modification. * Clean up miscellaneous outstanding to-do items (#466) Completes TODO items scattered throughout the code base and removes a number of outdated ones that have already been addressed. * Make buffering thread safe (#468) In addition to synced collections being thread safe individually, while in buffered mode the buffer accesses also have to be made thread safe for multithreaded operation to be safe. This pull request introduces sufficient locking mechanisms to support thread-safe reads from and writes to the buffers. * Feature/synced collection/unify buffering (#469) Unifies the implementation of the two different file buffering modes as much as possible using a shared base class. In addition, this fixes a couple of issues with the thread-safe buffering solution in #468 that only show up on PyPy where execution is fast enough to stress-test multithreaded execution. It also reduces thread locking scopes to the minimum possible. * Feature/synced collection/contexts (#470) Removes usage of contextlib.contextmanager and replaces it with custom context classes implementing the context manager protocol. The contextlib decorator has measurable overhead that these pre-instantiated context managers avoid. Furthermore, many of the context managers in synced collections follow a very similar counter-based pattern that is now generalized and shared between them. * Install pymongo on pypy. * Don't sync on construction. * Add comparison operators to SyncedList and make sure modifying the filename of JSONDict is safe. * Remove unnecessary constructor validation, providing both data and resource arguments (e.g. filename for a JSONDict) is valid. * Fix unused imports. * Feature/synced collection/replace jsondict (#472) The old JSONDict and SyncedAttrDict classes are replaced with the new ones from the SyncedCollections framework. The new classes are now used for the Job's statepoint and document attributes as well as the Project document. The state point is now stored in the new _StatePointDict class, which extends the new JSONDict class to afford greater control over saving and loading of data. Much of the internals of the Job class have also been simplified, with most of the complex logic for job migration and validation when the state point changes now contained within the _StatePointDict. * Replace old JSONDict with new BufferedJSONDict. * Verify that replacing BufferedJSONDict with MemoryBufferedJSONDict. * Remove largely redundant _reset_sp method. * Remove single-use internal functions in Job to reduce surface area for SyncedCollection integration. * Move logic from _init into init. * Working implementation of statepoint using new SyncedCollection. * Remove _check_manifest. * Expose loading explicitly to remove the need for internal laziness in the StatepointDict. * Simplify the code as much as possible by inlining move method and catching the correct error. * Improve documentation of context manager for statepoint loading. * Replace MemoryBufferedJSONDict in Project for now. * Add documentation of why jobs must be stored as a list in the statepoint. * Address PR comments. * Add back import. * Ensure _StatepointDict is always initialized in constructor. * Change _StatepointDict to validate id on load. * Refactor error handling into _StatepointDict class. * Update docstrings. * Update comment. * Fix some docstrings. * Remove redundant JobsCorruptedError check. * Rewrite reset_statepoint to not depend on creating another job. * Reduce direct accesses of internal attributes and do some simplification of the code. * Reraise errors in JSONCollection. * Change reset to require a non-None argument and to call _update internally. * Add reset_data method to provide clear access points of the _StatepointDict for the Job. * Create new internal method for handling resetting. * Move statepoint resetting logic into the statepoint object itself. * Stop accessing internal statepoint filename attribute directly and rely on validation on construction. * Make statepoint thread safe. * Some minor cleanup. * Remove now unnecessary protection of the filename key. * Explicitly document behavior of returning None from _load_from_resource. * Apply suggestions from code review Co-authored-by: Bradley Dice <[email protected]> * Rename SCJSONEncoder to SyncedCollectionJSONEncoder. * Only access old id once. * Move lazy attribute initialization into one location. * Address PR requests that don't cause any issues. * Remove the temporary state point file backup. * Make as many old buffer tests as possible. * Reset buffer size after test. * Last changes from PR review. Co-authored-by: Bradley Dice <[email protected]> * Fix synced collection support for 0d numpy arrays. * Add oldest supported version of pymongo and ensure that zarr/mongo collections don't fail on import. * Deprecate json module (#480) Change all non-deprecated modules to import the built-in json module instead of signac's and deprecate signac.core.json. * Feature/synced collection/reorg (#481) Reorganizes the package structure so that the synced_collections subpackage is now at the package root and is internally structured with subpackages for data types, backends, and buffers. * Move synced collections from core to package root. * Fix all import locations. * Reorganize internals of synced collection package. * Fix all imports for reorganized package. * Hide caching module since it's still experimental. * Address PR comments. * Make __all__ into an empty list not rather than a list containing an empty string. * Feature/synced collection/simplify global buffering (#482) Eliminate the global buffering mode across all backends in favor of a more targeted per-backend approach. * Enable per backend buffering. * Remove global buffering in favor of class-specific buffering. * Reintroduce warnings for deprecated functionality. * Remove truly global buffering and replace it with class-level buffering. * Document new features. * Feature/synced collection/deprecate old (#483) Deprecates the old _SyncedDict, SyncedAttrDict, and JSONDict classes, along with any associated functions and exceptions. * Deprecate old synced dict classes. * Move class deprecation warnings to constructors. * Feature/synced collection/fix buffer reload (#486) * Make sure data type is preserved when reloading from buffer after flush. * Add test of new error case. * Fix lots of documentation issues. * Address first round of PR comments. * Update changelog. * Fix mypy error. * Don't error check uid unless it's provided. * First pass to address PR comments. * Feature/synced collection/remove attr access (#504) This patch removes attribute-based access to synced dictionaries. This logic is moved to a new `AttrDict` mixin class that can be inherited by any other subclasses if this feature is desired. * Simplify definition of __setattr__ by relying on complete list of protected keys. * Move all attribute-based access to a separate mixin class AttrDict. * Rename SyncedAttrDict to SyncedDict. * Move synced_attr_dict to synced_dict. * Remove attribute-based access from existing backend dict types and add new validator to check string keys. * Isolate deprecated non-str key handling to the _convert_key_to_str json validator. * Add tests of the AttrDict behavior. * Use new attrdict based classes in signac and make all tests pass. * Remove support for inheriting protected attributes, they must be defined by the class itself. * Change initialization order to call super first wherever possible. * Address PR comments. * Address final round of PR comments. * Feature/synced collection/numpy (#503) Isolate all numpy logic into a single utility file so that handling of numpy arrays can be standardized. Also substantially improves test coverage, testing a large matrix of different numpy dtypes and shapes with different types of synced collections. The testing framework as a whole is refactored to simplify the code and reduce the amount of boilerplate required to add the new numpy tests. * Initial pass at isolating all numpy logic to a single file. * Use pytest to generate temporary directories. * Stop saving backend kwargs as class variables. * Remove most references to class-level variables in tests. * Remove the _backend_collection variable. * Remove unnecessary autouse fixtures. * Start adding comprehensive tests for numpy conversion. * Make sure locks are released even if saving fails. * Add tests of vector types and add tests for SyncedList as well as SyncedDict. * Use pytest to parametrize numpy types instead of looping manually. * Unify vector and scalar tests. * Stop testing squeezing and just test the relevant shapes directly. * Add test of reset. * Move numpy tests back to main file. * Remove _conert_numpy_scalar, which performed undesirable squeezing of 1-element 1d arrays, and replace usage with _convert_numpy. * Separate numpy testing into separate functions and limit supported behavior to only the necessary use cases. * Add a warning when implicit numpy conversion occurs. * Update changelog. * Address all PR comments aside from numpy conversion in type resolver. * Catch numpy warnings in signac numpy integration tests. * Support older versions of numpy for random number generation. * Fix isort issue introduced by rebase. * Address PR comments. * Allow AbstractTypeResolver to perform arbitrary preprocessing, delegating the numpy-specialized code to the caller and making it less confusing. * Add missing call to _convert_numpy. * Set NUMPY_SHAPES for MongoDB tests when numpy is not present. * Remove add_validators and specify validators in class definition (#507) * Remove add_validators classmethod and instead require validators to be defined at class definition time, preventing one application from modifying validation process for others and giving a standard means to completely override parent validators in parents. * Change type in docstring. * Don't use os.path.join where not needed. (#511) The extra work performed by os.path.join can be slow, so this PR replaces it with direct string concatenation of os.sep. * Don't use os.path.join where not needed. * Update signac/contrib/job.py Co-authored-by: Bradley Dice <[email protected]> * Disable recursive validation during recursive conversion of nested types. (#509) * Feature/synced collection/optimize jsondict validation (#508) Define a single validator for JSONAttrDict classes that combines the logic of other validators while reducing collection traversal costs. Also switch from converting numpy arrays to just bypassing the resolver's cache for them. * Use single separate validator for state points for performance. * Remove preprocessor from type resolver and instead use a blocklist that prevents caching data for certain types. * Reorder resolvers to optimize performance. * Make sure not to include strings as sequences. * Move state point validator to collection_json and use for all JSONAttrDict types. * Make sure to also check complex types. * Add back missing period lost during stash merge. * Address review comments. * Feature/synced collection/optimize protected key lookup (#510) Since the most common operation on protected keys is to check if some key is within the list of protected keys, this patch changes the `_PROTECTED_KEYS` attribute to a set for faster O(1) membership checks. * Switch protected keys from a sequence to a set for faster containment checks. * Change evaluation order of checks in setattr. * Address PR comments. * Defer statepoint instantiation, unify reset_statepoint logic (#497) * Defer state point initialization when lazy loading. * Allow validation to be disabled in SyncedCollection._update. * Unify reset_statepoint logic across methods. * Revert validation-related changes. * Add test, fix bug. * Update tests/test_job.py * Feature/synced collection/optimize (#513) * Add a resoler to fast-exit _from_base for non-collection types. * Optimize synced collection init method. * Remove validators property and access internal variable directly. * Add early exit for _convert_array. * Use locally cached root var instead of self._root. * Remove superfluous duplicate check. * Optionally skip validation in SyncedCollection _update. (#512) * Add option to trust the data source during _update. * Skip validation if JSON is valid. * Fix pre-commit. * Rename trust_source to _validate. * Set _validate=False. Co-authored-by: Vishav Sharma <[email protected]> Co-authored-by: Bradley Dice <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR optimizes all JSON-based SyncedCollections, including the buffered mode. I benchmarked a number of operations comparing the new JSONDict to the old one on two machines, a 2020 Macbook Pro and our Linux workstations using a slow network file system (as opposed to faster scratch space). I'm happy to share some plots for those interested. Brief summary: depending on the operation being benchmarked, whether buffering is enabled, and the operating system, I see performance improvements in the range of 1.3x-2.7x. In general the biggest performance improvements are observed for operations that change the dict since they touch all of the code paths that I optimized. Some more detailed information:
Motivation and Context
Types of Changes
1The change breaks (or has the potential to break) existing functionality.
Checklist:
If necessary: