Skip to content

Commit

Permalink
Feature/synced collections (#484)
Browse files Browse the repository at this point in the history
* Improve Sync Data Structures (#336)

Adds a generic framework for synced collections to signac. The abstract SyncedCollection defines the interface for any Collection-like object (as defined by Python's collections.abc.Collection) that is synced with some form of persistent storage. This concept generalizes the existing _SyncedDict and SyncedList classes by implementing a common framework for these. Critically, this framework decouples the act of saving to and loading from the backend from the recursive conversion between a SyncedCollection and its base data type (e.g. a SyncedDict to a dict and vice versa), the latter of which is necessary to ensure proper synchronization of nested collections. This decoupling makes it much easier to enable different back ends for a given data structure without having to modify the synchronization logic.

* Add alternative backends for synced_collection (#364)

Adds Redis, Zarr, and MongoDB backends.

* Added validation layer to SyncedCollection (#378)

SyncedCollections can now arbitrary validators to input data to ensure that data conforms to a desired specification. This layer can be employed to ensure, for instance, that all inputs are valid JSON.

* Added buffering to SyncedCollection (#363)

Adds the ability to buffer I/O in SyncedCollections using an in-memory cache to store data.

* Feature/synced collection/reorg (#437)

Reorganizes synced collections into their own subpackage and move all corresponding tests into a subdirectory for easier identification.

* Feature/synced collection/reorg tests (#438)

Cleans up and standardizes the testing architecture so that all backends can easily apply the exact same set of tets using a few minor extra pieces of information. Prevents duplicate execution of tests upon import, while also adding a large number of additional tests for backends that were not previously tested.

* Fix incomplete merge

* Remove lingering old file.

* Feature/synced collection/cleanup (#445)

Rename various methods and remove unnecessary ones from the public API. Standardize internal APIs and simplify the implementation of new backends by automating more subclass behaviors. Improve constructors to enable positional arguments. Improve interfaces for various backends by making it easier for the user to specify and access the precise storage mechanisms.

* Merge master, apply all relevant formatting tools, and add documentation (#446)

Makes the SyncedCollection framework adhere to black, isort, flake8, pydocstyle, and mypy requirements while adding sufficiently detailed documentation to all classes.

* Feature/synced collection/cleanup2 (#447)

Simplifies and standardizes file buffering implementation. Adds extra tests for SyncedAttrDict.update and simplifies its implementation to use _update.

* Feature/synced collection/optimization (#453)

Optimize various aspects of the SyncedCollection framework, including validators, abstract base class type lookups, and the core data structures.

* Remove unnecessary backend str from tests.

* Feature/synced collection/test mongodb redis (#464)

MongoDB and Redis will no longer be silently skipped on CI, so any changes that break support for those will be immediately discovered.

* Make SyncedCollections thread-safe (#463)

Operations that modify SyncedCollections (and their subclasses) now acquire object-specific mutexes prior to modification in order to guarantee consistency between threads. Thread safety can be enabled or disabled by the user.

* Implements an in-memory buffer for SyncedCollections (#462)

The new buffering mode is a variant on the old one that avoids unnecessary encoding, decoding, and in-memory updating by directly sharing memory between various collections accessing the same file in the buffer. This direct sharing allows all changes to be automatically persisted, avoiding any cache inconsistencies without the high overhead of JSON serialization for every single modification.

* Clean up miscellaneous outstanding to-do items (#466)

Completes TODO items scattered throughout the code base and removes a number of outdated ones that have already been addressed.

* Make buffering thread safe (#468)

In addition to synced collections being thread safe individually, while in buffered mode the buffer accesses also have to be made thread safe for multithreaded operation to be safe. This pull request introduces sufficient locking mechanisms to support thread-safe reads from and writes to the buffers.

* Feature/synced collection/unify buffering (#469)

Unifies the implementation of the two different file buffering modes as much as possible using a shared base class. In addition, this fixes a couple of issues with the thread-safe buffering solution in #468 that only show up on PyPy where execution is fast enough to stress-test multithreaded execution. It also reduces thread locking scopes to the minimum possible.

* Feature/synced collection/contexts (#470)

Removes usage of contextlib.contextmanager and replaces it with custom context classes implementing the context manager protocol. The contextlib decorator has measurable overhead that these pre-instantiated context managers avoid. Furthermore, many of the context managers in synced collections follow a very similar counter-based pattern that is now generalized and shared between them.

* Install pymongo on pypy.

* Don't sync on construction.

* Add comparison operators to SyncedList and make sure modifying the filename of JSONDict is safe.

* Remove unnecessary constructor validation, providing both data and resource arguments (e.g. filename for a JSONDict) is valid.

* Fix unused imports.

* Feature/synced collection/replace jsondict (#472)

The old JSONDict and SyncedAttrDict classes are replaced with the new ones from the SyncedCollections framework. The new classes are now used for the Job's statepoint and document attributes as well as the Project document. The state point is now stored in the new _StatePointDict class, which extends the new JSONDict class to afford greater control over saving and loading of data. Much of the internals of the Job class have also been simplified, with most of the complex logic for job migration and validation when the state point changes now contained within the _StatePointDict.

* Replace old JSONDict with new BufferedJSONDict.

* Verify that replacing BufferedJSONDict with MemoryBufferedJSONDict.

* Remove largely redundant _reset_sp method.

* Remove single-use internal functions in Job to reduce surface area for SyncedCollection integration.

* Move logic from _init into init.

* Working implementation of statepoint using new SyncedCollection.

* Remove _check_manifest.

* Expose loading explicitly to remove the need for internal laziness in the StatepointDict.

* Simplify the code as much as possible by inlining move method and catching the correct error.

* Improve documentation of context manager for statepoint loading.

* Replace MemoryBufferedJSONDict in Project for now.

* Add documentation of why jobs must be stored as a list in the statepoint.

* Address PR comments.

* Add back import.

* Ensure _StatepointDict is always initialized in constructor.

* Change _StatepointDict to validate id on load.

* Refactor error handling into _StatepointDict class.

* Update docstrings.

* Update comment.

* Fix some docstrings.

* Remove redundant JobsCorruptedError check.

* Rewrite reset_statepoint to not depend on creating another job.

* Reduce direct accesses of internal attributes and do some simplification of the code.

* Reraise errors in JSONCollection.

* Change reset to require a non-None argument and to call _update internally.

* Add reset_data method to provide clear access points of the _StatepointDict for the Job.

* Create new internal method for handling resetting.

* Move statepoint resetting logic into the statepoint object itself.

* Stop accessing internal statepoint filename attribute directly and rely on validation on construction.

* Make statepoint thread safe.

* Some minor cleanup.

* Remove now unnecessary protection of the filename key.

* Explicitly document behavior of returning None from _load_from_resource.

* Apply suggestions from code review

Co-authored-by: Bradley Dice <[email protected]>

* Rename SCJSONEncoder to SyncedCollectionJSONEncoder.

* Only access old id once.

* Move lazy attribute initialization into one location.

* Address PR requests that don't cause any issues.

* Remove the temporary state point file backup.

* Make as many old buffer tests as possible.

* Reset buffer size after test.

* Last changes from PR review.

Co-authored-by: Bradley Dice <[email protected]>

* Fix synced collection support for 0d numpy arrays.

* Add oldest supported version of pymongo and ensure that zarr/mongo collections don't fail on import.

* Deprecate json module (#480)

Change all non-deprecated modules to import the built-in json module instead of signac's and deprecate signac.core.json.

* Feature/synced collection/reorg (#481)

Reorganizes the package structure so that the synced_collections subpackage is now at the package root and is internally structured with subpackages for data types, backends, and buffers.

* Move synced collections from core to package root.

* Fix all import locations.

* Reorganize internals of synced collection package.

* Fix all imports for reorganized package.

* Hide caching module since it's still experimental.

* Address PR comments.

* Make __all__ into an empty list not rather than a list containing an empty string.

* Feature/synced collection/simplify global buffering (#482)

Eliminate the global buffering mode across all backends in favor of a more targeted per-backend approach.

* Enable per backend buffering.

* Remove global buffering in favor of class-specific buffering.

* Reintroduce warnings for deprecated functionality.

* Remove truly global buffering and replace it with class-level buffering.

* Document new features.

* Feature/synced collection/deprecate old (#483)

Deprecates the old _SyncedDict, SyncedAttrDict, and JSONDict classes, along with any associated functions and exceptions.

* Deprecate old synced dict classes.

* Move class deprecation warnings to constructors.

* Feature/synced collection/fix buffer reload (#486)

* Make sure data type is preserved when reloading from buffer after flush.

* Add test of new error case.

* Fix lots of documentation issues.

* Address first round of PR comments.

* Update changelog.

* Fix mypy error.

* Don't error check uid unless it's provided.

* First pass to address PR comments.

* Feature/synced collection/remove attr access (#504)

This patch removes attribute-based access to synced dictionaries. This logic is moved to a new `AttrDict` mixin class that can be inherited by any other subclasses if this feature is desired.

* Simplify definition of __setattr__ by relying on complete list of protected keys.

* Move all attribute-based access to a separate mixin class AttrDict.

* Rename SyncedAttrDict to SyncedDict.

* Move synced_attr_dict to synced_dict.

* Remove attribute-based access from existing backend dict types and add new validator to check string keys.

* Isolate deprecated non-str key handling to the _convert_key_to_str json validator.

* Add tests of the AttrDict behavior.

* Use new attrdict based classes in signac and make all tests pass.

* Remove support for inheriting protected attributes, they must be defined by the class itself.

* Change initialization order to call super first wherever possible.

* Address PR comments.

* Address final round of PR comments.

* Feature/synced collection/numpy (#503)

Isolate all numpy logic into a single utility file so that handling of numpy arrays can be standardized. Also substantially improves test coverage, testing a large matrix of different numpy dtypes and shapes with different types of synced collections. The testing framework as a whole is refactored to simplify the code and reduce the amount of boilerplate required to add the new numpy tests.

* Initial pass at isolating all numpy logic to a single file.

* Use pytest to generate temporary directories.

* Stop saving backend kwargs as class variables.

* Remove most references to class-level variables in tests.

* Remove the _backend_collection variable.

* Remove unnecessary autouse fixtures.

* Start adding comprehensive tests for numpy conversion.

* Make sure locks are released even if saving fails.

* Add tests of vector types and add tests for SyncedList as well as SyncedDict.

* Use pytest to parametrize numpy types instead of looping manually.

* Unify vector and scalar tests.

* Stop testing squeezing and just test the relevant shapes directly.

* Add test of reset.

* Move numpy tests back to main file.

* Remove _conert_numpy_scalar, which performed undesirable squeezing of 1-element 1d arrays, and replace usage with _convert_numpy.

* Separate numpy testing into separate functions and limit supported behavior to only the necessary use cases.

* Add a warning when implicit numpy conversion occurs.

* Update changelog.

* Address all PR comments aside from numpy conversion in type resolver.

* Catch numpy warnings in signac numpy integration tests.

* Support older versions of numpy for random number generation.

* Fix isort issue introduced by rebase.

* Address PR comments.

* Allow AbstractTypeResolver to perform arbitrary preprocessing, delegating the numpy-specialized code to the caller and making it less confusing.

* Add missing call to _convert_numpy.

* Set NUMPY_SHAPES for MongoDB tests when numpy is not present.

* Remove add_validators and specify validators in class definition (#507)

* Remove add_validators classmethod and instead require validators to be defined at class definition time, preventing one application from modifying validation process for others and giving a standard means to completely override parent validators in parents.

* Change type in docstring.

* Don't use os.path.join where not needed. (#511)

The extra work performed by os.path.join can be slow, so this PR replaces it with direct string concatenation of os.sep.

* Don't use os.path.join where not needed.

* Update signac/contrib/job.py

Co-authored-by: Bradley Dice <[email protected]>

* Disable recursive validation during recursive conversion of nested types. (#509)

* Feature/synced collection/optimize jsondict validation (#508)

Define a single validator for JSONAttrDict classes that combines the logic of other validators while reducing collection traversal costs. Also switch from converting numpy arrays to just bypassing the resolver's cache for them.

* Use single separate validator for state points for performance.

* Remove preprocessor from type resolver and instead use a blocklist that prevents caching data for certain types.

* Reorder resolvers to optimize performance.

* Make sure not to include strings as sequences.

* Move state point validator to collection_json and use for all JSONAttrDict types.

* Make sure to also check complex types.

* Add back missing period lost during stash merge.

* Address review comments.

* Feature/synced collection/optimize protected key lookup (#510)

Since the most common operation on protected keys is to check if some key is within the list of protected keys, this patch changes the `_PROTECTED_KEYS` attribute to a set for faster O(1) membership checks.

* Switch protected keys from a sequence to a set for faster containment checks.

* Change evaluation order of checks in setattr.

* Address PR comments.

* Defer statepoint instantiation, unify reset_statepoint logic (#497)

* Defer state point initialization when lazy loading.

* Allow validation to be disabled in SyncedCollection._update.

* Unify reset_statepoint logic across methods.

* Revert validation-related changes.

* Add test, fix bug.

* Update tests/test_job.py

* Feature/synced collection/optimize (#513)

* Add a resoler to fast-exit _from_base for non-collection types.

* Optimize synced collection init method.

* Remove validators property and access internal variable directly.

* Add early exit for _convert_array.

* Use locally cached root var instead of self._root.

* Remove superfluous duplicate check.

* Optionally skip validation in SyncedCollection _update. (#512)

* Add option to trust the data source during _update.

* Skip validation if JSON is valid.

* Fix pre-commit.

* Rename trust_source to _validate.

* Set _validate=False.

Co-authored-by: Vishav Sharma <[email protected]>
Co-authored-by: Bradley Dice <[email protected]>
  • Loading branch information
3 people authored Feb 20, 2021
2 parents 27d6c9f + 9a73e9a commit 29c0749
Show file tree
Hide file tree
Showing 55 changed files with 7,051 additions and 284 deletions.
1 change: 1 addition & 0 deletions .circleci/ci-oldest-reqs.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ h5py==2.8.0
numpy==1.13.3
packaging==15.0
pandas==1.0.0
pymongo==3.0.0
pytest-cov==2.10.1
pytest==6.2.1
tables==3.3.0
Expand Down
15 changes: 14 additions & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ jobs:
linux-python-39: &linux-template
docker:
- image: circleci/python:3.9
- image: circleci/mongo:latest
- image: circleci/redis:latest

environment:
BENCHMARKS: "RUN"
Expand Down Expand Up @@ -138,16 +140,22 @@ jobs:
<<: *linux-template
docker:
- image: circleci/python:3.8
- image: circleci/mongo:latest
- image: circleci/redis:latest

linux-python-37:
<<: *linux-template
docker:
- image: circleci/python:3.7
- image: circleci/mongo:latest
- image: circleci/redis:latest

linux-python-36-oldest:
<<: *linux-template
docker:
- image: circleci/python:3.6
- image: circleci/mongo:latest
- image: circleci/redis:latest
environment:
BENCHMARKS: "SKIP"
DEPENDENCIES: "OLDEST"
Expand Down Expand Up @@ -180,7 +188,12 @@ jobs:
${PYTHON} -m pip install --progress-bar off -U pip>=20.3
${PYTHON} -m pip install --progress-bar off -U codecov
${PYTHON} -m pip install --progress-bar off -U -r requirements/requirements-test.txt
${PYTHON} -m pip install --progress-bar off -U -r requirements/requirements-test-optional.txt
# For some reason Zarr doesn't install correctly on Windows (runs
# into pip SSL errors), so we skip that test.
grep -v zarr requirements-test-optional.txt > requirements-test-optional-windows.txt
${PYTHON} -m pip install --progress-bar off -U -r requirements-test-optional-windows.txt
${PYTHON} -m pip install --progress-bar off -U -e .
- run:
name: Run tests
Expand Down
1 change: 1 addition & 0 deletions changelog.txt
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ next
Added
+++++

- New ``SyncedCollection`` class and subclasses to replace ``JSONDict`` with more general support for different types of resources (such as MongoDB collections or Redis databases) and more complete support for different data types synchronized with files (#196, #234, #249, #316, #383, #397, #465, #484). This change introduces a minor-backwards incompatible change; for users making direct use of signac buffering, the ``force_write`` parameter is no longer respected. If the argument is passed, a warning will now be raised to indicate that it is ignored and will be removed in signac 2.0.
- Unified querying for state point and document filters using 'sp' and 'doc' as prefixes (#332, #514). This change introduces a minor backwards-incompatible change to the ``Collection`` index schema ('statepoint'->'sp'), but this does not affect any APIs, only indexes saved to file using a previous version of signac. Indexing APIs will be removed in signac 2.0.


Expand Down
108 changes: 108 additions & 0 deletions doc/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -206,3 +206,111 @@ signac.errors module
:members:
:undoc-members:
:show-inheritance:

synced\_collections package
===========================

Data Types
----------

synced\_collections.synced\_collection module
+++++++++++++++++++++++++++++++++++++++++++++

.. automodule:: signac.synced_collections.data_types.synced_collection
:members:
:private-members:
:show-inheritance:

synced\_collections.synced\_dict module
+++++++++++++++++++++++++++++++++++++++++++++

.. automodule:: signac.synced_collections.data_types.synced_dict
:members:
:show-inheritance:

synced\_collections.synced\_list module
+++++++++++++++++++++++++++++++++++++++++++++

.. automodule:: signac.synced_collections.data_types.synced_list
:members:
:show-inheritance:

Backends
--------

synced\_collections.backends.collection\_json module
+++++++++++++++++++++++++++++++++++++++++++

.. automodule:: signac.synced_collections.backends.collection_json
:members:
:show-inheritance:

synced\_collections.backends.collection\_mongodb module
++++++++++++++++++++++++++++++++++++++++++++++

.. automodule:: signac.synced_collections.backends.collection_mongodb
:members:
:show-inheritance:

synced\_collections.backends.collection\_redis module
++++++++++++++++++++++++++++++++++++++++++++

.. automodule:: signac.synced_collections.backends.collection_redis
:members:
:show-inheritance:

synced\_collections.backends.collection\_zarr module
+++++++++++++++++++++++++++++++++++++++++++

.. automodule:: signac.synced_collections.backends.collection_zarr
:members:
:show-inheritance:

Buffers
-------

synced\_collections.buffers.buffered\_collection module
+++++++++++++++++++++++++++++++++++++++++++++++

.. automodule:: signac.synced_collections.buffers.buffered_collection
:members:
:private-members:
:show-inheritance:

synced\_collections.buffers.file\_buffered\_collection module
+++++++++++++++++++++++++++++++++++++++++++++++++++++

.. automodule:: signac.synced_collections.buffers.file_buffered_collection
:members:
:show-inheritance:

synced\_collections.buffers.serialized\_file\_buffered\_collection module
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

.. automodule:: signac.synced_collections.buffers.serialized_file_buffered_collection
:members:
:show-inheritance:

synced\_collections.buffers.memory\_buffered\_collection module
+++++++++++++++++++++++++++++++++++++++++++++++++++++++

.. automodule:: signac.synced_collections.buffers.memory_buffered_collection
:members:
:show-inheritance:

Miscellaneous Modules
---------------------

synced\_collections.utils module
++++++++++++++++++++++++++++++++

.. automodule:: signac.synced_collections.utils
:members:
:show-inheritance:

synced\_collections.validators module
+++++++++++++++++++++++++++++++++++++

.. automodule:: signac.synced_collections.validators
:members:
:show-inheritance:
7 changes: 5 additions & 2 deletions doc/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -316,6 +316,9 @@ def __getattr__(cls, name):
intersphinx_mapping = {
"python": ("https://docs.python.org/3", None),
"pymongo": ("https://pymongo.readthedocs.io/en/stable/", None),
"pandas": ("https://pandas.pydata.org/docs/", None),
"h5py": ("https://docs.h5py.org/en/stable/", None),
"pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None),
"h5py": ("http://docs.h5py.org/en/stable/", None),
"zarr": ("https://zarr.readthedocs.io/en/stable", None),
"redis": ("https://redis-py.readthedocs.io/en/stable/", None),
"numcodecs": ("https://numcodecs.readthedocs.io/en/stable/", None),
}
3 changes: 3 additions & 0 deletions requirements/requirements-test-optional.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,7 @@ h5py==3.1.0; implementation_name=='cpython'
numpy==1.20.0
pandas==1.2.1; implementation_name=='cpython'
pymongo==3.11.2; implementation_name=='cpython'
redis==3.5.3
ruamel.yaml==0.16.12
tables==3.6.1; implementation_name=='cpython'
zarr==2.4.0; platform_system!='Windows'
6 changes: 3 additions & 3 deletions setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ ignore = E123,E126,E203,E226,E241,E704,W503,W504
match = ^((?!\.sync-zenodo-metadata|setup|benchmark|mpipool|connection|crypt|host|filesystems|indexing).)*\.py$
match-dir = ^((?!\.|tests|configobj|db).)*$
ignore-decorators = "deprecated"
ignore = D105, D107, D203, D204, D213
add-ignore = D105, D107, D203, D204, D213

[mypy]
ignore_missing_imports = True
Expand All @@ -34,8 +34,8 @@ omit =
*/signac/common/configobj/*.py

[tool:pytest]
filterwarnings =
ignore: .*[The indexing module | get_statepoint] is deprecated.*: DeprecationWarning
filterwarnings =
ignore: .*[The indexing module | get_statepoint | Use of.+as key] is deprecated.*: DeprecationWarning

[bumpversion:file:setup.py]

Expand Down
15 changes: 11 additions & 4 deletions signac/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,15 +27,21 @@
from .contrib import filesystems as fs
from .contrib import get_job, get_project, index, index_files, init_project
from .core.h5store import H5Store, H5StoreManager
from .core.jsondict import JSONDict
from .core.jsondict import buffer_reads_writes as buffered
from .core.jsondict import flush_all as flush
from .core.jsondict import get_buffer_load, get_buffer_size
from .core.jsondict import in_buffered_mode as is_buffered
from .db import get_database
from .diff import diff_jobs
from .synced_collections.backends.collection_json import (
BufferedJSONAttrDict as JSONDict,
)
from .version import __version__

# Alias some properties related to buffering into the signac namespace.
buffered = JSONDict.buffer_backend
is_buffered = JSONDict.backend_is_buffered
get_buffer_load = JSONDict.get_current_buffer_size
get_buffer_size = JSONDict.get_buffer_capacity
set_buffer_size = JSONDict.set_buffer_capacity

__all__ = [
"__version__",
"contrib",
Expand Down Expand Up @@ -69,6 +75,7 @@
"flush",
"get_buffer_size",
"get_buffer_load",
"set_buffer_size",
"JSONDict",
"H5Store",
"H5StoreManager",
Expand Down
2 changes: 1 addition & 1 deletion signac/contrib/collection.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

import argparse
import io
import json
import logging
import operator
import re
Expand All @@ -24,7 +25,6 @@
from math import isclose
from numbers import Number

from ..core import json
from .filterparse import parse_filter_arg
from .utility import _nested_dicts_to_dotted_keys, _to_hashable

Expand Down
4 changes: 2 additions & 2 deletions signac/contrib/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ class DestinationExistsError(Error, RuntimeError):
Parameters
----------
destination :
The destination object causing the error.
destination : str
The destination causing the error.
"""

Expand Down
3 changes: 1 addition & 2 deletions signac/contrib/filterparse.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,10 @@
# This software is licensed under the BSD 3-Clause License.
"""Parse the filter arguments."""

import json
import sys
from collections.abc import Mapping

from ..core import json


def _print_err(msg=None):
"""Print the provided message to stderr.
Expand Down
4 changes: 3 additions & 1 deletion signac/contrib/hashing.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
import hashlib
import json

from ..synced_collections.utils import SyncedCollectionJSONEncoder

# We must use the standard library json for exact consistency in formatting


Expand All @@ -27,7 +29,7 @@ def calc_id(spec):
Encoded hash in hexadecimal format.
"""
blob = json.dumps(spec, sort_keys=True)
blob = json.dumps(spec, cls=SyncedCollectionJSONEncoder, sort_keys=True)
m = hashlib.md5()
m.update(blob.encode())
return m.hexdigest()
4 changes: 2 additions & 2 deletions signac/contrib/import_export.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
"""Provides features for importing and exporting data."""

import errno
import json
import logging
import os
import re
Expand All @@ -16,7 +17,6 @@
from tempfile import TemporaryDirectory
from zipfile import ZIP_DEFLATED, ZipFile

from ..core import json
from .errors import DestinationExistsError, StatepointParsingError
from .utility import _dotted_dict_to_nested_dicts, _mkdir_p

Expand Down Expand Up @@ -766,7 +766,7 @@ def _copy_to_job_workspace(src, job, copytree):
raise DestinationExistsError(job)
raise
else:
job._init()
job.init()
return dst


Expand Down
Loading

0 comments on commit 29c0749

Please sign in to comment.