- Add Storm commands for easily adding, deleting, and listing layer push and pull configurations. (#2071)
- Fix
layer.getPropCount()
API for universal properties. (#2073) - Add a missing async yield in
Snap.addNodes()
. (#2074) - Constrain lmdb version due to unexpected behavior in
v1.1.0
. (#2075)
- Update user docs for Storm flow control and data model references. (#2066)
- Implement tag model based pruning behavior for controlling how individual tag trees are deleted from nodes. (#2067)
- Add model interfaces for defining common sets of properties for forms, starting with some file mime metadata. (#2040)
- Add
file:mime:msdoc
,file:mime:msxls
,file:mime:msppt
, andfile:mime:rtf
forms. (#2040) - Tweak the ival normalizer to auto-expand intervals with a single element. (#2070)
- Removed the experimental
spawn
feature of the Storm runtime. (#2068)
- Add a missing async yield statement in
View.getEdgeVerbs()
. (#2069)
- Correct incorrect references to the
synapse.tools.easycert
documentation. (#2065)
- Add support for storing model metadata for tags and support for enforcing tag trees using regular expressions. (#2056)
- Add
ou:contest:url
secondary property. (#2059) - Add
synapse.lib.autodoc
to collect some Storm documentation helpers into a single library. (#2034) - Add
tag.prune
Storm command to remove parent tags when removing a leaf tag from a node. (#2062) - Update the
msgpack
Python dependency to versionv1.0.2
. (#1735) - Add logs to Cell backup routines. (#2060)
- Export the Layer iterrows APIs to the CoreApi. (#2061)
- Do not connect to Aha servers when they are not needed. (#2058)
- Make the array property
ou:org:industries
a unique array property. (#2059) - Add permission checks to the Storm
movetag
command. (#2063) - Add permissions checks to the Storm
edges.del
command. (#2064)
- Add documentation for the
synapse.tools.genpkg
utility, for loading Storm packages into a Cortex. (#2057) - Refactor the Stormtypes documentation generation to make it data driven. (#2034)
- Add support for ndef based light edge definitions in the
syn.nodes
feed API. (#2051) (#2053) - Add ISIC codes to the
ou:industry
form. (#2054) (#2055) - Add secondary properties
:loc
,:latlong
, and:place
to theinet:web:action
andinet:web:logon
forms. (#2052) - Add secondary property
:enabled
to the formit:app:yara:rule
. (#2052) - Deprecate the
file:string
andou:member
forms, in favor of using light edges for storing those relationships. (#2052)
- Allow expression statments to be used in Storm filters. (#2041)
- Add
file:subfile:path
secondary property to record the path a file was stored in a parent file. The correspondingfile:subfile:name
property is marked as deprecated. (#2043) - Make the Axon
wget()
timeout a configurable parameter. (#2047) - Add a
Cortex.exportStorm()
on the Cortex which allows for exporting nodes from a Storm query which can be directly ingested with thesyn.nodes
feed function. If the data is serialized using msgpack and stored in a Axon, it can be added to a Cortex with the newCortex.feedFromAxon()
API. A new HTTP API,/api/v1/storm/export
, can be used to get a msgpacked file using this export interface. (#2045)
- Fix issues in the Layer push and pull loop code. (#2044) (#2048)
- Add missing
toprim()
andtostr()
calls for the Stormtypes Whois guid generation helpers. (#2046) - Fix behavior in the Storm lookup mode which failed to lookup some expected results. (#2049)
- Fix
$lib.pkg.get()
return value when the package is not present. (#2050)
- Fix a variable scoping issue causing a race condition. (#2042)
- Add a Storm
wget
command which will download a file from a URL using the Cortex Axon and yieldinet:urlfile
nodes. (#2035) - Add a
--diff
option to themerge
command to enumerate changes. (#2037) - Allow StormLib Layer API to dynamically update a Layer's logedits setting. (#2038)
- Add StormLib APIs for adding and deleting extended model properties, forms and tag properties. (#2039)
- Fix an issue with the JsonStor not created nested entries properly. (#2036)
- Correct the StormType
Queue.pop()
API to properly pop and return only the item at the specified index or the next entry in the Queue. This simplifies the intent behind the.pop()
operation; and removes thecull
andwait
parameters which were previously on the method. (#2032)
- Use
resp.iter_chunked
in the Axon.wget()
API to improve compatibility with some third party libraries. (#2030) - Require the use of a msgpack based deepcopy operation in handling storage nodes. (#2031)
- Fix for ambiguous whitespace in Storm command argument parsing. (#2033)
- Add APIs to remove decommissioned services from AHA servers.
- Add (optional) explicit network parameters to AHA APIs. (#2029)
- Add cell.isCellActive() API to differentiate leaders/mirrors. (#2028)
- Add pop() method to Storm list objects. (#2027)
- Fix bug in dry-run output of new merge command. (#2026)
- Make syncIndexEvents testing more resiliant
- Make syncIndexEvents yield more often when filtering results (#2025)
- Update push/pull tests to use new waittask() API
- Raise clear errors in ambiguous use of node.tagglobs() API
- Update model docs and examples for geo:latitude and geo:longitude
- Support deref form names in storm node add expressions (#2024)
- Update tests to normalize equality comparison values (#2023)
- Added axon.size() API and storm plumbing (#2020)
- Fix active coro issue uncovered with cluster testing (#2021)
- Added (BETA) RST pre-processor to embed Storm output into RST docs. (#1988)
- Added a
merge
command to allow per-node Layer merge operations to be done. (#2009) - Updated storm package format to include a semver version string. (#2016)
- Added telepath proxy getPipeline API to minimize round-trip delay. (#1615)
- Added Node properties iteration and setitem APIs to storm. (#2011)
- Fixes for active coro API and internal layer API name fixes. (#2018)
- Allow :prop -+> * join syntax. (#2015)
- Make getFormCount() API return a primitive dictionary. (#2014)
- Make StormVarListError messages more user friendly. (#2013)
2.17.0
was not published due to CI issues.
- Allow the
matchdef
used in theLayer.syncIndexEvents()
API to match on tagprop data. (#2010)
- Properly detect and raise a client side exception in Telepath generators when the underlying Link has been closed. (#2008)
- Refactor the Layer push/push test to not reach through the Layer API boundary. (#2012)
- Add documentation for Storm raw pivot syntax. (#2007)
- Add documentation for recently added Storm commands. (#2007)
- General cleanup and clarifications. (#2007)
- Replaced the View sync APIs introduced in
v2.14.0
with Layer specific sync APIs. (#2003) - Add
$lib.regex.matches()
and$lib.regex.search()
Stormtypes APIs for performing regular expression operations against text in Storm. (#1999) (#2005) - Add
synapse.tools.genpkg
for generating Storm packages and loading them into a Cortex. (#2004) - Refactored the StormDmon implementation to use a single async task and allow
the Dmons to be restarted via
$lib.dmon.bump(iden)
. This replaces the outer task / inner task paradigm that was previously present. Also add the ability to persistently disable and enable a StomDmon. (#1998) - Added
aha://
support to thesynapse.tools.pushfile
andsynapse.tools.pullfile
tools. (#2006)
- Properly handle whitespace in keyword arguments when calling functions in Storm. (#1997)
- Fix some garbage collection issues causing periodic pauses in a Cortex due to failing to close some generators used in the Storm Command AST node. (#2001) (#2002)
- Fix scope based permission checks in Storm. (#2000)
- Add two new Cortex APIs:
syncIndexEvents
andsyncLayerEvents
useful for external indexing. (#1948) (#1996) - LMDB Slab improvements: Allow dupfixed dbs, add
firstkey
method, inline_ispo2
, add HotCount deletion. (#1948) - Add method to merge sort sorted async generators. (#1948)
- Ensure parent FQDN exists even in out-of-order node edit playback. (#1995)
- Fix an issue with the new layer push / pull code. (#1994)
- Fix an issue with the url sanitization function when the path contains
an
@
character. (#1993)
- Add a
/api/v1/active
HTTPAPI to the Cell that can be used as an unauthenticated liveliness check. (#1987) - Add
$lib.pip.gen()
Stormtypes API for ephemeral queues and bulk data access in Storm. (#1986) - Add a
$lib.model.tagprop()
Stormtypes API for retrieving Tagprop definitions. (#1990) - Add efficient View and Layer push/pull configurations. (#1991) (#1992)
- Add
getAhaUrls()
to the Aha service to prepare for additional service discovery. (#1989) - Add a
/api/v1/auth/onepass/issue
HTTPAPI for an admin to mint a one-time password for a Cell user. (#1982)
- Make
aha://
urls honor local paths. (#1985)
2.14.0
was not published due to CI issues.
- Add
$lib.pkg.get()
StormTypes function to get the Storm Package definition for a given package by name. (#1983)
- The user account provisioned by the
aha:admin
could be locked out. Now, upon startup, if they have been locked out or had their admin status removed, they are unlocked and admin is reset. (#1984)
- Prevent OverflowError exceptions which could have resulted from lift operations with integer storage types. (#1980)
- Remove
inet:ipv4
norm routine wrap-around behavior for integers which are outside the normal bounds of IPv4 addresses. (#1979) - Fix
view.add
and fork related permissions. (#1981) - Read
telepath.yaml
when using thesynapse.tools.cellauth
tool. (#1981)
This release also includes the changes from v2.12.1, which was not released due to an issue with CI pipelines.
- Add the missing API
getPathObjs
on the JsonStorCell. (#1976) - Fix the HasRelPropCond AST node support for Storm pivprop operations. (#1972)
- Fix support for the
aha:registry
config parameter in a Cell to support an array of strings. (#1975) - Split the
Cortex.addForm()
Nexus handler into two parts to allow for safe event replay. (#1978) - Stop forking a large number of child layers in a View persistence test. (#1977)
- Add the missing API
getPathObjs
on the JsonStorCell. (#1976) - Fix the HasRelPropCond AST node support for Storm pivprop operations. (#1972)
- Fix support for the
aha:registry
config parameter in a Cell to support an array of strings. (#1975)
- Add a
onload
paramter to thestormpkg
definition. This represents a Storm query which is executed every time thestormpkg
is loaded in a Cortex. (#1971) (#1974) - Add the ability, in Storm, to unset variables, remove items from
dictionaries, and remove items from lists. This is done via assigning
$lib.undef
to the value to be removed. (#1970) - Add support for SOCKS proxy support for outgoing connections from an Axon
and Cortex, using the
'http:proxy
configuration option. This configuration value must be a valid string for theaiohttp_socks.ProxyConnector.from_url()
API. The SOCKS proxy is used by the Axon when downloading files; and by the Cortex when making HTTP connections inside of Storm. (#1968) - Add
aha:admin
to the Cell configuration to provide a common name that is used to create an admin user for remote access to the Cell via the Aha service. (#1969) - Add
auth:ctor
andauth:conf
config to the Cell in order to allow hooking the construction of theHiveAuth
object. (#1969)
- Optimize Storm lift and filter queries, so that more efficient lift operations may be performed in some cases. (#1966)
- Add a
Axon.wget()
API to allow the Axon to retrieve files directly from a URL. (#1965) - Add a JsonStor Cell, which allows for hierarchical storage and retrieval of JSON documents. (#1954)
- Add a Cortex HTTP API,
/api/v1/storm/call
. This behaves like theCoreApi.callStorm()
API. (#1967) - Add
:client:host
and:server:host
secondary properties to theinet:http:request
form. (#1955) - Add
:host
and:acct
secondary properties to theinet:search:query
form. (#1955) - Add a Telepath service discovery implementation, the Aha cell. The Aha APIs are currently provisional and subject to change. (#1954)
- The Storm
cron.at
command now supports a--now
flag to create a cron job which immediately executes. (#1963)
- Fix a cleanup race that caused occasional
test_lmdbslab_base
failures. (#1962) - Fix an issue with
EDIT_NODEDATA_SET
nodeedits missing theoldv
value. (#1961) - Fix an issue where
cron.cleanup
could have prematurely deleted some cron jobs. (#1963)
- Fix a CI issue which prevented the Python
sdist
package from being uploaded to PyPi. (#1960)
The v2.10.0
Synapse release contains support for Python 3.8. Docker images
are now built using a Python 3.8 image by default. There are also Python 3.7
images available as vertexproject/synapse:master-py37
and
vertexproject/synapse:v2.x.x-py37
.
- Python 3.8 release support for Docker and PyPi. (#1921) (#1956)
- Add support for adding extended forms to the Cortex. This allows users to define their own forms using the existing types which are available in the Synapse data model. (#1944)
- The Storm
and
andor
statements now short-circuit and will return when their logical condition is first met. This means that subsequent clauses in those statements may not be executed. (#1952) - Add a mechanism for Storm Services to specify commands which may require
privilege elevation to execute. An example of this may be to allow a command
to create nodes; without managning individual permissions on what nodes a
user may normally be allowed to create. Services using this mechanism wiill
use the
storm.asroot.cmd.<<cmd name>>
hierarchy to grant this permission. (#1953) (#1958) - Add
$lib.json
Stormtypes Library to convert between string data and primitives. (#1949) - Add a
parallel
command to allow for executing a portion of a Storm query in parallel. Add abackground
command to execute a Storm query as a detached task from the current query, capturing variables in the process. (#1931) (#1957) - Add a
$lib.exit()
function to StormTypes to allow for quickly exiting a Storm query. (#1931) - Add
$lib.bytes.upload()
to Stormtypes for streaming bytes into the Axon that the Cortex is configured with. (#1945) - Add Storm commands to manage locking and unlocking deprecated model properties. (#1909)
- Add
cron.cleanup
command to make it easy to clean up completed cron jobs. (#1942) - Add date of death properties and consistently named photo secondary properties. (#1929)
- Add model additions for representing education and awards. (#1930)
- Add additional account linkages to the
inet
model for users and groups. (#1946) - Add
inet:web:hashtag
as its own form, and add:hashtags
toinet:web:post
. (#1946) - Add
lang:translation
to capture language translations of texts in a more comprehensive way than olderlang
model forms did. Thelang:idiom
andlang:trans
forms have been marked as deprecated. (#1946) - Update the
ou
model to addou:attendee
andou:contest
andou:contest:result
forms. Several secondary properties related to conference attendance have been marked deprecated. (#1946) - The
ps:persona
andps:persona:has
forms have been marked as deprecated. (#1946) - Add
ps:contactlist
to allow collecting multipleps:contact
nodes together. (#1935) - Allow the Storm Service cmdargs to accept any valid model type in the
type
value. (#1923) (#1936) - Add
>
,<
,>=
and<=
comparators forinet:ipv4
type. (#1938) - Add configuration options to the Axon to limit the amount of data which can be stored in it. Add a configuration option the Cortex to limit the number of nodes which may be stored in a given Cortex. (#1950)
- Fix a potential incorrect length for Spooled sets during fallback. (#1937)
- Fix an issue with the Telepath
Client
object caching theirMethod
andGenrMethod
attributes across re-connections of the underlyingProxy
objects. (#1939) (#1941) - Fix a bug where a temporary spool slab cleanup failed to remove all files from the filesystem that were created when the slab was made. (#1940)
- Move exceptions which do not subclass
SynErr
out ofsynapse/exc.py
. (#1947) (#1951)
- Fix an issue where a Cortex migrated from a 01x release could overwrite entries in a Layer's historical nodeedit log. (#1934)
- Fix an issue with the layer definition schema. (#1927)
- Reuse existing an existing
DateTime
object when making time strings. This gives a slight performance boost for thesynapse.lib.time.repr()
function. (#1919) - Remove deprecated use of
loop
arguments when callingasyncio
primitives. (#1920) - Allow Storm Services to define a minimum required Synapse version by the Cortex. If the Cortex is not running the minimum version, the Cortex will not load (#1900)
- Only get the nxsindx in the
Layer.storeNodeEdits()
function if logging edits. (#1926) - Include the Node iden value in the
CantDelNode
exception when attempting to delete a Node failes due to existing references to the node. (#1926) - Take advantage of the LMDB append operation when possible. (#1912)
- Fix an issues in the Telepath Client where an exception thrown by a onlink function could cause additional linkloop tasks to be spawned. (#1924)
The v2.9.0
Synapse release contains an automatic Cortex Layer data
migration. The updated layer storage format reduces disk and memory
requirements for a layer. It is recommended to test this process with a
backup of a Cortex before updating a production Cortex.
In order to maximize the space savings from the new layer storage format,
after the Cortex has been migrated to v2.9.0
, one can take a cold
backup of the Cortex and restore the Cortex from that backup. This
compacts the LMDB databases which back the Layers and reclaims disk space
as a result. This is an optional step; as LMDB will eventually re-use the
existing space on disk.
If there are any questions about this, please reach out in the Synapse Slack channel so we can assist with any data migration questions.
- Optimize the layer storage format for memory size and performance. (#1877) (#1885) (#1899) (#1917)
- Initial support Python 3.8 compatibility for the core Synapse library. Additional 3.8 support (such as wheels and Docker images) will be available in future releases. (#1907)
- Add a read only Storm option to the Storm runtime. This option prevents executing commands or Stormtypes functions which may modify data in the Cortex. (#1869) (#1916)
- Allow the Telepath Dmon to disconnect clients using a ready status. (#1881)
- Ensure that there is only one online backup of a Cell occurring at a time. (#1883)
- Added
.lower()
,.strip()
,.lstrip()
and.rstrip()
methods to the Stormtypes Str object. These behave like the Pythonstr
methods. (#1886) (#1906) - When scraping text, defanged indicators are now refanged by default. (#1888)
- Normalize read-only property declarations to use booleans in the data model. (#1887)
- Add
lift.byverb
command to allow lifting nodes using a light edge verb. (#1890) - Add netblock and range lift helpers for
inet:ipv6
type, similar to the helpers forinet:ipv4
. (#1869) - Add a
edges.del
command to bulk remove light weight edges from nodes. (#1893) - The
yield
keyword in Storm now supports iterating over Stormtypes List and Set objects. (#1898) - Add
ou:contract
,ou:industry
andit:reveng:function:strings
forms to the data model. (#1894) - Add some display type-hinting to the data model for some string fields which may be multi-line fields. (#1892)
- Add
getFormCounts()
API to the Stormtypes View and Layer objects. (#1903) - Allow Cortex layers to report their total size on disk. This is exposed in
the Stormtypes
Layer.pack()
method for a layer. (#1910) - Expose the remote Storm Service name in the
$lib.service.get()
Stormtypes API. This allows getting a service object without knowing the name of the service as it was locally added to a Cortex. Also add a$lib.service.has()
API which allows checking to see if a service is available on a Cortex. (#1908) (#1915) - Add regular expression (
~=
) and prefix matching (^=
) expression comparators that can be used with logical expressions inside of Storm. (#1906) - Promote
CoreApi.addFeedData()
calls to tracked tasks which can be viewed and terminated. (#1918)
- Fixed a Storm bug where attempting to access an undeclared variable
silently fails. This will now raise a
NoSuchVar
exception. This is verified at runtime, not at syntax evaluation. (#1916) - Ensure that Storm HTTP APIs tear down the runtime task if the remote disconnects before consuming all of the messages. (#1889)
- Fix an issue where the
model.edge.list
command could block the ioloop for large Cortex. (#1890) - Fix a regex based lifting bug. (#1899)
- Fix a few possibly greedy points in the AST code which could have resulted in greedy CPU use. (#1902)
- When pivoting across light edges, if the destination form was not a valid form, nothing happened. Now a StormRuntimeError is raised if the destination form is not valid. (#1905)
- Fix an issue with spawn processes accessing lmdb databases after a slab resize event has occurred by the main process. (#1914)
- Fix a slab teardown race seen in testing Python 3.8 on MacOS. (#1914)
- The
0.1.x
to2.x.x
Migration tool and associated Cortex sync service has been removed from Synapse in the2.9.0
release.
- Clarify user documentation for pivot out and pivot in operations. (#1891)
- Add a deprecation policy for Synapse Data model elements. (#1895)
- Pretty print large data structures that may occur in the data model documentation. (#1897)
- Update Storm Lift documentation to add the
?=
operator. (#1904)
- Module updates to support generic organization identifiers, generic advertising identifiers, asnet6 and a few other secondary property additions. (#1879)
- Update the Cell backup APIs to perform a consistent backup across all slabs for a Cell. (#1873)
- Add support for a environment variable,
SYN_LOCKMEM_DISABLE
which will disable any memory locking of LMDB slabs. (#1882)
- The
0.1.x
to2.x.x
Migration tool and and associated Cortex sync service will be removed from Synapse in the2.9.0
release. In order to move forward to2.9.0
, please make sure that any Cortexes which still need to be migrated will first be migrated to2.8.x
prior to attempting to use2.9.x
.
- Add Synapse README content to the Pypi page. This was a community contribution from https://github.com/wesinator. (#1872)
- The
0.1.x
to2.x.x
Migration tool and and associated Cortex sync service will be removed from Synapse in the2.9.0
release. In order to move forward to2.9.0
, please make sure that any Cortexes which still need to be migrated will first be migrated to2.8.x
prior to attempting to use2.9.x
. (#1880)
- Remove duplicate words in a comment. This was a community contribution from enadjoe. (#1874)
- Fix a nested Nexus log event in Storm Service deletion. The
del
event causing Storm code execution could lead to nested Nexus events, which is incongruent with how Nexus change handlers work. This now spins off the Storm code in a free-running coroutine. This does change the servicedel
semantics since any support Storm packages a service had may be removed by the time the handler executes. (#1876) - Fix an issue where the
cull
parameter was not being passed to the multiqueue properly when calling.gets()
on a Storm Types Queue object. (#1876) - Pin the
nbconvert
package to a known working version, asv6.0.0
of that package broke the Synapse document generation by changing how templates work. (#1876) - Correct
min
andmax
integer examples in tagprop documentation and tests. (#1878)
- Update tests for additional test code coverage. This was a community contribution from blackout. (#1867)
- Add implicit links to documentation generated for Storm services, to allow for direct linking inside of documentation to specific Storm commands. (#1866)
- Add future support for deprecating model elements in the Synapse data model. This support will produce client and server side warnings when deprecated model elements are used or loaded by custom model extensions or CoreModules. (#1863)
- Update
FixedCache.put()
to avoid a cache miss. This was a community contribution from blackout. (#1868) - Fix the ioloop construction to be aware of
SYN_GREEDY_CORO
environment variable to put the ioloop into debug mode and log long-running coroutines. (#1870) - Fix how service permissions are checked in
$lib.service.get()
and$lib.service.wait()
Storm library calls. These APIs now first checkservice.get.<service iden>
before checkingservice.get.<service name>
permissions. A successfulservice.get.<service name>
check will result in a warning to the client and the server. (#1871)
- Refactor an Axon unit test to make it easier to test alternative Axon implementations. (#1862)
- Fix an issue in
synapse.tools.cmdr
where it did not ensure that the users Synapse directory was created before trying to open files in the directory. (#1860) (#1861)
- Fix an incorrect statement in our documentation about the intrinsic Axon that a Cortex creates being remotely accessible. (#1862)
- Add Telepath and HTTP API support to set and remove global Storm variables. (#1846)
- Add Cell level APIs for performing the backup of a Cell. These APIs are exposed inside of a Cortex via a Storm Library. (#1844)
- Add support for Cron name and doc fields to be editable. (#1848)
- Add support for Runtime-only (
runt
) nodes in the PivotOut operation (-> *
). (#1851) - Add
:nicks
and:names
secondary properties tops:person
andps:persona
types. (#1852) - Add a new
ou:position
form and a few associated secondary properties. (#1849) - Add a step to the CI build process to smoke test the sdist and wheel packages before publishing them to PyPI. (#1853)
- Add support for representing
nodedata
in the command hinting for Storm command implementations and expose it on thesyn:cmd
runt nodes. (#1850) - Add package level configuration data to Storm Packages in the
modconf
value of a package definition. This is added to the runtime variables when a Storm package is imported, and includes thesvciden
for packages which come from Storm Services. (#1855) - Add support for passing HTTP params when using
$lib.inet.http.*
functions to make HTTP calls in Storm. (#1856) - Log Storm queries made via the
callStorm()
andcount()
APIs. (#1857)
- Fix an issue were some Storm filter operations were not yielding CPU time appropriately. (#1845)
- Remove a reference to deprecated
eval()
API from quickstart documentation. (#1858)
- Support
+hh:mm
and+hh:mm
timezone offset parsing when normalizingtime
values. (#1833) - Enable making mirrors of Cortex mirrors work. (#1836)
- Remove read-only properties from
inet:flow
andinet:http:request
forms. (#1840) - Add support for setting nodedata and light edges in the
syn.nodes
ingest format. (#1839) - Sync the LMDB Slab replay log if it gets too large instead of waiting for a force commit operation. (#1838)
- Make the Agenda unit tests an actual component test to reduce test complexity. (#1837)
- Support glob patterns when specifying files to upload to an Axon with
synapse.tools.pushfile
. (#1837) - Use the node edit metadata to store and set the
.created
property on nodes, so that mirrors of Cortexes have consistent.created
timestamps. (#1765) - Support parent runtime variables being accessed during the execution of a
macro.exec
command. (#1841) - Setting tags from variable values in Storm now calls
s_stormtypes.tostr()
on the variable value. (#1843)
- The Storm
tree
command now catches the SynapseRecursionLimitHit
error and raises aStormRuntimeError
instead. TheRecursionLimitHit
being raised by that command was, in practice, confusing. (#1832) - Resolve memory leak issues related to callStorm and Base object teardowns with exceptions. (#1842)
- Add performance oriented counting APIs per layer, and expose them via Stormtypes. (#1813)
- Add the ability to clone a layer, primarily for benchmarking and testing purposes. (#1819)
- Update the benchmark script to run on remote Cortexes. (#1829)
- Sanitize passwords from Telepath URLs during specific cases where the URL may be logged. (#1830)
- Fix a few typos in docstrings. (#1831)
- Refactor the Nexus to remove leadership awareness. (#1785)
- Add support for client-side certificates in Telepath for SSL connections. (#1785)
- Add multi-dir support for CertDir. (#1785)
- Add a
--no-edges
option to the Stormgraph
command. (#1805) - Add
:doc:url
to thesyn:tag
form to allow recording a URL which may document a tag. (#1805) - Add
CoreApi.reqValidStorm()
and a/api/v1/reqvalidstorm
Cortex HTTPAPI endpoint to validate that a given Storm query is valid Storm syntax. (#1806) - Support Unicode white space in Storm. All Python s (Unicode white space + ASCII separators) is now treated as white space in Storm. (#1812)
- Refactor how StormLib and StormPrim objects access their object locals, and add them to a global registry to support runtime introspection of those classes. (#1804)
- Add smoke tests for the Docker containers built in CircleCI, as well as adding Docker healthchecks to the Cortex, Axon and Cryotank images. (#1815)
- Initialize the names of the default view and layer in a fresh Cortex to
default
. (#1814) - Add HTTPAPI endpoints for the Axon to upload, download and check for the existend of files. (#1817) (#1822) (#1824) (#1825)
- Add a
$lib.bytes.has()
API to check if the Axon a Cortex is configured with knows about a given sha256 value. (#1822) - Add initial model for prices, currences, securities and exchanges. (#1820)
- Add a
:author
field to theit:app:yara:rule
form. (#1821) - Add an experimental option to set the NexusLog as a
map_async
slab. (#1826) - Add an initial transportation model. (#1816)
- Add the ability to dereference an item, from a list of items, in Storm via index. (#1827)
- Add a generic
$lib.inet.http.request()
Stormlib function make HTTP requests with arbitrary verbs. (#1828)
- Fix an issue with the Docker builds for Synapse where the package was not being installed properly. (#1815)
- Update documentation for deploying Cortex mirrors. (#1811)
- Add automatically generated documentation for all the Storm
$lib...
functions and Storm Primitive types. (#1804) - Add examples of creating a given Form to the automatically generated documentation for the automatically generated datamodel documentation. (#1818)
- Add additional documentation for Cortex automation. (#1797)
- Add Devops documentation for the list of user permissions relevant to a Cell, Cortex and Axon. (#1823)
- Update the Storm
scrape
command to makerefs
light edges, instead ofedge:refs
nodes. (#1801) (#1803) - Add
:headers
and:response:headers
secondary properties to theinet:http:request
form as Array types, so that requests can be directly linked to headers. (#1800) - Add
:headers
secondary property to theinet:email:messaage
form as Array types, so that messages can be directly linked to headers. (#1800) - Add additional model elements to support recording additional data for binary reverse engineering. (#1802)
- Prohibit invalid rules from being set on a User or Role object. (#1798)
- Add
ps.list
andps.kill
commands to Storm, to allow introspecting the runtime tasks during (#1782) - Add an
autoadd
mode to Storm, which will extract basic indicators and make nodes from them when executed. This is a superset of the behavior in thelookup
mode. (#1795) - Support skipping directories in the
synapse.tools.backup
tool. (#1792) - Add prefix based lifting to the Hex type. (#1796)
- Fix an issue for prop pivot out syntax where the source data is an array type. (#1794)
- Add Synapse data model background on light edges and update the Storm data modification and pivot references for light edges. (#1784)
- Add additional terms to the Synapse glossary. (#1784)
- Add documentation for additional Storm commands. (#1784)
- Update documentation for Array types. (#1791)
- Add some small enhancements to the Cortex benchmarking script. (#1790)
- Fix an error in the help for the
macro.del
command. (#1786) - Fix rule indexing for the
synapse.tools.cellauth
tool to correctly print the rule offsets. (#1787) - Remove extraneous output from the Storm Parser output. (#1789)
- Rewrite the language (and private APIs) for the Storm
model.edge
related commands to remove references to extended properties. That was confusing language which was unclear for users. (#1789) - During 2.0.0 migrations, ensure that Cortex and Layer idens are unique; and make minimum 0.1.6 version requirement for migration. (#1788)
- The Axon test suite was missing a test for calling
Axon.get()
on a file it did not have. This is now included in the test suite. (#1783)
- Improve Synapse devops documentation hierarchy. Add note about Cell directories being persistent. (#1781)
- Add a
postAnit()
callback to thesynapse.lib.base.Base()
object which is called after the__anit__()
call chain is completed, but beforeBase.anit()
returns the object instance to the caller. This is used by the Cell to defer certain Nexus actions until the Cell has completed initializing all of its instance attributes. (#1768) - Make
synapse.lib.msgpack.en()
raise aSynErr.NotMsgpackSafe
exception instead of passing through the exception raised by msgpack. (#1768)
- Add a missing
toprim()
call in$lib.globals.set()
. (#1778) - Fix an issue in the quickstart documentation related to permissions. Thank you
enadjoe
for your contribution. (#1779) - Fix an Cell/Cortex startup issue which caused errors when starting up a Cortex when the last Nexus event was
replayed. This has a secondary effect that Cell implementers cannot be making Nexus changes during the
__anit__
methods. (#1768)
- Add a minimal Storm Service example to the developer documentation. (#1776)
- Reorganize the Synapse User Guide into a more hierarchical format. (#1777)
- Fill out additional glossary items. (#1780)
- Disallow command and bare string contensts from starting with
//
and/*
in Storm syntax. (#1769)
- Fix an issue in the autodoc tool which failed to account for Storm Service commands without cmdargs. (#1775)
- Add information about light edges to graph carving output. (#1762)
- Add a
geo:json
type andgeo:place:geojson
property to the model. (#1759) - Add the ability to record documentation for light edges. (#1760)
- Add the ability to delete and set items inside of a MultiQueue. (#1766)
- Refactor
v2.0.0
changelog documentation. (#1763) - Add Vertex branding to the Synapse documentation. (#1767)
- Update Backups documentation in the Devops guide. (#1764)
- Update the autodoc tool to generate documentation for Cell confdefs and StormService information. (#1772)
- Update to separate the devops guides into distinct sections. (#1772)
- Add documentation for how to do boot-time configuration for a a Synapse Cell. (#1772)
- Remove duplicate information about backups. (#1774)
Initial 2.0.0 release. See :ref:`200_changes` for notable new features and changes, as well as backwards incompatible changes.
For the Synapse 0.1.x changelog, see 01x Changelog located in the v0.1.x documentation.