improve the check of the arguments in "generate_sovrin_pool_transactions" #162

AlexanderShekhovcov · 2017-05-11T16:28:45Z

The changes:

clean and validate each of IP addresses for parameter --ips
check --nodes greater than zero
check --envName can be only 'test' or 'live'

…ons"

* changes for stp * removing redundant close in orientdb socket * adding tests for catchup scenario and 3 phase message outside water mark * fixing import in test * Fix 'infinit' looping on receiving messages when their amount is very high (#97) * fix 'infinit' looping on receiving messages when their amount is very high * move quotas to config * make zstack get params from config only once * add test for quota * increase message quotas to 100 * Feature Added: Jenkins (#99) * Moved util methods to NetwrorkInterfaces; fixes in raet stack * fix key generation * Hotfix: Deps (#101) * - Added a custom jsonpickle handler for enums to avoid the issue jsonpickle/jsonpickle#135 in jsonpickle 0.9.2 which is the canonical version for Ubuntu 16.04. (#98) - Corrected Cli.restoreWalletByPath method. * Merge branches 'stp' and 'zmq' of github.com:evernym/plenum into stp * handling infinite loop problem and fixing test delay * fixing import * bugfix in test * raising timeout * merge * change in ordered * move connection related exceptions to stp project * fix imports of moved exceptions * fixed dependencies * fixing name * temporarily disabling tests on Windows * renamed init_plenum_raet_keep to init_plenum_keys * removing unused code and changing script mode * Introduced failing of the pytets session in case any unexpected warnings have been made (#104) * changes to troubleshoot testing for coro not awaited * added support for warning checking across all tests * fixed scope for warnfilters fixture * - Updated the warnings filter. Made the rules for ResourceWarning: "unclosed file", ResourceWarning: "unclosed.*socket\.socket" and RuntimeWarning: "coroutine.*was never awaited" global since they are made by the garbage collector in unpredictable points. - Sorted the rules in the warnings filter alphabetically. * Specialized the warnings filter rule for RuntimeWarning about a coroutine was never awaited (to "msgAll" coroutine). * Added an explicit call of the garbage collector to the tear-down phase of "warncheck" fixture in order to ensure that warnings will be generated and recorded for all the unclosed resources. * Fixed the issue with that a coroutine "msgAll" object was never awaited. Removed the corresponding "ignore" rule from the warnings filter. * - Removed the rules not being actuated currently from the warnings filter. - Replaced use of line numbers with use of message patterns in the warnings filter rules. * Corrected the message pattern for a rule from the warnings filter. * Added an "ignore" rule for ResourceWarning: "unclosed event loop" to the warnings filter. * Returned back the warnings filter rules for DeprecationWarnings in jsonpickle package. Replaced use of line numbers with use of message patterns in these rules. * adding warning * removing looper and checking while closing orientdb * increasing a timeout * replacing warn with warning methods * fixed stp dependency * fixed tests * skip failing test, created SOV-881. * adding load script * updating setup * changing logs to print alias * setting resource limit for tests * handling exception while raising resource limits * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring (#106) * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring * removed unwanted error code, refactored list keyrings command so that it correctly works for agent cli * removed unused imports * making CLI accept hex seed of size 64 * move logging to stp repo * mitigating key error in pool manager * changing timeouts so the diconnect check happens twice * handling unavailabily of resource module * some corrects to gc * setting the lastOrdered sequence number of the replica after election * fixing obsolete imports * adding temporary info logs to show up during load testing, problem identified for PREPREPARE getting rejects, state tree has different root at beginning for different nodes * reverting config changes * using ordered json encoder * add declarations of common request handling methods to ReqHandler * refactor method commit * add docstring for ReqHandler, rearrange methods * rename *ReqHandlers to *RequestHandlers * add TODOs about fixing hierarchy of State and PruningState * rename validateReq to validate * add config=None argument to validate method to make sub and super signatures match * rename applyReq to apply * add super initialization, update usage of super methods * update docstring of commit * remove commitReq since it does the same as commit * adding timeout to test utility method * changes for load testing * sanitising log messages * Versioning fix (#113) * changes for request handlers * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * changing to rlp encoding * moving error codes to stp * upgrading version of stp * req_handler changes * fixing import * temp commit * Stp (#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * minor changes to serialization * Plenum test runner fix (#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * updating apply of request handler * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * leveldb HashStore * parametrize test timeouts (#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * removing commented code and fixing bugs in pool request handler * lowering Delta in pool_transactions conftest * fix jenkins build (#123) * fixed deps in setup.py * added pypi publishing * updated deps * updated deps stp-dev -> stp-perf-imp * removed unnecessary sources * updated setup.py to hold correct deps names and self name for publishing * fixed plenum for the latest stp (#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * sending last pre-prepare sequence number in consistency proof so already processed request can be ignored, re-enabling tests and re-adding proper timeouts * Feature Added: Deb build * Feature Added: Deb build * Do view change if a primary is disconnected (#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * processing stashed ordered requests after all ledgers have caught up and applying any request if mode changed while processing and removing obsolete code from pool manager * Unclosed file warnings (#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * refactoring, documentation and updating setup to get rid of pyorient * refactoring * Updated ledger dep * deprecating orientdb * check state equality too while checking ledger equality in tests * removing obsolete test helper method * Agent generalization (#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * Add default values for role and verkey in state and fix test timeouts * refactor * import logger * updated to use new API from shared lib * updated to use new API from shared lib * handle pyorient import * handle pyorient import * increment stp version * Enabled the tests in plenum.test.test_log_rotation module. (#135) * increment stp version * close files in tests * check for close method before calling it (#136) * minor fix in batch creation logic and caching primary replica no in node to avoid recomputation * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/common/test_network_setup.py # plenum/server/node.py # plenum/test/test_node.py # setup.py * add missing __init__ for member package * added missed __init__.py for common/member * skip temporarily a failing test * fix primary disconnection and new primary not being same as old for master instance * add more tests for primary disconnection, same primary not elected in the next view and refactor * add new test for minor network glitch with primary * add missing parameter to bootstrap test network (#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * minor change in a test and removing sip statements * fix the catchup tests (#140) * make tests a little robust * fix test, check for disconnections after processing sent and received messages and little documentation * Close SOV-976 (#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/test/node_catchup/test_new_node_catchup.py # setup.py * up stp to 0.1.42 (#148) * refactor, enable test, adding timeout for view change and create remotes as soon as nodestack starts * fix broken test * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (#149) * increment stp * increment ledger * up stp to 0.1.14 * move state and kv storage to 'state' repo * get rid of orientdb code * fix tests * fix generation of trustee txns * fix genesis txns for trustee * fix generation of trustee txns (#151) * remove obsolete code * client to attempt establishing connections with nodes on start * refactor monitor.isMasterThroughputTooLow * update method for forcing view change cna checking it; intoduce wait method for it * use provoke_and_wait_for_view_change instead of eventually and check * remove unused dependency and skip tests - removed unused dependencies - minor fixes - skip tests that fail in master too (we have separate tickets for this) * increment ledger * increment ledger * enable debug logging in tests * using reverted changes in state interface * increment state version * Removed redundant copying of the file "pool_transactions_sandbox" to the directory "~/.plenum". (It is already done using "data_files" argument of "setup" function call.) (#152) * speed up tests (#147) * speed up test_nodes_with_pool_txns.py tests * reduce the timeout in checkNodesConnected * speed up the pool_transactions test * fix the timeout in test_delay.py * fix the timeout in test_primary_election_case5.py * speed up monitoring tests * revert changes in test_instance_change_with_Delta.py * each test's running time is limited by TestRunningTimeLimitSec * increase pool interconnection timeout on zstack retry timeout * fix timeout in test_delay.py * fix tests, make them faster * speedup the tests, tidy test timeouts * add a timeout into waits * bump stp-dev==0.1.43 * Unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well (#153) * unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well * decrease number of nodes in test_instance_cannot_become_active_with_less_than_four_servers to 13 * fix change back HA, SOV-908 (#154) * fix * Fix testChangeNodeHaForNonPrimary (#157) * unskip test_change_non_primary_node_ha * fix waits name - it was renamed * verify genesis ledgers only if such option set in config * use primaryName instead of isPrimary to check that election is done * add numOfReelections parameter to waits.expectedPoolElectionTimeout * refactor test_node.py * set custom election timeout (numOfReelections=3) in changeNodeHa * remove debug prints * check VerKey is base58 for NODE txn, SOV-988 (#158) * client using a method which makes the code more testable, update to an old test with more checks and removing async from a method where it was not needed * Fix test testChangeNodeHaForPrimary (#160) * unskip testChangeNodeHaForPrimary * simplify conditional in changeNodeHa * node which is going down should not participate in a view change * change formating in testChangeNodeHaForPrimary * refactor to make method overridable and adding an argument to a test function * updated pypi package name for state repo and its version (#159) * Squashed commit of the following: create LedgerInfo class and replace collections of LedgerManager by fields * rename LedgerManager.ledgers to ledgerRegistry * fix usages of LedgerManager.ledgerRegistry * use correct attribute and renamed method * bring change from master and unskip a test * use correct attribute and set state to be committed when received from catchup * formatting * up ledger-3pc-batch version to 0.2.16 * up stp-3pc-batch version to 0.1.15 * improve the check of the arguments in "generate_sovrin_pool_transactions" (#162) * check client input for the NODE txn (#161) * fix testInstChangeWithLowerRatioThanDelta - decrease number of messages in a batch * decrease timeout to fix provoke_and_wait_for_view_change * fixing test * Fix post 3pc-batch merge (#163) * remove development options from Jenkinsfile * skip some broken tests * Update versions and skip windows build steps (#164) * switch off windows related build steps * update versions of ledger and stp * fix name * up ledger and stp version * skip test testNodeRequestingTxns * using correct timeout in tests * move some enums to correct location, using ledger's string encoding methods and test for stashing requests * bump dependency version * fix timeout in tests * make order of ledger sync customizable, add tranformer for transactions before adding to ledger, update seqno map db for transactions from catchup and update tests to check this * temporarily change config variable * fix regression where node starts catchup process if sent a consistency proff * bumping dependency version * bumping dependency version * bumping dependency version * Fix redundant reconnections and refactor (#165) * refactor retryForExpected * use maintainConnections in retryForExpected instead of connect * refactor resendRequests * up stp version to 0.1.49 * replace CLIENT_REQACK_TIMEOUT by CLIENT_REPLY_TIMEOUT when calling _filterExpected for REPLYs * up ledger to 0.2.19 * Add new helper-methods for conversion base58 to hex (#166) * increment state-trie and ledger versions (#168) * Make changes to support ChunkedFileStore (#167) * update _defaultStore of Ledger to make it correspond to original one * remove initialization of domain ledger - now it is done by ledger and file store it uses * replace manual copying of transaction files by usage of defaultFile * increase timeout for testNodeRequestingTxns * skip test testInstChangeWithLowerRatioThanDelta because it fails intermittently * up ledger version to 0.2.28 * Bump dependency version * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node (#172) * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node * Fixed failing tests * Fixed unsed imports * Update conftest.py * Increased test time global limit * Reverted timeouts * Added logs * Fixed filtering CPs * Fixed filtering CPs * Fixed filtering CPs * Input validation (#170) * Input sanitization: Add base logic and test cases * Input sanitization: add NODE and NYM txn, fixes * Input sanitization: implement node and client messages * roll away node to node validation * role field is option for NYM txn * fix tests * fixes for sovrin-node * implement validation for merkle root hash * uncomment new ConsistencyProof message implementation * add "nullable" property for FieldBase * fix usage of alphabet * add JsonField * add validation on message level * use hash size range instead of one value; use base58.alphabet instead of own list * fix usage hasSizes; made error comment more verbose * uncomment new implementation LedgerStatus message * uncomment new implementation for Prepare message and fix fields types * roll away node to node checkes * check each node connected * improve log messages * fix testAdd2NewNodes * Improvement of validation rules (#173) * fix Node txn, INDY-9 * add tests for the Node operation * - Provided all the disabled tests with the ticket references. (#176) - Enabled the tests testChangeNodeHaForPrimary and testChangeNodeHaForNonPrimary on non-Windows platforms. - Commented out parametrizers at the disabled tests in plenum.test.input_validation.test_common_checks module. * Make SERVICES field required for add NODE txn * Added another test scenario for catchup (#178) * Added another test scenario for catchup fix * stable release

* setting resource limit for tests * handling exception while raising resource limits * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring (#106) * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring * removed unwanted error code, refactored list keyrings command so that it correctly works for agent cli * removed unused imports * making CLI accept hex seed of size 64 * move logging to stp repo * mitigating key error in pool manager * changing timeouts so the diconnect check happens twice * handling unavailabily of resource module * some corrects to gc * setting the lastOrdered sequence number of the replica after election * fixing obsolete imports * adding temporary info logs to show up during load testing, problem identified for PREPREPARE getting rejects, state tree has different root at beginning for different nodes * reverting config changes * using ordered json encoder * add declarations of common request handling methods to ReqHandler * refactor method commit * add docstring for ReqHandler, rearrange methods * rename *ReqHandlers to *RequestHandlers * add TODOs about fixing hierarchy of State and PruningState * rename validateReq to validate * add config=None argument to validate method to make sub and super signatures match * rename applyReq to apply * add super initialization, update usage of super methods * update docstring of commit * remove commitReq since it does the same as commit * adding timeout to test utility method * changes for load testing * sanitising log messages * Versioning fix (#113) * changes for request handlers * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * changing to rlp encoding * moving error codes to stp * upgrading version of stp * req_handler changes * fixing import * temp commit * Stp (#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * minor changes to serialization * Plenum test runner fix (#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * updating apply of request handler * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * leveldb HashStore * parametrize test timeouts (#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * removing commented code and fixing bugs in pool request handler * lowering Delta in pool_transactions conftest * fix jenkins build (#123) * fixed deps in setup.py * added pypi publishing * updated deps * updated deps stp-dev -> stp-perf-imp * removed unnecessary sources * updated setup.py to hold correct deps names and self name for publishing * fixed plenum for the latest stp (#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * sending last pre-prepare sequence number in consistency proof so already processed request can be ignored, re-enabling tests and re-adding proper timeouts * Feature Added: Deb build * Feature Added: Deb build * Do view change if a primary is disconnected (#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * processing stashed ordered requests after all ledgers have caught up and applying any request if mode changed while processing and removing obsolete code from pool manager * Unclosed file warnings (#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * refactoring, documentation and updating setup to get rid of pyorient * refactoring * Updated ledger dep * deprecating orientdb * check state equality too while checking ledger equality in tests * removing obsolete test helper method * Agent generalization (#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * Add default values for role and verkey in state and fix test timeouts * refactor * import logger * updated to use new API from shared lib * updated to use new API from shared lib * handle pyorient import * handle pyorient import * increment stp version * Enabled the tests in plenum.test.test_log_rotation module. (#135) * increment stp version * close files in tests * check for close method before calling it (#136) * minor fix in batch creation logic and caching primary replica no in node to avoid recomputation * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/common/test_network_setup.py # plenum/server/node.py # plenum/test/test_node.py # setup.py * add missing __init__ for member package * added missed __init__.py for common/member * skip temporarily a failing test * fix primary disconnection and new primary not being same as old for master instance * add more tests for primary disconnection, same primary not elected in the next view and refactor * add new test for minor network glitch with primary * add missing parameter to bootstrap test network (#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * minor change in a test and removing sip statements * fix the catchup tests (#140) * make tests a little robust * fix test, check for disconnections after processing sent and received messages and little documentation * Close SOV-976 (#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/test/node_catchup/test_new_node_catchup.py # setup.py * up stp to 0.1.42 (#148) * refactor, enable test, adding timeout for view change and create remotes as soon as nodestack starts * fix broken test * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (#149) * increment stp * increment ledger * up stp to 0.1.14 * move state and kv storage to 'state' repo * get rid of orientdb code * fix tests * fix generation of trustee txns * fix genesis txns for trustee * fix generation of trustee txns (#151) * remove obsolete code * client to attempt establishing connections with nodes on start * refactor monitor.isMasterThroughputTooLow * update method for forcing view change cna checking it; intoduce wait method for it * use provoke_and_wait_for_view_change instead of eventually and check * remove unused dependency and skip tests - removed unused dependencies - minor fixes - skip tests that fail in master too (we have separate tickets for this) * increment ledger * increment ledger * enable debug logging in tests * using reverted changes in state interface * increment state version * Removed redundant copying of the file "pool_transactions_sandbox" to the directory "~/.plenum". (It is already done using "data_files" argument of "setup" function call.) (#152) * speed up tests (#147) * speed up test_nodes_with_pool_txns.py tests * reduce the timeout in checkNodesConnected * speed up the pool_transactions test * fix the timeout in test_delay.py * fix the timeout in test_primary_election_case5.py * speed up monitoring tests * revert changes in test_instance_change_with_Delta.py * each test's running time is limited by TestRunningTimeLimitSec * increase pool interconnection timeout on zstack retry timeout * fix timeout in test_delay.py * fix tests, make them faster * speedup the tests, tidy test timeouts * add a timeout into waits * bump stp-dev==0.1.43 * Unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well (#153) * unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well * decrease number of nodes in test_instance_cannot_become_active_with_less_than_four_servers to 13 * fix change back HA, SOV-908 (#154) * Fix testChangeNodeHaForNonPrimary (#157) * unskip test_change_non_primary_node_ha * fix waits name - it was renamed * verify genesis ledgers only if such option set in config * use primaryName instead of isPrimary to check that election is done * add numOfReelections parameter to waits.expectedPoolElectionTimeout * refactor test_node.py * set custom election timeout (numOfReelections=3) in changeNodeHa * remove debug prints * check VerKey is base58 for NODE txn, SOV-988 (#158) * client using a method which makes the code more testable, update to an old test with more checks and removing async from a method where it was not needed * Fix test testChangeNodeHaForPrimary (#160) * unskip testChangeNodeHaForPrimary * simplify conditional in changeNodeHa * node which is going down should not participate in a view change * change formating in testChangeNodeHaForPrimary * refactor to make method overridable and adding an argument to a test function * updated pypi package name for state repo and its version (#159) * Squashed commit of the following: create LedgerInfo class and replace collections of LedgerManager by fields * rename LedgerManager.ledgers to ledgerRegistry * fix usages of LedgerManager.ledgerRegistry * use correct attribute and renamed method * bring change from master and unskip a test * use correct attribute and set state to be committed when received from catchup * formatting * up ledger-3pc-batch version to 0.2.16 * up stp-3pc-batch version to 0.1.15 * improve the check of the arguments in "generate_sovrin_pool_transactions" (#162) * check client input for the NODE txn (#161) * fix testInstChangeWithLowerRatioThanDelta - decrease number of messages in a batch * decrease timeout to fix provoke_and_wait_for_view_change * fixing test * Fix post 3pc-batch merge (#163) * remove development options from Jenkinsfile * skip some broken tests * Update versions and skip windows build steps (#164) * switch off windows related build steps * update versions of ledger and stp * fix name * up ledger and stp version * skip test testNodeRequestingTxns * using correct timeout in tests * move some enums to correct location, using ledger's string encoding methods and test for stashing requests * bump dependency version * fix timeout in tests * make order of ledger sync customizable, add tranformer for transactions before adding to ledger, update seqno map db for transactions from catchup and update tests to check this * temporarily change config variable * fix regression where node starts catchup process if sent a consistency proff * bumping dependency version * bumping dependency version * bumping dependency version * Fix redundant reconnections and refactor (#165) * refactor retryForExpected * use maintainConnections in retryForExpected instead of connect * refactor resendRequests * up stp version to 0.1.49 * replace CLIENT_REQACK_TIMEOUT by CLIENT_REPLY_TIMEOUT when calling _filterExpected for REPLYs * up ledger to 0.2.19 * Add new helper-methods for conversion base58 to hex (#166) * increment state-trie and ledger versions (#168) * Make changes to support ChunkedFileStore (#167) * update _defaultStore of Ledger to make it correspond to original one * remove initialization of domain ledger - now it is done by ledger and file store it uses * replace manual copying of transaction files by usage of defaultFile * increase timeout for testNodeRequestingTxns * skip test testInstChangeWithLowerRatioThanDelta because it fails intermittently * up ledger version to 0.2.28 * Bump dependency version * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node (#172) * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node * Fixed failing tests * Fixed unsed imports * Update conftest.py * Increased test time global limit * Reverted timeouts * Added logs * Fixed filtering CPs * Fixed filtering CPs * Fixed filtering CPs * Input validation (#170) * Input sanitization: Add base logic and test cases * Input sanitization: add NODE and NYM txn, fixes * Input sanitization: implement node and client messages * roll away node to node validation * role field is option for NYM txn * fix tests * fixes for sovrin-node * implement validation for merkle root hash * uncomment new ConsistencyProof message implementation * add "nullable" property for FieldBase * fix usage of alphabet * add JsonField * add validation on message level * use hash size range instead of one value; use base58.alphabet instead of own list * fix usage hasSizes; made error comment more verbose * uncomment new implementation LedgerStatus message * uncomment new implementation for Prepare message and fix fields types * roll away node to node checkes * uncomment some new message implementations * use HexField instead of MerkleRootHashField for state_root and rxn_root in PrePrepare * make message about hash length in MerkleRootField more verbose * limit timestamp field by 253402290000.0 * modify randomString to remove length limitation * add tests for merkle root field validation * make randomString work not only with indexable containers * refactor tests for merkle root * fix timestamp validation * add test for timestamp validation * update validation of timestamp * check each node connected * improve log messages * fix testAdd2NewNodes * Improvement of validation rules (#173) * fix Node txn, INDY-9 * add tests for the Node operation * add checking that number of init parameters of message is equal to number of fields declared in schema * use OrderedDict in __dict__ * add test for merkle, fix logic * use OrderedDict for _fields and update all realted methods * - Provided all the disabled tests with the ticket references. (#176) - Enabled the tests testChangeNodeHaForPrimary and testChangeNodeHaForNonPrimary on non-Windows platforms. - Commented out parametrizers at the disabled tests in plenum.test.input_validation.test_common_checks module. * more removal of op to the top since it can affect checking of a number of args * Make SERVICES field required for add NODE txn * Added another test scenario for catchup (#178) * override items, keys, values methods of Mapping in MessageBase to make it compatible * Added another test scenario for catchup fix * clean MessageBase * INDY-73: Removed a commented out @skipif decorator (#177) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * Add base58 str class; destnode; destnym; * use - as default ppSeqNo instead of -1 * overtride __ftr__ for MessageBase * comment checking of presence of SERVICE field * move alphabet validation from _specific_validation to avoid accidental overriding of it * INDY-73: Added ticket references to test stubs (#180) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * - Added ticket references to the not implemented tests which had not contained them yet. - Uncommented testStopScriptIfNodeIsRunning and disabled it since it fails. * Added raising NotImplementedError to not implemented tests. * Move all specific checks to from validate method * Eliminate multiple inheritance * Enabled and updated testNodesConnectsWhenOneNodeIsLate (#181) * Enabled and updated testNodesConnectsWhenOneNodeIsLate * updated assertions logic * INDY-109 (#184) * updates to test helpers and change in forwarding requests logic (#187) * Use secure random number generator for randomString() Current usage of random.choice() is not secure so replaced that with libsodium provided randombytes_uniform() which is secure and also available on all platforms where libsodium is available * Fixes for INDY-71 (#185) * Refactor check_endpoint_valid func * Fix validation, INDY-71 * Fix imports * Do not connect to a new, non-active node * Remove confusing comments * Minor refactor * Fix test testAddInactiveNodeThenActivate * Validation for view change and 3pc messages (#193) * add exception for type error and raise it in MessageValidator * make error message in ChooseField more verbose * uncomment new implementations of Reelection, Primary and InstanceChange messages * update and fix testInstanceChangeMsgTypeChecking * fix testDiscardInstChngMsgFrmPastView * use NonNegativeField instead of TimestampFied inTieAmountField for its seconds part * make name pf test for merkle root field more standard * move tests for fields to own package * add missing empty line * remove ununsed imports * refactor test for timestamp * remove todo about code duplication since it is not valid anymore * add tests for some field types * make IterableField check type of inner field and that it is not none * add test for hex field * add test for ledger id * add test for request id field * override __eq__ for MessageBase * use NonNegativeNumberField instead of TimestampField in RequestIdentifierField * use IdentifierField instead of NonEmptyStringField in RequestIdentifierField * update test for request identifier * add test for IdentifierField * update test for RequestIdentifier field * implement validation for IdentifierField and reimplement DestNodeField using it * uncomment new implementations for Propagate, Prepare, Commit, Ordered * Refactor base58 check; add Verkey check; dest node; dest nym; merkelroot; * move test for base58 field to field validation package * remove unused import * add checking of max length hash to merkle tree root field test * update test for identifier field * update test for request identifier field * fix formatting * fix checking of zero * remove unused commented line * rename test_non_negative_number to test_positive_number * Make verkey validator accept empty strings * Add super's __init__ params to child calsses * Improve getMatchedHelpableMsg logic (#195) * Use original batch parameters in all tests except the ones that use batches and use alternate of eventuallyAll to check a collection of functions under a timeout (#190) * updates to test helpers and change in forwarding requests logic * overriding batch params in tests * use correct timeouts in tests * accounting for nomination delay * Disable view change (#196) * repaired calls to deprecated methods * Added unsafe 'disable_view_change' for testing - Added 'unsafe' attribute to default config. "unsafe" is a set of attributes that can set certain behaviors that are not safe, for example, 'disable_view_change' disables view changes from happening. This might be useful in testing scenarios, but never in a live network. - Added 'unsafe' behavior 'disable_view_change' which causes Monitor to always return false when asked if master instance performance is degraded. - Refactored view change fixture so it could be reused to test new 'disable_view_change' behavior. * fixed import for missed fixture in test * Fix validation types (#197) * Enable and update testNodeDiscardMessageFromUnknownView (#194) * added equality method for MessageBase * added ability to cancel created delayers * enabled and updated testNodeDiscardMessageFromUnknownView * updated test to recent changes in node logic * reverted recent changes not necessary for now * Catch up demoted (#201) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * Fixes for dead catchup (#202) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * up stp version to 0.1.56 * up stp version to 0.1.57 * fixed tests * fix test * Disable the test: INDY-147. * fix README * fix setup.py * up stp version to 0.1.10

* rename validateReq to validate * add config=None argument to validate method to make sub and super signatures match * rename applyReq to apply * add super initialization, update usage of super methods * update docstring of commit * remove commitReq since it does the same as commit * adding timeout to test utility method * changes for load testing * sanitising log messages * Versioning fix (#113) * changes for request handlers * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * changing to rlp encoding * moving error codes to stp * upgrading version of stp * req_handler changes * fixing import * temp commit * Stp (#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * minor changes to serialization * Plenum test runner fix (#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * updating apply of request handler * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * leveldb HashStore * parametrize test timeouts (#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * removing commented code and fixing bugs in pool request handler * lowering Delta in pool_transactions conftest * fix jenkins build (#123) * fixed deps in setup.py * added pypi publishing * updated deps * updated deps stp-dev -> stp-perf-imp * removed unnecessary sources * updated setup.py to hold correct deps names and self name for publishing * fixed plenum for the latest stp (#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * sending last pre-prepare sequence number in consistency proof so already processed request can be ignored, re-enabling tests and re-adding proper timeouts * Feature Added: Deb build * Feature Added: Deb build * Do view change if a primary is disconnected (#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * processing stashed ordered requests after all ledgers have caught up and applying any request if mode changed while processing and removing obsolete code from pool manager * Unclosed file warnings (#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * refactoring, documentation and updating setup to get rid of pyorient * refactoring * Updated ledger dep * deprecating orientdb * check state equality too while checking ledger equality in tests * removing obsolete test helper method * Agent generalization (#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * Add default values for role and verkey in state and fix test timeouts * refactor * import logger * updated to use new API from shared lib * updated to use new API from shared lib * handle pyorient import * handle pyorient import * increment stp version * Enabled the tests in plenum.test.test_log_rotation module. (#135) * increment stp version * close files in tests * check for close method before calling it (#136) * minor fix in batch creation logic and caching primary replica no in node to avoid recomputation * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/common/test_network_setup.py # plenum/server/node.py # plenum/test/test_node.py # setup.py * add missing __init__ for member package * added missed __init__.py for common/member * skip temporarily a failing test * fix primary disconnection and new primary not being same as old for master instance * add more tests for primary disconnection, same primary not elected in the next view and refactor * add new test for minor network glitch with primary * add missing parameter to bootstrap test network (#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * minor change in a test and removing sip statements * fix the catchup tests (#140) * make tests a little robust * fix test, check for disconnections after processing sent and received messages and little documentation * Close SOV-976 (#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/test/node_catchup/test_new_node_catchup.py # setup.py * up stp to 0.1.42 (#148) * refactor, enable test, adding timeout for view change and create remotes as soon as nodestack starts * fix broken test * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (#149) * increment stp * increment ledger * up stp to 0.1.14 * move state and kv storage to 'state' repo * get rid of orientdb code * fix tests * fix generation of trustee txns * fix genesis txns for trustee * fix generation of trustee txns (#151) * remove obsolete code * client to attempt establishing connections with nodes on start * refactor monitor.isMasterThroughputTooLow * update method for forcing view change cna checking it; intoduce wait method for it * use provoke_and_wait_for_view_change instead of eventually and check * remove unused dependency and skip tests - removed unused dependencies - minor fixes - skip tests that fail in master too (we have separate tickets for this) * increment ledger * increment ledger * enable debug logging in tests * using reverted changes in state interface * increment state version * Removed redundant copying of the file "pool_transactions_sandbox" to the directory "~/.plenum". (It is already done using "data_files" argument of "setup" function call.) (#152) * speed up tests (#147) * speed up test_nodes_with_pool_txns.py tests * reduce the timeout in checkNodesConnected * speed up the pool_transactions test * fix the timeout in test_delay.py * fix the timeout in test_primary_election_case5.py * speed up monitoring tests * revert changes in test_instance_change_with_Delta.py * each test's running time is limited by TestRunningTimeLimitSec * increase pool interconnection timeout on zstack retry timeout * fix timeout in test_delay.py * fix tests, make them faster * speedup the tests, tidy test timeouts * add a timeout into waits * bump stp-dev==0.1.43 * Unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well (#153) * unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well * decrease number of nodes in test_instance_cannot_become_active_with_less_than_four_servers to 13 * fix change back HA, SOV-908 (#154) * Fix testChangeNodeHaForNonPrimary (#157) * unskip test_change_non_primary_node_ha * fix waits name - it was renamed * verify genesis ledgers only if such option set in config * use primaryName instead of isPrimary to check that election is done * add numOfReelections parameter to waits.expectedPoolElectionTimeout * refactor test_node.py * set custom election timeout (numOfReelections=3) in changeNodeHa * remove debug prints * check VerKey is base58 for NODE txn, SOV-988 (#158) * client using a method which makes the code more testable, update to an old test with more checks and removing async from a method where it was not needed * Fix test testChangeNodeHaForPrimary (#160) * unskip testChangeNodeHaForPrimary * simplify conditional in changeNodeHa * node which is going down should not participate in a view change * change formating in testChangeNodeHaForPrimary * refactor to make method overridable and adding an argument to a test function * updated pypi package name for state repo and its version (#159) * Squashed commit of the following: create LedgerInfo class and replace collections of LedgerManager by fields * rename LedgerManager.ledgers to ledgerRegistry * fix usages of LedgerManager.ledgerRegistry * use correct attribute and renamed method * bring change from master and unskip a test * use correct attribute and set state to be committed when received from catchup * formatting * up ledger-3pc-batch version to 0.2.16 * up stp-3pc-batch version to 0.1.15 * improve the check of the arguments in "generate_sovrin_pool_transactions" (#162) * check client input for the NODE txn (#161) * fix testInstChangeWithLowerRatioThanDelta - decrease number of messages in a batch * decrease timeout to fix provoke_and_wait_for_view_change * fixing test * Fix post 3pc-batch merge (#163) * remove development options from Jenkinsfile * skip some broken tests * Update versions and skip windows build steps (#164) * switch off windows related build steps * update versions of ledger and stp * fix name * up ledger and stp version * skip test testNodeRequestingTxns * using correct timeout in tests * move some enums to correct location, using ledger's string encoding methods and test for stashing requests * bump dependency version * fix timeout in tests * make order of ledger sync customizable, add tranformer for transactions before adding to ledger, update seqno map db for transactions from catchup and update tests to check this * temporarily change config variable * fix regression where node starts catchup process if sent a consistency proff * bumping dependency version * bumping dependency version * bumping dependency version * Fix redundant reconnections and refactor (#165) * refactor retryForExpected * use maintainConnections in retryForExpected instead of connect * refactor resendRequests * up stp version to 0.1.49 * replace CLIENT_REQACK_TIMEOUT by CLIENT_REPLY_TIMEOUT when calling _filterExpected for REPLYs * up ledger to 0.2.19 * Add new helper-methods for conversion base58 to hex (#166) * increment state-trie and ledger versions (#168) * Make changes to support ChunkedFileStore (#167) * update _defaultStore of Ledger to make it correspond to original one * remove initialization of domain ledger - now it is done by ledger and file store it uses * replace manual copying of transaction files by usage of defaultFile * increase timeout for testNodeRequestingTxns * skip test testInstChangeWithLowerRatioThanDelta because it fails intermittently * up ledger version to 0.2.28 * Bump dependency version * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node (#172) * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node * Fixed failing tests * Fixed unsed imports * Update conftest.py * Increased test time global limit * Reverted timeouts * Added logs * Fixed filtering CPs * Fixed filtering CPs * Fixed filtering CPs * Input validation (#170) * Input sanitization: Add base logic and test cases * Input sanitization: add NODE and NYM txn, fixes * Input sanitization: implement node and client messages * roll away node to node validation * role field is option for NYM txn * fix tests * fixes for sovrin-node * implement validation for merkle root hash * uncomment new ConsistencyProof message implementation * add "nullable" property for FieldBase * fix usage of alphabet * add JsonField * add validation on message level * use hash size range instead of one value; use base58.alphabet instead of own list * fix usage hasSizes; made error comment more verbose * uncomment new implementation LedgerStatus message * uncomment new implementation for Prepare message and fix fields types * roll away node to node checkes * uncomment some new message implementations * use HexField instead of MerkleRootHashField for state_root and rxn_root in PrePrepare * make message about hash length in MerkleRootField more verbose * limit timestamp field by 253402290000.0 * modify randomString to remove length limitation * add tests for merkle root field validation * make randomString work not only with indexable containers * refactor tests for merkle root * fix timestamp validation * add test for timestamp validation * update validation of timestamp * check each node connected * improve log messages * fix testAdd2NewNodes * Improvement of validation rules (#173) * fix Node txn, INDY-9 * add tests for the Node operation * add checking that number of init parameters of message is equal to number of fields declared in schema * use OrderedDict in __dict__ * add test for merkle, fix logic * use OrderedDict for _fields and update all realted methods * - Provided all the disabled tests with the ticket references. (#176) - Enabled the tests testChangeNodeHaForPrimary and testChangeNodeHaForNonPrimary on non-Windows platforms. - Commented out parametrizers at the disabled tests in plenum.test.input_validation.test_common_checks module. * more removal of op to the top since it can affect checking of a number of args * Make SERVICES field required for add NODE txn * Added another test scenario for catchup (#178) * override items, keys, values methods of Mapping in MessageBase to make it compatible * Added another test scenario for catchup fix * clean MessageBase * INDY-73: Removed a commented out @skipif decorator (#177) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * Add base58 str class; destnode; destnym; * use - as default ppSeqNo instead of -1 * overtride __ftr__ for MessageBase * comment checking of presence of SERVICE field * move alphabet validation from _specific_validation to avoid accidental overriding of it * INDY-73: Added ticket references to test stubs (#180) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * - Added ticket references to the not implemented tests which had not contained them yet. - Uncommented testStopScriptIfNodeIsRunning and disabled it since it fails. * Added raising NotImplementedError to not implemented tests. * Move all specific checks to from validate method * Eliminate multiple inheritance * Enabled and updated testNodesConnectsWhenOneNodeIsLate (#181) * Enabled and updated testNodesConnectsWhenOneNodeIsLate * updated assertions logic * INDY-109 (#184) * updates to test helpers and change in forwarding requests logic (#187) * Use secure random number generator for randomString() Current usage of random.choice() is not secure so replaced that with libsodium provided randombytes_uniform() which is secure and also available on all platforms where libsodium is available * Fixes for INDY-71 (#185) * Refactor check_endpoint_valid func * Fix validation, INDY-71 * Fix imports * Do not connect to a new, non-active node * Remove confusing comments * Minor refactor * Fix test testAddInactiveNodeThenActivate * Validation for view change and 3pc messages (#193) * add exception for type error and raise it in MessageValidator * make error message in ChooseField more verbose * uncomment new implementations of Reelection, Primary and InstanceChange messages * update and fix testInstanceChangeMsgTypeChecking * fix testDiscardInstChngMsgFrmPastView * use NonNegativeField instead of TimestampFied inTieAmountField for its seconds part * make name pf test for merkle root field more standard * move tests for fields to own package * add missing empty line * remove ununsed imports * refactor test for timestamp * remove todo about code duplication since it is not valid anymore * add tests for some field types * make IterableField check type of inner field and that it is not none * add test for hex field * add test for ledger id * add test for request id field * override __eq__ for MessageBase * use NonNegativeNumberField instead of TimestampField in RequestIdentifierField * use IdentifierField instead of NonEmptyStringField in RequestIdentifierField * update test for request identifier * add test for IdentifierField * update test for RequestIdentifier field * implement validation for IdentifierField and reimplement DestNodeField using it * uncomment new implementations for Propagate, Prepare, Commit, Ordered * Refactor base58 check; add Verkey check; dest node; dest nym; merkelroot; * move test for base58 field to field validation package * remove unused import * add checking of max length hash to merkle tree root field test * update test for identifier field * update test for request identifier field * fix formatting * fix checking of zero * remove unused commented line * rename test_non_negative_number to test_positive_number * Make verkey validator accept empty strings * Add super's __init__ params to child calsses * Improve getMatchedHelpableMsg logic (#195) * Use original batch parameters in all tests except the ones that use batches and use alternate of eventuallyAll to check a collection of functions under a timeout (#190) * updates to test helpers and change in forwarding requests logic * overriding batch params in tests * use correct timeouts in tests * accounting for nomination delay * Disable view change (#196) * repaired calls to deprecated methods * Added unsafe 'disable_view_change' for testing - Added 'unsafe' attribute to default config. "unsafe" is a set of attributes that can set certain behaviors that are not safe, for example, 'disable_view_change' disables view changes from happening. This might be useful in testing scenarios, but never in a live network. - Added 'unsafe' behavior 'disable_view_change' which causes Monitor to always return false when asked if master instance performance is degraded. - Refactored view change fixture so it could be reused to test new 'disable_view_change' behavior. * fixed import for missed fixture in test * Fix validation types (#197) * Enable and update testNodeDiscardMessageFromUnknownView (#194) * added equality method for MessageBase * added ability to cancel created delayers * enabled and updated testNodeDiscardMessageFromUnknownView * updated test to recent changes in node logic * reverted recent changes not necessary for now * Catch up demoted (#201) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * Fixes for dead catchup (#202) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * up stp version to 0.1.56 * up stp version to 0.1.57 * fixed tests * fix test * Disable the test: INDY-147. * Make random string generation efficient The current algorithm requires calling the libsodium random function 'size' times where size is the expected length of random string Changed this so that this will require at the most '(size/64)' number of calls. Also we do not need the chars list now we always use only alphabets and digits available in hex this makes it very easy and still gives us what we need. Every instance of usage of this function has been checked and nowhere the 'chars' parameter is used * Change to pass exact size to libsodium randombytes() function This probably is cryptographically more accurate * Fix catchup when commit (#203) * remove 3PC messages that are already ordered during catch-up * Added more tests * fix catch-up in between 3pc msgs * fix tests * update validation rule for verkey (#208) * fix an incorrect test (#207) * fix an incorrect test * refactoring test * remove un-necessary parameter (#210) * Revert unordered batches before starting catch-up (not after) (#209) * Revert unordered batches before starting catch-up (not after) * add test for revert of the state before catchup * improve the test * Do not enqueue old pre-prepares (#211) * ignore pp requests with ppSeqNo less than expected * minor refactor * Optimisations and a testing util (#212) * add todo, fixing log levels, commenting out an expensive area of code and adding capability to diable monitor in tests * an optimisation avoiding re-calculation of merkle proof * remove commented code and add test * update names * add test fils for validators * add test for commit message * add test for ConsistencyProof message * update test for Commit message * add test for InstanceChange message * add test for LedgerStatus message * add test for nomination message * add test for Ordered message * add tests for Prepare and PrePrepare messages * add test for Primary message * add test for PROPAGATE message * add test for Reelection message * remove unused imports * Election bug fix (#215) * Election bug fix if `f` nodes do not send Primary and the node is the last one to send Primary, then it was not able to decide the primary, since a node was doing "primary setting" only on receiving a Primary message, changed that to try "setting primary" after sending Primary too * update test with more checks

* add super initialization, update usage of super methods * update docstring of commit * remove commitReq since it does the same as commit * adding timeout to test utility method * changes for load testing * sanitising log messages * Versioning fix (#113) * changes for request handlers * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * changing to rlp encoding * moving error codes to stp * upgrading version of stp * req_handler changes * fixing import * temp commit * Stp (#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * minor changes to serialization * Plenum test runner fix (#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * updating apply of request handler * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * leveldb HashStore * parametrize test timeouts (#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * removing commented code and fixing bugs in pool request handler * lowering Delta in pool_transactions conftest * fix jenkins build (#123) * fixed deps in setup.py * added pypi publishing * updated deps * updated deps stp-dev -> stp-perf-imp * removed unnecessary sources * updated setup.py to hold correct deps names and self name for publishing * fixed plenum for the latest stp (#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * sending last pre-prepare sequence number in consistency proof so already processed request can be ignored, re-enabling tests and re-adding proper timeouts * Feature Added: Deb build * Feature Added: Deb build * Do view change if a primary is disconnected (#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * processing stashed ordered requests after all ledgers have caught up and applying any request if mode changed while processing and removing obsolete code from pool manager * Unclosed file warnings (#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * refactoring, documentation and updating setup to get rid of pyorient * refactoring * Updated ledger dep * deprecating orientdb * check state equality too while checking ledger equality in tests * removing obsolete test helper method * Agent generalization (#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * Add default values for role and verkey in state and fix test timeouts * refactor * import logger * updated to use new API from shared lib * updated to use new API from shared lib * handle pyorient import * handle pyorient import * increment stp version * Enabled the tests in plenum.test.test_log_rotation module. (#135) * increment stp version * close files in tests * check for close method before calling it (#136) * minor fix in batch creation logic and caching primary replica no in node to avoid recomputation * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/common/test_network_setup.py # plenum/server/node.py # plenum/test/test_node.py # setup.py * add missing __init__ for member package * added missed __init__.py for common/member * skip temporarily a failing test * fix primary disconnection and new primary not being same as old for master instance * add more tests for primary disconnection, same primary not elected in the next view and refactor * add new test for minor network glitch with primary * add missing parameter to bootstrap test network (#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * minor change in a test and removing sip statements * fix the catchup tests (#140) * make tests a little robust * fix test, check for disconnections after processing sent and received messages and little documentation * Close SOV-976 (#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/test/node_catchup/test_new_node_catchup.py # setup.py * up stp to 0.1.42 (#148) * refactor, enable test, adding timeout for view change and create remotes as soon as nodestack starts * fix broken test * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (#149) * increment stp * increment ledger * up stp to 0.1.14 * move state and kv storage to 'state' repo * get rid of orientdb code * fix tests * fix generation of trustee txns * fix genesis txns for trustee * fix generation of trustee txns (#151) * remove obsolete code * client to attempt establishing connections with nodes on start * refactor monitor.isMasterThroughputTooLow * update method for forcing view change cna checking it; intoduce wait method for it * use provoke_and_wait_for_view_change instead of eventually and check * remove unused dependency and skip tests - removed unused dependencies - minor fixes - skip tests that fail in master too (we have separate tickets for this) * increment ledger * increment ledger * enable debug logging in tests * using reverted changes in state interface * increment state version * Removed redundant copying of the file "pool_transactions_sandbox" to the directory "~/.plenum". (It is already done using "data_files" argument of "setup" function call.) (#152) * speed up tests (#147) * speed up test_nodes_with_pool_txns.py tests * reduce the timeout in checkNodesConnected * speed up the pool_transactions test * fix the timeout in test_delay.py * fix the timeout in test_primary_election_case5.py * speed up monitoring tests * revert changes in test_instance_change_with_Delta.py * each test's running time is limited by TestRunningTimeLimitSec * increase pool interconnection timeout on zstack retry timeout * fix timeout in test_delay.py * fix tests, make them faster * speedup the tests, tidy test timeouts * add a timeout into waits * bump stp-dev==0.1.43 * Unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well (#153) * unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well * decrease number of nodes in test_instance_cannot_become_active_with_less_than_four_servers to 13 * fix change back HA, SOV-908 (#154) * Fix testChangeNodeHaForNonPrimary (#157) * unskip test_change_non_primary_node_ha * fix waits name - it was renamed * verify genesis ledgers only if such option set in config * use primaryName instead of isPrimary to check that election is done * add numOfReelections parameter to waits.expectedPoolElectionTimeout * refactor test_node.py * set custom election timeout (numOfReelections=3) in changeNodeHa * remove debug prints * check VerKey is base58 for NODE txn, SOV-988 (#158) * client using a method which makes the code more testable, update to an old test with more checks and removing async from a method where it was not needed * Fix test testChangeNodeHaForPrimary (#160) * unskip testChangeNodeHaForPrimary * simplify conditional in changeNodeHa * node which is going down should not participate in a view change * change formating in testChangeNodeHaForPrimary * refactor to make method overridable and adding an argument to a test function * updated pypi package name for state repo and its version (#159) * Squashed commit of the following: create LedgerInfo class and replace collections of LedgerManager by fields * rename LedgerManager.ledgers to ledgerRegistry * fix usages of LedgerManager.ledgerRegistry * use correct attribute and renamed method * bring change from master and unskip a test * use correct attribute and set state to be committed when received from catchup * formatting * up ledger-3pc-batch version to 0.2.16 * up stp-3pc-batch version to 0.1.15 * improve the check of the arguments in "generate_sovrin_pool_transactions" (#162) * check client input for the NODE txn (#161) * fix testInstChangeWithLowerRatioThanDelta - decrease number of messages in a batch * decrease timeout to fix provoke_and_wait_for_view_change * fixing test * Fix post 3pc-batch merge (#163) * remove development options from Jenkinsfile * skip some broken tests * Update versions and skip windows build steps (#164) * switch off windows related build steps * update versions of ledger and stp * fix name * up ledger and stp version * skip test testNodeRequestingTxns * using correct timeout in tests * move some enums to correct location, using ledger's string encoding methods and test for stashing requests * bump dependency version * fix timeout in tests * make order of ledger sync customizable, add tranformer for transactions before adding to ledger, update seqno map db for transactions from catchup and update tests to check this * temporarily change config variable * fix regression where node starts catchup process if sent a consistency proff * bumping dependency version * bumping dependency version * bumping dependency version * Fix redundant reconnections and refactor (#165) * refactor retryForExpected * use maintainConnections in retryForExpected instead of connect * refactor resendRequests * up stp version to 0.1.49 * replace CLIENT_REQACK_TIMEOUT by CLIENT_REPLY_TIMEOUT when calling _filterExpected for REPLYs * up ledger to 0.2.19 * Add new helper-methods for conversion base58 to hex (#166) * increment state-trie and ledger versions (#168) * Make changes to support ChunkedFileStore (#167) * update _defaultStore of Ledger to make it correspond to original one * remove initialization of domain ledger - now it is done by ledger and file store it uses * replace manual copying of transaction files by usage of defaultFile * increase timeout for testNodeRequestingTxns * skip test testInstChangeWithLowerRatioThanDelta because it fails intermittently * up ledger version to 0.2.28 * Bump dependency version * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node (#172) * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node * Fixed failing tests * Fixed unsed imports * Update conftest.py * Increased test time global limit * Reverted timeouts * Added logs * Fixed filtering CPs * Fixed filtering CPs * Fixed filtering CPs * Input validation (#170) * Input sanitization: Add base logic and test cases * Input sanitization: add NODE and NYM txn, fixes * Input sanitization: implement node and client messages * roll away node to node validation * role field is option for NYM txn * fix tests * fixes for sovrin-node * implement validation for merkle root hash * uncomment new ConsistencyProof message implementation * add "nullable" property for FieldBase * fix usage of alphabet * add JsonField * add validation on message level * use hash size range instead of one value; use base58.alphabet instead of own list * fix usage hasSizes; made error comment more verbose * uncomment new implementation LedgerStatus message * uncomment new implementation for Prepare message and fix fields types * roll away node to node checkes * uncomment some new message implementations * use HexField instead of MerkleRootHashField for state_root and rxn_root in PrePrepare * make message about hash length in MerkleRootField more verbose * limit timestamp field by 253402290000.0 * modify randomString to remove length limitation * add tests for merkle root field validation * make randomString work not only with indexable containers * refactor tests for merkle root * fix timestamp validation * add test for timestamp validation * update validation of timestamp * check each node connected * improve log messages * fix testAdd2NewNodes * Improvement of validation rules (#173) * fix Node txn, INDY-9 * add tests for the Node operation * add checking that number of init parameters of message is equal to number of fields declared in schema * use OrderedDict in __dict__ * add test for merkle, fix logic * use OrderedDict for _fields and update all realted methods * - Provided all the disabled tests with the ticket references. (#176) - Enabled the tests testChangeNodeHaForPrimary and testChangeNodeHaForNonPrimary on non-Windows platforms. - Commented out parametrizers at the disabled tests in plenum.test.input_validation.test_common_checks module. * more removal of op to the top since it can affect checking of a number of args * Make SERVICES field required for add NODE txn * Added another test scenario for catchup (#178) * override items, keys, values methods of Mapping in MessageBase to make it compatible * Added another test scenario for catchup fix * clean MessageBase * INDY-73: Removed a commented out @skipif decorator (#177) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * Add base58 str class; destnode; destnym; * use - as default ppSeqNo instead of -1 * overtride __ftr__ for MessageBase * comment checking of presence of SERVICE field * move alphabet validation from _specific_validation to avoid accidental overriding of it * INDY-73: Added ticket references to test stubs (#180) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * - Added ticket references to the not implemented tests which had not contained them yet. - Uncommented testStopScriptIfNodeIsRunning and disabled it since it fails. * Added raising NotImplementedError to not implemented tests. * Move all specific checks to from validate method * Eliminate multiple inheritance * Enabled and updated testNodesConnectsWhenOneNodeIsLate (#181) * Enabled and updated testNodesConnectsWhenOneNodeIsLate * updated assertions logic * INDY-109 (#184) * updates to test helpers and change in forwarding requests logic (#187) * Use secure random number generator for randomString() Current usage of random.choice() is not secure so replaced that with libsodium provided randombytes_uniform() which is secure and also available on all platforms where libsodium is available * Fixes for INDY-71 (#185) * Refactor check_endpoint_valid func * Fix validation, INDY-71 * Fix imports * Do not connect to a new, non-active node * Remove confusing comments * Minor refactor * Fix test testAddInactiveNodeThenActivate * Validation for view change and 3pc messages (#193) * add exception for type error and raise it in MessageValidator * make error message in ChooseField more verbose * uncomment new implementations of Reelection, Primary and InstanceChange messages * update and fix testInstanceChangeMsgTypeChecking * fix testDiscardInstChngMsgFrmPastView * use NonNegativeField instead of TimestampFied inTieAmountField for its seconds part * make name pf test for merkle root field more standard * move tests for fields to own package * add missing empty line * remove ununsed imports * refactor test for timestamp * remove todo about code duplication since it is not valid anymore * add tests for some field types * make IterableField check type of inner field and that it is not none * add test for hex field * add test for ledger id * add test for request id field * override __eq__ for MessageBase * use NonNegativeNumberField instead of TimestampField in RequestIdentifierField * use IdentifierField instead of NonEmptyStringField in RequestIdentifierField * update test for request identifier * add test for IdentifierField * update test for RequestIdentifier field * implement validation for IdentifierField and reimplement DestNodeField using it * uncomment new implementations for Propagate, Prepare, Commit, Ordered * Refactor base58 check; add Verkey check; dest node; dest nym; merkelroot; * move test for base58 field to field validation package * remove unused import * add checking of max length hash to merkle tree root field test * update test for identifier field * update test for request identifier field * fix formatting * fix checking of zero * remove unused commented line * rename test_non_negative_number to test_positive_number * Make verkey validator accept empty strings * Add super's __init__ params to child calsses * Improve getMatchedHelpableMsg logic (#195) * Use original batch parameters in all tests except the ones that use batches and use alternate of eventuallyAll to check a collection of functions under a timeout (#190) * updates to test helpers and change in forwarding requests logic * overriding batch params in tests * use correct timeouts in tests * accounting for nomination delay * Disable view change (#196) * repaired calls to deprecated methods * Added unsafe 'disable_view_change' for testing - Added 'unsafe' attribute to default config. "unsafe" is a set of attributes that can set certain behaviors that are not safe, for example, 'disable_view_change' disables view changes from happening. This might be useful in testing scenarios, but never in a live network. - Added 'unsafe' behavior 'disable_view_change' which causes Monitor to always return false when asked if master instance performance is degraded. - Refactored view change fixture so it could be reused to test new 'disable_view_change' behavior. * fixed import for missed fixture in test * Fix validation types (#197) * Enable and update testNodeDiscardMessageFromUnknownView (#194) * added equality method for MessageBase * added ability to cancel created delayers * enabled and updated testNodeDiscardMessageFromUnknownView * updated test to recent changes in node logic * reverted recent changes not necessary for now * Catch up demoted (#201) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * Fixes for dead catchup (#202) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * up stp version to 0.1.56 * up stp version to 0.1.57 * fixed tests * fix test * Disable the test: INDY-147. * Make random string generation efficient The current algorithm requires calling the libsodium random function 'size' times where size is the expected length of random string Changed this so that this will require at the most '(size/64)' number of calls. Also we do not need the chars list now we always use only alphabets and digits available in hex this makes it very easy and still gives us what we need. Every instance of usage of this function has been checked and nowhere the 'chars' parameter is used * Change to pass exact size to libsodium randombytes() function This probably is cryptographically more accurate * Fix catchup when commit (#203) * remove 3PC messages that are already ordered during catch-up * Added more tests * fix catch-up in between 3pc msgs * fix tests * update validation rule for verkey (#208) * fix an incorrect test (#207) * fix an incorrect test * refactoring test * remove un-necessary parameter (#210) * Revert unordered batches before starting catch-up (not after) (#209) * Revert unordered batches before starting catch-up (not after) * add test for revert of the state before catchup * improve the test * Do not enqueue old pre-prepares (#211) * ignore pp requests with ppSeqNo less than expected * minor refactor * Optimisations and a testing util (#212) * add todo, fixing log levels, commenting out an expensive area of code and adding capability to diable monitor in tests * an optimisation avoiding re-calculation of merkle proof * remove commented code and add test * update names * add test fils for validators * add test for commit message * add test for ConsistencyProof message * update test for Commit message * add test for InstanceChange message * add test for LedgerStatus message * add test for nomination message * add test for Ordered message * add tests for Prepare and PrePrepare messages * add test for Primary message * add test for PROPAGATE message * add test for Reelection message * remove unused imports * Election bug fix (#215) * Election bug fix if `f` nodes do not send Primary and the node is the last one to send Primary, then it was not able to decide the primary, since a node was doing "primary setting" only on receiving a Primary message, changed that to try "setting primary" after sending Primary too * update test with more checks * removed obsolete test testStopScriptIfNodeIsRunning (#220) * Fix processing Propagate msg, INDY-245 (#221) * Bugfix/non human readable error (#222) * handling case with odd-length verkey * updated exception message, check it in test * Stable release

* changes for stp * removing redundant close in orientdb socket * adding tests for catchup scenario and 3 phase message outside water mark * fixing import in test * Fix 'infinit' looping on receiving messages when their amount is very high (hyperledger#97) * fix 'infinit' looping on receiving messages when their amount is very high * move quotas to config * make zstack get params from config only once * add test for quota * increase message quotas to 100 * Feature Added: Jenkins (hyperledger#99) * Moved util methods to NetwrorkInterfaces; fixes in raet stack * fix key generation * Hotfix: Deps (hyperledger#101) * - Added a custom jsonpickle handler for enums to avoid the issue jsonpickle/jsonpickle#135 in jsonpickle 0.9.2 which is the canonical version for Ubuntu 16.04. (hyperledger#98) - Corrected Cli.restoreWalletByPath method. * Merge branches 'stp' and 'zmq' of github.com:evernym/plenum into stp * handling infinite loop problem and fixing test delay * fixing import * bugfix in test * raising timeout * merge * change in ordered * move connection related exceptions to stp project * fix imports of moved exceptions * fixed dependencies * fixing name * temporarily disabling tests on Windows * renamed init_plenum_raet_keep to init_plenum_keys * removing unused code and changing script mode * Introduced failing of the pytets session in case any unexpected warnings have been made (hyperledger#104) * changes to troubleshoot testing for coro not awaited * added support for warning checking across all tests * fixed scope for warnfilters fixture * - Updated the warnings filter. Made the rules for ResourceWarning: "unclosed file", ResourceWarning: "unclosed.*socket\.socket" and RuntimeWarning: "coroutine.*was never awaited" global since they are made by the garbage collector in unpredictable points. - Sorted the rules in the warnings filter alphabetically. * Specialized the warnings filter rule for RuntimeWarning about a coroutine was never awaited (to "msgAll" coroutine). * Added an explicit call of the garbage collector to the tear-down phase of "warncheck" fixture in order to ensure that warnings will be generated and recorded for all the unclosed resources. * Fixed the issue with that a coroutine "msgAll" object was never awaited. Removed the corresponding "ignore" rule from the warnings filter. * - Removed the rules not being actuated currently from the warnings filter. - Replaced use of line numbers with use of message patterns in the warnings filter rules. * Corrected the message pattern for a rule from the warnings filter. * Added an "ignore" rule for ResourceWarning: "unclosed event loop" to the warnings filter. * Returned back the warnings filter rules for DeprecationWarnings in jsonpickle package. Replaced use of line numbers with use of message patterns in these rules. * adding warning * removing looper and checking while closing orientdb * increasing a timeout * replacing warn with warning methods * fixed stp dependency * fixed tests * skip failing test, created SOV-881. * adding load script * updating setup * changing logs to print alias * setting resource limit for tests * handling exception while raising resource limits * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring (hyperledger#106) * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring * removed unwanted error code, refactored list keyrings command so that it correctly works for agent cli * removed unused imports * making CLI accept hex seed of size 64 * move logging to stp repo * mitigating key error in pool manager * changing timeouts so the diconnect check happens twice * handling unavailabily of resource module * some corrects to gc * setting the lastOrdered sequence number of the replica after election * fixing obsolete imports * adding temporary info logs to show up during load testing, problem identified for PREPREPARE getting rejects, state tree has different root at beginning for different nodes * reverting config changes * using ordered json encoder * add declarations of common request handling methods to ReqHandler * refactor method commit * add docstring for ReqHandler, rearrange methods * rename *ReqHandlers to *RequestHandlers * add TODOs about fixing hierarchy of State and PruningState * rename validateReq to validate * add config=None argument to validate method to make sub and super signatures match * rename applyReq to apply * add super initialization, update usage of super methods * update docstring of commit * remove commitReq since it does the same as commit * adding timeout to test utility method * changes for load testing * sanitising log messages * Versioning fix (hyperledger#113) * changes for request handlers * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * changing to rlp encoding * moving error codes to stp * upgrading version of stp * req_handler changes * fixing import * temp commit * Stp (hyperledger#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * minor changes to serialization * Plenum test runner fix (hyperledger#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * updating apply of request handler * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (hyperledger#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (hyperledger#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * leveldb HashStore * parametrize test timeouts (hyperledger#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * removing commented code and fixing bugs in pool request handler * lowering Delta in pool_transactions conftest * fix jenkins build (hyperledger#123) * fixed deps in setup.py * added pypi publishing * updated deps * updated deps stp-dev -> stp-perf-imp * removed unnecessary sources * updated setup.py to hold correct deps names and self name for publishing * fixed plenum for the latest stp (hyperledger#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (hyperledger#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * sending last pre-prepare sequence number in consistency proof so already processed request can be ignored, re-enabling tests and re-adding proper timeouts * Feature Added: Deb build * Feature Added: Deb build * Do view change if a primary is disconnected (hyperledger#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * processing stashed ordered requests after all ledgers have caught up and applying any request if mode changed while processing and removing obsolete code from pool manager * Unclosed file warnings (hyperledger#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * refactoring, documentation and updating setup to get rid of pyorient * refactoring * Updated ledger dep * deprecating orientdb * check state equality too while checking ledger equality in tests * removing obsolete test helper method * Agent generalization (hyperledger#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * Add default values for role and verkey in state and fix test timeouts * refactor * import logger * updated to use new API from shared lib * updated to use new API from shared lib * handle pyorient import * handle pyorient import * increment stp version * Enabled the tests in plenum.test.test_log_rotation module. (hyperledger#135) * increment stp version * close files in tests * check for close method before calling it (hyperledger#136) * minor fix in batch creation logic and caching primary replica no in node to avoid recomputation * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/common/test_network_setup.py # plenum/server/node.py # plenum/test/test_node.py # setup.py * add missing __init__ for member package * added missed __init__.py for common/member * skip temporarily a failing test * fix primary disconnection and new primary not being same as old for master instance * add more tests for primary disconnection, same primary not elected in the next view and refactor * add new test for minor network glitch with primary * add missing parameter to bootstrap test network (hyperledger#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * minor change in a test and removing sip statements * fix the catchup tests (hyperledger#140) * make tests a little robust * fix test, check for disconnections after processing sent and received messages and little documentation * Close SOV-976 (hyperledger#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/test/node_catchup/test_new_node_catchup.py # setup.py * up stp to 0.1.42 (hyperledger#148) * refactor, enable test, adding timeout for view change and create remotes as soon as nodestack starts * fix broken test * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (hyperledger#149) * increment stp * increment ledger * up stp to 0.1.14 * move state and kv storage to 'state' repo * get rid of orientdb code * fix tests * fix generation of trustee txns * fix genesis txns for trustee * fix generation of trustee txns (hyperledger#151) * remove obsolete code * client to attempt establishing connections with nodes on start * refactor monitor.isMasterThroughputTooLow * update method for forcing view change cna checking it; intoduce wait method for it * use provoke_and_wait_for_view_change instead of eventually and check * remove unused dependency and skip tests - removed unused dependencies - minor fixes - skip tests that fail in master too (we have separate tickets for this) * increment ledger * increment ledger * enable debug logging in tests * using reverted changes in state interface * increment state version * Removed redundant copying of the file "pool_transactions_sandbox" to the directory "~/.plenum". (It is already done using "data_files" argument of "setup" function call.) (hyperledger#152) * speed up tests (hyperledger#147) * speed up test_nodes_with_pool_txns.py tests * reduce the timeout in checkNodesConnected * speed up the pool_transactions test * fix the timeout in test_delay.py * fix the timeout in test_primary_election_case5.py * speed up monitoring tests * revert changes in test_instance_change_with_Delta.py * each test's running time is limited by TestRunningTimeLimitSec * increase pool interconnection timeout on zstack retry timeout * fix timeout in test_delay.py * fix tests, make them faster * speedup the tests, tidy test timeouts * add a timeout into waits * bump stp-dev==0.1.43 * Unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well (hyperledger#153) * unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well * decrease number of nodes in test_instance_cannot_become_active_with_less_than_four_servers to 13 * fix change back HA, SOV-908 (hyperledger#154) * fix * Fix testChangeNodeHaForNonPrimary (hyperledger#157) * unskip test_change_non_primary_node_ha * fix waits name - it was renamed * verify genesis ledgers only if such option set in config * use primaryName instead of isPrimary to check that election is done * add numOfReelections parameter to waits.expectedPoolElectionTimeout * refactor test_node.py * set custom election timeout (numOfReelections=3) in changeNodeHa * remove debug prints * check VerKey is base58 for NODE txn, SOV-988 (hyperledger#158) * client using a method which makes the code more testable, update to an old test with more checks and removing async from a method where it was not needed * Fix test testChangeNodeHaForPrimary (hyperledger#160) * unskip testChangeNodeHaForPrimary * simplify conditional in changeNodeHa * node which is going down should not participate in a view change * change formating in testChangeNodeHaForPrimary * refactor to make method overridable and adding an argument to a test function * updated pypi package name for state repo and its version (hyperledger#159) * Squashed commit of the following: create LedgerInfo class and replace collections of LedgerManager by fields * rename LedgerManager.ledgers to ledgerRegistry * fix usages of LedgerManager.ledgerRegistry * use correct attribute and renamed method * bring change from master and unskip a test * use correct attribute and set state to be committed when received from catchup * formatting * up ledger-3pc-batch version to 0.2.16 * up stp-3pc-batch version to 0.1.15 * improve the check of the arguments in "generate_sovrin_pool_transactions" (hyperledger#162) * check client input for the NODE txn (hyperledger#161) * fix testInstChangeWithLowerRatioThanDelta - decrease number of messages in a batch * decrease timeout to fix provoke_and_wait_for_view_change * fixing test * Fix post 3pc-batch merge (hyperledger#163) * remove development options from Jenkinsfile * skip some broken tests * Update versions and skip windows build steps (hyperledger#164) * switch off windows related build steps * update versions of ledger and stp * fix name * up ledger and stp version * skip test testNodeRequestingTxns * using correct timeout in tests * move some enums to correct location, using ledger's string encoding methods and test for stashing requests * bump dependency version * fix timeout in tests * make order of ledger sync customizable, add tranformer for transactions before adding to ledger, update seqno map db for transactions from catchup and update tests to check this * temporarily change config variable * fix regression where node starts catchup process if sent a consistency proff * bumping dependency version * bumping dependency version * bumping dependency version * Fix redundant reconnections and refactor (hyperledger#165) * refactor retryForExpected * use maintainConnections in retryForExpected instead of connect * refactor resendRequests * up stp version to 0.1.49 * replace CLIENT_REQACK_TIMEOUT by CLIENT_REPLY_TIMEOUT when calling _filterExpected for REPLYs * up ledger to 0.2.19 * Add new helper-methods for conversion base58 to hex (hyperledger#166) * increment state-trie and ledger versions (hyperledger#168) * Make changes to support ChunkedFileStore (hyperledger#167) * update _defaultStore of Ledger to make it correspond to original one * remove initialization of domain ledger - now it is done by ledger and file store it uses * replace manual copying of transaction files by usage of defaultFile * increase timeout for testNodeRequestingTxns * skip test testInstChangeWithLowerRatioThanDelta because it fails intermittently * up ledger version to 0.2.28 * Bump dependency version * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node (hyperledger#172) * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node * Fixed failing tests * Fixed unsed imports * Update conftest.py * Increased test time global limit * Reverted timeouts * Added logs * Fixed filtering CPs * Fixed filtering CPs * Fixed filtering CPs * Input validation (hyperledger#170) * Input sanitization: Add base logic and test cases * Input sanitization: add NODE and NYM txn, fixes * Input sanitization: implement node and client messages * roll away node to node validation * role field is option for NYM txn * fix tests * fixes for sovrin-node * implement validation for merkle root hash * uncomment new ConsistencyProof message implementation * add "nullable" property for FieldBase * fix usage of alphabet * add JsonField * add validation on message level * use hash size range instead of one value; use base58.alphabet instead of own list * fix usage hasSizes; made error comment more verbose * uncomment new implementation LedgerStatus message * uncomment new implementation for Prepare message and fix fields types * roll away node to node checkes * check each node connected * improve log messages * fix testAdd2NewNodes * Improvement of validation rules (hyperledger#173) * fix Node txn, INDY-9 * add tests for the Node operation * - Provided all the disabled tests with the ticket references. (hyperledger#176) - Enabled the tests testChangeNodeHaForPrimary and testChangeNodeHaForNonPrimary on non-Windows platforms. - Commented out parametrizers at the disabled tests in plenum.test.input_validation.test_common_checks module. * Make SERVICES field required for add NODE txn * Added another test scenario for catchup (hyperledger#178) * Added another test scenario for catchup fix * stable release Signed-off-by: Andrei Goncharov <[email protected]> Sign-off-executed-by: toktar <[email protected]> Approved-at: h-master

* setting resource limit for tests * handling exception while raising resource limits * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring (hyperledger#106) * moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring * removed unwanted error code, refactored list keyrings command so that it correctly works for agent cli * removed unused imports * making CLI accept hex seed of size 64 * move logging to stp repo * mitigating key error in pool manager * changing timeouts so the diconnect check happens twice * handling unavailabily of resource module * some corrects to gc * setting the lastOrdered sequence number of the replica after election * fixing obsolete imports * adding temporary info logs to show up during load testing, problem identified for PREPREPARE getting rejects, state tree has different root at beginning for different nodes * reverting config changes * using ordered json encoder * add declarations of common request handling methods to ReqHandler * refactor method commit * add docstring for ReqHandler, rearrange methods * rename *ReqHandlers to *RequestHandlers * add TODOs about fixing hierarchy of State and PruningState * rename validateReq to validate * add config=None argument to validate method to make sub and super signatures match * rename applyReq to apply * add super initialization, update usage of super methods * update docstring of commit * remove commitReq since it does the same as commit * adding timeout to test utility method * changes for load testing * sanitising log messages * Versioning fix (hyperledger#113) * changes for request handlers * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * changing to rlp encoding * moving error codes to stp * upgrading version of stp * req_handler changes * fixing import * temp commit * Stp (hyperledger#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * minor changes to serialization * Plenum test runner fix (hyperledger#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * updating apply of request handler * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (hyperledger#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (hyperledger#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * leveldb HashStore * parametrize test timeouts (hyperledger#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * removing commented code and fixing bugs in pool request handler * lowering Delta in pool_transactions conftest * fix jenkins build (hyperledger#123) * fixed deps in setup.py * added pypi publishing * updated deps * updated deps stp-dev -> stp-perf-imp * removed unnecessary sources * updated setup.py to hold correct deps names and self name for publishing * fixed plenum for the latest stp (hyperledger#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (hyperledger#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * sending last pre-prepare sequence number in consistency proof so already processed request can be ignored, re-enabling tests and re-adding proper timeouts * Feature Added: Deb build * Feature Added: Deb build * Do view change if a primary is disconnected (hyperledger#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * processing stashed ordered requests after all ledgers have caught up and applying any request if mode changed while processing and removing obsolete code from pool manager * Unclosed file warnings (hyperledger#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * refactoring, documentation and updating setup to get rid of pyorient * refactoring * Updated ledger dep * deprecating orientdb * check state equality too while checking ledger equality in tests * removing obsolete test helper method * Agent generalization (hyperledger#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * Add default values for role and verkey in state and fix test timeouts * refactor * import logger * updated to use new API from shared lib * updated to use new API from shared lib * handle pyorient import * handle pyorient import * increment stp version * Enabled the tests in plenum.test.test_log_rotation module. (hyperledger#135) * increment stp version * close files in tests * check for close method before calling it (hyperledger#136) * minor fix in batch creation logic and caching primary replica no in node to avoid recomputation * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/common/test_network_setup.py # plenum/server/node.py # plenum/test/test_node.py # setup.py * add missing __init__ for member package * added missed __init__.py for common/member * skip temporarily a failing test * fix primary disconnection and new primary not being same as old for master instance * add more tests for primary disconnection, same primary not elected in the next view and refactor * add new test for minor network glitch with primary * add missing parameter to bootstrap test network (hyperledger#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * minor change in a test and removing sip statements * fix the catchup tests (hyperledger#140) * make tests a little robust * fix test, check for disconnections after processing sent and received messages and little documentation * Close SOV-976 (hyperledger#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/test/node_catchup/test_new_node_catchup.py # setup.py * up stp to 0.1.42 (hyperledger#148) * refactor, enable test, adding timeout for view change and create remotes as soon as nodestack starts * fix broken test * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (hyperledger#149) * increment stp * increment ledger * up stp to 0.1.14 * move state and kv storage to 'state' repo * get rid of orientdb code * fix tests * fix generation of trustee txns * fix genesis txns for trustee * fix generation of trustee txns (hyperledger#151) * remove obsolete code * client to attempt establishing connections with nodes on start * refactor monitor.isMasterThroughputTooLow * update method for forcing view change cna checking it; intoduce wait method for it * use provoke_and_wait_for_view_change instead of eventually and check * remove unused dependency and skip tests - removed unused dependencies - minor fixes - skip tests that fail in master too (we have separate tickets for this) * increment ledger * increment ledger * enable debug logging in tests * using reverted changes in state interface * increment state version * Removed redundant copying of the file "pool_transactions_sandbox" to the directory "~/.plenum". (It is already done using "data_files" argument of "setup" function call.) (hyperledger#152) * speed up tests (hyperledger#147) * speed up test_nodes_with_pool_txns.py tests * reduce the timeout in checkNodesConnected * speed up the pool_transactions test * fix the timeout in test_delay.py * fix the timeout in test_primary_election_case5.py * speed up monitoring tests * revert changes in test_instance_change_with_Delta.py * each test's running time is limited by TestRunningTimeLimitSec * increase pool interconnection timeout on zstack retry timeout * fix timeout in test_delay.py * fix tests, make them faster * speedup the tests, tidy test timeouts * add a timeout into waits * bump stp-dev==0.1.43 * Unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well (hyperledger#153) * unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well * decrease number of nodes in test_instance_cannot_become_active_with_less_than_four_servers to 13 * fix change back HA, SOV-908 (hyperledger#154) * Fix testChangeNodeHaForNonPrimary (hyperledger#157) * unskip test_change_non_primary_node_ha * fix waits name - it was renamed * verify genesis ledgers only if such option set in config * use primaryName instead of isPrimary to check that election is done * add numOfReelections parameter to waits.expectedPoolElectionTimeout * refactor test_node.py * set custom election timeout (numOfReelections=3) in changeNodeHa * remove debug prints * check VerKey is base58 for NODE txn, SOV-988 (hyperledger#158) * client using a method which makes the code more testable, update to an old test with more checks and removing async from a method where it was not needed * Fix test testChangeNodeHaForPrimary (hyperledger#160) * unskip testChangeNodeHaForPrimary * simplify conditional in changeNodeHa * node which is going down should not participate in a view change * change formating in testChangeNodeHaForPrimary * refactor to make method overridable and adding an argument to a test function * updated pypi package name for state repo and its version (hyperledger#159) * Squashed commit of the following: create LedgerInfo class and replace collections of LedgerManager by fields * rename LedgerManager.ledgers to ledgerRegistry * fix usages of LedgerManager.ledgerRegistry * use correct attribute and renamed method * bring change from master and unskip a test * use correct attribute and set state to be committed when received from catchup * formatting * up ledger-3pc-batch version to 0.2.16 * up stp-3pc-batch version to 0.1.15 * improve the check of the arguments in "generate_sovrin_pool_transactions" (hyperledger#162) * check client input for the NODE txn (hyperledger#161) * fix testInstChangeWithLowerRatioThanDelta - decrease number of messages in a batch * decrease timeout to fix provoke_and_wait_for_view_change * fixing test * Fix post 3pc-batch merge (hyperledger#163) * remove development options from Jenkinsfile * skip some broken tests * Update versions and skip windows build steps (hyperledger#164) * switch off windows related build steps * update versions of ledger and stp * fix name * up ledger and stp version * skip test testNodeRequestingTxns * using correct timeout in tests * move some enums to correct location, using ledger's string encoding methods and test for stashing requests * bump dependency version * fix timeout in tests * make order of ledger sync customizable, add tranformer for transactions before adding to ledger, update seqno map db for transactions from catchup and update tests to check this * temporarily change config variable * fix regression where node starts catchup process if sent a consistency proff * bumping dependency version * bumping dependency version * bumping dependency version * Fix redundant reconnections and refactor (hyperledger#165) * refactor retryForExpected * use maintainConnections in retryForExpected instead of connect * refactor resendRequests * up stp version to 0.1.49 * replace CLIENT_REQACK_TIMEOUT by CLIENT_REPLY_TIMEOUT when calling _filterExpected for REPLYs * up ledger to 0.2.19 * Add new helper-methods for conversion base58 to hex (hyperledger#166) * increment state-trie and ledger versions (hyperledger#168) * Make changes to support ChunkedFileStore (hyperledger#167) * update _defaultStore of Ledger to make it correspond to original one * remove initialization of domain ledger - now it is done by ledger and file store it uses * replace manual copying of transaction files by usage of defaultFile * increase timeout for testNodeRequestingTxns * skip test testInstChangeWithLowerRatioThanDelta because it fails intermittently * up ledger version to 0.2.28 * Bump dependency version * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node (hyperledger#172) * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node * Fixed failing tests * Fixed unsed imports * Update conftest.py * Increased test time global limit * Reverted timeouts * Added logs * Fixed filtering CPs * Fixed filtering CPs * Fixed filtering CPs * Input validation (hyperledger#170) * Input sanitization: Add base logic and test cases * Input sanitization: add NODE and NYM txn, fixes * Input sanitization: implement node and client messages * roll away node to node validation * role field is option for NYM txn * fix tests * fixes for sovrin-node * implement validation for merkle root hash * uncomment new ConsistencyProof message implementation * add "nullable" property for FieldBase * fix usage of alphabet * add JsonField * add validation on message level * use hash size range instead of one value; use base58.alphabet instead of own list * fix usage hasSizes; made error comment more verbose * uncomment new implementation LedgerStatus message * uncomment new implementation for Prepare message and fix fields types * roll away node to node checkes * uncomment some new message implementations * use HexField instead of MerkleRootHashField for state_root and rxn_root in PrePrepare * make message about hash length in MerkleRootField more verbose * limit timestamp field by 253402290000.0 * modify randomString to remove length limitation * add tests for merkle root field validation * make randomString work not only with indexable containers * refactor tests for merkle root * fix timestamp validation * add test for timestamp validation * update validation of timestamp * check each node connected * improve log messages * fix testAdd2NewNodes * Improvement of validation rules (hyperledger#173) * fix Node txn, INDY-9 * add tests for the Node operation * add checking that number of init parameters of message is equal to number of fields declared in schema * use OrderedDict in __dict__ * add test for merkle, fix logic * use OrderedDict for _fields and update all realted methods * - Provided all the disabled tests with the ticket references. (hyperledger#176) - Enabled the tests testChangeNodeHaForPrimary and testChangeNodeHaForNonPrimary on non-Windows platforms. - Commented out parametrizers at the disabled tests in plenum.test.input_validation.test_common_checks module. * more removal of op to the top since it can affect checking of a number of args * Make SERVICES field required for add NODE txn * Added another test scenario for catchup (hyperledger#178) * override items, keys, values methods of Mapping in MessageBase to make it compatible * Added another test scenario for catchup fix * clean MessageBase * INDY-73: Removed a commented out @skipif decorator (hyperledger#177) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * Add base58 str class; destnode; destnym; * use - as default ppSeqNo instead of -1 * overtride __ftr__ for MessageBase * comment checking of presence of SERVICE field * move alphabet validation from _specific_validation to avoid accidental overriding of it * INDY-73: Added ticket references to test stubs (hyperledger#180) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * - Added ticket references to the not implemented tests which had not contained them yet. - Uncommented testStopScriptIfNodeIsRunning and disabled it since it fails. * Added raising NotImplementedError to not implemented tests. * Move all specific checks to from validate method * Eliminate multiple inheritance * Enabled and updated testNodesConnectsWhenOneNodeIsLate (hyperledger#181) * Enabled and updated testNodesConnectsWhenOneNodeIsLate * updated assertions logic * INDY-109 (hyperledger#184) * updates to test helpers and change in forwarding requests logic (hyperledger#187) * Use secure random number generator for randomString() Current usage of random.choice() is not secure so replaced that with libsodium provided randombytes_uniform() which is secure and also available on all platforms where libsodium is available * Fixes for INDY-71 (hyperledger#185) * Refactor check_endpoint_valid func * Fix validation, INDY-71 * Fix imports * Do not connect to a new, non-active node * Remove confusing comments * Minor refactor * Fix test testAddInactiveNodeThenActivate * Validation for view change and 3pc messages (hyperledger#193) * add exception for type error and raise it in MessageValidator * make error message in ChooseField more verbose * uncomment new implementations of Reelection, Primary and InstanceChange messages * update and fix testInstanceChangeMsgTypeChecking * fix testDiscardInstChngMsgFrmPastView * use NonNegativeField instead of TimestampFied inTieAmountField for its seconds part * make name pf test for merkle root field more standard * move tests for fields to own package * add missing empty line * remove ununsed imports * refactor test for timestamp * remove todo about code duplication since it is not valid anymore * add tests for some field types * make IterableField check type of inner field and that it is not none * add test for hex field * add test for ledger id * add test for request id field * override __eq__ for MessageBase * use NonNegativeNumberField instead of TimestampField in RequestIdentifierField * use IdentifierField instead of NonEmptyStringField in RequestIdentifierField * update test for request identifier * add test for IdentifierField * update test for RequestIdentifier field * implement validation for IdentifierField and reimplement DestNodeField using it * uncomment new implementations for Propagate, Prepare, Commit, Ordered * Refactor base58 check; add Verkey check; dest node; dest nym; merkelroot; * move test for base58 field to field validation package * remove unused import * add checking of max length hash to merkle tree root field test * update test for identifier field * update test for request identifier field * fix formatting * fix checking of zero * remove unused commented line * rename test_non_negative_number to test_positive_number * Make verkey validator accept empty strings * Add super's __init__ params to child calsses * Improve getMatchedHelpableMsg logic (hyperledger#195) * Use original batch parameters in all tests except the ones that use batches and use alternate of eventuallyAll to check a collection of functions under a timeout (hyperledger#190) * updates to test helpers and change in forwarding requests logic * overriding batch params in tests * use correct timeouts in tests * accounting for nomination delay * Disable view change (hyperledger#196) * repaired calls to deprecated methods * Added unsafe 'disable_view_change' for testing - Added 'unsafe' attribute to default config. "unsafe" is a set of attributes that can set certain behaviors that are not safe, for example, 'disable_view_change' disables view changes from happening. This might be useful in testing scenarios, but never in a live network. - Added 'unsafe' behavior 'disable_view_change' which causes Monitor to always return false when asked if master instance performance is degraded. - Refactored view change fixture so it could be reused to test new 'disable_view_change' behavior. * fixed import for missed fixture in test * Fix validation types (hyperledger#197) * Enable and update testNodeDiscardMessageFromUnknownView (hyperledger#194) * added equality method for MessageBase * added ability to cancel created delayers * enabled and updated testNodeDiscardMessageFromUnknownView * updated test to recent changes in node logic * reverted recent changes not necessary for now * Catch up demoted (hyperledger#201) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * Fixes for dead catchup (hyperledger#202) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * up stp version to 0.1.56 * up stp version to 0.1.57 * fixed tests * fix test * Disable the test: INDY-147. * fix README * fix setup.py * up stp version to 0.1.10 Signed-off-by: Muzychenko Victor <[email protected]> Sign-off-executed-by: toktar <[email protected]> Approved-at: h-master

* rename validateReq to validate * add config=None argument to validate method to make sub and super signatures match * rename applyReq to apply * add super initialization, update usage of super methods * update docstring of commit * remove commitReq since it does the same as commit * adding timeout to test utility method * changes for load testing * sanitising log messages * Versioning fix (hyperledger#113) * changes for request handlers * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * changing to rlp encoding * moving error codes to stp * upgrading version of stp * req_handler changes * fixing import * temp commit * Stp (hyperledger#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * minor changes to serialization * Plenum test runner fix (hyperledger#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * updating apply of request handler * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (hyperledger#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (hyperledger#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * leveldb HashStore * parametrize test timeouts (hyperledger#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * removing commented code and fixing bugs in pool request handler * lowering Delta in pool_transactions conftest * fix jenkins build (hyperledger#123) * fixed deps in setup.py * added pypi publishing * updated deps * updated deps stp-dev -> stp-perf-imp * removed unnecessary sources * updated setup.py to hold correct deps names and self name for publishing * fixed plenum for the latest stp (hyperledger#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (hyperledger#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * sending last pre-prepare sequence number in consistency proof so already processed request can be ignored, re-enabling tests and re-adding proper timeouts * Feature Added: Deb build * Feature Added: Deb build * Do view change if a primary is disconnected (hyperledger#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * processing stashed ordered requests after all ledgers have caught up and applying any request if mode changed while processing and removing obsolete code from pool manager * Unclosed file warnings (hyperledger#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * refactoring, documentation and updating setup to get rid of pyorient * refactoring * Updated ledger dep * deprecating orientdb * check state equality too while checking ledger equality in tests * removing obsolete test helper method * Agent generalization (hyperledger#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * Add default values for role and verkey in state and fix test timeouts * refactor * import logger * updated to use new API from shared lib * updated to use new API from shared lib * handle pyorient import * handle pyorient import * increment stp version * Enabled the tests in plenum.test.test_log_rotation module. (hyperledger#135) * increment stp version * close files in tests * check for close method before calling it (hyperledger#136) * minor fix in batch creation logic and caching primary replica no in node to avoid recomputation * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/common/test_network_setup.py # plenum/server/node.py # plenum/test/test_node.py # setup.py * add missing __init__ for member package * added missed __init__.py for common/member * skip temporarily a failing test * fix primary disconnection and new primary not being same as old for master instance * add more tests for primary disconnection, same primary not elected in the next view and refactor * add new test for minor network glitch with primary * add missing parameter to bootstrap test network (hyperledger#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * minor change in a test and removing sip statements * fix the catchup tests (hyperledger#140) * make tests a little robust * fix test, check for disconnections after processing sent and received messages and little documentation * Close SOV-976 (hyperledger#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/test/node_catchup/test_new_node_catchup.py # setup.py * up stp to 0.1.42 (hyperledger#148) * refactor, enable test, adding timeout for view change and create remotes as soon as nodestack starts * fix broken test * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (hyperledger#149) * increment stp * increment ledger * up stp to 0.1.14 * move state and kv storage to 'state' repo * get rid of orientdb code * fix tests * fix generation of trustee txns * fix genesis txns for trustee * fix generation of trustee txns (hyperledger#151) * remove obsolete code * client to attempt establishing connections with nodes on start * refactor monitor.isMasterThroughputTooLow * update method for forcing view change cna checking it; intoduce wait method for it * use provoke_and_wait_for_view_change instead of eventually and check * remove unused dependency and skip tests - removed unused dependencies - minor fixes - skip tests that fail in master too (we have separate tickets for this) * increment ledger * increment ledger * enable debug logging in tests * using reverted changes in state interface * increment state version * Removed redundant copying of the file "pool_transactions_sandbox" to the directory "~/.plenum". (It is already done using "data_files" argument of "setup" function call.) (hyperledger#152) * speed up tests (hyperledger#147) * speed up test_nodes_with_pool_txns.py tests * reduce the timeout in checkNodesConnected * speed up the pool_transactions test * fix the timeout in test_delay.py * fix the timeout in test_primary_election_case5.py * speed up monitoring tests * revert changes in test_instance_change_with_Delta.py * each test's running time is limited by TestRunningTimeLimitSec * increase pool interconnection timeout on zstack retry timeout * fix timeout in test_delay.py * fix tests, make them faster * speedup the tests, tidy test timeouts * add a timeout into waits * bump stp-dev==0.1.43 * Unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well (hyperledger#153) * unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well * decrease number of nodes in test_instance_cannot_become_active_with_less_than_four_servers to 13 * fix change back HA, SOV-908 (hyperledger#154) * Fix testChangeNodeHaForNonPrimary (hyperledger#157) * unskip test_change_non_primary_node_ha * fix waits name - it was renamed * verify genesis ledgers only if such option set in config * use primaryName instead of isPrimary to check that election is done * add numOfReelections parameter to waits.expectedPoolElectionTimeout * refactor test_node.py * set custom election timeout (numOfReelections=3) in changeNodeHa * remove debug prints * check VerKey is base58 for NODE txn, SOV-988 (hyperledger#158) * client using a method which makes the code more testable, update to an old test with more checks and removing async from a method where it was not needed * Fix test testChangeNodeHaForPrimary (hyperledger#160) * unskip testChangeNodeHaForPrimary * simplify conditional in changeNodeHa * node which is going down should not participate in a view change * change formating in testChangeNodeHaForPrimary * refactor to make method overridable and adding an argument to a test function * updated pypi package name for state repo and its version (hyperledger#159) * Squashed commit of the following: create LedgerInfo class and replace collections of LedgerManager by fields * rename LedgerManager.ledgers to ledgerRegistry * fix usages of LedgerManager.ledgerRegistry * use correct attribute and renamed method * bring change from master and unskip a test * use correct attribute and set state to be committed when received from catchup * formatting * up ledger-3pc-batch version to 0.2.16 * up stp-3pc-batch version to 0.1.15 * improve the check of the arguments in "generate_sovrin_pool_transactions" (hyperledger#162) * check client input for the NODE txn (hyperledger#161) * fix testInstChangeWithLowerRatioThanDelta - decrease number of messages in a batch * decrease timeout to fix provoke_and_wait_for_view_change * fixing test * Fix post 3pc-batch merge (hyperledger#163) * remove development options from Jenkinsfile * skip some broken tests * Update versions and skip windows build steps (hyperledger#164) * switch off windows related build steps * update versions of ledger and stp * fix name * up ledger and stp version * skip test testNodeRequestingTxns * using correct timeout in tests * move some enums to correct location, using ledger's string encoding methods and test for stashing requests * bump dependency version * fix timeout in tests * make order of ledger sync customizable, add tranformer for transactions before adding to ledger, update seqno map db for transactions from catchup and update tests to check this * temporarily change config variable * fix regression where node starts catchup process if sent a consistency proff * bumping dependency version * bumping dependency version * bumping dependency version * Fix redundant reconnections and refactor (hyperledger#165) * refactor retryForExpected * use maintainConnections in retryForExpected instead of connect * refactor resendRequests * up stp version to 0.1.49 * replace CLIENT_REQACK_TIMEOUT by CLIENT_REPLY_TIMEOUT when calling _filterExpected for REPLYs * up ledger to 0.2.19 * Add new helper-methods for conversion base58 to hex (hyperledger#166) * increment state-trie and ledger versions (hyperledger#168) * Make changes to support ChunkedFileStore (hyperledger#167) * update _defaultStore of Ledger to make it correspond to original one * remove initialization of domain ledger - now it is done by ledger and file store it uses * replace manual copying of transaction files by usage of defaultFile * increase timeout for testNodeRequestingTxns * skip test testInstChangeWithLowerRatioThanDelta because it fails intermittently * up ledger version to 0.2.28 * Bump dependency version * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node (hyperledger#172) * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node * Fixed failing tests * Fixed unsed imports * Update conftest.py * Increased test time global limit * Reverted timeouts * Added logs * Fixed filtering CPs * Fixed filtering CPs * Fixed filtering CPs * Input validation (hyperledger#170) * Input sanitization: Add base logic and test cases * Input sanitization: add NODE and NYM txn, fixes * Input sanitization: implement node and client messages * roll away node to node validation * role field is option for NYM txn * fix tests * fixes for sovrin-node * implement validation for merkle root hash * uncomment new ConsistencyProof message implementation * add "nullable" property for FieldBase * fix usage of alphabet * add JsonField * add validation on message level * use hash size range instead of one value; use base58.alphabet instead of own list * fix usage hasSizes; made error comment more verbose * uncomment new implementation LedgerStatus message * uncomment new implementation for Prepare message and fix fields types * roll away node to node checkes * uncomment some new message implementations * use HexField instead of MerkleRootHashField for state_root and rxn_root in PrePrepare * make message about hash length in MerkleRootField more verbose * limit timestamp field by 253402290000.0 * modify randomString to remove length limitation * add tests for merkle root field validation * make randomString work not only with indexable containers * refactor tests for merkle root * fix timestamp validation * add test for timestamp validation * update validation of timestamp * check each node connected * improve log messages * fix testAdd2NewNodes * Improvement of validation rules (hyperledger#173) * fix Node txn, INDY-9 * add tests for the Node operation * add checking that number of init parameters of message is equal to number of fields declared in schema * use OrderedDict in __dict__ * add test for merkle, fix logic * use OrderedDict for _fields and update all realted methods * - Provided all the disabled tests with the ticket references. (hyperledger#176) - Enabled the tests testChangeNodeHaForPrimary and testChangeNodeHaForNonPrimary on non-Windows platforms. - Commented out parametrizers at the disabled tests in plenum.test.input_validation.test_common_checks module. * more removal of op to the top since it can affect checking of a number of args * Make SERVICES field required for add NODE txn * Added another test scenario for catchup (hyperledger#178) * override items, keys, values methods of Mapping in MessageBase to make it compatible * Added another test scenario for catchup fix * clean MessageBase * INDY-73: Removed a commented out @skipif decorator (hyperledger#177) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * Add base58 str class; destnode; destnym; * use - as default ppSeqNo instead of -1 * overtride __ftr__ for MessageBase * comment checking of presence of SERVICE field * move alphabet validation from _specific_validation to avoid accidental overriding of it * INDY-73: Added ticket references to test stubs (hyperledger#180) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * - Added ticket references to the not implemented tests which had not contained them yet. - Uncommented testStopScriptIfNodeIsRunning and disabled it since it fails. * Added raising NotImplementedError to not implemented tests. * Move all specific checks to from validate method * Eliminate multiple inheritance * Enabled and updated testNodesConnectsWhenOneNodeIsLate (hyperledger#181) * Enabled and updated testNodesConnectsWhenOneNodeIsLate * updated assertions logic * INDY-109 (hyperledger#184) * updates to test helpers and change in forwarding requests logic (hyperledger#187) * Use secure random number generator for randomString() Current usage of random.choice() is not secure so replaced that with libsodium provided randombytes_uniform() which is secure and also available on all platforms where libsodium is available * Fixes for INDY-71 (hyperledger#185) * Refactor check_endpoint_valid func * Fix validation, INDY-71 * Fix imports * Do not connect to a new, non-active node * Remove confusing comments * Minor refactor * Fix test testAddInactiveNodeThenActivate * Validation for view change and 3pc messages (hyperledger#193) * add exception for type error and raise it in MessageValidator * make error message in ChooseField more verbose * uncomment new implementations of Reelection, Primary and InstanceChange messages * update and fix testInstanceChangeMsgTypeChecking * fix testDiscardInstChngMsgFrmPastView * use NonNegativeField instead of TimestampFied inTieAmountField for its seconds part * make name pf test for merkle root field more standard * move tests for fields to own package * add missing empty line * remove ununsed imports * refactor test for timestamp * remove todo about code duplication since it is not valid anymore * add tests for some field types * make IterableField check type of inner field and that it is not none * add test for hex field * add test for ledger id * add test for request id field * override __eq__ for MessageBase * use NonNegativeNumberField instead of TimestampField in RequestIdentifierField * use IdentifierField instead of NonEmptyStringField in RequestIdentifierField * update test for request identifier * add test for IdentifierField * update test for RequestIdentifier field * implement validation for IdentifierField and reimplement DestNodeField using it * uncomment new implementations for Propagate, Prepare, Commit, Ordered * Refactor base58 check; add Verkey check; dest node; dest nym; merkelroot; * move test for base58 field to field validation package * remove unused import * add checking of max length hash to merkle tree root field test * update test for identifier field * update test for request identifier field * fix formatting * fix checking of zero * remove unused commented line * rename test_non_negative_number to test_positive_number * Make verkey validator accept empty strings * Add super's __init__ params to child calsses * Improve getMatchedHelpableMsg logic (hyperledger#195) * Use original batch parameters in all tests except the ones that use batches and use alternate of eventuallyAll to check a collection of functions under a timeout (hyperledger#190) * updates to test helpers and change in forwarding requests logic * overriding batch params in tests * use correct timeouts in tests * accounting for nomination delay * Disable view change (hyperledger#196) * repaired calls to deprecated methods * Added unsafe 'disable_view_change' for testing - Added 'unsafe' attribute to default config. "unsafe" is a set of attributes that can set certain behaviors that are not safe, for example, 'disable_view_change' disables view changes from happening. This might be useful in testing scenarios, but never in a live network. - Added 'unsafe' behavior 'disable_view_change' which causes Monitor to always return false when asked if master instance performance is degraded. - Refactored view change fixture so it could be reused to test new 'disable_view_change' behavior. * fixed import for missed fixture in test * Fix validation types (hyperledger#197) * Enable and update testNodeDiscardMessageFromUnknownView (hyperledger#194) * added equality method for MessageBase * added ability to cancel created delayers * enabled and updated testNodeDiscardMessageFromUnknownView * updated test to recent changes in node logic * reverted recent changes not necessary for now * Catch up demoted (hyperledger#201) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * Fixes for dead catchup (hyperledger#202) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * up stp version to 0.1.56 * up stp version to 0.1.57 * fixed tests * fix test * Disable the test: INDY-147. * Make random string generation efficient The current algorithm requires calling the libsodium random function 'size' times where size is the expected length of random string Changed this so that this will require at the most '(size/64)' number of calls. Also we do not need the chars list now we always use only alphabets and digits available in hex this makes it very easy and still gives us what we need. Every instance of usage of this function has been checked and nowhere the 'chars' parameter is used * Change to pass exact size to libsodium randombytes() function This probably is cryptographically more accurate * Fix catchup when commit (hyperledger#203) * remove 3PC messages that are already ordered during catch-up * Added more tests * fix catch-up in between 3pc msgs * fix tests * update validation rule for verkey (hyperledger#208) * fix an incorrect test (hyperledger#207) * fix an incorrect test * refactoring test * remove un-necessary parameter (hyperledger#210) * Revert unordered batches before starting catch-up (not after) (hyperledger#209) * Revert unordered batches before starting catch-up (not after) * add test for revert of the state before catchup * improve the test * Do not enqueue old pre-prepares (hyperledger#211) * ignore pp requests with ppSeqNo less than expected * minor refactor * Optimisations and a testing util (hyperledger#212) * add todo, fixing log levels, commenting out an expensive area of code and adding capability to diable monitor in tests * an optimisation avoiding re-calculation of merkle proof * remove commented code and add test * update names * add test fils for validators * add test for commit message * add test for ConsistencyProof message * update test for Commit message * add test for InstanceChange message * add test for LedgerStatus message * add test for nomination message * add test for Ordered message * add tests for Prepare and PrePrepare messages * add test for Primary message * add test for PROPAGATE message * add test for Reelection message * remove unused imports * Election bug fix (hyperledger#215) * Election bug fix if `f` nodes do not send Primary and the node is the last one to send Primary, then it was not able to decide the primary, since a node was doing "primary setting" only on receiving a Primary message, changed that to try "setting primary" after sending Primary too * update test with more checks Signed-off-by: Andrei Goncharov <[email protected]> Sign-off-executed-by: toktar <[email protected]> Approved-at: h-master

* add super initialization, update usage of super methods * update docstring of commit * remove commitReq since it does the same as commit * adding timeout to test utility method * changes for load testing * sanitising log messages * Versioning fix (hyperledger#113) * changes for request handlers * [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16 * changing to rlp encoding * moving error codes to stp * upgrading version of stp * req_handler changes * fixing import * temp commit * Stp (hyperledger#116) * use common stp interface for changing of node's HA * get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces. * fix tests; increment stp version * add AuthMode enum instead of auto parameter * fixed testConnectWithoutKeySharingFails test * increased test timeout * minor changes to serialization * Plenum test runner fix (hyperledger#115) * Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file. * Corrected test results summary patterns in plenum test runner. * updating apply of request handler * fix import of OP_FIELD_NAME * up version of stp to 0.1.24 * Agent issuer wallet restoration (hyperledger#117) * supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence * minor changes in list keyrings to show issuer wallet for agent as well * minor change to print base wallet first and then any other related wallets * up stp version to 0.1.26 * skip tests in test_new_node_catchup.py * scheduling primary selection * Skip failing tests (hyperledger#122) * skeip test testChangeNodeHaForNonPrimary due to SOV-941 * skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940 * leveldb HashStore * parametrize test timeouts (hyperledger#120) * rename expectedWait to expectedNodeInterconnectionTime * add signature for other time expect functions * use named timeouts in conftest * move totalConnections from waits to utils * use named timeout in ensureElectionsDone * use float instead of int for seconds * use default args of ensureElectionsDone where it is possible * use named argument for timeouts * use named timeout in helper * use names for parameters * rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one * use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts * add comments for two functions with similar names * rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived * rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests * add 'returns' to docstrings * fix passing of args to waits.expectedElectionTimeout * use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible * update doc lines * create waitForViewChange and use it in ensureView * replace usages of checkViewNoForNodes with eventually by waitForViewChange * add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck * rename checkMsg to checkMessageReceived * rename sendMsgAndCheck to sendMessageAndCheckDelivery * add docstring to sendMessageAndCheckDelivery * remove unused helper function * add expectedGetReadyTimeout and use it in checkPoolReady * rename overrideTimeout parameter to customTimeout in checkNodesConnected * use default timeout of checkNodesConnected * create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them * add todo for ensureDisconnectedToNodes * update waits.expectedPoolLedgerCheck * add todo for checkNodesParticipating * add requestIds parameter for waitForSufficientRepliesForRequests * update docstring of waitForSufficientRepliesForRequests * fix waitForSufficientRepliesForRequests * remove unused imports from test_log_filtering * use named timeout in test_status_command * use waits.expectedTransactionExecutionTime in testTransactions * refactor testTransactions * use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper * use named timeout in test_basic_client_commands * use named timeout in helper.checkRequest * create waitClientConnected and use it instead of checkClientConnected with eventually * add docstrings * rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it * rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout * rename checkAllNodesStarted to waitAllNodesStarted * fix default value of customTimeout of checkPoolReady * create waitAllNodesUp and use it instead of checkAllNodesUp with eventually * create waitReplyCount and use instead of checkReplyCount and eventually * use named timeouts in test_client * use named timeouts in some more tests * add basic implementation for waits.expectedCatchupTime and use it * make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter * use named timeout in testNodeDiscardMessageFromUnknownView * use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary * rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default * update timeouts in testNodeDiscardMessageFromUnknownView * create waits.expectedRequestStashingTime and use it * add TODO to test_catchup_delayed_nodes * create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality * use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality * use named timeout in testOrderingCase2 * add waits.expectedNominationTimeout and use it * use named timeout in some more tests * add missing empty lines * update waits * add 'TODO[slow-factor]' * update timeouts in the tests * fix testTestNodeDelay and missed import * skip testMultipleRequests test * skip testStatusAfterClientAdded test * fix testInstChangeWithLowerRatioThanDelta * fix test_new_node_catchup.py * fix testChangeHaPersistsPostNodesRestart * fix testAdd2NewNodes * increase expectedElectionTimeout timeout * rename logger.warn => logger.warning * tune timeouts in the tests * make sections in waits.py * add --repat for runner.py * increase expectedCatchupTime timeout * improve runner.py * tune the test timeouts * skip some catchup tests * parametrize test timeouts * rm eventually.py * fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults * fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers * tune propagate and preprepare test timeouts * skip testNumOf*WithFPlusOneFaults * removing commented code and fixing bugs in pool request handler * lowering Delta in pool_transactions conftest * fix jenkins build (hyperledger#123) * fixed deps in setup.py * added pypi publishing * updated deps * updated deps stp-dev -> stp-perf-imp * removed unnecessary sources * updated setup.py to hold correct deps names and self name for publishing * fixed plenum for the latest stp (hyperledger#127) * fixed plenum for the latest stp * increment stp version * archive runner.py output for all agents (hyperledger#107) * archive runner.py results * using env variable NODE_NAME instaed of func param for artifacts * configured archiveArtifacts to allow empty/no archive * sending last pre-prepare sequence number in consistency proof so already processed request can be ignored, re-enabling tests and re-adding proper timeouts * Feature Added: Deb build * Feature Added: Deb build * Do view change if a primary is disconnected (hyperledger#128) * check whether function is a partial function in getCallableName * add tests for view change when primary goes down * start view change if primary went offline * use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded * processing stashed ordered requests after all ledgers have caught up and applying any request if mode changed while processing and removing obsolete code from pool manager * Unclosed file warnings (hyperledger#124) * Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter. * Fixed some causes of ResourceWarnings about unclosed files. * - Fixed some causes of ResourceWarnings about unclosed files. - Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function. * - Fixed the rest causes of ResourceWarnings about unclosed files. - Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead). * Updated stp-dev dependency to the new version. * Reverted update of stp-dev dependency. * Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher). * refactoring, documentation and updating setup to get rid of pyorient * refactoring * Updated ledger dep * deprecating orientdb * check state equality too while checking ledger equality in tests * removing obsolete test helper method * Agent generalization (hyperledger#126) * refactored test_network_setup; added some role-base helper classes for members of the network * fixed issue with wrong import and minor refactoring * more work on tests * bugfixes * interim changes during refactoring * adding init_utils * cleaning up before keys * removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine * added a check in DidVerifier to check that we have a verkey * fixed unreliable tmpdir test fixture * modifications to tests to make more robust when running from command line * changed spyable to be able to monkey patch a single method * Added a run_till_quiet, waits for quiet prods This function will wait till there is a quiet period from the prodables. Dependent on prodables correctly reporting events. Will run forever if there is always activity. * small tweek to log format to keep | alined * changes to fix some issues happening in end-to-end test * using same seed for creating keys * ignore log files * added a small test for verifier * removed some excpetion class that were moved to STP * init Local keys for client in test network * Add default values for role and verkey in state and fix test timeouts * refactor * import logger * updated to use new API from shared lib * updated to use new API from shared lib * handle pyorient import * handle pyorient import * increment stp version * Enabled the tests in plenum.test.test_log_rotation module. (hyperledger#135) * increment stp version * close files in tests * check for close method before calling it (hyperledger#136) * minor fix in batch creation logic and caching primary replica no in node to avoid recomputation * fix testNumOf*WithFPlusOneFaults tests * Updated stp dep * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/common/test_network_setup.py # plenum/server/node.py # plenum/test/test_node.py # setup.py * add missing __init__ for member package * added missed __init__.py for common/member * skip temporarily a failing test * fix primary disconnection and new primary not being same as old for master instance * add more tests for primary disconnection, same primary not elected in the next view and refactor * add new test for minor network glitch with primary * add missing parameter to bootstrap test network (hyperledger#143) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * minor change in a test and removing sip statements * fix the catchup tests (hyperledger#140) * make tests a little robust * fix test, check for disconnections after processing sent and received messages and little documentation * Close SOV-976 (hyperledger#144) * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux * add missing parameter to bootstrap test network * add missing parameter to bootstrap test network * [Closes SOV-976] Unable to create the genesis transaction files * Merge branches '3pc-batch' and 'master' of github.com:evernym/plenum into 3pc-batch # Conflicts: # plenum/test/node_catchup/test_new_node_catchup.py # setup.py * up stp to 0.1.42 (hyperledger#148) * refactor, enable test, adding timeout for view change and create remotes as soon as nodestack starts * fix broken test * [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (hyperledger#149) * increment stp * increment ledger * up stp to 0.1.14 * move state and kv storage to 'state' repo * get rid of orientdb code * fix tests * fix generation of trustee txns * fix genesis txns for trustee * fix generation of trustee txns (hyperledger#151) * remove obsolete code * client to attempt establishing connections with nodes on start * refactor monitor.isMasterThroughputTooLow * update method for forcing view change cna checking it; intoduce wait method for it * use provoke_and_wait_for_view_change instead of eventually and check * remove unused dependency and skip tests - removed unused dependencies - minor fixes - skip tests that fail in master too (we have separate tickets for this) * increment ledger * increment ledger * enable debug logging in tests * using reverted changes in state interface * increment state version * Removed redundant copying of the file "pool_transactions_sandbox" to the directory "~/.plenum". (It is already done using "data_files" argument of "setup" function call.) (hyperledger#152) * speed up tests (hyperledger#147) * speed up test_nodes_with_pool_txns.py tests * reduce the timeout in checkNodesConnected * speed up the pool_transactions test * fix the timeout in test_delay.py * fix the timeout in test_primary_election_case5.py * speed up monitoring tests * revert changes in test_instance_change_with_Delta.py * each test's running time is limited by TestRunningTimeLimitSec * increase pool interconnection timeout on zstack retry timeout * fix timeout in test_delay.py * fix tests, make them faster * speedup the tests, tidy test timeouts * add a timeout into waits * bump stp-dev==0.1.43 * Unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well (hyperledger#153) * unskip testProtocolInstanceCannotBecomeActiveWithLessThanFourServers since it works well * decrease number of nodes in test_instance_cannot_become_active_with_less_than_four_servers to 13 * fix change back HA, SOV-908 (hyperledger#154) * Fix testChangeNodeHaForNonPrimary (hyperledger#157) * unskip test_change_non_primary_node_ha * fix waits name - it was renamed * verify genesis ledgers only if such option set in config * use primaryName instead of isPrimary to check that election is done * add numOfReelections parameter to waits.expectedPoolElectionTimeout * refactor test_node.py * set custom election timeout (numOfReelections=3) in changeNodeHa * remove debug prints * check VerKey is base58 for NODE txn, SOV-988 (hyperledger#158) * client using a method which makes the code more testable, update to an old test with more checks and removing async from a method where it was not needed * Fix test testChangeNodeHaForPrimary (hyperledger#160) * unskip testChangeNodeHaForPrimary * simplify conditional in changeNodeHa * node which is going down should not participate in a view change * change formating in testChangeNodeHaForPrimary * refactor to make method overridable and adding an argument to a test function * updated pypi package name for state repo and its version (hyperledger#159) * Squashed commit of the following: create LedgerInfo class and replace collections of LedgerManager by fields * rename LedgerManager.ledgers to ledgerRegistry * fix usages of LedgerManager.ledgerRegistry * use correct attribute and renamed method * bring change from master and unskip a test * use correct attribute and set state to be committed when received from catchup * formatting * up ledger-3pc-batch version to 0.2.16 * up stp-3pc-batch version to 0.1.15 * improve the check of the arguments in "generate_sovrin_pool_transactions" (hyperledger#162) * check client input for the NODE txn (hyperledger#161) * fix testInstChangeWithLowerRatioThanDelta - decrease number of messages in a batch * decrease timeout to fix provoke_and_wait_for_view_change * fixing test * Fix post 3pc-batch merge (hyperledger#163) * remove development options from Jenkinsfile * skip some broken tests * Update versions and skip windows build steps (hyperledger#164) * switch off windows related build steps * update versions of ledger and stp * fix name * up ledger and stp version * skip test testNodeRequestingTxns * using correct timeout in tests * move some enums to correct location, using ledger's string encoding methods and test for stashing requests * bump dependency version * fix timeout in tests * make order of ledger sync customizable, add tranformer for transactions before adding to ledger, update seqno map db for transactions from catchup and update tests to check this * temporarily change config variable * fix regression where node starts catchup process if sent a consistency proff * bumping dependency version * bumping dependency version * bumping dependency version * Fix redundant reconnections and refactor (hyperledger#165) * refactor retryForExpected * use maintainConnections in retryForExpected instead of connect * refactor resendRequests * up stp version to 0.1.49 * replace CLIENT_REQACK_TIMEOUT by CLIENT_REPLY_TIMEOUT when calling _filterExpected for REPLYs * up ledger to 0.2.19 * Add new helper-methods for conversion base58 to hex (hyperledger#166) * increment state-trie and ledger versions (hyperledger#168) * Make changes to support ChunkedFileStore (hyperledger#167) * update _defaultStore of Ledger to make it correspond to original one * remove initialization of domain ledger - now it is done by ledger and file store it uses * replace manual copying of transaction files by usage of defaultFile * increase timeout for testNodeRequestingTxns * skip test testInstChangeWithLowerRatioThanDelta because it fails intermittently * up ledger version to 0.2.28 * Bump dependency version * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node (hyperledger#172) * [Closes SOV-980] Bug Fixed: A node should start catchup process if it realises that it has lagged behind the other node * Fixed failing tests * Fixed unsed imports * Update conftest.py * Increased test time global limit * Reverted timeouts * Added logs * Fixed filtering CPs * Fixed filtering CPs * Fixed filtering CPs * Input validation (hyperledger#170) * Input sanitization: Add base logic and test cases * Input sanitization: add NODE and NYM txn, fixes * Input sanitization: implement node and client messages * roll away node to node validation * role field is option for NYM txn * fix tests * fixes for sovrin-node * implement validation for merkle root hash * uncomment new ConsistencyProof message implementation * add "nullable" property for FieldBase * fix usage of alphabet * add JsonField * add validation on message level * use hash size range instead of one value; use base58.alphabet instead of own list * fix usage hasSizes; made error comment more verbose * uncomment new implementation LedgerStatus message * uncomment new implementation for Prepare message and fix fields types * roll away node to node checkes * uncomment some new message implementations * use HexField instead of MerkleRootHashField for state_root and rxn_root in PrePrepare * make message about hash length in MerkleRootField more verbose * limit timestamp field by 253402290000.0 * modify randomString to remove length limitation * add tests for merkle root field validation * make randomString work not only with indexable containers * refactor tests for merkle root * fix timestamp validation * add test for timestamp validation * update validation of timestamp * check each node connected * improve log messages * fix testAdd2NewNodes * Improvement of validation rules (hyperledger#173) * fix Node txn, INDY-9 * add tests for the Node operation * add checking that number of init parameters of message is equal to number of fields declared in schema * use OrderedDict in __dict__ * add test for merkle, fix logic * use OrderedDict for _fields and update all realted methods * - Provided all the disabled tests with the ticket references. (hyperledger#176) - Enabled the tests testChangeNodeHaForPrimary and testChangeNodeHaForNonPrimary on non-Windows platforms. - Commented out parametrizers at the disabled tests in plenum.test.input_validation.test_common_checks module. * more removal of op to the top since it can affect checking of a number of args * Make SERVICES field required for add NODE txn * Added another test scenario for catchup (hyperledger#178) * override items, keys, values methods of Mapping in MessageBase to make it compatible * Added another test scenario for catchup fix * clean MessageBase * INDY-73: Removed a commented out @skipif decorator (hyperledger#177) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * Add base58 str class; destnode; destnym; * use - as default ppSeqNo instead of -1 * overtride __ftr__ for MessageBase * comment checking of presence of SERVICE field * move alphabet validation from _specific_validation to avoid accidental overriding of it * INDY-73: Added ticket references to test stubs (hyperledger#180) * Removed a commented out @skipif decorator. * Updated a ticket reference at a disabled test. * - Added ticket references to the not implemented tests which had not contained them yet. - Uncommented testStopScriptIfNodeIsRunning and disabled it since it fails. * Added raising NotImplementedError to not implemented tests. * Move all specific checks to from validate method * Eliminate multiple inheritance * Enabled and updated testNodesConnectsWhenOneNodeIsLate (hyperledger#181) * Enabled and updated testNodesConnectsWhenOneNodeIsLate * updated assertions logic * INDY-109 (hyperledger#184) * updates to test helpers and change in forwarding requests logic (hyperledger#187) * Use secure random number generator for randomString() Current usage of random.choice() is not secure so replaced that with libsodium provided randombytes_uniform() which is secure and also available on all platforms where libsodium is available * Fixes for INDY-71 (hyperledger#185) * Refactor check_endpoint_valid func * Fix validation, INDY-71 * Fix imports * Do not connect to a new, non-active node * Remove confusing comments * Minor refactor * Fix test testAddInactiveNodeThenActivate * Validation for view change and 3pc messages (hyperledger#193) * add exception for type error and raise it in MessageValidator * make error message in ChooseField more verbose * uncomment new implementations of Reelection, Primary and InstanceChange messages * update and fix testInstanceChangeMsgTypeChecking * fix testDiscardInstChngMsgFrmPastView * use NonNegativeField instead of TimestampFied inTieAmountField for its seconds part * make name pf test for merkle root field more standard * move tests for fields to own package * add missing empty line * remove ununsed imports * refactor test for timestamp * remove todo about code duplication since it is not valid anymore * add tests for some field types * make IterableField check type of inner field and that it is not none * add test for hex field * add test for ledger id * add test for request id field * override __eq__ for MessageBase * use NonNegativeNumberField instead of TimestampField in RequestIdentifierField * use IdentifierField instead of NonEmptyStringField in RequestIdentifierField * update test for request identifier * add test for IdentifierField * update test for RequestIdentifier field * implement validation for IdentifierField and reimplement DestNodeField using it * uncomment new implementations for Propagate, Prepare, Commit, Ordered * Refactor base58 check; add Verkey check; dest node; dest nym; merkelroot; * move test for base58 field to field validation package * remove unused import * add checking of max length hash to merkle tree root field test * update test for identifier field * update test for request identifier field * fix formatting * fix checking of zero * remove unused commented line * rename test_non_negative_number to test_positive_number * Make verkey validator accept empty strings * Add super's __init__ params to child calsses * Improve getMatchedHelpableMsg logic (hyperledger#195) * Use original batch parameters in all tests except the ones that use batches and use alternate of eventuallyAll to check a collection of functions under a timeout (hyperledger#190) * updates to test helpers and change in forwarding requests logic * overriding batch params in tests * use correct timeouts in tests * accounting for nomination delay * Disable view change (hyperledger#196) * repaired calls to deprecated methods * Added unsafe 'disable_view_change' for testing - Added 'unsafe' attribute to default config. "unsafe" is a set of attributes that can set certain behaviors that are not safe, for example, 'disable_view_change' disables view changes from happening. This might be useful in testing scenarios, but never in a live network. - Added 'unsafe' behavior 'disable_view_change' which causes Monitor to always return false when asked if master instance performance is degraded. - Refactored view change fixture so it could be reused to test new 'disable_view_change' behavior. * fixed import for missed fixture in test * Fix validation types (hyperledger#197) * Enable and update testNodeDiscardMessageFromUnknownView (hyperledger#194) * added equality method for MessageBase * added ability to cancel created delayers * enabled and updated testNodeDiscardMessageFromUnknownView * updated test to recent changes in node logic * reverted recent changes not necessary for now * Catch up demoted (hyperledger#201) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * Fixes for dead catchup (hyperledger#202) * add tests for the issues with catch-up after demotion/disconnection * updates replica's last_ordered_pp_seq_no after catch-up * disconnect a demoted node * up stp version to 0.1.56 * up stp version to 0.1.57 * fixed tests * fix test * Disable the test: INDY-147. * Make random string generation efficient The current algorithm requires calling the libsodium random function 'size' times where size is the expected length of random string Changed this so that this will require at the most '(size/64)' number of calls. Also we do not need the chars list now we always use only alphabets and digits available in hex this makes it very easy and still gives us what we need. Every instance of usage of this function has been checked and nowhere the 'chars' parameter is used * Change to pass exact size to libsodium randombytes() function This probably is cryptographically more accurate * Fix catchup when commit (hyperledger#203) * remove 3PC messages that are already ordered during catch-up * Added more tests * fix catch-up in between 3pc msgs * fix tests * update validation rule for verkey (hyperledger#208) * fix an incorrect test (hyperledger#207) * fix an incorrect test * refactoring test * remove un-necessary parameter (hyperledger#210) * Revert unordered batches before starting catch-up (not after) (hyperledger#209) * Revert unordered batches before starting catch-up (not after) * add test for revert of the state before catchup * improve the test * Do not enqueue old pre-prepares (hyperledger#211) * ignore pp requests with ppSeqNo less than expected * minor refactor * Optimisations and a testing util (hyperledger#212) * add todo, fixing log levels, commenting out an expensive area of code and adding capability to diable monitor in tests * an optimisation avoiding re-calculation of merkle proof * remove commented code and add test * update names * add test fils for validators * add test for commit message * add test for ConsistencyProof message * update test for Commit message * add test for InstanceChange message * add test for LedgerStatus message * add test for nomination message * add test for Ordered message * add tests for Prepare and PrePrepare messages * add test for Primary message * add test for PROPAGATE message * add test for Reelection message * remove unused imports * Election bug fix (hyperledger#215) * Election bug fix if `f` nodes do not send Primary and the node is the last one to send Primary, then it was not able to decide the primary, since a node was doing "primary setting" only on receiving a Primary message, changed that to try "setting primary" after sending Primary too * update test with more checks * removed obsolete test testStopScriptIfNodeIsRunning (hyperledger#220) * Fix processing Propagate msg, INDY-245 (hyperledger#221) * Bugfix/non human readable error (hyperledger#222) * handling case with odd-length verkey * updated exception message, check it in test * Stable release Signed-off-by: Devin Fisher <[email protected]> Sign-off-executed-by: toktar <[email protected]> Approved-at: h-master

improve the check of the arguments in "generate_sovrin_pool_transacti…

1097122

…ons"

aigoncharov approved these changes May 12, 2017

View reviewed changes

aigoncharov merged commit 549e00d into hyperledger:master May 12, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve the check of the arguments in "generate_sovrin_pool_transactions" #162

improve the check of the arguments in "generate_sovrin_pool_transactions" #162

AlexanderShekhovcov commented May 11, 2017 •

edited

Loading

improve the check of the arguments in "generate_sovrin_pool_transactions" #162

improve the check of the arguments in "generate_sovrin_pool_transactions" #162

Conversation

AlexanderShekhovcov commented May 11, 2017 • edited Loading

AlexanderShekhovcov commented May 11, 2017 •

edited

Loading