Skip to content

Commit

Permalink
Stable release (#156)
Browse files Browse the repository at this point in the history
* proposed abstraction for network interface

* initial commit for zstack

* test in progress

* fixing problem with verify key location

* change in remote

* test passing in promiscous mode

* adding linger time but it does not help

* using disconnect on remote and changing from old style to new style coroutines

* completing test

* comments

* removed sleeps

* refactoring tests and KITZStack

* making basic node tests work

* fixing tests in test_node_connection.py

* changes for pool transactions

* more changes

* fixing more tests

* fixing more tests

* adding whitelist message

* adding whitelist message for node catchup tests

* added ELEC tag to logs for troubleshooting

* fixing a bug where a node with a primary was nominating itself during re-elections

* in the middle

* patching non-test node and client objects to make script tests work

* added dummy placeholders to allow tests to run until abstraction of networking is complete

* all tests except one work

* changes in scripts

* changes for endpoint stack

* commenting a log message which makes some tests run slow

* fixes in scripts

* not printing some logs on CLI

* creating node params file

* digest recalculation preventaion

* checking for port in use for both tcp and udp

* ensured new log entries didn't show up in CLI

* printing exception with the log

* beginnig to abolish key sharing

* changes for key sharing

* fixing a bug with req reply store and some tests

* log messages

* fixing logs

* removing accidentally committed code

* add pattern for hidden files to gitignore

* adding disconnect detection

* fixing bug for 3 phase messages prior to stable checkpoint

* 'gc'ing requests only when checkpoint achived on protocol instances

* fixing bug

* resolving some race conditions around checkpoints

* add option for supresing stdout logging

* removing linger and handling exception while sending message through listener

* closing auth, stopping prodables when logging error results in BlowUp and upgrating tests

* Consistency proof request should specify the target seqNo

* skipping a test and raising timeout

* skipping test

* skipping test

* skipping test

* skipping test

* initial refactoring to move secure transport abstraction into a different project

* minor changes

* moved some modules to stp

* moving node and client stacks

* fixed some improper imports

* more changes

* fixed keystroke error

* not calling sorted container's method from a newer api, resolving a bug in catchup when transactions are requested after timeout

* [SOV-768] Fixed port not available exception getting ignored and tests failing on windows and linux based system

* Removed unused import

* flexible timeout and catching exception

* Fixed error for port not available exception on windows

* Removed extra lines at the end of exceptions file

* Moved common logic related to transport to stp

* using new names from stp

* changes for stp

* removing redundant close in orientdb socket

* adding tests for catchup scenario and 3 phase message outside water mark

* fixing import in test

* Fix 'infinit' looping on receiving messages when their amount is very high (#97)

* fix 'infinit' looping on receiving messages when their amount is very high

* move quotas to config

* make zstack get params from config only once

* add test for quota

* increase message quotas to 100

* Feature Added: Jenkins (#99)

* Moved util methods to NetwrorkInterfaces; fixes in raet stack

* fix key generation

* Hotfix: Deps (#101)

* - Added a custom jsonpickle handler for enums to avoid the issue jsonpickle/jsonpickle#135 in jsonpickle 0.9.2 which is the canonical version for Ubuntu 16.04. (#98)

- Corrected Cli.restoreWalletByPath method.

* Merge branches 'stp' and 'zmq' of github.com:evernym/plenum into stp

* handling infinite loop problem and fixing test delay

* fixing import

* bugfix in test

* raising timeout

* merge

* change in ordered

* move connection related exceptions to stp project

* fix imports of moved exceptions

* fixed dependencies

* temporarily disabling tests on Windows

* renamed init_plenum_raet_keep to init_plenum_keys

* removing unused code and changing script mode

* Introduced failing of the pytets session in case any unexpected warnings have been made (#104)

* changes to troubleshoot testing for coro not awaited

* added support for warning checking across all tests

* fixed scope for warnfilters fixture

* - Updated the warnings filter. Made the rules for ResourceWarning: "unclosed file", ResourceWarning: "unclosed.*socket\.socket" and RuntimeWarning: "coroutine.*was never awaited" global since they are made by the garbage collector in unpredictable points.
- Sorted the rules in the warnings filter alphabetically.

* Specialized the warnings filter rule for RuntimeWarning about a coroutine was never awaited (to "msgAll" coroutine).

* Added an explicit call of the garbage collector to the tear-down phase of "warncheck" fixture in order to ensure that warnings will be generated and recorded for all the unclosed resources.

* Fixed the issue with that a coroutine "msgAll" object was never awaited. Removed the corresponding "ignore" rule from the warnings filter.

* - Removed the rules not being actuated currently from the warnings filter.
- Replaced use of line numbers with use of message patterns in the warnings filter rules.

* Corrected the message pattern for a rule from the warnings filter.

* Added an "ignore" rule for ResourceWarning: "unclosed event loop" to the warnings filter.

* Returned back the warnings filter rules for DeprecationWarnings in jsonpickle package. Replaced use of line numbers with use of message patterns in these rules.

* adding warning

* removing looper and checking while closing orientdb

* increasing a timeout

* replacing warn with warning methods

* fixed stp dependency

* fixed tests

* skip failing test, created SOV-881.

* changing logs to print alias

* setting resource limit for tests

* handling exception while raising resource limits

* moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring (#106)

* moved wallet persistence and restoration logic to helper methods so that it can be re-utilized from different places, some other minor refactoring

* removed unwanted error code, refactored list keyrings command so that it correctly works for agent cli

* removed unused imports

* making CLI accept hex seed of size 64

* move logging to stp repo

* mitigating key error in pool manager

* changing timeouts so the diconnect check happens twice

* handling unavailabily of resource module

* adding timeout to test utility method

* Versioning fix (#113)

* [Closes SOV-905] Big Fixed: Orientdb is not installing with the latest build 0.3.16

* moving error codes to stp

* upgrading version of stp

* Stp (#116)

* use common stp interface for changing of node's HA

* get rid of explicitly calling zmq and raet; move all transport stuff into stp interfaces.

* fix tests; increment stp version

* add AuthMode enum instead of auto parameter

* fixed testConnectWithoutKeySharingFails test

* increased test timeout

* Plenum test runner fix (#115)

* Fixed a bug in plenum test runner with an incorrect test results summary pattern. Previously it had been expected that the line with the test results summary must begin with equality signs and end with them. But this line is padded with equality signs only if it is shorter than 80 characters. Now the test results summary patterns don't require equality signs. Instead they may match only the last line in the file.

* Corrected test results summary patterns in plenum test runner.

* fix import of OP_FIELD_NAME

* up version of stp to 0.1.24

* Agent issuer wallet restoration (#117)

* supplied keys parameter as True to wallet encoding and decoding functions so that dictionary keys which are custom objects restore properly, otherwise it used to be restored as string oppossed to actual object it was before persistence

* minor changes in list keyrings to show issuer wallet for agent as well

* minor change to print base wallet first and then any other related wallets

* up stp version to 0.1.26

* skip tests in test_new_node_catchup.py

* scheduling primary selection

* Skip failing tests (#122)

* skeip test testChangeNodeHaForNonPrimary due to SOV-941

* skip test testProtocolInstanceCannotBecomeActiveWithLessThanFourServers due to SOV-940

* parametrize test timeouts (#120)

* rename expectedWait to expectedNodeInterconnectionTime

* add signature for other time expect functions

* use named timeouts in conftest

* move totalConnections from waits to utils

* use named timeout in ensureElectionsDone

* use float instead of int for seconds

* use default args of ensureElectionsDone where it is possible

* use named argument for timeouts

* use named timeout in helper

* use names for parameters

* rename 'timeoutPerReq' of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies to 'customTimeoutPerRequest' to emphasize the fact that there is a default one

* use default timeout instead of custom timeout of checkSufficientRepliesForRequests and sendReqsToNodesAndVerifySuffReplies where it can be used; use named argument for timeouts

* add comments for two functions with similar names

* rename checkSufficientRepliesRecvd to checkSufficientRepliesReceived

* rename checkSufficientRepliesForRequests to waitForSufficientRepliesForRequests

* add 'returns' to docstrings

* fix passing of args to waits.expectedElectionTimeout

* use waitForSufficientRepliesForRequests and it's default timeout instead of checkSufficientRepliesReceived everywhere it is possible

* update doc lines

* create waitForViewChange and use it in ensureView

* replace usages of checkViewNoForNodes with eventually by waitForViewChange

* add waits.expectedNodeToNodeMessageDeliveryTime and use it in sendMsgAndCheck

* rename checkMsg to checkMessageReceived

* rename sendMsgAndCheck to sendMessageAndCheckDelivery

* add docstring to sendMessageAndCheckDelivery

* remove unused helper function

* add expectedGetReadyTimeout and use it in checkPoolReady

* rename overrideTimeout parameter to customTimeout in checkNodesConnected

* use default timeout of checkNodesConnected

* create expectedPoolLedgerCheck and expectedClientConnectionTimeout waits and use them

* add todo for ensureDisconnectedToNodes

* update waits.expectedPoolLedgerCheck

* add todo for checkNodesParticipating

* add requestIds parameter for waitForSufficientRepliesForRequests

* update docstring of waitForSufficientRepliesForRequests

* fix waitForSufficientRepliesForRequests

* remove unused imports from test_log_filtering

* use named timeout in test_status_command

* use waits.expectedTransactionExecutionTime in testTransactions

* refactor testTransactions

* use waitRequestSuccess and waitBalanceChange in test_cli_with_auction_plugin, move them to test.cli.helper

* use named timeout in test_basic_client_commands

* use named timeout in helper.checkRequest

* create waitClientConnected and use it instead of checkClientConnected with eventually

* add docstrings

* rename checkNodeStarted to waitNodeStarted and use named timeout 'expectedNodeStartUpTimeout' in it

* rename expectedGetReadyTimeout to expectedPoolGetReadyTimeout

* rename checkAllNodesStarted to waitAllNodesStarted

* fix default value of customTimeout of checkPoolReady

* create waitAllNodesUp and use it instead of checkAllNodesUp with eventually

* create waitReplyCount and use instead of checkReplyCount and eventually

* use named timeouts in test_client

* use named timeouts in some more tests

* add basic implementation for waits.expectedCatchupTime and use it

* make expectedCatchupTime get custom ConsistencyProofsTimeout as parameter

* use named timeout in testNodeDiscardMessageFromUnknownView

* use waits.expectedElectionTimeout and rename timeout arg to custonTimeout in checkEveryNodeHasAtMostOnePrimary

* rename timeout argument of plenum/test/node_catchup/test_discard_view_no.py to customTimeout and make it used named timeouts from waits as default

* update timeouts in testNodeDiscardMessageFromUnknownView

* create waits.expectedRequestStashingTime and use it

* add TODO to test_catchup_delayed_nodes

* create waitNodeLedgersEquality and use it instead of directo usage of checkNodeLedgersEquality

* use waits.expectedPoolLedgerCheck in waitNodeLedgersEquality

* use named timeout in testOrderingCase2

* add waits.expectedNominationTimeout and use it

* use named timeout in some more tests

* add missing empty lines

* update waits

* add 'TODO[slow-factor]'

* update timeouts in the tests

* fix testTestNodeDelay and missed import

* skip testMultipleRequests test

* skip testStatusAfterClientAdded test

* fix testInstChangeWithLowerRatioThanDelta

* fix test_new_node_catchup.py

* fix testChangeHaPersistsPostNodesRestart

* fix testAdd2NewNodes

* increase expectedElectionTimeout timeout

* rename logger.warn => logger.warning

* tune timeouts in the tests

* make sections in waits.py

* add --repat for runner.py

* increase expectedCatchupTime timeout

* improve runner.py

* tune the test timeouts

* skip some catchup tests

* parametrize test timeouts

* rm eventually.py

* fix testQueueingReqFromFutureView, testNumOfPrePrepareWithFPlusOneFaults, testNumOfPrepareWithFPlusOneFaults

* fix testProtocolInstanceCannotBecomeActiveWithLessThanFourServers

* tune propagate and preprepare test timeouts

* skip testNumOf*WithFPlusOneFaults

* fixed plenum for the latest stp (#127)

* fixed plenum for the latest stp

* increment stp version

* archive runner.py output for all agents (#107)

* archive runner.py results

* using env variable NODE_NAME instaed of func param for artifacts

* configured archiveArtifacts to allow empty/no archive

* Do view change if a primary is disconnected (#128)

* check whether function is a partial function in getCallableName

* add tests for view change when primary goes down

* start view change if primary went offline

* use startViewChangeIfPrimaryWentOffline instead of doElectionIfNeeded

* Unclosed file warnings (#124)

* Removed "ignore" rule for ResourceWarning about an unclosed file from the warnings filter.

* Fixed some causes of ResourceWarnings about unclosed files.

* - Fixed some causes of ResourceWarnings about unclosed files.
- Corrected plenum.common.txn_util.updateGenesisPoolTxnFile function.

* - Fixed the rest causes of ResourceWarnings about unclosed files.
- Removed TimeAndSizeRotatingFileHandler class which is not used anymore (the class with the same name from stp is now used instead).

* Updated stp-dev dependency to the new version.

* Reverted update of stp-dev dependency.

* Skipped the tests in plenum.test.test_log_rotation module since they require stp-dev dependency of a newer version (0.1.28 or higher).

* Agent generalization (#126)

* refactored test_network_setup; added some role-base helper classes for members of the network

* fixed issue with wrong import and minor refactoring

* more work on tests

* bugfixes

* interim changes during refactoring

* adding init_utils

* cleaning up before keys

* removed type hint causing issues; this issue is fixed in Python 3.5.3 with a new typing.Coroutine

* added a check in DidVerifier to check that we have a verkey

* fixed unreliable tmpdir test fixture

* modifications to tests to make more robust when running from command line

* changed spyable to be able to monkey patch a single method

* Added a run_till_quiet, waits for quiet prods

This function will wait till there is a quiet period from the
prodables. Dependent on prodables correctly reporting events.
Will run forever if there is always activity.

* small tweek to log format to keep | alined

* changes to fix some issues happening in end-to-end test

* using same seed for creating keys

* ignore log files

* added a small test for verifier

* removed some excpetion class that were moved to STP

* init Local keys for client in test network

* updated to use new API from shared lib

* Enabled the tests in plenum.test.test_log_rotation module. (#135)

* check for close method before calling it (#136)

* fix testNumOf*WithFPlusOneFaults tests

* Updated stp dep

* added missed __init__.py for common/member

* add missing parameter to bootstrap test network (#143)

* [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux

* add missing parameter to bootstrap test network

* add missing parameter to bootstrap test network

* [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux

* add missing parameter to bootstrap test network

* add missing parameter to bootstrap test network

* fix the catchup tests (#140)

* Close SOV-976 (#144)

* [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux

* add missing parameter to bootstrap test network

* add missing parameter to bootstrap test network

* [Closes SOV-947] Sovrin-node test testTrusteeCannotChangeVerkey fails on Linux

* add missing parameter to bootstrap test network

* add missing parameter to bootstrap test network

* [Closes SOV-976] Unable to create the genesis transaction files

* up stp to 0.1.42 (#148)

* [Closes SOV-981] Bug Fixed: Unable to create pool transaction file on the client machine without passing it a node number (#149)

* fix generation of trustee txns

* fix generation of trustee txns (#151)

* fix
  • Loading branch information
Andrei Goncharov authored May 3, 2017
1 parent 993c90c commit 664bc55
Show file tree
Hide file tree
Showing 19 changed files with 282 additions and 167 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -68,3 +68,6 @@ docs/source/api_docs/

# hidden files
.*

# log files
*.log
20 changes: 4 additions & 16 deletions Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,10 @@ def testUbuntu = {

testEnv.inside('--network host') {
echo 'Ubuntu Test: Install dependencies'
testHelpers.installDeps()
testHelpers.install()

echo 'Ubuntu Test: Test'
def resFile = "test-result.${NODE_NAME}.txt"
try {
sh "python runner.py --pytest \"python -m pytest\" --output \"$resFile\""
}
finally {
archiveArtifacts allowEmptyArchive: true, artifacts: "$resFile"
}
testHelpers.testRunner(resFile: "test-result.${NODE_NAME}.txt")
}
}
finally {
Expand Down Expand Up @@ -71,16 +65,10 @@ def testWindowsNoDocker = {

testHelpers.createVirtualEnvAndExecute({ python, pip ->
echo 'Windows No Docker Test: Install dependencies'
testHelpers.installDepsBat(python, pip)
testHelpers.install(python: python, pip: pip, isVEnv: true)

echo 'Windows No Docker Test: Test'
def resFile = "test-result.${NODE_NAME}.txt"
try {
bat "${python} runner.py --pytest \"${python} -m pytest\" --output \"$resFile\""
}
finally {
archiveArtifacts allowEmptyArchive: true, artifacts: "$resFile"
}
testHelpers.testRunner(resFile: "test-result.${NODE_NAME}.txt", python: python)
})
}
finally {
Expand Down
3 changes: 2 additions & 1 deletion plenum/client/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ def __init__(self,
# TODO: Have a way for a client to have a user friendly name. Does it
# matter now, it used to matter in some CLI exampples in the past.
# self.name = name
self.name = self.stackName
self.name = self.stackName or 'Client~' + str(id(self))

cha = None
# If client information already exists is RAET then use that
Expand Down Expand Up @@ -127,6 +127,7 @@ def __init__(self,
self.created = time.perf_counter()

# noinspection PyCallingNonCallable
# TODO I think this is a bug here, sighex is getting passed in the seed parameter
self.nodestack = self.nodeStackClass(stackargs,
self.handleOneNodeMsg,
self.nodeReg,
Expand Down
37 changes: 37 additions & 0 deletions plenum/common/exceptions.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
from plenum.server.suspicion_codes import Suspicion
from re import compile

from plenum.server.suspicion_codes import Suspicion
Expand All @@ -9,6 +10,23 @@ def __init__(self, identifier=None, reqId=None):
self.reqId = reqId


class NodeError(Exception):
pass


class PortNotAvailableForNodeWebServer(NodeError):
pass


class RemoteError(NodeError):
def __init__(self, remote):
self.remote = remote


class RemoteNotFound(RemoteError):
pass


class BaseExc(Exception):
# def __init__(self, code: int=None, reason: str=None):
# self.code = code
Expand Down Expand Up @@ -212,6 +230,25 @@ class OrientDBNotRunning(GraphStorageNotAvailable):
pass


class WalletError(Exception):
pass


class WalletNotSet(WalletError):
pass


class WalletNotInitialized(WalletError):
pass


class PortNotAvailable(OSError):
def __init__(self, port):
self.port = port
super().__init__("port not available: {}".format(port))


class OperationError(Exception):
def __init__(self, error):
super().__init__("error occurred during operation: {}".format(error))

12 changes: 8 additions & 4 deletions plenum/common/keygen_utils.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import os

from plenum.common.stacks import nodeStackClass
from stp_core.crypto.util import randomSeed
from stp_zmq.util import createCertsFromKeys

from plenum.common.constants import CLIENT_STACK_SUFFIX
Expand All @@ -19,12 +20,15 @@ def initRemoteKeys(name, baseDir, sigseed, verkey, override=False, config=None):
override=override)



def initNodeKeysForBothStacks(name, baseDir, sigseed, override=False, config=None):
nodeStackClass.initLocalKeys(name, baseDir, sigseed, override=override)
nodeStackClass.initLocalKeys(name + CLIENT_STACK_SUFFIX, baseDir, sigseed,
override=override)
# `sigseed` is initailsed to keep the seed same for both stacks.
# Both node and client stacks need to have same keys
sigseed = sigseed or randomSeed()

nodeStackClass.initLocalKeys(name + CLIENT_STACK_SUFFIX, baseDir, sigseed,
override=override)
return nodeStackClass.initLocalKeys(name, baseDir, sigseed,
override=override)


def areKeysSetup(name, baseDir, config=None):
Expand Down
Loading

0 comments on commit 664bc55

Please sign in to comment.