endpoint: force wallet resync #1268

KtorZ · 2020-01-15T12:40:52Z

Issue Number

Overview

92c2aa0
extend swagger specification with 'forceResync'
As an extreme measure to force re-sync the wallet
206affc
extend API with 'forceResync' operation
- Added a 'forceResync' link
Extended the servant Client
Provide an initial straightforward implementation re-using 'rollbackBlocks'
3d9bc77
implement 'forceResync' handler properly
Taking care of concurrent workers possibly acting on the database
at the same time as well as restarting the restoration worker at
the right point
4b06845
add integration test for the 'forceResync' endpoint
Doesn't pass currently because of the in-memory sqlite implementation.
It works like a charm with the file database though:

(Right before making the request, observe the thread id #142)
[cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node.

(Upon receiving the request)
[cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos

(Turning off the worker, rolling back, and switching the worker back on)
[cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over.
[cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0
[cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0

(Responding to the request)
[cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s

(And later, the worker restarted, applying blocks from the start, new thread id #528).
[cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1]
[cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1]
[cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2]
[cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3]

[...]

(Eventually, reaches the tip)
[cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating
[cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored
[cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s)
[cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100]

d196fd1
preserve in-memory database states between calls
For the in-memory Sqlite database, we do actually preserve the database
after the 'action' is done. This allows for calling 'withDatabase'
several times within the same execution and get back the same database.
The memory is only cleaned up when calling 'removeDatabase', to mimic
the way the file database works!

Without this, code like:

  Registry.remove re wid
  registerWorker ctx wid

would lead to weird behavior where the wallet wid which existed when
the worker was stopped didn't exist anymore when register the worker
again. This was because, killing the worker closes the connection
to the database and re-opening the "same" connection on the in-memory
database actually yield a completely different memory representation!

423296b
revise 'expectResponseCode' to give more details on failure
It was kinda hard to understand what was going on from the error message.
This is now slightly better and at least, allow some reasonning
5eae1fa
re-generate nix machinery
fab6e17
fix database closing error handling
- Added some additional log messages
Added an extra exception handler
Swapped worker removal with database closing
Reviewed how the in-memory database got cleaned up to avoid race condition
9babcc2
use on-disk database in integration tests

Comments

lib/core-integration/src/Test/Integration/Scenario/API/Wallets.hs

paweljakubas · 2020-01-15T13:01:59Z

lib/core-integration/src/Test/Integration/Scenario/API/Wallets.hs

+    -- 3. The wallet eventually re-sync
+    eventually $ do
+        v <- request @wallet ctx (Link.getWallet @style w) Default Empty
+        verify v [ expectFieldSatisfy @IO #state (== (ApiT Ready)) ]


we are fine here to check only ApiWallet's state field, right? If there is no point 2 we are obviously satisfying point 3. It would be nice (although I do not know if it is easy) to show in point 2 upon forcing that state is Syncing ... so something happens as a result of force resync.

It would be nice (although I do not know if it is easy) to show in point 2 upon forcing that state is Syncing ...

that would be great. +1

Also I think that it would be nice to assert for few other stuff (maybe as separate scenarios):

balance (total, available, rewards?)

delegation status (if it the wallet was delegating is should be delegating still)

addresses (if it makes sense ?)

transaction history (whatever it should be after resync)

I 100% agree with you guys and I actually thought of this. The main problem I have here is .... race-conditions.

Technically speaking, right after force-resyncing a wallet, your tip should be at genesis. But, by the time you make the request to produce some assertion... your wallet might as well have restored itself :| ... So, it's kinda hard to come up with reliable assertions that still make it possible to assess the behavior, precisely because this endpoint is meant to be used in situations where something went wrong / wasn't supposed to happen.

So here, the scenario is much more about exercising the endpoint and making sure that, nothing wrong really happen, whereas the real confirmation that it works come from the logging output.

I understand that race conditions are the preventing the addition of the "Syncing" assertion? But checking for the balance, delegation, addresses, transaction history should be doable right? (That would also add confidence that nothing wrong happened with the wallet after resync and it is still usable... perhaps making transaction after resync could be also a test that's nice to have )

I've added a few tests but, nothing really satisfactory in terms of testing states. If you guys have ideas, I'll happily do this in another PR (merging as is).

lib/core/src/Cardano/Wallet/Api.hs

paweljakubas · 2020-01-15T13:06:57Z

lib/core/src/Cardano/Wallet/Api/Server.hs

+    df = ctx ^. dbFactory @s @k
+    -- NOTE Safe because it happens without any worker running.
+    safeRollback = do
+        let tr' = Registry.transformTrace wid tr


we will have log as coming from withLogging about request..I wonder if we do not want to say more here. although we will have also additional logs from rollbackBlocks below. But maybe removing Registry is worth logging

logs look as follows:

[cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...]

I think it's quite sufficient here 🤔

indeed, logs looks good. Worker has exited... logs removing of registry. and as expected there are logs from rollbackBlocks

paweljakubas · 2020-01-15T13:08:41Z

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

+        -- after the 'action' is done. This allows for calling 'withDatabase'
+        -- several times within the same execution and get back the same
+        -- database. The memory is only cleaned up when calling
+        -- 'removeDatabase', to mimic the way the file database works!


paweljakubas

I left some comments/questions.

What about adding the test that shows that when using force resync any incoming transaction will be erased whereas outgoing transactions will be put back as pending?

paweljakubas · 2020-01-15T13:13:51Z

lib/core/src/Cardano/Wallet/DB.hs

        -- ^ Creates a new or use an existing database, maintaining an open
        -- connection so long as necessary

    , removeDatabase :: WalletId -> IO ()
        -- ^ Erase any trace of the database
-    } deriving (Generic)
+    }


{-# LANGUAGE DeriveGeneric #-}

not needed after this change

piotr-iohk · 2020-01-15T13:26:03Z

lib/core-integration/src/Test/Integration/Scenario/API/Wallets.hs

+    -- 3. The wallet eventually re-sync
+    eventually $ do
+        v <- request @wallet ctx (Link.getWallet @style w) Default Empty
+        verify v [ expectFieldSatisfy @IO #state (== (ApiT Ready)) ]


It would be nice (although I do not know if it is easy) to show in point 2 upon forcing that state is Syncing ...

that would be great. +1

Also I think that it would be nice to assert for few other stuff (maybe as separate scenarios):

balance (total, available, rewards?)

delegation status (if it the wallet was delegating is should be delegating still)

addresses (if it makes sense ?)

transaction history (whatever it should be after resync)

piotr-iohk · 2020-01-15T13:47:33Z

specifications/api/swagger.yaml

+
+        Force the wallet to rewind back to the genesis block and sync again with the chain.
+        Any incoming transaction will be erased whereas outgoing transactions will be put back
+        as pending. This action should only be taken as an extreme measure to resolve potential


I don't understand this part. Is it for all the transactions, even if they are in ledger?

Why I'm asking: when you delete a wallet and create it again you have your transaction history available... My understanding and expectation of the forceResync would be to be an equivalent of delete and create (but without the need of deleting and creating a wallet again), am I wrong? (I bet I am...)

@piotr-iohk force-resync is not equivalent to deleting and re-creating the wallet. It's actually equivalent to (and is a) rollback to the genesis block. So, after this, some transaction will be put back in pending such that, users can still track them.
The annoying part with deleting and re-creating your wallet is also that you completely loose track of your wallet history. If you attempted to made a payment, it won't show up unless visible in the ledger. But you may still have transaction floating around in mempools that don't show up in your wallet. Whereas, by using force resync here, you'll still see them.

Any incoming transaction will be erased whereas outgoing transactions will be put back as pending.

I think this statement may be somewhat misleading. After reading it I thought that after doing force resync all my incoming txs will be erased and not visible at all and outgoing will become pending. And it will be a finite state. But if they are in the ledger they will be discovered again and visible again. Hence the first comment, cause it was a bit surprising to me.

KtorZ · 2020-01-15T15:52:34Z

bors r+

1268: endpoint: force wallet resync r=KtorZ a=KtorZ # Issue Number  #1146 # Overview  - 06ee934 extend swagger specification with 'forceResync' As an extreme measure to force re-sync the wallet - 0938999 extend API with 'forceResync' operation - Added a 'forceResync' link - Extended the servant Client - Provide an initial straightforward implementation re-using 'rollbackBlocks' - 8f2b197 implement 'forceResync' handler properly Taking care of concurrent workers possibly acting on the database at the same time as well as restarting the restoration worker at the right point - 7d3e281 add integration test for the 'forceResync' endpoint Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ``` - e134d52 preserve in-memory database states between calls For the in-memory Sqlite database, we do actually preserve the database after the 'action' is done. This allows for calling 'withDatabase' several times within the same execution and get back the same database. The memory is only cleaned up when calling 'removeDatabase', to mimic the way the file database works! Without this, code like: ```hs Registry.remove re wid registerWorker ctx wid ``` would lead to weird behavior where the wallet `wid` which existed when the worker was stopped didn't exist anymore when register the worker again. This was because, killing the worker closes the connection to the database and re-opening the "same" connection on the in-memory database actually yield a completely different memory representation! - 1577706 revise 'expectResponseCode' to give more details on failure It was kinda hard to understand what was going on from the error message. This is now slightly better and at least, allow some reasonning # Comments  ![Screenshot from 2020-01-15 16-50-37](https://user-images.githubusercontent.com/5680256/72448387-3373ee80-37b7-11ea-962d-60c507ea4617.png)  Co-authored-by: KtorZ <[email protected]>

iohk-bors · 2020-01-15T15:56:19Z

Build failed

buildkite/cardano-wallet

As an extreme measure to force re-sync the wallet

- Added a 'forceResync' link - Extended the servant Client - Provide an initial straightforward implementation re-using 'rollbackBlocks'

Taking care of concurrent workers possibly acting on the database at the same time as well as restarting the restoration worker at the right point

Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ```

For the in-memory Sqlite database, we do actually preserve the database after the 'action' is done. This allows for calling 'withDatabase' several times within the same execution and get back the same database. The memory is only cleaned up when calling 'removeDatabase', to mimic the way the file database works! Without this, code like: ```hs Registry.remove re wid registerWorker ctx wid ``` would lead to weird behavior where the wallet `wid` which existed when the worker was stopped didn't exist anymore when register the worker again. This was because, killing the worker closes the connection to the database and re-opening the "same" connection on the in-memory database actually yield a completely different memory representation!

It was kinda hard to understand what was going on from the error message. This is now slightly better and at least, allow some reasonning

KtorZ · 2020-01-15T17:14:10Z

bors r+

1268: endpoint: force wallet resync r=KtorZ a=KtorZ # Issue Number  #1146 # Overview  - 06ee934 extend swagger specification with 'forceResync' As an extreme measure to force re-sync the wallet - 0938999 extend API with 'forceResync' operation - Added a 'forceResync' link - Extended the servant Client - Provide an initial straightforward implementation re-using 'rollbackBlocks' - 8f2b197 implement 'forceResync' handler properly Taking care of concurrent workers possibly acting on the database at the same time as well as restarting the restoration worker at the right point - 7d3e281 add integration test for the 'forceResync' endpoint Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ``` - e134d52 preserve in-memory database states between calls For the in-memory Sqlite database, we do actually preserve the database after the 'action' is done. This allows for calling 'withDatabase' several times within the same execution and get back the same database. The memory is only cleaned up when calling 'removeDatabase', to mimic the way the file database works! Without this, code like: ```hs Registry.remove re wid registerWorker ctx wid ``` would lead to weird behavior where the wallet `wid` which existed when the worker was stopped didn't exist anymore when register the worker again. This was because, killing the worker closes the connection to the database and re-opening the "same" connection on the in-memory database actually yield a completely different memory representation! - 1577706 revise 'expectResponseCode' to give more details on failure It was kinda hard to understand what was going on from the error message. This is now slightly better and at least, allow some reasonning # Comments  ![Screenshot from 2020-01-15 16-50-37](https://user-images.githubusercontent.com/5680256/72448387-3373ee80-37b7-11ea-962d-60c507ea4617.png)  Co-authored-by: KtorZ <[email protected]>

iohk-bors · 2020-01-15T17:26:56Z

Build failed

buildkite/cardano-wallet

KtorZ · 2020-01-16T15:40:06Z

bors r+

1268: endpoint: force wallet resync r=KtorZ a=KtorZ # Issue Number  #1146 # Overview  - 06ee934 extend swagger specification with 'forceResync' As an extreme measure to force re-sync the wallet - 0938999 extend API with 'forceResync' operation - Added a 'forceResync' link - Extended the servant Client - Provide an initial straightforward implementation re-using 'rollbackBlocks' - 8f2b197 implement 'forceResync' handler properly Taking care of concurrent workers possibly acting on the database at the same time as well as restarting the restoration worker at the right point - 7d3e281 add integration test for the 'forceResync' endpoint Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ``` - e134d52 preserve in-memory database states between calls For the in-memory Sqlite database, we do actually preserve the database after the 'action' is done. This allows for calling 'withDatabase' several times within the same execution and get back the same database. The memory is only cleaned up when calling 'removeDatabase', to mimic the way the file database works! Without this, code like: ```hs Registry.remove re wid registerWorker ctx wid ``` would lead to weird behavior where the wallet `wid` which existed when the worker was stopped didn't exist anymore when register the worker again. This was because, killing the worker closes the connection to the database and re-opening the "same" connection on the in-memory database actually yield a completely different memory representation! - 1577706 revise 'expectResponseCode' to give more details on failure It was kinda hard to understand what was going on from the error message. This is now slightly better and at least, allow some reasonning # Comments  ![Screenshot from 2020-01-15 16-50-37](https://user-images.githubusercontent.com/5680256/72448387-3373ee80-37b7-11ea-962d-60c507ea4617.png)  Co-authored-by: KtorZ <[email protected]>

- Added some additional log messages - Added an extra exception handler - Swapped worker removal with database closing - Reviewed how the in-memory database got cleaned up to avoid race condition

iohk-bors · 2020-01-16T15:50:31Z

Canceled

KtorZ · 2020-01-16T15:51:25Z

bors r+

1268: endpoint: force wallet resync r=KtorZ a=KtorZ # Issue Number  #1146 # Overview  - 92c2aa0 extend swagger specification with 'forceResync' As an extreme measure to force re-sync the wallet - 206affc extend API with 'forceResync' operation - Added a 'forceResync' link - Extended the servant Client - Provide an initial straightforward implementation re-using 'rollbackBlocks' - 3d9bc77 implement 'forceResync' handler properly Taking care of concurrent workers possibly acting on the database at the same time as well as restarting the restoration worker at the right point - 4b06845 add integration test for the 'forceResync' endpoint Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ``` - d196fd1 preserve in-memory database states between calls For the in-memory Sqlite database, we do actually preserve the database after the 'action' is done. This allows for calling 'withDatabase' several times within the same execution and get back the same database. The memory is only cleaned up when calling 'removeDatabase', to mimic the way the file database works! Without this, code like: ```hs Registry.remove re wid registerWorker ctx wid ``` would lead to weird behavior where the wallet `wid` which existed when the worker was stopped didn't exist anymore when register the worker again. This was because, killing the worker closes the connection to the database and re-opening the "same" connection on the in-memory database actually yield a completely different memory representation! - 423296b revise 'expectResponseCode' to give more details on failure It was kinda hard to understand what was going on from the error message. This is now slightly better and at least, allow some reasonning - 5eae1fa re-generate nix machinery - fab6e17 fix database closing error handling - Added some additional log messages - Added an extra exception handler - Swapped worker removal with database closing - Reviewed how the in-memory database got cleaned up to avoid race condition - 9babcc2 use on-disk database in integration tests # Comments  ![Screenshot from 2020-01-15 16-50-37](https://user-images.githubusercontent.com/5680256/72448387-3373ee80-37b7-11ea-962d-60c507ea4617.png)  Co-authored-by: KtorZ <[email protected]>

iohk-bors · 2020-01-16T16:12:08Z

Build failed

buildkite/cardano-wallet

KtorZ · 2020-01-16T16:20:12Z

bors retry

1268: endpoint: force wallet resync r=KtorZ a=KtorZ # Issue Number  #1146 # Overview  - 92c2aa0 extend swagger specification with 'forceResync' As an extreme measure to force re-sync the wallet - 206affc extend API with 'forceResync' operation - Added a 'forceResync' link - Extended the servant Client - Provide an initial straightforward implementation re-using 'rollbackBlocks' - 3d9bc77 implement 'forceResync' handler properly Taking care of concurrent workers possibly acting on the database at the same time as well as restarting the restoration worker at the right point - 4b06845 add integration test for the 'forceResync' endpoint Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ``` - d196fd1 preserve in-memory database states between calls For the in-memory Sqlite database, we do actually preserve the database after the 'action' is done. This allows for calling 'withDatabase' several times within the same execution and get back the same database. The memory is only cleaned up when calling 'removeDatabase', to mimic the way the file database works! Without this, code like: ```hs Registry.remove re wid registerWorker ctx wid ``` would lead to weird behavior where the wallet `wid` which existed when the worker was stopped didn't exist anymore when register the worker again. This was because, killing the worker closes the connection to the database and re-opening the "same" connection on the in-memory database actually yield a completely different memory representation! - 423296b revise 'expectResponseCode' to give more details on failure It was kinda hard to understand what was going on from the error message. This is now slightly better and at least, allow some reasonning - 5eae1fa re-generate nix machinery - fab6e17 fix database closing error handling - Added some additional log messages - Added an extra exception handler - Swapped worker removal with database closing - Reviewed how the in-memory database got cleaned up to avoid race condition - 9babcc2 use on-disk database in integration tests # Comments  ![Screenshot from 2020-01-15 16-50-37](https://user-images.githubusercontent.com/5680256/72448387-3373ee80-37b7-11ea-962d-60c507ea4617.png)  Co-authored-by: KtorZ <[email protected]>

KtorZ · 2020-01-16T17:18:40Z

CI is stuck with "Check Cabal Reconfigure" for I am not sure what the reason is but I don't think it's remotely related to the PR, so merging all-the-same :|

iohk-bors · 2020-01-16T17:23:43Z

Timed out

KtorZ requested review from paweljakubas and piotr-iohk January 15, 2020 12:40

KtorZ self-assigned this Jan 15, 2020

KtorZ changed the title ~~Ktor z/force resync~~ endpoint: force wallet resync Jan 15, 2020

KtorZ commented Jan 15, 2020

View reviewed changes

lib/core-integration/src/Test/Integration/Scenario/API/Wallets.hs Outdated Show resolved Hide resolved

paweljakubas reviewed Jan 15, 2020

View reviewed changes

lib/core/src/Cardano/Wallet/Api.hs Outdated Show resolved Hide resolved

paweljakubas reviewed Jan 15, 2020

View reviewed changes

lib/core/src/Cardano/Wallet/Api.hs Outdated Show resolved Hide resolved

paweljakubas reviewed Jan 15, 2020

View reviewed changes

paweljakubas approved these changes Jan 15, 2020

View reviewed changes

paweljakubas reviewed Jan 15, 2020

View reviewed changes

piotr-iohk reviewed Jan 15, 2020

View reviewed changes

KtorZ force-pushed the KtorZ/force-resync branch 2 times, most recently from 0ca057d to 1577706 Compare January 15, 2020 15:49

KtorZ added 7 commits January 15, 2020 18:04

extend swagger specification with 'forceResync'

92c2aa0

As an extreme measure to force re-sync the wallet

extend API with 'forceResync' operation

206affc

- Added a 'forceResync' link - Extended the servant Client - Provide an initial straightforward implementation re-using 'rollbackBlocks'

implement 'forceResync' handler properly

3d9bc77

Taking care of concurrent workers possibly acting on the database at the same time as well as restarting the restoration worker at the right point

revise 'expectResponseCode' to give more details on failure

423296b

It was kinda hard to understand what was going on from the error message. This is now slightly better and at least, allow some reasonning

re-generate nix machinery

5eae1fa

KtorZ force-pushed the KtorZ/force-resync branch from 1577706 to 5eae1fa Compare January 15, 2020 17:05

KtorZ added 2 commits January 16, 2020 16:50

fix database closing error handling

fab6e17

- Added some additional log messages - Added an extra exception handler - Swapped worker removal with database closing - Reviewed how the in-memory database got cleaned up to avoid race condition

use on-disk database in integration tests

9babcc2

KtorZ force-pushed the KtorZ/force-resync branch from 4ed4d42 to 9babcc2 Compare January 16, 2020 15:50

KtorZ merged commit 2681d91 into master Jan 16, 2020

KtorZ deleted the KtorZ/force-resync branch January 16, 2020 17:18

KtorZ mentioned this pull request Jan 18, 2020

Wallet backend running with file db crashes on integration tests #1271

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

endpoint: force wallet resync #1268

endpoint: force wallet resync #1268

KtorZ commented Jan 15, 2020 •

edited

Loading

paweljakubas Jan 15, 2020 •

edited

Loading

piotr-iohk Jan 15, 2020

KtorZ Jan 15, 2020

piotr-iohk Jan 15, 2020

KtorZ Jan 15, 2020

paweljakubas Jan 15, 2020 •

edited

Loading

KtorZ Jan 15, 2020

paweljakubas Jan 15, 2020

paweljakubas Jan 15, 2020

paweljakubas left a comment •

edited

Loading

paweljakubas Jan 15, 2020

piotr-iohk Jan 15, 2020

piotr-iohk Jan 15, 2020

KtorZ Jan 15, 2020

piotr-iohk Jan 15, 2020

KtorZ commented Jan 15, 2020

iohk-bors bot commented Jan 15, 2020

KtorZ commented Jan 15, 2020

iohk-bors bot commented Jan 15, 2020

KtorZ commented Jan 16, 2020

iohk-bors bot commented Jan 16, 2020

KtorZ commented Jan 16, 2020

iohk-bors bot commented Jan 16, 2020

KtorZ commented Jan 16, 2020

KtorZ commented Jan 16, 2020

iohk-bors bot commented Jan 16, 2020

endpoint: force wallet resync #1268

endpoint: force wallet resync #1268

Conversation

KtorZ commented Jan 15, 2020 • edited Loading

Issue Number

Overview

Comments

paweljakubas Jan 15, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paweljakubas Jan 15, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paweljakubas left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KtorZ commented Jan 15, 2020

iohk-bors bot commented Jan 15, 2020

Build failed

KtorZ commented Jan 15, 2020

iohk-bors bot commented Jan 15, 2020

Build failed

KtorZ commented Jan 16, 2020

iohk-bors bot commented Jan 16, 2020

Canceled

KtorZ commented Jan 16, 2020

iohk-bors bot commented Jan 16, 2020

Build failed

KtorZ commented Jan 16, 2020

KtorZ commented Jan 16, 2020

iohk-bors bot commented Jan 16, 2020

Timed out

KtorZ commented Jan 15, 2020 •

edited

Loading

paweljakubas Jan 15, 2020 •

edited

Loading

paweljakubas Jan 15, 2020 •

edited

Loading

paweljakubas left a comment •

edited

Loading