Zero balance reported on the wallet despite receiving funds from faucet

[piotr-iohk]

Context

Issue encountered by @alan-mcnicholas on his wallet connected to nightly testnet (in Daedalus)
Wallet had some funds from the faucet. After some time (after launching Daedalus again some time later) it was showing zero balance, despite there is transaction in the history.

The same wallet restored on different wallet backend shows balance correctly.

Steps to Reproduce

No steps to reproduce. Occurred once so far.

Expected behavior

Balance should be correct.

Actual behavior

Apparently the balance was shown correctly at the beginning, but after launching Daedalus again next morning the balance was 0.

Wallet db: wallets.tar.gz

Logs: Uploading Logs.tar.gz…

Daedalus:

Transaction details:

{
  "inserted_at": {
    "time": "2019-12-07T13:08:42Z",
    "block": {
      "height": {
        "quantity": 2987,
        "unit": "block"
      },
      "epoch_number": 3,
      "slot_number": 7073
    }
  },
  "status": "in_ledger",
  "amount": {
    "quantity": 10000000000,
    "unit": "lovelace"
  },
  "inputs": [
    {
      "id": "3e55108546d9525296f3c4e45e255781c3b27accee84a3e6ceea5ce972132948",
      "index": 255
    }
  ],
  "direction": "incoming",
  "outputs": [
    {
      "amount": {
        "quantity": 10000000000,
        "unit": "lovelace"
      },
      "address": "addr1ssgjwyddpxx6wq9msl4vxtzy8mfgmy3s53mx3w2ztrwjw3tsmtvncqrcgpy3p0tv8jsy5sghlpz787cxyrqerps8p76887d4rt2rsww8lrk3gz"
    }
  ],
  "depth": {
    "quantity": 8245,
    "unit": "block"
  },
  "id": "20425e2e42f01e0e6aed9c00cf6880776e95b5afc9c06224b9c7e7a04ec1a42d"
}

---

Resolution

• Make sure that the wallet and its network layer are still in sync when the wallet has rolled back ==> Revisions of the logic handling roll backs  #1246

---

QA

[alan-mcnicholas]

Uploading logs + wallet db from 0 balance instance.
0balanceIssue.zip

[KtorZ]

Okay. I think I found the source of the issue while sleeping 🙃

1. We have a piece of code in the networking layer that we call "the chain follower" and that takes care of following a particular node's chain. A client can ask for next blocks to the follower which yields one of the three responses:

   1. Roll Forward (when new blocks are found)
   2. Roll Backward (when the node has switched to a different chain)
   3. AwaitReply (when there's no new blocks yet).
      Upon receiving a "roll backward" (with an associated point of rollback), the wallet will "roll back" to that point and continue following the chain from that point.

2. The database does not keep track of every blocks it sees for this would be too expensive in terms of storage (and would rapidly impact the overall performance of a wallet). Therefore, the wallet database engine:

   1. Makes sparse checkpoints at regular intervals, more dense near the tip.
   2. Prunes old checkpoints as it move forwards.
      The total number of checkpoints highly depends on the k / epoch_stability parameter from the protocol, known as the max length of a chain fork (i.e., the maximum number of blocks that we could potentially roll back). This value is set to 10 for the testnet. There are two important properties with this engine:
   3. No matter what, it always keep the most recent checkpoint and the genesis checkpoint. As a consequence, if we receives blocks by batches of 10, we'll make one checkpoint every 10 blocks, though pruning may discard old blocks quite quickly.
   4. When rolling back, the database will rollback to the most recent checkpoint that is lower or equal to the point of rollback.

3. The protocol has an active slot coefficient, which determines the density of blocks over time. For the incentivized testnet, this is set to 10%, meaning that 90% of the slots are empty, on average. Combined with the epoch stability, it means that, for every 10 slots, there's also a non null likelihood to have not a single block (and therefore, have 10 slots with the same "block height").

Now, the first two pieces are working well (as per testing) separately but, put together, we may run into a non-obvious issue that is made more likely because of 3, especially when catching up with a network (where the density of checkpoints is small).

The scenario would be as follow:

1. The wallet starts syncing and initialize the chain follower to a certain point (based on known checkpoints).
2. The chain follower starts consuming blocks,and rapidly, switches to a new chain that is not far off.
3. The database is asked to rollback to a close point for which there's no checkpoint. So it rolls back to a point further in the past (possibly, the genesis block).
4. The chain follower continues yielding blocks, as-if the wallet engine was now at the point of rollback. Yet, there's now a discrepancy between where the wallet is vs where the chain follower thinks it it.

As a mitigation, it seems that we would need some sort of acknowledgement after a RollBackward command from the chain follower to confirm to which point the wallet has actually rolled backward. Or simply, recompute an intersection after each roll back.

[KtorZ]

Closing this one in hope. Will re-open if the problem arises again.