Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework output_pos index (better txhashset transactional support) #3235

Closed
antiochp opened this issue Feb 20, 2020 · 0 comments · Fixed by #3236
Closed

Rework output_pos index (better txhashset transactional support) #3235

antiochp opened this issue Feb 20, 2020 · 0 comments · Fixed by #3236
Assignees
Milestone

Comments

@antiochp
Copy link
Member

Recent exploration into the implementation of an index of "recently seen" kernels led us down the path of maintaining an "undo" list per block. This way we can robustly handle duplicate kernels, maintaining the pos of the most recent instance of the kernel in the index, while being able to handle rewind scenarios involving previous instances of the kernel.

Related - #3228

We do something similar to this for the output_pos index currently where we track a bitmap of spent output pos per block. And during rewind we use this per-block bitmap to ensure the UTXO set is consistent with rewound chain state. Note that this allows the utxo set to be rewound reliably but leaves "false-positives" in the output_pos index as we do not rewind the index itself.

So we have always used the output_pos index as non-authoritative. During rewind we can (and will expect to) encounter "false-positive" results from the index. We may find an entry in the index for a given output, but we do not yet know for sure if this index entry is accurate. So we go to the PMMR itself for authoritative view of chain state.

The process is roughly -

  • look in output_pos for possible result
    • if nothing in index then output is spent
    • if we find a result in the index, look in the output PMMR
  • compare output commitment between index and PMMR
    • if match then output is unspent
    • otherwise output is spent

It became apparent during the kernel index exploration that there is a better way to do this that can also be applied to the output_pos index.
Rather than storing a bitmap of spent output positions which is by definition unsorted, we can store a vec of output positions where the sort order is consistent with the inputs to the block.
This provides the ability to map positions to input commitments themselves. Rewind can now take advantage of this to ensure the output_pos index remains consistent with the rewound chain state. The output_pos index can be maintained transactionally alongside block processing, in both regular (apply new block) and rewound directions.
This eliminates the possibility for "false positives" in the index lookup. If we find an entry in the index then we know the output is unspent. If we do not find an entry, the output is spent, or never existed.

This involves a change to what we store in the db. We need to store a vec of output_pos (and associated block heights) rather than the existing serialized bitmap per block.
This also involves changes to both apply_block() and the rewind_block impls to update the output_pos as necessary, within the txhashset extension (and therefore transactionally).

Reworking the output_pos index to handle transactional rewind in this way gives us a clear path forward to use the same approach for the kernel_pos index. Rather than "spending" outputs we replace kernels with more recent instances (in the "recently seen" index).

We have this working on a branch. PR coming shortly for these changes. Ideally we can get this into 3.1.0-beta and allow it to be sufficiently tested and released as part of 3.1.0.
Tracking this here so we can tag it as scheduled for inclusion in 3.1.0.

@antiochp antiochp added this to the 3.1.0 milestone Feb 20, 2020
@antiochp antiochp self-assigned this Feb 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant