Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollbacks issue in stake pool worker #1200

Closed
KtorZ opened this issue Dec 17, 2019 · 0 comments
Closed

Rollbacks issue in stake pool worker #1200

KtorZ opened this issue Dec 17, 2019 · 0 comments
Assignees

Comments

@KtorZ
Copy link
Member

KtorZ commented Dec 17, 2019

Context

Relates to #1181.

Information -
Version v2019-12-16 4639180
Platform All
Installation N/A

Steps to Reproduce

  1. If the node rolls back to a previous epoch, it may happen that it discards a few delegation certificates. However, the database can only handle rollbacks with an "epoch precision" (it discards data that are strictly or greater to the epoch of rollback). But, we may still discover certificates that are known within the same epoch and, end up in situation like the following:
SQLite3 returned ErrorConstraint while attempting to perform step: UNIQUE constraint failed: pool_registration.pool_id

Expected behavior

  • Rollbacks doesn't lead the stake pools database in a blocked state.

Actual behavior

  • If this error is encountered, we'll simply keep looping and re-run into this error forever.

Resolution

  • Modified the database (in a non backward-compatible) way to store slot id instead of mere epoch number allowing for handling rollbacks correctly ==> Fix pool registration rollback #1201

  • Handled db unsafe migration for stake pools by re-creating the pool database from scratch if an old database is encountered (forcing a re-sync) ==> Fix pool registration rollback #1201


QA

@KtorZ KtorZ self-assigned this Dec 17, 2019
iohk-bors bot added a commit that referenced this issue Dec 18, 2019
1201: Fix pool registration rollback r=KtorZ a=KtorZ

# Issue Number

<!-- Put here a reference to the issue this PR relates to and which requirements it tackles -->

#1200

# Overview

<!-- Detail in a few bullet points the work accomplished in this PR -->

- commit 18852c0
    modify rollback property to capture the rollback in-within epoch problem. It now fails

- commit 3a4d4fd
    store registration with full slot and not only epoch to allow correct rollback!
    
    This is however a DB breaking change which requires migration...
    Yet, there's no way to migrate from an existing database to this new
    schema. Since however this is a preferable solution and, re-syncing
    the pools data is cheap, we'll simply discards any existing database
    that has an invalid format and, resync from scratch :)

- commit f21f4e3
    add a regression test to show that migration from an old database doesn't work out of the box
    Currently fails with:
    
    ```
    PersistError 
    Database migration: manual intervention required.
    The unsafe actions are prefixed by '***' below:
    CREATE TEMP TABLE \"pool_registration_backup\"(\"pool_id\" VARCHAR NOT NULL,\"slot\" INTEGER NOT NULL,\"margin\" INTEGER NOT NULL,\"cost\" INTEGER NOT NULL, PRIMARY KEY (\"pool_id\"));
    INSERT INTO \"pool_registration_backup\"(\"pool_id\",\"margin\",\"cost\") SELECT \"pool_id\",\"margin\",\"cost\" FROM \"pool_registration\";
    *** DROP TABLE \"pool_registration\";
    CREATE TABLE \"pool_registration\"(\"pool_id\" VARCHAR NOT NULL,\"slot\" INTEGER NOT NULL,\"margin\" INTEGER NOT NULL,\"cost\" INTEGER NOT NULL, PRIMARY KEY (\"pool_id\"));
    INSERT INTO \"pool_registration\" SELECT \"pool_id\",\"slot\",\"margin\",\"cost\" FROM \"pool_registration_backup\";
    DROP TABLE \"pool_registration_backup\";"
    ```

- commit 440e099 
    handle migration error in stake pool db layer
    As described in the previous commit.

- commit 112927b
    better assertions on logs message for db migration tests



# Comments

<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: KtorZ <[email protected]>
iohk-bors bot added a commit that referenced this issue Dec 18, 2019
1201: Fix pool registration rollback r=KtorZ a=KtorZ

# Issue Number

<!-- Put here a reference to the issue this PR relates to and which requirements it tackles -->

#1200

# Overview

<!-- Detail in a few bullet points the work accomplished in this PR -->

- commit 18852c0
    modify rollback property to capture the rollback in-within epoch problem. It now fails

- commit 3a4d4fd
    store registration with full slot and not only epoch to allow correct rollback!
    
    This is however a DB breaking change which requires migration...
    Yet, there's no way to migrate from an existing database to this new
    schema. Since however this is a preferable solution and, re-syncing
    the pools data is cheap, we'll simply discards any existing database
    that has an invalid format and, resync from scratch :)

- commit f21f4e3
    add a regression test to show that migration from an old database doesn't work out of the box
    Currently fails with:
    
    ```
    PersistError 
    Database migration: manual intervention required.
    The unsafe actions are prefixed by '***' below:
    CREATE TEMP TABLE \"pool_registration_backup\"(\"pool_id\" VARCHAR NOT NULL,\"slot\" INTEGER NOT NULL,\"margin\" INTEGER NOT NULL,\"cost\" INTEGER NOT NULL, PRIMARY KEY (\"pool_id\"));
    INSERT INTO \"pool_registration_backup\"(\"pool_id\",\"margin\",\"cost\") SELECT \"pool_id\",\"margin\",\"cost\" FROM \"pool_registration\";
    *** DROP TABLE \"pool_registration\";
    CREATE TABLE \"pool_registration\"(\"pool_id\" VARCHAR NOT NULL,\"slot\" INTEGER NOT NULL,\"margin\" INTEGER NOT NULL,\"cost\" INTEGER NOT NULL, PRIMARY KEY (\"pool_id\"));
    INSERT INTO \"pool_registration\" SELECT \"pool_id\",\"slot\",\"margin\",\"cost\" FROM \"pool_registration_backup\";
    DROP TABLE \"pool_registration_backup\";"
    ```

- commit 440e099 
    handle migration error in stake pool db layer
    As described in the previous commit.

- commit 112927b
    better assertions on logs message for db migration tests



# Comments

<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: KtorZ <[email protected]>
@KtorZ KtorZ closed this as completed Dec 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant