Storage migration for Atree register inlining

[fxamacker]

onflow/atree#292 requires storage migration.

Several PRs such as onflow/atree#330 changed the way data is stored in accounts on Flow. Future PRs related to onflow/atree#292 will make further changes to Atree-based storage format. E.g.

Given this, we will need to migrate existing data to the new Atree storage format used by PRs related to epic onflow/atree#292.

Two main items are proceeding in parallel

• @janezpodhostnik - Storage migration program is in PR Atree storage migration #4633.
• @fxamacker - Atree Inlining is in PRs tracked by epic Optimize atree to reduce Flow's mtrie size and reduce number of reads atree#292 such as
  • Atree Register Inlining and Data Deduplication atree#342

TODO

A. Storage migration dry runs without Atree inlining:

• 1. testnet data - dry-run storage migration without Atree inlining
• 2. mainnet data - dry-run storage migration without Atree inlining
  • succeeded WITH VALIDATION ENABLED using Oct 25, 2023 mainnet snapshot using Atree storage migration #4633 (thanks @janezpodhostnik!)
    • peak memory use (e.g. max rss reported by time -a)
    • total duration and atree migration duration
• 3. optimize storage migration if needed
  • Migrate payloads by reference #5123
  • Change migration progress log to be more verbose #5122
  • Migrate-by-account changes #5128
  • Deduplicate contract names migration #5143
  • OPTIONAL Change progress loggers to reduce logging overhead #5140
• 4. OPTIONAL add option to migration program to use atree.CheckStorageHealth (thanks @bluesign for suggestion)
  • We now log unmigrated registers during migration, manually checked this after dry-runs, etc. so maybe this can be skipped for now?
• 5. Replace hash-based validation of migrated Cadence values to use Equal() #5204

Next steps (external to this issue's main PR) that can proceed in parallel include:

• Atree Register Inlining and Data Deduplication atree#342
• Add readonly iterators and support value mutations only from non-readonly iterators atree#345
• Update smoke test for atree inlining atree#348
• Make smoke tests check recently added data deduplication feature atree#350
• Integrate with atree inlining and data deduplication cadence#2882
• Add feature to support mutation for array and map iterators atree#359

B. Storage migration dry runs with Atree inlining:

These require Atree Inlining and Data Deduplication, Cadence Integration, etc. listed above.

• 1. testnet data - dry-run storage migration with Atree inlining
• 2. mainnet data - dry-run storage migration with Atree inlining
• 3. OPTIONAL optimize storage migration if needed

C. Actual storage migration with atree register inlining:

TBD based on results of dry runs.

[fxamacker]

Closing this issue (reopen if needed).

Integration of Atree inlining and Cadence v0.42.x with flow-go is tracked in issue #5296.