Proposal for suite structure, test types, and configs

[protolambda]

First of all, credits to @djrtwo and others who have been working on the test formats. Much of the work in this issue is a copy/adaption of theirs.

Motivation:

• Unify and standardize the tests. I am looking to create all different test types to get my Go "executable spec" fully conforming with the Python version. Having all these issues open in a limbo state does not help. Better get something reasonable down in the mean time. Polish it later.
• Dealing with constants that need to be readable and truely constant is a pain-point for me. Separate configurations would help A LOT. See here.
• Test types are fundamental, and a test-runner shouldn't be running through 1000s of files to just execute e.g. the fork-choice tests. It's also easier to implement test runners that are at least somewhat encapsulated to the package/crate/module/whatever that actually runs the tests.
• Putting a lot of unrelated test cases into one file makes things harder to understand. Let's avoid that.
• Having one standard format for test-suite headers would be nice.

Definitions

Test types

• shuffling: tests validator shuffling
• bls: tests crypto functions of BLS
• ssz: tests serialization
• hash_tree_root: hash-tree-roots (incl. and excl. of signatures)
• fork_choice: tests the choice between blocks when determining the head of the chain
• deltas: tests the computation of rewards and penalties, as defined by the spec as deltas.
• state_transition: originally beacon_state, contains big transition with list of blocks
• epoch_transition: tests sub-transitions within an epoch transition
• block_transition: tests sub-transitions within a block transition

Test suites

A test suite is simply a YAML file in the directory of its test-type.

Suites are:

• minimal
• descriptive
• versioned by fork
• configured

Format

title: <string, short, one line>
summary: <string, average, 1-3 lines>
fork: <string, first the phase name, then the spec version>
config: <string, reference to a config file, without extension>

test_cases: <key,value map, values being maps defining a test case each>
   ...

The choice for just a fork field, and no version field, stems from #27 and #28

An clear example of the format:

title: rewards and penalties
summary: Cover changes during epoch processing of rewards and penalties
fork: phase0-0.5.0
config: phase0_minimal

test_cases:
   ...

Configs

Configs are defined separately, each in its own YAML file.

Since configs are re-used across tests and we don't want to read them as random variables, each field should have a comment. I.e. the default phase0 config should just copy the phase 0 spec comments, alternative versions, like a minimal config, should describe the choice for changes.

The format is really simple, just the names of the constants, followed by their value:

# Minimal amount of shards, with a few different shard committees
SHARD_COUNT: 8
# Lower, less secure, not to be used in production.
TARGET_COMMITTEE_SIZE: 16
...

Test case types

shuffling

Tests validator shuffling. See: #10.

Format:

input:
  epoch: <int>
  validators: <list of validators>
output: <list of committees>
seed: <bytes 32, hex-encoded with prefix, string>

A validator is defined as:

activation_epoch: <int>
exit_epoch: <int, can be the int value of FAR_FUTURE>
original_index: <int>

A committee is defined as an inline list of integers, example:

[3,4,5,1,8]

bls

Tests crypto functions of BLS. See #16

ssz

Tests serialization. See #13

hash_tree_root

Tests hash-tree-roots (incl. and excl. of signatures). See #14

fork_choice

Tests the choice between blocks when determining the head of the chain. See #12. (note: proposing some opinionated changes here, to be discussed later, and - -> _)

# Can be different from first block, just to make sure it still handles fork-choice well / aborts properly.
start: <block-id>
blocks:
  - id: <block-id>
    parent: <block-id>
weights:
  # if no matching key for block-id, weight = 0
  - <block-id>: <int>
  - ...
head: <block-id, not present if start or weight is invalid>

deltas

Tests the computation of rewards and penalties, as defined by the spec as deltas.
This is a new idea, but worthwhile imho to get one of the most important parts readily testable. Clients that don't work with "deltas" vectors can just wrap their rewarding/slashing functions with a simple function that collects the calculations.

The signature is basically: state -> (rewards, penalties) (with state being the point of the state starting from the rewards/penalties entry in the epoch transition)

Format to be proposed later.

state_transition

Tests the bigger transitions with a list of blocks, like a so-called "smoke-test". See: beacon_state, #21, same test_case format, except no per-case config (per suite now).

Format:

# all fields of BeaconState
initial_state: <key/value map>
# A list of blocks to be processed sequentially on top of the initial state
blocks: <list of key/value maps, each encoding a block>
# A subset of fields of BeaconState containing the expected values of the resulting state
expected_state: <key/value map>
# Hash_tree_root(state) after processing to the latest block in blocks
expected_state_root: <optional, 32-byte hex string>    

epoch_transition

Tests sub-transitions within an epoch transition.

Similar to full state_transition test format:

Format:

# transition handle
handle: <an epoch sub-transition handle>
# all fields of BeaconState, in the state just before the epoch sub-transition processing
initial_state: <key/value map>
# A subset of fields of BeaconState containing the expected values of the resulting state, just after executing the sub-state transition
expected_state: <key/value map>
# Hash_tree_root(expected_state)
expected_state_root: <optional, 32-byte hex string>    

Epoch sub-transition handles:

• eth1
• justification
• crosslinks
• rewards_and_penalties
• ejections
• validator_registry
• slashings
• exit_queue
• finish

block_transition

Tests sub-transitions within a block transition, with a single block input in addition to the initial state.

Similar to full state_transition test format:

Format:

# transition handle
handle: <a block sub-transition handle>
# all fields of BeaconState
initial_state: <key/value map>
# The block being processed, in the state of just before the block sub-transition processing.
block: <key/value map, a single block>
# A subset of fields of BeaconState containing the expected values of the resulting state
expected_state: <key/value map>
# Hash_tree_root(state) after processing to the latest block in blocks
expected_state_root: <optional, 32-byte hex string>    

Block sub-transition handles:

• header
• randao
• eth1
• proposer_slashings
• attester_slashings
• attestations
• deposits
• voluntary_exits
• transfers

Structure

The configs and tests define the top-level folder contents,
with tests containing collections of typed test-suites, named by type.

Such tree structure could look like this:

├── configs
│   ├── phase0_minimal.yaml
│   ├── phase0.yaml
│   └── phase1.yaml
└── tests
    ├── bls
    │   ├── verify_multi.yaml
    │   ├── verify_single.yaml
    │   ├── aggregation.yaml
    │   └── sign_msg.yaml
    ├── block_transitions
    │   └── deposits.yaml
    ├── deltas
    │   ├── crosslinks.yaml
    │   └── justification.yaml
    ├── epoch_transitions
    │   ├── eth1.yaml
    │   └── rewards_and_penalties.yaml
    ├── fork_choice
    │   ├── long_chains.yaml
    │   ├── half_half.yaml
    │   └── big_branch_factor.yaml
    ├── hash_tree_root
    │   ├── dynamic_size.yaml
    │   ├── edge_cases.yaml
    │   ├── deep_nested.yaml
    │   ├── hash_tree_root.yaml
    │   └── signed_root.yaml
    ├── shuffling
    │   ├── big_shuffling.yaml
    │   ├── edge_cases.yaml
    │   └── permute_index.yaml
    ├── ssz
    │   ├── common.yaml
    │   └── serialization.yaml
    └── total_transitions
        ├── epochs_1.yaml
        ├── epochs_n.yaml
        ├── fast_chain.yaml
        ├── genesis.yaml
        ├── slots_0.yaml
        ├── slots_1.yaml
        ├── slots_n.yaml
        └── slow_chain.yaml

Now let's get something like this standardized, and I can start implementing it :)

[protolambda]

This may also come close to getting #15 done.

[djrtwo]

block_transition has blocks defined rather than block. This should be singular, no?

[protolambda]

@djrtwo Good spot, yes, see comment. Will fix type description

[jannikluhn]

Regarding versions: Do we want to support ranges? (e.g. >=0.5.0 or something)? This seems useful in the long run, but it might be overkill for now (as test runners likely won't implement it anyways).