Merge pull request #23 from input-output-hk/bwbush/peras-iosim

Statistical analysis of block production and statistics-aware quickcheck

Logbook.md

@@ -1,5 +1,17 @@
 ## 2024-02-14
 
+### BB on testing statistical properties
+
+1. Ran and statistically analyzed an experiment [using this Jupyter notebook](peras-iosim/analyses/block-production/ReadMe.ipynb) to verify the correctness of slot leadership and block production for the `PseudoPraos` protocol.
+2. Research QuickCheck-based approaches for handling approximate equality and statistical properties.
+    - There don't seem to be special-purpose libraries of this.
+    - QuickCheck and its extension provide some control over coverage and sampling, but this doesn't address the fundamental problem of determining when a failure is a valid statistical outlier vs an error.
+3. Revised QuickCheck test for block production to use a three-sigma confidence interval. Eventually, we'll need to make the sigma level tunable.
+
+Other revisions to `peras-iosim`:
+- Added an explicit parameter for total stake to `Peras.IOSim.Simulate.Types.Parameters`.
+- Changed serialization of `Peras.Chain` so that it doesn't keeply next JSON, since that causes most JSON parsers to crash for long chains.
+
 ### AB on Network Modeling
 
 Reading about Rust's FFI: https://www.michaelfbryan.com/rust-ffi-guide/setting_up.html

peras-iosim/analyses/block-production/experiment.sh

@@ -0,0 +1,40 @@
+#/usr/bin/env bash
+
+PERAS_IOSIM=../../../dist-newstyle/build/x86_64-linux/ghc-9.6.3/peras-iosim-0.1.0.0/x/peras-iosim/noopt/build/peras-iosim/peras-iosim
+
+cat << EOI > tmp-results.csv
+Seed,Active Slot Coefficient,End Slot,Total Stake,Node Stake,Blocks Produced
+EOI
+
+for i in {1..1000}
+do
+
+SEED=$RANDOM
+TOTAL_STAKE=1000
+END_SLOT=7200
+ASC=0.05
+
+cat << EOI > tmp-network.yaml
+randomSeed: $SEED
+peerCount: 1
+downstreamCount: 0
+totalStake: $TOTAL_STAKE
+maximumStake: $TOTAL_STAKE
+messageDelay: 0.35
+endSlot: $END_SLOT
+EOI
+
+cat << EOI > tmp-protocol.yaml
+protocol: PseudoPraos
+activeSlotCoefficient: $ASC
+EOI
+
+echo "$i: $SEED"
+
+"$PERAS_IOSIM" --parameter-file tmp-network.yaml --protocol-file tmp-protocol.yaml --result-file tmp-result.json
+
+jq -r '.exitStates.N1 | "'"$SEED","$ASC","$END_SLOT","$TOTAL_STAKE"',\(.stake),\(.preferredChain.blocks|length)"' tmp-result.json >> tmp-results.csv
+
+done
+
+sort -ur tmp-results.csv > results.csv