Skip to content

Commit

Permalink
Merge pull request #5057 from stacks-network/docs/local-mutantion-tes…
Browse files Browse the repository at this point in the history 
…ting

docs & script for running cargo mutants locally on CI limitation
  • Loading branch information
@kantai
kantai authored Aug 9, 2024
2 parents beeebe6 + 5cc974e commit 4867505
Show file tree
Hide file tree
Showing 2 changed files with 233 additions and 0 deletions.
87 changes: 87 additions & 0 deletions contrib/tools/local-mutation-testing.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
#!/bin/bash

set -euo pipefail

# Install cargo-mutants
cargo install --version 24.7.1 cargo-mutants --locked

# Create diff file between current branch and develop branch
git diff origin/develop...HEAD > git.diff

# Remove git diff files about removed/renamed files
awk '
/^diff --git/ {
diff_line = $0
getline
if ($0 !~ /^(deleted file mode|similarity index)/) {
print diff_line
print
}
}
!/^(diff --git|deleted file mode|similarity index|rename from|rename to)/ {print}
' git.diff > processed.diff

# Extract mutants based on the processed diff
cargo mutants --in-diff processed.diff --list > all_mutants.txt

# Create a directory for organizing mutants
mkdir -p mutants_by_package

# Organize mutants into files based on their main folder
while IFS= read -r line; do
package=$(echo "$line" | cut -d'/' -f1)

case $package in
"stackslib")
echo "$line" >> "mutants_by_package/stackslib.txt"
;;
"testnet")
echo "$line" >> "mutants_by_package/stacks-node.txt"
;;
"stacks-signer")
echo "$line" >> "mutants_by_package/stacks-signer.txt"
;;
*)
echo "$line" >> "mutants_by_package/small-packages.txt"
;;
esac
done < all_mutants.txt

# Function to run mutants for a package
run_mutants() {
local package=$1
local threshold=$2
local output_dir=$3
local mutant_file="mutants_by_package/${package}.txt"

if [ ! -f "$mutant_file" ]; then
echo "No mutants found for $package"
return 0
fi

local regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "$mutant_file" | paste -sd'|' -)
local mutant_count=$(cargo mutants -F "$regex_pattern" -E ": replace .{1,2} with .{1,2} in " --list | wc -l)

if [ "$mutant_count" -gt "$threshold" ]; then
echo "Running mutants for $package ($mutant_count mutants)"
RUST_BACKTRACE=1 BITCOIND_TEST=1 \
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
-F "$regex_pattern" \
-E ": replace .{1,2} with .{1,2} in " \
--output "$output_dir" \
--test-tool=nextest \
--package "$package" \
-- --all-targets --test-threads 1 || true

echo $? > "${output_dir}/exit_code.txt"
else
echo "Skipping $package, only $mutant_count mutants (threshold: $threshold)"
fi

return 0
}

# Run mutants for each wanted package
run_mutants "stacks-signer" 500 "./stacks-signer_mutants" || true
run_mutants "stacks-node" 540 "./stacks-node_mutants" || true
run_mutants "stackslib" 72 "./stackslib_mutants" || true
146 changes: 146 additions & 0 deletions docs/mutation-testing.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
# Mutation Testing

This document describes how to run mutation testing locally to mimic the outcome of a PR, without the CI limitation it provides by timing out after 6 hours.
[Here is the script](../contrib/tools/local-mutation-testing.sh) to run the tests locally by running the mutants created by the changes between `HEAD` and develop.
It does automatically all the steps explained below.

From the root level of the stacks-core repository run
```sh
./contrib/tools/local-mutation-testing.sh
```

## Prerequirements

Install the cargo mutants library
```sh
cargo install --version 24.7.1 cargo-mutants --locked
```


## Steps
1. Be on source branch you would use for the PR.
2. Create diff file comparing this branch with the `develop` branch
```sh
git diff origin/develop..HEAD > git.diff
```
3. Clean up the diff file and create auxiliary files
```sh
awk '
/^diff --git/ {
diff_line = $0
getline
if ($0 !~ /^(deleted file mode|similarity index)/) {
print diff_line
print
}
}
!/^(diff --git|deleted file mode|similarity index|rename from|rename to)/ {print}
' git.diff > processed.diff
# Extract mutants based on the processed diff
cargo mutants --in-diff processed.diff --list > all_mutants.txt
# Create a directory for organizing mutants
mkdir -p mutants_by_package
# Organize mutants into files based on their main folder
while IFS= read -r line; do
package=$(echo "$line" | cut -d'/' -f1)
case $package in
"stackslib")
echo "$line" >> "mutants_by_package/stackslib.txt"
;;
"testnet")
echo "$line" >> "mutants_by_package/stacks-node.txt"
;;
"stacks-signer")
echo "$line" >> "mutants_by_package/stacks-signer.txt"
;;
*)
echo "$line" >> "mutants_by_package/small-packages.txt"
;;
esac
done < all_mutants.txt
```
4. Based on the package required to run the mutants for
a. Stackslib package
```sh
regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "mutants_by_package/stackslib.txt" | paste -sd'|' -)
RUST_BACKTRACE=1 BITCOIND_TEST=1 \
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
-F "$regex_pattern" \
-E ": replace .{1,2} with .{1,2} in " \
--output "./stackslib_mutants" \
--test-tool=nextest \
-- --all-targets --test-threads 1
```
b. Stacks-node (testnet) package
```sh
regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "mutants_by_package/testnet.txt" | paste -sd'|' -)
RUST_BACKTRACE=1 BITCOIND_TEST=1 \
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
-F "$regex_pattern" \
-E ": replace .{1,2} with .{1,2} in " \
--output "./testnet_mutants" \
--test-tool=nextest \
-- --all-targets --test-threads 1
```
c. Stacks-signer
```sh
regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "mutants_by_package/stacks-signer.txt" | paste -sd'|' -)
RUST_BACKTRACE=1 BITCOIND_TEST=1 \
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
-F "$regex_pattern" \
-E ": replace .{1,2} with .{1,2} in " \
--output "./stacks-signer_mutants" \
--test-tool=nextest \
-- --all-targets --test-threads 1
```
d. All other packages combined
```sh
regex_pattern=$(sed 's/[][()\.^$*+?{}|]/\\&/g' "mutants_by_package/small-packages.txt" | paste -sd'|' -)
cargo mutants --timeout-multiplier 1.5 --no-shuffle -vV \
-F "$regex_pattern" \
-E ": replace .{1,2} with .{1,2} in " \
--output "./small-packages_mutants" \
--test-tool=nextest \
-- --all-targets --test-threads 1
```

## How to run one specific mutant to test it

Example of output which had a missing mutant
```sh
MISSED stacks-signer/src/runloop.rs:424:9: replace <impl SignerRunLoop for RunLoop<Signer, T>>::run_one_pass -> Option<Vec<SignerResult>> with None in 3.0s build + 9.3s test
```
Example of fix for it
```sh
RUST_BACKTRACE=1 BITCOIND_TEST=1 \
cargo mutants -vV \
-F "replace process_stackerdb_event" \
-E ": replace <impl SignerRunLoop for RunLoop<Signer, T>>::run_one_pass -> Option<Vec<SignerResult>> with None in " \
--test-tool=nextest \
-- \
--run-ignored all \
--fail-fast \
--test-threads 1
```
General command to run
```sh
RUST_BACKTRACE=1 BITCOIND_TEST=1 \
cargo mutants -vV \
-F "replace process_stackerdb_event" \
-E ": replace [modify this] with [modify this] in " \
--test-tool=nextest \
-- \
--run-ignored all \
--fail-fast \
--test-threads 1
```

0 comments on commit 4867505

Please sign in to comment.