Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race building execution and builder block with cutoff #5223

Merged
merged 23 commits into from
Mar 6, 2023

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Mar 1, 2023

We have scenarios where builder api responded with block but execution api took forever to resolve causing an avoidable delay in block proposal (and hence orphaned block)

This PR races building execution and builder block with cutoff of 3 seconds post which the race resolves with whoever wins

  • testcases for racePromisesWithCutoff utility
    image

Closes #5219

@g11tech g11tech requested a review from a team as a code owner March 1, 2023 16:09
@g11tech g11tech marked this pull request as draft March 1, 2023 16:09
@wemeetagain
Copy link
Member

Can this be done with somehting like:

function allSettledWithTimeout<T>(promises: Promise<T>, timeoutMs: number) {
  const timerPromise = new Promise((_resolve, reject) => setTimeout(reject, timeoutMs));
  return Promise.allSettled(promises.map((promise) => Promise.race(promise, timerPromise)));
}

@github-actions
Copy link
Contributor

github-actions bot commented Mar 1, 2023

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: d517bd5 Previous: 0d638ba Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 505.56 us/op 838.37 us/op 0.60
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 53.634 us/op 43.776 us/op 1.23
BLS verify - blst-native 1.2247 ms/op 1.1907 ms/op 1.03
BLS verifyMultipleSignatures 3 - blst-native 2.5011 ms/op 2.4204 ms/op 1.03
BLS verifyMultipleSignatures 8 - blst-native 5.3569 ms/op 5.1937 ms/op 1.03
BLS verifyMultipleSignatures 32 - blst-native 19.906 ms/op 18.756 ms/op 1.06
BLS aggregatePubkeys 32 - blst-native 26.481 us/op 24.993 us/op 1.06
BLS aggregatePubkeys 128 - blst-native 102.65 us/op 97.900 us/op 1.05
getAttestationsForBlock 54.348 ms/op 53.032 ms/op 1.02
isKnown best case - 1 super set check 263.00 ns/op 253.00 ns/op 1.04
isKnown normal case - 2 super set checks 269.00 ns/op 249.00 ns/op 1.08
isKnown worse case - 16 super set checks 271.00 ns/op 267.00 ns/op 1.01
CheckpointStateCache - add get delete 5.2720 us/op 5.2240 us/op 1.01
validate gossip signedAggregateAndProof - struct 2.7927 ms/op 2.7727 ms/op 1.01
validate gossip attestation - struct 1.3253 ms/op 1.3239 ms/op 1.00
pickEth1Vote - no votes 1.2903 ms/op 1.2231 ms/op 1.05
pickEth1Vote - max votes 9.2449 ms/op 10.171 ms/op 0.91
pickEth1Vote - Eth1Data hashTreeRoot value x2048 8.5384 ms/op 8.4952 ms/op 1.01
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 13.565 ms/op 13.247 ms/op 1.02
pickEth1Vote - Eth1Data fastSerialize value x2048 632.36 us/op 672.80 us/op 0.94
pickEth1Vote - Eth1Data fastSerialize tree x2048 4.5307 ms/op 7.5671 ms/op 0.60
bytes32 toHexString 486.00 ns/op 497.00 ns/op 0.98
bytes32 Buffer.toString(hex) 338.00 ns/op 350.00 ns/op 0.97
bytes32 Buffer.toString(hex) from Uint8Array 537.00 ns/op 560.00 ns/op 0.96
bytes32 Buffer.toString(hex) + 0x 338.00 ns/op 347.00 ns/op 0.97
Object access 1 prop 0.16800 ns/op 0.16900 ns/op 0.99
Map access 1 prop 0.16100 ns/op 0.15600 ns/op 1.03
Object get x1000 6.6680 ns/op 6.4740 ns/op 1.03
Map get x1000 0.61000 ns/op 0.60000 ns/op 1.02
Object set x1000 51.936 ns/op 51.181 ns/op 1.01
Map set x1000 43.837 ns/op 42.532 ns/op 1.03
Return object 10000 times 0.23710 ns/op 0.23030 ns/op 1.03
Throw Error 10000 times 4.2236 us/op 4.0266 us/op 1.05
fastMsgIdFn sha256 / 200 bytes 3.5560 us/op 3.3680 us/op 1.06
fastMsgIdFn h32 xxhash / 200 bytes 292.00 ns/op 305.00 ns/op 0.96
fastMsgIdFn h64 xxhash / 200 bytes 395.00 ns/op 380.00 ns/op 1.04
fastMsgIdFn sha256 / 1000 bytes 11.755 us/op 11.475 us/op 1.02
fastMsgIdFn h32 xxhash / 1000 bytes 416.00 ns/op 432.00 ns/op 0.96
fastMsgIdFn h64 xxhash / 1000 bytes 471.00 ns/op 470.00 ns/op 1.00
fastMsgIdFn sha256 / 10000 bytes 104.77 us/op 103.98 us/op 1.01
fastMsgIdFn h32 xxhash / 10000 bytes 2.0150 us/op 1.9210 us/op 1.05
fastMsgIdFn h64 xxhash / 10000 bytes 1.4320 us/op 1.3430 us/op 1.07
enrSubnets - fastDeserialize 64 bits 1.3280 us/op 1.2900 us/op 1.03
enrSubnets - ssz BitVector 64 bits 486.00 ns/op 480.00 ns/op 1.01
enrSubnets - fastDeserialize 4 bits 171.00 ns/op 167.00 ns/op 1.02
enrSubnets - ssz BitVector 4 bits 486.00 ns/op 474.00 ns/op 1.03
prioritizePeers score -10:0 att 32-0.1 sync 2-0 112.88 us/op 100.88 us/op 1.12
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 147.55 us/op 133.50 us/op 1.11
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 182.37 us/op 163.78 us/op 1.11
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 316.54 us/op 297.84 us/op 1.06
prioritizePeers score 0:0 att 64-1 sync 4-1 371.03 us/op 358.57 us/op 1.03
array of 16000 items push then shift 1.6393 us/op 1.6111 us/op 1.02
LinkedList of 16000 items push then shift 8.8980 ns/op 8.6260 ns/op 1.03
array of 16000 items push then pop 90.427 ns/op 86.950 ns/op 1.04
LinkedList of 16000 items push then pop 9.0300 ns/op 8.5880 ns/op 1.05
array of 24000 items push then shift 2.4636 us/op 2.3545 us/op 1.05
LinkedList of 24000 items push then shift 9.1710 ns/op 8.7110 ns/op 1.05
array of 24000 items push then pop 85.000 ns/op 78.142 ns/op 1.09
LinkedList of 24000 items push then pop 8.8800 ns/op 8.2920 ns/op 1.07
intersect bitArray bitLen 8 13.926 ns/op 12.959 ns/op 1.07
intersect array and set length 8 86.245 ns/op 75.754 ns/op 1.14
intersect bitArray bitLen 128 45.216 ns/op 43.096 ns/op 1.05
intersect array and set length 128 1.1843 us/op 1.0409 us/op 1.14
Buffer.concat 32 items 2.8120 us/op 2.9990 us/op 0.94
Uint8Array.set 32 items 2.7880 us/op 2.4540 us/op 1.14
pass gossip attestations to forkchoice per slot 2.3746 ms/op 3.1549 ms/op 0.75
computeDeltas 3.0764 ms/op 3.0749 ms/op 1.00
computeProposerBoostScoreFromBalances 1.7864 ms/op 1.8511 ms/op 0.97
altair processAttestation - 250000 vs - 7PWei normalcase 2.2045 ms/op 3.4115 ms/op 0.65
altair processAttestation - 250000 vs - 7PWei worstcase 3.4106 ms/op 5.1327 ms/op 0.66
altair processAttestation - setStatus - 1/6 committees join 144.94 us/op 151.47 us/op 0.96
altair processAttestation - setStatus - 1/3 committees join 289.95 us/op 300.79 us/op 0.96
altair processAttestation - setStatus - 1/2 committees join 367.78 us/op 417.63 us/op 0.88
altair processAttestation - setStatus - 2/3 committees join 511.24 us/op 505.12 us/op 1.01
altair processAttestation - setStatus - 4/5 committees join 672.63 us/op 718.96 us/op 0.94
altair processAttestation - setStatus - 100% committees join 785.09 us/op 845.49 us/op 0.93
altair processBlock - 250000 vs - 7PWei normalcase 17.700 ms/op 18.271 ms/op 0.97
altair processBlock - 250000 vs - 7PWei normalcase hashState 27.812 ms/op 30.993 ms/op 0.90
altair processBlock - 250000 vs - 7PWei worstcase 49.546 ms/op 57.209 ms/op 0.87
altair processBlock - 250000 vs - 7PWei worstcase hashState 65.383 ms/op 80.962 ms/op 0.81
phase0 processBlock - 250000 vs - 7PWei normalcase 1.9605 ms/op 2.6923 ms/op 0.73
phase0 processBlock - 250000 vs - 7PWei worstcase 28.090 ms/op 33.639 ms/op 0.84
altair processEth1Data - 250000 vs - 7PWei normalcase 444.98 us/op 651.38 us/op 0.68
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 6.7720 us/op 13.959 us/op 0.49
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 20.019 us/op 49.911 us/op 0.40
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 10.006 us/op 12.739 us/op 0.79
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 6.5170 us/op 11.496 us/op 0.57
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 74.267 us/op 106.74 us/op 0.70
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 593.99 us/op 721.40 us/op 0.82
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 899.18 us/op 914.43 us/op 0.98
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 848.67 us/op 908.70 us/op 0.93
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.1647 ms/op 2.3572 ms/op 0.92
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.5732 ms/op 1.6613 ms/op 0.95
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 3.7457 ms/op 3.7943 ms/op 0.99
Tree 40 250000 create 301.98 ms/op 288.39 ms/op 1.05
Tree 40 250000 get(125000) 175.26 ns/op 174.82 ns/op 1.00
Tree 40 250000 set(125000) 898.73 ns/op 851.67 ns/op 1.06
Tree 40 250000 toArray() 16.892 ms/op 18.048 ms/op 0.94
Tree 40 250000 iterate all - toArray() + loop 16.362 ms/op 21.203 ms/op 0.77
Tree 40 250000 iterate all - get(i) 64.583 ms/op 69.549 ms/op 0.93
MutableVector 250000 create 10.169 ms/op 10.843 ms/op 0.94
MutableVector 250000 get(125000) 6.3830 ns/op 6.4480 ns/op 0.99
MutableVector 250000 set(125000) 272.73 ns/op 257.66 ns/op 1.06
MutableVector 250000 toArray() 2.7697 ms/op 2.7896 ms/op 0.99
MutableVector 250000 iterate all - toArray() + loop 3.0589 ms/op 2.8992 ms/op 1.06
MutableVector 250000 iterate all - get(i) 1.7239 ms/op 1.5754 ms/op 1.09
Array 250000 create 2.3233 ms/op 2.4507 ms/op 0.95
Array 250000 clone - spread 996.42 us/op 1.1835 ms/op 0.84
Array 250000 get(125000) 0.49400 ns/op 0.55700 ns/op 0.89
Array 250000 set(125000) 0.56600 ns/op 0.64400 ns/op 0.88
Array 250000 iterate all - loop 77.548 us/op 85.988 us/op 0.90
effectiveBalanceIncrements clone Uint8Array 300000 22.143 us/op 24.779 us/op 0.89
effectiveBalanceIncrements clone MutableVector 300000 302.00 ns/op 377.00 ns/op 0.80
effectiveBalanceIncrements rw all Uint8Array 300000 159.74 us/op 163.21 us/op 0.98
effectiveBalanceIncrements rw all MutableVector 300000 74.242 ms/op 81.527 ms/op 0.91
phase0 afterProcessEpoch - 250000 vs - 7PWei 112.24 ms/op 110.29 ms/op 1.02
phase0 beforeProcessEpoch - 250000 vs - 7PWei 41.830 ms/op 32.604 ms/op 1.28
altair processEpoch - mainnet_e81889 295.42 ms/op 323.13 ms/op 0.91
mainnet_e81889 - altair beforeProcessEpoch 60.532 ms/op 47.797 ms/op 1.27
mainnet_e81889 - altair processJustificationAndFinalization 17.448 us/op 17.369 us/op 1.00
mainnet_e81889 - altair processInactivityUpdates 5.3050 ms/op 5.7761 ms/op 0.92
mainnet_e81889 - altair processRewardsAndPenalties 67.877 ms/op 68.232 ms/op 0.99
mainnet_e81889 - altair processRegistryUpdates 2.8340 us/op 2.7730 us/op 1.02
mainnet_e81889 - altair processSlashings 590.00 ns/op 476.00 ns/op 1.24
mainnet_e81889 - altair processEth1DataReset 661.00 ns/op 461.00 ns/op 1.43
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2295 ms/op 1.3322 ms/op 0.92
mainnet_e81889 - altair processSlashingsReset 4.2800 us/op 3.8350 us/op 1.12
mainnet_e81889 - altair processRandaoMixesReset 7.2810 us/op 4.1870 us/op 1.74
mainnet_e81889 - altair processHistoricalRootsUpdate 1.0030 us/op 688.00 ns/op 1.46
mainnet_e81889 - altair processParticipationFlagUpdates 2.7720 us/op 2.5200 us/op 1.10
mainnet_e81889 - altair processSyncCommitteeUpdates 589.00 ns/op 434.00 ns/op 1.36
mainnet_e81889 - altair afterProcessEpoch 124.57 ms/op 120.78 ms/op 1.03
phase0 processEpoch - mainnet_e58758 354.56 ms/op 316.36 ms/op 1.12
mainnet_e58758 - phase0 beforeProcessEpoch 134.96 ms/op 129.03 ms/op 1.05
mainnet_e58758 - phase0 processJustificationAndFinalization 14.933 us/op 17.443 us/op 0.86
mainnet_e58758 - phase0 processRewardsAndPenalties 64.333 ms/op 58.502 ms/op 1.10
mainnet_e58758 - phase0 processRegistryUpdates 8.9590 us/op 7.6850 us/op 1.17
mainnet_e58758 - phase0 processSlashings 521.00 ns/op 491.00 ns/op 1.06
mainnet_e58758 - phase0 processEth1DataReset 510.00 ns/op 456.00 ns/op 1.12
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 969.86 us/op 1.0117 ms/op 0.96
mainnet_e58758 - phase0 processSlashingsReset 4.6220 us/op 3.2410 us/op 1.43
mainnet_e58758 - phase0 processRandaoMixesReset 4.3450 us/op 4.1860 us/op 1.04
mainnet_e58758 - phase0 processHistoricalRootsUpdate 586.00 ns/op 630.00 ns/op 0.93
mainnet_e58758 - phase0 processParticipationRecordUpdates 4.0650 us/op 3.7680 us/op 1.08
mainnet_e58758 - phase0 afterProcessEpoch 92.781 ms/op 99.369 ms/op 0.93
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.1913 ms/op 1.2767 ms/op 0.93
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4509 ms/op 1.4497 ms/op 1.00
altair processInactivityUpdates - 250000 normalcase 20.605 ms/op 20.598 ms/op 1.00
altair processInactivityUpdates - 250000 worstcase 26.488 ms/op 22.449 ms/op 1.18
phase0 processRegistryUpdates - 250000 normalcase 6.6960 us/op 6.4250 us/op 1.04
phase0 processRegistryUpdates - 250000 badcase_full_deposits 230.76 us/op 231.32 us/op 1.00
phase0 processRegistryUpdates - 250000 worstcase 0.5 129.20 ms/op 111.66 ms/op 1.16
altair processRewardsAndPenalties - 250000 normalcase 66.213 ms/op 56.246 ms/op 1.18
altair processRewardsAndPenalties - 250000 worstcase 68.909 ms/op 57.838 ms/op 1.19
phase0 getAttestationDeltas - 250000 normalcase 6.1442 ms/op 6.3454 ms/op 0.97
phase0 getAttestationDeltas - 250000 worstcase 6.3822 ms/op 6.6512 ms/op 0.96
phase0 processSlashings - 250000 worstcase 3.5833 ms/op 3.4413 ms/op 1.04
altair processSyncCommitteeUpdates - 250000 172.08 ms/op 166.65 ms/op 1.03
BeaconState.hashTreeRoot - No change 249.00 ns/op 255.00 ns/op 0.98
BeaconState.hashTreeRoot - 1 full validator 50.457 us/op 49.504 us/op 1.02
BeaconState.hashTreeRoot - 32 full validator 481.14 us/op 524.41 us/op 0.92
BeaconState.hashTreeRoot - 512 full validator 5.3590 ms/op 5.1842 ms/op 1.03
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 60.892 us/op 62.700 us/op 0.97
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 855.52 us/op 852.20 us/op 1.00
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 11.299 ms/op 11.058 ms/op 1.02
BeaconState.hashTreeRoot - 1 balances 50.251 us/op 49.015 us/op 1.03
BeaconState.hashTreeRoot - 32 balances 442.26 us/op 444.69 us/op 0.99
BeaconState.hashTreeRoot - 512 balances 4.2346 ms/op 4.2434 ms/op 1.00
BeaconState.hashTreeRoot - 250000 balances 74.429 ms/op 74.369 ms/op 1.00
aggregationBits - 2048 els - zipIndexesInBitList 14.409 us/op 16.757 us/op 0.86
regular array get 100000 times 31.028 us/op 43.137 us/op 0.72
wrappedArray get 100000 times 31.036 us/op 32.765 us/op 0.95
arrayWithProxy get 100000 times 15.276 ms/op 16.188 ms/op 0.94
ssz.Root.equals 513.00 ns/op 537.00 ns/op 0.96
byteArrayEquals 516.00 ns/op 534.00 ns/op 0.97
shuffle list - 16384 els 6.5452 ms/op 6.6648 ms/op 0.98
shuffle list - 250000 els 96.038 ms/op 97.569 ms/op 0.98
processSlot - 1 slots 8.2240 us/op 8.4890 us/op 0.97
processSlot - 32 slots 1.2773 ms/op 1.2981 ms/op 0.98
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 184.54 us/op 186.33 us/op 0.99
getCommitteeAssignments - req 1 vs - 250000 vc 2.7976 ms/op 2.7886 ms/op 1.00
getCommitteeAssignments - req 100 vs - 250000 vc 3.9670 ms/op 3.9871 ms/op 0.99
getCommitteeAssignments - req 1000 vs - 250000 vc 4.3020 ms/op 4.2710 ms/op 1.01
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.6000 ns/op 4.1600 ns/op 1.11
state getBlockRootAtSlot - 250000 vs - 7PWei 929.89 ns/op 708.41 ns/op 1.31
computeProposers - vc 250000 10.506 ms/op 10.546 ms/op 1.00
computeEpochShuffling - vc 250000 100.70 ms/op 101.45 ms/op 0.99
getNextSyncCommittee - vc 250000 170.61 ms/op 166.89 ms/op 1.02

by benchmarkbot/action

@g11tech
Copy link
Contributor Author

g11tech commented Mar 1, 2023

Can this be done with somehting like:

function allSettledWithTimeout<T>(promises: Promise<T>, timeoutMs: number) {
  const timerPromise = new Promise((_resolve, reject) => setTimeout(reject, timeoutMs));
  return Promise.allSettled(promises.map((promise) => Promise.race(promise, timerPromise)));
}

problem with this is ( i think) it will force timeout both execution/blinded block at 3 sec for e.g. , but we want to resolve for whoever completes first post cutoff if all promises exceeds cutoff

let me see how i can code it up in a cleaner manner

packages/utils/src/promise.ts Outdated Show resolved Hide resolved
@@ -1,7 +1,7 @@
{
"compilerOptions": {
"target": "es2019",
"lib": ["es2020", "esnext.bigint", "es2020.string", "es2020.symbol.wellknown", "dom"],
"lib": ["es2020", "esnext.bigint", "es2020.string", "es2020.symbol.wellknown", "dom", "es2021.promise"],
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

required for Promise.any

@g11tech
Copy link
Contributor Author

g11tech commented Mar 2, 2023

@wemeetagain i have updated the racefn using the inbuild primitives, let me know how it looks now

@twoeths twoeths added this to the v1.6.0 milestone Mar 3, 2023
@@ -11,13 +11,20 @@ import {ValidatorStore, BuilderSelection} from "./validatorStore.js";
import {BlockDutiesService, GENESIS_SLOT} from "./blockDuties.js";

const ETH_TO_WEI = BigInt("1000000000000000000");
const BLOCK_PRODUCTION_RACE_CUTOFF_MS = 3_000;
const BLOCK_PRODUCTION_RACE_TIMEOUT_MS = 12_000;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these values optimal (or good enough to being with)?

Copy link
Member

@nflaig nflaig Mar 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you know how other CL clients handle this and what cutoff they use?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these values make sense to me

@g11tech g11tech marked this pull request as ready for review March 3, 2023 13:13
Copy link
Member

@nflaig nflaig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks really clean and well tested!

precutoff = "precutoff-return",
/** cutoff reached as some were pending till cutoff **/
cutoff = "cutoff-reached",
/** atleast one resolved till cutoff so no race required */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/** atleast one resolved till cutoff so no race required */
/** at least one resolved till cutoff so no race required */

return mapStatusesToResponses(promisesStates);
}

// Post deadline resolve with any of the promise or all rejected before timeout
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deadline is the cutoff here, right?

Suggested change
// Post deadline resolve with any of the promise or all rejected before timeout
// Post cutoff resolve with any of the promise or all rejected before timeout

// or rejected (-ve number)
// Third item in testcase row is the expected value or error message in string
// Last item is expected events
const testcases: [string, number[], (string | Error)[], RaceEvent[]][] = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for a bunch of test cases 👍

@g11tech g11tech merged commit fa13bdb into unstable Mar 6, 2023
@g11tech g11tech deleted the g11tech/build-block-race branch March 6, 2023 07:21
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.6.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Vc keeps waiting for both builder and engine blocks
4 participants