Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check for snap server #6609

Merged
merged 50 commits into from
Jul 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
81840ff
commit building version
pinges Feb 13, 2024
40d9ea1
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
pinges Feb 13, 2024
7f2fd43
EthPeer add isServingSnap to be able to make sure that we have enough…
pinges Feb 23, 2024
dcbb895
add real check for snap server using a snap task
pinges Feb 26, 2024
5449acc
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
pinges Feb 27, 2024
d3455c9
some cleanup
pinges Feb 29, 2024
f2d7beb
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
pinges Feb 29, 2024
01f9101
add log for snap server check
pinges Feb 29, 2024
5d7f1f1
change the way we handle the max incoming connections
pinges Mar 1, 2024
80534d3
fix unit tests
pinges Mar 5, 2024
edbc9d4
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
pinges Mar 5, 2024
734bb89
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
pinges Mar 6, 2024
ea43fd0
fix acceptance test
pinges Mar 7, 2024
d6a2999
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
pinges Mar 7, 2024
98fbcce
merge main
pinges Mar 26, 2024
fbe6b52
Merge branch 'main' into CheckForSnapServer
macfarla Apr 3, 2024
446ef1b
merge
macfarla Apr 3, 2024
4203a55
merge
macfarla Apr 11, 2024
61d7265
merge
macfarla Apr 12, 2024
de6899f
extra logging from #6924
macfarla Apr 12, 2024
272c1bb
formatting
macfarla Apr 12, 2024
ae030e7
delete version metadata
macfarla Apr 12, 2024
68934cc
reuse ethPeers
macfarla Apr 12, 2024
a95e4f4
make a switching peer task
macfarla Apr 12, 2024
13fb105
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
macfarla Apr 12, 2024
83ef98f
add client info to disconnect message
macfarla Apr 15, 2024
5af6bbb
check for non-deprecated sync modes
macfarla Apr 15, 2024
9bed81f
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
macfarla Apr 15, 2024
4f4e847
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
macfarla Apr 15, 2024
87690b9
logging
macfarla Apr 15, 2024
c3c5bb3
formatting
macfarla Apr 15, 2024
5d57f1d
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
macfarla Apr 15, 2024
14528e8
merge
pinges Jun 13, 2024
b148347
fix after merge
pinges Jun 13, 2024
12c8350
stop prefering snap servers after sync
pinges Jun 18, 2024
1354979
fix compile
pinges Jun 18, 2024
048765f
add tests and some clean-up
pinges Jun 20, 2024
de7d55b
make constant
pinges Jun 20, 2024
a3e17d1
merge and fix conflicts
pinges Jun 20, 2024
0099f11
add newline
pinges Jun 20, 2024
f1f2073
spotless
pinges Jun 20, 2024
13807b1
fix tests, fix trailing peers, and clean up
pinges Jun 25, 2024
9bf96ff
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
pinges Jun 25, 2024
ab4e7d6
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
pinges Jun 26, 2024
6e98ee9
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
pinges Jun 27, 2024
626af31
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
pinges Jun 27, 2024
1a2b324
Merge branch 'main' into CheckForSnapServer
pinges Jul 1, 2024
2a84c29
review change
pinges Jul 3, 2024
dd0b697
Merge branch 'main' of github.com:hyperledger/besu into CheckForSnapS…
pinges Jul 3, 2024
f4d90f0
Merge branch 'CheckForSnapServer' of github.com:pinges/besu into Chec…
pinges Jul 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions besu/src/main/java/org/hyperledger/besu/RunnerBuilder.java
Original file line number Diff line number Diff line change
Expand Up @@ -688,13 +688,14 @@ public Runner build() {
.map(nodePerms -> PeerPermissions.combine(nodePerms, defaultPeerPermissions))
.orElse(defaultPeerPermissions);

final EthPeers ethPeers = besuController.getEthPeers();
macfarla marked this conversation as resolved.
Show resolved Hide resolved

LOG.info("Detecting NAT service.");
final boolean fallbackEnabled = natMethod == NatMethod.AUTO || natMethodFallbackEnabled;
final NatService natService = new NatService(buildNatManager(natMethod), fallbackEnabled);
final NetworkBuilder inactiveNetwork = caps -> new NoopP2PNetwork();
final NetworkBuilder activeNetwork =
caps -> {
final EthPeers ethPeers = besuController.getEthPeers();
return DefaultP2PNetwork.builder()
.vertx(vertx)
.nodeKey(nodeKey)
Expand All @@ -709,8 +710,8 @@ public Runner build() {
.blockchain(context.getBlockchain())
.blockNumberForks(besuController.getGenesisConfigOptions().getForkBlockNumbers())
.timestampForks(besuController.getGenesisConfigOptions().getForkBlockTimestamps())
.allConnectionsSupplier(ethPeers::getAllConnections)
.allActiveConnectionsSupplier(ethPeers::getAllActiveConnections)
.allConnectionsSupplier(ethPeers::streamAllConnections)
.allActiveConnectionsSupplier(ethPeers::streamAllActiveConnections)
.maxPeers(ethPeers.getMaxPeers())
.build();
};
Expand All @@ -721,9 +722,10 @@ public Runner build() {
.subProtocols(subProtocols)
.network(p2pEnabled ? activeNetwork : inactiveNetwork)
.metricsSystem(metricsSystem)
.ethPeersShouldConnect(ethPeers::shouldTryToConnect)
.build();

besuController.getEthPeers().setRlpxAgent(networkRunner.getRlpxAgent());
ethPeers.setRlpxAgent(networkRunner.getRlpxAgent());

final P2PNetwork network = networkRunner.getNetwork();
// ForkId in Ethereum Node Record needs updating when we transition to a new protocol spec
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,6 @@
import org.hyperledger.besu.ethereum.core.Difficulty;
import org.hyperledger.besu.ethereum.core.MiningParameters;
import org.hyperledger.besu.ethereum.core.PrivacyParameters;
import org.hyperledger.besu.ethereum.core.Synchronizer;
import org.hyperledger.besu.ethereum.eth.EthProtocol;
import org.hyperledger.besu.ethereum.eth.EthProtocolConfiguration;
import org.hyperledger.besu.ethereum.eth.SnapProtocol;
Expand Down Expand Up @@ -77,6 +76,7 @@
import org.hyperledger.besu.ethereum.eth.transactions.TransactionPool;
import org.hyperledger.besu.ethereum.eth.transactions.TransactionPoolConfiguration;
import org.hyperledger.besu.ethereum.eth.transactions.TransactionPoolFactory;
import org.hyperledger.besu.ethereum.forkid.ForkIdManager;
import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
import org.hyperledger.besu.ethereum.mainnet.ProtocolSpec;
import org.hyperledger.besu.ethereum.p2p.config.NetworkingConfiguration;
Expand Down Expand Up @@ -604,6 +604,12 @@ public BesuController build() {
final int maxMessageSize = ethereumWireProtocolConfiguration.getMaxMessageSize();
final Supplier<ProtocolSpec> currentProtocolSpecSupplier =
() -> protocolSchedule.getByBlockHeader(blockchain.getChainHeadHeader());
final ForkIdManager forkIdManager =
new ForkIdManager(
blockchain,
genesisConfigOptions.getForkBlockNumbers(),
genesisConfigOptions.getForkBlockTimestamps(),
ethereumWireProtocolConfiguration.isLegacyEth64ForkIdEnabled());
final EthPeers ethPeers =
new EthPeers(
getSupportedProtocol(),
Expand All @@ -615,7 +621,9 @@ public BesuController build() {
nodeKey.getPublicKey().getEncodedBytes(),
maxPeers,
maxRemotelyInitiatedPeers,
randomPeerPriority);
randomPeerPriority,
syncConfig.getSyncMode(),
forkIdManager);

final EthMessages ethMessages = new EthMessages();
final EthMessages snapMessages = new EthMessages();
Expand Down Expand Up @@ -681,13 +689,14 @@ public BesuController build() {
ethMessages,
scheduler,
peerValidators,
Optional.empty());
Optional.empty(),
forkIdManager);

final PivotBlockSelector pivotBlockSelector =
createPivotSelector(
protocolSchedule, protocolContext, ethContext, syncState, metricsSystem, blockchain);

final Synchronizer synchronizer =
final DefaultSynchronizer synchronizer =
createSynchronizer(
protocolSchedule,
worldStateStorageCoordinator,
Expand All @@ -697,6 +706,16 @@ public BesuController build() {
ethProtocolManager,
pivotBlockSelector);

ethPeers.setTrailingPeerRequirementsSupplier(synchronizer::calculateTrailingPeerRequirements);

if (SyncMode.isSnapSync(syncConfig.getSyncMode())
|| SyncMode.isCheckpointSync(syncConfig.getSyncMode())) {
synchronizer.subscribeInSync((b) -> ethPeers.snapServerPeersNeeded(!b));
usmansaleem marked this conversation as resolved.
Show resolved Hide resolved
ethPeers.snapServerPeersNeeded(true);
} else {
ethPeers.snapServerPeersNeeded(false);
}

protocolContext.setSynchronizer(Optional.of(synchronizer));

final Optional<SnapProtocolManager> maybeSnapProtocolManager =
Expand Down Expand Up @@ -809,7 +828,7 @@ private TrieLogPruner createTrieLogPruner(
* @param pivotBlockSelector the pivot block selector
* @return the synchronizer
*/
protected Synchronizer createSynchronizer(
protected DefaultSynchronizer createSynchronizer(
final ProtocolSchedule protocolSchedule,
final WorldStateStorageCoordinator worldStateStorageCoordinator,
final ProtocolContext protocolContext,
Expand Down Expand Up @@ -1000,6 +1019,7 @@ protected String getSupportedProtocol() {
* @param scheduler the scheduler
* @param peerValidators the peer validators
* @param mergePeerFilter the merge peer filter
* @param forkIdManager the fork id manager
* @return the eth protocol manager
*/
protected EthProtocolManager createEthProtocolManager(
Expand All @@ -1012,7 +1032,8 @@ protected EthProtocolManager createEthProtocolManager(
final EthMessages ethMessages,
final EthScheduler scheduler,
final List<PeerValidator> peerValidators,
final Optional<MergePeerFilter> mergePeerFilter) {
final Optional<MergePeerFilter> mergePeerFilter,
final ForkIdManager forkIdManager) {
return new EthProtocolManager(
protocolContext.getBlockchain(),
networkId,
Expand All @@ -1026,8 +1047,7 @@ protected EthProtocolManager createEthProtocolManager(
mergePeerFilter,
synchronizerConfiguration,
scheduler,
genesisConfigOptions.getForkBlockNumbers(),
genesisConfigOptions.getForkBlockTimestamps());
forkIdManager);
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
import org.hyperledger.besu.ethereum.eth.sync.state.SyncState;
import org.hyperledger.besu.ethereum.eth.transactions.TransactionPool;
import org.hyperledger.besu.ethereum.eth.transactions.TransactionPoolConfiguration;
import org.hyperledger.besu.ethereum.forkid.ForkIdManager;
import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
import org.hyperledger.besu.ethereum.p2p.config.SubProtocolConfiguration;
import org.hyperledger.besu.ethereum.storage.StorageProvider;
Expand Down Expand Up @@ -242,7 +243,8 @@ protected EthProtocolManager createEthProtocolManager(
final EthMessages ethMessages,
final EthScheduler scheduler,
final List<PeerValidator> peerValidators,
final Optional<MergePeerFilter> mergePeerFilter) {
final Optional<MergePeerFilter> mergePeerFilter,
final ForkIdManager forkIdManager) {
return besuControllerBuilderSchedule
.get(0L)
.createEthProtocolManager(
Expand All @@ -255,7 +257,8 @@ protected EthProtocolManager createEthProtocolManager(
ethMessages,
scheduler,
peerValidators,
mergePeerFilter);
mergePeerFilter,
forkIdManager);
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@
import org.hyperledger.besu.ethereum.eth.sync.backwardsync.BackwardSyncContext;
import org.hyperledger.besu.ethereum.eth.sync.state.SyncState;
import org.hyperledger.besu.ethereum.eth.transactions.TransactionPool;
import org.hyperledger.besu.ethereum.forkid.ForkIdManager;
import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
import org.hyperledger.besu.ethereum.worldstate.WorldStateArchive;
Expand Down Expand Up @@ -97,7 +98,8 @@ protected EthProtocolManager createEthProtocolManager(
final EthMessages ethMessages,
final EthScheduler scheduler,
final List<PeerValidator> peerValidators,
final Optional<MergePeerFilter> mergePeerFilter) {
final Optional<MergePeerFilter> mergePeerFilter,
final ForkIdManager forkIdManager) {

var mergeContext = protocolContext.getConsensusContext(MergeContext.class);

Expand Down Expand Up @@ -126,7 +128,8 @@ protected EthProtocolManager createEthProtocolManager(
ethMessages,
scheduler,
peerValidators,
filterToUse);
filterToUse,
forkIdManager);

return ethProtocolManager;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@
import org.hyperledger.besu.ethereum.chain.MutableBlockchain;
import org.hyperledger.besu.ethereum.core.MiningParameters;
import org.hyperledger.besu.ethereum.core.PrivacyParameters;
import org.hyperledger.besu.ethereum.core.Synchronizer;
import org.hyperledger.besu.ethereum.eth.EthProtocolConfiguration;
import org.hyperledger.besu.ethereum.eth.manager.EthContext;
import org.hyperledger.besu.ethereum.eth.manager.EthMessages;
Expand All @@ -49,6 +48,7 @@
import org.hyperledger.besu.ethereum.eth.sync.state.SyncState;
import org.hyperledger.besu.ethereum.eth.transactions.TransactionPool;
import org.hyperledger.besu.ethereum.eth.transactions.TransactionPoolConfiguration;
import org.hyperledger.besu.ethereum.forkid.ForkIdManager;
import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
import org.hyperledger.besu.ethereum.storage.StorageProvider;
import org.hyperledger.besu.ethereum.worldstate.DataStorageConfiguration;
Expand Down Expand Up @@ -156,7 +156,8 @@ protected EthProtocolManager createEthProtocolManager(
final EthMessages ethMessages,
final EthScheduler scheduler,
final List<PeerValidator> peerValidators,
final Optional<MergePeerFilter> mergePeerFilter) {
final Optional<MergePeerFilter> mergePeerFilter,
final ForkIdManager forkIdManager) {
return mergeBesuControllerBuilder.createEthProtocolManager(
protocolContext,
synchronizerConfiguration,
Expand All @@ -167,7 +168,8 @@ protected EthProtocolManager createEthProtocolManager(
ethMessages,
scheduler,
peerValidators,
mergePeerFilter);
mergePeerFilter,
forkIdManager);
}

@Override
Expand Down Expand Up @@ -212,7 +214,7 @@ protected PluginServiceFactory createAdditionalPluginServices(
}

@Override
protected Synchronizer createSynchronizer(
protected DefaultSynchronizer createSynchronizer(
final ProtocolSchedule protocolSchedule,
final WorldStateStorageCoordinator worldStateStorageCoordinator,
final ProtocolContext protocolContext,
Expand All @@ -222,15 +224,14 @@ protected Synchronizer createSynchronizer(
final PivotBlockSelector pivotBlockSelector) {

DefaultSynchronizer sync =
(DefaultSynchronizer)
super.createSynchronizer(
protocolSchedule,
worldStateStorageCoordinator,
protocolContext,
ethContext,
syncState,
ethProtocolManager,
pivotBlockSelector);
super.createSynchronizer(
protocolSchedule,
worldStateStorageCoordinator,
protocolContext,
ethContext,
syncState,
ethProtocolManager,
pivotBlockSelector);

if (genesisConfigOptions.getTerminalTotalDifficulty().isPresent()) {
LOG.info(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@
import org.hyperledger.besu.consensus.common.bft.network.PeerConnectionTracker;
import org.hyperledger.besu.datatypes.Address;
import org.hyperledger.besu.ethereum.p2p.network.ProtocolManager;
import org.hyperledger.besu.ethereum.p2p.peers.Peer;
import org.hyperledger.besu.ethereum.p2p.rlpx.connections.PeerConnection;
import org.hyperledger.besu.ethereum.p2p.rlpx.wire.Capability;
import org.hyperledger.besu.ethereum.p2p.rlpx.wire.Message;
Expand Down Expand Up @@ -108,11 +107,6 @@ public void handleNewConnection(final PeerConnection peerConnection) {
peers.add(peerConnection);
}

@Override
public boolean shouldConnect(final Peer peer, final boolean incoming) {
return false; // for now the EthProtocolManager takes care of this
}

@Override
public void handleDisconnect(
final PeerConnection peerConnection,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,11 @@ public ForkIdManager(
checkNotNull(blockchain);
checkNotNull(blockNumberForks);
this.chainHeadSupplier = blockchain::getChainHeadHeader;
this.genesisHash = blockchain.getGenesisBlock().getHash();
try {
this.genesisHash = blockchain.getGenesisBlock().getHash();
} catch (Exception e) {
throw new RuntimeException(e);
}
this.blockNumbersForkIds = new ArrayList<>();
this.timestampsForkIds = new ArrayList<>();
this.legacyEth64 = legacyEth64;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ protected boolean removeEldestEntry(final Map.Entry<Hash, Boolean> eldest) {
private final PeerReputation reputation = new PeerReputation();
private final Map<PeerValidator, Boolean> validationStatus = new ConcurrentHashMap<>();
private final Bytes id;
private boolean isServingSnap = false;

private static final Map<Integer, Integer> roundMessages;

Expand Down Expand Up @@ -393,6 +394,14 @@ public RequestManager.ResponseStream getSnapTrieNode(
requestManagers.get(SnapProtocol.NAME).get(SnapV1.GET_TRIE_NODES), getTrieNodes);
}

public void setIsServingSnap(final boolean isServingSnap) {
this.isServingSnap = isServingSnap;
}

public boolean isServingSnap() {
return isServingSnap;
}

private RequestManager.ResponseStream sendRequest(
final RequestManager requestManager, final MessageData messageData) throws PeerNotConnected {
lastRequestTimestamp = clock.millis();
Expand Down Expand Up @@ -582,9 +591,9 @@ public String getProtocolName() {
}

/**
* Return A read-only snapshot of this peer's current {@code chainState} }
* Return A read-only snapshot of this peer's current {@code chainState}
*
* @return A read-only snapshot of this peer's current {@code chainState} }
* @return A read-only snapshot of this peer's current {@code chainState}
*/
public ChainHeadEstimate chainStateSnapshot() {
return chainHeadState.getSnapshot();
Expand Down Expand Up @@ -629,14 +638,17 @@ public boolean hasSupportForMessage(final int messageCode) {
@Override
public String toString() {
return String.format(
"PeerId: %s %s, validated? %s, disconnected? %s, client: %s, %s, %s",
"PeerId: %s %s, validated? %s, disconnected? %s, client: %s, %s, %s, isServingSnap %s, has height %s, connected for %s ms",
getLoggableId(),
reputation,
isFullyValidated(),
isDisconnected(),
connection.getPeerInfo().getClientId(),
connection,
connection.getPeer().getEnodeURLString());
connection.getPeer().getEnodeURLString(),
isServingSnap,
chainHeadState.getEstimatedHeight(),
System.currentTimeMillis() - connection.getInitiatedAt());
}

@Nonnull
Expand Down
Loading
Loading