Skip to content

Commit

Permalink
Fix eth_getLogs issue (#1351)
Browse files Browse the repository at this point in the history
Fix some issues which caused some log to be missing when calling the eth_getLogs method
- Setting up a cache version
- Add a check integrity of the cache
- Fix a lock issue

Signed-off-by: Karim TAAM <[email protected]>
  • Loading branch information
matkt authored Sep 10, 2020
1 parent c67870a commit 65bb27c
Show file tree
Hide file tree
Showing 16 changed files with 411 additions and 86 deletions.
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@
### Additions and Improvements

### Bug Fixes
* Fix logs queries missing results against chain head [\#1351](https://github.com/hyperledger/besu/pull/1351)

#### Previously identified known issues

- [Logs queries missing results against chain head](KNOWN_ISSUES.md#Logs-queries-missing-results-against-chain-head)
- [Eth/65 loses peers](KNOWN_ISSUES.md#eth65-loses-peers)
- [Fast sync when running Besu on cloud providers](KNOWN_ISSUES.md#fast-sync-when-running-besu-on-cloud-providers)
- [Privacy users with private transactions created using v1.3.4 or earlier](KNOWN_ISSUES.md#privacy-users-with-private-transactions-created-using-v134-or-earlier)
Expand Down
8 changes: 0 additions & 8 deletions KNOWN_ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,6 @@ in the current release are provided in the [Changelog](CHANGELOG.md).

Known issues are open issues categorized as [Very High or High impact](https://wiki.hyperledger.org/display/BESU/Defect+Prioritisation+Policy).

## Logs queries missing results against chain head

When using `eth_getLogs` against the head of Goerli to retrieve Eth2 deposit log events, [some results seem to be missing](https://github.com/hyperledger/besu/issues/1153).

Workaround -> Use `eth_getLogs` against historical blocks rather than the chain head directly.

A fix for this issue is actively being worked on.

## Eth/65 loses peers

From v1.4.4, `eth/65` is [disabled by default](https://github.com/hyperledger/besu/pull/741).
Expand Down
4 changes: 2 additions & 2 deletions besu/src/main/java/org/hyperledger/besu/Runner.java
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@
import org.hyperledger.besu.ethereum.api.graphql.GraphQLHttpService;
import org.hyperledger.besu.ethereum.api.jsonrpc.JsonRpcHttpService;
import org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketService;
import org.hyperledger.besu.ethereum.api.query.AutoTransactionLogBloomCachingService;
import org.hyperledger.besu.ethereum.api.query.TransactionLogBloomCacher;
import org.hyperledger.besu.ethereum.api.query.cache.AutoTransactionLogBloomCachingService;
import org.hyperledger.besu.ethereum.api.query.cache.TransactionLogBloomCacher;
import org.hyperledger.besu.ethereum.chain.Blockchain;
import org.hyperledger.besu.ethereum.p2p.network.NetworkRunner;
import org.hyperledger.besu.ethereum.p2p.peers.EnodeURL;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@
import static com.google.common.base.Preconditions.checkNotNull;
import static com.google.common.base.Preconditions.checkState;
import static org.hyperledger.besu.cli.DefaultCommandValues.MANDATORY_LONG_FORMAT_HELP;
import static org.hyperledger.besu.ethereum.api.query.TransactionLogBloomCacher.BLOCKS_PER_BLOOM_CACHE;
import static org.hyperledger.besu.ethereum.api.query.cache.TransactionLogBloomCacher.BLOCKS_PER_BLOOM_CACHE;

import org.hyperledger.besu.controller.BesuController;
import org.hyperledger.besu.ethereum.api.query.TransactionLogBloomCacher;
import org.hyperledger.besu.ethereum.api.query.cache.TransactionLogBloomCacher;
import org.hyperledger.besu.ethereum.chain.MutableBlockchain;
import org.hyperledger.besu.ethereum.eth.manager.EthScheduler;
import org.hyperledger.besu.metrics.noop.NoOpMetricsSystem;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcResponse;
import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcSuccessResponse;
import org.hyperledger.besu.ethereum.api.query.BlockchainQueries;
import org.hyperledger.besu.ethereum.api.query.TransactionLogBloomCacher;
import org.hyperledger.besu.ethereum.api.query.cache.TransactionLogBloomCacher;
import org.hyperledger.besu.ethereum.core.BlockHeader;

import java.util.Map;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcResponse;
import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcSuccessResponse;
import org.hyperledger.besu.ethereum.api.query.BlockchainQueries;
import org.hyperledger.besu.ethereum.api.query.TransactionLogBloomCacher;
import org.hyperledger.besu.ethereum.api.query.cache.TransactionLogBloomCacher;

import java.util.Map;
import java.util.Optional;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,10 @@
package org.hyperledger.besu.ethereum.api.query;

import static com.google.common.base.Preconditions.checkArgument;
import static org.hyperledger.besu.ethereum.api.query.TransactionLogBloomCacher.BLOCKS_PER_BLOOM_CACHE;
import static org.hyperledger.besu.ethereum.api.query.cache.TransactionLogBloomCacher.BLOCKS_PER_BLOOM_CACHE;

import org.hyperledger.besu.ethereum.api.handlers.RpcMethodTimeoutException;
import org.hyperledger.besu.ethereum.api.query.cache.TransactionLogBloomCacher;
import org.hyperledger.besu.ethereum.chain.Blockchain;
import org.hyperledger.besu.ethereum.chain.TransactionLocation;
import org.hyperledger.besu.ethereum.core.Account;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,20 @@
*
* SPDX-License-Identifier: Apache-2.0
*/
package org.hyperledger.besu.ethereum.api.query;
package org.hyperledger.besu.ethereum.api.query.cache;

import static org.hyperledger.besu.ethereum.api.query.cache.LogBloomCacheMetadata.DEFAULT_VERSION;

import org.hyperledger.besu.ethereum.chain.Blockchain;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.Optional;
import java.util.OptionalLong;
import java.util.stream.Stream;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
Expand All @@ -31,7 +35,6 @@ public class AutoTransactionLogBloomCachingService {
private final Blockchain blockchain;
private final TransactionLogBloomCacher transactionLogBloomCacher;
private OptionalLong blockAddedSubscriptionId = OptionalLong.empty();
private OptionalLong chainReorgSubscriptionId = OptionalLong.empty();

public AutoTransactionLogBloomCachingService(
final Blockchain blockchain, final TransactionLogBloomCacher transactionLogBloomCacher) {
Expand All @@ -46,22 +49,26 @@ public void start() {
if (!cacheDir.toFile().exists() || !cacheDir.toFile().isDirectory()) {
Files.createDirectory(cacheDir);
}
final LogBloomCacheMetadata logBloomCacheMetadata =
LogBloomCacheMetadata.lookUpFrom(cacheDir);
if (logBloomCacheMetadata.getVersion() == 0) {
try (Stream<Path> walk = Files.walk(cacheDir)) {
walk.filter(Files::isRegularFile).map(Path::toFile).forEach(File::delete);
} catch (Exception e) {
LOG.error("Failed to update cache {}", e.getMessage());
}
new LogBloomCacheMetadata(DEFAULT_VERSION).writeToDirectory(cacheDir);
}

blockAddedSubscriptionId =
OptionalLong.of(
blockchain.observeBlockAdded(
event -> {
if (event.isNewCanonicalHead()) {
transactionLogBloomCacher.cacheLogsBloomForBlockHeader(
event.getBlock().getHeader(), Optional.empty(), true);
event.getBlock().getHeader(), Optional.empty());
}
}));
chainReorgSubscriptionId =
OptionalLong.of(
blockchain.observeChainReorg(
(blockWithReceipts, __) ->
transactionLogBloomCacher.cacheLogsBloomForBlockHeader(
blockWithReceipts.getHeader(), Optional.empty(), true)));

transactionLogBloomCacher
.getScheduler()
.scheduleFutureTask(transactionLogBloomCacher::cacheAll, Duration.ofMinutes(1));
Expand All @@ -73,6 +80,5 @@ public void start() {
public void stop() {
LOG.info("Shutting down Auto transaction logs caching service.");
blockAddedSubscriptionId.ifPresent(blockchain::removeObserver);
chainReorgSubscriptionId.ifPresent(blockchain::removeChainReorgObserver);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
/*
* Copyright ConsenSys AG.
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
* SPDX-License-Identifier: Apache-2.0
*/
package org.hyperledger.besu.ethereum.api.query.cache;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.nio.file.Path;

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class LogBloomCacheMetadata {
private static final Logger LOG = LogManager.getLogger();

public static final int DEFAULT_VERSION = 1;

private static final String METADATA_FILENAME = "CACHE_METADATA.json";
private static final ObjectMapper MAPPER = new ObjectMapper();
private final int version;

@JsonCreator
public LogBloomCacheMetadata(@JsonProperty("version") final int version) {
this.version = version;
}

public int getVersion() {
return version;
}

public static LogBloomCacheMetadata lookUpFrom(final Path dataDir) throws IOException {
LOG.info("Lookup cache metadata file in data directory: {}", dataDir.toString());
return resolveDatabaseMetadata(getDefaultMetadataFile(dataDir));
}

public void writeToDirectory(final Path dataDir) throws IOException {
MAPPER.writeValue(getDefaultMetadataFile(dataDir), this);
}

private static File getDefaultMetadataFile(final Path dataDir) {
return dataDir.resolve(METADATA_FILENAME).toFile();
}

private static LogBloomCacheMetadata resolveDatabaseMetadata(final File metadataFile)
throws IOException {
LogBloomCacheMetadata databaseMetadata;
try {
databaseMetadata = MAPPER.readValue(metadataFile, LogBloomCacheMetadata.class);
} catch (FileNotFoundException fnfe) {
databaseMetadata = new LogBloomCacheMetadata(0);
} catch (JsonProcessingException jpe) {
throw new IllegalStateException(
String.format("Invalid metadata file %s", metadataFile.getAbsolutePath()), jpe);
}
return databaseMetadata;
}
}
Loading

0 comments on commit 65bb27c

Please sign in to comment.