Provide ASIC resistant PoW scheme for BSQ swaps

[stejbac]

This PR adds an implementation of the Equihash memory-hard Proof of Work scheme for DoS/spam attack protection, to replace the present simple Hashcash scheme. It also adds a simple versioning system to make it easier to migrate from one algorithm to another in case of failure to provide adequate DoS attach protection, that is, if an attacker spamming the BSQ swap offer book has too much of a speed/efficiency advantage over an honest user, e.g. due to use of FPGAs, GPUs, etc.

The Equihash scheme is asymmetric, meaning that producing a PoW by solving a hashing puzzle is slow and consumes up to MBs of memory, but verification is instant. The puzzle solutions are larger than that of the present Hashcash scheme (where it is just a 64-bit counter/nonce) but still fairly small - 10's of bytes to a few kB in size, depending on the puzzle parameters. So they shouldn't bloat the BSQ swap offer payloads significantly.

--

Note that one of the commits (5da8df2) touches the DAO packages (VoteResultService), although the code change is trivial.

Also, the last commit (e0595aa) makes a breaking change the proto serialisation, by changing the Filter/PoW difficulty field to a linear-scale floating point -- that commit isn't essential to the PR, though it would cause less disruption to include it before the 1.8.0 release.

[chimp1984]

@stejbac
I think the predicates in FilterManager and HashCashService are just complicating and bloating the APIs. The intention was that they might be useful for testing but at the end they are unnecessary. I would prefer to inline them.
I created 2 commits which I will post here... Feel free to add them if you agree to the change. I think its easier that way as that I make a PR to you repo (can do that as well if you prefer).

[chimp1984]

Index: common/src/main/java/bisq/common/crypto/ProofOfWorkService.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/common/src/main/java/bisq/common/crypto/ProofOfWorkService.java b/common/src/main/java/bisq/common/crypto/ProofOfWorkService.java
--- a/common/src/main/java/bisq/common/crypto/ProofOfWorkService.java	(revision e0595aa284ce18ab7602f1126d23014da336ef69)
+++ b/common/src/main/java/bisq/common/crypto/ProofOfWorkService.java	(revision e0e70bd804fec36844987bc1a29051a24e3cbdc4)
@@ -19,9 +19,9 @@
 
 import com.google.common.base.Preconditions;
 
+import java.util.Arrays;
 import java.util.Optional;
 import java.util.concurrent.CompletableFuture;
-import java.util.function.BiPredicate;
 
 import lombok.Getter;
 
@@ -58,15 +58,13 @@
     public boolean verify(ProofOfWork proofOfWork,
                           String itemId,
                           String ownerId,
-                          double controlDifficulty,
-                          BiPredicate<byte[], byte[]> challengeValidation,
-                          BiPredicate<Double, Double> difficultyValidation) {
+                          double controlDifficulty) {
 
         Preconditions.checkArgument(proofOfWork.getVersion() == version);
 
         byte[] controlChallenge = getChallenge(itemId, ownerId);
-        return challengeValidation.test(proofOfWork.getChallenge(), controlChallenge) &&
-                difficultyValidation.test(proofOfWork.getDifficulty(), controlDifficulty) &&
+        return Arrays.equals(proofOfWork.getChallenge(), controlChallenge) &&
+                proofOfWork.getDifficulty() >= controlDifficulty &&
                 verify(proofOfWork);
     }
 }
Index: core/src/main/java/bisq/core/filter/FilterManager.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/core/src/main/java/bisq/core/filter/FilterManager.java b/core/src/main/java/bisq/core/filter/FilterManager.java
--- a/core/src/main/java/bisq/core/filter/FilterManager.java	(revision e0595aa284ce18ab7602f1126d23014da336ef69)
+++ b/core/src/main/java/bisq/core/filter/FilterManager.java	(revision e0e70bd804fec36844987bc1a29051a24e3cbdc4)
@@ -37,9 +37,9 @@
 import bisq.common.app.Version;
 import bisq.common.config.Config;
 import bisq.common.config.ConfigFileEditor;
-import bisq.common.crypto.ProofOfWorkService;
 import bisq.common.crypto.KeyRing;
 import bisq.common.crypto.ProofOfWork;
+import bisq.common.crypto.ProofOfWorkService;
 
 import org.bitcoinj.core.ECKey;
 import org.bitcoinj.core.Sha256Hash;
@@ -58,7 +58,6 @@
 
 import java.math.BigInteger;
 
-import java.util.Arrays;
 import java.util.Collection;
 import java.util.Collections;
 import java.util.Date;
@@ -66,7 +65,6 @@
 import java.util.List;
 import java.util.Set;
 import java.util.concurrent.CopyOnWriteArrayList;
-import java.util.function.BiPredicate;
 import java.util.function.Consumer;
 
 import java.lang.reflect.Method;
@@ -88,12 +86,6 @@
     private static final String BANNED_SEED_NODES = "bannedSeedNodes";
     private static final String BANNED_BTC_NODES = "bannedBtcNodes";
 
-    private final BiPredicate<byte[], byte[]> challengeValidation = Arrays::equals;
-    // We only require a new pow if difficulty has increased
-    private final BiPredicate<Double, Double> difficultyValidation =
-            (value, controlValue) -> value - controlValue >= 0;
-
-
     ///////////////////////////////////////////////////////////////////////////////////////////
     // Listener
     ///////////////////////////////////////////////////////////////////////////////////////////
@@ -501,9 +493,7 @@
         }
         return service.get().verify(offer.getBsqSwapOfferPayload().get().getProofOfWork(),
                 offer.getId(), offer.getOwnerNodeAddress().toString(),
-                filter.getPowDifficulty(),
-                challengeValidation,
-                difficultyValidation);
+                filter.getPowDifficulty());
     }
 
     public List<Integer> getEnabledPowVersions() {

[chimp1984]

Index: common/src/main/java/bisq/common/crypto/HashCashService.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/common/src/main/java/bisq/common/crypto/HashCashService.java b/common/src/main/java/bisq/common/crypto/HashCashService.java
--- a/common/src/main/java/bisq/common/crypto/HashCashService.java	(revision e0e70bd804fec36844987bc1a29051a24e3cbdc4)
+++ b/common/src/main/java/bisq/common/crypto/HashCashService.java	(revision d54f1aafc04601ec350b083d06b24b4774a08d6b)
@@ -22,24 +22,18 @@
 
 import java.nio.charset.StandardCharsets;
 
-import java.util.Arrays;
 import java.util.concurrent.CompletableFuture;
-import java.util.function.BiPredicate;
 
 import lombok.extern.slf4j.Slf4j;
 
 /**
  * HashCash implementation for proof of work
- * It doubles required work by difficulty increase (adding one leading zero).
+ * It doubles required work by log2Difficulty increase (adding one leading zero).
  *
  * See https://www.hashcash.org/papers/hashcash.pdf
  */
 @Slf4j
 public class HashCashService extends ProofOfWorkService {
-    // Default validations. Custom implementations might use tolerance.
-    private static final BiPredicate<byte[], byte[]> isChallengeValid = Arrays::equals;
-    private static final BiPredicate<Integer, Integer> isDifficultyValid = Integer::equals;
-
     HashCashService() {
         super(0);
     }
@@ -50,105 +44,35 @@
         return mint(payload, challenge, difficulty);
     }
 
-    @Override
-    public byte[] getChallenge(String itemId, String ownerId) {
-        return getBytes(itemId + ownerId);
-    }
-
-    static CompletableFuture<ProofOfWork> mint(byte[] payload,
+    public CompletableFuture<ProofOfWork> mint(byte[] payload,
                                                byte[] challenge,
                                                double difficulty) {
-        return HashCashService.mint(payload,
-                challenge,
-                difficulty,
-                HashCashService::testDifficulty);
-    }
-
-    @Override
-    boolean verify(ProofOfWork proofOfWork) {
-        return verify(proofOfWork,
-                proofOfWork.getChallenge(),
-                toNumLeadingZeros(proofOfWork.getDifficulty()));
-    }
-
-    static boolean verify(ProofOfWork proofOfWork,
-                          byte[] controlChallenge,
-                          int controlLog2Difficulty) {
-        return HashCashService.verify(proofOfWork,
-                controlChallenge,
-                controlLog2Difficulty,
-                HashCashService::testDifficulty);
-    }
-
-    static boolean verify(ProofOfWork proofOfWork,
-                          byte[] controlChallenge,
-                          int controlLog2Difficulty,
-                          BiPredicate<byte[], byte[]> challengeValidation,
-                          BiPredicate<Integer, Integer> difficultyValidation) {
-        return HashCashService.verify(proofOfWork,
-                controlChallenge,
-                controlLog2Difficulty,
-                challengeValidation,
-                difficultyValidation,
-                HashCashService::testDifficulty);
-    }
-
-    private static boolean testDifficulty(byte[] result, int log2Difficulty) {
-        return HashCashService.numberOfLeadingZeros(result) > log2Difficulty;
-    }
-
-
-    ///////////////////////////////////////////////////////////////////////////////////////////
-    // Generic
-    ///////////////////////////////////////////////////////////////////////////////////////////
-
-    static CompletableFuture<ProofOfWork> mint(byte[] payload,
-                                               byte[] challenge,
-                                               double difficulty,
-                                               BiPredicate<byte[], Integer> testDifficulty) {
         return CompletableFuture.supplyAsync(() -> {
             long ts = System.currentTimeMillis();
             int log2Difficulty = toNumLeadingZeros(difficulty);
-            byte[] result;
+            byte[] hash;
             long counter = 0;
             do {
-                result = toSha256Hash(payload, challenge, ++counter);
+                hash = toSha256Hash(payload, challenge, ++counter);
             }
-            while (!testDifficulty.test(result, log2Difficulty));
+            while (numberOfLeadingZeros(hash) <= log2Difficulty);
             ProofOfWork proofOfWork = new ProofOfWork(payload, counter, challenge, difficulty, System.currentTimeMillis() - ts, 0);
             log.info("Completed minting proofOfWork: {}", proofOfWork);
             return proofOfWork;
         });
     }
 
-    static boolean verify(ProofOfWork proofOfWork,
-                          byte[] controlChallenge,
-                          int controlLog2Difficulty,
-                          BiPredicate<byte[], Integer> testDifficulty) {
-        return verify(proofOfWork,
-                controlChallenge,
-                controlLog2Difficulty,
-                HashCashService.isChallengeValid,
-                HashCashService.isDifficultyValid,
-                testDifficulty);
-    }
-
-    static boolean verify(ProofOfWork proofOfWork,
-                          byte[] controlChallenge,
-                          int controlLog2Difficulty,
-                          BiPredicate<byte[], byte[]> challengeValidation,
-                          BiPredicate<Integer, Integer> difficultyValidation,
-                          BiPredicate<byte[], Integer> testDifficulty) {
-        return challengeValidation.test(proofOfWork.getChallenge(), controlChallenge) &&
-                difficultyValidation.test(toNumLeadingZeros(proofOfWork.getDifficulty()), controlLog2Difficulty) &&
-                verify(proofOfWork, testDifficulty);
-    }
-
-    private static boolean verify(ProofOfWork proofOfWork, BiPredicate<byte[], Integer> testDifficulty) {
-        byte[] hash = HashCashService.toSha256Hash(proofOfWork.getPayload(),
+    @Override
+    boolean verify(ProofOfWork proofOfWork) {
+        byte[] hash = toSha256Hash(proofOfWork.getPayload(),
                 proofOfWork.getChallenge(),
                 proofOfWork.getCounter());
-        return testDifficulty.test(hash, toNumLeadingZeros(proofOfWork.getDifficulty()));
+        return numberOfLeadingZeros(hash) > toNumLeadingZeros(proofOfWork.getDifficulty());
+    }
+
+    @Override
+    public byte[] getChallenge(String itemId, String ownerId) {
+        return getBytes(itemId + ownerId);
     }
 
 
Index: common/src/test/java/bisq/common/crypto/HashCashServiceTest.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/common/src/test/java/bisq/common/crypto/HashCashServiceTest.java b/common/src/test/java/bisq/common/crypto/HashCashServiceTest.java
--- a/common/src/test/java/bisq/common/crypto/HashCashServiceTest.java	(revision e0e70bd804fec36844987bc1a29051a24e3cbdc4)
+++ b/common/src/test/java/bisq/common/crypto/HashCashServiceTest.java	(revision d54f1aafc04601ec350b083d06b24b4774a08d6b)
@@ -77,7 +77,7 @@
         List<ProofOfWork> tokens = new ArrayList<>();
         for (int i = 0; i < numTokens; i++) {
             byte[] challenge = UUID.randomUUID().toString().getBytes(StandardCharsets.UTF_8);
-            tokens.add(HashCashService.mint(payload, challenge, difficulty).get());
+            tokens.add(new HashCashService().mint(payload, challenge, difficulty).get());
         }
         double size = tokens.size();
         long ts2 = System.currentTimeMillis();

[chimp1984]

@stejbac
As far as I understand (and tested) the behaviour in case of 2 pow version (e.g. [0,1] or [1,0] set as enabled versions) is that the first is the preferred one and both are valid. So if I had an offer with hashcash pow and we add Equihash as first item I dont need to recreate the pow. But for a new offer Equihash will be used. If we switch from [0] to [1] then all my offers need to recreate the pow.
If difficulty from filter decreases there is no need to recreate the pow of an existing offer, only for increases pow need to be redone.

The difficulty value to be set is 2 pow x, e.g. 2 pow 18 = 262144 means 18 leading zeros for hashcash. It is roughly the number if iterations for hashcash. Values in the range of 300 000-800 000 produce on my machine a pow duration of about 0.5-4 sec.

Could we add some documentation to the Filter so that the Filter maintainer finds quickly some guidance when he sets or changes the values?

[stejbac]

@chimp1984 I've just pushed those two patches now (as a single commit).

Yes, the list of enabled version numbers is in order of preference, so that the first is preferred when creating PoWs and they are both allowed when verifying PoWs. Adding Equihash to the list (in any order) won't cause Hashcash PoWs to be recreated, unless Hashcash is also removed from the list.

Generation times for Hashcash & Equihash are intended to be roughly the same for any given difficulty (at least when it's a power-of-two), with a difficulty of 2 pow n meaning the difficulty of finding > n leading zeros for Hashcash (or equivalently finding a hash with exactly n leading zeros), so 2 * (2 pow n) Hashcash iterations on average, i.e. 524288 average iterations for a difficulty of 2 pow 18 = 262144.

I can attempt to update the BSQ Swaps wiki, as information there about the difficulty would be out of date with this PR.

[stejbac]

I just discovered a trivial bug in the iterator returned by Equihash.IntListMultimap::get, which was causing puzzle solutions (and hence Equihash proofs-of-work) to be invalid about 3% of the time. I've committed a fix (0ee6175), pushed above.

[chimp1984]

@stejbac Is there a commit outstanding or is the PR ready for merge?

[chimp1984]

utACK

[ripcurlx]

utACK - based on #5858 (review)