in progress: Experimental Testnet

[kilpatty]

I wanted to open up this issue for some discussion on a community run "experimental testnet" after speaking with @tynes.

There have been a number of improvements in hsd that should be tested in a testnet-like ecosystem, mainly the new PoW scheme.

There are also a number of improvements that are "on-the-way" such as Elligator, adjustments to the Airdrop Claim process, and anything else that we think might help Handshake succeed.

I'm proposing a testnet that is run by the community, but is explicitly tagged as experimental - that is, users who are joining this testnet should know that it can break and might be rolled over. As these above improvements feel ready to be included into the experimental testnet, we can either force all seed node operators to update, or roll the chain. The man purpose being to test these experimental updates without launching a new testnet each time.

Discussion Points:

• Where is this maintained? A branch on HSD or a forked repo.
• How important is HNSD for this process - is HNSD not supporting a new feature a blocker?
• Do we keep the old testnet alive? This might provide a more stable (albeit outdated) environment for those who don't want to have the upkeep of an experimental testnet.
• Starting with or without Elligator
• What's our strategy for marking a feature as ready to be included into the testnet.
• Seed operators

I think we should set a date within the near future to launch this testnet, and then see how it goes. I'm happy to transition HNScan over to it (And keep the old testnet alive as well for the time being).

The holidays are approaching, so I'm going to suggest that we attempt to spin this up before Thanksgiving (And ultimately the next developer call), that being said - it might be in our interest to wait until that call to discuss the above points.

EDIT: Somethings I'd be interested in seeing merged in *before we launch the initial version of this (mainly to prevent the amount of re-rolling).

• chain: disable goosig after 1 year + 1 month #305 (This would be a hard, hard fork if merged in after, so would love to see it pre).
• name claims: allowing soft-fork to increase name reservation period.  #301 (on me to finish this one).

[mslipper]

I'd be in favor of this until there is a new "official" testnet release. It'll make testing Bob and some of our other internal projects much easier. Throwing in my 2c worth for discussion point:

• HSD branch/forked repo: A branch on HSD would be ideal, since such a branch would imply buy-in from the canonical implementation.
• HNSD: AFAIK, there's nothing preventing HNSD from being updated after the launch of the experimental testnet and any desynchronization between HSD and HNSD will have to be addressed prior to mainnet anyway. I don't think such a discrepancy should block an experimental testnet.
• Old testnet: I don't think we should explicitly shepherd people away from the old testnet, but at the same time hash power might naturally shift away from it depending on the experimental testnet's usage. Just leaving it alone and letting nature take its course will probably be sufficient. (@kilpatty this is separate from whatever network HNScan supports - supporting both would be nice but I understand that might be a lot of work.)
• Elligators: I suggest we keep the experimental testnet in line with whatever is in HSD's master branch. We'd add Elligators once they land on master.
• Feature-ready: See above - if it lands on master, it's probably ready.
• Seed operators: Kyokan can help with this. We have existing infrastructure running long-lived public HSD nodes.

[kilpatty]

@mslipper all of your thoughts pretty much align with exactly how I was thinking as well.

I'll work on opening a PR of master -> experiemtal-testnet. Anyone who wants to run a seed node can comment on that PR with the Key@IP of the node, and we start start working on getting that merged in. We'll have to regenerate the testnet genesis block, but I believe the scripts are included so that shouldn't be too much work.

The only other suggestions I have is that we either A. change ports (to make things easier to run both the old testnet and this one), or B. change the location, so that it lives inside of the .hsd folder in testnet-experimental vs testnet. If we do that, it should be a pretty trivial transition for most people in terms of supporting it.

I'm going to spend some time today and tomorrow building out support for multiple networks on hnscan. Will link here when it's done-ish.

[chjj]

@kilpatty, @mslipper, this sounds good. I really hate running seed nodes. :)

I feel like we're probably ready for another testnet: I spent last night getting the new airdrop and faucet trees built (along with some updates to the name list). This was also partially necessary to fix a DoS vector in the goosig RNG (which was present for the entire lifespan of the last testnet). Airdrop recipients are also now limited to 10 keys per account in order to give them plausible deniability with regards to their subtree size (we could opt for a more complicated padding scheme, but it's very tricky). Anyway, airdrops should work on master again.

The two things on my mind right now are schnorr and resource serialization. I think I'll merge schnorr in to master tonight. It'd also be nice to move to some simpler resource serialization, but I don't think it's necessary to have before a new testnet. And since this testnet is experimental, we could probably just roll out a new serialization as it's running.

That brings me to HNSD: I've been hesitant to merge any new resource serialization in order to avoid the pain of implementing it in HNSD. HNSD is very far behind right now, so I don't think we need it to start a new testnet. It can come into play later.

I think #301 and #305 are probably worthwhile. I want to review more of this after I get some sleep (odd schedule right now).

[kilpatty]

Just to bump this, I'm planning to merge in multi-network support in hnscan today (HNScan/HNScan#233). Once I get that merged in, hnscan will be able to support the experimental testnet as well as the old one.

I'll plan to push up a PR to hsd today merging master into a new branch (experimental-testnet), and we can start working towards launching that most likely next week.

Will plan to try to get my airdrop claim PR push up within a day or two as well!

[pinheadmz]

Outcome of developer call today:

 - use simnet
 - use telegram to coordinate
 - expect rapid rollovers and hard forks
 - start without elligator
 - focus on fork of handshake-org/hsd
 - maintain HNSCan Testnet4 node just for PIA/hnsd
 - each of 3 organizations on call will maintain a simnet seed node
 - modify network params to speed up auctions and rollout

[tynes]

Any updates on these efforts?

I haven't seen any activity on https://github.com/Handshake-Protocol/hsd

[pinheadmz]

@tynes I think the simnet work will be based on a branch of hsd, not a new repo.
If @boymanjor can set that up and maybe allow you and/or myself commit access we can get started. Namebase, Purse and Urkel will launch seed nodes, and we need to get key:IP from them.

[tynes]

Pull Requests can be opened against this branch: https://github.com/handshake-org/hsd/tree/experimental-testnet

[chjj]

Just an update: the tree has been rebuilt once more. I think it's the last time we will build the airdrop tree (hopefully). I'm finally happy with the scheme.

The airdrop tree now pads all subtrees to 16 leaves, creating a perfect merkle tree for each user which shouldn't leak any privacy. Each subtree has an HKDF seed associated with it. The same seed is encrypted to each of the user's keys such that they can regenerate the synthetic leaves to verify that we didn't do anything malicious (like inserting our own keys).

The only downside here is that the merkle tree is 4x bigger (~100mb). I hope that's not an issue for recipients, but maybe something we can talk about. We could possibly reduce it to 8 leaves per user, but note that 8 leaves == 4 original user keys.

I also put a lot of work into improving goosig. I think it can be considered consensus-ready now. I'm very confident in it.

I've pushed some code to commit to all of these tree roots in the genesis block. I'm probably going to be breaking master more often now.

[tynes]

The experimental testnet was populated with seeds and should be ready to go.
Some network params were tweaked to speed up the bidding process.

https://github.com/handshake-org/hsd/tree/experimental-testnet

[pinheadmz]

@tynes is it okay to close this issue? The exp-test branch is public and simnet is currently running on its second iteration.

[kilpatty]

It may be worthwhile to keep this open, even if only to provide updates on the current status of the simnet for new users, but otherwise I'm down to close this.

[tynes]

I agree with @kilpatty, we should leave it open so that people don't require tribal knowledge to participate. Updating the name of the issue to remove proposal

[pinheadmz]

I move (again) to close this issue, I think now its time. Simnet is dead long live mainnet!

[pinheadmz]

Closing due to inactivity, destruction of the simnet network and launch of MAINET! 🎉