spruned 0.0.10

spruned version 0.0.10

Welcome to a new spruned release: moonlight.
No new features on the frontend but definitely a lot of fixes and cleanups under the hood.
Thanks to @dakk for his first-time contributions!

This release stabilizes the new data repositories and in-app flows.
It's warmly advised to upgrade as soon as possible, especially from <= 0.0.7, due to important protocols changes and performances improvements.

Changes:

• use the proper entry point to load the P2P interface before the headers sync is done (fetch synced blocks also during the header sync) #85
• upgrade Electrum protocol methods (headers) and bring back some testnet servers #86
• implement a micro dependencies injection function to fix Database Is Closed Error #91
• fix an issue on DB versioning during migrations with Python 3.6.4
• improve asyncio usage, replace old-fashioned decorators with the latest async\await syntax

Available also on pypi and dockerhub.

spruned 0.0.9

spruned 0.0.9:

• bugfixing on the getblock api

spruned 0.0.8

spruned version 0.0.8

• add database migrations. This is tricky and may fail. In case, delete the storage and sync headers again (I hope not).
• improve the efficiency of SQLite DB (fewer calls, less size) by changing data serialization and queries. reduces the size by roughly a half.
• improve the efficiency on the P2P side when new block headers are notified (should fix race conditions of issue #85 )
• persist and index raw transactions when fetched via getblock (reduce network calls)
• improve the efficiency of the getblock verbose calls by using the per-block transactions index, avoid to load the entire block data
• increase the cache agent speed by using the per-block transactions index instead of the leveldb iterator

There are a lot of changes that may break the code, but they are very needed and useful.

The basic idea is to improve data indexing so that it can also be used in the future for other purposes (layerthree).

The release is available on pypi and docker hub.

spruned 0.0.7.2

Release 0.0.7.2:

• fixed a memory leak in the mempool volatile repository
• deprecated python 3.5.2, min version is now python 3.5.3
• added support for python 3.7.2
• requirements update

Also available on https://cloud.docker.com/u/gdassori/repository/docker/gdassori/spruned

spruned 0.0.7

release 0.0.7 changelog:

• mempool support (see notes)
• quorum based fee estimation improvements (see notes)
• electrum protocol bump to v1.4
• Github anchored docker distribution (https://hub.docker.com/r/gdassori/spruned, see notes)
• jsonrpc floats formatting bugfix (now with fixed precision)
• rpcport bugfix (was 8332 also in testnet unless explicit)

notes

Mempool:

The code was in since a while, zeromq already shared some observers. Looks stable enough to be released.

$ spruned --mempool-size 100

enables the mempool with 100Mbytes buffer.

Fee estimation:

The previous quorum service had its own connectionpool. That was slow. The fee estimation service now uses the same connectionpool as the other API.

Docker:

I have seen many spruned containers on dockerhub, but I'd like to maintain a Github-anchored container on my username (gdassori, the same as Github).

spruned 0.0.6

Welcome to spruned 0.0.6!

This new release comes with a major update: zeromq support.
All the Core 0MQ topics are supported: hashblock, rawblock, hashtx, rawtx.

Summary:

• new checkpoints: up to block 568150 (00000000000000000001ae8ead7f279a3f7038967a147a0fb35acb83ff16fd82)
• new feature: zeromq
• new electrum testnet servers (public servers are unreliable, I'm going to support also the testnet network)
• jsonrpc: gettxout floats was returned as strings
• jsonrpc: getblockchaininfo difficulty&chainwork are mocks but the syntax is now valid (was null values)
• arguments: --daemon is deprecated, the argument is now useless and will be removed in next versions.

spruned 0.0.5

Long time passed since a new release!

Here comes spruned 0.0.5:

• lot of bugfixing all around
• a robusts P2P connectionpool with a 100% getblock hitrate
• an electrum servers quorum-based fee estimation
• proxies support
• tor support
• less memory usage for low end devices

An extended RPC API set:

== Blockchain ==
getbestblockhash
getblock "blockhash" ( verbosity )
getblockchaininfo
getblockcount
getblockhash height
getblockheader "hash" ( verbose )
gettxout "txid" n ( include_mempool )

== Rawtransactions ==
getrawtransaction "txid" ( verbose )
sendrawtransaction "hexstring" ( allowhighfees )

== Util ==
estimatefee nblocks
estimatesmartfee conf_target ("estimate_mode")
uptime

== Network ==
getpeerinfo
getnetworkinfo

== Wallet ==
validateaddress

== Partially emulated for compatibility ==
getchaintxstats
getmininginfo
getnettotals

spruned 0.0.3 beta 5

good news! spruned 0.0.3b5 comes with a lot of bugfixing and improvements.

• bloom filters: the default filter is the genesis address, new txids are no more notified by p2p peers.
• network: disconnect the peers not aligned to the best well known height (from electrum servers)
• estimatefee: a fee group is established gathering data from multiple electrum peers to provide an average median fee and avoid casual changes.
• scoring system: every peer now has a score, lower scores are disconnected, many unwanted behaviors can lower the scores
• caching: smarter cache hooks to reduce network load

spruned 0.0.2 beta 1

spruned v0.0.2b1, release notes:

• performances improvements on non verbose block downloads
• peers discovery and scoring, generally the p2p connection pool is more reliable
• bugfixes
• improved block caching
• loglevels adjustment
• global application speedup

On a raspberry zero the expected behaviour for an healthy connections pool in spruned (may take a bit to establish) on the bitcoin mainnet blockchain, with non verbose blocks, is the following:

2018-09-14 22:08:15,830 - p2p - INFO - Block 00000000000000000023cf82610d8cc7b929b91917653bd5e1bcaafb21d1df27 served in 1.3266s
2018-09-14 22:08:23,721 - p2p - INFO - Block 00000000000000000014916b46b8f5f3e722a5bcbd738adf550a9d765583ea73 served in 4.6273s
2018-09-14 22:08:29,499 - p2p - INFO - Block 0000000000000000002007fe9d65dab839839430852f9e449fd9f82dd2acd191 served in 3.2248s
2018-09-14 22:08:33,518 - p2p - INFO - Block 0000000000000000000173baf61960c70f87b5fea81ee0eb84759123a0e72df1 served in 3.1161s
2018-09-14 22:08:38,170 - p2p - INFO - Block 00000000000000000008c49c77029c568d1f81d065d5158a7f493f60785c2628 served in 1.8292s
2018-09-14 22:08:42,040 - p2p - INFO - Block 000000000000000000024ccfb7e1f25076e9806f0f024384dad89909b8f9c79a served in 1.2097s
2018-09-14 22:08:46,331 - p2p - INFO - Block 00000000000000000008cba9967e020fe782271e073b274f35d74ab056d7ba9e served in 1.3243s
2018-09-14 22:08:55,389 - p2p - INFO - Block 00000000000000000008b108f2df965b85b280b006e368e638f51e81a26c16fe served in 8.2317s
2018-09-14 22:09:00,315 - p2p - INFO - Block 0000000000000000001ce948fee7fd82629a9879172f36e6e75906a1e13a797f served in 3.3176s
2018-09-14 22:09:14,242 - p2p - INFO - Block 0000000000000000001ab81222886b6350bd38427fa112e45f466f13c2fc0868 served in 11.4722s
2018-09-14 22:09:20,615 - p2p - INFO - Block 000000000000000000275bd8ea62d34a88be77ad9fc87becaf5e42d68cb93f44 served in 3.7804s
2018-09-14 22:09:27,853 - p2p - INFO - Block 0000000000000000002575234fd5d0e64f3136279dedfe56133ee91117f2d558 served in 4.5819s
2018-09-14 22:09:32,679 - p2p - INFO - Block 00000000000000000018bf050790126eb480fd3dc4a972c262ead1efb566ca37 served in 0.4589s
2018-09-14 22:09:35,250 - p2p - INFO - Block 0000000000000000000c4e009379dac3e2ca6cf73b65bf444bbeadcf0ab40ba6 served in 0.8339s
2018-09-14 22:09:46,326 - p2p - INFO - Block 000000000000000000242fb80cb0a2a326615eb70d2b6ef5e414f975e9da860a served in 5.2715s
2018-09-14 22:09:50,992 - p2p - INFO - Block 000000000000000000144c8a7971d481aec0a2a919b86bec7b62a694dece599f served in 0.2485s
2018-09-14 22:09:55,140 - p2p - INFO - Block 000000000000000000129581394f545d6e92d02e60b38596a8aef379071b3ce6 served in 3.7976s
2018-09-14 22:09:57,822 - p2p - INFO - Block 00000000000000000025fb7aa34dec9d3b34d6a69f47d6a167142fe45ca7b0cb served in 0.4071s
2018-09-14 22:10:01,797 - p2p - INFO - Block 00000000000000000010a50966d4e94be3303c9be1ec6abaf4db887fce986b64 served in 3.1227s
2018-09-14 22:10:09,170 - p2p - INFO - Block 0000000000000000002074376e4e970004a743ba77996d731dc76475030b23d7 served in 6.0753s
2018-09-14 22:10:13,941 - p2p - INFO - Block 0000000000000000000950a95793a7df49498223d6740eb0602ee33cc1d5f465 served in 1.0358s
2018-09-14 22:10:16,873 - p2p - INFO - Block 00000000000000000016a28cb550844773f4902cf2341c950e7c097dc8864f75 served in 1.7823s
2018-09-14 22:10:20,663 - p2p - INFO - Block 00000000000000000025725c6f369e79102c22785a31e0bc4f9d23c1482f9f28 served in 3.2550s
2018-09-14 22:10:24,418 - p2p - INFO - Block 000000000000000000219684c0fff4354fc809100661529fe785ef3b308e7646 served in 1.2439s
2018-09-14 22:10:25,494 - p2p - INFO - Block 0000000000000000001acf094496f5a272ea3f63a1e5931a7a656bdca09333f7 served in 0.7833s
2018-09-14 22:10:32,608 - p2p - INFO - Block 00000000000000000007fb360457edf480fb3b013f85dcadb05e5e76250ceeda served in 6.2963s
2018-09-14 22:10:35,351 - p2p - INFO - Block 00000000000000000025a15f04d553d949d399620c6d138072a3bb8cda1c9c28 served in 0.1448s
2018-09-14 22:10:44,016 - p2p - INFO - Block 0000000000000000000b9dd0be851d2d152650a4ebd69437c8759432fa124f17 served in 8.4693s
2018-09-14 22:10:49,692 - p2p - INFO - Block 0000000000000000002083418fecf1a0b18d6181ecf657eaf32cddeb0e2c457f served in 1.9351s
2018-09-14 22:10:55,623 - p2p - INFO - Block 000000000000000000208e9ca72f117fd88db5bc458155eb7e874b0116446185 served in 3.1718s
2018-09-14 22:10:58,927 - p2p - INFO - Block 00000000000000000012efa79ee5de7149564bfc51e7bc805821ae59e555162e served in 0.8526s
2018-09-14 22:11:03,326 - p2p - INFO - Block 0000000000000000000076d945ca3b37d7e4469c39275855882a7c9b048d666f served in 3.4746s
2018-09-14 22:11:11,638 - p2p - INFO - Block 0000000000000000000e4bc5f4d5dc1195687173de35b016b0829221e3a6dc8d served in 4.0430s
2018-09-14 22:11:15,551 - p2p - INFO - Block 00000000000000000027c756c58b7b6f0547b73f87ca0a33c0bafc16aaed6481 served in 0.7130s
2018-09-14 22:11:17,212 - p2p - INFO - Block 0000000000000000000ec77fafb4fdbe426be01acd7b4fcdcf90be6ee9b5c14f served in 0.6503s
2018-09-14 22:11:18,980 - p2p - INFO - Block 000000000000000000248dae091988a7a85f0e96324f1433f072731d2823dbd8 served in 0.6084s
2018-09-14 22:11:20,248 - p2p - INFO - Block 000000000000000000112df0ff2fb9ebfde59cbccd8ad75e3d00050801d03cac served in 0.6265s
2018-09-14 22:11:21,501 - p2p - INFO - Block 0000000000000000001277a2c765f728be3dc2bc851755bb3d7b1e4d2f092618 served in 0.4725s
2018-09-14 22:11:23,259 - p2p - INFO - Block 000000000000000000275fd719bffd6030c62bb797914e88ff3148aa638e94b9 served in 0.9213s
2018-09-14 22:11:26,400 - p2p - INFO - Block 00000000000000000014d23aacb7a3352c140dd0d78be37a542683893a3d3803 served in 1.4890s
2018-09-14 22:11:31,336 - p2p - INFO - Block 0000000000000000000b78f225be0474be732910e0939c6a25cbcbb57c4e1e81 served in 2.8372s
2018-09-14 22:11:34,762 - p2p - INFO - Block 0000000000000000001b4f0c376b279d291b03acc5c8ac6bfc2fe9be88995ae4 served in 2.6523s
2018-09-14 22:11:39,830 - p2p - INFO - Block 0000000000000000001700afee9531eb988f88dc4381b4ddd074b7f0f1a4df94 served in 1.7372s
2018-09-14 22:11:45,138 - p2p - INFO - Block 0000000000000000000895948986bdea42a08804d6aefb5f4970e99a6146ec02 served in 1.7867s
2018-09-14 22:11:59,602 - p2p - INFO - Block 0000000000000000002196a0b8b916e1099497fc14ac610fedd09c6c56af5ec2 served in 8.9184s
2018-09-14 22:12:06,099 - p2p - INFO - Block 00000000000000000020a37db7d78b140cafac1b87cc3fdd85c9833d20a11e97 served in 1.6268s
2018-09-14 22:12:17,475 - p2p - INFO - Block 0000000000000000001ed139a204a70b98f9c9c052c3c6cc7af7deb5659b3be3 served in 1.9709s
2018-09-14 22:12:19,925 - p2p - INFO - Block 0000000000000000000c3b4ee73e5a176c463fa004414df9c5be8f539764dbc9 served in 0.3514s
2018-09-14 22:12:23,122 - p2p - INFO - Block 00000000000000000022869966d1cd97943d51d37d92fc16434953804374ce74 served in 2.1919s
2018-09-14 22:12:26,489 - p2p - INFO - Block 0000000000000000001bdd33d6694124c652397da61ed07e14801235866487fe served in 2.5332s
2018-09-14 22:12:29,549 - p2p - INFO - Block 00000000000000000010b1460409458cbda4bfdba880716a9eef9e47d5842b3d served in 0.2512s
2018-09-14 22:12:32,856 - p2p - INFO - Block 0000000000000000001ab229b37ab88513342f28cfa0c32263552fd99276b969 served in 2.9045s

Optimization of verbose blocks may be addressed in future releases.

spruned 0.0.2 alpha 7

P2P Improvements:

• Use getaddr to obtain, persist and connect to new peers
• Ban peers after too many failures