Merge pull request #2996 from stacks-network/develop

Changes for version 2.05.0.1.0

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,21 @@
+<!--
+  IMPORTANT
+  Pull requests are ideal for making small changes to this project. However, they are NOT an appropriate venue to introducing non-trivial or breaking changes to the codebase.
+
+  For introducing non-trivial or breaking changes to the codebase, please follow the SIP (Stacks Improvement Proposal) process documented here:
+  https://github.com/blockstack/stacks-blockchain/blob/master/sip/sip-000-stacks-improvement-proposal-process.md.
+-->
+
+### Description
+
+### Applicable issues
+- fixes #
+
+### Additional info (benefits, drawbacks, caveats)
+
+### Checklist
+- [ ] Test coverage for new or modified code paths
+- [ ] Changelog is updated
+- [ ] Required documentation changes (e.g., `docs/rpc/openapi.yaml` and `rpc-endpoints.md` for v2 endpoints, `event-dispatcher.md` for new events)
+- [ ] New clarity functions have corresponding PR in `clarity-benchmarking` repo
+- [ ] New integration test(s) added to `bitcoin-tests.yml`

.github/actions/bitcoin-int-tests/Dockerfile.code-cov

@@ -2,7 +2,7 @@ FROM rust:bullseye AS test
 
 WORKDIR /build
 
-RUN rustup override set nightly && \
+RUN rustup override set nightly-2022-01-14 && \
     rustup component add llvm-tools-preview && \
     cargo install grcov
 

.github/actions/bitcoin-int-tests/Dockerfile.generic.bitcoin-tests

@@ -6,7 +6,7 @@ COPY . .
 
 WORKDIR /src/testnet/stacks-node
 
-RUN rustup override set nightly && \
+RUN rustup override set nightly-2022-01-14 && \
     rustup component add llvm-tools-preview && \
     cargo install grcov
 

.github/actions/bitcoin-int-tests/Dockerfile.large-genesis

@@ -9,7 +9,7 @@ RUN cd / && tar -xvzf bitcoin-0.20.0-x86_64-linux-gnu.tar.gz
 
 RUN ln -s /bitcoin-0.20.0/bin/bitcoind /bin/
 
-RUN rustup override set nightly && \
+RUN rustup override set nightly-2022-01-14 && \
     rustup component add llvm-tools-preview && \
     cargo install grcov
 

.github/workflows/bitcoin-tests.yml

@@ -5,6 +5,11 @@ name: stacks-bitcoin-integration-tests
 on:
   pull_request:
 
+concurrency:
+  group: stacks-bitcoin-integration-tests-${{ github.ref }}
+  # Only cancel in progress if this is for a PR
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
 jobs:
   build-integration-image:
     runs-on: ubuntu-latest
@@ -43,6 +48,7 @@ jobs:
           - tests::neon_integrations::bitcoind_forking_test
           - tests::neon_integrations::should_fix_2771
           - tests::neon_integrations::pox_integration_test
+          - tests::neon_integrations::mining_events_integration_test
           - tests::bitcoin_regtest::bitcoind_integration_test
           - tests::should_succeed_handling_malformed_and_valid_txs
           - tests::neon_integrations::size_overflow_unconfirmed_microblocks_integration_test
@@ -53,10 +59,15 @@ jobs:
           - tests::neon_integrations::filter_low_fee_tx_integration_test
           - tests::neon_integrations::filter_long_runtime_tx_integration_test
           - tests::neon_integrations::mining_transactions_is_fair
+          - tests::neon_integrations::fuzzed_median_fee_rate_estimation_test_window5
+          - tests::neon_integrations::fuzzed_median_fee_rate_estimation_test_window10
+          - tests::neon_integrations::use_latest_tip_integration_test
+          - tests::neon_integrations::test_flash_block_skip_tenure
           - tests::epoch_205::test_dynamic_db_method_costs
           - tests::epoch_205::transition_empty_blocks
           - tests::epoch_205::test_cost_limit_switch_version205
           - tests::epoch_205::test_exact_block_costs
+          - tests::epoch_205::bigger_microblock_streams_in_2_05
     steps:
       - uses: actions/checkout@v2
       - name: Download docker image

.github/workflows/stacks-blockchain.yml

@@ -12,6 +12,11 @@ on:
         description: 'The tag to create (optional)'
         required: false
 
+concurrency:
+  group: stacks-blockchain-${{ github.ref }}
+  # Only cancel in progress if this is for a PR
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
 jobs:
   # Notify Slack channel of workflow start
   notify-start:
@@ -244,7 +249,7 @@ jobs:
       - create-release
     strategy:
       matrix:
-        platform: [ windows-x64, macos-x64, linux-x64, linux-musl-x64, linux-armv7, linux-arm64 ]
+        platform: [ windows-x64, macos-x64, macos-arm64, linux-x64, linux-musl-x64, linux-armv7, linux-arm64 ]
 
     steps:
       - uses: actions/checkout@v2

CHANGELOG.md

@@ -5,6 +5,79 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to the versioning scheme outlined in the [README.md](README.md).
 
+## [2.05.0.1.0]
+
+### Added 
+- A new fee estimator intended to produce fewer over-estimates, by having less
+  sensitivity to outliers. Its characteristic features are: 1) use a window to
+  forget past estimates instead of exponential averaging, 2) use weighted
+  percentiles, so that bigger transactions influence the estimates more, 3)
+  assess empty space in blocks as having paid the "minimum fee", so that empty
+  space is accounted for, 4) use random "fuzz" so that in busy times the fees can
+  change dynamically. (#2972)
+- Implements anti-entropy protocol for querying transactions from other 
+  nodes' mempools. Before, nodes wouldn't sync mempool contents with one another.
+  (#2884)
+- Structured logging in the mining code paths. This will shine light 
+  on what happens to transactions (successfully added, skipped or errored) that the
+  miner considers while buildings blocks. (#2975)
+- Added the mined microblock event, which includes information on transaction
+  events that occurred in the course of mining (will provide insight
+  on whether a transaction was successfully added to the block,
+  skipped, or had a processing error). (#2975)
+- For v2 endpoints, can now specify the `tip` parameter to `latest`. If 
+  `tip=latest`, the node will try to run the query off of the latest tip. (#2778)
+- Adds the /v2/headers endpoint, which returns a sequence of SIP-003-encoded 
+  block headers and consensus hashes (see the ExtendedStacksHeader struct that 
+  this PR adds to represent this data). (#2862)
+- Adds the /v2/data_var endpoint, which returns a contract's data variable 
+  value and a MARF proof of its existence. (#2862)
+- Fixed a bug in the unconfirmed state processing logic that could lead to a
+  denial of service (node crash) for nodes that mine microblocks (#2970)
+- Added prometheus metric that tracks block fullness by logging the percentage of each
+  cost dimension that is consumed in a given block (#3025).  
+
+
+### Changed
+- Updated the mined block event. It now includes information on transaction 
+  events that occurred in the course of mining (will provide insight
+  on whether a transaction was successfully added to the block, 
+  skipped, or had a processing error). (#2975)
+- Updated some of the logic in the block assembly for the miner and the follower
+  to consolidate similar logic. Added functions `setup_block` and `finish_block`.
+  (#2946)
+- Makes the p2p state machine more reactive to newly-arrived 
+  `BlocksAvailable` and `MicroblocksAvailable` messages for block and microblock 
+  streams that this node does not have. If such messages arrive during an inventory 
+  sync, the p2p state machine will immediately transition from the inventory sync 
+  work state to the block downloader work state, and immediately proceed to fetch 
+  the available block or microblock stream. (#2862)
+- Nodes will push recently-obtained blocks and microblock streams to outbound
+  neighbors if their cached inventories indicate that they do not yet have them
+(#2986).
+- Nodes will no longer perform full inventory scans on their peers, except
+  during boot-up, in a bid to minimize block-download stalls (#2986).
+- Nodes will process sortitions in parallel to downloading the Stacks blocks for
+  a reward cycle, instead of doing these tasks sequentially (#2986).
+- The node's runloop will coalesce and expire stale requests to mine blocks on
+  top of parent blocks that are no longer the chain tip (#2969).
+- Several database indexes have been updated to avoid table scans, which
+  significantly improves most RPC endpoint speed and cuts node spin-up time in
+half (#2989, #3005).
+- Fixed a rare denial-of-service bug whereby a node that processes a very deep
+  burnchain reorg can get stuck, and be rendered unable to process further
+sortitions.  This has never happened in production, but it can be replicated in
+tests (#2989).
+- Updated what indices are created, and ensures that indices are created even 
+  after the database is initialized (#3029).
+
+### Fixed 
+- Updates the lookup key for contracts in the pessimistic cost estimator. Before, contracts
+  published by different principals with the same name would have had the same 
+  key in the cost estimator. (#2984)
+- Fixed a few prometheus metrics to be more accurate compared to `/v2` endpoints 
+  when polling data (#2987)
+
 ## [2.05.0.0.0]
 
 This software update is a consensus changing release and the
@@ -16,12 +89,12 @@ directories before the 2.05 consensus changes activate (Bitcoin height
 run a 2.05 node afterwards, you must start from a new chainstate
 directory.
 
-## Added
+### Added
 
 - At height 713,000 a new `costs-2` contract will be launched by the
   Stacks boot address.
 
-## Changed
+### Changed
 
 - Stacks blocks whose parents are mined >= 713,000 will use default costs
   from the new `costs-2` contract.
@@ -31,7 +104,7 @@ directory.
 - Stacks blocks whose parents are mined >= 713,000 will use the new block
   limit defined in SIP-012.
 
-## Fixed
+### Fixed
 
 - Miners are now more aggressive in calculating their block limits
   when confirming microblocks (#2916)
@@ -43,15 +116,15 @@ selection logic in the default miner to prioritize by an estimated fee
 rate instead of raw fee. This release's chainstate directory is
 compatible with chainstate directories from 2.0.11.3.0.
 
-## Added
+### Added
 
 - FeeEstimator and CostEstimator interfaces. These can be controlled
   via node configuration options. See the `README.md` for more
   information on the configuration.
 - New fee rate estimation endpoint `/v2/fees/transaction` (#2872). See
   `docs/rpc/openapi.yaml` for more information.
 
-## Changed
+### Changed
 
 - Prioritize transaction inclusion in blocks by estimated fee rates (#2859).
 - MARF sqlite connections will now use `mmap`'ed connections with up to 256MB