In p2p/dial.go, conn.flags was accessed without using sync/atomic.
This race is fixed by removing the access.

In p2p/enode/iter_test.go, a similar race is resolved by writing the field atomically.

Co-authored-by: Felix Lange <[email protected]>

…hereum#23437)

This fixes a rare crash which could occur when two different errors happened
in the same bloombits.MatcherSession.

Before:

316M .git/modules/tests/

After:

171M .git/modules/tests/

Co-authored-by: Felix Lange <[email protected]>
Co-authored-by: Sina Mahmoodi <[email protected]>

Co-authored-by: Felix Lange <[email protected]>

* core: make txpool reject too sudden changes

* core: add some metrics to txpool

ethereum#23426)" (ethereum#23456)

This reverts commit c368f72.

The new linter version is built with go 1.17 and thus includes the go vet
check for mismatched +build and go:build lines.

Fortunately, no new warnings are reported with this update.

This PR adds functionality to the evm t8n to calculate ethash difficulty. If the caller does not provide a currentDifficulty, but instead provides the parentTimestamp (well, semi-optional, will default to 0 if not given), and parentDifficulty, we can calculate it for him.

The caller can also provide a parentUncleHash. In most, but not all cases, the parent uncle hash also affects the formula. If no such hash is provided (or, if the empty all-zero hash is provided), it's assumed that there were no uncles.

…ereum#23419)

Generated by go1.17 fmt ./...

As per benchmark results below, these changes speed up encoding/decoding of
consensus objects a bit.

    name                             old time/op    new time/op    delta
    EncodeRLP/legacy-header-8           384ns ± 1%     331ns ± 3%  -13.83%  (p=0.000 n=7+8)
    EncodeRLP/london-header-8           411ns ± 1%     359ns ± 2%  -12.53%  (p=0.000 n=8+8)
    EncodeRLP/receipt-for-storage-8     251ns ± 0%     239ns ± 0%   -4.97%  (p=0.000 n=8+8)
    EncodeRLP/receipt-full-8            319ns ± 0%     300ns ± 0%   -5.89%  (p=0.000 n=8+7)
    EncodeRLP/legacy-transaction-8      389ns ± 1%     387ns ± 1%     ~     (p=0.099 n=8+8)
    EncodeRLP/access-transaction-8      607ns ± 0%     581ns ± 0%   -4.26%  (p=0.000 n=8+8)
    EncodeRLP/1559-transaction-8        627ns ± 0%     606ns ± 1%   -3.44%  (p=0.000 n=8+8)
    DecodeRLP/legacy-header-8           831ns ± 1%     813ns ± 1%   -2.20%  (p=0.000 n=8+8)
    DecodeRLP/london-header-8           824ns ± 0%     804ns ± 1%   -2.44%  (p=0.000 n=8+7)

* rlp: pass length to byteArrayBytes

This makes it possible to inline byteArrayBytes. For arrays, the length is known
at encoder construction time, so the call to v.Len() can be avoided.

* rlp: avoid IsNil for pointer encoding

It's actually cheaper to use Elem first, because it performs less checks
on the value. If the pointer was nil, the result of Elem is 'invalid'.

* rlp: minor optimizations for slice/array encoding

For empty slices/arrays, we can avoid storing a list header entry in the
encoder buffer. Also avoid doing the tail check at encoding time because
it is already known at encoder construction time.

WriteTo is unused, and go vet warns about it because it
doesn't match the usual signature of the io.WriterTo method.

Currently rpc.BlockNumber is marshalled to JSON as a numeric value, which is
wrong because BlockNumber.UnmarshalJSON() wants it to either be hex-encoded
or string "earliest"/"latest"/"pending". As a result, the call chain

    rpc.BlockNumberOrHashWithNumber(123) -> json.Marshal() -> json.Unmarshal()

fails with error "cannot unmarshal object into Go value of type string".

…thereum#23470)

Support for this Ubuntu release has ended on July 22 2021 and
Launchpad no longer accepts uploads for it.

* accounts: new AddBackends method in manager

* node,cmd/geth: mv accman backend init to cmd/geth

* node,cmd/geth: mv scrypt config downstreawm from node

* accounts: use static buffer size for accman sub chan

minor fix

* accounts,cmd/geth: update accman backends through its event loop

* accounts,node: add comments

* accounts: un-export newBackendEvent

* accounts: use chan instead of wg in newBlockEvent

* node: rename isKeyDirEphem

* accounts,cmd: AddBackends->AddBackend

* accounts: fix potential blocking when adding backend

…ealer correctly (ethereum#23312)" (ethereum#23466)

This reverts commit 57feabe.

…tion (ethereum#23476)

Currently, setDefaults overwrites the transaction input value if only input is provided. This causes personal_sendTransaction to estimate the gas based on a transaction with empty data. eth_estimateGas never calls setDefaults so it was unaffected by this.

* core/vm: rework jumpdest analysis benchmarks

For BenchmarkJumpdestOpAnalysis use fixed code size of ~1.2MB
and classic benchmark loop.

* core/vm: clear bitvec in jumpdest analysis benchmark

strorage -> storage

* core: fix race conditions in txpool

* core: fixed races in the txpool

* core: rebased on master

* core: move reheap mutex

* core: renamed mutex

* core: revert Reheap changes

* Add missing copyright header

* Update stacktrie_test.go

Co-authored-by: Péter Szilágyi <[email protected]>

* docs: add post-mortem

* Update docs/postmortems/2021-08-22-split-postmortem.md

Co-authored-by: Sina Mahmoodi <[email protected]>

* Update docs/postmortems/2021-08-22-split-postmortem.md

Co-authored-by: Sina Mahmoodi <[email protected]>

* Update docs/postmortems/2021-08-22-split-postmortem.md

Co-authored-by: Sina Mahmoodi <[email protected]>

* Update docs/postmortems/2021-08-22-split-postmortem.md

* Update 2021-08-22-split-postmortem.md

* Update docs/postmortems/2021-08-22-split-postmortem.md

Co-authored-by: Sina Mahmoodi <[email protected]>

Co-authored-by: Sina Mahmoodi <[email protected]>

* internal/debug: remove deprecated flags

The removed flags are removed in the main portion of geth, this removes it internally too.

* internal/debug: remove legacy --debug and legacy --backtrace flag

* Update flags.go

Co-authored-by: Martin Holst Swende <[email protected]>

* remove rpc flags

* remove legacy rpc flags

* remove legacy rpc flags

* remove legacy rpc commands

* (hopefully) fix most of the build errors

* fix build errors

https://app.travis-ci.com/github/ethereum/go-ethereum/jobs/530318686

* cmd/utils: fix syntax error

* empty commit to unbreak travis ci

* fix syntax error

* syntax fixes

* syntax fixes

* fix

fixes "cmd/geth/usage.go:234:7: expected '(', found init (typecheck)"

* fix

* various fixes in usage.go

* various fixes in flags.go

* adds extra space

reverts the spacing to how it was before I resolved the merge conflict

* more fixes in usage.go

* fix

fix for cmd/geth/usage.go:243:17: expected operand, found ':=' (typecheck) in travis

* Update cmd/utils/flags.go

Co-authored-by: Martin Holst Swende <[email protected]>

* fix error

 fixes these errors:
cmd/utils/flags_legacy.go:21:2: "strings" imported but not used (typecheck)

	"strings"

	^

cmd/utils/flags_legacy.go:24:2: "github.com/ethereum/go-ethereum/node" imported but not used (typecheck)

	"github.com/ethereum/go-ethereum/node"

	^

* goimports

Co-authored-by: Martin Holst Swende <[email protected]>

This change is a rewrite of the freezer code.

When writing ancient chain data to the freezer, the previous version first encoded each
individual item to a temporary buffer, then wrote the buffer. For small item sizes (for
example, in the block hash freezer table), this strategy causes a lot of system calls for
writing tiny chunks of data. It also allocated a lot of temporary []byte buffers.

In the new version, we instead encode multiple items into a re-useable batch buffer, which
is then written to the file all at once. This avoids performing a system call for every
inserted item.

To make the internal batching work, the ancient database API had to be changed. While
integrating this new API in BlockChain.InsertReceiptChain, additional optimizations were
also added there.

Co-authored-by: Felix Lange <[email protected]>

…9903)

This PR ensures that wiping all data associated with a node (apart from its nodekey)
will not generate already used sequence number for the ENRs, since all remote nodes
would reject them until they out-number the previously published largest one.

The big complication with this scheme is that every local update to the ENR can
potentially bump the sequence number by one. In order to ensure that local updates
do not outrun the clock, the sequence number is a millisecond-precision timestamp,
and updates are throttled to occur at most once per millisecond.

Co-authored-by: Felix Lange <[email protected]>

…thereum#23542)

This fixes crashes on 32bit builds.

This fixes crashes in txpool tests on 32bit platforms.

les: duplicate downloader and fetcher to allow progressive refactoring

Updated example genesis.json configuration section, added londonBlock within config.

* cmd/evm: transaction validation tool

* cmd/evm: add hash to t9n tool

* cmd/evm: lint nits

* cmd/evm: nitpicks

* core: cmd: invert disableMemory

* core: fix missed inversion

* cmd/evm: preserve Flags but change default value

* Apply suggestions from code review

Co-authored-by: Martin Holst Swende <[email protected]>

Co-authored-by: Martin Holst Swende <[email protected]>

…thereum#23568)

* cmd/devp2p/internal/ethtest: only use eth66 if eth66 is negotiated

* cmd/devp2p/internal/ethtest: switch on concrete type not pointer

* cmd/devp2p/internal/ethtest: switch on concrete type not pointer

This change introduces 2 new optional methods; `enter()` and `exit()` for js tracers, and makes `step()` optiona. The two new methods are invoked when entering and exiting a call frame (but not invoked for the outermost scope, which has it's own methods). Currently these are the data fields passed to each of them:

    enter: type (opcode), from, to, input, gas, value
    exit: output, gasUsed, error

The PR also comes with a re-write of the callTracer. As a backup we keep the previous tracing script under the name `callTracerLegacy`. Behaviour of both tracers are equivalent for the most part, although there are some small differences (improvements), where the new tracer is more correct / has more information.

* ethclient/gethclient: fix flaky test (due to map key ordering)

* accounts/keystore: fix test failing due to rand collision due to low time resolution on windows

Spelling update

…um#23605)

* internal: support optional filter expression for debug.stacks

* internal/debug: fix string regexp

* internal/debug: support searching for line numbers too

internal/debug: make gosimple linter happy

go.mod: fix module file to prevent go from updating it during build

* add explicit note about typing exit in console

* Add note about typing exit as alternative

Closes ethereum#23522

Co-authored-by: Martin Holst Swende <[email protected]>

* rpc: set scheme for ws and ipc conns to the server

* rpc: review fixes

* rpc: bugfix

* rpc: empty default scheme

This PR adds a 30s timeout for the remote part to answer a ping message, thus detecting (silent) disconnnects

* eth/tracers: add re-write of 4byte tracer using enter/exit

* eth/tracers: fix 4byte indent

…unt to trie" (ethereum#23567)

* core/state: abstracted "write account to trie" method

* fix appveyor build

* Apply suggestions from code review

Co-authored-by: Martin Holst Swende <[email protected]>

* review feedback

* core/state/accounts: move Account to core/types

* core/types: rename Account -> StateAccount

* core/state: restore EncodeRLP for stateObject

* core/types: add the missing file

* more review feedback

* more review feedback

Co-authored-by: Martin Holst Swende <[email protected]>
Co-authored-by: Felix Lange <[email protected]>

This PR implements a new debug method, which I've talked briefly about to some other client developers. It allows the caller to obtain the intermediate state roots for a block (which might be either a canon block or a 'bad' block).

…thereum#23147)

* core/types: rm extranous check in test

* core/rawdb: add lightweight types for block logs

* core/rawdb,eth: use lightweight accessor for log filtering

* core/rawdb: add bench for decoding into rlpLogs

* accounts/abi/bind: check event signature before parsing

* remove redundant break line

…#23632)

This PR fixes an issue in traceChain, where the statedb Commit operation was performed asynchronously with dereference-operations agains the underlying trie.Database instance. Due to how the reference counting works within the trie database (where parent count is recursively updated when new parents are added), doing dereferencing in the middle of Commit can cause the refcount to become wrong, leading to an inconsistent state. 

This was fixed by doing Commit/Deref from the same routine.

* Simplify peer known block/txns cache

* Address minor changes

* Add more minor comments

* Minor changes from review

Fixes ethereum#23640

internal/ethapi: support retrieving header RLPs too

* eth/tracers: fix calltracer fault handling

* eth/tracers: fix calltracer indentation

internal/ethapi: make header/block rlp retrieval canonical

The test did not synchronize with per-case goroutines, and thus didn't notice
that some tests were just hanging. This change adds missing synchronization
and fixes the broken tests.

…ereum#23656)

…um#23646)

This PR adds a method to the debug namespace, to iterate over the blocks and check where we have the roots on disk.

This update includes fixes for the *BSD build and go module vendoring.

This resolves a long-standing TODO. The point of copying the address is
to ensure that all data referenced by types.Transaction is independent of the
data passed into the constructor.

Co-authored-by: Felix Lange <[email protected]>

This change removes misuses of sync.WaitGroup in BlockChain. Before this change,
block insertion modified the WaitGroup counter in order to ensure that Stop would wait
for pending operations to complete. This was racy and could even lead to crashes
if Stop was called at an unfortunate time. The issue is resolved by adding a specialized
'closable' mutex, which prevents chain modifications after stopping while also
synchronizing writers with each other.

Co-authored-by: Felix Lange <[email protected]>

…#23683)

Fixes ethereum#23681

After the fix I get the address 0x6d6d02e83c4ced98204e20126acf27e9d87b8af2 for the
tx mentioned in the ticket, which agrees with etherscan.

Go 1.17.2 fixes some miscompilation issues on amd64 and a runtime issue with timers.
While the upgrade is not strictly necessary for go-ethereum right now, it is still good
to be up-to-date.

This ensures that all miner goroutines have exited before stopping the blockchain. 

Co-authored-by: Felix Lange <[email protected]>

This fixes a data race on worker.current by moving the call to StopPrefetcher
into the main loop.

The commit also contains fixes for two other races in unit tests of unrelated packages.