Commit Graph

165 Commits

Author SHA1 Message Date
a1313b5b1e trie: make hasher parallel when number of changes are large (#20488)
* trie: make hasher parallel when number of changes are large

* trie: remove unused field dirtyCount

* trie: rename unhashedCount/unhashed
2020-02-04 14:02:38 +02:00
5a9c96454e trie: separate hashes and committer, collapse on commit
* trie:  make db insert use size instead of full data

* core/state: minor optimization in state onleaf allocation

* trie: implement dedicated committer and hasher

* trie: use dedicated committer/hasher

* trie: linter nitpicks

* core/state, trie: avoid unnecessary storage trie load+commit

* trie: review feedback, mainly docs + minor changes

* trie: start deprecating old hasher

* trie: fix misspell+lint

* trie: deprecate hasher.go, make proof framework use new hasher

* trie: rename pure_committer/hasher to committer/hasher

* trie, core/state: fix review concerns

* trie: more review concerns

* trie: make commit collapse into hashnode, don't touch dirtyness

* trie: goimports fixes

* trie: remove panics
2020-02-03 17:28:30 +02:00
9b09c0fc83 * trie: utilize callbacks instead of amassing lists in ref/unref (#20529)
* trie/tests: add benchmarks and update trie tests

* trie: update benchmark tests

* trie: utilize callbacks instead of amassing lists of hashes in database ref/unref

* trie: replace remaining non-callback based accesses
2020-01-17 13:59:45 +02:00
4f7b7f84ae add node.go unit test file node_test.go (#20028)
* add node.go unit test file node_test.go

* add node_test.go file license and rollback trie_test.go

* fix unused variable v

* trie: fix license year

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-01-07 10:31:20 +01:00
13ccf6016e trie: track dirty cache metrics, track clean writes on commit 2019-12-02 12:23:35 +02:00
08611cfd75 trie: remove dead code (#20405) 2019-11-28 12:47:35 +02:00
9d67222f4e trie: replace bigcache with fastcache (#19971) 2019-11-25 10:58:15 +02:00
0abcf03fde trie: remove unused code (#20366) 2019-11-22 17:24:48 +02:00
ecdbb402ee trie: remove node ordering slice in sync batch (#19929)
When we flush a batch of trie nodes into database during the state
sync, we should guarantee that all children should be flushed before
parent.

Actually the trie nodes commit order is strict by: children -> parent.
But when we flush all ready nodes into db, we don't need the order
anymore since

    (1) they are all ready nodes (no more dependency)
    (2) underlying database provides write atomicity
2019-10-28 18:50:11 +01:00
df6c08a485 core, trie: decode the value for storage dump (#19943)
* core, trie: decode the value for storage dump

* core/state: address comment
2019-08-12 17:14:40 +03:00
5183483c53 core/state, p2p/discover, trie, whisper: avoid unnecessary conversions (#19870)
No need to convert these types.
2019-07-22 10:30:09 +03:00
dcc4adfcd7 cmd/geth: wrong memory size sanitizing on OpenBSD (#19793) 2019-07-05 13:13:21 +03:00
536b3b416c cosensus, core, eth, params, trie: fixes + clique history cap 2019-05-16 10:39:35 +03:00
80469bea0c all: integrate the freezer with fast sync
* all: freezer style syncing

core, eth, les, light: clean up freezer relative APIs

core, eth, les, trie, ethdb, light: clean a bit

core, eth, les, light: add unit tests

core, light: rewrite setHead function

core, eth: fix downloader unit tests

core: add receipt chain insertion test

core: use constant instead of hardcoding table name

core: fix rollback

core: fix setHead

core/rawdb: remove canonical block first and then iterate side chain

core/rawdb, ethdb: add hasAncient interface

eth/downloader: calculate ancient limit via cht first

core, eth, ethdb: lots of fixes

* eth/downloader: print ancient disable log only for fast sync
2019-05-16 10:39:32 +03:00
9effd64290 core, eth, trie: bloom filter for trie node dedup during fast sync (#19489)
* core, eth, trie: bloom filter for trie node dedup during fast sync

* eth/downloader, trie: address review comments

* core, ethdb, trie: restart fast-sync bloom construction now and again

* eth/downloader: initialize fast sync bloom on startup

* eth: reenable eth/62 until we properly remove it
2019-05-13 15:28:01 +03:00
14868a37fb trie: clarify why verifyProof doesn't check hashes (#19530)
* trie: fix merkle proof

* trie: use hasher instead of allocate keccack256 every time

* trie: add comments
2019-05-07 15:06:07 +03:00
4a4abc41d4 trie: approximate the wasted cache metaspace closer 2019-04-12 11:43:16 +03:00
4bf0d11e7c trie: there's no point in retrieving the metaroot 2019-04-05 13:09:28 +03:00
59e1953246 core, ethdb, trie: mode dirty data to clean cache on flush (#19307)
This PR is a more advanced form of the dirty-to-clean cacher (#18995),
where we reuse previous database write batches as datasets to uncache,
saving a dirty-trie-iteration and a dirty-trie-rlp-reencoding per block.
2019-03-26 15:48:31 +01:00
876f357364 trie: disable fnv64a hashing of hashes for bigcache (#19314)
* trie: disable fnv64a hashing of hashes for bigcache

* trie/database: add very important period
2019-03-22 17:13:28 +02:00
91eec1251c cmd, core, eth, trie: get rid of trie cache generations (#19262)
* cmd, core, eth, trie: get rid of trie cache generations

* core, trie: get rid of remainder of cache gen boilerplate
2019-03-14 15:25:12 +02:00
8111b9dda5 ethdb, trie: tiny API tidy-up from the database rework pr 2019-03-12 12:32:02 +02:00
054412e335 all: clean up and proerly abstract database access 2019-03-06 13:35:03 +02:00
514a9472ad trie: prefer nil slices over zero-length slices (#19084) 2019-02-19 14:50:11 +01:00
4f85c2b88b trie: fix error in node decoding (#19111) 2019-02-16 16:16:12 +01:00
bb7c786b09 trie: add missing unlock call in error case (#18985) 2019-02-04 12:42:46 +01:00
33d233d3e1 vendor, crypto, swarm: switch over to upstream sha3 package 2019-01-04 09:26:07 +02:00
2843001ac2 trie: fix overflow in write cache parent tracking (#18165)
trie/database: fix overflow in parent tracking
2018-11-22 15:14:31 +02:00
d136e985e8 trie: go fmt package 2018-11-16 16:35:39 +02:00
68be45e5f8 trie: return hasher to pool (#18116)
* trie: return hasher to pool

* trie: minor code formatting fix
2018-11-16 11:50:48 +02:00
434dd5bc00 cmd, core, eth, light, trie: add trie read caching layer 2018-11-15 12:22:13 +02:00
53634f1e04 trie: remove unused originalRoot field (#17862) 2018-10-08 13:16:16 +02:00
6a33954731 core, eth, trie: use common/prque (#17508) 2018-09-03 17:33:21 +02:00
70398d300d trie: fix typo (#17498) 2018-08-24 21:08:48 +03:00
11bbc66082 eth, trie: fix tracer GC which accidentally pruned the metaroot 2018-08-09 12:33:30 +03:00
8051a0768a trie: fix comment typo (#17350) 2018-08-08 16:08:40 +03:00
cf05ef9106 p2p, swarm, trie: avoid copying slices in loops (#17265) 2018-08-07 13:56:40 +03:00
8a9c31a307 trie: handle removing the freshest node too 2018-07-30 16:31:17 +03:00
319098cc1c trie: fix a temporary memory leak in the memcache 2018-07-02 15:47:33 +03:00
d926bf2c7e trie: cache collapsed tries node, not rlp blobs (#16876)
The current trie memory database/cache that we do pruning on stores
trie nodes as binary rlp encoded blobs, and also stores the node
relationships/references for GC purposes. However, most of the trie
nodes (everything apart from a value node) is in essence just a
collection of references.

This PR switches out the RLP encoded trie blobs with the
collapsed-but-not-serialized trie nodes. This permits most of the
references to be recovered from within the node data structure,
avoiding the need to track them a second time (expensive memory wise).
2018-06-21 11:28:05 +02:00
a3267ed929 trie: don't report the root flushlist as an alloc 2018-06-11 14:32:13 +03:00
ea06da0892 trie: avoid unnecessary slicing on shortnode decoding (#16917)
optimization code
2018-06-07 11:48:36 +03:00
e8ea5aa0d5 trie: reduce hasher allocations (#16896)
* trie: reduce hasher allocations

name    old time/op    new time/op    delta
Hash-8    4.05µs ±12%    3.56µs ± 9%  -12.13%  (p=0.000 n=20+19)

name    old alloc/op   new alloc/op   delta
Hash-8    1.30kB ± 0%    0.66kB ± 0%  -49.15%  (p=0.000 n=20+20)

name    old allocs/op  new allocs/op  delta
Hash-8      11.0 ± 0%       8.0 ± 0%  -27.27%  (p=0.000 n=20+20)

* trie: bump initial buffer cap in hasher
2018-06-05 15:06:29 +03:00
143c4341d8 core, eth, trie: streaming GC for the trie cache (#16810)
* core, eth, trie: streaming GC for the trie cache

* trie: track memcache statistics
2018-06-04 10:47:43 +03:00
38c7eb0f26 trie: rename TrieSync to Sync and improve hexToKeybytes (#16804)
This removes a golint warning: type name will be used as trie.TrieSync by
other packages, and that stutters; consider calling this Sync.

In hexToKeybytes len(hex) is even and (even+1)/2 == even/2, remove the +1.
2018-05-29 17:48:43 +02:00
d6ed2f67a8 eth, node, trie: fix minor typos (#16802) 2018-05-24 15:55:20 +03:00
54294b45b1 Merge pull request #16803 from karalabe/trie-avoid-funccall
trie: cleaner logic, one less func call
2018-05-24 15:54:00 +03:00
d31802312a trie: cleaner logic, one less func call 2018-05-24 13:46:45 +03:00
c934c06cc1 trie: support proof generation from the iterator 2018-05-23 13:02:20 +03:00
0fe47e98c4 trie: fixes to comply with golint (#16771) 2018-05-21 23:41:31 +03:00