Commit Graph

188 Commits

Author SHA1 Message Date
carllin
06f84c65f1 Fix rooted accounts cleanup, simplify locking (#12194)
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-28 16:04:46 -07:00
carllin
00b36e6db2 Fix ref-count for multiple stores to the same pubkey in a slot, fixes zero lamport purge detection (#12462)
Co-authored-by: Carl Lin <carl@solana.com>
2020-09-25 18:54:48 -07:00
carllin
55be8d4016 Move dropping AppendVecs outside lock (#12408)
* Move drop outside lock

Co-authored-by: Carl Lin <carl@solana.com>
2020-09-23 14:17:49 -07:00
sakridge
3930cb865a Add keccak-secp256k1 instruction (#11839)
* Implement keccak-secp256k1 instruction

Verifies eth addreses with ecrecover function

* Move secp256k1 test
2020-09-15 18:23:21 -07:00
Ryo Onodera
de4a613610 Check bank capitalization (#11927)
* Check bank capitalization

* Simplify and unify capitalization calculation

* Improve and add tests

* Avoid overflow and inhibit automatic restart

* Fix test

* Tweak checked sum for cap. and add tests

* Fix broken build after merge conflicts..

* Rename to ClusterType

* Rename confusing method

* Clarify comment

* Verify cap. in rent and inflation tests

Co-authored-by: Stephen Akridge <sakridge@gmail.com>
2020-09-12 01:48:06 +09:00
Ryo Onodera
c274e26eb8 Prevent unbound memory growth by blockstore_processor (#12110)
* Prevent unbound memory growth by blockstore_processor

* Promote log to info! considering infrequency

* Exclude the time of freeing from interval...

* Skip not-shrinkable slots even if forced

* Add comment
2020-09-09 12:05:10 +09:00
Ryo Onodera
53b8ea4464 Rename to ClusterType and restore devnet compat. (#12068)
* Rename to ClusterType and restore devnet compat.

* De-duplicate parse code and add comments

* Adjust default Devnet genesis & reduce it in tests
2020-09-08 23:55:09 +09:00
carllin
a13efc52b3 Add unlock epochs for blake3 (#12054)
Co-authored-by: Carl <carl@solana.com>
2020-09-04 15:21:15 -07:00
carllin
af08221aec Switch account hashing to blake3 (#11969)
* Switch account hashing to blake3

Co-authored-by: Carl <carl@solana.com>
2020-09-02 00:37:36 -07:00
sakridge
770d3d383c Accounts hash calculation metrics (#11433) 2020-08-06 21:39:10 -07:00
Michael Vines
eefcf484cb clippy 2020-08-03 18:35:15 +00:00
Ryo Onodera
3e4f49f9c9 Fix race condition between shrinking and cleaning (#11235)
* Fix race condition between shrinking and cleaning

* Minor formatting

* fix ci

* Update comments

* More update comment

* Adjust fn naming
2020-07-29 06:06:14 +09:00
Michael Vines
e5d8c4383f Add Bank support for "upgrade epochs" where all non-vote transactions will be rejected 2020-07-17 08:48:42 -07:00
sakridge
8bf3a0aa05 accounts_clean: Convert stack dependency calculation with iterative (#11067)
* accounts_clean: Convert stack dependency calculation with iterative

* optimize clean with by creating a reverse-lookup hashset of the affected
keys

* Add dependency bench

reduce bench

* Huge clean
2020-07-15 06:49:22 -07:00
Kristofer Peterson
ed5a2f2a90 Fold bank serialisation into serde snapshot (#10581)
* Move bank (de)serialisation logic from bank and snapshot_utils to serde_snapshot.
Add sanity assertions between genesis config and bank fields on deserialisation.
Atomically update atomic bool in quote_for_specialization_detection().
Use same genesis config when restoring snapshots in test cases.

* Tidy up namings and duplicate structs to version

* Apply struct renames to tests

* Update abi hashes

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-07-13 23:00:59 +09:00
sakridge
edf2b1ee85 Check for deleting key, make sure list is empty again (#11007) 2020-07-11 20:38:34 -07:00
sakridge
631f051c68 Add jitter to cleanup to prevent all nodes cleaning at the same time (#10936) 2020-07-10 11:54:45 -07:00
sakridge
6429042b6e Add --print-account-stats/--snapshot-archive-path arguments to ledger-tool (#10945)
Allows for seeing how the accounts are being stored
and specifying a different snapshot source directory.
2020-07-08 09:32:11 -07:00
Ryo Onodera
39b3ac6a8d Introduce automatic ABI maintenance mechanism (2/2; rollout) (#8012)
* Introduce automatic ABI maintenance mechanism (2/2; rollout)

* Fix stable clippy

* Change to symlink

* Freeze abi of Tower

* fmt...

* Improve dev-experience!

* Update BankSlotDelta

$ diff -u /tmp/abi8/*7dg6BreYxTuxiVz6aLvk3p2Z7GQk2cJqfGvC9h4FAoSj* /tmp/abi8/*9chBcbXVJ4fK7uGgydQzam5aHipaAKFw6V4LDFpjbE4w*
--- /tmp/abi8/bank__BankSlotDelta_frozen_abi__test_abi_digest_7dg6BreYxTuxiVz6aLvk3p2Z7GQk2cJqfGvC9h4FAoSj      2020-06-18 18:01:22.831228087 +0900
+++ /tmp/abi8/bank__BankSlotDelta_frozen_abi__test_abi_digest_9chBcbXVJ4fK7uGgydQzam5aHipaAKFw6V4LDFpjbE4w      2020-07-03 15:59:58.430695244 +0900
@@ -140,7 +140,7 @@
                                                         field u8
                                                             primitive u8
                                                         field solana_sdk::instruction::InstructionError
-                                                            enum InstructionError (variants = 34)
+                                                            enum InstructionError (variants = 35)
                                                                 variant(0) GenericError (unit)
                                                                 variant(1) InvalidArgument (unit)
                                                                 variant(2) InvalidInstructionData (unit)
@@ -176,6 +176,7 @@
                                                                 variant(31) CallDepth (unit)
                                                                 variant(32) MissingAccount (unit)
                                                                 variant(33) ReentrancyNotAllowed (unit)
+                                                                variant(34) MaxSeedLengthExceeded (unit)
                                                     variant(9) CallChainTooDeep (unit)
                                                     variant(10) MissingSignatureForFee (unit)
                                                     variant(11) InvalidAccountIndex (unit)

* Fix some merge conflicts...
2020-07-06 20:22:23 +09:00
sakridge
832d47317e Move clean accounts to background service (#10898) 2020-07-02 22:25:17 -07:00
Greg Fitzgerald
0550b893b0 Fix typos (#10675)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2020-06-17 20:54:52 -07:00
carllin
8bd62d78eb Fix race in remove_unrooted_slot (#10607)
* Fix race

* clippy fixes

* Rename and add comment

Co-authored-by: Carl <carl@solana.com>
2020-06-15 19:41:26 -07:00
Ryo Onodera
dfe72d5242 Optimize stale slot shrinking for previously cleaned roots (#10099)
* Prioritize shrinking of previously cleaned roots

* measure time of stale slot shrinking

* Disable shrink for test

* shrink: budgeting, store count, force for snapshot

* Polish implementation and fix tests

* Fix ci..

* Clean up a bit

* Further polish implementation and fix/add tests

* Rebase fixes

* Remove unneeded Default for AccountStorageEntry

* Address review comments

* More cleanup

* More cleanup
2020-06-12 14:51:43 +09:00
Boqin Qin
0a638a8f17 runtime: fix possible deadlock in accounts_db (#10469) 2020-06-10 19:37:15 +09:00
Kristofer Peterson
e23340d89e Clippy cleanup for all targets and nighly rust (also support 1.44.0) (#10445)
* address warnings from 'rustup run beta cargo clippy --workspace'

minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs

expect some performance improvement AccountsDB::clean_accounts()

* address warnings from 'rustup run beta cargo clippy --workspace --tests'

* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'

* rustfmt

* fix warning stragglers

* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned

* Remove lock around JsonRpcRequestProcessor (#10417)

automerge

* make ancestors parameter optional to avoid forcing construction of empty hash maps

Co-authored-by: Greg Fitzgerald <greg@solana.com>
2020-06-09 09:38:14 +09:00
Michael Vines
e3b444834f Add more logging while unpacking snapshots (#10266) 2020-05-27 10:41:05 -07:00
Kristofer Peterson
b7a32f01c0 Multi-version snapshot support (#9980)
* Multi-version snapshot support

* rustfmt

* Remove CLI options and runtime support for selection output snapshot version.
Address some clippy complaints.

* Muzzle clippy type complexity warning.

Despite clippy's suggestion, it is not currently possible to create type aliases
for traits and so everything within the 'Box<...>' cannot be type aliased.

This then leaves creating full blown traits, and either implementing
said traits by closure (somehow) or moving the closures into new structs
implementing said traits which seems a bit of a palaver.

Alternatively it is possible to define and use the type alias 'type ResultBox<T> = Result<Box<T>>'
which does seems rather pointless and not a great reduction in complexity but is enough to keep clippy happy.

In the end I simply went with squelching the clippy warning.

* Remove now unused Serialize/Deserialize trait implementations for AccountStorageEntry and AppendVec

* refactor versioned de/serialisers

* rename serde_utils to serde_snapshot

* move call to accounts_db.generate_index() back down to context_accountsdb_from_stream()

* update version 1.1.1 to 1.2.0
remove nested use of serialize_bytes

* cleanups

* Add back measurement of account storage entry serialization.
Remove construction of Vec and HashMap temporaries during serialization.

* consolidate serialisation test cases into serde_snapshot.
clean up leakage of implementation details in serde_snapshot.

* move short term / legacy snapshot code into child module

* add serialize_iter_as_tuple

* preliminary integration of following commit

commit 6d58b73c47294bfb93465d5a83cd2175660b6e6d
Author: Ryo Onodera <ryoqun@gmail.com>
Date:   Wed May 20 14:02:02 2020 +0900

    Confine snapshot 1.1 relic to versioned codepath

* refactored serde_snapshot, rustfmt
legacy accounts_db format now "owns" both leading u64s, legacy bank_rc format has none

* reduce type complexity (clippy)
2020-05-23 02:54:24 +09:00
Kristofer Peterson
58ef02f02b 9951 clippy errors in the test suite (#10030)
automerge
2020-05-15 09:35:43 -07:00
Ryo Onodera
1eb40c3fe0 Introduce eager rent collection (#9527)
* Switch AccountsIndex.account_maps from HashMap to BTreeMap

* Introduce eager rent collection

* Start to add tests

* Avoid too short eager rent collection cycles

* Add more tests

* Add more tests...

* Refacotr!!!!!!

* Refactoring follow up

* More tiny cleanups

* Don't rewrite 0-lamport accounts to be deterministic

* Refactor a bit

* Do hard fork, restore tests, and perf. mitigation

* Fix build...

* Refactor and add switch over for testnet (TdS)

* Use to_be_bytes

* cleanup

* More tiny cleanup

* Rebase cleanup

* Set Bank::genesis_hash when resuming from snapshot

* Reorder fns and clean ups

* Better naming and commenting

* Yet more naming clarifications

* Make prefix width strictly uniform for 2-base partition_count

* Fix typo...

* Revert cluster-dependent gate

* kick ci?

* kick ci?

* kick ci?
2020-05-13 16:22:14 +09:00
Justin Starry
5cc252d471 Remove hash field from account (#9915) 2020-05-12 23:39:46 +08:00
Ryo Onodera
2ac50177a6 Include account.owner into account hash (#9917)
automerge
2020-05-07 13:01:11 -07:00
carllin
3442f36f8a Repair alternate versions of dead slots (#9805)
Co-authored-by: Carl <carl@solana.com>
2020-05-05 14:07:21 -07:00
Michael Vines
230df0ec0c Upgrade to Rust 1.43.0 (#9754) 2020-04-29 18:02:05 -07:00
Ryo Onodera
9918539229 Introduce type alias Ancestors (#9699)
* Introduce type alias AncestorList

* Rename AncestorList => Ancestors
2020-04-27 11:07:03 +09:00
sakridge
217828a849 Reduce accounts_db prints (#9469) 2020-04-15 09:05:54 -07:00
sakridge
99655206c8 Calculate account refs fix (#9447) 2020-04-11 12:52:10 -07:00
Ryo Onodera
b28ec430e4 Introduce background stale AppendVec shrink mechanism (#9219)
* Introduce background AppendVec shrink mechanism

* Support ledger tool

* Clean up

* save

* save

* Fix CI

* More clean up

* Add tests

* Clean up yet more

* Use account.hash...

* Fix typo....

* Add comment

* Rename accounts_cleanup_service
2020-04-06 17:30:23 +09:00
sakridge
b1771b92ec Calculate ref counts earlier to prevent bad clean (#9147) 2020-03-29 14:42:34 -07:00
Ryo Onodera
2623c71ed3 Use type aliases/resulting var names consistently (#9060) 2020-03-26 13:08:56 +09:00
Justin Starry
ef3af104ae Return appropriate error for invalid program account (#9047)
automerge
2020-03-25 13:23:05 -07:00
sakridge
45348b2c83 Remove accounts unwrap (#9062)
automerge
2020-03-25 10:21:30 -07:00
sakridge
c530fbd22b Remove thread-priority crate which is not cross-platform (#9023) 2020-03-23 12:18:52 -07:00
sakridge
4b397d15b3 Accounts cleanup service and perf improvements (#8799)
* Use atomic for ref count instead of taking rwlock

* Accounts cleanup service

* Review comments
2020-03-23 08:50:23 -07:00
Michael Vines
88ba8439fc Add frozen account support (#8989)
automerge
2020-03-22 11:10:04 -07:00
sakridge
4dd0367136 Rwlock storage opt (#9006)
* Remove unecessary account paths rwlock

* Remove path rwlock in accounts_db and optimize storage critical section
2020-03-22 10:04:03 -07:00
sakridge
3ea556bc24 Drop storage lock (#8997) 2020-03-21 13:37:52 -07:00
Ryo Onodera
4bbf09f582 Enable conservative out-of-bound snapshot cleaning (#8811)
* Enable conservative out-of-bound snapshot cleaning

* Add tests
2020-03-13 14:44:00 +09:00
Ryo Onodera
952cd38b7b Avoid early clean and bad snapshot by ref-counting (#8724)
* Avoid early clean and bad snapshot by ref-counting

* Add measure

* Clean ups

* clean ups
2020-03-13 14:14:37 +09:00
Michael Vines
f146c92e88 Always and fully normalize stored 0-lamport accts. (#8657) 2020-03-05 09:14:40 -07:00
sakridge
b084c1d437 Remove accounts hack and correctly restore accounts store counts (#8569)
* Remove accounts hack and correctly restore append-vec counts

* Add test
2020-03-03 20:48:55 -08:00