* Make account shrink configurable #17544 (#17778)
1. Added both options for measuring space usage using total accounts usage and for individual store shrink ratio using an enum. Validator CLI options: --accounts-shrink-optimize-total-space and --accounts-shrink-ratio
2. Added code for selecting candidates based on total usage in a separate function select_candidates_by_total_usage
3. Added unit tests for the new functions added
4. The default implementations is kept at 0.8 shrink ratio with --accounts-shrink-optimize-total-space set to true
Fixes#17544
(cherry picked from commit 269d995832)
# Conflicts:
# core/tests/snapshots.rs
# ledger/src/bank_forks_utils.rs
# runtime/src/accounts_db.rs
# runtime/src/snapshot_utils.rs
* fix some merge errors
Co-authored-by: Lijun Wang <83639177+lijunwangs@users.noreply.github.com>
Co-authored-by: Jeff Washington (jwash) <wash678@gmail.com>
* verify bank hash on startup with ledger tool option (#17939)
(cherry picked from commit f558b9b6bf)
# Conflicts:
# core/tests/snapshots.rs
# ledger/src/bank_forks_utils.rs
# runtime/src/snapshot_utils.rs
* fix merge errors
Co-authored-by: Jeff Washington (jwash) <75863576+jeffwashington@users.noreply.github.com>
Co-authored-by: Jeff Washington (jwash) <wash678@gmail.com>
* Account for possibility of cache flush in load()
* More cleaning
* More cleaning
* Remove unused method and some comment cleaning
* Fix typo
* Make the detected impossible purge race panic()!
* Finally revert to original .expect()
* Fix typos...
* Add assertion for max_root for easier reasoning
* Reframe races with LoadHint as possible opt.
* Fix test
* Make race bug tests run longer for less flaky
* Delay the clone-in-lock slow path even for RPC
* Make get_account panic-free & add its onchain ver.
* Fix rebase conflicts...
* Clean up
* Clean up comment
* Revert fn name change
* Fix flaky test...
* fmt...
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
* Add load_largest_accounts bench
* Check lamports before address filter
* Use BinaryHeap, add Accounts test
* Use pubkey reference in the min-heap
Also, flatten code with early returns
Co-authored-by: Greg Fitzgerald <greg@solana.com>
* Simpler cap by including sysvars and native programs
* Fix tests
* Add comment
* revert some unrelated code
* Update test_bank_update_sysvar_account for cap.
* Test cap. for add_native_program using new helper
* Improve the cap adjustment with new tests
* Fix typo...
* Adjust test for improved code coverage
* Rename simpler_capitalization => simple_capitalization
* More rename and bonus commenting
* Check bank capitalization
* Simplify and unify capitalization calculation
* Improve and add tests
* Avoid overflow and inhibit automatic restart
* Fix test
* Tweak checked sum for cap. and add tests
* Fix broken build after merge conflicts..
* Rename to ClusterType
* Rename confusing method
* Clarify comment
* Verify cap. in rent and inflation tests
Co-authored-by: Stephen Akridge <sakridge@gmail.com>
* accounts_clean: Convert stack dependency calculation with iterative
* optimize clean with by creating a reverse-lookup hashset of the affected
keys
* Add dependency bench
reduce bench
* Huge clean
* Focus bench on squash
Squash performance does not depend on adding a small number accounts. It mainly depends on total number of accounts that need to be hashed during freeze operation. New bank means a clone of accounts db, so we don't get previous errors in log.
* Fix fmt and add slot counter
* runtime: Add bench for accounts::hash_internal_state
* fixup! cargo fmt
* fixup! cargo clippy
* fixup! Use a more representitive number of accounts
* fixup! More descriptive name for accounts creation helper
