gaspersic/solana
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
16,785 Commits 81 Branches 357 Tags
8ee5fbc5c0b51a1f70b4831baed1ffe2cc65c133
Commit Graph

36 Commits

Author SHA1 Message Date
Michael Vines
b8837c04ec Reformat imports to a consistent style for imports 
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
 2021-12-03 09:19:13 -08:00
Tyera Eulberg
a3bef2e537 Fix shreds-to-hours/days estimations (#19477) 2021-08-30 13:16:06 -06:00
Alexander Meißner
6514096a67 chore: cargo +nightly clippy --fix -Z unstable-options 2021-06-18 10:42:46 -07:00
Ryo Onodera
1f97b2365f Avoid full-range compactions with periodic filtered b.g. ones (#16697) 
* Update rocksdb to v0.16.0

* Promote the infrequent and important log to info!

* Force background compaction by ttl without manual compaction

* Fix test

* Support no compaction mode in test_ledger_cleanup_compaction

* Fix comment

* Make compaction_interval customizable

* Avoid major compaction with periodic filtering...

* Adress lazy_static, special cfs and range check

* Clean up a bit and add comment

* Add comment

* More comments...

* Config code cleanup

* Add comment

* Use .conflicts_with()

* Nullify unneeded delete_range ops for special CFs

* Some clean ups

* Clarify the locking intention

* Ensure special CFs' consistency with PurgeType::CompactionFilter

* Fix comment

* Fix bad copy paste

* Fix various types...

* Don't use tuples

* Add a unit test for compaction_filter

* Fix typo...

* Remove flag and just use new behavior always

* Fix wrong condition negation...

* Doc. about no set_last_purged_slot in purge_slots

* Write a test and fix off-by-one bug....

* Apply suggestions from code review

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* Follow up to github review suggestions

* Fix line-wrapping

* Fix conflict

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
 2021-05-28 16:42:56 +09:00
Michael Vines
b242f82696 Reduce test-validator ledger size 2021-04-05 08:37:29 -07:00
sakridge
5b8f046c67 More configurable rocksdb compaction (#15213) 
rocksdb compaction can cause long stalls, so
make it more configurable to try and reduce those stalls
and also to coordinate between multiple nodes to not induce
stall at the same time.
 2021-02-14 10:16:30 -08:00
sakridge
583cec922b Remove ledger purge batching (#10830) 2020-06-29 14:44:35 -07:00
Michael Vines
aa6832964c ledger_cleanup_service: compact at a slower rate than purging (#10414) 2020-06-04 21:06:06 -07:00
Tyera Eulberg
eee9a08376 Purge TransactionStatus and AddressSignatures exactly from ledger-tool (#10358) 
* Add failing test

* Add execution path to purge primary-index columns exactly

* Fail gracefully if older TransactionStatus rocksdb keys are present

* Remove columns_empty check for special columns

* Move blockstore purge methods to submodule

* Remove unused column empty check
 2020-06-02 19:49:31 -06:00
Michael Vines
156387aba4 LedgerCleanupService no longer causes an OOM and actually purges (#10199) 
* cleanup_ledger() now services new_root_receiver while purging
* purge_slots() now fully deletes before compacting
* Add ledger pruning grafana graph
 2020-05-24 21:41:54 -07:00
sakridge
a9b82cf95b Enable disk metrics (#10009) 2020-05-12 15:24:39 -07:00
sakridge
fa20963b93 Revert shred fs (#9712) 
* Revert "Untar is called for shred archives that do not exist. (#9565)"

This reverts commit 729cb5eec6.

* Revert "Dont insert shred payload into rocksdb (#9366)"

This reverts commit 5ed39de8c5.
 2020-04-24 15:04:23 -07:00
anatoly yakovenko
5ed39de8c5 Dont insert shred payload into rocksdb (#9366) 
automerge
 2020-04-16 18:20:55 -07:00
sakridge
a8b8c2f438 Move slow compaction test to integration (#9506) 2020-04-15 11:54:03 -07:00
sakridge
be77bdef12 Allow lower shred count (#9410) 2020-04-09 16:36:44 -07:00
sakridge
a61ddb6f61 max_ledger_slots -> max_ledger_shreds (#9198) 
automerge
 2020-03-31 17:21:19 -07:00
sakridge
73e99cc513 Ledger cleanup fixes (#9131) 
* Fix purging happening every slot when cleanup service is not started at slot 0
* Purge by shred count instead of slots since slots can have variable
number of shreds
 2020-03-30 19:02:12 -07:00
Michael Vines
fb98df76b7 4x DEFAULT_MAX_LEDGER_SLOTS to give nodes 3 hours of slots to repair from (#8388) 
automerge
 2020-02-21 15:04:02 -08:00
carllin
4ffd7693d6 Add lock to make sure slot-based locktree calls are safe (#7993) 2020-01-28 13:45:41 -08:00
Michael Vines
ebf6e1c0e9 --limit-ledger-size now accepts an optional slot count value 2020-01-20 14:20:30 -07:00
Greg Fitzgerald
b5dba77056 Rename blocktree to blockstore (#7757) 
automerge
 2020-01-13 13:13:52 -08:00
Greg Fitzgerald
a707c9410e More thiserror (#7183) 
* Less solana_core::result. Module now private.

* Drop solana_core::result dependency from a few more modules

* Fix warning

* Cleanup

* Fix typo
 2020-01-02 20:50:43 -07:00
Sagar Dhawan
6a9005645a Update "limit-ledger-size" to use DeleteRange for much faster deletes (#7515) 
* Update "limit-ledger-size" to use DeleteRange for much faster deletes

* Update core/src/ledger_cleanup_service.rs

Co-Authored-By: Michael Vines <mvines@gmail.com>

* Rewrite more idiomatically

* Move max_ledger_slots to a fn for clippy

* Remove unused import

* Detect when all columns have been purged and fix a bug in deletion

* Check that more than 1 column is actually deleted

* Add helper to test that ledger meets minimum slot bounds

* Remove manual batching of deletes

* Refactor to keep some N slots older than the highest root

* Define MAX_LEDGER_SLOTS that ledger_cleanup_service will try to keep around

* Refactor compact range
 2019-12-18 11:50:09 -08:00
Sunny Gleason
323673c3c0 Add compact_cf calls to reclaim storage during ledger slot purge (#7264) 2019-12-18 10:29:46 -05:00
Sunny Gleason
83218c479a code cleanup, storage_size() was Option<u64>, now Result<u64> (#7424) 2019-12-12 14:55:30 -05:00
Sunny Gleason
06415de8ee change blocktree*::storage_size() to return Option<u64> to handle live fs changes (#7401) 2019-12-10 19:12:49 -05:00
Sunny Gleason
c00216e3be feat: ledger size and cleanup metrics (#7335) 2019-12-06 22:32:45 -05:00
carllin
43e2301e2c Fix roots overrunning broadcast (#6884) 
* Add trusted pathway for insert_shreds to avoid checks
 2019-11-14 00:32:07 -08:00
Greg Fitzgerald
a3a830e1ab Delete Service trait (#6921) 2019-11-13 11:12:09 -07:00
Greg Fitzgerald
30a08f4282 Cleanup ledger macros (#6916) 
automerge
 2019-11-13 07:14:09 -08:00
Greg Fitzgerald
5468be2ef9 Add solana-ledger crate (#6415) 
automerge
 2019-10-18 09:28:51 -07:00
Jack May
e8d88f3237 Split SDK's timing.rs (#5823) 2019-09-06 14:30:56 -07:00
Pankaj Garg
3d3b03a123 Verify signature of recovered shred before adding them to blocktree (#5811) 
* Verify signature of recovered shred before adding them to blocktree

* fix failing tests, and review comments
 2019-09-05 18:20:30 -07:00
Pankaj Garg
3b0d48e3b8 Remove blocktree blob references (#5691) 
* Remove blocktree blob references

* fixes and cleanup

* replace uninitialized() call with MaybeUninit

* fix bench
 2019-09-03 21:32:51 -07:00
Sagar Dhawan
535df0026d Fixes for Blocktree space amplification and slot deletion (#5266) 
* Fixes for Blocktree space amplification and slot deletion
 2019-07-24 17:28:08 -07:00
Sagar Dhawan
a07b17b9b5 Drop older slots in the ledger (#5188) 
* Add facility to delete blocktree columns in range

* Add ledger cleanup service

* Add local_cluster test
 2019-07-20 13:13:55 -07:00