gaspersic/solana
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Avoid full-range compactions with periodic filtered b.g. ones (backport #16697) (#17741)

Browse Source 
* Avoid full-range compactions with periodic filtered b.g. ones (#16697)

* Update rocksdb to v0.16.0

* Promote the infrequent and important log to info!

* Force background compaction by ttl without manual compaction

* Fix test

* Support no compaction mode in test_ledger_cleanup_compaction

* Fix comment

* Make compaction_interval customizable

* Avoid major compaction with periodic filtering...

* Adress lazy_static, special cfs and range check

* Clean up a bit and add comment

* Add comment

* More comments...

* Config code cleanup

* Add comment

* Use .conflicts_with()

* Nullify unneeded delete_range ops for special CFs

* Some clean ups

* Clarify the locking intention

* Ensure special CFs' consistency with PurgeType::CompactionFilter

* Fix comment

* Fix bad copy paste

* Fix various types...

* Don't use tuples

* Add a unit test for compaction_filter

* Fix typo...

* Remove flag and just use new behavior always

* Fix wrong condition negation...

* Doc. about no set_last_purged_slot in purge_slots

* Write a test and fix off-by-one bug....

* Apply suggestions from code review

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* Follow up to github review suggestions

* Fix line-wrapping

* Fix conflict

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
(cherry picked from commit 1f97b2365f)

# Conflicts:
#	ledger/src/blockstore_db.rs

* Fix conflict

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
This commit is contained in:
mergify[bot]
2021-06-04 14:38:02 +00:00
committed by GitHub
parent ae42413d57
commit 5e7db52087
9 changed files with 644 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
core/src/ledger_cleanup_service.rs
View File

@@ -207,11 +207,25 @@ impl LedgerCleanupService {
);
let mut purge_time = Measure::start("purge_slots");
blockstore.purge_slots(
purge_first_slot,
lowest_cleanup_slot,
PurgeType::PrimaryIndex,
PurgeType::CompactionFilter,
);
// Update only after purge operation.
// Safety: This value can be used by compaction_filters shared via Arc<AtomicU64>.
// Compactions are async and run as a multi-threaded background job. However, this
// shouldn't cause consistency issues for iterators and getters because we have
// already expired all affected keys (older than or equal to lowest_cleanup_slot)
// by the above `purge_slots`. According to the general RocksDB design where SST
// files are immutable, even running iterators aren't affected; the database grabs
// a snapshot of the live set of sst files at iterator's creation.
// Also, we passed the PurgeType::CompactionFilter, meaning no delete_range for
// transaction_status and address_signatures CFs. These are fine because they
// don't require strong consistent view for their operation.
blockstore.set_max_expired_slot(lowest_cleanup_slot);
purge_time.stop();
info!("{}", purge_time);