* tracks erasure coding shreds' indices explicitly (#21822)
The indices for erasure coding shreds are tied to data shreds:
https://github.com/solana-labs/solana/blob/90f41fd9b/ledger/src/shred.rs#L921
However with the upcoming changes to erasure schema, there will be more
erasure coding shreds than data shreds and we can no longer infer coding
shreds indices from data shreds.
The commit adds constructs to track coding shreds indices explicitly.
(cherry picked from commit 65d59f4ef0)
# Conflicts:
# core/benches/retransmit_stage.rs
# core/benches/shredder.rs
# core/src/broadcast_stage/broadcast_duplicates_run.rs
# core/src/broadcast_stage/broadcast_fake_shreds_run.rs
# core/src/broadcast_stage/fail_entry_verification_broadcast_run.rs
# core/src/window_service.rs
# ledger/src/blockstore.rs
# ledger/src/shred.rs
# ledger/tests/shred.rs
* removes mergify merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* removes next_shred_index from return value of entries to shreds api (#21961)
next-shred-index is already readily available from returned data shreds.
The commit simplifies the api for upcoming changes to erasure coding
schema which will require explicit tracking of indices for coding shreds
as well as data shreds.
(cherry picked from commit 89d66c3210)
# Conflicts:
# core/benches/shredder.rs
# core/src/broadcast_stage/broadcast_duplicates_run.rs
# core/src/broadcast_stage/broadcast_fake_shreds_run.rs
# core/src/broadcast_stage/fail_entry_verification_broadcast_run.rs
# core/src/broadcast_stage/standard_broadcast_run.rs
# gossip/src/duplicate_shred.rs
# ledger/src/blockstore.rs
# ledger/src/shred.rs
# ledger/tests/shred.rs
* removes mergify merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* uses Option<Slot> for SlotMeta.parent_slot (#21808)
SlotMeta.parent_slot for the head of a detached chain of slots is
unknown and that is indicated by u64::MAX which lacks type-safety:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L203-L205
The commit changes the type to Option<Slot>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.
(cherry picked from commit 8d980f07ba)
# Conflicts:
# core/src/repair_generic_traversal.rs
# ledger-tool/src/main.rs
# ledger/src/blockstore.rs
* removes mergify merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* uses Option<u64> for SlotMeta.last_index (#21775)
SlotMeta.last_index may be unknown and current code is using u64::MAX to
indicate that:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L169-L174
This lacks type-safety and can introduce bugs if not always checked for
Several instances of slot_meta.last_index + 1 are also subject to
overflow.
This commit updates the type to Option<u64>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.
(cherry picked from commit e08139f949)
# Conflicts:
# core/src/repair_generic_traversal.rs
# ledger/src/blockstore.rs
# ledger/src/blockstore_meta.rs
* removes mergify merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* adds back ErasureMeta::first_coding_index field (#21623)
https://github.com/solana-labs/solana/pull/16646
removed first_coding_index since the field is currently redundant and
always equal to fec_set_index.
However, with upcoming changes to erasure coding schema, this will no
longer be the same as fec_set_index and so requires a separate field to
represent.
(cherry picked from commit 49ba09b333)
# Conflicts:
# ledger/src/blockstore.rs
* removes mergify merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Problem
Blockstore currently removes erasure shreds if the data shreds are
successfully recovered on insert, which is an issue if we want to
serve coding shreds over repair.
Summary of Changes
This diff keeps all coding shreds even on successful recovery and
changes change the signature of prev_inserted_codes to immutable
reference to ensure its immunity.
Fixes#20968
(cherry picked from commit 3aa49e2c69)
Co-authored-by: Yueh-Hsuan Chiang <93241502+yhchiang-sol@users.noreply.github.com>
* consolidate execute timings to a common module
* Add execute metrics
* Add metrics for executor creation
* Use saturating_add_assign macro
Co-authored-by: Trent Nelson <trent@solana.com>
Co-authored-by: Carl Lin <carl@solana.com>
* implements copy-on-write for staked-nodes (#19090)
Bank::staked_nodes and Bank::epoch_staked_nodes redundantly clone
staked-nodes HashMap even though an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/a9014cece/runtime/src/vote_account.rs#L77
This commit implements copy-on-write semantics for staked-nodes by
wrapping the underlying HashMap in Arc<...>.
(cherry picked from commit f302774cf7)
# Conflicts:
# runtime/src/bank.rs
# runtime/src/stakes.rs
# runtime/src/vote_account.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* get_signatures_for_address does not correctly account for result sets that span local and Bigtable sources (#22115)
* get_signatures_for_address does not correctly account for result sets that span Blockstore and Bigtable.
This causes Bigtable to return `RowNotFound` until the new tx is uploaded.
Check that `before` exists in Bigtable, and if not, set it to `None` to return the full data set.
References #21442Closes#22110
* Differentiate between before sig not found and no newer signatures
* Dedupe bigtable results to account for potential upload race
Co-authored-by: Tyera Eulberg <tyera@solana.com>
(cherry picked from commit bac6821e19)
# Conflicts:
# ledger/src/blockstore.rs
* Fix conflicts
Co-authored-by: Omar Kilani <omar.kilani@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
* makes solana_runtime::vote_account::VoteAccount private
VoteAccount is an implementation detail, and should not be public.
Only ArcVoteAccount is the public type.
(cherry picked from commit 1403eaeefc)
# Conflicts:
# runtime/src/vote_account.rs
* renames solana_runtime::vote_account::VoteAccount
Rename:
VoteAccount -> VoteAccountInner # the private type
ArcVoteAccount -> VoteAccount # the public type
(cherry picked from commit 00e5e12906)
# Conflicts:
# core/src/progress_map.rs
# ledger/src/blockstore_processor.rs
# ledger/src/staking_utils.rs
# runtime/src/bank.rs
# runtime/src/epoch_stakes.rs
# runtime/src/serde_snapshot/tests.rs
# runtime/src/stakes.rs
# runtime/src/vote_account.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* removes redundant args from Shredder::try_recovery (#21226)
Shredder::try_recovery is already taking a Vec<Shred> as an argument. All the
other arguments are embedded in the shreds, and are so redundant.
(cherry picked from commit 5fb0ab9d00)
# Conflicts:
# ledger/src/shred.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* adds back position field to coding-shred-header (#21600)
https://github.com/solana-labs/solana/pull/17004
removed position field from coding-shred-header because as it stands the
field is redundant and unused.
However, with the upcoming changes to erasure coding schema this field
will no longer be redundant and needs to be populated.
(cherry picked from commit cd17f63d81)
# Conflicts:
# core/src/window_service.rs
# ledger/src/blockstore.rs
# ledger/src/shred.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* Use max_complete_transaction_status_slot in BigTableUploadService
* Use node root to limit BigTableUploadService
(cherry picked from commit 9e043d2055)
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
