* shrink batches when over 80% of the space is wasted (#23066)
* shrink batches when over 80% of the space is wasted
(cherry picked from commit 83d31c9e65)
# Conflicts:
# core/benches/sigverify_stage.rs
# core/src/sigverify_stage.rs
# perf/src/sigverify.rs
* fixup!
Co-authored-by: anatoly yakovenko <anatoly@solana.com>
* improves sigverify discard_excess_packets performance (#22577)
As shown by the added benchmark, current code does worse if there is a
spam address plus a lot of unique addresses.
on current master:
test bench_packet_discard_many_senders ... bench: 1,997,960 ns/iter (+/- 103,715)
test bench_packet_discard_mixed_senders ... bench: 14,256,116 ns/iter (+/- 534,865)
test bench_packet_discard_single_sender ... bench: 1,306,809 ns/iter (+/- 61,992)
with this commit:
test bench_packet_discard_many_senders ... bench: 1,644,025 ns/iter (+/- 83,715)
test bench_packet_discard_mixed_senders ... bench: 1,089,789 ns/iter (+/- 86,324)
test bench_packet_discard_single_sender ... bench: 955,234 ns/iter (+/- 55,953)
(cherry picked from commit dcf44d2523)
# Conflicts:
# core/src/sigverify_stage.rs
* removes mergify merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
avoid bad performance of remove(0) for a single sender
(cherry picked from commit 49443406fd)
# Conflicts:
# core/src/sigverify_stage.rs
Co-authored-by: sakridge <sakridge@gmail.com>
The indices for erasure coding shreds are tied to data shreds:
https://github.com/solana-labs/solana/blob/90f41fd9b/ledger/src/shred.rs#L921
However with the upcoming changes to erasure schema, there will be more
erasure coding shreds than data shreds and we can no longer infer coding
shreds indices from data shreds.
The commit adds constructs to track coding shreds indices explicitly.
(cherry picked from commit 65d59f4ef0)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
next-shred-index is already readily available from returned data shreds.
The commit simplifies the api for upcoming changes to erasure coding
schema which will require explicit tracking of indices for coding shreds
as well as data shreds.
(cherry picked from commit 89d66c3210)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
- decouple cost_model from cost_tracker; allowing one cost_model
instance being shared within a validator;
- update cost_model api to calculate_cost(&self...)->transaction_cost
* Add separate vote processing tpu port
* Add feature to send to tpu vote port
* Add vote rejecting sigverify mode
* use packet.meta.is_simple_vote_tx in place of deserialization
* consolidate code that identifies vote tx atcommon path for cpu and gpu
* new key for feature set
* banking forward tpu vote
* add tpu vote port to dockerfile and other review changes
* Simplify thread id compare
* fix a test; updated cluster_info ABI change
Co-authored-by: Tao Zhu <tao@solana.com>
Co-authored-by: sakridge <sakridge@gmail.com>
* windows: Make solana-test-validator work
The important changes to get this going on Windows:
* ledger lock needs to be done on a file instead of the directory
* IPC service needs to use the Windows pipe naming scheme
* always disable the JIT
* file logging not possible yet because we can't redirect stderr,
but this will change once env_logger fixes the pipe output target!
* Integrate review feedback
* Move transaction sanitization earlier in the pipeline
* Renamed HashedTransaction to SanitizedTransaction
* Implement deref for sanitized transaction
* bring back process_transactions test method
* Use sanitized transactions for cost model calculation
Broadcast stage and retransmit stage should arrange nodes on turbine
broadcast tree in exactly same order. Additionally any changes to this
ordering (e.g. updating how unstaked nodes are handled) requires feature
gating to keep the cluster in sync.
Current implementation is scattered out over several public methods and
exposes too much of implementation details (e.g. usize indices into
peers vector) which makes code changes and checking for feature
activations more difficult.
This commit encapsulates turbine peer computations into a new struct,
and only exposes two public methods, get_broadcast_peer and
get_retransmit_peers, for call-sites.
* Add stats and counter around cost model ops, mainly:
- calculate transaction cost
- check transaction can fit in a block
- update block cost tracker after transactions are added to block
- replay_stage to update/insert execution cost to table
* Change mutex on cost_tracker to RwLock
* removed cloning cost_tracker for local use, as the metrics show clone is very expensive.
* acquire and hold locks for block of TXs, instead of acquire and release per transaction;
* remove redundant would_fit check from cost_tracker update execution path
* refactor cost checking with less frequent lock acquiring
* avoid many Transaction_cost heap allocation when calculate cost, which
is in the hot path - executed per transaction.
* create hashmap with new_capacity to reduce runtime heap realloc.
* code review changes: categorize stats, replace explicit drop calls, concisely initiate to default
* address potential deadlock by acquiring locks one at time
* replay stage feeds back realtime per-program execution cost to cost model;
* program cost execution table is initialized into empty table, no longer populated with hardcoded numbers;
* changed cost unit to microsecond, using value collected from mainnet;
* add ExecuteCostTable with fixed capacity for security concern, when its limit is reached, programs with old age AND less occurrence will be pushed out to make room for new programs.
* Create solana-poh crate
* Move BigTableUploadService to solana-ledger
* Add solana-rpc to workspace
* Move dependencies to solana-rpc
* Move remaining rpc modules to solana-rpc
* Single use statement solana-poh
* Single use statement solana-rpc
* * Add following to banking_stage:
1. CostModel as immutable ref shared between threads, to provide estimated cost for transactions.
2. CostTracker which is shared between threads, tracks transaction costs for each block.
* replace hard coded program ID with id() calls
* Add Account Access Cost as part of TransactionCost. Account Access cost are weighted differently between read and write, signed and non-signed.
* Establish instruction_execution_cost_table, add function to update or insert instruction cost, unit tested. It is read-only for now; it allows Replay to insert realtime instruction execution costs to the table.
* add test for cost_tracker atomically try_add operation, serves as safety guard for future changes
* check cost against local copy of cost_tracker, return transactions that would exceed limit as unprocessed transaction to be buffered; only apply bank processed transactions cost to tracker;
* bencher to new banking_stage with max cost limit to allow cost model being hit consistently during bench iterations
* Move gossip modules to solana-gossip
* Update Protocol abi digest due to move
* Move gossip benches and hook up CI
* Remove unneeded Result entries
* Single use statements