This consists of:
- syscalls
- passing return data from invoked to invoker
- printing to stable log
- rust and C SDK changes
(cherry picked from commit 53b47b87b2)
* Cost Model to limit transactions which are not parallelizeable (#16694)
* * Add following to banking_stage:
1. CostModel as immutable ref shared between threads, to provide estimated cost for transactions.
2. CostTracker which is shared between threads, tracks transaction costs for each block.
* replace hard coded program ID with id() calls
* Add Account Access Cost as part of TransactionCost. Account Access cost are weighted differently between read and write, signed and non-signed.
* Establish instruction_execution_cost_table, add function to update or insert instruction cost, unit tested. It is read-only for now; it allows Replay to insert realtime instruction execution costs to the table.
* add test for cost_tracker atomically try_add operation, serves as safety guard for future changes
* check cost against local copy of cost_tracker, return transactions that would exceed limit as unprocessed transaction to be buffered; only apply bank processed transactions cost to tracker;
* bencher to new banking_stage with max cost limit to allow cost model being hit consistently during bench iterations
* replay stage feed back program cost (#17731)
* replay stage feeds back realtime per-program execution cost to cost model;
* program cost execution table is initialized into empty table, no longer populated with hardcoded numbers;
* changed cost unit to microsecond, using value collected from mainnet;
* add ExecuteCostTable with fixed capacity for security concern, when its limit is reached, programs with old age AND less occurrence will be pushed out to make room for new programs.
* investigate system performance test degradation (#17919)
* Add stats and counter around cost model ops, mainly:
- calculate transaction cost
- check transaction can fit in a block
- update block cost tracker after transactions are added to block
- replay_stage to update/insert execution cost to table
* Change mutex on cost_tracker to RwLock
* removed cloning cost_tracker for local use, as the metrics show clone is very expensive.
* acquire and hold locks for block of TXs, instead of acquire and release per transaction;
* remove redundant would_fit check from cost_tracker update execution path
* refactor cost checking with less frequent lock acquiring
* avoid many Transaction_cost heap allocation when calculate cost, which
is in the hot path - executed per transaction.
* create hashmap with new_capacity to reduce runtime heap realloc.
* code review changes: categorize stats, replace explicit drop calls, concisely initiate to default
* address potential deadlock by acquiring locks one at time
* Persist cost table to blockstore (#18123)
* Add `ProgramCosts` Column Family to blockstore, implement LedgerColumn; add `delete_cf` to Rocks
* Add ProgramCosts to compaction excluding list alone side with TransactionStatusIndex in one place: `excludes_from_compaction()`
* Write cost table to blockstore after `replay_stage` replayed active banks; add stats to measure persist time
* Deletes program from `ProgramCosts` in blockstore when they are removed from cost_table in memory
* Only try to persist to blockstore when cost_table is changed.
* Restore cost table during validator startup
* Offload `cost_model` related operations from replay main thread to dedicated service thread, add channel to send execute_timings between these threads;
* Move `cost_update_service` to its own module; replay_stage is now decoupled from cost_model.
* log warning when channel send fails (#18391)
* Aggregate cost_model into cost_tracker (#18374)
* * aggregate cost_model into cost_tracker, decouple it from banking_stage to prevent accidental deadlock. * Simplified code, removed unused functions
* review fixes
* update ledger tool to restore cost table from blockstore (#18489)
* update ledger tool to restore cost model from blockstore when compute-slot-cost
* Move initialize_cost_table into cost_model, so the function can be tested and shared between validator and ledger-tool
* refactor and simplify a test
* manually fix merge conflicts
* Per-program id timings (#17554)
* more manual fixing
* solve a merge conflict
* featurize cost model
* more merge fix
* cost model uses compute_unit to replace microsecond as cost unit
(#18934)
* Reject blocks for costs above the max block cost (#18994)
* Update block max cost limit to fix performance regession (#19276)
* replace function with const var for better readability (#19285)
* Add few more metrics data points (#19624)
* periodically report sigverify_stage stats (#19674)
* manual merge
* cost model nits (#18528)
* Accumulate consumed units (#18714)
* tx wide compute budget (#18631)
* more manual merge
* ignore zerorize drop security
* - update const cost values with data collected by #19627
- update cost calculation to closely proposed fee schedule #16984
* add transaction cost histogram metrics (#20350)
* rebase to 1.7.15
* add tx count and thread id to stats (#20451)
each stat reports and resets when slot changes
* remove cost_model feature_set
* ignore vote transactions from cost model
Co-authored-by: sakridge <sakridge@gmail.com>
Co-authored-by: Jeff Biseda <jbiseda@gmail.com>
Co-authored-by: Jack May <jack@solana.com>
* Add separate vote processing tpu port
* Add feature to send to tpu vote port
* Add vote rejecting sigverify mode
* use packet.meta.is_simple_vote_tx in place of deserialization
* consolidate code that identifies vote tx atcommon path for cpu and gpu
* new key for feature set
* banking forward tpu vote
* add tpu vote port to dockerfile and other review changes
* Simplify thread id compare
* fix a test; updated cluster_info ABI change
Co-authored-by: Tao Zhu <tao@solana.com>
* Check seed length before trying to cal program address (#19699)
(cherry picked from commit 778b2adbea)
* resolve conflicts
Co-authored-by: Jack May <jack@solana.com>
* stake: Advance `credits_observed` on activation epoch (#19309)
* stake: Advance `credits_observed` on activation epoch
* Add test for merging stakes just after activation
(cherry picked from commit 2c3bdedea3)
# Conflicts:
# runtime/src/bank.rs
* Fix merge issues
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
* stake: Remove v2 program references (#19308)
* stake: Remove v2 program references
* Remove stake v2 feature, along with stake rewrite
(cherry picked from commit 73aa004c59)
# Conflicts:
# runtime/src/bank.rs
* Fix merge conflict
* Fix uses of old `stake` function
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
* system-program: Move lamports == 0 check on transfers
* Address feedback
* Update stake split to explicitly allocate + assign
* Update stake tests referring to split instruction
* Revert whitespace
* Update split instruction index in test
* Remove unnecessary `assign_with_seed` from `split_with_seed`
* Fix stake instruction parser
* Update test to allow splitting into account with lamports
(cherry picked from commit 8f5e773caf)
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
* Add missing ProgramError to InstructionError mappings
* add note
* Clarify process of adding new program error
(cherry picked from commit 83b9a046d1)
Co-authored-by: Jack May <jack@solana.com>
* Always bail if program modifies a ro account (#17569)
(cherry picked from commit a3240aebde)
* resolve conflicts
* nudge
Co-authored-by: Jack May <jack@solana.com>
* feat: allow SetAuthority in CLI calls
* chore: clippy match_like_matches_macro
* chore: clippy match_like_matches_macro
* chore: rename CLI to CPI
* chore: move check for cpi authorised instruction to syscalls
* chore: add set_upgrade_authority cpi test
* chore: assert upgrade authority was changed
* feat: gate set_upgrade_authority via cpi with a feature
* chore: move feature to the end of the list
* chore: remove white spaces
* chore: remove white spaces
* chore: update comment to rerun build
* Drop write lock on sysvars
* adds env var for demoting sysvar write lock demotion
* moves demote logic to is_writable
* feature gates sysvar write lock demotion
* adds builtins to write lock demotion
* adds system program id to builtins
* adds Feature111...
* adds an abi-freeze test
* mvines set of builtin program keys
Co-authored-by: Michael Vines <mvines@gmail.com>
* update tests
* adds bpf loader keys
* Add test sysvar
* Plumb demote_sysvar to is_writable
* more plumbing of demote_sysvar_write_locks to is_writable
* patches test_program_bpf_instruction_introspection
* hard codes demote_sysvar_write_locks to false for serialization/encoding methods
* Revert "hard codes demote_sysvar_write_locks to false for serialization/encoding methods"
This reverts commit ae3e2d2e777437bddd753933097a210dcbc1b1fc.
* change the hardcoded ones to demote_sysvar_write_locks=true
* Use data_as_mut_slice
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
