* improves sigverify discard_excess_packets performance (#22577)
As shown by the added benchmark, current code does worse if there is a
spam address plus a lot of unique addresses.
on current master:
test bench_packet_discard_many_senders ... bench: 1,997,960 ns/iter (+/- 103,715)
test bench_packet_discard_mixed_senders ... bench: 14,256,116 ns/iter (+/- 534,865)
test bench_packet_discard_single_sender ... bench: 1,306,809 ns/iter (+/- 61,992)
with this commit:
test bench_packet_discard_many_senders ... bench: 1,644,025 ns/iter (+/- 83,715)
test bench_packet_discard_mixed_senders ... bench: 1,089,789 ns/iter (+/- 86,324)
test bench_packet_discard_single_sender ... bench: 955,234 ns/iter (+/- 55,953)
(cherry picked from commit dcf44d2523)
# Conflicts:
# core/benches/sigverify_stage.rs
# core/src/sigverify_stage.rs
* removes mergify merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* discard time histogram
* closer to the if
* update
(cherry picked from commit e616a7ebfc)
# Conflicts:
# core/src/sigverify_stage.rs
Co-authored-by: anatoly yakovenko <anatoly@solana.com>
* consolidate execute timings to a common module
* Add execute metrics
* Add metrics for executor creation
* Use saturating_add_assign macro
Co-authored-by: Trent Nelson <trent@solana.com>
Co-authored-by: Carl Lin <carl@solana.com>
avoid bad performance of remove(0) for a single sender
(cherry picked from commit 49443406fd)
# Conflicts:
# core/src/sigverify_stage.rs
Co-authored-by: sakridge <sakridge@gmail.com>
* implements copy-on-write for staked-nodes (#19090)
Bank::staked_nodes and Bank::epoch_staked_nodes redundantly clone
staked-nodes HashMap even though an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/a9014cece/runtime/src/vote_account.rs#L77
This commit implements copy-on-write semantics for staked-nodes by
wrapping the underlying HashMap in Arc<...>.
(cherry picked from commit f302774cf7)
# Conflicts:
# runtime/src/bank.rs
# runtime/src/stakes.rs
# runtime/src/vote_account.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* Split up local cluster tests into separate CI steps (#22295)
* Split up local cluster tests into separate CI steps
* Update buildkite-pipeline.sh
(cherry picked from commit 0e1afcbb26)
# Conflicts:
# local-cluster/tests/local_cluster.rs
* resolve conflicts
Co-authored-by: Justin Starry <justin@solana.com>
* Flip iter operations to keep associated address/header/packets together (#22245)
Flip iter operations to keep associated address/header/packets together
Before this change, if cast_socket_addr() returned a None for any
address/header pair, the subsequent zip() would misalign the
address/header pair and packet. So, this change zips all three together,
then does filter_map() so keep things aligned.
Additionally, compute total_size inline to avoid running through packets
a second time.
(cherry picked from commit 20b61e28b6)
# Conflicts:
# streamer/src/recvmmsg.rs
* removes backport merge conflicts
Co-authored-by: steviez <steven@solana.com>
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* removes epoch_authorized_voters from VoteTracker (#22192)
https://github.com/solana-labs/solana/pull/22169
verifies authorized-voter early on in vote-listener pipeline; and so
VoteTracker no longer needs to maintain and check for epoch authorized
voters.
(cherry picked from commit 69d71f8f86)
# Conflicts:
# core/src/cluster_info_vote_listener.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* re-calibrate limit based on mainnet data, see issue #21917
(cherry picked from commit d743c2917c)
# Conflicts:
# runtime/src/block_cost_limits.rs
* set secp256k1 cost similar to sigverify
(cherry picked from commit a2a7e91ad6)
* removes backport merge conflicts
Co-authored-by: Tao Zhu <tao@solana.com>
* checks for authorized voter early on in the vote-listener pipeline (#22169)
Before votes are verified that they are signed by the authorized voter,
they might be dropped in verified-vote-packets code. If there are
enough many spam votes from unauthorized voters, this may potentially
drop valid votes but keep the false ones.
https://github.com/solana-labs/solana/blob/57986f982/core/src/verified_vote_packets.rs#L165-L168
(cherry picked from commit c0c6038654)
# Conflicts:
# core/src/cluster_info_vote_listener.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* get_signatures_for_address does not correctly account for result sets that span local and Bigtable sources (#22115)
* get_signatures_for_address does not correctly account for result sets that span Blockstore and Bigtable.
This causes Bigtable to return `RowNotFound` until the new tx is uploaded.
Check that `before` exists in Bigtable, and if not, set it to `None` to return the full data set.
References #21442Closes#22110
* Differentiate between before sig not found and no newer signatures
* Dedupe bigtable results to account for potential upload race
Co-authored-by: Tyera Eulberg <tyera@solana.com>
(cherry picked from commit bac6821e19)
# Conflicts:
# ledger/src/blockstore.rs
* Fix conflicts
Co-authored-by: Omar Kilani <omar.kilani@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
* makes solana_runtime::vote_account::VoteAccount private
VoteAccount is an implementation detail, and should not be public.
Only ArcVoteAccount is the public type.
(cherry picked from commit 1403eaeefc)
# Conflicts:
# runtime/src/vote_account.rs
* renames solana_runtime::vote_account::VoteAccount
Rename:
VoteAccount -> VoteAccountInner # the private type
ArcVoteAccount -> VoteAccount # the public type
(cherry picked from commit 00e5e12906)
# Conflicts:
# core/src/progress_map.rs
# ledger/src/blockstore_processor.rs
# ledger/src/staking_utils.rs
# runtime/src/bank.rs
# runtime/src/epoch_stakes.rs
# runtime/src/serde_snapshot/tests.rs
# runtime/src/stakes.rs
# runtime/src/vote_account.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* removes redundant args from Shredder::try_recovery (#21226)
Shredder::try_recovery is already taking a Vec<Shred> as an argument. All the
other arguments are embedded in the shreds, and are so redundant.
(cherry picked from commit 5fb0ab9d00)
# Conflicts:
# ledger/src/shred.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* new net-stats require a new table (#21996)
(cherry picked from commit 3fe942ab30)
# Conflicts:
# core/src/system_monitor_service.rs
* merge
Co-authored-by: Jeff Biseda <jbiseda@gmail.com>
* Add json support for feature sets; also print output after feature list
* Move stringifying into Display implementation
(cherry picked from commit dcd2854829)
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
- Adds feature reject_section_virtual_address_file_offset_mismatch.
- Adds feature start_verify_shift32_imm.
- Enables enable_symbol_and_section_labels only in the rbpf-cli.
* Moves syscall_registry into the rbpf Executable constructor.
* Adds the reject_unresolved_syscalls flag which is only set when deploying programs via the CLI.
* Add Accountsdb plugin documentations (#21746)
Add the public facing documentation about the plugin framework: explaining the interface, how to load plugin and the example PostgreSQL plugin implementation.
Updated the rust documentation for the plugin interfaces for accounts and slot.
This changes are targeted for v1.8. Information about transactions will be updated later.
* fmt issue
* SDK: Add stdlib.h include to pull in `abort()` (#21700)
(cherry picked from commit 923720f529)
# Conflicts:
# sdk/bpf/c/inc/sol/assert.h
* Fix merge conflict
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
* adds back position field to coding-shred-header (#21600)
https://github.com/solana-labs/solana/pull/17004
removed position field from coding-shred-header because as it stands the
field is redundant and unused.
However, with the upcoming changes to erasure coding schema this field
will no longer be redundant and needs to be populated.
(cherry picked from commit cd17f63d81)
# Conflicts:
# core/src/window_service.rs
# ledger/src/blockstore.rs
# ledger/src/shred.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
- Adds feature reject_section_virtual_address_file_offset_mismatch.
- Adds feature start_verify_shift32_imm.
- Enables enable_symbol_and_section_labels only in the rbpf-cli.
* Moves syscall_registry into the rbpf Executable constructor.
* Adds the reject_unresolved_syscalls flag which is only set when deploying programs via the CLI.
* Ledger-tool: only require ledger dir when necessary (#21575)
* Don't canonicalize ledger_path unless ledger_path will be used
* Single use statement
(cherry picked from commit 2b269dbe0e)
# Conflicts:
# ledger-tool/src/main.rs
* Fix conflicts
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
Otherwise getHealth fails if account hashes are not propagated.
(cherry picked from commit 9886366977)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
For writing tests, it is often desirable to create a certain situation
that would be tedious, or even impossible to create by sending
individual transactions.
For example, a common attack vector on Solana is to create a copy of
some existing account that stores configuration data, but manipulate the
data stored there, and then to call a program and pass in the
manipulated account, instead of the real account.
If one wants to test this, one option is to write a program that you can
call to write arbitrary data into a new account account (and possibly
change its owner), then include that program in the test context, and
send a transaction to call it. This is extremely tedious, and developers
are not going to bother doing it. I myself would rather fork
solana-program-test to add this `set_account` method, than to write that
program. Having a “god mode” method to just write an account, lowers the
barrier to writing comprehensive tests.
A second reason for introducing this method, is defense in depth. There
may be states of the bank that are not reachable *yet* by only sending
transactions, but that you might want to test against either way. For
example, right now, the balance of a stake account cannot decrease
without going through the stake program. But what if Solana were to
introduce slashing in the future, and you want to ensure your program is
robust against decreases in stake account balance? Right now there is no
way to test this, but by introducing this “god mode” to write accounts,
the situation becomes testable.
(cherry picked from commit e7fa412465)
Using dashes prevents the code from building, which is annoying
if you quickly want to test some hack.
With underscore we get the same effect but the code still builds.
(cherry picked from commit 46aed1b7f9)
Co-authored-by: Benno Fünfstück <benno.fuenfstueck@gmail.com>
* bank: factor tx log getter out to method on `TransactionLogCollector`
(cherry picked from commit efedb55705)
# Conflicts:
# runtime/src/bank.rs
* bank: Add failing test for `TransactionLogCollector::get_logs_for_address()`
(cherry picked from commit dafdc15dfe)
# Conflicts:
# runtime/src/bank.rs
* bank: don't panic when getting logs from malformed TransactionLogCollector
(cherry picked from commit e558ad4c71)
* bank: `TransactionLogCollector` - only push address mappings after pushing a log message
(cherry picked from commit 41cc7af7cd)
Co-authored-by: Trent Nelson <trent@solana.com>
* Use max_complete_transaction_status_slot in BigTableUploadService
* Use node root to limit BigTableUploadService
(cherry picked from commit 9e043d2055)
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Moves syscall_registry into the rbpf Executable constructor.
* Adds the reject_unresolved_syscalls flag which is only set when deploying programs via the CLI.
(cherry picked from commit 8d5c04e257)
Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
* Add function for changing thread's nice value
Linux only.
* Add validator option to change niceness of snapshot packager thread
* Add validator option to change niceness of RPC server threads
Fixes https://github.com/solana-labs/solana/issues/14556
* Run `./scripts/cargo-for-all-lock-files.sh tree`
* report pubsub stats every 2s (#21192)
(cherry picked from commit 7659a2edc2)
* remove use of Duration::MAX
Co-authored-by: Jeff Biseda <jbiseda@gmail.com>
Summary of Changes
Added the reference postgresql.conf
Prepare slot update statement to reduce overhead in updating slot
Support custom connection string
Allow the plugin to panic on replication issues to ensure consistency
* Add missing websocket methods to rust RPC PubSub client (#21065)
- Added accountSubscribe, programSubscribe, slotSubscribe and rootSubscribe to rust RpcClient
- Removed duplication on cleanup threads
- Moved RPCVote from rpc/ to client/rpc_response
(cherry picked from commit a0f9e0e8ee)
# Conflicts:
# Cargo.lock
# client-test/Cargo.toml
# core/tests/client.rs
* Fix conflicts
* Make test result not depend on TestValidator setup
Co-authored-by: Manuel Gil <manugildev@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
* Different error if block is not available
* Add slot to error message
* Make and use helper function
* Check finalized path as well
Co-authored-by: Tyera Eulberg <tyera@solana.com>
(cherry picked from commit 700e42d556)
Co-authored-by: sakridge <sakridge@gmail.com>
* Add compute budget program as a noop (#20992)
(cherry picked from commit 1e2bef76e3)
# Conflicts:
# sdk/src/feature_set.rs
* resolve conflicts
Co-authored-by: Jack May <jack@solana.com>
* - cost_tracker is data member of a bank, it can report metrics when bank is frozen (#20802)
- removed cost_tracker_stats and histogram
- move stats reporting outside of bank freeze
(cherry picked from commit c2bfce90b3)
# Conflicts:
# Cargo.lock
# core/src/banking_stage.rs
# core/src/replay_stage.rs
# core/src/tvu.rs
# ledger-tool/src/main.rs
# programs/bpf/Cargo.lock
# runtime/Cargo.toml
# runtime/src/cost_tracker.rs
* manual fix merge conflicts
Co-authored-by: Tao Zhu <82401714+taozhu-chicago@users.noreply.github.com>
Co-authored-by: Tao Zhu <tao@solana.com>
* tpu-client: Move `send_messages_with_spinner` from program (#20960)
We have too many ways of sending transactions, and too many
reimplementations of the same logic all over the place.
The program deploy logic and stake-o-matic currently make the
most use of the TPU client, so this merges their implementations into
one place to be reused by both. Yay for consolidation!
(cherry picked from commit 5f7b60576f)
# Conflicts:
# cli/src/program.rs
# client/src/mock_sender.rs
* Fix merge issues, use older APIs
* Update mock sender fee to match block height
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
* Force a recent version of the openssl crate to allow this to build on M1 macs
(cherry picked from commit 920159fc63)
# Conflicts:
# Cargo.lock
# programs/bpf_loader/Cargo.toml
* Run cargo check
(cherry picked from commit 8efc577374)
# Conflicts:
# programs/bpf/Cargo.lock
* Resolve merge conflicts
Co-authored-by: Matt Wilde <matthewcwilde@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
* Accountsdb plugin postgres -- bulk insertion at startup (#20763)
Use bulk insertion to Postgres at startup to reduce time taken for initial snapshot restore for postgres plugin. Avoid duplicate writes of accounts at startup. Doing account plugin notification and indexing in parallel.
Improved error handling for postgres plugin to show the real db issues for debug purpose
Added more metrics for postgres plugin.
Refactored plugin centric code out to a sub module from accounts_db and added unit tests
* Fixed the unit test failures
* Update old GitHub links in 'Developing with Rust' docs
* exclude_entrypoint -> no-entrypoint in 'Developing with Rust'
(cherry picked from commit f729dec321)
Co-authored-by: Brian Anderson <andersrb@gmail.com>
# Problem
The initial proposal ruled out implementing BigTable queries for
the `getBlockTime` RPC, but then it was implemented a couple months
later. Indicating that the functionality was never implemented in
the "implemented-proposals" document is a little confusing, so let's
bring the document in line with what actually happened. 🦾
# Summary of Changes
Remove the blurb about how `getBlockTime` was going to be deprecated
and add it to the list of calls that didn't yet support BigTable
queries at the time the proposal was written.
(cherry picked from commit 0c7bade0b2)
Co-authored-by: Arthur Burkart <arthur@presynce.com>
* - make cost_tracker a member of bank, remove shared instance from TPU; (#20627)
- decouple cost_model from cost_tracker; allowing one cost_model
instance being shared within a validator;
- update cost_model api to calculate_cost(&self...)->transaction_cost
(cherry picked from commit 7496b5784b)
# Conflicts:
# core/src/banking_stage.rs
# ledger-tool/src/main.rs
# runtime/src/bank.rs
# runtime/src/cost_model.rs
# runtime/src/cost_tracker.rs
* manual fix merge conflicts
Co-authored-by: Tao Zhu <82401714+taozhu-chicago@users.noreply.github.com>
Co-authored-by: Tao Zhu <tao@solana.com>
Allow u16::MAX to be specified for the instruction index. This makes it
possible to specify the current instruction, so it is not necessary to
know the instruction number.
Solang requires a method for verify ed25519 signatures. Add a new
builtin program at address Ed25519SigVerify111111111111111111111111111
which takes any number of ed25519 signature, public key, and message.
If any of the signatures fails to verify, an error is returned.
The changes for the web3.js package will go into another commit, since
the tests test against a released solana node. Adding web3.js ed25519
testing will break CI.
(cherry picked from commit b491354e51)
Conflicts:
Cargo.lock
Cargo.toml
programs/bpf/Cargo.lock
runtime/Cargo.toml
sdk/src/feature_set.rs
sdk/src/transaction.rs
sdk/src/transaction/sanitized.rs
Rather than pass in individual features, pass in the entire feature set
so that we can add the ed25519 program feature in a later commit.
(cherry picked from commit 0f62771f42)
Conflicts:
banks-server/src/banks_server.rs
core/src/banking_stage.rs
programs/secp256k1/src/lib.rs
rpc/src/rpc.rs
runtime/src/bank.rs
sdk/src/transaction.rs
sdk/src/transaction/sanitized.rs
* Improve program-test process_transaction() speed by reducing sleep duration in banks-server (#20508)
* banks_server: Reduce sleep duration for local server
This speeds up process_transaction_with_commitment_and_context()
and thus most program tests by a lot.
* Plumb tick duration through poh config and signature polling
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
(cherry picked from commit bea181eba9)
# Conflicts:
# banks-server/src/banks_server.rs
# program-test/src/lib.rs
* Fix merge issues
Co-authored-by: Christian Kamm <ckamm@delightful-solutions.de>
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
* - move cost tracker into bank, so each bank has its own cost tracker; (#20527)
- move related modules to runtime
(cherry picked from commit 005d6863fd)
# Conflicts:
# Cargo.lock
# core/benches/banking_stage.rs
# core/src/banking_stage.rs
# core/src/lib.rs
# core/src/tvu.rs
# ledger-tool/src/main.rs
# ledger/src/blockstore_processor.rs
# programs/bpf/Cargo.lock
# runtime/Cargo.toml
# runtime/src/cost_model.rs
* manual fix merge conflicts
Co-authored-by: Tao Zhu <82401714+taozhu-chicago@users.noreply.github.com>
Co-authored-by: Tao Zhu <tao@solana.com>
This consists of:
- syscalls
- passing return data from invoked to invoker
- printing to stable log
- rust and C SDK changes
(cherry picked from commit 53b47b87b2)
When making a snapshot archive, we used to shell out and call `tar -S`
for sparse file support. The tar crate supports sparse files, so no
need to do this anymore.
Fixes#10860
(cherry picked from commit 68cc71409e)
# Conflicts:
# runtime/src/snapshot_utils.rs
* Accountsdb stream plugin improvement (#20419)
Support using connection pooling and use multiple threads to do Postgres db operations. The performance is improved from 1500 RPS to 40,000 RPS measured during validator start.
Support multiple plugins at the same time.
* Fixed a fmt issue
The docker image fails with:
/usr/bin/solana-run.sh: line 66: ./fetch-spl.sh: No such file or directory
In the solana docker image, scripts/run.sh is copied to
/usr/bin/solana-run.sh and fetch-spl.sh to /usr/bin/fetch-spl.sh. This
means that the line:
cd "$(dirname "$0")/.."
means we're doing a "cd /usr", which means we can't find fetch-spl.sh or
spl-genesis-args.sh (i.e., the error above).
(cherry picked from commit 2762f6f96f)
Co-authored-by: Sean Young <sean@mess.org>
* adds metrics for number of nodes vs number of pubkeys (#20512)
(cherry picked from commit 0da661de62)
# Conflicts:
# gossip/src/cluster_info_metrics.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Merge AccountsDb plugin framework to v1.8 (#20518)
Summary of Changes
Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows
Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.
The code comprises 4 major parts:
accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
* Cost Model to limit transactions which are not parallelizeable (#16694)
* * Add following to banking_stage:
1. CostModel as immutable ref shared between threads, to provide estimated cost for transactions.
2. CostTracker which is shared between threads, tracks transaction costs for each block.
* replace hard coded program ID with id() calls
* Add Account Access Cost as part of TransactionCost. Account Access cost are weighted differently between read and write, signed and non-signed.
* Establish instruction_execution_cost_table, add function to update or insert instruction cost, unit tested. It is read-only for now; it allows Replay to insert realtime instruction execution costs to the table.
* add test for cost_tracker atomically try_add operation, serves as safety guard for future changes
* check cost against local copy of cost_tracker, return transactions that would exceed limit as unprocessed transaction to be buffered; only apply bank processed transactions cost to tracker;
* bencher to new banking_stage with max cost limit to allow cost model being hit consistently during bench iterations
* replay stage feed back program cost (#17731)
* replay stage feeds back realtime per-program execution cost to cost model;
* program cost execution table is initialized into empty table, no longer populated with hardcoded numbers;
* changed cost unit to microsecond, using value collected from mainnet;
* add ExecuteCostTable with fixed capacity for security concern, when its limit is reached, programs with old age AND less occurrence will be pushed out to make room for new programs.
* investigate system performance test degradation (#17919)
* Add stats and counter around cost model ops, mainly:
- calculate transaction cost
- check transaction can fit in a block
- update block cost tracker after transactions are added to block
- replay_stage to update/insert execution cost to table
* Change mutex on cost_tracker to RwLock
* removed cloning cost_tracker for local use, as the metrics show clone is very expensive.
* acquire and hold locks for block of TXs, instead of acquire and release per transaction;
* remove redundant would_fit check from cost_tracker update execution path
* refactor cost checking with less frequent lock acquiring
* avoid many Transaction_cost heap allocation when calculate cost, which
is in the hot path - executed per transaction.
* create hashmap with new_capacity to reduce runtime heap realloc.
* code review changes: categorize stats, replace explicit drop calls, concisely initiate to default
* address potential deadlock by acquiring locks one at time
* Persist cost table to blockstore (#18123)
* Add `ProgramCosts` Column Family to blockstore, implement LedgerColumn; add `delete_cf` to Rocks
* Add ProgramCosts to compaction excluding list alone side with TransactionStatusIndex in one place: `excludes_from_compaction()`
* Write cost table to blockstore after `replay_stage` replayed active banks; add stats to measure persist time
* Deletes program from `ProgramCosts` in blockstore when they are removed from cost_table in memory
* Only try to persist to blockstore when cost_table is changed.
* Restore cost table during validator startup
* Offload `cost_model` related operations from replay main thread to dedicated service thread, add channel to send execute_timings between these threads;
* Move `cost_update_service` to its own module; replay_stage is now decoupled from cost_model.
* log warning when channel send fails (#18391)
* Aggregate cost_model into cost_tracker (#18374)
* * aggregate cost_model into cost_tracker, decouple it from banking_stage to prevent accidental deadlock. * Simplified code, removed unused functions
* review fixes
* update ledger tool to restore cost table from blockstore (#18489)
* update ledger tool to restore cost model from blockstore when compute-slot-cost
* Move initialize_cost_table into cost_model, so the function can be tested and shared between validator and ledger-tool
* refactor and simplify a test
* manually fix merge conflicts
* Per-program id timings (#17554)
* more manual fixing
* solve a merge conflict
* featurize cost model
* more merge fix
* cost model uses compute_unit to replace microsecond as cost unit
(#18934)
* Reject blocks for costs above the max block cost (#18994)
* Update block max cost limit to fix performance regession (#19276)
* replace function with const var for better readability (#19285)
* Add few more metrics data points (#19624)
* periodically report sigverify_stage stats (#19674)
* manual merge
* cost model nits (#18528)
* Accumulate consumed units (#18714)
* tx wide compute budget (#18631)
* more manual merge
* ignore zerorize drop security
* - update const cost values with data collected by #19627
- update cost calculation to closely proposed fee schedule #16984
* add transaction cost histogram metrics (#20350)
* rebase to 1.7.15
* add tx count and thread id to stats (#20451)
each stat reports and resets when slot changes
* remove cost_model feature_set
* ignore vote transactions from cost model
Co-authored-by: sakridge <sakridge@gmail.com>
Co-authored-by: Jeff Biseda <jbiseda@gmail.com>
Co-authored-by: Jack May <jack@solana.com>
if there is no newline at the end of the file, this export is glued to the rest of the code and generates a syntax error like this
```bash
if [ -f ~/.git-completion.bash ]; then
. ~/.git-completion.bash
fiexport PATH="/Users/user/.local/share/solana/install/active_release/bin:$PATH"
```
(cherry picked from commit 87c0d8d9e7)
Co-authored-by: OleG <emptystamp@gmail.com>
* Deploy error is buffer is too small
* missing file
(cherry picked from commit de8331eeaf)
# Conflicts:
# cli/tests/fixtures/noop.so
Co-authored-by: Jack May <jack@solana.com>
* Add separate vote processing tpu port
* Add feature to send to tpu vote port
* Add vote rejecting sigverify mode
* use packet.meta.is_simple_vote_tx in place of deserialization
* consolidate code that identifies vote tx atcommon path for cpu and gpu
* new key for feature set
* banking forward tpu vote
* add tpu vote port to dockerfile and other review changes
* Simplify thread id compare
* fix a test; updated cluster_info ABI change
Co-authored-by: Tao Zhu <tao@solana.com>
* removes Slot from TransmitShreds (#19327)
An earlier version of the code was funneling through stakes along with
shreds to broadcast:
https://github.com/solana-labs/solana/blob/b67ffab37/core/src/broadcast_stage.rs#L127
This was changed to only slots as stakes computation was pushed further
down the pipeline in:
https://github.com/solana-labs/solana/pull/18971
However shreds themselves embody which slot they belong to. So pairing
them with slot is redundant and adds rooms for bugs should they become
inconsistent.
(cherry picked from commit 1deb4add81)
# Conflicts:
# core/benches/cluster_info.rs
# core/src/broadcast_stage.rs
# core/src/broadcast_stage/broadcast_duplicates_run.rs
# core/src/broadcast_stage/fail_entry_verification_broadcast_run.rs
# core/src/broadcast_stage/standard_broadcast_run.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* removes packet-count metrics from retransmit stage
Working towards sending shreds (instead of packets) to retransmit stage
so that shreds recovered from erasure codes are as well retransmitted.
Following commit will add these metrics back to window-service, earlier
in the pipeline.
(cherry picked from commit bf437b0336)
# Conflicts:
# core/src/retransmit_stage.rs
* adds packet/shred count stats to window-service
Adding back these metrics from the earlier commit which removed them
from retransmit stage.
(cherry picked from commit 8198a7eae1)
* removes erroneous uses of Arc<...> from retransmit stage
(cherry picked from commit 6e413331b5)
# Conflicts:
# core/src/retransmit_stage.rs
# core/src/tvu.rs
* sends shreds (instead of packets) to retransmit stage
Working towards channelling through shreds recovered from erasure codes
to retransmit stage.
(cherry picked from commit 3efccbffab)
# Conflicts:
# core/src/retransmit_stage.rs
* returns completed-data-set-info from insert_data_shred
instead of opaque (u32, u32) which are then converted to
CompletedDataSetInfo at the call-site.
(cherry picked from commit 3c71670bd9)
# Conflicts:
# ledger/src/blockstore.rs
* retransmits shreds recovered from erasure codes
Shreds recovered from erasure codes have not been received from turbine
and have not been retransmitted to other nodes downstream. This results
in more repairs across the cluster which is slower.
This commit channels through recovered shreds to retransmit stage in
order to further broadcast the shreds to downstream nodes in the tree.
(cherry picked from commit 7a8807b8bb)
# Conflicts:
# core/src/retransmit_stage.rs
# core/src/window_service.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* removes raw indexing from streamer (#19183)
Raw indexing is verbose and error-prone. This same code had an indexing
bug causing validator nodes panic just a few months ago:
https://github.com/solana-labs/solana/commit/482b8c6be
(cherry picked from commit 8229a4fbf6)
# Conflicts:
# streamer/Cargo.toml
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* sends slots (instead of stakes) through broadcast flow
Current broadcast code is computing stakes for each slot before sending
them down the channel:
https://github.com/solana-labs/solana/blob/049fb0417/core/src/broadcast_stage/standard_broadcast_run.rs#L208-L228https://github.com/solana-labs/solana/blob/0cf52e206/core/src/broadcast_stage.rs#L342-L349
Since the stakes are a function of epoch the slot belongs to (and so
does not necessarily change from one slot to another), forwarding the
slot itself would allow better caching downstream.
In addition we need to invalidate the cache if the epoch changes (which
the current code does not do), and that requires to know which slot (and
so epoch) current broadcasted shreds belong to:
https://github.com/solana-labs/solana/blob/19bd30262/core/src/broadcast_stage/standard_broadcast_run.rs#L332-L344
(cherry picked from commit 44b11154ca)
# Conflicts:
# core/src/broadcast_stage/broadcast_duplicates_run.rs
# core/src/broadcast_stage/standard_broadcast_run.rs
* implements cluster-nodes cache
Cluster nodes are cached keyed by the respective epoch from which stakes
are obtained, and so if epoch changes cluster-nodes will be recomputed.
A time-to-live eviction policy is enforced to refresh entries in case
gossip contact-infos are updated.
(cherry picked from commit ecc1c7957f)
* uses cluster-nodes cache in retransmit stage
The new cluster-nodes cache will:
* ensure cluster-nodes are recalculated if the epoch (and so the epoch
staked nodes) changes.
* encapsulate time-to-live eviction policy.
(cherry picked from commit 30bec3921e)
* uses cluster-nodes cache in broadcast-stage
* Current caching mechanism does not update cluster-nodes when the epoch
(and so epoch staked nodes) changes:
https://github.com/solana-labs/solana/blob/19bd30262/core/src/broadcast_stage/standard_broadcast_run.rs#L332-L344
* Additionally, the cache update has a concurrency bug in which the
thread which does compare_and_swap may be blocked when it tries to
obtain the write-lock on cache, while other threads will keep running
ahead with the outdated cache (since the atomic timestamp is already
updated).
In the new ClusterNodesCache, entries are keyed by epoch, and so if
epoch changes cluster-nodes will be recalculated. The time-to-live
eviction policy is also encapsulated and rigidly enforced.
(cherry picked from commit aa32738dd5)
# Conflicts:
# core/src/broadcast_stage/broadcast_duplicates_run.rs
# core/src/broadcast_stage/fail_entry_verification_broadcast_run.rs
# core/src/broadcast_stage/standard_broadcast_run.rs
* unifies cluster-nodes computation & caching across turbine stages
Broadcast-stage is using epoch_staked_nodes based on the same slot that
shreds belong to:
https://github.com/solana-labs/solana/blob/049fb0417/core/src/broadcast_stage/standard_broadcast_run.rs#L208-L228https://github.com/solana-labs/solana/blob/0cf52e206/core/src/broadcast_stage.rs#L342-L349
But retransmit-stage is using bank-epoch of the working-bank:
https://github.com/solana-labs/solana/blob/19bd30262/core/src/retransmit_stage.rs#L272-L289
So the two are not consistent at epoch boundaries where some nodes may
have a working bank (or similarly a root bank) lagging other nodes. As a
result the node which obtains a packet may construct turbine broadcast
tree inconsistently with its parent node in the tree and so some packets
may fail to reach all nodes in the tree.
(cherry picked from commit 50d0e830c9)
* adds fallback & metric for when epoch staked-nodes are none
(cherry picked from commit fb69f45f14)
* allows only one thread to update cluster-nodes cache entry for an epoch
If two threads simultaneously call into ClusterNodesCache::get for the
same epoch, and the cache entry is outdated, then both threads recompute
cluster-nodes for the epoch and redundantly overwrite each other.
This commit wraps ClusterNodesCache entries in Arc<Mutex<...>>, so that
when needed only one thread does the computations to update the entry.
(cherry picked from commit eaf927cf49)
* falls back on working-bank if root-bank::epoch-staked-nodes is none
bank.get_leader_schedule_epoch(shred_slot)
is one epoch after epoch_schedule.get_epoch(shred_slot).
At epoch boundaries, shred is already one epoch after the root-slot. So
we need epoch-stakes 2 epochs ahead of the root. But the root bank only
has epoch-stakes for one epoch ahead, and as a result looking up epoch
staked-nodes from the root-bank fails.
To be backward compatible with the current master code, this commit
implements a fallback on working-bank if epoch staked-nodes obtained
from the root-bank is none.
(cherry picked from commit e4be00fece)
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Rationalize usage of sendmmsg(2). Skip packets which failed to send and track failures.
(cherry picked from commit ae5ad5cf9b)
Co-authored-by: Jeff Biseda <jbiseda@gmail.com>
* Revert "temporarily disable new audit"
This reverts commit 3dfbd95ddc.
* Bump version of zeroize_derive from v1.0.0 to v1.2.0
(cherry picked from commit 0c62a6fe3f)
Co-authored-by: Justin Starry <justin@solana.com>
* sigverify to identify and mark simple vote transaction (#20021)
* check vote tx at get_packet_offsets to cover both cpu and gpu paths
* add pubkey_len to PacketOffsets to reduce the redundant bytes counting
* allow vote to have 1 or 2 sigs (#20082)
* windows: Make solana-test-validator work (#20099)
* windows: Make solana-test-validator work
The important changes to get this going on Windows:
* ledger lock needs to be done on a file instead of the directory
* IPC service needs to use the Windows pipe naming scheme
* always disable the JIT
* file logging not possible yet because we can't redirect stderr,
but this will change once env_logger fixes the pipe output target!
* Integrate review feedback
(cherry picked from commit 567f30aa1a)
# Conflicts:
# validator/src/bin/solana-test-validator.rs
# validator/src/lib.rs
# validator/src/main.rs
* Fix merge conflicts
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
* Add stable_log output when a program is loaded as native code instead of BPF
(cherry picked from commit 34f5020457)
* Add ProgramTest::add_builtin_program()
This permits the unit testing of builtin programs in the ProgramTest environment
(cherry picked from commit 830ca369f1)
Co-authored-by: Michael Vines <mvines@gmail.com>
* move `./run.sh` into `./scripts`
(cherry picked from commit 92e343da26)
* add some guidance in place of `./run.sh`
(cherry picked from commit 33de7b856f)
Co-authored-by: Trent Nelson <trent@solana.com>
* client: Add retry logic on Pubsub 429s (#19990)
(cherry picked from commit e9b066d497)
* Use exponential backoff for older version of tungstenite
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
* Optimize RPC pubsub for multiple clients with the same subscription (#18943)
* reimplement rpc pubsub with a broadcast queue
* update tests for new pubsub implementation
* fix: fix review suggestions
* chore(rpc): add additional pubsub metrics
* integrate max subscriptions check into SubscriptionTracker to reduce locking
* separate subscription control from tracker
* limit memory usage of items in pubsub broadcast queue, improve error handling
* add more pubsub metrics
* add final count metrics to pubsub
* add metric for total number of subscriptions
* fix small review suggestions
* remove by_params from SubscriptionTracker and add node_progress_watchers map instead
* add subscription tracker tests
* add metrics for number of pubsub notifications as a counter
* ignore clippy lint in TokenCounter
* fix underflow in token counter
* reduce queue capacity in pubsub tests
* fix(rpc): fix test timeouts
* fix race in account subscription test
* Add RpcSubscriptions::new_for_tests
Co-authored-by: Pavel Strakhov <p.strakhov@iconic.vc>
Co-authored-by: Nikita Podoliako <n.podoliako@zubr.io>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
(cherry picked from commit 65227f44dc)
# Conflicts:
# Cargo.lock
# core/Cargo.toml
# core/src/replay_stage.rs
# core/src/validator.rs
# replica-node/src/replica_node.rs
# rpc/Cargo.toml
* Fix conflicts (and standardize naming to make future subscription backports easier
Co-authored-by: Pavel Strakhov <ri@idzaaus.org>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
* rpc: performance fix for getProgramAccounts
The accounts were gradually pushed into a vector, which produced
significant slowdowns for very large responses.
* rpc: rewrite loops using iterators
Co-authored-by: Christian Kamm <ckamm@delightful-solutions.de>
(cherry picked from commit f1bbf1d8b0)
Co-authored-by: Christian Kamm <mail@ckamm.de>
In many Linux distros such as NixOS, the directory in which packages are
installed is assumed to be read-only. To work around this, it is
expected that the filepaths which packaged CLI tools take in are able to
be freely configured through either 1) command-line flags, 2)
environment variables, or 3) a configuration file.
In this commit, environment variables 'BPF_SDK_PATH' and 'BPF_OUT_PATH',
which map respectively to command-line flags '--bpf-sdk-path' and
'--bpf-out-dir', are now handled in cargo-build-bpf.
Additionally, given that arbitrary filepaths may now be set in which the
BPF SDK is located, the requirement in which
'$BPF_SDK_PATH/dependencies/bpf-tools' must strictly be a symbolic link
to the directory '$HOME/.cache/solana/${bpf-tools.version}/bpf-tools has
been relaxed.
Ideally, the directory in which bpf-tools is expected to be downloaded
to and stored should be configurable. Though, this commit serves as a
temporary fix which enables NixOS users to now start being able to build
applications with the Solana SDK.
Co-authored-by: Kenta Iwasaki <kenta@lithdew.net>
* Check seed length before trying to cal program address (#19699)
(cherry picked from commit 778b2adbea)
* resolve conflicts
Co-authored-by: Jack May <jack@solana.com>
* cli: show upgradeable program accounts (#19431)
(cherry picked from commit 57bbbb83a4)
* resolve conflicts
* nudge
Co-authored-by: Jack May <jack@solana.com>
The parameter is now a required third argument. This is because authorized
withdrawer should never be the same as vote account keypair or validator
identity keypair for security reasons.
Added a --allow-unsafe-authorized-withdrawer to override this restriction if
necessary.
(cherry picked from commit e288459cf2)
* stake: Advance `credits_observed` on activation epoch (#19309)
* stake: Advance `credits_observed` on activation epoch
* Add test for merging stakes just after activation
(cherry picked from commit 2c3bdedea3)
# Conflicts:
# runtime/src/bank.rs
* Fix merge issues
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
In order to debug this panic on the clusters:
panicked at 'assertion failed: (vote_index as usize) <
MAX_LOCKOUT_HISTORY', core/src/cluster_info.rs:1012:9
(cherry picked from commit d7051b0d21)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* Only encode data with <128 bytes in bs58
* Use same MAX_BS58_BYTES const in rpc
* Fix test
* Spell out base
Co-authored-by: Tyera Eulberg <tyera@solana.com>
(cherry picked from commit 967746abbf)
Co-authored-by: sakridge <sakridge@gmail.com>
* Add note about ASK keyword to paper wallet doc
Per https://github.com/solana-labs/solana/issues/17325#issuecomment-844434585, this change adds a note to the paper wallet docs mentioning the use of the `ASK` keyword.
Also specifes that the `solana-keygen pubkey prompt://` command returns a derived bip44 base address.
(I'm new to Solana development and this was a point of confusion for me, only resolved by finding this issue)
* Update docs/src/wallet-guide/paper-wallet.md
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
(cherry picked from commit 3be5715f45)
Co-authored-by: Brent Fitzgerald <burnto@gmail.com>
* stake: Remove v2 program references (#19308)
* stake: Remove v2 program references
* Remove stake v2 feature, along with stake rewrite
(cherry picked from commit 73aa004c59)
# Conflicts:
# runtime/src/bank.rs
* Fix merge conflict
* Fix uses of old `stake` function
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
* docs: Mainnet Beta inflation has been enabled for quite some time
(cherry picked from commit 169ded9a70)
* validator: Trusted validators are now called known validators
(cherry picked from commit e0bc5fa690)
* docs: trust minimize
(cherry picked from commit 40613161a0)
* docs: correct known validator operator
(cherry picked from commit eced50d103)
* docs: Remove decommissioned testnet archetype validator
(cherry picked from commit a587eec20b)
* docs: update devnet start args with new validators
(cherry picked from commit 2a877ae06e)
Co-authored-by: Trent Nelson <trent@solana.com>
Cross cluster gossip contamination is causing cluster-slots hash map to
contain a lot of bogus values and consume too much memory:
https://github.com/solana-labs/solana/issues/17789
If a node is using the same identity key across clusters, then these
erroneous values might not be filtered out by shred-versions check,
because one of the variants of the contact-info will have matching
shred-version:
https://github.com/solana-labs/solana/issues/17789#issuecomment-896304969
The cluster-slots hash-map is bounded and trimmed at the lower end by
the current root. This commit also discards slots epochs ahead of the
root.
(cherry picked from commit 563aec0b4d)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* Add AsRef<AccountInfo> for AccountInfo
(cherry picked from commit 930465e67c)
* add test for AsRef
(cherry picked from commit d10e37a829)
Co-authored-by: Kirill Fomichev <fanatid@ya.ru>
* Fixing missing pubsub notification for programSubscribe and logsSubscribe (#19092)
#18587: programSubscribe is missing notifications randomly. The issue is because of two reasons
Not all rooted slots get OptimisticallyConfirmed notifications
The OptimisticallyConfirmed notifications can be out of order for slots: slot A and B with A < B can see notification for B first before A.
Summary of Changes
Changed OptimisticallyConfirmedBankTracker to send notifications for parent banks if they have not been notified yet. We use a new variable last_notified_slot to track that.
Tests:
With my validator running against testnet, before the fix, it was failing 75% of time, with the fix, it is passing consistently. Using the program mentioned in #18587.
(cherry picked from commit 1a372a792e)
* Use v1.7 api
Co-authored-by: Lijun Wang <83639177+lijunwangs@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
* Handle 0-lamport account in index generation
* rename duplicate to dirty keys
Co-authored-by: Carl Lin <carl@solana.com>
(cherry picked from commit 5a4979f25f)
Co-authored-by: sakridge <sakridge@gmail.com>
* Use last_valid_block_height in services and client apps (#19163)
* Add deprecated tag to Bank::get_blockhash_last_valid_slot
* Update SendTransactionService to use last_valid_block_height
* Update solana-tokens to use last_valid_block_height
* Remove dangling file
* Update solana program to use last_valid_block_height
* Update Banks crates to use last_valid_block_height
(cherry picked from commit 5970083b4d)
# Conflicts:
# cli/src/program.rs
* Fix conflict
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
* Add some docs for RpcClient and friends (#18748)
* Add some docs for RpcSender, HttpSender, MockSender
* Support SimulateTransaction in MockSender
* Add docs for RpcClient constructors
* Add some more RpcClient examples
* rustfmt
* Reflow docs in rpc_client and friends
(cherry picked from commit 5dcfd7ce74)
# Conflicts:
# client/src/mock_sender.rs
* Fix conflicts
Co-authored-by: Brian Anderson <andersrb@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
* Fix serialization of parameters in RpcClient::get_block_production_with_config (#18998)
Params must be an array or null.
(cherry picked from commit 58f395257b)
# Conflicts:
# client/src/mock_sender.rs
* Fix conflict
Co-authored-by: Brian Anderson <andersrb@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
* Auto-generate shell completions for solana-cli (issue #8879 and #14005)
Implement `completion` SubCommand for solana-cli, which outputs
completion script to stdout and exits the process. The script generation
handled completely by clap.
In order to implement the generation, one minor design change was
necessary regarding the creation of clap `App`.
Previously: One part of App initialization was in the `app` function,
and some other arguments and subcommands were added later directly in
the `main` function.
Now: The whole construction of App was moved to `get_clap_app` function.
P.S. I wasn't sure if constructing App separately had visual importance,
so both constructing parts are still separate in `base_clap_app` and
`final_clap_app` functions. But they sure could be in one single
function.
* Dereplicode match expr, fix clippy warning.
* Move clap App construction into separate module
Also join two parts of the construction into a single function
* Fix tests
* Apply rustfmt lints
(cherry picked from commit 9d0a937a05)
Co-authored-by: theonekeyg <34949189+theonekeyg@users.noreply.github.com>
* filters crds values in parallel when responding to gossip pull-requests (#18877)
When responding to gossip pull-requests, filter_crds_values takes a lot of time
while holding onto read-lock:
https://github.com/solana-labs/solana/blob/f51d64868/gossip/src/crds_gossip_pull.rs#L509-L566
This commit will filter-crds-values in parallel using rayon thread-pools.
(cherry picked from commit f1198fc6d5)
# Conflicts:
# gossip/src/cluster_info.rs
# gossip/src/crds_gossip_pull.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* backport new column families from master to 1.6 (#18743)
* backporting bank_hash and program_costs column families from master to 1.6 for rocksdb backward compatibility
* missed a line to allow dead code
* include code for purge
* Exclude stubbed ProgramCosts column from compaction (#18840)
Co-authored-by: Tao Zhu <82401714+taozhu-chicago@users.noreply.github.com>
This feature requires clang to support -march=native on M1. Before this change,
cargo build would output a build error when building for aarch64.
(cherry picked from commit 8a9b7f5ef2)
# Conflicts:
# ledger/Cargo.toml
* Refactor account encoding to povide helper w/out querying Bank
* Use new method in get_program_accounts to properly return length err
(cherry picked from commit 3eecb6f4ae)
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* removes id from push_lowest_slot args (#18645)
push_lowest_slot cannot sign the new crds-value unless the id (pubkey)
argument passed-in is the same pubkey as in ClusterInfo::keypair(), in
which case the id argument is redundant:
https://github.com/solana-labs/solana/blob/bb41cf346/gossip/src/cluster_info.rs#L824-L845
Additionally, the lookup is done with self.id(), but insert is done with
the id argument, which is logically a bug.
(cherry picked from commit c90af3cd63)
# Conflicts:
# gossip/src/cluster_info.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* refactor(rpc_client): simplfy send and confirm transaction flow
* chore: configurable num of retries for send and confirm
(cherry picked from commit 70234dfdf4)
Co-authored-by: hrls <viktor.kharitonovich@gmail.com>
* removes id and shred_version from CrdsGossip (#18505)
ClusterInfo is the gateway to CrdsGossip function calls, and it already
has node's pubkey and shred version (full ContactInfo and Keypair in
fact).
Duplicating these data in CrdsGossip adds redundancy and possibility for
bugs should they not be consistent with ClusterInfo.
(cherry picked from commit 4e1333fbe6)
# Conflicts:
# gossip/src/cluster_info.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* Refactoring: Unify account_deps and accounts (#17898)
* Changes ThisInvokeContext::get_account() to use accounts instead of pre_accounts.
* Adds explicit keys to accounts to make them symmetric to account_deps.
* Appends account_deps to accounts in transaction loading and removes account_deps everywhere else.
(cherry picked from commit 7462c27d07)
# Conflicts:
# program-test/src/lib.rs
# runtime/src/bank.rs
# runtime/src/message_processor.rs
* fix conflicts
Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
Co-authored-by: Justin Starry <justin@solana.com>
Current implementation of weighted_shuffle:
https://github.com/solana-labs/solana/blob/b08f8bd1b/gossip/src/weighted_shuffle.rs#L11-L37
uses a heuristic which results in biased samples.
For example, if the weights are [1, 10, 100], then the 3rd index should
come first 100 times more often than the 1st index. However,
weighted_shuffle is picking the 3rd index 200+ times more often than the
1st index, showing a disproportional bias in favor of higher weights.
This commit implements weighted shuffle using binary indexed tree to
maintain cumulative sum of weights while sampling. The resulting samples
are demonstrably unbiased and precisely proportional to the weights.
Additionally the iterator interface allows to skip computations when
not all indices are processed.
Of the use cases of weighted_shuffle, changing turbine code requires
feature-gating to keep the cluster in sync. That is not updated in
this commit, but can be done together with future updates to turbine.
(cherry picked from commit dba42c57b4)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
The repair-peers cache is reset each time repair service loop runs,
and so computed repeatedly for the same slots:
https://github.com/solana-labs/solana/blob/d2b07dca9/core/src/repair_service.rs#L275
This commit uses an LRU cache to persists repair-peers for each slot.
In addition to LRU eviction rules, in order to avoid re-using outdated
data, each entry also has 10 seconds TTL.
(cherry picked from commit a0551b4054)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Broadcast stage and retransmit stage should arrange nodes on turbine
broadcast tree in exactly same order. Additionally any changes to this
ordering (e.g. updating how unstaked nodes are handled) requires feature
gating to keep the cluster in sync.
Current implementation is scattered out over several public methods and
exposes too much of implementation details (e.g. usize indices into
peers vector) which makes code changes and checking for feature
activations more difficult.
This commit encapsulates turbine peer computations into a new struct,
and only exposes two public methods, get_broadcast_peer and
get_retransmit_peers, for call-sites.
(cherry picked from commit 04787be8b1)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* Report 404 instead of 502 when a snapshot file is not found
This provides the client with more useful information about why their
request failed
(cherry picked from commit 40d696fcbc)
* As a last resort try to download an uncompressed snapshot
`solana-test-validator` creates uncompressed snapshots and it can be
useful to attach another validator to a `solana-test-validator` cluster
from time to time
(cherry picked from commit e36247d187)
Co-authored-by: Michael Vines <mvines@gmail.com>
I wanted to use this sysvar to get a recent block hash, but I didn't
know whether the first or the last entry contains the most recent block
hash.
By calling it for mainnet, printing the results, and comparing that to
the recent blocks on solanabeach.io/blocks, I discovered that the
entries are ordered from most recent to least recent. Document this to
save future readers the trouble.
(cherry picked from commit 94ab0eb49f)
Co-authored-by: Ruud van Asseldonk <ruud@chorus.one>
* Add proposal for supporting big transactions
* account index program
* fix formatting
* review feedback
* Add cost changes section
* Add cost section and more attack details
* fix lint
* document metadata changes
* nit
* rpc details
* add index meta struct
* add additional proposal and chagne title
* rename proposal file
* rename to address map and rewrite tx format
* no more appends, limit mapping size to 256
* update dos section
* add note about readonly
* restructure message to use enum
* cleanup
(cherry picked from commit 191519188d)
Co-authored-by: Justin Starry <justin@solana.com>
* Try all peers during pull-request in case of offline nodes
* fix clippy err
(cherry picked from commit f4fb5de545)
Co-authored-by: Ashwin Sekar <ashwin@solana.com>
* Generate epoch-completed time from block times
* Add annotation when block times not available
(cherry picked from commit f2b0d562b0)
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Zelcore wallet support for SOL/SPLs info added
I have added a quick infodump about Zelcore wallet supporting various features of the Solana ecosystem. I looked at previous commits and it seemed adding to the top of the file was the norm, but please feel free to move around if needed.
* Zelcore wallet info moved to bottom of file.
(cherry picked from commit 8fc4c2f375)
Co-authored-by: trial123Zel <38964930+trial123Zel@users.noreply.github.com>
* filters crds values obtained through gossip by their shred version (#18072)
filter_by_shred_version does not check the shred-version of the owner of
the crds-value. It only checks the shred-version of the node which is
relaying the value:
https://github.com/solana-labs/solana/blob/5cc073420/gossip/src/cluster_info.rs#L2274-L2289
So crds-values with different shred versions can still pass through this
function as long as they are relayed by a node with matching shred
version; and so, a single node can bridge different shred values
through-out the cluster.
(cherry picked from commit 69a5f0e6cd)
# Conflicts:
# gossip/src/cluster_info.rs
* removes backport merge conflicts
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* Keep track of dirty stores on remove accounts to clean
and not zero_lamport key set
* Only dirty when count==0?
* Add another clean
(cherry picked from commit 3b1738c000)
Co-authored-by: sakridge <sakridge@gmail.com>
* adds shred-version to ip-echo-server response
When starting a validator, the node initially joins gossip with
shred_verison = 0, until it adopts the entrypoint's shred-version:
https://github.com/solana-labs/solana/blob/9b182f408/validator/src/main.rs#L417
Depending on the load on the entrypoint, this adopting entrypoint
shred-version through gossip sometimes becomes very slow, and causes
several problems in gossip because we have to partially support
shred_version == 0 which is a source of leaking crds values from one
cluster to another. e.g. see
https://github.com/solana-labs/solana/pull/17899
and the other linked issues there.
In order to remove shred_version == 0 from gossip, this commit adds
shred-version to ip-echo-server response. Once the entrypoints are
updated, on validator start-up, if --expected_shred_version is not
specified we will obtain shred-version from the entrypoint using
ip-echo-server.
(cherry picked from commit 598093b5db)
# Conflicts:
# Cargo.lock
# net-utils/Cargo.toml
# programs/bpf/Cargo.lock
* removes backport merge conflicts
* obtains shred-version from entrypoint's ip-echo-server in validator-main
(cherry picked from commit 58e115275a)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* eliminate flatten and sort in hash calculation
* reduce critical section time
* remove now no-longer necessary test code
* conflict with reset bins to 0 pr
(cherry picked from commit bf97627021)
Co-authored-by: Jeff Washington (jwash) <75863576+jeffwashington@users.noreply.github.com>
Crds values of nodes with different shred versions are creeping into
gossip table resulting in runtime issues as the one addressed in:
https://github.com/solana-labs/solana/pull/17899
This commit works towards enforcing more checks and filtering based on
shred version by adding necessary mapping and api to gossip table.
Once populated, pubkey->shred-version mapping persists as long as there
are any values associated with the pubkey.
(cherry picked from commit 5a99fa3790)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* Make account shrink configurable #17544 (#17778)
1. Added both options for measuring space usage using total accounts usage and for individual store shrink ratio using an enum. Validator CLI options: --accounts-shrink-optimize-total-space and --accounts-shrink-ratio
2. Added code for selecting candidates based on total usage in a separate function select_candidates_by_total_usage
3. Added unit tests for the new functions added
4. The default implementations is kept at 0.8 shrink ratio with --accounts-shrink-optimize-total-space set to true
Fixes#17544
(cherry picked from commit 269d995832)
# Conflicts:
# core/tests/snapshots.rs
# ledger/src/bank_forks_utils.rs
# runtime/src/accounts_db.rs
# runtime/src/snapshot_utils.rs
* fix some merge errors
Co-authored-by: Lijun Wang <83639177+lijunwangs@users.noreply.github.com>
Co-authored-by: Jeff Washington (jwash) <wash678@gmail.com>
* Skip shrink when it doesn't save anything (#17405)
(cherry picked from commit 14c52ab018)
# Conflicts:
# runtime/src/accounts_db.rs
* fix merge error
Co-authored-by: sakridge <sakridge@gmail.com>
Co-authored-by: Jeff Washington (jwash) <wash678@gmail.com>
* verify bank hash on startup with ledger tool option (#17939)
(cherry picked from commit f558b9b6bf)
# Conflicts:
# core/tests/snapshots.rs
# ledger/src/bank_forks_utils.rs
# runtime/src/snapshot_utils.rs
* fix merge errors
Co-authored-by: Jeff Washington (jwash) <75863576+jeffwashington@users.noreply.github.com>
Co-authored-by: Jeff Washington (jwash) <wash678@gmail.com>
* add metrics for startup
* roll timings up higher
* fix test
* fix duplicate
(cherry picked from commit 471b34132e)
# Conflicts:
# ledger/src/bank_forks_utils.rs
# runtime/src/snapshot_utils.rs
conflicts because #17778 is not present.
Co-authored-by: Jeff Washington (jwash) <75863576+jeffwashington@users.noreply.github.com>
Refactor a few functions that are on the load-from-snapshot path, to facilitate
adding in incremental snapshots more easily.
Additionally, add some tests and doc comments.
(cherry picked from commit 1953543274)
Co-authored-by: Brooks Prumo <brooks@solana.com>
* Refactor stake program into solana_program (#17906)
* Move stake state / instructions into solana_program
* Update account-decoder
* Update cli and runtime
* Update all other parts
* Commit Cargo.lock changes in programs/bpf
* Update cli stake instruction import
* Allow integer arithmetic
* Update ABI digest
* Bump rust mem instruction count
* Remove useless structs
* Move stake::id() -> stake::program::id()
* Re-export from solana_sdk and mark deprecated
* Address feedback
* Run cargo fmt
* Run cargo fmt post cherry-pick
* calculate_capitalization uses hash calculation
* feedback
* remove debugging code, clean up slot math
Co-authored-by: Jeff Washington (jwash) <75863576+jeffwashington@users.noreply.github.com>
Reports of excessive GPU memory usage and errors
from cudaHostRegister. There are some cases where pinning is
not required.
(cherry picked from commit eeee75c5be)
Co-authored-by: sakridge <sakridge@gmail.com>
* fix minor typos and punctuation
* fix minor typos and punctuation
* rewording for clarity and typo corrections
* rewording for clarity and typo corrections
* rewording for clarity and typo corrections
Co-authored-by: Gregg Dourgarian <greggd@aidacreative.com>
(cherry picked from commit 54155f875a)
Co-authored-by: Haik Dulgarian <greggd@tempworks.com>
* update dependence version for gag to leatest support windows
* fix compile on windows
* add Cargo.lock
(cherry picked from commit e0ab5ee4f8)
Co-authored-by: Govlzkoy <gotope@users.noreply.github.com>
* rework hash calculation to not keep slot and write version
* refactor functions and add tests
* always use multiple slot code path
(cherry picked from commit b5bb91b50f)
Co-authored-by: Jeff Washington (jwash) <75863576+jeffwashington@users.noreply.github.com>
* Accounts dumping logic
* Add test for interaction between cache flush and remove_unrooted_slot()
* Update comments
* Rename
* renaming
* Add more comments
* Renaming
* Fixup test and bad check
(cherry picked from commit bbcdf073ba)
Co-authored-by: carllin <carl@solana.com>
* system-program: Move lamports == 0 check on transfers
* Address feedback
* Update stake split to explicitly allocate + assign
* Update stake tests referring to split instruction
* Revert whitespace
* Update split instruction index in test
* Remove unnecessary `assign_with_seed` from `split_with_seed`
* Fix stake instruction parser
* Update test to allow splitting into account with lamports
(cherry picked from commit 8f5e773caf)
Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
* document ProgramTest::new
* simplify ProgramTest::new doc-string
* make ProgramTest::add_program noisier
`add_program` (and `new`, implicitly) now prints a warning when the user
supplies a bogus program name to a ProgramTest and invokes `test-bpf`.
Additionally, it is now impossible to ask for a regular `test` and for
the generated ProgramTest to load BPF code instead of native code.
Previously, this was caused by a precedence issue: BPF code would always
be preferred over native if the program name was valid, regardless of
user choice.
(cherry picked from commit 2aaf55795f)
Co-authored-by: xuoe <alex@psi.io>
If the crds entry belongs to the caller itself, then the caller will
always have the more recent version of it, regardless of it being
filtered out by the bloom filter or not.
The exception is node-instance types which are meant to detect duplicate
running instances, and those are exempted.
(cherry picked from commit 7cf6e66ddd)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
epoch-slots may be overwritten before they are written to crds table:
https://github.com/solana-labs/solana/issues/17711
This commit writes new epoch-slots to crds table synchronously with
push_epoch_slots. The functions is still not thread-safe as commented in
the code, however currently only one threads is invoking this code.
(cherry picked from commit 60b0a13444)
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
* use slots returned from get_snapshot_storages to sort
* add tests
(cherry picked from commit 9388aaca15)
# Conflicts:
# runtime/src/accounts_db.rs
# runtime/src/sorted_storages.rs
Co-authored-by: Jeff Washington (jwash) <wash678@gmail.com>
* Avoid full-range compactions with periodic filtered b.g. ones (#16697)
* Update rocksdb to v0.16.0
* Promote the infrequent and important log to info!
* Force background compaction by ttl without manual compaction
* Fix test
* Support no compaction mode in test_ledger_cleanup_compaction
* Fix comment
* Make compaction_interval customizable
* Avoid major compaction with periodic filtering...
* Adress lazy_static, special cfs and range check
* Clean up a bit and add comment
* Add comment
* More comments...
* Config code cleanup
* Add comment
* Use .conflicts_with()
* Nullify unneeded delete_range ops for special CFs
* Some clean ups
* Clarify the locking intention
* Ensure special CFs' consistency with PurgeType::CompactionFilter
* Fix comment
* Fix bad copy paste
* Fix various types...
* Don't use tuples
* Add a unit test for compaction_filter
* Fix typo...
* Remove flag and just use new behavior always
* Fix wrong condition negation...
* Doc. about no set_last_purged_slot in purge_slots
* Write a test and fix off-by-one bug....
* Apply suggestions from code review
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
* Follow up to github review suggestions
* Fix line-wrapping
* Fix conflict
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
(cherry picked from commit 1f97b2365f)
# Conflicts:
# ledger/src/blockstore_db.rs
* Fix conflict
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
* Add missing ProgramError to InstructionError mappings
* add note
* Clarify process of adding new program error
(cherry picked from commit 83b9a046d1)
Co-authored-by: Jack May <jack@solana.com>
* Make the sys-tuner oneliner actually copy-pastable
* Use `command -v`
(cherry picked from commit 41975016b9)
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
* Always bail if program modifies a ro account (#17569)
(cherry picked from commit a3240aebde)
* resolve conflicts
* nudge
Co-authored-by: Jack May <jack@solana.com>
When building the master branch, please make sure you are using the latest stable rust version by running:
Please make sure you are always using the latest stable rust version by running:
```bash
$ rustup update
```
When building a specific release branch, you should check the rust version in `ci/rust-version.sh` and if necessary, install that version by running:
```bash
$ rustup install VERSION
```
Note that if this is not the latest rust version on your machine, cargo commands may require an [override](https://rust-lang.github.io/rustup/overrides.html) in order to use the correct version.
On Linux systems you may need to install libssl-dev, pkg-config, zlib1g-dev, etc. On Ubuntu:
```bash
@@ -116,13 +110,13 @@ send us that patch!
All claims, content, designs, algorithms, estimates, roadmaps,
specifications, and performance measurements described in this project
are done with the Solana Foundation's ("SF") good faith efforts. It is up to
are done with the Solana Foundation's ("SF") best efforts. It is up to
the reader to check and validate their accuracy and truthfulness.
Furthermore, nothing in this project constitutes a solicitation for
Furthermore nothing in this project constitutes a solicitation for
investment.
Any content produced by SF or developer resources that SF provides, are
for educational and inspirational purposes only. SF does not encourage,
for educational and inspiration purposes only. SF does not encourage,
induce or sanction the deployment, integration or use of any such
applications (including the code comprising the Solana blockchain
protocol) in violation of applicable laws or regulations and hereby
This design describes how the cluster slashes leaders that produce duplicate
blocks.
Leaders that produce multiple blocks for the same slot increase the number of
potential forks that the cluster has to resolve.
## Primitives
1. gossip_root: Nodes now gossip their current root
2. gossip_duplicate_slots: Nodes can gossip up to `N` duplicate slot proofs.
3.`DUPLICATE_THRESHOLD`: The minimum percentage of stake that needs to vote on a fork with version `X` of a duplicate slot, in order for that fork to become votable.
## Protocol
1. When WindowStage detects a duplicate slot proof `P`, it checks the new `gossip_root` to see if `<= 1/3` of the nodes have rooted a slot `S >= P`. If so, it pushes a proof to `gossip_duplicate_slots` to gossip. WindowStage then signals ReplayStage about this duplicate slot `S`. These proofs can be purged from gossip once the validator sees > 2/3 of people gossiping roots `R > S`.
2. When ReplayStage receives the signal for a duplicate slot `S` from `1)` above, the validator monitors gossip and replay waiting for`>= DUPLICATE_THRESHOLD` votes for the same hash which implies the same version of the slot. If this conditon is met for some version with hash `H` of slot `S`, this is then known as the `duplicate_confirmed` version of the slot.
Before a duplicate slot `S` is `duplicate_confirmed`, it's first excluded from the vote candidate set in the fork choice rules. In addition, ReplayStage also resets PoH to the *latest* ancestor of the *earliest*`non-duplicate/confirmed_duplicate_slot`, so that block generation can start happening on the earliest known *safe* block.
Some notes about the `DUPLICATE_THRESHOLD`. In the cases below, assume `DUPLICATE_THRESHOLD = 52`:
a) If less than `2 * DUPLICATE_THRESHOLD - 1` percentage of the network is malicious, then there can only be one such `duplicate_confirmed` version of the slot. With `DUPLICATE_THRESHOLD = 52`, this is
a malcious tolerance of `4%`
b) The liveness of the network is at most `1 - DUPLICATE_THRESHOLD - SWITCH_THRESHOLD`. This is because if you need at least `SWITCH_THRESHOLD` percentage of the stake voting on a different fork in order to switch off of a duplicate fork that has `< DUPLICATE_THRESHOLD` stake voting on it, and is *not*`duplicate_confirmed`. For `DUPLICATE_THRESHOLD = 52` and `DUPLICATE_THRESHOLD = 38`, this implies a liveness tolerance of `10%`.
For example in the situation below, validators that voted on `2` can't vote any further on fork `2` because it's been removed from fork choice. Now slot 6 better have enough stake for a switching proof, or the network halts.
```text
|-------- 2 (51% voted, then detected this slot was a duplicate and removed this slot from fork choice)
0---|
|---------- 6 (39%)
```
3. Switching proofs need to be extended to allow including vote hashes from different versions of the same same slot (detected through 1). Right now this is not supported since switching proofs can
only be built using votes from banks in BankForks, and two different versions of the same slot cannot
simultaneously exist in BankForks. For instance:
```text
|-------- 2
|
0------------- 1 ------ 2'
|
|---------- 6
```
Imagine each version of slot 2 and 2' have `DUPLICATE_THRESHOLD / 2` of the votes on them, so neither duplicate can be confirmed. At most slot 6 has `1 - DUPLICATE_THRESHOLD / 2` of the votes
on it, which is less than the switching threshold. Thus, in order for validators voting on `2` or `2'` to switch to slot 6, and make progress, they need to incorporate votes from the other version of the slot into their switching proofs.
### The repair problem.
Now what happens if one of the following occurs:
1) Due to network blips/latencies, some validators fail to observe the gossip votes before they are overwritten by newer votes? Then some validators may conclude a slot `S` is `duplicate_confirmed` while others don't.
2) Due to lockouts, no version of duplicate slot `S` reaches `duplicate_confirmed` status, but one of its descendants may reach `duplicate_confirmed` after those lockouts expire, which by definition, means `S` is also `duplicate_confirmed`.
3) People who are catching up and don't see the votes in gossip encounter a dup block and can't make progress.
We assume that given a network is eventually stable, if at least one correct validator observed `S` is `duplicate_confirmed`, then if `S` is part of the heaviest fork, then eventually all validators will observe some descendant of `S` is duplicate confirmed.
This problem we need to solve is modeled simply by the below scenario:
```text
1 -> 2 (duplicate) -> 3 -> 4 (duplicate)
```
Assume the following:
1. Due to gossiping duplciate proofs, we assume everyone will eventually see duplicate proofs for 2 and 4, so everyone agrees to remove them from fork choice until they are `duplicate_confirmed`.
2. Due to lockouts, `> DUPLICATE_THRESHOLD` of the stake votes on 4, but not 2. This means at least `DUPLICATE_THRESHOLD` of people have the "correct" version of both slots 2 and 4.
3. However, the remaining `1-DUPLICATE_THRESHOLD` of people have wrong version of 2. This means in replay, their slot 3 will be marked dead, *even though the faulty slot is 2*. The goal is to get these people on the right fork again.
Possible solution:
1. Change `EpochSlots` to signal when a bank is frozen, not when a slot is complete. If we see > `DUPLICATE_THRESHOLD` have frozen the dead slot 3, then we attempt recovery. Note this does not mean that all `DUPLICATE_THRESHOLD` have frozen the same version of the bank, it's just a signal to us that something may be wrong with our version of the bank.
2. Recovery takes the form of a special repair request, `RepairDuplicateConfirmed(dead_slot, Vec<(Slot, Hash)>)`, which specifies a dead slot, and then a vector of `(slot, hash)` of `N` of its latest parents.
3. The repairer sees this request and responds with the correct hash only if any element of the `(slot, hash)` vector is both `duplicate_confirmed` and the hash doesn't match the requester's hash in the vector.
4. Once the requester sees the "correct" hash is different than their frozen hash, they dump the block so that they can accept a new block, and ask the network for the block with the correct hash.
Of course the repairer might lie to you, and you'll get the wrong version of the block, in which case you'll end up with another dead block and repeat the procedure.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
