1684 Commits

Author SHA1 Message Date
mergify[bot]
0e6b476cbf More set_root metrics (backport #21286) (#21290)
* More set_root metrics (#21286)

(cherry picked from commit 398af132a5)

# Conflicts:
#	core/src/replay_stage.rs

* Fix conflict

Co-authored-by: sakridge <sakridge@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-11-16 02:29:33 +00:00
mergify[bot]
6a6dd86262 throttle store_cached when cache size is too large (#21188) (#21279)
* throttle store_cached when cache size is too large

* reduce max delay

* 100ms max

* 10ms max delay

Co-authored-by: Jeff Washington (jwash) <75863576+jeffwashington@users.noreply.github.com>
2021-11-15 19:57:27 +00:00
Ivan Mironov
8072635967 Add validator options to change priority of snapshot packager and RPC threads (v1.8) (#21020)
* Add function for changing thread's nice value

Linux only.

* Add validator option to change niceness of snapshot packager thread

* Add validator option to change niceness of RPC server threads

Fixes https://github.com/solana-labs/solana/issues/14556

* Run `./scripts/cargo-for-all-lock-files.sh tree`
2021-11-15 09:50:45 -08:00
mergify[bot]
2992a7154a Refactor vote state to remove double negative (backport #21244) (#21250)
* Refactor vote state to remove double negative (#21244)

(cherry picked from commit ef29d2d172)

# Conflicts:
#	runtime/src/stakes.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2021-11-13 01:27:23 +00:00
Tyera Eulberg
09ef4d12f7 Bump version to 1.8.4 (#21232) (#21235)
Co-authored-by: sakridge <sakridge@gmail.com>
2021-11-10 15:20:41 -07:00
mergify[bot]
de94c4e867 Add set_root bank drop logging (#21144) (#21194)
(cherry picked from commit 706b60b5c8)

Co-authored-by: carllin <carl@solana.com>
2021-11-10 11:58:28 -07:00
Tyera Eulberg
74684a107c Revert "Bump version to 1.8.4 (#21232)"
This reverts commit 19b3ba0442.
2021-11-10 11:19:39 -07:00
sakridge
19b3ba0442 Bump version to 1.8.4 (#21232) 2021-11-10 16:38:44 +01:00
Justin Starry
89b2a3d0ae Refactor RentDebits to use hashmap instead of vec (v1.8 backport) (#21176)
* Refactor `RentDebits` to use hashmap instead of vec (v1.8 backport)

* Fix rent debits test (#21177)
2021-11-05 08:52:14 +00:00
mergify[bot]
56fc58a2b5 Simplify replay vote tracking by using packet metadata (backport #21112) (#21149)
* Simplify replay vote tracking by using packet metadata (#21112)

(cherry picked from commit 140a5f633d)

# Conflicts:
#	core/src/banking_stage.rs
#	ledger-tool/src/main.rs
#	rpc/src/rpc.rs
#	runtime/src/bank.rs
#	runtime/src/bank_utils.rs
#	runtime/src/stakes.rs
#	sdk/src/transaction/sanitized.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2021-11-05 00:16:34 +00:00
sakridge
c14864a608 Add accounts stats for space amplification (#21151) 2021-11-03 16:41:29 +01:00
mergify[bot]
780302af56 Add set_root timing metrics (#21119) (#21135)
(cherry picked from commit 1e5212e60d)

Co-authored-by: sakridge <sakridge@gmail.com>
2021-11-02 20:17:04 +00:00
Tyera Eulberg
8dd3c1ece1 Bump version to v1.8.3 (#21040) 2021-10-28 11:17:11 -06:00
mergify[bot]
66caead016 Add compute budget noops (backport #20992) (#21014)
* Add compute budget program as a noop (#20992)

(cherry picked from commit 1e2bef76e3)

# Conflicts:
#	sdk/src/feature_set.rs

* resolve conflicts

Co-authored-by: Jack May <jack@solana.com>
2021-10-27 12:47:35 -07:00
mergify[bot]
de1f60fb2d Refactor cost tracker metrics reporting (backport #20802) (#20933)
* - cost_tracker is data member of a bank, it can report metrics when bank is frozen (#20802)

- removed cost_tracker_stats and histogram
- move stats reporting outside of bank freeze

(cherry picked from commit c2bfce90b3)

# Conflicts:
#	Cargo.lock
#	core/src/banking_stage.rs
#	core/src/replay_stage.rs
#	core/src/tvu.rs
#	ledger-tool/src/main.rs
#	programs/bpf/Cargo.lock
#	runtime/Cargo.toml
#	runtime/src/cost_tracker.rs

* manual fix merge conflicts

Co-authored-by: Tao Zhu <82401714+taozhu-chicago@users.noreply.github.com>
Co-authored-by: Tao Zhu <tao@solana.com>
2021-10-27 16:48:20 +00:00
mergify[bot]
7e7f8ef5f0 Report timing info for stakes cache updates from txs (backport #20856) (#20884)
* Report timing info for stakes cache updates from txs (#20856)

(cherry picked from commit 735016661b)

# Conflicts:
#	runtime/src/bank.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2021-10-26 20:04:32 +00:00
Lijun Wang
782d143489 Accountsdb plugin write ordering (#20948) (#20964)
Use the write_version in the Accounts's meta data so that account write with lower write_version would not overwrite the higher ones.
2021-10-26 00:05:40 -07:00
Lijun Wang
66630804de Accountsdb plugin postgres -- bulk insertion at startup (#20763) (#20931)
* Accountsdb plugin postgres -- bulk insertion at startup (#20763)

Use bulk insertion to Postgres at startup to reduce time taken for initial snapshot restore for postgres plugin. Avoid duplicate writes of accounts at startup. Doing account plugin notification and indexing in parallel.

Improved error handling for postgres plugin to show the real db issues for debug purpose
Added more metrics for postgres plugin.
Refactored plugin centric code out to a sub module from accounts_db and added unit tests

* Fixed the unit test failures
2021-10-25 09:18:32 -07:00
Trent Nelson
23b6ce7980 Bump version to 1.8.2 2021-10-21 00:43:40 -06:00
mergify[bot]
440ccd189e Add program heap bump instruction (backport #20607) (#20815)
* Add program heap bump instruction (#20607)

(cherry picked from commit 58164517e4)

* nudge

Co-authored-by: Jack May <jack@solana.com>
2021-10-20 23:05:57 +00:00
mergify[bot]
63835ec214 prior to panicing with cap mismatch, try other calculation (#20292) (#20804)
(cherry picked from commit fa5b091b4c)

Co-authored-by: Jeff Washington (jwash) <75863576+jeffwashington@users.noreply.github.com>
2021-10-20 02:26:23 +00:00
Jon Cinque
564cc95b00 runtime: Add foundation stake pool withdraw authority (#20797)
(cherry picked from commit cb2bd65858)
2021-10-19 17:56:09 -07:00
mergify[bot]
28eb6ff796 Invoke cost tracker from its bank (backport #20627) (#20800)
* - make cost_tracker a member of bank, remove shared instance from TPU; (#20627)

- decouple cost_model from cost_tracker; allowing one cost_model
  instance being shared within a validator;
- update cost_model api to calculate_cost(&self...)->transaction_cost

(cherry picked from commit 7496b5784b)

# Conflicts:
#	core/src/banking_stage.rs
#	ledger-tool/src/main.rs
#	runtime/src/bank.rs
#	runtime/src/cost_model.rs
#	runtime/src/cost_tracker.rs

* manual fix merge conflicts

Co-authored-by: Tao Zhu <82401714+taozhu-chicago@users.noreply.github.com>
Co-authored-by: Tao Zhu <tao@solana.com>
2021-10-20 00:22:38 +00:00
Sean Young
0a6bb84aec feat: add ed25519 signature verify program
Solang requires a method for verify ed25519 signatures. Add a new
builtin program at address Ed25519SigVerify111111111111111111111111111
which takes any number of ed25519 signature, public key, and message.
If any of the signatures fails to verify, an error is returned.

The changes for the web3.js package will go into another commit, since
the tests test against a released solana node. Adding web3.js ed25519
testing will break CI.

(cherry picked from commit b491354e51)

Conflicts:
	Cargo.lock
	Cargo.toml
	programs/bpf/Cargo.lock
	runtime/Cargo.toml
	sdk/src/feature_set.rs
	sdk/src/transaction.rs
	sdk/src/transaction/sanitized.rs
2021-10-18 15:41:24 +01:00
Sean Young
c8f6a0817b verify_precompiles needs FeatureSet
Rather than pass in individual features, pass in the entire feature set
so that we can add the ed25519 program feature in a later commit.

(cherry picked from commit 0f62771f42)

 Conflicts:
	banks-server/src/banks_server.rs
	core/src/banking_stage.rs
	programs/secp256k1/src/lib.rs
	rpc/src/rpc.rs
	runtime/src/bank.rs
	sdk/src/transaction.rs
	sdk/src/transaction/sanitized.rs
2021-10-18 15:41:24 +01:00
mergify[bot]
778f37b12d fix unstable test (#20645) (#20663)
(cherry picked from commit 220fd41bbc)

Co-authored-by: Tao Zhu <82401714+taozhu-chicago@users.noreply.github.com>
2021-10-13 20:02:42 +00:00
mergify[bot]
400a88786a aggregate cost_tracker to bank (backport #20527) (#20622)
* - move cost tracker into bank, so each bank has its own cost tracker; (#20527)

- move related modules to runtime

(cherry picked from commit 005d6863fd)

# Conflicts:
#	Cargo.lock
#	core/benches/banking_stage.rs
#	core/src/banking_stage.rs
#	core/src/lib.rs
#	core/src/tvu.rs
#	ledger-tool/src/main.rs
#	ledger/src/blockstore_processor.rs
#	programs/bpf/Cargo.lock
#	runtime/Cargo.toml
#	runtime/src/cost_model.rs

* manual fix merge conflicts

Co-authored-by: Tao Zhu <82401714+taozhu-chicago@users.noreply.github.com>
Co-authored-by: Tao Zhu <tao@solana.com>
2021-10-13 05:07:09 +00:00
Sean Young
927d3b5e0d Add return data implementation
This consists of:
 - syscalls
 - passing return data from invoked to invoker
 - printing to stable log
 - rust and C SDK changes

(cherry picked from commit 53b47b87b2)
2021-10-12 18:31:42 +01:00
Brooks Prumo
df929bda38 Do not shell out for tar (#19043)
When making a snapshot archive, we used to shell out and call `tar -S`
for sparse file support.  The tar crate supports sparse files, so no
need to do this anymore.

Fixes #10860

(cherry picked from commit 68cc71409e)

# Conflicts:
#	runtime/src/snapshot_utils.rs
2021-10-12 15:36:51 +00:00
mergify[bot]
9acf708344 Remove support for dynamically loaded native programs (backport #20444) (#20560)
* Remove support for dynamically loaded native programs (#20444)

(cherry picked from commit 785fcb63f5)

# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	program-runtime/src/instruction_processor.rs
#	programs/failure/Cargo.toml
#	programs/failure/tests/failure.rs
#	programs/noop/Cargo.toml
#	programs/ownable/Cargo.toml
#	programs/ownable/src/ownable_processor.rs
#	runtime/src/bank.rs
#	runtime/tests/noop.rs
#	sdk/src/feature_set.rs

* resolve conflicts

Co-authored-by: Jack May <jack@solana.com>
2021-10-11 23:55:39 +00:00
Lijun Wang
7d0494fcaa Merge AccountsDb plugin framework to v1.8 (#20518)
Merge AccountsDb plugin framework to v1.8 (#20518)
Summary of Changes

Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows

Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.

The code comprises 4 major parts:

accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-10-07 14:15:05 -07:00
Tao Zhu
348ba57b12 Bump version to 1.8.1 2021-10-06 17:57:06 -07:00
Tao Zhu
db85d659b9 Cost model 1.7 (#20188)
* Cost Model to limit transactions which are not parallelizeable (#16694)

* * Add following to banking_stage:
  1. CostModel as immutable ref shared between threads, to provide estimated cost for transactions.
  2. CostTracker which is shared between threads, tracks transaction costs for each block.

* replace hard coded program ID with id() calls

* Add Account Access Cost as part of TransactionCost. Account Access cost are weighted differently between read and write, signed and non-signed.

* Establish instruction_execution_cost_table, add function to update or insert instruction cost, unit tested. It is read-only for now; it allows Replay to insert realtime instruction execution costs to the table.

* add test for cost_tracker atomically try_add operation, serves as safety guard for future changes

* check cost against local copy of cost_tracker, return transactions that would exceed limit as unprocessed transaction to be buffered; only apply bank processed transactions cost to tracker;

* bencher to new banking_stage with max cost limit to allow cost model being hit consistently during bench iterations

* replay stage feed back program cost (#17731)

* replay stage feeds back realtime per-program execution cost to cost model;

* program cost execution table is initialized into empty table, no longer populated with hardcoded numbers;

* changed cost unit to microsecond, using value collected from mainnet;

* add ExecuteCostTable with fixed capacity for security concern, when its limit is reached, programs with old age AND less occurrence will be pushed out to make room for new programs.

* investigate system performance test degradation  (#17919)

* Add stats and counter around cost model ops, mainly:
- calculate transaction cost
- check transaction can fit in a block
- update block cost tracker after transactions are added to block
- replay_stage to update/insert execution cost to table

* Change mutex on cost_tracker to RwLock

* removed cloning cost_tracker for local use, as the metrics show clone is very expensive.

* acquire and hold locks for block of TXs, instead of acquire and release per transaction;

* remove redundant would_fit check from cost_tracker update execution path

* refactor cost checking with less frequent lock acquiring

* avoid many Transaction_cost heap allocation when calculate cost, which
is in the hot path - executed per transaction.

* create hashmap with new_capacity to reduce runtime heap realloc.

* code review changes: categorize stats, replace explicit drop calls, concisely initiate to default

* address potential deadlock by acquiring locks one at time

* Persist cost table to blockstore (#18123)

* Add `ProgramCosts` Column Family to blockstore, implement LedgerColumn; add `delete_cf` to Rocks
* Add ProgramCosts to compaction excluding list alone side with TransactionStatusIndex in one place: `excludes_from_compaction()`

* Write cost table to blockstore after `replay_stage` replayed active banks; add stats to measure persist time
* Deletes program from `ProgramCosts` in blockstore when they are removed from cost_table in memory
* Only try to persist to blockstore when cost_table is changed.
* Restore cost table during validator startup

* Offload `cost_model` related operations from replay main thread to dedicated service thread, add channel to send execute_timings between these threads;
* Move `cost_update_service` to its own module; replay_stage is now decoupled from cost_model.

* log warning when channel send fails (#18391)

* Aggregate cost_model into cost_tracker (#18374)

* * aggregate cost_model into cost_tracker, decouple it from banking_stage to prevent accidental deadlock. * Simplified code, removed unused functions

* review fixes

* update ledger tool to restore cost table from blockstore (#18489)

* update ledger tool to restore cost model from blockstore when compute-slot-cost

* Move initialize_cost_table into cost_model, so the function can be tested and shared between validator and ledger-tool

* refactor and simplify a test

* manually fix merge conflicts

* Per-program id timings (#17554)

* more manual fixing

* solve a merge conflict

* featurize cost model

* more merge fix

* cost model uses compute_unit to replace microsecond as cost unit
(#18934)

* Reject blocks for costs above the max block cost (#18994)

* Update block max cost limit to fix performance regession (#19276)

* replace function with const var for better readability (#19285)

* Add few more metrics data points (#19624)

* periodically report sigverify_stage stats (#19674)

* manual merge

* cost model nits (#18528)

* Accumulate consumed units (#18714)

* tx wide compute budget (#18631)

* more manual merge

* ignore zerorize drop security

* - update const cost values with data collected by #19627
- update cost calculation to closely proposed fee schedule #16984

* add transaction cost histogram metrics (#20350)

* rebase to 1.7.15

* add tx count and thread id to stats (#20451)
each stat reports and resets when slot changes

* remove cost_model feature_set

* ignore vote transactions from cost model

Co-authored-by: sakridge <sakridge@gmail.com>
Co-authored-by: Jeff Biseda <jbiseda@gmail.com>
Co-authored-by: Jack May <jack@solana.com>
2021-10-06 15:55:29 -06:00
Trent Nelson
a4df784e82 Bump version to 1.8.0 2021-10-06 15:48:23 -06:00
Justin Starry
d922971ec6 Optimize stakes cache and rewards at epoch boundaries (backport #20432) (#20472)
* Optimize stakes cache and rewards at epoch boundaries (backport #20432)

* fix conflicts
2021-10-06 16:15:27 +00:00
mergify[bot]
95ac00d30a Make rewards tracer async friendly (backport #20452) (#20456)
* Make rewards tracer async friendly (#20452)

(cherry picked from commit 250a8503fe)

# Conflicts:
#	Cargo.lock
#	ledger-tool/Cargo.toml
#	runtime/src/bank.rs

* fix conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2021-10-06 11:20:50 +00:00
mergify[bot]
9f4f8fc9e9 Add struct and convenience methods to track stake activation status (backport #20392) (#20425)
* Add struct and convenience methods to track stake activation status (#20392)

* Add struct and convenience methods to track stake activation status

* fix nits

* rename

(cherry picked from commit 0ddb34a0b4)

# Conflicts:
#	runtime/src/stakes.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2021-10-05 04:33:30 +00:00
Tyera Eulberg
734b380cdb Bump version to v1.7.15 (#20338) 2021-09-30 10:51:34 -06:00
Tyera Eulberg
5bdb824267 Remove original feature gating (#20334) 2021-09-29 22:36:51 -06:00
sakridge
fec15f69f4 Increment 1.7 version (#20316) 2021-09-29 15:37:45 -04:00
sakridge
257ddbeee1 Tpu vote 1.7 (#20187)
* Add separate vote processing tpu port

* Add feature to send to tpu vote port

* Add vote rejecting sigverify mode

* use packet.meta.is_simple_vote_tx in place of deserialization

* consolidate code that identifies vote tx atcommon path for cpu and gpu

* new key for feature set

* banking forward tpu vote

* add tpu vote port to dockerfile and other review changes

* Simplify thread id compare

* fix a test; updated cluster_info ABI change

Co-authored-by: Tao Zhu <tao@solana.com>
2021-09-29 18:12:58 +02:00
mergify[bot]
a005a6b816 Restore ability for programs to upgrade themselves (backport #20265) (#20295)
* Restore ability for programs to upgrade themselves (#20265)

* Make helper associated fn

* Add feature definition

* Add handling to preserve program-id write lock when upgradeable loader is present; restore bpf upgrade-self test

* Use single feature

(cherry picked from commit 2cd9dc99b6)

# Conflicts:
#	runtime/src/accounts.rs
#	sdk/program/src/message.rs
#	sdk/program/src/message/mapped.rs
#	sdk/program/src/message/sanitized.rs
#	sdk/src/feature_set.rs

* Fix conflicts

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-09-29 01:34:33 +00:00
Trent Nelson
7b4638aa0b runtime: remove inactive delegation from stakes cache 2021-09-22 20:43:34 -06:00
sakridge
70d556782b Bump 1.7 version (#19943) 2021-09-16 13:16:09 -06:00
sakridge
ca83167cfc Only allow votes when root distance gets too high (#19933) 2021-09-16 16:48:51 +02:00
mergify[bot]
54ad080bf2 Fix native invoke writable privileges (backport #19750) (#19860)
* Fix native invoke writable privileges (#19750)

* Fix native invoke writable privileges

* build downstream spl bpf programs for tests

(cherry picked from commit 00d7981f64)

# Conflicts:
#	program-runtime/src/instruction_processor.rs
#	runtime/src/message_processor.rs
#	sdk/src/feature_set.rs

* resolve conflictds

Co-authored-by: Jack May <jack@solana.com>
2021-09-16 01:42:26 +00:00
mergify[bot]
97bd521725 Add logging after taking a bank snapshot (backport #19891) (#19920)
* Add logging after taking a bank snapshot (#19891)

(cherry picked from commit 8e3c420414)

* Fix conflict

Co-authored-by: Brooks Prumo <brooks@solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-09-15 14:29:33 -06:00
mergify[bot]
b9a0156a93 Return error if Transaction contains writable executable or ProgramData accounts (backport #19629) (#19730)
* Return error if Transaction contains writable executable or ProgramData accounts (#19629)

* Return error if Transaction locks an executable as writable

* Return error if a ProgramData account is writable but the upgradable loader isn't present

* Remove unreachable clause

* Fixup bpf tests

* Review comments

* Add new TransactionError

* Disallow writes to any upgradeable-loader account when loader not present; remove is_upgradeable_loader_present exception for all other executables

(cherry picked from commit 38bbb77989)

# Conflicts:
#	programs/bpf/tests/programs.rs
#	runtime/src/accounts.rs
#	runtime/src/bank.rs
#	sdk/src/transaction.rs
#	storage-proto/proto/transaction_by_addr.proto
#	storage-proto/src/convert.rs

* Fix conflicts

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-09-09 14:02:07 +00:00
Tyera Eulberg
376b21c6e7 Tiny dedupe 2021-09-04 01:43:23 -06:00
mergify[bot]
fcda5d4a7d Demote write locks on transaction program ids (backport #19593) (#19633)
* Demote write locks on transaction program ids (#19593)

* Add feature

* Demote write lock on program ids

* Fixup bpf tests

* Update MappedMessage::is_writable

* Comma nit

* Review comments

(cherry picked from commit decec3cd8b)

# Conflicts:
#	core/src/banking_stage.rs
#	core/src/cost_model.rs
#	core/src/cost_tracker.rs
#	ledger-tool/src/main.rs
#	program-runtime/src/instruction_processor.rs
#	programs/bpf/tests/programs.rs
#	programs/bpf_loader/src/syscalls.rs
#	rpc/src/transaction_status_service.rs
#	runtime/src/accounts.rs
#	runtime/src/bank.rs
#	runtime/src/message_processor.rs
#	sdk/benches/serialize_instructions.rs
#	sdk/program/src/message/mapped.rs
#	sdk/program/src/message/sanitized.rs
#	sdk/src/transaction/sanitized.rs

* Fix conflicts

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-09-04 06:46:09 +00:00