Commit Graph

264 Commits

Author SHA1 Message Date
mergify[bot]
84644ce994 Ignore delinquent stake on exit (#20367) (#20381)
* add --ignore-delinquency flag to validator exit and wait-for-restart-window sub commands

* Fix a merge issue

* Add missing variable declaration

* Remove empty line to help CI checks pass

* run rustfmt

* Change argument wording for clarity and verbosity

* Change --ignore-delinquent-stake to --max-delinquent-stake

* cargo fmtgit add validator/src/main.rsgit add validator/src/main.rs

* Adjust per mvines

* Formatting

* Improve input validation

* Please automate cargo fmt somehow

(cherry picked from commit fc5dd7f3bc)

Co-authored-by: Michael <68944931+michaelh-laine@users.noreply.github.com>
2021-10-12 19:14:30 +00:00
mergify[bot]
353cb8efa5 solana-validator wait-for-restart-window command now accepts an optional --identity argument (backport #18684) (#20609)
* wait-for-restart-window command now accepts an optional --identity argument

(cherry picked from commit c418e8f370)

# Conflicts:
#	validator/src/main.rs

* Fixed cherry-pick conflicts

Co-authored-by: Michael Vines <mvines@gmail.com>
Co-authored-by: Steven Czabaniuk <steven@solana.com>
2021-10-12 06:15:30 +00:00
Justin Starry
d922971ec6 Optimize stakes cache and rewards at epoch boundaries (backport #20432) (#20472)
* Optimize stakes cache and rewards at epoch boundaries (backport #20432)

* fix conflicts
2021-10-06 16:15:27 +00:00
Michael Vines
00b03897e1 Default --rpc-bind-address to 127.0.0.1 when --private-rpc is provided and --bind-address is not
(cherry picked from commit 221343e849)
2021-10-04 16:58:46 -07:00
sakridge
257ddbeee1 Tpu vote 1.7 (#20187)
* Add separate vote processing tpu port

* Add feature to send to tpu vote port

* Add vote rejecting sigverify mode

* use packet.meta.is_simple_vote_tx in place of deserialization

* consolidate code that identifies vote tx atcommon path for cpu and gpu

* new key for feature set

* banking forward tpu vote

* add tpu vote port to dockerfile and other review changes

* Simplify thread id compare

* fix a test; updated cluster_info ABI change

Co-authored-by: Tao Zhu <tao@solana.com>
2021-09-29 18:12:58 +02:00
mergify[bot]
085f5f945d uses tikv_jemallocator::Jemalloc as the global allocator (backport #20149) (#20166)
* uses tikv_jemallocator::Jemalloc as the global allocator (#20149)

https://github.com/solana-labs/solana/pull/16346
switched default allocator from jemalloc to system allocator, but that
has shown regressions in form of higher ram usage causing nodes go OOM:
https://discord.com/channels/428295358100013066/439194979856809985/890413193858539580

This commit sets jemalloc as the default allocator.

(cherry picked from commit 2cf081d863)

# Conflicts:
#	Cargo.lock

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-09-24 17:37:57 +00:00
mergify[bot]
b112e4a8aa windows: Make solana-test-validator work (backport #20099) (#20123)
* windows: Make solana-test-validator work (#20099)

* windows: Make solana-test-validator work

The important changes to get this going on Windows:

* ledger lock needs to be done on a file instead of the directory
* IPC service needs to use the Windows pipe naming scheme
* always disable the JIT
* file logging not possible yet because we can't redirect stderr,
but this will change once env_logger fixes the pipe output target!

* Integrate review feedback

(cherry picked from commit 567f30aa1a)

# Conflicts:
#	validator/src/bin/solana-test-validator.rs
#	validator/src/lib.rs
#	validator/src/main.rs

* Fix merge conflicts

Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
2021-09-24 12:59:12 +00:00
mergify[bot]
03da3eaa81 Optimize RPC pubsub for multiple clients with the same subscription (backport #18943) (#19987)
* Optimize RPC pubsub for multiple clients with the same subscription (#18943)

* reimplement rpc pubsub with a broadcast queue

* update tests for new pubsub implementation

* fix: fix review suggestions

* chore(rpc): add additional pubsub metrics

* integrate max subscriptions check into SubscriptionTracker to reduce locking

* separate subscription control from tracker

* limit memory usage of items in pubsub broadcast queue, improve error handling

* add more pubsub metrics

* add final count metrics to pubsub

* add metric for total number of subscriptions

* fix small review suggestions

* remove by_params from SubscriptionTracker and add node_progress_watchers map instead

* add subscription tracker tests

* add metrics for number of pubsub notifications as a counter

* ignore clippy lint in TokenCounter

* fix underflow in token counter

* reduce queue capacity in pubsub tests

* fix(rpc): fix test timeouts

* fix race in account subscription test

* Add RpcSubscriptions::new_for_tests

Co-authored-by: Pavel Strakhov <p.strakhov@iconic.vc>
Co-authored-by: Nikita Podoliako <n.podoliako@zubr.io>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
(cherry picked from commit 65227f44dc)

# Conflicts:
#	Cargo.lock
#	core/Cargo.toml
#	core/src/replay_stage.rs
#	core/src/validator.rs
#	replica-node/src/replica_node.rs
#	rpc/Cargo.toml

* Fix conflicts (and standardize naming to make future subscription backports easier

Co-authored-by: Pavel Strakhov <ri@idzaaus.org>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-09-20 06:00:08 +00:00
mergify[bot]
caea9c99cd Minimize trust (backport #19279) (#19335)
* docs: Mainnet Beta inflation has been enabled for quite some time

(cherry picked from commit 169ded9a70)

* validator: Trusted validators are now called known validators

(cherry picked from commit e0bc5fa690)

* docs: trust minimize

(cherry picked from commit 40613161a0)

* docs: correct known validator operator

(cherry picked from commit eced50d103)

* docs: Remove decommissioned testnet archetype validator

(cherry picked from commit a587eec20b)

* docs: update devnet start args with new validators

(cherry picked from commit 2a877ae06e)

Co-authored-by: Trent Nelson <trent@solana.com>
2021-08-20 06:24:43 +00:00
mergify[bot]
7584262f47 add validator option --accounts-db-skip-shrink (backport #19028) (#19128)
* add validator option --accounts-db-skip-shrink (#19028)

* add validator option --accounts-db-skip-shrink

* typo

(cherry picked from commit 3280ae3e9f)

# Conflicts:
#	core/tests/snapshots.rs
#	ledger/src/bank_forks_utils.rs
#	ledger/src/blockstore_processor.rs
#	replica-node/src/replica_node.rs
#	runtime/src/snapshot_utils.rs

* Fix conflicts

Co-authored-by: Jeff Washington (jwash) <75863576+jeffwashington@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-08-10 10:07:27 -06:00
mergify[bot]
b142ef5f8b Improve target CPU feature checks (backport #18940) (#18958)
* Improve check for Apple M1 silicon under Rosetta

(cherry picked from commit 59641623d1)

# Conflicts:
#	core/src/validator.rs

* test-validator: start logging asap

(cherry picked from commit ee65ffb3c8)

* test-validator: move feature check earlier in startup

(cherry picked from commit e641f257ef)

* validator: add avx2 runtime check

(cherry picked from commit c435f7b3e3)

* validator: start logging asap

(cherry picked from commit ed8285c096)

* validator: check target CPU features earlier

(cherry picked from commit 8ed0cd0fff)

* validator: remove disused cuda config argument

(cherry picked from commit 71f6d839f9)

Co-authored-by: Trent Nelson <trent@solana.com>
2021-07-30 06:47:22 +00:00
mergify[bot]
eacc69efba adds validator flag to allow private ip addresses (backport #18850) (#18975)
* adds validator flag to allow private ip addresses (#18850)

(cherry picked from commit d2d5f36a3c)

# Conflicts:
#	accounts-cluster-bench/Cargo.toml
#	bench-tps/Cargo.toml
#	cli/Cargo.toml
#	core/benches/cluster_info.rs
#	core/src/banking_stage.rs
#	core/src/broadcast_stage.rs
#	core/src/broadcast_stage/broadcast_duplicates_run.rs
#	core/src/broadcast_stage/fail_entry_verification_broadcast_run.rs
#	core/src/broadcast_stage/standard_broadcast_run.rs
#	core/src/cluster_slots_service.rs
#	core/src/repair_service.rs
#	core/src/tvu.rs
#	core/src/validator.rs
#	dos/Cargo.toml
#	gossip/src/cluster_info.rs
#	gossip/src/crds_gossip_pull.rs
#	gossip/src/crds_gossip_push.rs
#	gossip/src/gossip_service.rs
#	local-cluster/Cargo.toml
#	local-cluster/src/cluster_tests.rs
#	local-cluster/tests/local_cluster.rs
#	rpc/Cargo.toml
#	rpc/src/rpc.rs
#	tokens/Cargo.toml
#	validator/Cargo.toml
#	validator/src/main.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-07-29 21:43:24 +00:00
mergify[bot]
363b75619f adds shred-version to ip-echo-server response (backport #18066) (#18113)
* adds shred-version to ip-echo-server response

When starting a validator, the node initially joins gossip with
shred_verison = 0, until it adopts the entrypoint's shred-version:
https://github.com/solana-labs/solana/blob/9b182f408/validator/src/main.rs#L417

Depending on the load on the entrypoint, this adopting entrypoint
shred-version through gossip sometimes becomes very slow, and causes
several problems in gossip because we have to partially support
shred_version == 0 which is a source of leaking crds values from one
cluster to another. e.g. see
https://github.com/solana-labs/solana/pull/17899
and the other linked issues there.

In order to remove shred_version == 0 from gossip, this commit adds
shred-version to ip-echo-server response. Once the entrypoints are
updated, on validator start-up, if --expected_shred_version is not
specified we will obtain shred-version from the entrypoint using
ip-echo-server.

(cherry picked from commit 598093b5db)

# Conflicts:
#	Cargo.lock
#	net-utils/Cargo.toml
#	programs/bpf/Cargo.lock

* removes backport merge conflicts

* obtains shred-version from entrypoint's ip-echo-server in validator-main

(cherry picked from commit 58e115275a)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-06-21 23:04:08 +00:00
mergify[bot]
0e7512a225 Fix Nightly Clippy Warnings (backport #18065) (#18070)
* chore: cargo +nightly clippy --fix -Z unstable-options

(cherry picked from commit 6514096a67)

# Conflicts:
#	core/src/banking_stage.rs
#	core/src/cost_model.rs
#	core/src/cost_tracker.rs
#	core/src/execute_cost_table.rs
#	core/src/replay_stage.rs
#	core/src/tvu.rs
#	ledger-tool/src/main.rs
#	programs/bpf_loader/build.rs
#	rbpf-cli/src/main.rs
#	sdk/cargo-build-bpf/src/main.rs
#	sdk/cargo-test-bpf/src/main.rs
#	sdk/src/secp256k1_instruction.rs

* chore: cargo fmt

(cherry picked from commit 789f33e8db)

* Updates BPF program assert_instruction_count tests.

(cherry picked from commit c1e03f3410)

# Conflicts:
#	programs/bpf/tests/programs.rs

* Resolve conflicts

Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-06-18 20:02:48 +00:00
mergify[bot]
e9c234d89f Make account shrink configurable #17544 (backport #17778) (#18013)
* Make account shrink configurable #17544 (#17778)

1. Added both options for measuring space usage using total accounts usage and for individual store shrink ratio using an enum. Validator CLI options: --accounts-shrink-optimize-total-space and --accounts-shrink-ratio
2. Added code for selecting candidates based on total usage in a separate function select_candidates_by_total_usage
3. Added unit tests for the new functions added
4. The default implementations is kept at 0.8 shrink ratio with --accounts-shrink-optimize-total-space set to true

Fixes #17544

(cherry picked from commit 269d995832)

# Conflicts:
#	core/tests/snapshots.rs
#	ledger/src/bank_forks_utils.rs
#	runtime/src/accounts_db.rs
#	runtime/src/snapshot_utils.rs

* fix some merge errors

Co-authored-by: Lijun Wang <83639177+lijunwangs@users.noreply.github.com>
Co-authored-by: Jeff Washington (jwash) <wash678@gmail.com>
2021-06-17 17:32:03 +00:00
mergify[bot]
b472dac6b3 validator: expose max active pubsub subscriptions to CLI (#18036)
(cherry picked from commit 5efc48fc69)

Co-authored-by: Trent Nelson <trent@solana.com>
2021-06-17 08:02:20 +00:00
mergify[bot]
e247625025 Create solana-poh and move remaining rpc modules to solana-rpc (backport #17698) (#17745)
* Create solana-poh and move remaining rpc modules to solana-rpc (#17698)

* Create solana-poh crate

* Move BigTableUploadService to solana-ledger

* Add solana-rpc to workspace

* Move dependencies to solana-rpc

* Move remaining rpc modules to solana-rpc

* Single use statement solana-poh

* Single use statement solana-rpc

(cherry picked from commit 544b3c0d17)

# Conflicts:
#	Cargo.lock
#	banking-bench/Cargo.toml
#	core/Cargo.toml
#	core/benches/banking_stage.rs
#	local-cluster/Cargo.toml
#	rpc/Cargo.toml
#	stake-monitor/Cargo.toml
#	validator/Cargo.toml

* Fix conflicts & versions

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-06-04 18:19:08 +00:00
mergify[bot]
5e7db52087 Avoid full-range compactions with periodic filtered b.g. ones (backport #16697) (#17741)
* Avoid full-range compactions with periodic filtered b.g. ones (#16697)

* Update rocksdb to v0.16.0

* Promote the infrequent and important log to info!

* Force background compaction by ttl without manual compaction

* Fix test

* Support no compaction mode in test_ledger_cleanup_compaction

* Fix comment

* Make compaction_interval customizable

* Avoid major compaction with periodic filtering...

* Adress lazy_static, special cfs and range check

* Clean up a bit and add comment

* Add comment

* More comments...

* Config code cleanup

* Add comment

* Use .conflicts_with()

* Nullify unneeded delete_range ops for special CFs

* Some clean ups

* Clarify the locking intention

* Ensure special CFs' consistency with PurgeType::CompactionFilter

* Fix comment

* Fix bad copy paste

* Fix various types...

* Don't use tuples

* Add a unit test for compaction_filter

* Fix typo...

* Remove flag and just use new behavior always

* Fix wrong condition negation...

* Doc. about no set_last_purged_slot in purge_slots

* Write a test and fix off-by-one bug....

* Apply suggestions from code review

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* Follow up to github review suggestions

* Fix line-wrapping

* Fix conflict

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
(cherry picked from commit 1f97b2365f)

# Conflicts:
#	ledger/src/blockstore_db.rs

* Fix conflict

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-06-04 14:38:02 +00:00
Lijun Wang
54f0fc9f0f Use type alias for DownloadProgress callback (#17518)
Convert to use type alias for the callback and cascade the changes to callers. Thanks @jeffwashington for the help making it possible.
Changed the closure for the progress update in the validator main to FnMut and modify the abort count in the closure which is more reliable.
2021-05-26 13:26:07 -07:00
Tyera Eulberg
9a5330b7eb Move gossip modules into solana-gossip crate (#17352)
* Move gossip modules to solana-gossip

* Update Protocol abi digest due to move

* Move gossip benches and hook up CI

* Remove unneeded Result entries

* Single use statements
2021-05-26 09:15:46 -06:00
Lijun Wang
4c17243157 snapshot download enhancement (#17415)
1. Allow the validator bootstrap code to specify the minimal snapshot download speed. If the snapshot download speed is detected below that, a different RPC can be retried. The default is 10MB/sec.

2. To prevent spinning on a number of sub-optimal choices and not making progress, the abort/retry logic is implemented with the following safe guards:
2.1 at maximum we do this retry for 5 times -- this number is configurable with default 5.
2.2 if the download in one notification round (5 second) is more than 2%, do not do retry -- it is not as bad anyway.
2.3 if the remaining estimate time is less than 1 minutes, do not abort retry as it will be done quickly anyway.
2.4 We do this abort/retry logic only at the first notification to avoid wasting download efforts -- the reasoning is being opportunistic and too greedy may not achieve overall shorter download time.

3. The download_snapshot and download_file is modified with the option allowing caller to notified of download progress via a callback. This allows the business logic of retrying to the place it belongs.
2021-05-25 09:32:12 -07:00
Tyera Eulberg
41ec1c8d50 Add blockstore-root-scan for api nodes on boot (#17402)
* Add blockstore-root-scan for api nodes on boot

* Ensure cluster-confirmed root and parents are set as root in blockstore in load_frozen_forks()

* Plumb rpc-scan-and-fix-roots validator flag
2021-05-24 13:24:47 -06:00
sakridge
a8dca3976b Refactor genesis download/load/check functions (#17276)
* Refactor genesis ingest functions

* Consolidate genesis.bin/genesis.tar.bz2 references
2021-05-24 16:45:36 +02:00
Tyera Eulberg
827355a6b1 Create solana-rpc crate and move subscriptions (#17320)
* Move non_circulating_supply to runtime

* Add solana-rpc crate and move max_slots

* Move subscriptions to solana-rpc

* Single use statements
2021-05-19 00:54:28 -06:00
Tyera Eulberg
27004f1b76 Return error for excluded secondary-index keys (#17193)
* Add runtime helpers to check secondary indexes for key

* Add custom rpc error

* Check secondary-index key inclusion in rpc

* Clone complete AccountSecondaryIndexes into rpc to avoid bank query
2021-05-13 21:04:21 +00:00
Lijun Wang
9c42a89a43 Issue #17008 -- make snapshot archives to hold on to configurable. (#17158)
* purge_old_snapshot_archives is changed to take an extra argument 'maximum_snapshots_to_retain' to control the max number of latest snapshot archives to retain. Note the oldest snapshot is always retained as before and is not subjected to this new options.
* The validator and ledger-tool executables are modified with a CLI argument --maximum-snapshots-to-retain. And the options are propagated down the call chains. Their corresponding shell scripts were changed accordingly.
* SnapshotConfig is modified to have an extra field for the maximum_snapshots_to_retain
* Unit tests are developed to cover purge_old_snapshot_archives
2021-05-12 10:32:27 -07:00
Jeff Washington (jwash)
7d96f78821 include/exclude keys on account secondary index (#17110)
* AccountSecondaryIndexes.include/exclude

* use normal scan if key is not indexed

* add a test to ask for a scan for an excluded secondary index

* fix up cli args
2021-05-11 22:06:22 +00:00
Michael Vines
9ba2c53b85 Add --tower argument to specify where tower files are persisted 2021-05-05 12:20:39 -07:00
Michael Vines
6004c0abf5 getLeaderSchedule now supports filtered results based on validator identity 2021-04-21 17:59:26 -07:00
Tyera Eulberg
0924c2d070 Add port and gossip options to solana-test-validator (#16696) 2021-04-21 02:40:52 +00:00
Tyera Eulberg
70f3f7e679 Move obsolete rpc endpoints to separate api for removal (#16500)
* Move obsolete rpc methods to separate api for removal

* Remove obsolete method from docs

* Fix test using obs method
2021-04-12 20:33:40 -06:00
Michael Vines
05ad979a2d canonicalize authorized voter filepath 2021-04-12 17:52:55 -07:00
Michael Vines
2229b70c4e Add authorized-voter add/remove-all commands 2021-04-12 15:55:28 -07:00
Michael Vines
ef30943c5c Fix up App formatting 2021-04-11 22:32:56 -07:00
Michael Vines
a679aebc82 wait-for-restart-window works again for unstaked nodes 2021-04-04 16:52:39 +00:00
Michael Vines
c8c89dd5f7 wait-for-restart-window now indicates how far away the next restart window is 2021-04-02 22:45:32 -07:00
sakridge
b99ae8f334 Skip leader slots until a vote lands (#15607) 2021-03-25 18:54:51 -07:00
DimAn
20b53eb4b4 Revert to snapshots 2 2021-03-17 10:31:01 -07:00
DimAn
0b42379ed7 Revert to snapshots
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-03-17 10:31:01 -07:00
DimAn
a43b3674c7 add missed suggestion 2021-03-17 10:31:01 -07:00
DimAn
cfb01e26dd Apply suggestions from code review
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-03-17 10:31:01 -07:00
DimAn
0209d334bd Add option for separate snapshot location
(cherry picked from commit 6126878f509c69e23480a5ec22b3271e2b16e072)
2021-03-17 10:31:01 -07:00
Michael Vines
eab182188a Fix delinquent stake display 2021-03-13 18:58:04 +00:00
Michael Vines
aa2b2d6b75 Default --ledger arg to "ledger" for solana-validator and solana-ledger-tool 2021-03-11 13:09:30 -08:00
Michael Vines
ac8ccee6b8 Add --force and --monitor options to exit subcommand 2021-03-11 05:17:31 +00:00
Michael Vines
c836cd85c3 Default to the BPF JIT. Use the solana-validator --no-bpf-jit flag to disable 2021-03-10 08:37:35 -08:00
Michael Vines
76ddc93358 The --identity argument is now required 2021-03-06 19:08:37 -08:00
Michael Vines
04d11ca6c6 Slow down solana-validator monitor refresh interval when talking to a real node 2021-03-07 00:51:02 +00:00
sakridge
d09112fa6d PoH batch size calibration (#15717) 2021-03-05 16:01:21 -08:00
Michael Vines
bd13262b42 Add validator startup process reporting before RPC is available 2021-03-05 08:03:36 -08:00