gaspersic/solana
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
11,312 Commits 81 Branches 357 Tags
ad5ef9cc485cf49daa25d8a3c44e6600b9d35fd8
Commit Graph

232 Commits

Author SHA1 Message Date
mergify[bot]
39eae50024 implements DataBudget using atomics (#12856) (#13157) 
(cherry picked from commit 05cf15a382)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
 2020-10-26 18:27:45 +00:00
mergify[bot]
7d66cba7f4 Remove spammy invalid rpc log (#13100) (#13101) 
(cherry picked from commit c95f6c4b83)

Co-authored-by: Justin Starry <justin@solana.com>
 2020-10-23 08:21:33 +00:00
mergify[bot]
b90b46fee1 Allow nodes to advertise a different rpc address over gossip (#13053) (#13077) 
* Allow nodes to advertise a different rpc address over gossip

* Feedback

(cherry picked from commit 8b0242a5d8)

Co-authored-by: Justin Starry <justin@solana.com>
 2020-10-22 04:44:41 +00:00
Michael Vines
5ad0ccdfe1 cargo fmt 2020-10-21 20:27:57 -07:00
Michael Vines
422bb3c526 Run codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand 2020-10-21 20:27:57 -07:00
mergify[bot]
fbe5a89e74 retains hash value of outdated responses received from pull requests (#12513) (#12603) 
pull_response_fail_inserts has been increasing:
https://cdn.discordapp.com/attachments/478692221441409024/759096187587657778/pull_response_fail_insert.png
but for outdated values which fail to insert:
https://github.com/solana-labs/solana/blob/a5c3fc14b3/core/src/crds_gossip_pull.rs#L332-L344
https://github.com/solana-labs/solana/blob/a5c3fc14b3/core/src/crds.rs#L104-L108
are not recorded anywhere, and so the next pull request may obtain the
same redundant payload again, unnecessary taking bandwidth.

This commit holds on to the hashes of failed-inserts for a while, similar
to purged_values:
https://github.com/solana-labs/solana/blob/a5c3fc14b3/core/src/crds_gossip_pull.rs#L380
and filter them out for the next pull request:
https://github.com/solana-labs/solana/blob/a5c3fc14b3/core/src/crds_gossip_pull.rs#L204

(cherry picked from commit 1866521df6)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
 2020-10-01 01:47:20 +00:00
mergify[bot]
55b0e9e9c7 builds crds filters in parallel (bp #12360) (#12571) 
* builds crds filters in parallel (#12360)

Based on run-time profiles, the majority time of new_pull_requests is
spent building bloom filters, in hashing and bit-vec ops.

This commit builds crds filters in parallel using rayon constructs. The
added benchmark shows ~5x speedup (4-core machine, 8 threads).

(cherry picked from commit 537bbde22e)

# Conflicts:
#	core/Cargo.toml

* resolves mergify merge conflict

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
 2020-09-30 01:08:16 +00:00
mergify[bot]
a07e90516b separates out ClusterInfo::{gossip,listen} thread-pools (#12535) (#12547) 
https://github.com/solana-labs/solana/pull/12402
moved gossip-work threads:
https://github.com/solana-labs/solana/blob/afd9bfc45/core/src/cluster_info.rs#L2330-L2334
to ClusterInfo::new as a new field in the ClusterInfo struct:
https://github.com/solana-labs/solana/blob/35208c5ee/core/src/cluster_info.rs#L249
So that they can be shared between listen and gossip threads:
https://github.com/solana-labs/solana/blob/afd9bfc45/core/src/gossip_service.rs#L54-L67

However, in testing https://github.com/solana-labs/solana/pull/12360
it turned out this will cause breakage:
https://buildkite.com/solana-labs/solana/builds/31646
https://buildkite.com/solana-labs/solana/builds/31651
https://buildkite.com/solana-labs/solana/builds/31655
Whereas with separate thread pools all is good. It might be the case
that one thread is slowing down the other by exhausting the thread-pool
whereas with separate thread-pools we get fair scheduling guarantees
from the os.

This commit reverts https://github.com/solana-labs/solana/pull/12402
and instead adds separate thread-pools for listen and gossip threads:
https://github.com/solana-labs/solana/blob/afd9bfc45/core/src/gossip_service.rs#L54-L67

(cherry picked from commit 0d5258b6d3)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
 2020-09-29 11:27:47 +00:00
mergify[bot]
bc3aa53e02 Runtime feature activation framework (bp #12376) (#12497) 
* Runtime feature activation framework

(cherry picked from commit 93259f0bae)

# Conflicts:
#	runtime/src/bank.rs

* Add feature set identifier to gossiped version information

(cherry picked from commit 35f5f9fc7b)

# Conflicts:
#	Cargo.lock
#	version/Cargo.toml

* Port instructions sysvar and secp256k1 program activation to FeatureSet

(cherry picked from commit c10da16d7b)

# Conflicts:
#	runtime/src/bank.rs
#	runtime/src/message_processor.rs

* Add feature management commands

(cherry picked from commit 93ed0ab2bb)

# Conflicts:
#	Cargo.lock
#	cli/Cargo.toml

* Make test_process_rest_api less fragile

(cherry picked from commit 7526bb96f3)

* Remove id field

(cherry picked from commit cc6ba1e131)

* FeatureSet test

(cherry picked from commit 92406cf9a0)

* cargo fmt

(cherry picked from commit 199940d683)

* cli review feedback

(cherry picked from commit 3a2b8c5e5b)

* Rename active() to is_active()

(cherry picked from commit e39fac9f01)

* Resolve merge conflicts

* Remove continues from compute_active_feature_set()

Co-authored-by: Michael Vines <mvines@gmail.com>
 2020-09-26 17:49:53 +00:00
mergify[bot]
72365bb9d2 moves gossip-work thread pool cons to ClusterInfo::new (#12402) (#12458) 
(cherry picked from commit 42f1ef8acb)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
 2020-09-24 20:01:03 +00:00
mergify[bot]
b5c7ad3a9b Add new validator options for running in more restrictive environments (bp #12191) (#12218) 
* Add --restricted-repair-only-mode flag

(cherry picked from commit 63a67f415e)

* Add --gossip-validator argument

(cherry picked from commit daae638781)

* Documenet how to reduce validator port exposure

(cherry picked from commit c8f03c7f6d)

Co-authored-by: Michael Vines <mvines@gmail.com>
 2020-09-15 04:26:23 +00:00
Greg Fitzgerald
e938925b95 Add BanksClient (#11721) 
Cherry-picked from #10728, but without the changes to solana-tokens
 2020-08-19 22:24:24 -06:00
mergify[bot]
2c642d4639 Filter push/pulls from spies (#11620) (#11703) 
* Filter push/pulls from spies

* Don't pull from peers with shred version == 0, don't push to people with shred_version == 0

Co-authored-by: Carl <carl@solana.com>
(cherry picked from commit 0f0a2ddafe)

Co-authored-by: carllin <wumu727@gmail.com>
 2020-08-19 03:20:43 +00:00
mergify[bot]
311d9a56c4 Add incoming pull response counter (#11591) (#11592) 
(cherry picked from commit 54137e3446)

Co-authored-by: sakridge <sakridge@gmail.com>
 2020-08-12 22:17:50 +00:00
mergify[bot]
69e53ec92a Gossip log (#11555) (#11562) 
Co-authored-by: Carl <carl@solana.com>
(cherry picked from commit 1b238dd63e)

Co-authored-by: carllin <wumu727@gmail.com>
 2020-08-11 22:55:18 +00:00
mergify[bot]
61af485732 filter out old gossip pull requests (#11448) (#11553) 
* init

* builds

* stats

* revert

* tests

* clippy

* add some jitter

* shorter jitter timer

* update

* fixup! update

* use saturating_sub

* fix filters

(cherry picked from commit 713851b68d)

Co-authored-by: anatoly yakovenko <anatoly@solana.com>
 2020-08-11 19:42:32 +00:00
Michael Vines
eefcf484cb clippy 2020-08-03 18:35:15 +00:00
Ryo Onodera
39b3ac6a8d Introduce automatic ABI maintenance mechanism (2/2; rollout) (#8012) 
* Introduce automatic ABI maintenance mechanism (2/2; rollout)

* Fix stable clippy

* Change to symlink

* Freeze abi of Tower

* fmt...

* Improve dev-experience!

* Update BankSlotDelta

$ diff -u /tmp/abi8/*7dg6BreYxTuxiVz6aLvk3p2Z7GQk2cJqfGvC9h4FAoSj* /tmp/abi8/*9chBcbXVJ4fK7uGgydQzam5aHipaAKFw6V4LDFpjbE4w*
--- /tmp/abi8/bank__BankSlotDelta_frozen_abi__test_abi_digest_7dg6BreYxTuxiVz6aLvk3p2Z7GQk2cJqfGvC9h4FAoSj      2020-06-18 18:01:22.831228087 +0900
+++ /tmp/abi8/bank__BankSlotDelta_frozen_abi__test_abi_digest_9chBcbXVJ4fK7uGgydQzam5aHipaAKFw6V4LDFpjbE4w      2020-07-03 15:59:58.430695244 +0900
@@ -140,7 +140,7 @@
                                                         field u8
                                                             primitive u8
                                                         field solana_sdk::instruction::InstructionError
-                                                            enum InstructionError (variants = 34)
+                                                            enum InstructionError (variants = 35)
                                                                 variant(0) GenericError (unit)
                                                                 variant(1) InvalidArgument (unit)
                                                                 variant(2) InvalidInstructionData (unit)
@@ -176,6 +176,7 @@
                                                                 variant(31) CallDepth (unit)
                                                                 variant(32) MissingAccount (unit)
                                                                 variant(33) ReentrancyNotAllowed (unit)
+                                                                variant(34) MaxSeedLengthExceeded (unit)
                                                     variant(9) CallChainTooDeep (unit)
                                                     variant(10) MissingSignatureForFee (unit)
                                                     variant(11) InvalidAccountIndex (unit)

* Fix some merge conflicts...
 2020-07-06 20:22:23 +09:00
sakridge
d9b389f510 Reduce logging lines (#10835) 2020-06-29 15:57:28 -07:00
Greg Fitzgerald
6ee222363e Move BankForks to solana_runtime (#10637) 
* Move BankForks to solana_runtime

* Update imports
 2020-06-17 15:27:03 +00:00
sakridge
1eca9b19ab Entry verify cleanup and gossip counters (#10632) 
* Add prune message counter

* Switch to us verification time to match other counters

* Add separate transaction/poh verify timing
 2020-06-16 14:00:29 -07:00
anatoly yakovenko
ba83e4ca50 Fix fannout gossip bench (#10509) 
* Gossip benchmark

* Rayon tweaking

* push pulls

* fanout to max nodes

* fixup! fanout to max nodes

* fixup! fixup! fanout to max nodes

* update

* multi vote test

* fixup prune

* fast propagation

* fixups

* compute up to 95%

* test for specific tx

* stats

* stats

* fixed tests

* rename

* track a lagging view of which nodes have the local node in their active set in the local received_cache

* test fixups

* dups are old now

* dont prune your own origin

* send vote to tpu

* tests

* fixed tests

* fixed test

* update

* ignore scale

* lint

* fixup

* fixup

* fixup

* cleanup

Co-authored-by: Stephen Akridge <sakridge@gmail.com>
 2020-06-13 22:03:38 -07:00
sakridge
4c140acb3b ClusterInfo cleanup (#10504) 
automerge
 2020-06-10 17:00:17 -07:00
sakridge
6eb5ef6ac7 Add back missing pull_response success counter (#10491) 2020-06-10 09:17:57 -07:00
sakridge
ecb6959720 Optimize process pull responses (#10460) 
* Batch process pull responses

* Generate pull requests at 1/2 rate

* Do filtering work of process_pull_response in read lock

Only take write lock to insert if needed.
 2020-06-09 17:08:13 -07:00
anatoly yakovenko
832d324a23 Revert "Gossip PullRequests tend to return a lot of duplicates. (#10326)" (#10455) 
This reverts commit 31e20eff82.
 2020-06-09 07:27:00 -07:00
Kristofer Peterson
e23340d89e Clippy cleanup for all targets and nighly rust (also support 1.44.0) (#10445) 
* address warnings from 'rustup run beta cargo clippy --workspace'

minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs

expect some performance improvement AccountsDB::clean_accounts()

* address warnings from 'rustup run beta cargo clippy --workspace --tests'

* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'

* rustfmt

* fix warning stragglers

* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned

* Remove lock around JsonRpcRequestProcessor (#10417)

automerge

* make ancestors parameter optional to avoid forcing construction of empty hash maps

Co-authored-by: Greg Fitzgerald <greg@solana.com>
 2020-06-09 09:38:14 +09:00
Greg Fitzgerald
af8c21c559 Remove lock around JsonRpcRequestProcessor (#10417) 
automerge
 2020-06-07 20:54:03 -07:00
sakridge
0645a0c96d Gossip cleanup remove duplicate gossip metrics and name worker threads (#10435) 
Refactor into functions
 2020-06-06 15:05:45 -07:00
sakridge
3d2230f1a9 Add pull request count metrics (#10421) 2020-06-05 09:36:31 -07:00
anatoly yakovenko
31e20eff82 Gossip PullRequests tend to return a lot of duplicates. (#10326) 
* filter messages that are likely to be pushed from the response

* tests

* tests

* wait to start filtering responses, and push stats to influx

* wait to start filtering responses, and push stats to influx

* reduce the timers to match the publish self timeout

* fmt

* fmt
 2020-06-05 08:01:45 -07:00
sakridge
ef37b82ffa More cluster stats and add epoch stakes cache in retransmit stage (#10345) 
* More cluster info metrics for push request/response counts

* Cache staked peers for the epoch
 2020-06-01 08:37:54 -07:00
Kristofer Peterson
fb4d8e1f62 cleanup clippy tests (#10172) 
automerge
 2020-05-29 00:26:06 -07:00
sakridge
3f508b37fd Skip gossip requests with different shred version and split lock (#10240) 2020-05-28 11:38:13 -07:00
sakridge
7ebd8ee531 Cluster info metrics (#10215) 2020-05-25 15:03:34 -07:00
carllin
3aae98c8be Add switching vote instruction (#10197) 
* Add switching vote

* Make sure vote size stays under gossip limit

Co-authored-by: Carl <carl@solana.com>
 2020-05-24 15:38:35 -07:00
Kristofer Peterson
58ef02f02b 9951 clippy errors in the test suite (#10030) 
automerge
 2020-05-15 09:35:43 -07:00
Jack May
eb1acaf927 Remove archiver and storage program (#9992) 
automerge
 2020-05-14 18:22:47 -07:00
Michael Vines
4e4a21f9b7 solana-gossip spy can now specify a shred version (#10040) 2020-05-13 19:37:40 -07:00
Michael Vines
2521f75c18 Advertise node software version in gossip (#9981) 
* Advertise node version in gossip

* Remove solana_clap_utils::version! macro
 2020-05-11 15:02:01 -07:00
Michael Vines
16ddd001f6 Gossip no longer pushes/pulls from nodes with a different shred version (#9868) 2020-05-05 20:15:19 -07:00
anatoly yakovenko
a0514eb2ae thiserror, docs, remove general Failure case (#9741) 
automerge
 2020-04-29 18:12:51 -07:00
anatoly yakovenko
8ef097bf6f Input values are not sanitized after they are deserialized, making it far too easy for Leo to earn SOL (#9706) 
* sanitize gossip protocol messages
* sanitize transactions
* crds protocol sanitize
 2020-04-27 11:06:00 -07:00
carllin
bab3502260 Push down cluster_info lock (#9594) 
* Push down cluster_info lock

* Rework budget decrement

Co-authored-by: Carl <carl@solana.com>
 2020-04-21 12:54:45 -07:00
sakridge
65a9658b13 Budget for gossip traffic (#9550) 2020-04-18 22:11:17 -07:00
Michael Vines
7b5cdf6adf Reduce ReceiveUpdates log spam 2020-04-17 13:32:19 -07:00
Michael Vines
83a96c557d Ensure --dynamic-port-range is wide enough 2020-04-14 12:21:05 -07:00
Michael Vines
8cda974552 Fix flaky new_archiver_external_ip_test (#9457) 
automerge
 2020-04-12 11:49:34 -07:00
sakridge
ae5a6419d4 ReceiveUpdates spams the log, adjust the threshold higher (#9429) 2020-04-10 10:21:46 -07:00
Michael Vines
85feca305b Avoid port conflict in new_with_external_ip_test_gossip 2020-04-09 20:27:54 -07:00