Commit Graph

1980 Commits

Author SHA1 Message Date
mergify[bot]
bb4943d8a7 samples repair peers using WeightedIndex (#13919) (#13938)
To output one random sample, weighted_best generates n random numbers:
https://github.com/solana-labs/solana/blob/f751a5d4e/core/src/weighted_shuffle.rs#L38-L63
WeightedIndex does so with only one random number:
https://github.com/rust-random/rand/blob/eb02f0e46/src/distributions/weighted_index.rs#L223-L240
Additionally, if the index is already constructed, it only does a total
of O(log(n)) amount of work; which can be achieved if RepairCache,
caches the weighted index:
https://github.com/solana-labs/solana/blob/f751a5d4e/core/src/serve_repair.rs#L83

Also, the repair-peers code can be reorganized to have fewer redundant
unlock-then-lock code.

(cherry picked from commit c3048b451d)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-12-03 19:48:09 +00:00
Michael Vines
38d7d7df50 Bump version to v1.4.15 2020-12-03 01:55:39 +00:00
mergify[bot]
da1796f97a Nonce cleanup followup (bp #13868) (#13922)
* runtime: Factor out `DurableNoncePartial` finishing logic and add tests

(cherry picked from commit 8df2a4bac0)

* accounts: Don't assume fee-payer is the first account

(cherry picked from commit 47af5933ca)

* accounts: Replace nonce_rollback unreachable block with descriptive panic

(cherry picked from commit be7760caa1)

* sdk: Check owner when verifying nonce accounts

(cherry picked from commit 274312ebb5)

* runtime: Replace `HashAgeKind` with `NonceRollbackInfo`

(cherry picked from commit 404fc1570d)

* Make `Accounts::is_non_loader_key()` a method on `Message`

(cherry picked from commit 17defbff13)

Co-authored-by: Trent Nelson <trent@solana.com>
2020-12-02 21:34:21 +00:00
mergify[bot]
72aa24db2c Remove rpc_banks from validator (bp #13882) (#13905)
* Remove rpc_banks from validator (#13882)

* Remove rpc_banks from validator

* Bump abi-digest

(cherry picked from commit 10c81a2448)

# Conflicts:
#	core/src/validator.rs

* Fix conflicts

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-12-02 04:47:42 +00:00
Michael Vines
a1828a0e45 Reduce the number of snapshots
(cherry picked from commit 73111b005f)
2020-12-01 13:05:24 -08:00
mergify[bot]
446e02bad1 Add logging in check_blockstore_max_root (#13887) (#13888)
(cherry picked from commit 8fd1e55805)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-12-01 09:01:09 +00:00
mergify[bot]
535c7a44db Strengthen EpochSlots sanitization (#13873)
(cherry picked from commit 90d557d916)

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-12-01 00:04:01 +00:00
mergify[bot]
3559062c00 caches vote-state de-serialized from vote accounts (#13795) (#13862)
Gossip and other places repeatedly de-serialize vote-state stored in
vote accounts. Ideally the first de-serialization should cache the
result.

This commit adds new VoteAccount type which lazily de-serializes
VoteState from Account data and caches the result internally.

Serialize and Deserialize traits are manually implemented to match
existing code. So, despite changes to frozen_abi, this commit should be
backward compatible.

(cherry picked from commit e1793e5a13)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-11-30 18:36:23 +00:00
sakridge
cea722ac09 Bump version to v1.4.14 (#13852) 2020-11-29 18:20:19 +00:00
Michael Vines
f5eb5a3ba6 Bump version to v1.4.13 2020-11-26 06:59:28 +00:00
mergify[bot]
a09ee672a6 Separate blockstore checks for not (yet) rooted and cleaned up (#13814) (#13817)
(cherry picked from commit 4ff0f0949a)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-11-26 00:17:06 +00:00
mergify[bot]
fe4c59e38d Add solana logs command (#13812)
(cherry picked from commit 4ef2da0ff0)

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-11-25 22:22:23 +00:00
mergify[bot]
305d5d97d8 Nonce updates (#13799) (#13810)
* runtime: Add `FeeCalculator` resolution method to `HashAgeKind`

* runtime: Plumb fee-collected accounts for failed nonce tx rollback

* runtime: Use fee-collected nonce/fee account for nonced TX error rollback

* runtime: Add test for failed nonced TX accounts rollback

* Fee payer test

* fixup: replace nonce account when it pays the fee

* fixup: nonce fee-payer collect test

* fixup: fixup: clippy/fmt for replace...

* runtime: Test for `HashAgeKind::fee_calculator()`

* Clippy

Co-authored-by: Trent Nelson <trent@solana.com>
(cherry picked from commit b70abdc645)

Co-authored-by: sakridge <sakridge@gmail.com>
2020-11-25 18:18:04 +00:00
mergify[bot]
96d8ee9e07 Add base64+zstd encoding for RPC account data (#13796)
(cherry picked from commit 215ddecaa5)

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-11-25 03:20:20 +00:00
Tyera Eulberg
fd9e003ae1 Bump version to 1.4.12 (#13792) 2020-11-24 22:53:36 +00:00
mergify[bot]
c736c4633e Clean up default commitment handling for subscriptions (#13781)
(cherry picked from commit 61ab2072bd)

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-11-24 08:10:11 +00:00
Tyera Eulberg
7099d07fd3 Bump version to 1.4.11 (#13780) 2020-11-24 06:30:35 +00:00
Tyera Eulberg
e3b47d22d7 Check SlotNotRooted if confirmed block not found in blockstore or bigtable (#13778) 2020-11-24 04:23:05 +00:00
mergify[bot]
225d15bde8 processes pull-request callers only once per unique caller (#13750) (#13753)
process_pull_requests acquires a write lock on crds table to update
records timestamp for each of the pull-request callers:
https://github.com/solana-labs/solana/blob/3087c9049/core/src/crds_gossip_pull.rs#L287-L300
However, pull-requests overlap a lot in callers and this function ends
up doing a lot of redundant duplicate work.

This commit obtains unique callers before acquiring an exclusive lock on
crds table.

(cherry picked from commit 26bf2b7e45)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-11-22 19:08:33 +00:00
mergify[bot]
6dc62bfb77 Fix fragile tests in prep of stake rewrite pr (#13654) (#13721)
* Fix fragile tests in prep of stake rewrite pr

* Restore BOOTSTRAP_VALIDATOR_LAMPORTS where appropriate

* Further clean up

* Further clean up

* Aligh with other call site change

* Remove false warn!

* fix ci!

(cherry picked from commit b74d7b5758)

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-11-20 09:34:46 +00:00
mergify[bot]
b5baa966ac Fix avx check with newest nightly compiler (#13465) (#13719)
(cherry picked from commit c644b05c54)

Co-authored-by: sakridge <sakridge@gmail.com>
2020-11-20 01:23:16 +00:00
mergify[bot]
ff38a46af6 sanitizes bloom filters to avoid division by zero (#13714) (#13718)
Pull requests received over the wire can cause a validator to panic
because of division by zero in bloom filters:
https://github.com/solana-labs/solana/blob/af08ba93e/runtime/src/bloom.rs#L86-L88

(cherry picked from commit a8c29505f0)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-11-20 01:07:30 +00:00
mergify[bot]
e085b580b5 makes crds fields private (#13703) (#13708)
Crds fields should maintain several invariants between themselves, so
exposing them as public fields can be bug prone. In addition these
invariants are asserted on every write:
https://github.com/solana-labs/solana/blob/9668dd85d/core/src/crds.rs#L138-L154
https://github.com/solana-labs/solana/blob/9668dd85d/core/src/crds.rs#L239-L262
which adds extra instructions and is not optimal. Should these fields be
private the asserts will be redundant.

(cherry picked from commit b58f69297f)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-11-19 22:26:28 +00:00
mergify[bot]
fbe4e95e6a breaks prunes data into chunks to fit into packets (#13613) (#13698)
Validator logs show that prune messages are dropped because they exceed
packet data size:
https://github.com/solana-labs/solana/blob/f25c969ad/perf/src/packet.rs#L90-L92
This can exacerbate gossip traffic by redundantly increasing push
messages across network. The workaround is to break prunes into smaller
chunks and send over in multiple messages.

(cherry picked from commit 1ffab5de77)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-11-19 17:53:03 +00:00
mergify[bot]
15f6b6ccd6 RPC: Demote missing block error to warning (#13685)
It frightens the tourists

(cherry picked from commit f2a1a0ac5c)

Co-authored-by: Trent Nelson <trent@solana.com>
2020-11-19 07:05:32 +00:00
Tyera Eulberg
855cf9a362 Bump version to 1.4.10 (#13657) 2020-11-18 01:29:58 +00:00
mergify[bot]
d031bbcf2e Quiet notification logs when no subscriptions (#13629) (#13637)
(cherry picked from commit 3e4acba72f)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-11-17 08:26:15 +00:00
mergify[bot]
c183c3a5ec Improve TestValidator instantiation (bp #13627) (#13635)
* Improve TestValidator instantiation (#13627)

* Add TestValidator::new_with_fees constructor, and warning for low bootstrap_validator_lamports

* Add logging to solana-tokens integration test to help catch low bootstrap_validator_lamports in the future

* Reasonable TestValidator mint_lamports

(cherry picked from commit ef99689592)

# Conflicts:
#	tokens/Cargo.toml

* Fix conflict

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-11-17 01:08:58 -07:00
mergify[bot]
b42cda32ff packs more crds-values in a single gossip packet (#13500) (#13604)
split_gossip_messages:
https://github.com/solana-labs/solana/blob/a97c04b40/core/src/cluster_info.rs#L1536-L1574
splits crds-values into chunks to fit into a gossip packet. However it is
using a global upper-bound for the header-size across all protocols:
https://github.com/solana-labs/solana/blob/a97c04b40/core/src/cluster_info.rs#L90-L93
This can be wasteful as the specific gossip protocol can have smaller
header than this upper-bound (e.g. Protocol::PushMessage is 170 bytes
smaller). Adding more crds-values in one gossip packet can avoid the
overheads of separate packets and reduce total number of bytes sent over
the wire.

This commit updates the splitting function to take a max-chunk-size
argument. At call-site, this value is set to the size of the protocol
which the values are sent over.

(cherry picked from commit 5e8490ab9d)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-11-15 19:49:18 +00:00
mergify[bot]
2344391c48 indexes nodes' contact infos in crds table (#13553) (#13601)
In several places in gossip code, the entire crds table is scanned only
to filter out nodes' contact infos. Currently on mainnet, crds table is
of size ~70k, while there are only ~470 nodes. So the full table scan is
inefficient. Instead we may maintain an index of only nodes' contact
infos.

(cherry picked from commit cbea9ebc34)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-11-15 17:58:39 +00:00
mergify[bot]
207d13e429 Disable the PubSub vote subscription by default (#13600)
The --rpc-pubsub-enable-vote-subscription flag may be used to enable it.
The current vote subscription is problematic because it emits a
notification for *every* vote, so hundreds a second in a real cluster.
Critically it's also missing information about *who* is voting,
rendering all those notifications practically useless.

Until these two issues can be resolved, the vote subscription is not
much more than a potential DoS vector.

(cherry picked from commit 5d72e52ad0)

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-11-14 22:06:38 +00:00
mergify[bot]
360f166f5a Add counter metrics to rpc-subscriptions (#13596) (#13598)
(cherry picked from commit 88ae321d3f)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-11-14 21:09:43 +00:00
Michael Vines
b4deeb8e36 Add stable program logging for BPF and native programs 2020-11-14 08:26:54 -08:00
Trent Nelson
b838aba840 Bump version to 1.4.9 2020-11-13 17:50:10 -07:00
mergify[bot]
63cf168fef Send pubsub metrics to metrics server (#13584) (#13586)
(cherry picked from commit 34bf80ba9c)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-11-13 20:56:04 +00:00
mergify[bot]
71ea198a07 Stop searching for incorrect shred version after a minute (#13512) (#13582)
(cherry picked from commit c1f3f9d27b)

Co-authored-by: sakridge <sakridge@gmail.com>
2020-11-13 18:01:16 +00:00
mergify[bot]
44fffcbb1c filters out offline nodes from pull options (#13533) (#13551)
Inactive nodes are still observing incoming gossip traffic:
https://discord.com/channels/428295358100013066/670512312339398668/776140351291260968
likely because of pull-requests.

Previous related issues and commits:
https://github.com/solana-labs/solana/issues/12409
https://github.com/solana-labs/solana/pull/12620
https://github.com/solana-labs/solana/pull/12674

This commit implements same logic as
https://github.com/solana-labs/solana/pull/12674
to exclude inactive nodes from pull options, with the same periodic
retry logic for offline staked nodes in order to mitigate eclipse
attack.

(cherry picked from commit 4e4e12b384)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-11-12 18:43:04 +00:00
Ryo Onodera
e14c2f94f4 Bump version to v1.4.8 (#13552) 2020-11-12 17:43:31 +00:00
mergify[bot]
437c356626 Discard pre hard fork persisted tower if hard-forking (#13536) (#13550)
* Discard pre hard fork persisted tower if hard-forking

* Relax config.require_tower

* Add cluster test

* nits

* Remove unnecessary check

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
Co-authored-by: Carl Lin <carl@solana.com>
(cherry picked from commit 9821a7754c)

Co-authored-by: carllin <wumu727@gmail.com>
2020-11-12 15:43:43 +00:00
mergify[bot]
2cb9ca5966 Fix slow/stuck unstaking due to toggling in epoch (#13501) (#13535)
* Fix slow/stuck unstaking due to toggling in epoch

* nits

* nits

* Add stake_program_v2 feature status check to cli

Co-authored-by: Tyera Eulberg <tyera@solana.com>
(cherry picked from commit 89b474e192)

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-11-11 22:27:10 +00:00
mergify[bot]
4f247a232f Validator: Periodically log what we're waiting for during --wait-for-supermajority (#13531)
(cherry picked from commit 38f15e41b5)

Co-authored-by: Trent Nelson <trent@solana.com>
2020-11-11 21:29:50 +00:00
mergify[bot]
e9eba97299 Fix signature access (#13491) (#13503)
(cherry picked from commit 70c4626efe)

Co-authored-by: sakridge <sakridge@gmail.com>
2020-11-10 17:53:43 +00:00
mergify[bot]
8104895a07 Send RPC notification when account is deleted (#13440) (#13496)
* Send RPC notification when account is deleted

* Remove unwrap

(cherry picked from commit a97c04b400)

Co-authored-by: Justin Starry <justin@solana.com>
2020-11-10 13:07:51 +00:00
Tyera Eulberg
c9e646b86b Bump version to v1.4.7 (#13488) 2020-11-10 05:55:26 +00:00
mergify[bot]
9627bfced3 propagates errors out of Packet::from_data (#13445) (#13470)
Packet::from_data is ignoring serialization errors:
https://github.com/solana-labs/solana/blob/d08c3232e/sdk/src/packet.rs#L42-L48
This is likely never useful as the packet will be sent over the wire
taking bandwidth but at the receiving end will either fail to
deserialize or it will be invalid.
This commit will propagate the errors out of the function to the
call-site, allowing the call-site to handle the error.

(cherry picked from commit 73ac104df2)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-11-08 16:25:36 +00:00
Michael Vines
e738bf1c9a Bump version to v1.4.6 2020-11-07 02:49:14 +00:00
mergify[bot]
98095b6f8d drops older gossip packets when load shedding (#13364) (#13423)
Gossip drops incoming packets when overloaded:
https://github.com/solana-labs/solana/blob/f6a73098a/core/src/cluster_info.rs#L2462-L2475
However newer packets are dropped in favor of the older ones.
This is probably not ideal as newer packets are more likely to contain
more recent data, so dropping them will keep the validator state
lagging.

(cherry picked from commit 7f4debdad5)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-11-05 18:30:00 +00:00
mergify[bot]
a2c32d7d0e shares the lock on gossip when processing prune messages (#13339) (#13422)
Processing prune messages acquires an exclusive lock on gossip:
https://github.com/solana-labs/solana/blob/55b0428ff/core/src/cluster_info.rs#L1824-L1825
This can be reduced to a shared lock if active-sets are changed to use
atomic bloom filters:
https://github.com/solana-labs/solana/blob/55b0428ff/core/src/crds_gossip_push.rs#L50

(cherry picked from commit 8f0796436a)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-11-05 17:18:22 +00:00
mergify[bot]
ed97a2578d measures processing time of each kind of gossip packets (#13366) (#13418)
(cherry picked from commit 118ce47b97)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-11-05 16:46:25 +00:00
mergify[bot]
60d316c9fd Bump spl-token and spl-memo crate versions (#13400)
Co-authored-by: Michael Vines <mvines@gmail.com>
2020-11-04 23:16:41 +00:00