2355 Commits

Author SHA1 Message Date
bc31378797 Trim extra shred bytes in blockstore (#16602)
Strip the zero-padding off of data shreds before insertion into blockstore

Co-authored-by: Stephen Akridge <sakridge@gmail.com>
Co-authored-by: Nathan Hawkins <utsl@utsl.org>
2021-04-27 17:40:41 -05:00
3b8d6b59fb records hash of values purged by expired pull-responses (#16800)
process_pull_responses should record hash of values purged by expired
responses (as well as unexpired ones):
https://github.com/solana-labs/solana/blob/c1829dd00/core/src/crds_gossip_pull.rs#L385-L387

otherwise, these values are not excluded from following pull-requests
(from likely different nodes):
https://github.com/solana-labs/solana/blob/c1829dd00/core/src/crds_gossip_pull.rs#L447-L452

and would waste bandwidth should they be included in subsequent
pull-responses.
2021-04-27 12:06:49 +00:00
0f3ac51cf1 limits to data_header.size when combining shreds' payloads (#16708)
Shredder::deshred is ignoring data_header.size when combining shreds' payloads:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/shred.rs#L940-L961

Also adding more sanity checks on the alignment of data shreds indices.
2021-04-27 12:04:44 +00:00
59fc33635a Add getVoteAccounts RPC method parameter to restrict results to a single vote account 2021-04-27 04:27:15 +00:00
9706512115 removes old runtime feature gates in gossip and turbine (#16633) 2021-04-26 17:12:02 +00:00
ca14c18998 owner -> owner() (#16782) 2021-04-23 22:49:47 +00:00
63436cc2bf Disable flaky test_poh_service (#16772) 2021-04-23 12:14:11 -05:00
2c82f2154d retains crds values if the origin is still active (#16576)
Local timestamps are updated for records associated with a pubkey if the
origin is still active:
https://github.com/solana-labs/solana/blob/c8ed14c64/core/src/crds.rs#L301-L311

However this is done inconsistently on some gossip paths (pull requests
and pull responses) but not all (e.g. push messages). Additionally
update_record_timestamp is inefficient since there can be ~800 values
associated with each pubkey.

This commit updates records timestamps only on contact-infos; and,
instead utilizes origin's timestamp when purging old values.
2021-04-23 15:14:49 +00:00
03194145c0 removes first_coding_index from erasure recovery code (#16646)
first_coding_index is the same as the set_index and is so redundant:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/blockstore_meta.rs#L49-L60
2021-04-23 12:00:37 +00:00
75b8434b76 Add TPU client for sending txs to the current leader tpu port (#16736)
* Add TPU client for sending txs to the current leader tpu port

* Update tpu_client.rs
2021-04-23 09:35:12 +08:00
636b5987af Update getLeaderSchedule options (#16749) 2021-04-22 19:27:30 +00:00
6004c0abf5 getLeaderSchedule now supports filtered results based on validator identity 2021-04-21 17:59:26 -07:00
91b6888e15 verify_pubkey() now takes a ref 2021-04-21 14:43:49 -07:00
4c94f8933f Ingest votes from gossip into fork choice (#16560) 2021-04-21 14:40:35 -07:00
a1ef2bd74d Ignore flaky test_pull_request_time_pruning 2021-04-21 12:07:36 -07:00
37b8587d4e expands number of erasure coding shreds in the last batch in slots (#16484)
Number of parity coding shreds is always less than the number of data
shreds in FEC blocks:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L719

Data shreds are batched in chunks of 32 shreds each:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L714

However the very last batch of data shreds in a slot can be small, in
which case the loss rate can be exacerbated.

This commit expands the number of coding shreds in the last FEC block in
slots to: 64 - number of data shreds; so that FEC blocks are always 64
data and parity coding shreds each.

As a consequence of this, the last FEC block has more parity coding
shreds than data shreds. So for some shred indices we will have a coding
shred but no data shreds. This should not cause any kind of overlapping
FEC blocks as in:
https://github.com/solana-labs/solana/pull/10095
since this is done only for the very last batch in a slot, and the next
slot will reset the shred index.
2021-04-21 12:47:50 +00:00
0924c2d070 Add port and gossip options to solana-test-validator (#16696) 2021-04-21 02:40:52 +00:00
34addee882 getVoteAccounts: Limit the length of the epoch_credits array 2021-04-20 14:42:28 -07:00
8e69dd42c1 Add non-default repair nonce values (#16512)
* Track outstanding nonces in repair

* Rework outstanding requests to use lru cache and randomize nonces

Co-authored-by: Carl <carl@solana.com>
2021-04-20 09:37:33 -07:00
bc90e04e64 uses current local timestamp when recording purged values
CrdsGossipPull.purged_values is meant to record recently purged values
so that they are excluded from imminent pull requests, until the entire
cluster have synced to the updated value:
https://github.com/solana-labs/solana/blob/c826cddbb/core/src/crds_gossip_pull.rs#L449-L454

However, VersionedCrdsValue.local_timestamp represents the local time
when the value was last updated, and given that crds values may have
different timeouts based on stake, it does not necessarily represent how
recently the value was purged:
https://github.com/solana-labs/solana/blob/c826cddbb/core/src/crds.rs#L75-L76

As such, recording current local timestamp when purging values is more
appropriate. Additionally, purge_purged assumes that the purge_values is
sorted in timestamps when draining the old ones; which is not true if
those timestamps are VersionedCrdsValue.local_timestamp:
https://github.com/solana-labs/solana/blob/c826cddbb/core/src/crds_gossip_pull.rs#L563-L571
2021-04-20 11:21:00 +00:00
a7e65c0034 RPC: use finalized as default pubsub commitment level (#16659)
* RPC: use finalized as default pubsub commitment level

* update docs

* Fix tests
2021-04-20 08:19:54 +00:00
c8b474cd0b Send votes to next leader's TPU instead of our TPU 2021-04-20 00:38:21 -07:00
4aa753ff01 rename threads: 15 char limit (#16625) 2021-04-19 12:16:58 -05:00
b06e93fe5b Increase test timeout 2021-04-18 20:55:02 -07:00
a911ae00ba clippy 2021-04-18 20:55:02 -07:00
e405747409 Revert "Add limit and shrink policy for recycler (#15320)"
This reverts commit c2e8814dce.
2021-04-18 19:29:24 +00:00
6907a2366e Remove unnecessary clone 2021-04-17 10:23:13 -07:00
974e6dd2c1 Deprecate "confirmed" RpcClient methods (#16520)
* Remove obsolete client methods

* Deprecate GetConfirmed client methods

* Rename Confirmed config structs, with appropriate deprecation

* Fixup client apps

* Map RpcRequest to deprecated when targeting older nodes
2021-04-15 17:00:14 -06:00
7dfb51c0b4 Cli: move airdrop to rpc requests (#16557)
* Add recent_blockhash to requestAirdrop

* Move tx confirmation to separate method

* Add RpcClient airdrop methods

* Request cli airdrop via RpcClient

* Pass optional faucet_addr into TestValidator and fix tests

* Update client/src/rpc_client.rs

Co-authored-by: Michael Vines <mvines@gmail.com>

Co-authored-by: Michael Vines <mvines@gmail.com>
2021-04-15 06:25:23 +00:00
d92721aab9 uses timeouts based on stake for filtering pull responses (#16549)
filter_pull_responses is using default timeout when discarding pull
responses (except for ContactInfo):
https://github.com/solana-labs/solana/blob/f804ce63c/core/src/crds_gossip_pull.rs#L349-L350

But purging code uses timeouts based on stake:
https://github.com/solana-labs/solana/blob/f804ce63c/core/src/cluster_info.rs#L1867-L1870

So the crds value will not be purged from the sender's table and will be
sent again over the next pull request.
2021-04-14 20:18:00 +00:00
f35a6a8be0 prioritizes contact-infos in pull responses (#16541)
Expired crds values where the contact-info does not exist are wasted:
https://github.com/solana-labs/solana/blob/f804ce63c/core/src/crds_gossip_pull.rs#L353-L378
and then are sent again over the next pull-request.

Also, the stake of the first response (which can be anything) is used to
weight all pull-responses to a node, while the rest of responses can
have different stake.
https://github.com/solana-labs/solana/blob/f804ce63c/core/src/cluster_info.rs#L2231
2021-04-14 18:45:20 +00:00
f0c150cfb9 Fix channel panic in tests (#16503)
* Fix channel panic

* Add exit signal to PohRecorder because Crossbeam doesnt drop objects inside dropped channel
2021-04-14 12:07:21 -05:00
37afa00ffb Rpc: deprecate getConfirmed endpoints (#16502)
* Deprecate getConfirmed methods in rpc

* Add new methods to docs

* Move deprecated rpc methods to separate docs section

* Add note to docs about removal timing
2021-04-13 01:50:15 -06:00
85eb37fab0 Merge pull request from GHSA-8v47-8c53-wwrc
* Track transaction check time separately from account loads

* banking packet process metrics

* Remove signature clone in status cache lookup

* Reduce allocations when converting packets to transactions

* Add blake3 hash of transaction messages in status cache

* Bug fixes

* fix tests and run fmt

* Address feedback

* fix simd tx entry verification

* Fix rebase

* Feedback

* clean up

* Add tests

* Remove feature switch and fall back to signature check

* Bump programs/bpf Cargo.lock

* clippy

* nudge benches

* Bump `BankSlotDelta` frozen ABI hash`

* Add blake3 to sdk/programs/Cargo.lock

* nudge bpf tests

* short circuit status cache checks

Co-authored-by: Trent Nelson <trent@solana.com>
2021-04-13 00:28:08 -06:00
70f3f7e679 Move obsolete rpc endpoints to separate api for removal (#16500)
* Move obsolete rpc methods to separate api for removal

* Remove obsolete method from docs

* Fix test using obs method
2021-04-12 20:33:40 -06:00
2229b70c4e Add authorized-voter add/remove-all commands 2021-04-12 15:55:28 -07:00
17a173ebb5 Clean up build warning 2021-04-12 15:55:28 -07:00
dc7030ffaa Allow fork choice to support multiple versions of a slot (#16266) 2021-04-12 01:00:59 -07:00
99b3aab703 Track gossip vote updates per hash for replay stage (#16421)
* Track gossip vote updates per hash for replay stage
2021-04-10 17:34:45 -07:00
54a04bac3d Apple M1 compatibility (#16346)
Co-authored-by: Christian Drappi <christiandrappi@Christians-MacBook-Pro.local>
2021-04-09 17:21:01 -07:00
8bc0bdd40b Fill in not-yet-finalized block-time if possible (#16460) 2021-04-09 20:25:47 +00:00
22a18a68e3 stops consuming pinned vectors with a recycler (#16441)
If the vector is pinned and has a recycler, From<PinnedVec>
implementation of Vec should clone (instead of consuming) the underlying
vector so that the next allocation of a PinnedVec will recycle an
already pinned one.
2021-04-09 16:55:24 +00:00
b08cff9e77 Simplify some pattern-matches (#16402)
When those match an exact combinator on Option / Result.

Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust).
2021-04-08 12:40:37 -06:00
bb9d2fd07a Cli: use get_inflation_rewards and limit epochs queried (#16408)
* Fix block-with-limit when not finalized blocks found

* Enable confirmed commitment in getInflationReward

* Use get_inflation_rewards in cli

* Line up rewards output

* Add range validator

* Change cli epoch arg -> num epochs

* Add solana inflation rewards subcommand

* Consolidate epoch rewards meta
2021-04-08 10:57:33 -06:00
e501fa5f0b Rpc: introduce get_inflation_reward rpc call (#16278)
* feat: introduce get_inflation_reward rpc call

* fix: style suggestions

* fix: more style changes and match how other rpc functions are defined

* feat: get reward for a single epoch

* feat: default to the most recent epoch

* fix: don't factor out get_confirmed_block

* style: introduce from impl for RpcEncodingConfigWrapper

* style: bring commitment into variable

* feat: support multiple pubkeys for get_inflation_reward

* feat: add get_inflation_reward to rpc client

* feat: return rewards in order

* fix: rename pubkeys to addresses

* docs: introduce jsonrpc docs for get_inflation_reward

* style: early return in map (not sure which is more idiomatic)

* fix: call the rpc client function args addresses as well

* fix: style

* fix: filter out only addresses we care about

* style: make this more idiomatic

* fix: change rpc client epoch to optional and include some docs edits

* feat: filter out rent rewards in get_inflation_reward

* feat: add option epoch config param to get_inflation_reward

* feat: rpc client get_inflation_reward takes epoch instead of config and some filter staking and voting rewards
2021-04-06 18:10:53 -07:00
1219842a96 No wallclock throttle tests (#16396) 2021-04-05 19:40:16 -07:00
b71875df61 cluster-info: Get rid of some integer math while we're here 2021-04-06 00:09:37 +00:00
b6b08706b9 cluster-info: Don't subtract non-shred spies from node count 2021-04-06 00:09:37 +00:00
7a2a39093d validator: Use a const for wait for supermajority threshold 2021-04-05 17:29:37 -06:00
b242f82696 Reduce test-validator ledger size 2021-04-05 08:37:29 -07:00