Commit Graph

2737 Commits

Author SHA1 Message Date
carllin
a4cf7d72be Filter out outdated slots (#22450)
* Filter out outdated slots

* Fixup error

(cherry picked from commit 4ab7d6c23e)

# Conflicts:
#	core/src/consensus.rs
#	local-cluster/tests/local_cluster.rs
#	programs/vote/src/vote_state/mod.rs
#	sdk/src/feature_set.rs
2022-02-07 23:47:06 +00:00
mergify[bot]
dd6c365bd9 Resolve conflicts (#22905)
Co-authored-by: carllin <carl@solana.com>
2022-02-05 06:47:18 +00:00
mergify[bot]
9ea025315e removes VoteTracker::new in favor of VoteTracker::default (#22941) (#22946)
VoteTracker::new does not need a bank and is so redundant:
https://github.com/solana-labs/solana/blob/5a230f418/core/src/cluster_info_vote_listener.rs#L103-L107
(cherry picked from commit 27aaf9df85)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-04 20:59:01 +00:00
mergify[bot]
c43cef79b5 Add quic port for accepting transactions (#22753) (#22937)
using quinn library

streamer: Sign TLS cert with validator identity key

Handle multiple incoming chunks

(cherry picked from commit 5a230f418d)

Co-authored-by: sakridge <sakridge@gmail.com>
2022-02-04 20:53:27 +00:00
Tao Zhu
f7091811d4 Allow buffered packets be consumed if bank is active, regardless leader schedule 2022-02-03 16:56:27 -06:00
Tao Zhu
15ef1827bf push live packets straight to buffer, leader only process packets from buffer 2022-02-03 16:56:27 -06:00
mergify[bot]
69e207ca58 rpc: use minimal mode by default (backport #22734) (#22879)
* rpc: use minimal mode by default

(cherry picked from commit eac4a6df68)

# Conflicts:
#	local-cluster/tests/local_cluster.rs

* test-validator-bin: reinstate full rpc method set

Co-authored-by: Trent Nelson <trent@solana.com>
2022-02-03 08:25:49 +00:00
mergify[bot]
fb8db79e63 adds reverse lookup index to cluster-nodes (#22892) (#22894)
retransmit has to exclude slot leader from set of nodes for each shred;
which currently requires a linear scan:
https://github.com/solana-labs/solana/blob/e3b137066/core/src/cluster_nodes.rs#L238-L242

This commit adds a reverse lookup index to avoid linear scan.

(cherry picked from commit dccbddad80)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-02 21:15:51 +00:00
mergify[bot]
237347847b caches WeightedShuffle struct in ClusterNodes (#22877) (#22889)
Instead of reconstructing WeightedShuffle struct for each shred
broadcast or retransmit, we can use the same struct with minimal
mutations.

(cherry picked from commit e3b137066d)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-02 16:54:57 +00:00
mergify[bot]
c715bc93cf removes Rng field from WeightedShuffle struct (#22850) (#22868)
(cherry picked from commit 45e09664b8)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-01 17:33:52 +00:00
mergify[bot]
4dfbb4347c includes zero weighted entries in WeightedShuffle (#22829) (#22847)
Current WeightedShuffle implementation excludes zero weighted entries
from the shuffle:
https://github.com/solana-labs/solana/blob/13e631dcf/gossip/src/weighted_shuffle.rs#L29-L30

Though mathematically this might make more sense, for our use-cases
(turbine specifically), this results in less efficient code:
https://github.com/solana-labs/solana/blob/13e631dcf/core/src/cluster_nodes.rs#L409-L430

This commit changes the implementation so that zero weighted indices are
also included in the shuffle but appear only at the end after non-zero
weighted indices.

(cherry picked from commit 604ca9316c)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-31 18:16:40 +00:00
Michael Vines
2368e09d89 Add vote account address to vote subscription
(cherry picked from commit 331b953551)

# Conflicts:
#	core/src/cluster_info_vote_listener.rs
#	rpc/src/rpc_pubsub.rs
#	rpc/src/rpc_subscriptions.rs
2022-01-27 11:19:26 -08:00
mergify[bot]
b0e0410003 Set the correct root in block commitment cache initialization (#22750) (#22757)
* Set the correct root in block commitment cache initialization

* clean up test

* bump

(cherry picked from commit d9c259a231)

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-27 03:44:59 +00:00
mergify[bot]
d9a5f714e1 Refactor: Rename variables and helper method to PohRecorder (#22676) (#22688)
* Refactor: Rename leader_first_tick_height field

* Refactor: add `PohRecorder::slot_for_tick_height` helper

* Refactor: Add type for poh leader status

(cherry picked from commit 1240217a73)

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-23 15:34:44 +08:00
Anatoly Yakovenko
620a80b581 sigverify -- dedupe bloom filter too slow followups 2022-01-21 23:59:41 -07:00
Trent Nelson
8bc4cc90d2 Bump version to 1.9.6 2022-01-21 20:15:43 -07:00
mergify[bot]
91bc44931f Add hidden cli option to allow validator reports replayed transaction cost metrics (backport #22369) (#22519)
* Add hidden cli option to allow validator reports replayed transaction cost metrics (#22369)

* add hidden cli option to allow validator reports replayed transaction cost detail metrics

* Update validator/src/main.rs

Co-authored-by: Michael Vines <mvines@gmail.com>

* - rebase master, using unbounded instead of channel; dowgrade to datapoint_trace

* removed cli arg, prefer log at trace

Co-authored-by: Michael Vines <mvines@gmail.com>
(cherry picked from commit a724fa2347)

# Conflicts:
#	core/src/tvu.rs

* fix conflict

Co-authored-by: Tao Zhu <82401714+taozhu-chicago@users.noreply.github.com>
Co-authored-by: Tao Zhu <tao@solana.com>
2022-01-21 15:05:41 -07:00
mergify[bot]
35ca3182ba Add estimated and actual block cost units metrics (backport #22326) (#22517)
* Add estimated and actual block cost units metrics (#22326)

* - report cost details for transactions selected to be packed into block;
- report estimated execution units packed into block, and actual units and time after execution

* revert reporting per-transaction details

* rollup transaction cost details (eg signature cost, wirte lock, data cost and execution costs) into block stats

* change naming from units to cu, use struct to replace tuple

(cherry picked from commit 1309a9cea0)

# Conflicts:
#	core/src/banking_stage.rs
#	core/src/qos_service.rs

* fix conflicts

Co-authored-by: Tao Zhu <82401714+taozhu-chicago@users.noreply.github.com>
Co-authored-by: Tao Zhu <tao@solana.com>
2022-01-21 15:05:19 -07:00
anatoly yakovenko
bf45f5b88e Faster dedup v1.9 (#22638)
Faster dedup port of #22607
2022-01-21 11:21:28 -08:00
mergify[bot]
e9e35fd7bd system-monitor-service: support percentages from bigger numbers (#22598)
(cherry picked from commit cca3dbc76d)

Co-authored-by: Trent Nelson <trent@solana.com>
2022-01-20 11:37:52 +00:00
Trent Nelson
66b94b86a9 banking-stage: remove unused stats fields 2022-01-20 06:09:10 +00:00
mergify[bot]
59f406d78a Refactor: move simple vote parsing to runtime (backport #22537) (#22587)
* Refactor: move simple vote parsing to runtime (#22537)

(cherry picked from commit 7f20c6149e)

# Conflicts:
#	core/src/cluster_info_vote_listener.rs
#	core/src/verified_vote_packets.rs
#	programs/vote/src/vote_transaction.rs
#	rpc/src/rpc_subscriptions.rs
#	runtime/src/bank.rs
#	runtime/src/bank_utils.rs
#	runtime/src/vote_sender_types.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-20 04:51:50 +00:00
mergify[bot]
dbf9a32883 Optimize packet dedup (#22571) (#22585)
* Use bloom filter to dedup packets

* dedup first

* Update bloom/src/bloom.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* fixup

* fixup

* fixup

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
(cherry picked from commit d343713f61)

# Conflicts:
#	Cargo.lock
#	core/Cargo.toml
#	core/src/banking_stage.rs
#	core/src/sigverify_stage.rs
#	gossip/Cargo.toml
#	perf/Cargo.toml
#	programs/bpf/Cargo.lock
#	runtime/Cargo.toml

Co-authored-by: anatoly yakovenko <anatoly@solana.com>
2022-01-20 02:51:49 +00:00
mergify[bot]
37e9076db0 Add PacketBatch packet_indexes stat (#22564) (#22575)
* collect stats on packet batch indicies

* cleanup

* cleanup

* cleanup

* change name

(cherry picked from commit 650882217c)

# Conflicts:
#	core/src/banking_stage.rs

Co-authored-by: buffalu <85544055+buffalu@users.noreply.github.com>
2022-01-20 00:05:05 +00:00
mergify[bot]
f77ea5f324 improves sigverify discard_excess_packets performance (backport #22577) (#22580)
* improves sigverify discard_excess_packets performance (#22577)

As shown by the added benchmark, current code does worse if there is a
spam address plus a lot of unique addresses.

on current master:
test bench_packet_discard_many_senders  ... bench:   1,997,960 ns/iter (+/- 103,715)
test bench_packet_discard_mixed_senders ... bench:  14,256,116 ns/iter (+/- 534,865)
test bench_packet_discard_single_sender ... bench:   1,306,809 ns/iter (+/- 61,992)

with this commit:
test bench_packet_discard_many_senders  ... bench:   1,644,025 ns/iter (+/- 83,715)
test bench_packet_discard_mixed_senders ... bench:   1,089,789 ns/iter (+/- 86,324)
test bench_packet_discard_single_sender ... bench:     955,234 ns/iter (+/- 55,953)

(cherry picked from commit dcf44d2523)

# Conflicts:
#	core/src/sigverify_stage.rs

* removes mergify merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-19 20:14:10 +00:00
mergify[bot]
c9df037dae Track discard time of excess packets in sigverify (#22554) (#22570)
* discard time histogram

* closer to the if

* update

(cherry picked from commit e616a7ebfc)

Co-authored-by: anatoly yakovenko <anatoly@solana.com>
2022-01-19 01:52:30 +00:00
mergify[bot]
2b87d99479 Use VecDeque instead of Vec in sigverify stage (#22538) (#22550)
avoid bad performance of remove(0) for a single sender

(cherry picked from commit 49443406fd)

# Conflicts:
#	core/src/sigverify_stage.rs

Co-authored-by: sakridge <sakridge@gmail.com>
2022-01-19 01:46:34 +00:00
mergify[bot]
2546ef4ad6 metrics for generate new bank forks (#22492) (#22548)
* metrics for generate new bank forks

* fixed

* Apply suggestions from code review

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* --fixup

* fixup!

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
(cherry picked from commit 2d94e6e5d3)

# Conflicts:
#	core/src/replay_stage.rs

Co-authored-by: anatoly yakovenko <anatoly@solana.com>
2022-01-19 01:45:25 +00:00
Tao Zhu
7a421fe602 Port counting vote CUs to block cost to v1.9 2022-01-14 10:49:43 -06:00
mergify[bot]
4ee6bc9a93 downgrade individual per-program-timing to trace to reduce writes to influx (#22471)
(cherry picked from commit 6614727be8)

Co-authored-by: Tao Zhu <tao@solana.com>
2022-01-13 02:37:47 +00:00
mergify[bot]
58dcc451a9 Prevent rent-paying account creation (backport #22292) (#22428)
* Prevent rent-paying account creation (#22292)

* Fixup typo

* Add new feature

* Add new TransactionError

* Add framework for checking account state before and after transaction processing

* Fail transactions that leave new rent-paying accounts

* Only check rent-state of writable tx accounts

* Review comments: combine process_result success behavior; log and metrics before feature activation

* Fix tests that assume rent-exempt accounts are okay

* Remove test no longer relevant

* Remove native/sysvar special case

* Move metrics submission to report legacy->legacy rent paying transitions as well

(cherry picked from commit 637e366b18)

# Conflicts:
#	runtime/src/bank.rs
#	runtime/src/lib.rs

* Fix conflicts and rework for TransactionRefCells

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2022-01-11 23:17:03 +00:00
Trent Nelson
cb49ae21b4 Bump version to v1.9.5 2022-01-08 21:17:51 +00:00
Justin Starry
1f00926874 Add runtime support for address table lookups (backport #22223) (#22354) 2022-01-08 07:57:04 +08:00
mergify[bot]
7b1da62763 Add execute metrics (backport #22296) (#22335)
* move `ExecuteTimings` from `runtime::bank` to `program_runtime::timings`

(cherry picked from commit 7d32909e17)

# Conflicts:
#	core/Cargo.toml
#	ledger/Cargo.toml
#	programs/bpf/Cargo.lock

* Add execute metrics

(cherry picked from commit b25e4a200b)

* Add metrics for executor creation

(cherry picked from commit 848b6dfbdd)

* Add helper macro for `AddAssign`ing with saturating arithmetic

(cherry picked from commit deb9344e49)

* Use saturating_add_assign macro

(cherry picked from commit 72fc6096a0)

* Consolidate process instruction execution timings to own struct

(cherry picked from commit 390ef0fbcd)

Co-authored-by: Trent Nelson <trent@solana.com>
Co-authored-by: Carl Lin <carl@solana.com>
2022-01-07 09:11:18 +00:00
Will Hickey
ba0188a36d Bump version to 1.9.4 (#22304) 2022-01-05 12:02:36 -06:00
mergify[bot]
8578429c4d Refactor: Improve type safety and readability of transaction execution (backport #22215) (#22289)
* Refactor: Improve type safety and readability of transaction execution (#22215)

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-05 23:01:15 +08:00
mergify[bot]
fb0e5adc7e patches bug in recv_mmsg when npkts != nrecv (backport #22276) (#22280)
* removes total-size from return value of recv_mmsg

(cherry picked from commit 4b24499916)

* patches bug in recv_mmsg when npkts != nrecv

If recv_mmsg receives 2 packets where the first one is filtered out,
then it returns npkts == 1:
https://github.com/solana-labs/solana/blob/01a096adc/streamer/src/recvmmsg.rs#L104-L115

But then streamer::packet::recv_from will erroneously keep the 1st
packet and drop the 2nd one:
https://github.com/solana-labs/solana/blob/01a096adc/streamer/src/packet.rs#L34-L49

To avoid this bug, this commit updates recv_mmsg to always return total
number of received packets. If socket address cannot be correctly
obtained, it is left as the default value which is UNSPECIFIED:
https://github.com/solana-labs/solana/blob/01a096adc/sdk/src/packet.rs#L145

(cherry picked from commit 379feecae5)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-04 23:42:52 +00:00
mergify[bot]
d84b994451 shrinks size of Packet.Meta (backport #22224) (#22273)
* removes seed and slot fields from Packet.Meta

507367e6ac
updated window-service to send shreds (as opposed to packets) to
retransmit-stage and so seed and slot fields in Packet.Meta are unused:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L27-L28

(cherry picked from commit aa9f7ed7e8)

* uses std::net::IpAddr type for Packet.Meta.addr

(cherry picked from commit 73a7741c49)

# Conflicts:
#	streamer/src/streamer.rs

* adds bitflags to Packet.Meta

Instead of a separate bool type for each flag, all the flags can be
encoded in a type-safe bitflags encoded in a single u8:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L19-L31

(cherry picked from commit 01a096adc8)

# Conflicts:
#	sdk/Cargo.toml

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-04 17:38:29 +00:00
mergify[bot]
185f52b712 Lower vote-only-mode to 400 (#22210) (#22272)
(cherry picked from commit 2486e21ffe)

Co-authored-by: sakridge <sakridge@gmail.com>
2022-01-04 15:15:16 +00:00
mergify[bot]
f8bf478fde Fix bug, add error specific timings (#22225) (#22252)
(cherry picked from commit 005592998d)

Co-authored-by: carllin <carl@solana.com>
2022-01-04 02:53:59 +00:00
mergify[bot]
35fb47d1ce removes epoch_authorized_voters from VoteTracker (backport #22192) (#22248)
* removes epoch_authorized_voters from VoteTracker (#22192)

https://github.com/solana-labs/solana/pull/22169
verifies authorized-voter early on in vote-listener pipeline; and so
VoteTracker no longer needs to maintain and check for epoch authorized
voters.

(cherry picked from commit 69d71f8f86)

# Conflicts:
#	core/src/cluster_info_vote_listener.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-04 01:51:24 +00:00
mergify[bot]
ca451ea23e checks for authorized voter early on in the vote-listener pipeline (backport #22169) (#22206)
* checks for authorized voter early on in the vote-listener pipeline (#22169)

Before votes are verified that they are signed by the authorized voter,
they might be dropped in verified-vote-packets code. If there are
enough many spam votes from unauthorized voters, this may potentially
drop valid votes but keep the false ones.
https://github.com/solana-labs/solana/blob/57986f982/core/src/verified_vote_packets.rs#L165-L168

(cherry picked from commit c0c6038654)

# Conflicts:
#	core/src/cluster_info_vote_listener.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-31 21:46:12 +00:00
mergify[bot]
113d261a2c Count compute units even when transaction errors (backport #22182) (#22199)
* Count compute units even when transaction errors (#22182)

(cherry picked from commit d06e6c7425)

# Conflicts:
#	program-runtime/src/invoke_context.rs
#	runtime/src/cost_model.rs
#	runtime/src/message_processor.rs

* Resolve conflicts

Co-authored-by: carllin <carl@solana.com>
2021-12-31 21:14:00 +00:00
carllin
7efd0391e9 Revert "Count compute units even when transaction errors (backport #22059) (#22154)" (#22175)
This reverts commit 401c542d2a.
2021-12-30 02:39:25 -05:00
mergify[bot]
6a556c5adb Stream additional block metadata via plugin (#22023) (#22179)
* Stream additional block metadata through plugin
blockhash, block_height, block_time, rewards are streamed

(cherry picked from commit f14928a970)

Co-authored-by: Lijun Wang <83639177+lijunwangs@users.noreply.github.com>
2021-12-30 05:44:12 +00:00
mergify[bot]
531f36c571 Don't forward packets received from TPU forwards port (#22078) (#22171)
* Don't forward packets received from TPU forwards port

* Add banking stage test

(cherry picked from commit b1d9a2e60e)

Co-authored-by: Justin Starry <justin@solana.com>
2021-12-30 10:42:28 +08:00
mergify[bot]
9fb67f9b07 Prevent log spam (#22148) (#22152)
(cherry picked from commit f061059e45)

Co-authored-by: carllin <carl@solana.com>
2021-12-29 08:28:48 +00:00
mergify[bot]
401c542d2a Count compute units even when transaction errors (backport #22059) (#22154)
* Count compute units even when transaction errors (#22059)

(cherry picked from commit eaa8c67bde)

# Conflicts:
#	program-runtime/src/invoke_context.rs
#	runtime/src/bank.rs
#	runtime/src/message_processor.rs

* Fix merge conflicts

Co-authored-by: carllin <carl@solana.com>
2021-12-29 08:04:12 +00:00
mergify[bot]
810ca36eae skip reporting all-zero stats (#22054)
(cherry picked from commit 9c5d82557a)

Co-authored-by: Tao Zhu <tao@solana.com>
2021-12-28 07:03:13 +00:00
mergify[bot]
16f821ea8c Ensure AncestorHashesSerice selects an open port (#21919) (#21997)
(cherry picked from commit 7f6fb6937a)

Co-authored-by: carllin <wumu727@gmail.com>
2021-12-28 06:46:59 +00:00