Commit Graph

15269 Commits

Author SHA1 Message Date
13eb2e4687 Add sbf-tools version to cargo target cache name on CI agents (#23026)
(cherry picked from commit c7aa7fb66b)

Co-authored-by: Dmitri Makarov <dmakarov@alumni.stanford.edu>
2022-02-09 21:56:07 +00:00
fa264cf7ca Remove cost model from end of slot filtering (#23021) 2022-02-08 22:32:13 -05:00
c478510a7f solana-validator monitor: Remove getMaxRetransmitSlot RPC method usage (backport #23001) (#23009)
* monitor: Remove getMaxRetransmitSlot RPC method usage

(cherry picked from commit dcd4ea9111)

# Conflicts:
#	validator/src/dashboard.rs

* rebase

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-02-08 19:33:46 +00:00
4ce59bbbb5 Reject close of active vote accounts (backport #22651) (#22895)
* Reject close of active vote accounts (#22651)

* 10461 Reject close of vote accounts unless it earned no credits in the previous epoch. This is checked by comparing current epoch (from clock sysvar) with the most recent epoch with credits in vote state.

(cherry picked from commit 75563f6c7b)

# Conflicts:
#	programs/vote/src/vote_processor.rs
#	programs/vote/src/vote_state/mod.rs
#	runtime/src/bank.rs
#	sdk/program/src/instruction.rs
#	sdk/program/src/program_error.rs
#	storage-proto/proto/transaction_by_addr.proto
#	storage-proto/src/convert.rs

* Resolve merge conflicts

* lint

* Clippy cleanup

* Add import to test module

* remove vote processor

* Update test_abi_digest hash

* cleanup

Co-authored-by: Will Hickey <csu_hickey@yahoo.com>
Co-authored-by: Will Hickey <will.hickey@solana.com>
2022-02-07 16:15:32 +00:00
ad9fceec8e Update syscall base costs (#22930)
(cherry picked from commit 2aa113fd8c)

Co-authored-by: Dmitri Makarov <dmakarov@alumni.stanford.edu>
2022-02-07 05:11:14 +00:00
43f791bffb patches bug in recv_mmsg when npkts != nrecv (backport #22276) (#22970)
* removes total-size from return value of recv_mmsg

(cherry picked from commit 4b24499916)

* patches bug in recv_mmsg when npkts != nrecv

If recv_mmsg receives 2 packets where the first one is filtered out,
then it returns npkts == 1:
https://github.com/solana-labs/solana/blob/01a096adc/streamer/src/recvmmsg.rs#L104-L115

But then streamer::packet::recv_from will erroneously keep the 1st
packet and drop the 2nd one:
https://github.com/solana-labs/solana/blob/01a096adc/streamer/src/packet.rs#L34-L49

To avoid this bug, this commit updates recv_mmsg to always return total
number of received packets. If socket address cannot be correctly
obtained, it is left as the default value which is UNSPECIFIED:
https://github.com/solana-labs/solana/blob/01a096adc/sdk/src/packet.rs#L145

(cherry picked from commit 379feecae5)

# Conflicts:
#	streamer/src/recvmmsg.rs

* removes mergify merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-06 17:14:07 +00:00
1570026c2c tracks erasure coding shreds indices explicitly (backport #21822) (#22969)
* tracks erasure coding shreds' indices explicitly (#21822)

The indices for erasure coding shreds are tied to data shreds:
https://github.com/solana-labs/solana/blob/90f41fd9b/ledger/src/shred.rs#L921

However with the upcoming changes to erasure schema, there will be more
erasure coding shreds than data shreds and we can no longer infer coding
shreds indices from data shreds.

The commit adds constructs to track coding shreds indices explicitly.

(cherry picked from commit 65d59f4ef0)

# Conflicts:
#	core/benches/retransmit_stage.rs
#	core/benches/shredder.rs
#	core/src/broadcast_stage/broadcast_duplicates_run.rs
#	core/src/broadcast_stage/broadcast_fake_shreds_run.rs
#	core/src/broadcast_stage/fail_entry_verification_broadcast_run.rs
#	core/src/window_service.rs
#	ledger/src/blockstore.rs
#	ledger/src/shred.rs
#	ledger/tests/shred.rs

* removes mergify merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-06 02:42:32 +00:00
b8e813a735 Implement json output for solana ping (backport #22959) (#22967)
* Implement json output for solana ping (#22959)

(cherry picked from commit d2c89213ff)

# Conflicts:
#	cli/src/cluster_query.rs

* Fix conflicts

Co-authored-by: Tyera Eulberg <tyera@solana.com>
2022-02-05 23:34:51 +00:00
89ec164787 removes next_shred_index from return value of entries to shreds api (backport #21961) (#22965)
* removes next_shred_index from return value of entries to shreds api (#21961)

next-shred-index is already readily available from returned data shreds.
The commit simplifies the api for upcoming changes to erasure coding
schema which will require explicit tracking of indices for coding shreds
as well as data shreds.

(cherry picked from commit 89d66c3210)

# Conflicts:
#	core/benches/shredder.rs
#	core/src/broadcast_stage/broadcast_duplicates_run.rs
#	core/src/broadcast_stage/broadcast_fake_shreds_run.rs
#	core/src/broadcast_stage/fail_entry_verification_broadcast_run.rs
#	core/src/broadcast_stage/standard_broadcast_run.rs
#	gossip/src/duplicate_shred.rs
#	ledger/src/blockstore.rs
#	ledger/src/shred.rs
#	ledger/tests/shred.rs

* removes mergify merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-05 23:05:40 +00:00
2f8991c312 simplifies ShredIndex api (backport #21932) (#22962)
* simplifies ShredIndex api (#21932)

(cherry picked from commit efd64a3862)

# Conflicts:
#	ledger/src/blockstore.rs

* removes mergify merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-05 18:13:00 +00:00
59dc5eb5b8 adds ErasureSetId identifying erasure coding sets of shreds (backport #21928) (#22953)
* adds ErasureSetId identifying erasure coding sets of shreds (#21928)

(cherry picked from commit 8183f28636)

# Conflicts:
#	ledger/src/blockstore.rs

* removes mergify merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-05 00:08:50 +00:00
9361a50445 adds ShredId uniquely identifying each shred (backport #21820) (#22940)
* adds ShredId uniquely identifying each shred (#21820)

(cherry picked from commit 4ceb2689f5)

# Conflicts:
#	ledger/src/blockstore.rs

* removes mergify merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-04 20:45:39 +00:00
86e369ec8e uses Option<Slot> for SlotMeta.parent_slot (backport #21808) (#22936)
* uses Option<Slot> for SlotMeta.parent_slot (#21808)

SlotMeta.parent_slot for the head of a detached chain of slots is
unknown and that is indicated by u64::MAX which lacks type-safety:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L203-L205

The commit changes the type to Option<Slot>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.

(cherry picked from commit 8d980f07ba)

# Conflicts:
#	core/src/repair_generic_traversal.rs
#	ledger-tool/src/main.rs
#	ledger/src/blockstore.rs

* removes mergify merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-04 16:30:31 +00:00
5136ed00a8 Bump bpf-tools to v1.23 (#22928)
(cherry picked from commit a9d9a5095b)

Co-authored-by: Dmitri Makarov <dmakarov@alumni.stanford.edu>
2022-02-04 04:54:50 +00:00
70ec2cd244 Return actual committed transactions from process_transactions() (backport #22802) (#22904)
* Return actual committed transactions from process_transactions()

* resolve conflicts

* fixup comment

* Fixup banking_stage-dropped_tx_before_forwarding counter

* Count cost model throttled transactions

* fixup tx_count moved

* Fixup tests

* remove qos service

* Cleanup clippy

Co-authored-by: Carl Lin <carl@solana.com>
2022-02-04 03:52:11 +00:00
430cdf679e rpc: use minimal mode by default (backport #22734) (#22878)
* rpc: use minimal mode by default

(cherry picked from commit eac4a6df68)

# Conflicts:
#	core/src/validator.rs
#	local-cluster/tests/common.rs
#	local-cluster/tests/local_cluster.rs
#	replica-node/tests/local_replica.rs
#	rpc/src/rpc_service.rs
#	transaction-dos/src/main.rs

* test-validator-bin: reinstate full rpc method set

Co-authored-by: Trent Nelson <trent@solana.com>
2022-02-03 21:11:58 +00:00
f32c33dd80 uses Option<u64> for SlotMeta.last_index (backport #21775) (#22915)
* uses Option<u64> for SlotMeta.last_index (#21775)

SlotMeta.last_index may be unknown and current code is using u64::MAX to
indicate that:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L169-L174

This lacks type-safety and can introduce bugs if not always checked for
Several instances of slot_meta.last_index + 1 are also subject to
overflow.

This commit updates the type to Option<u64>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.

(cherry picked from commit e08139f949)

# Conflicts:
#	core/src/repair_generic_traversal.rs
#	ledger/src/blockstore.rs
#	ledger/src/blockstore_meta.rs

* removes mergify merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-03 20:47:58 +00:00
6e46511cc8 adds back ErasureMeta::first_coding_index field (backport #21623) (#22912)
* adds back ErasureMeta::first_coding_index field (#21623)

https://github.com/solana-labs/solana/pull/16646
removed first_coding_index since the field is currently redundant and
always equal to fec_set_index.
However, with upcoming changes to erasure coding schema, this will no
longer be the same as fec_set_index and so requires a separate field to
represent.

(cherry picked from commit 49ba09b333)

# Conflicts:
#	ledger/src/blockstore.rs

* removes mergify merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-03 17:59:13 +00:00
59dd876d79 Persist coding shreds (backport #21214) (#22891)
* Persist coding shreds (#21214)

(cherry picked from commit 38fcfb7542)

# Conflicts:
#	ledger/src/blockstore.rs

* removes mergify merge conflicts

Co-authored-by: Yueh-Hsuan Chiang <93241502+yhchiang-sol@users.noreply.github.com>
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-02 19:01:41 +00:00
d1188d7c08 Keep all erasure coding shreds even on successful recovery (#20968) (#21052) (#22890)
Problem
Blockstore currently removes erasure shreds if the data shreds are
successfully recovered on insert, which is an issue if we want to
serve coding shreds over repair.

Summary of Changes
This diff keeps all coding shreds even on successful recovery and
changes change the signature of prev_inserted_codes to immutable
reference to ensure its immunity.

Fixes #20968

(cherry picked from commit 3aa49e2c69)

Co-authored-by: Yueh-Hsuan Chiang <93241502+yhchiang-sol@users.noreply.github.com>
2022-02-02 17:09:07 +00:00
89bc329cd2 docs-ci: prebuild cli bin with output to appease TravisCI hang check (#22883)
(cherry picked from commit 2fda90e414)

Co-authored-by: Trent Nelson <trent@solana.com>
2022-02-02 08:40:56 +00:00
6837d7691c adds more sanity checks to shreds (backport #21675) (#22869)
* adds more sanity checks to shreds (#21675)

(cherry picked from commit 8063273d09)

# Conflicts:
#	ledger/src/blockstore.rs

* removes mergify merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-02-01 18:13:19 +00:00
0afd9d81df Clean up before credits_auto_rewind (#22839) (#22865)
* Clean up before credits_auto_rewind

* Use `=` intead of `|=` for mutable bool

(cherry picked from commit 545c97f903)

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2022-02-01 15:29:03 +00:00
add26be71b rpc-sts: dedupe before initial send (#22855)
(cherry picked from commit 9f1f7aff2b)

Co-authored-by: Trent Nelson <trent@solana.com>
2022-02-01 01:26:03 +00:00
ac36954af0 Update ping to transfer to self, with rotating amount (backport #22657) (#22853)
* Update ping to transfer to self, with rotating amount (#22657)

* Update ping to transfer to self, with rotating amount

* Remove balance check

(cherry picked from commit 90689585ef)

# Conflicts:
#	cli/src/cluster_query.rs

* Fix api diff

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2022-02-01 00:40:14 +00:00
528980d037 changes Shred::parent return type to Option<Slot> (backport #21370) (#22844)
* changes Shred::parent return type to Option<Slot> (#21370)

Shred::parent can return garbage if the struct fields are invalid:
https://github.com/solana-labs/solana/blob/8a50b6302/ledger/src/shred.rs#L446-L453

The commit adds more sanity checks and changes the return type to Option<Slot>.

(cherry picked from commit dd338b6c9f)

# Conflicts:
#	ledger/src/shred.rs

* removes mergify merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-31 16:37:34 +00:00
375a939000 Add new_from_parent() timings (#22744) (#22805)
Co-authored-by: carllin <carl@solana.com>
2022-01-28 10:43:37 +00:00
7dd7cea0c7 Improve poh recorder metrics (backport #22730) (#22763)
* Improve poh recorder metrics (#22730)

* Improve poh recorder metrics

* Add metric for poh service send record

* feedback

* clean up

(cherry picked from commit 115b488807)

# Conflicts:
#	poh/src/poh_recorder.rs

* fix conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-28 06:30:35 +00:00
597342a643 Always contact release.solana.com over https (#22794)
(cherry picked from commit bd86459a94)

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-01-27 21:44:47 +00:00
367e23fe22 Restrict the Mergify copy command to core contributors (backport #22789) (#22791)
* Restrict the Mergify copy command to core contributors

(cherry picked from commit c0638439be)

# Conflicts:
#	.mergify.yml

* Update .mergify.yml

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-01-27 17:11:32 +00:00
92cc75b3ae test_ed25519 fails if we randomly select index 1 (#22779)
(cherry picked from commit c1b543c74d)

# Conflicts:
#       sdk/src/ed25519_instruction.rs

Co-authored-by: Sean Young <sean@mess.org>
2022-01-27 13:02:42 +00:00
fdc1b046bc Set the correct root in block commitment cache initialization (#22750) (#22756)
* Set the correct root in block commitment cache initialization

* clean up test

* bump

(cherry picked from commit d9c259a231)

# Conflicts:
#	replica-node/src/replica_node.rs

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-27 02:49:28 +00:00
6979d01e4c spl-associated-token-account: Add feature for new program (backport #22648) (#22718)
* spl-associated-token-account: Add feature for new program (#22648)

* spl-associated-token-account: Add feature for new program

* Address feedback

(cherry picked from commit fc21af4e6e)

# Conflicts:
#	runtime/src/bank.rs
#	runtime/src/lib.rs
#	sdk/src/feature_set.rs

* Fix merge conflicts

Co-authored-by: Jon Cinque <jon.cinque@gmail.com>
2022-01-25 00:16:18 +00:00
a8c56a1dfa Move discard check before generate offsets (#22684) (#22706)
(cherry picked from commit 7569f282c6)

Co-authored-by: sakridge <sakridge@gmail.com>
2022-01-24 16:03:12 +00:00
5d27a7f4c4 Refactor: Rename variables and helper method to PohRecorder (backport #22676) (#22687)
* Refactor: Rename variables and helper method to `PohRecorder` (#22676)

* Refactor: Rename leader_first_tick_height field

* Refactor: add `PohRecorder::slot_for_tick_height` helper

* Refactor: Add type for poh leader status

(cherry picked from commit 1240217a73)

# Conflicts:
#	core/src/replay_stage.rs
#	poh/src/poh_recorder.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-23 04:16:53 +00:00
de46df6b1b Perf: Only check executors cache for executable bpf program ids (backport #22624) (#22628)
* Perf: Only check executors cache for executable bpf program ids (#22624)

* Only check executors cache for executable bpf program ids

* switch to native loader check

* clean up tests

* fix tests

* clippy

(cherry picked from commit 7d34a7acac)

# Conflicts:
#	runtime/src/bank.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-22 14:12:29 +08:00
4b7450e89e Perf: Store deserialized sysvars in the sysvars cache (backport #22455) (#22627)
* Perf: Store deserialized sysvars in the sysvars cache (#22455)

* resolve conflicts

* remove bench
2022-01-22 14:09:05 +08:00
f3126f7e77 sigverify -- dedupe bloom filter too slow followups 2022-01-22 05:57:24 +00:00
9b90162564 Bump version to 1.8.15 (#22670) 2022-01-22 11:15:07 +08:00
9c01d90c70 Handle already discarded packets in discard_excess_packets (#22594) (#22634)
(cherry picked from commit 38b02bbcc0)

Co-authored-by: sakridge <sakridge@gmail.com>
v1.8.14
2022-01-22 01:10:31 +00:00
edf1954817 Faster dedup v1.8 (#22619)
* Faster dedup
2022-01-21 08:19:55 -08:00
59eee75d65 Speed up packet dedup and fix benches (backport #22592) (#22611)
* Speed up packet dedup and fix benches (#22592)

* Speed up packet dedup and fix benches

* fix tests

* allow int arithmetic in bench

(cherry picked from commit a2d251ce1e)

# Conflicts:
#	perf/src/sigverify.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-21 07:34:57 +00:00
c2dd9a006d Fetch sysvars from invoke context for vote program (backport #22444) (#22621)
* Fetch sysvars from invoke context for vote program (#22444)

* resolve conflicts
2022-01-21 02:59:33 +00:00
8eb0a1091a system-monitor-service: support percentages from bigger numbers (#22597)
(cherry picked from commit cca3dbc76d)

Co-authored-by: Trent Nelson <trent@solana.com>
2022-01-20 23:22:48 +00:00
cb5106a15b Dedup v1.8 (#22584)
backport
2022-01-19 21:31:55 -08:00
2d5957a4b4 Refactor: move sysvar cache to new module (#22586) 2022-01-20 03:27:03 +00:00
99846eea12 Add PacketBatch packet_indexes stat (#22564) (#22574)
* collect stats on packet batch indicies

* cleanup

* cleanup

* cleanup

* change name

(cherry picked from commit 650882217c)

# Conflicts:
#	core/src/banking_stage.rs

Co-authored-by: buffalu <85544055+buffalu@users.noreply.github.com>
2022-01-20 00:48:18 +00:00
78b82dedb1 improves sigverify discard_excess_packets performance (backport #22577) (#22579)
* improves sigverify discard_excess_packets performance (#22577)

As shown by the added benchmark, current code does worse if there is a
spam address plus a lot of unique addresses.

on current master:
test bench_packet_discard_many_senders  ... bench:   1,997,960 ns/iter (+/- 103,715)
test bench_packet_discard_mixed_senders ... bench:  14,256,116 ns/iter (+/- 534,865)
test bench_packet_discard_single_sender ... bench:   1,306,809 ns/iter (+/- 61,992)

with this commit:
test bench_packet_discard_many_senders  ... bench:   1,644,025 ns/iter (+/- 83,715)
test bench_packet_discard_mixed_senders ... bench:   1,089,789 ns/iter (+/- 86,324)
test bench_packet_discard_single_sender ... bench:     955,234 ns/iter (+/- 55,953)

(cherry picked from commit dcf44d2523)

# Conflicts:
#	core/benches/sigverify_stage.rs
#	core/src/sigverify_stage.rs

* removes mergify merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-19 21:23:34 +00:00
db942269f0 Stop caching sysvars, instead load them ahead of time (backport #21108) (#22572)
* Bump version to v1.8.14

* Stop caching sysvars, instead load them ahead of time (backport #21108)

Co-authored-by: Trent Nelson <trent@solana.com>
2022-01-19 17:42:43 +08:00
eede487797 Bump version to v1.8.14 (#22573) 2022-01-19 17:40:37 +08:00