2355 Commits

Author SHA1 Message Date
7a421fe602 Port counting vote CUs to block cost to v1.9 2022-01-14 10:49:43 -06:00
4ee6bc9a93 downgrade individual per-program-timing to trace to reduce writes to influx (#22471)
(cherry picked from commit 6614727be8)

Co-authored-by: Tao Zhu <tao@solana.com>
2022-01-13 02:37:47 +00:00
58dcc451a9 Prevent rent-paying account creation (backport #22292) (#22428)
* Prevent rent-paying account creation (#22292)

* Fixup typo

* Add new feature

* Add new TransactionError

* Add framework for checking account state before and after transaction processing

* Fail transactions that leave new rent-paying accounts

* Only check rent-state of writable tx accounts

* Review comments: combine process_result success behavior; log and metrics before feature activation

* Fix tests that assume rent-exempt accounts are okay

* Remove test no longer relevant

* Remove native/sysvar special case

* Move metrics submission to report legacy->legacy rent paying transitions as well

(cherry picked from commit 637e366b18)

# Conflicts:
#	runtime/src/bank.rs
#	runtime/src/lib.rs

* Fix conflicts and rework for TransactionRefCells

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2022-01-11 23:17:03 +00:00
1f00926874 Add runtime support for address table lookups (backport #22223) (#22354) 2022-01-08 07:57:04 +08:00
7b1da62763 Add execute metrics (backport #22296) (#22335)
* move `ExecuteTimings` from `runtime::bank` to `program_runtime::timings`

(cherry picked from commit 7d32909e17)

# Conflicts:
#	core/Cargo.toml
#	ledger/Cargo.toml
#	programs/bpf/Cargo.lock

* Add execute metrics

(cherry picked from commit b25e4a200b)

* Add metrics for executor creation

(cherry picked from commit 848b6dfbdd)

* Add helper macro for `AddAssign`ing with saturating arithmetic

(cherry picked from commit deb9344e49)

* Use saturating_add_assign macro

(cherry picked from commit 72fc6096a0)

* Consolidate process instruction execution timings to own struct

(cherry picked from commit 390ef0fbcd)

Co-authored-by: Trent Nelson <trent@solana.com>
Co-authored-by: Carl Lin <carl@solana.com>
2022-01-07 09:11:18 +00:00
8578429c4d Refactor: Improve type safety and readability of transaction execution (backport #22215) (#22289)
* Refactor: Improve type safety and readability of transaction execution (#22215)

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-05 23:01:15 +08:00
fb0e5adc7e patches bug in recv_mmsg when npkts != nrecv (backport #22276) (#22280)
* removes total-size from return value of recv_mmsg

(cherry picked from commit 4b24499916)

* patches bug in recv_mmsg when npkts != nrecv

If recv_mmsg receives 2 packets where the first one is filtered out,
then it returns npkts == 1:
https://github.com/solana-labs/solana/blob/01a096adc/streamer/src/recvmmsg.rs#L104-L115

But then streamer::packet::recv_from will erroneously keep the 1st
packet and drop the 2nd one:
https://github.com/solana-labs/solana/blob/01a096adc/streamer/src/packet.rs#L34-L49

To avoid this bug, this commit updates recv_mmsg to always return total
number of received packets. If socket address cannot be correctly
obtained, it is left as the default value which is UNSPECIFIED:
https://github.com/solana-labs/solana/blob/01a096adc/sdk/src/packet.rs#L145

(cherry picked from commit 379feecae5)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-04 23:42:52 +00:00
d84b994451 shrinks size of Packet.Meta (backport #22224) (#22273)
* removes seed and slot fields from Packet.Meta

507367e6ac
updated window-service to send shreds (as opposed to packets) to
retransmit-stage and so seed and slot fields in Packet.Meta are unused:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L27-L28

(cherry picked from commit aa9f7ed7e8)

* uses std::net::IpAddr type for Packet.Meta.addr

(cherry picked from commit 73a7741c49)

# Conflicts:
#	streamer/src/streamer.rs

* adds bitflags to Packet.Meta

Instead of a separate bool type for each flag, all the flags can be
encoded in a type-safe bitflags encoded in a single u8:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L19-L31

(cherry picked from commit 01a096adc8)

# Conflicts:
#	sdk/Cargo.toml

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-04 17:38:29 +00:00
185f52b712 Lower vote-only-mode to 400 (#22210) (#22272)
(cherry picked from commit 2486e21ffe)

Co-authored-by: sakridge <sakridge@gmail.com>
2022-01-04 15:15:16 +00:00
f8bf478fde Fix bug, add error specific timings (#22225) (#22252)
(cherry picked from commit 005592998d)

Co-authored-by: carllin <carl@solana.com>
2022-01-04 02:53:59 +00:00
35fb47d1ce removes epoch_authorized_voters from VoteTracker (backport #22192) (#22248)
* removes epoch_authorized_voters from VoteTracker (#22192)

https://github.com/solana-labs/solana/pull/22169
verifies authorized-voter early on in vote-listener pipeline; and so
VoteTracker no longer needs to maintain and check for epoch authorized
voters.

(cherry picked from commit 69d71f8f86)

# Conflicts:
#	core/src/cluster_info_vote_listener.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-04 01:51:24 +00:00
ca451ea23e checks for authorized voter early on in the vote-listener pipeline (backport #22169) (#22206)
* checks for authorized voter early on in the vote-listener pipeline (#22169)

Before votes are verified that they are signed by the authorized voter,
they might be dropped in verified-vote-packets code. If there are
enough many spam votes from unauthorized voters, this may potentially
drop valid votes but keep the false ones.
https://github.com/solana-labs/solana/blob/57986f982/core/src/verified_vote_packets.rs#L165-L168

(cherry picked from commit c0c6038654)

# Conflicts:
#	core/src/cluster_info_vote_listener.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-31 21:46:12 +00:00
113d261a2c Count compute units even when transaction errors (backport #22182) (#22199)
* Count compute units even when transaction errors (#22182)

(cherry picked from commit d06e6c7425)

# Conflicts:
#	program-runtime/src/invoke_context.rs
#	runtime/src/cost_model.rs
#	runtime/src/message_processor.rs

* Resolve conflicts

Co-authored-by: carllin <carl@solana.com>
2021-12-31 21:14:00 +00:00
7efd0391e9 Revert "Count compute units even when transaction errors (backport #22059) (#22154)" (#22175)
This reverts commit 401c542d2a.
2021-12-30 02:39:25 -05:00
6a556c5adb Stream additional block metadata via plugin (#22023) (#22179)
* Stream additional block metadata through plugin
blockhash, block_height, block_time, rewards are streamed

(cherry picked from commit f14928a970)

Co-authored-by: Lijun Wang <83639177+lijunwangs@users.noreply.github.com>
2021-12-30 05:44:12 +00:00
531f36c571 Don't forward packets received from TPU forwards port (#22078) (#22171)
* Don't forward packets received from TPU forwards port

* Add banking stage test

(cherry picked from commit b1d9a2e60e)

Co-authored-by: Justin Starry <justin@solana.com>
2021-12-30 10:42:28 +08:00
9fb67f9b07 Prevent log spam (#22148) (#22152)
(cherry picked from commit f061059e45)

Co-authored-by: carllin <carl@solana.com>
2021-12-29 08:28:48 +00:00
401c542d2a Count compute units even when transaction errors (backport #22059) (#22154)
* Count compute units even when transaction errors (#22059)

(cherry picked from commit eaa8c67bde)

# Conflicts:
#	program-runtime/src/invoke_context.rs
#	runtime/src/bank.rs
#	runtime/src/message_processor.rs

* Fix merge conflicts

Co-authored-by: carllin <carl@solana.com>
2021-12-29 08:04:12 +00:00
810ca36eae skip reporting all-zero stats (#22054)
(cherry picked from commit 9c5d82557a)

Co-authored-by: Tao Zhu <tao@solana.com>
2021-12-28 07:03:13 +00:00
16f821ea8c Ensure AncestorHashesSerice selects an open port (#21919) (#21997)
(cherry picked from commit 7f6fb6937a)

Co-authored-by: carllin <wumu727@gmail.com>
2021-12-28 06:46:59 +00:00
be0bcd85ed tracks erasure coding shreds' indices explicitly (#21822) (#22094)
The indices for erasure coding shreds are tied to data shreds:
https://github.com/solana-labs/solana/blob/90f41fd9b/ledger/src/shred.rs#L921

However with the upcoming changes to erasure schema, there will be more
erasure coding shreds than data shreds and we can no longer infer coding
shreds indices from data shreds.

The commit adds constructs to track coding shreds indices explicitly.

(cherry picked from commit 65d59f4ef0)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-23 19:38:50 +00:00
e3a4b98432 removes Select in favor of recv_timeout/try_iter (#21981) (#22001)
crossbeam_channel::Select::ready_timeout might return with success spuriously.

(cherry picked from commit 7476dfeec0)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-18 19:37:07 +00:00
91657ba8fe new net-stats require a new table (#21996) (#22000)
(cherry picked from commit 3fe942ab30)

Co-authored-by: Jeff Biseda <jbiseda@gmail.com>
2021-12-18 10:26:16 +00:00
35ee48bec9 RPC Block Subscription (backport #21787) (#21992)
* RPC Block Subscription (#21787)

* add stuff

* compiling

* add notify block

* wip

* feat: add blockSubscribe pubsub method

* address PR comments

Co-authored-by: Lucas B <buffalu@jito.network>
Co-authored-by: Zano <segfaultdoctor@protonmail.com>
(cherry picked from commit 76098dd42a)

# Conflicts:
#	Cargo.lock
#	client-test/Cargo.toml
#	rpc/src/rpc_subscriptions.rs

* Fix conflicts

Co-authored-by: segfaultdoctor <seg@jito.network>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-12-18 01:43:37 +00:00
e572678176 removes next_shred_index from return value of entries to shreds api (#21961) (#21980)
next-shred-index is already readily available from returned data shreds.
The commit simplifies the api for upcoming changes to erasure coding
schema which will require explicit tracking of indices for coding shreds
as well as data shreds.

(cherry picked from commit 89d66c3210)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-17 17:57:57 +00:00
f4521002b9 Clean up demote program write lock feature (backport #21949) (#21969)
* Clean up demote program write lock feature (#21949)

* Clean up demote program write lock feature

* fix test

(cherry picked from commit 6ff0be6a82)

# Conflicts:
#	programs/bpf_loader/src/syscalls.rs
#	runtime/src/accounts.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2021-12-17 04:45:22 +00:00
41ea597256 Fix subtraction overflow (#21871) (#21901)
(cherry picked from commit cb395abff7)

Co-authored-by: carllin <carl@solana.com>
2021-12-14 23:22:47 +00:00
a056fd88cb uses Option<Slot> for SlotMeta.parent_slot (backport #21808) (#21899)
* uses Option<Slot> for SlotMeta.parent_slot (#21808)

SlotMeta.parent_slot for the head of a detached chain of slots is
unknown and that is indicated by u64::MAX which lacks type-safety:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L203-L205

The commit changes the type to Option<Slot>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.

(cherry picked from commit 8d980f07ba)

# Conflicts:
#	ledger-tool/src/main.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-14 21:42:57 +00:00
2f1816d1db adds ShredId uniquely identifying each shred (backport #21820) (#21897)
* adds ShredId uniquely identifying each shred (#21820)

(cherry picked from commit 4ceb2689f5)

# Conflicts:
#	ledger/src/blockstore.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-14 21:03:08 +00:00
9956afb2bd uses Option<u64> for SlotMeta.last_index (#21775) (#21806)
SlotMeta.last_index may be unknown and current code is using u64::MAX to
indicate that:
https://github.com/solana-labs/solana/blob/6c108c8fc/ledger/src/blockstore_meta.rs#L169-L174

This lacks type-safety and can introduce bugs if not always checked for
Several instances of slot_meta.last_index + 1 are also subject to
overflow.

This commit updates the type to Option<u64>. Backward compatibility is
maintained by customizing serde serialize/deserialize implementations.

(cherry picked from commit e08139f949)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-11 17:39:05 +00:00
01941cf3de Rename Packets to PacketBatch (backport #21794) (#21805)
* Rename Packets to PacketBatch (#21794)

(cherry picked from commit 254ef3e7b6)

# Conflicts:
#	core/src/verified_vote_packets.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2021-12-11 16:53:23 +00:00
8cacf82cb8 adds more sanity checks to shreds (#21675) (#21734)
(cherry picked from commit 8063273d09)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-09 18:44:43 +00:00
b4c8e095bd adds back position field to coding-shred-header (#21600) (#21620)
https://github.com/solana-labs/solana/pull/17004
removed position field from coding-shred-header because as it stands the
field is redundant and unused.
However, with the upcoming changes to erasure coding schema this field
will no longer be redundant and needs to be populated.

(cherry picked from commit cd17f63d81)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-05 16:40:22 +00:00
9c6b95e1e1 fix distance calculation in get_closest_completion (#21601) 2021-12-03 22:36:46 -08:00
1430b58a6d Remove deprecated slow epoch boundary methods (#21568) 2021-12-03 17:59:10 +00:00
b8837c04ec Reformat imports to a consistent style for imports
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
2021-12-03 09:19:13 -08:00
b78f5b6032 Refactor: Cleanup InstructionProcessor (#21404)
* Moves create_message(), native_invoke() and process_cross_program_instruction()
from the InstructionProcessor to the InvokeContext so that they can have a useful "self" parameter.

* Moves InstructionProcessor into InvokeContext and Bank.

* Moves ExecuteDetailsTimings into its own file.

* Moves Executor into invoke_context.rs

* Moves PreAccount into its own file.

* impl AbiExample for BuiltinPrograms
2021-12-01 08:54:42 +01:00
ba9dfa0d22 Remove frozen account support 2021-11-29 08:38:11 -08:00
9edfc5936d Refactor accounts.rs with Justin's comments to improve lock accounts (#21406)
with results code path.
- fix a bug that could unlock accounts that weren't locked
- add test to the refactored function
- skip enumerating transaction accounts if qos results is an error
- add #[must_use] annotation
- avoid clone error in results
- add qos error code to unlock_accounts match statement
- remove unnecessary AbiExample
2021-11-23 21:17:55 +00:00
c29838fce1 Accountsdb plugin transaction part 3: Transaction Notifier (#21374)
The TransactionNotifierInterface interface for notifying transactions.
Changes to transaction_status_service to notify the notifier of the transaction data.
Interface to query the plugin's interest in transaction data
2021-11-23 09:55:53 -08:00
2602e7c3bc Fix flaky test (#21402)
* the async test is flaky on ci

* fix unstable test by increasing stats repoting time
2021-11-23 09:47:17 -06:00
dd338b6c9f changes Shred::parent return type to Option<Slot> (#21370)
Shred::parent can return garbage if the struct fields are invalid:
https://github.com/solana-labs/solana/blob/8a50b6302/ledger/src/shred.rs#L446-L453

The commit adds more sanity checks and changes the return type to Option<Slot>.
2021-11-23 14:45:26 +00:00
cd5a39ee43 the async test is flaky on ci (#21365) 2021-11-22 18:16:20 -06:00
87831e7f8d start system monitor earlier in validator so we get memory stats at startup (#21372) 2021-11-22 14:37:17 -06:00
f31ca8ba8c Report cluster slots size (#21380) 2021-11-22 17:47:58 +01:00
2ed7e3af89 prioritize slot repairs for unknown last index and close to completion (#21070) 2021-11-19 19:17:30 -08:00
0bda0c3e0c Add bank drop service (#21322) 2021-11-19 17:20:18 +01:00
48dfdfb4d5 changes Blockstore::is_shred_duplicate arg type to ShredType 2021-11-19 14:16:39 +00:00
57057f8d39 uses enum for shred type
Current code is using u8 which does not have any type-safety and can
contain invalid values:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L167

Checks for invalid shred-types are scattered through the code:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/blockstore.rs#L849-L851
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L346-L348

The commit uses enum for shred type with #[repr(u8)]. Backward
compatibility is maintained by implementing Serialize and Deserialize
compatible with u8, and adding a test to assert that.
2021-11-19 14:16:39 +00:00
b30c94ce55 ClusterInfoVoteListener send only missing votes to BankingStage (#20873) 2021-11-18 15:20:41 -08:00