Commit Graph

17046 Commits

Author SHA1 Message Date
34984ed16e v1.9: Only examine explicit tx accounts for rent state (#22442)
* Add failing test

* Fix: only examine accounts explicitly included in a tx
2022-01-11 20:55:10 -07:00
f4d1577337 Refactor: consolidate memo extraction for each message version (#22422) (#22435)
(cherry picked from commit 35a5dd9c45)

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-12 00:23:03 +00:00
58dcc451a9 Prevent rent-paying account creation (backport #22292) (#22428)
* Prevent rent-paying account creation (#22292)

* Fixup typo

* Add new feature

* Add new TransactionError

* Add framework for checking account state before and after transaction processing

* Fail transactions that leave new rent-paying accounts

* Only check rent-state of writable tx accounts

* Review comments: combine process_result success behavior; log and metrics before feature activation

* Fix tests that assume rent-exempt accounts are okay

* Remove test no longer relevant

* Remove native/sysvar special case

* Move metrics submission to report legacy->legacy rent paying transitions as well

(cherry picked from commit 637e366b18)

# Conflicts:
#	runtime/src/bank.rs
#	runtime/src/lib.rs

* Fix conflicts and rework for TransactionRefCells

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2022-01-11 23:17:03 +00:00
f0695ef6d9 optimizes ReadOnlyAccountsCache LRU eviction implementation (backport #22403) (#22426)
* optimizes ReadOnlyAccountsCache LRU eviction implementation (#22403)

ReadOnlyAccountsCache is using a background thread, table scan and sort
to implement LRU eviction policy:
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L66-L73
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L186-L191
https://github.com/solana-labs/solana/blob/eaa52bc93/runtime/src/read_only_accounts_cache.rs#L222

DashMap internally locks each shard when accessed; so a table scan in
the background thread can create a lot of lock contention.

This commit adds an index-list queue containing cached keys in the order
that they are accessed. Each hash-map entry also includes its index into
this queue.
When an item is first entered into the cache, it is added to the end of
the queue. Also each time an entry is looked up from the cache it is
moved to the end of queue. As a result, items in the queue are always
sorted in the order that they have last been accessed. When doing LRU
eviction, cache entries are evicted from the front of the queue.
Using index-list, all queue operations above are O(1) with low overhead
and so above achieves an efficient implementation of LRU cache eviction
policy.

(cherry picked from commit a49ef49f87)

# Conflicts:
#	Cargo.lock
#	programs/bpf/Cargo.lock
#	runtime/Cargo.toml
#	runtime/src/accounts_db.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-11 19:09:57 +00:00
41b0d6cca3 limits gossip vote stats to the top most voted slots (#22416) (#22418)
(cherry picked from commit 49da347d84)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-10 23:14:57 +00:00
ae77a52c97 wrap create executor timings datapoint in a module (#22398)
(cherry picked from commit 428575f9ae)

Co-authored-by: Trent Nelson <trent@solana.com>
2022-01-09 06:41:44 +00:00
133314e58c bank: fix executor cache metrics (#22396)
(cherry picked from commit 3b4aad9df1)

Co-authored-by: Trent Nelson <trent@solana.com>
2022-01-09 06:06:35 +00:00
cb49ae21b4 Bump version to v1.9.5 2022-01-08 21:17:51 +00:00
a9ebba5643 Clarify docs of minimum_balance (#22385) (#22387)
(cherry picked from commit 0f94e1d3a2)

Co-authored-by: Evan Conrad <evan@roomservice.dev>
2022-01-08 20:07:57 +00:00
8ce65878da improve multi executor cache addition (#22382)
Co-authored-by: Jack May <jack@solana.com>
v1.9.4
2022-01-08 13:03:46 +00:00
a4ca18a54d add excutor creation trace timings 2022-01-08 05:25:37 -07:00
7cb147fdcd Executor cache count primer (backport #22333) (#22375)
* bank: prime new executor cache entry use-counts

(cherry picked from commit 4ce48307bb)

* --amend

(cherry picked from commit ad3cb0bc93)

Co-authored-by: Trent Nelson <trent@solana.com>
2022-01-08 11:01:34 +00:00
2d693be9fa remove per program timings from blockstore processor ledger replay (#22370) (#22372)
(cherry picked from commit 813006b33b)

Co-authored-by: carllin <carl@solana.com>
2022-01-08 08:43:48 +00:00
50e716fc80 bank: Add executors cache metrics (#22368)
(cherry picked from commit 6d76db1de5)

Co-authored-by: Trent Nelson <trent@solana.com>
2022-01-08 01:34:53 +00:00
1f00926874 Add runtime support for address table lookups (backport #22223) (#22354) 2022-01-08 07:57:04 +08:00
662c6be51e removes CowCachedExecutors (#22343) (#22363)
Copy-on-write semantics for cached executors can be implemented by a
simple Arc<CachedExecutors> as opposed to CowCachedExecutors:
https://github.com/solana-labs/solana/blob/f1e2598ba/runtime/src/bank.rs#L244-L247

This will also avoid the need for double locking as in:
https://github.com/solana-labs/solana/blob/f1e2598ba/runtime/src/bank.rs#L3490-L3491
https://github.com/solana-labs/solana/blob/f1e2598ba/runtime/src/bank.rs#L3525-L3526

(cherry picked from commit c2389fc209)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-07 16:04:13 +00:00
9761f5b67f Add aarch64-apple-darwin publish tarball step (#22356)
(cherry picked from commit e2aa932e97)

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-01-07 10:17:11 +00:00
7b1da62763 Add execute metrics (backport #22296) (#22335)
* move `ExecuteTimings` from `runtime::bank` to `program_runtime::timings`

(cherry picked from commit 7d32909e17)

# Conflicts:
#	core/Cargo.toml
#	ledger/Cargo.toml
#	programs/bpf/Cargo.lock

* Add execute metrics

(cherry picked from commit b25e4a200b)

* Add metrics for executor creation

(cherry picked from commit 848b6dfbdd)

* Add helper macro for `AddAssign`ing with saturating arithmetic

(cherry picked from commit deb9344e49)

* Use saturating_add_assign macro

(cherry picked from commit 72fc6096a0)

* Consolidate process instruction execution timings to own struct

(cherry picked from commit 390ef0fbcd)

Co-authored-by: Trent Nelson <trent@solana.com>
Co-authored-by: Carl Lin <carl@solana.com>
2022-01-07 09:11:18 +00:00
2f97fee71a Cleanup ledger-tool analyze-storage command (#22310) (#22352)
* Make ledger-tool analyze-storage use Blockstore::open()

Opening a large ledger may require setting a larger open file descriptor
limit. Blockstore::open() does this whereas the underlying Database
object that analyze-storage was opening does not.

* Move key_size call lookup to take advantage of traits

* Fix typo where analyze worked on wrong column

* Make analyze-storage analyze all columns

(cherry picked from commit 9f1f64e384)

Co-authored-by: steviez <steven@solana.com>
2022-01-07 07:47:27 +00:00
3ae674dd28 Increase timeout of local-cluster-slow CI step 2022-01-07 15:31:10 +08:00
8214bc9db4 Retain executor cache counts (#22322) (#22341)
(cherry picked from commit f1e2598baa)

Co-authored-by: Jack May <jack@solana.com>
2022-01-06 19:00:29 +00:00
1132def37c Split up local cluster tests into separate CI steps (backport #22295) (#22303)
* Split up local cluster tests into separate CI steps (#22295)

* Split up local cluster tests into separate CI steps

* Update buildkite-pipeline.sh

(cherry picked from commit 0e1afcbb26)

# Conflicts:
#	local-cluster/tests/local_cluster.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-06 17:02:45 +00:00
7267ebaaf2 Consume from AccountsDataMeter (backport #21994) (#22323)
* Consume from AccountsDataMeter (#21994)

(cherry picked from commit 1460f00e0f)

# Conflicts:
#	program-runtime/src/invoke_context.rs

* fixup! conflicts

* fix tests for v1.9

* fixup! clippy

Co-authored-by: Brooks Prumo <brooks@solana.com>
2022-01-06 17:01:02 +00:00
4be6e52a4f cache executors on failed transactions (backport #22308) (#22328)
* cache executors on failed transactions (#22308)

(cherry picked from commit 12e160269e)

# Conflicts:
#	program-runtime/src/invoke_context.rs
#	runtime/src/bank.rs

* resolve conflicts

Co-authored-by: Jack May <jack@solana.com>
2022-01-06 09:14:48 +00:00
e7348243b4 [ledger-tool]compare_blocks (#22229) (#22330)
* 1.made load_credentials accept credential path as a parameter. 2.partial implement bigtable comparasion function

* finding missing blocks in bigtables in a specified range

* refactor compare-blocks,add unit test for missing_blocks and fmt

* compare-block fix last block bug

* refactor compare-block and improve wording

* Update ledger-tool/src/bigtable.rs

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* update compare-block command-line description

* style:improve wording/naming/code style

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
(cherry picked from commit d9220652ad)

Co-authored-by: pieceofr <komimi.p@gmail.com>
2022-01-06 08:55:26 +00:00
fc0c74d722 Only sum accounts data len from non-zero lamport accounts (#22309) (#22317)
(cherry picked from commit ab13e39518)

Co-authored-by: Brooks Prumo <brooks@solana.com>
2022-01-06 02:43:54 +00:00
687cd4779e Add AccountsDataMeter to InvokeContext (#21813) (#22299)
(cherry picked from commit 800472ddf5)

Co-authored-by: Brooks Prumo <brooks@solana.com>
2022-01-06 01:31:11 +00:00
b28d7050ab Update default --dynamic-port-range values to include some room for additional ports that may be added in the future (#22321)
(cherry picked from commit 37ebd9bd9e)

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-01-06 01:29:06 +00:00
6d72acfd6d --dynamic-port-range now requires at least 12 ports 2022-01-05 16:12:28 -08:00
840ec0686e Fix broken build from bpf/tests/programs.rs (#22312)
These tests were broken due to PR #22289
2022-01-05 15:06:15 -06:00
ba0188a36d Bump version to 1.9.4 (#22304) 2022-01-05 12:02:36 -06:00
05b9a2f203 fix(rpc): recreate dead and uncleaned subscriptions (#22281) (#22294)
(cherry picked from commit c1995c647b)

Co-authored-by: Nikita <bananaelecitrus@gmail.com>
2022-01-05 17:16:12 +00:00
8578429c4d Refactor: Improve type safety and readability of transaction execution (backport #22215) (#22289)
* Refactor: Improve type safety and readability of transaction execution (#22215)

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
v1.9.3
2022-01-05 23:01:15 +08:00
87f4a1f4b6 Bank gets accounts data len delta from MessageProcessor::process_message() (#22288)
(cherry picked from commit 635337d2ff)

Co-authored-by: Brooks Prumo <brooks@solana.com>
2022-01-05 05:50:03 +00:00
17411f9b4c Add accounts_data_len to Bank (#21781) (#22285)
(cherry picked from commit eeb97fe7ce)

Co-authored-by: Brooks Prumo <brooks@solana.com>
2022-01-05 02:24:52 +00:00
fb0e5adc7e patches bug in recv_mmsg when npkts != nrecv (backport #22276) (#22280)
* removes total-size from return value of recv_mmsg

(cherry picked from commit 4b24499916)

* patches bug in recv_mmsg when npkts != nrecv

If recv_mmsg receives 2 packets where the first one is filtered out,
then it returns npkts == 1:
https://github.com/solana-labs/solana/blob/01a096adc/streamer/src/recvmmsg.rs#L104-L115

But then streamer::packet::recv_from will erroneously keep the 1st
packet and drop the 2nd one:
https://github.com/solana-labs/solana/blob/01a096adc/streamer/src/packet.rs#L34-L49

To avoid this bug, this commit updates recv_mmsg to always return total
number of received packets. If socket address cannot be correctly
obtained, it is left as the default value which is UNSPECIFIED:
https://github.com/solana-labs/solana/blob/01a096adc/sdk/src/packet.rs#L145

(cherry picked from commit 379feecae5)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-04 23:42:52 +00:00
f4ded6fb6b Updates to the address lookup table proposal (#22269) (#22270)
Co-authored-by: Justin Starry <justin@solana.com>
2022-01-04 23:38:51 +00:00
f89bf7b939 Compute accounts data len during generate_index() (#21757) (#22277)
(cherry picked from commit ec7e17787e)

Co-authored-by: Brooks Prumo <brooks@solana.com>
2022-01-04 22:44:51 +00:00
c99aed4abf Update address map proposal to improve dev experience (#21576) (#22283)
* Update address map proposal to improve dev experience

* another revision to match implementation

(cherry picked from commit 0224a8b127)

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-04 22:11:59 +00:00
edfd8c1717 Fix program log filtering (#22133) (#22151)
(cherry picked from commit c7b0917e1a)

Co-authored-by: Jack May <jack@solana.com>
2022-01-04 21:56:49 +00:00
09dbf069e8 Add test to enforce that program id account info for CPI is optional (#22069) (#22103)
* Update tests to demonstrate that program id account info for CPI is optional

* Clean up comments that say that program id account info is required

(cherry picked from commit ec7536faf6)

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-04 21:47:48 +00:00
9764d4349b Add return types to generate_index() (#21735) (#22275)
(cherry picked from commit 1528f85112)

Co-authored-by: Brooks Prumo <brooks@solana.com>
2022-01-04 18:27:40 +00:00
d84b994451 shrinks size of Packet.Meta (backport #22224) (#22273)
* removes seed and slot fields from Packet.Meta

507367e6ac
updated window-service to send shreds (as opposed to packets) to
retransmit-stage and so seed and slot fields in Packet.Meta are unused:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L27-L28

(cherry picked from commit aa9f7ed7e8)

* uses std::net::IpAddr type for Packet.Meta.addr

(cherry picked from commit 73a7741c49)

# Conflicts:
#	streamer/src/streamer.rs

* adds bitflags to Packet.Meta

Instead of a separate bool type for each flag, all the flags can be
encoded in a type-safe bitflags encoded in a single u8:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L19-L31

(cherry picked from commit 01a096adc8)

# Conflicts:
#	sdk/Cargo.toml

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-04 17:38:29 +00:00
185f52b712 Lower vote-only-mode to 400 (#22210) (#22272)
(cherry picked from commit 2486e21ffe)

Co-authored-by: sakridge <sakridge@gmail.com>
2022-01-04 15:15:16 +00:00
3b59f67562 Limit number of accounts that a transaction can lock (backport #22201) (#22263)
* Limit number of accounts that a transaction can lock (#22201)

(cherry picked from commit 2b5e00d36d)

# Conflicts:
#	accountsdb-plugin-postgres/src/postgres_client/postgres_client_transaction.rs
#	runtime/src/accounts.rs
#	runtime/src/bank.rs
#	sdk/src/feature_set.rs
#	sdk/src/transaction/error.rs
#	storage-proto/proto/transaction_by_addr.proto
#	storage-proto/src/convert.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-04 11:34:34 +00:00
7d2589e2ac Documentation typos (#22262) (#22268)
* Fix typo markdown link

* Add missing punctuation full stop

(cherry picked from commit 9665da9d0b)

Co-authored-by: glihm <dev@glihm.net>
2022-01-04 11:15:06 +00:00
77558c315d Fixed issue #22124 -- missing historical data if slot updated later. (#22193) (#22259)
* Fixed issue #22124 -- missing historical data if slot updated later.

* Fixed a couple of comments

(cherry picked from commit 5b6027bef0)

Co-authored-by: Lijun Wang <83639177+lijunwangs@users.noreply.github.com>
2022-01-04 07:18:58 +00:00
464d533da3 Flip iter operations to keep associated address/header/packets together (#22245) (#22257)
Flip iter operations to keep associated address/header/packets together

Before this change, if cast_socket_addr() returned a None for any
address/header pair, the subsequent zip() would misalign the
address/header pair and packet. So, this change zips all three together,
then does filter_map() so keep things aligned.

Additionally, compute total_size inline to avoid running through packets
a second time.

(cherry picked from commit 20b61e28b6)

Co-authored-by: steviez <steven@solana.com>
2022-01-04 07:06:06 +00:00
f8bf478fde Fix bug, add error specific timings (#22225) (#22252)
(cherry picked from commit 005592998d)

Co-authored-by: carllin <carl@solana.com>
2022-01-04 02:53:59 +00:00
35fb47d1ce removes epoch_authorized_voters from VoteTracker (backport #22192) (#22248)
* removes epoch_authorized_voters from VoteTracker (#22192)

https://github.com/solana-labs/solana/pull/22169
verifies authorized-voter early on in vote-listener pipeline; and so
VoteTracker no longer needs to maintain and check for epoch authorized
voters.

(cherry picked from commit 69d71f8f86)

# Conflicts:
#	core/src/cluster_info_vote_listener.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-04 01:51:24 +00:00