Commit Graph

15203 Commits

Author SHA1 Message Date
10c40c9c2d Remove program-runtime/src/invoke_context.rs 2022-01-13 23:59:02 +08:00
1ef48f369e downgrade individual per-program-timing to trace to reduce writes to influx
(cherry picked from commit 6614727be8)
2022-01-12 22:01:56 -06:00
05e75ae937 Don't forward packets received from TPU forwards port (backport #22078) (#22180)
* Don't forward packets received from TPU forwards port (#22078)

* resolve conflicts
2022-01-10 14:36:56 +08:00
a7aa5202ab bank: fix executor cache metrics (#22395)
(cherry picked from commit 3b4aad9df1)

Co-authored-by: Trent Nelson <trent@solana.com>
2022-01-09 05:55:08 +00:00
9aa7821277 Bump version to v1.8.13 2022-01-08 12:27:11 -07:00
fee86726f2 improve multi executor cache addition (#22381)
(cherry picked from commit 4a9f4e2505)

Co-authored-by: Jack May <jack@solana.com>
v1.8.12
2022-01-08 12:52:16 +00:00
8a470d3ae8 add excutor creation trace timings 2022-01-08 04:46:31 -07:00
8c6df1f234 bank: prime new executor cache entry use-counts (#22374)
Co-authored-by: Trent Nelson <trent@solana.com>
2022-01-08 10:14:05 +00:00
7cbfab0958 remove per program timings from blockstore processor ledger replay (#22370) (#22371)
(cherry picked from commit 813006b33b)

Co-authored-by: carllin <carl@solana.com>
2022-01-08 08:33:31 +00:00
6130466a21 Executor cache metrics (backport #22332) (#22367)
* Add helper macro for `AddAssign`ing with saturating arithmetic

* bank: Add executors cache metrics

Co-authored-by: Trent Nelson <trent@solana.com>
2022-01-08 02:19:36 +00:00
299a59f458 removes CowCachedExecutors (backport #22343) (#22362)
* removes CowCachedExecutors (#22343)

Copy-on-write semantics for cached executors can be implemented by a
simple Arc<CachedExecutors> as opposed to CowCachedExecutors:
https://github.com/solana-labs/solana/blob/f1e2598ba/runtime/src/bank.rs#L244-L247

This will also avoid the need for double locking as in:
https://github.com/solana-labs/solana/blob/f1e2598ba/runtime/src/bank.rs#L3490-L3491
https://github.com/solana-labs/solana/blob/f1e2598ba/runtime/src/bank.rs#L3525-L3526

(cherry picked from commit c2389fc209)

# Conflicts:
#	runtime/src/bank.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-07 16:18:00 +00:00
b30c726d22 Add aarch64-apple-darwin publish tarball step (#22355)
(cherry picked from commit e2aa932e97)

Co-authored-by: Michael Vines <mvines@gmail.com>
2022-01-07 10:13:46 +00:00
069bb5e32f Increase timeout of local-cluster-slow CI step 2022-01-07 15:31:28 +08:00
cad0e7f04f cache executors on failed transactions (backport #22308) (#22327)
* cache executors on failed transactions (#22308)

(cherry picked from commit 12e160269e)

# Conflicts:
#	program-runtime/src/invoke_context.rs
#	programs/bpf_loader/src/lib.rs
#	runtime/src/bank.rs

* resolve conflicts

Co-authored-by: Jack May <jack@solana.com>
2022-01-06 15:38:39 -08:00
c90fa6643e Retain executor cache counts (#22322) (#22340)
(cherry picked from commit f1e2598baa)

Co-authored-by: Jack May <jack@solana.com>
2022-01-06 19:11:05 +00:00
54db774203 Split up local cluster tests into separate CI steps (backport #22295) (#22302)
* Split up local cluster tests into separate CI steps (#22295)

* Split up local cluster tests into separate CI steps

* Update buildkite-pipeline.sh

(cherry picked from commit 0e1afcbb26)

# Conflicts:
#	local-cluster/tests/local_cluster.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-06 15:26:56 +00:00
57b5ce5731 [ledger-tool]compare_blocks (#22229) (#22329)
* 1.made load_credentials accept credential path as a parameter. 2.partial implement bigtable comparasion function

* finding missing blocks in bigtables in a specified range

* refactor compare-blocks,add unit test for missing_blocks and fmt

* compare-block fix last block bug

* refactor compare-block and improve wording

* Update ledger-tool/src/bigtable.rs

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* update compare-block command-line description

* style:improve wording/naming/code style

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
(cherry picked from commit d9220652ad)

Co-authored-by: pieceofr <komimi.p@gmail.com>
2022-01-06 08:47:08 +00:00
e0d933f940 perf: skip updating already cached executors if unmodified (backport #22300) (#22315)
* Skip updating already cached executors if unmodified

(cherry picked from commit f2ed6f09ee)

# Conflicts:
#	program-runtime/src/invoke_context.rs
#	runtime/src/bank.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-06 02:04:55 +00:00
e2559f20df fix(rpc): recreate dead and uncleaned subscriptions (#22281) (#22293)
(cherry picked from commit c1995c647b)

Co-authored-by: Nikita <bananaelecitrus@gmail.com>
2022-01-05 10:23:34 +00:00
7b9ca3e9d9 Flip iter operations to keep associated address/header/packets together (backport #22245) (#22256)
* Flip iter operations to keep associated address/header/packets together (#22245)

Flip iter operations to keep associated address/header/packets together

Before this change, if cast_socket_addr() returned a None for any
address/header pair, the subsequent zip() would misalign the
address/header pair and packet. So, this change zips all three together,
then does filter_map() so keep things aligned.

Additionally, compute total_size inline to avoid running through packets
a second time.

(cherry picked from commit 20b61e28b6)

# Conflicts:
#	streamer/src/recvmmsg.rs

* removes backport merge conflicts

Co-authored-by: steviez <steven@solana.com>
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-04 16:31:59 +00:00
f0de9b43be Lower vote-only-mode to 400 (#22210) (#22271)
(cherry picked from commit 2486e21ffe)

Co-authored-by: sakridge <sakridge@gmail.com>
2022-01-04 14:17:31 +00:00
6d7b64b140 Fix bug, add error specific timings (#22225) (#22251)
(cherry picked from commit 005592998d)

Co-authored-by: carllin <carl@solana.com>
2022-01-04 09:23:49 +00:00
f506851ca1 removes epoch_authorized_voters from VoteTracker (backport #22192) (#22250)
* removes epoch_authorized_voters from VoteTracker (#22192)

https://github.com/solana-labs/solana/pull/22169
verifies authorized-voter early on in vote-listener pipeline; and so
VoteTracker no longer needs to maintain and check for epoch authorized
voters.

(cherry picked from commit 69d71f8f86)

# Conflicts:
#	core/src/cluster_info_vote_listener.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-04 02:02:56 +00:00
b70113e201 Correctly set CI_COMMIT when Buildkite provides HEAD instead of a real commit 2022-01-03 17:40:35 -08:00
6093c7a218 Switch from arm64-apple-darwin to aarch64-apple-darwin to align with Rust's target names 2022-01-03 17:27:09 -08:00
b9777e10ee Add support for arm64-apple-darwin release/channel artifacts 2022-01-03 17:27:05 -08:00
d840d56565 Use experimential docker virtualization framework for arm64
(cherry picked from commit ed0b47c6f8)
2022-01-03 17:27:00 -08:00
afb8df0c62 Correctly set CI_OS_NAME for macOs buildkite agents 2022-01-03 17:26:54 -08:00
84d535b89e re-calibrate limit based on mainnet data (backport #21995) (#22211)
* re-calibrate limit based on mainnet data, see issue #21917

(cherry picked from commit d743c2917c)

# Conflicts:
#	runtime/src/block_cost_limits.rs

* set secp256k1 cost similar to sigverify

(cherry picked from commit a2a7e91ad6)

* removes backport merge conflicts

Co-authored-by: Tao Zhu <tao@solana.com>
2022-01-03 19:03:41 +00:00
de1a9f84ac Bump rbpf to v0.2.21 (#22216) (#22218)
(cherry picked from commit 9139be89b7)
2022-01-01 21:43:28 +00:00
22bbd91843 Count compute units even when transaction errors (backport #22182) (#22198)
* Resolve conflicts

* Return err

Co-authored-by: carllin <carl@solana.com>
2022-01-01 13:16:56 +00:00
1babd07faf Increment execution timings on errors as well (backport #22053) (#22071)
* Increment execution timings on errors as well (#22053)

(cherry picked from commit 37f6777ceb)

# Conflicts:
#	programs/bpf_loader/src/lib.rs

* Fix conflicts

Co-authored-by: carllin <carl@solana.com>
2022-01-01 13:04:34 +00:00
0592b5568f Bump local cluster timeout (#22214) 2022-01-01 06:02:38 -05:00
ce98feac4e Exit early on BigTable error (#22200) (#22208)
(cherry picked from commit 0b1b36f088)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2022-01-01 02:05:36 +00:00
a53dd611c0 uses enum for shred type (backport #21333) (#22147)
* uses enum for shred type

Current code is using u8 which does not have any type-safety and can
contain invalid values:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L167

Checks for invalid shred-types are scattered through the code:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/blockstore.rs#L849-L851
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L346-L348

The commit uses enum for shred type with #[repr(u8)]. Backward
compatibility is maintained by implementing Serialize and Deserialize
compatible with u8, and adding a test to assert that.

(cherry picked from commit 57057f8d39)

# Conflicts:
#	core/src/retransmit_stage.rs
#	gossip/src/cluster_info.rs
#	ledger/Cargo.toml
#	ledger/src/blockstore.rs
#	ledger/src/shred.rs

* changes Blockstore::is_shred_duplicate arg type to ShredType

(cherry picked from commit 48dfdfb4d5)

# Conflicts:
#	ledger/src/blockstore.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-31 16:47:45 +00:00
ec970f9d69 checks for authorized voter early on in the vote-listener pipeline (backport #22169) (#22188)
* checks for authorized voter early on in the vote-listener pipeline (#22169)

Before votes are verified that they are signed by the authorized voter,
they might be dropped in verified-vote-packets code. If there are
enough many spam votes from unauthorized voters, this may potentially
drop valid votes but keep the false ones.
https://github.com/solana-labs/solana/blob/57986f982/core/src/verified_vote_packets.rs#L165-L168

(cherry picked from commit c0c6038654)

# Conflicts:
#	core/src/cluster_info_vote_listener.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-30 17:52:36 +00:00
7793a11b65 Bump solana_rbpf to version v0.2.20 (#22166) 2021-12-30 17:34:11 +01:00
4deac1daa4 Update install/src/command.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
(cherry picked from commit 29edb130cc)
2021-12-30 07:30:38 -08:00
4c36a93665 Add connect timeout and change overall timeout to None
(cherry picked from commit 3c1416091e)
2021-12-30 07:30:38 -08:00
052309227f fix: Installer increase download req timeout from 30 seconds to 6 minutes
(cherry picked from commit a1912f8400)
2021-12-30 07:30:38 -08:00
3b895104f3 ClusterInfoVoteListener send only missing votes to BankingStage (backport #20873) (#22135)
* ClusterInfoVoteListener send only missing votes to BankingStage (#20873)

(cherry picked from commit b30c94ce55)

# Conflicts:
#	core/src/cluster_info_vote_listener.rs
#	core/src/verified_vote_packets.rs
#	gossip/src/cluster_info.rs
#	local-cluster/tests/local_cluster.rs
#	runtime/src/bank.rs

* Resolve conflicts

* Remove Select

* Fixup tests

Co-authored-by: carllin <carl@solana.com>
2021-12-30 06:38:49 +00:00
24b0fc8927 get_signatures_for_address does not correctly account for result sets that span local and Bigtable sources (backport #22115) (#22167)
* get_signatures_for_address does not correctly account for result sets that span local and Bigtable sources (#22115)

* get_signatures_for_address does not correctly account for result sets that span Blockstore and Bigtable.

This causes Bigtable to return `RowNotFound` until the new tx is uploaded.

Check that `before` exists in Bigtable, and if not, set it to `None` to return the full data set.

References #21442
Closes #22110

* Differentiate between before sig not found and no newer signatures

* Dedupe bigtable results to account for potential upload race

Co-authored-by: Tyera Eulberg <tyera@solana.com>
(cherry picked from commit bac6821e19)

# Conflicts:
#	ledger/src/blockstore.rs

* Fix conflicts

Co-authored-by: Omar Kilani <omar.kilani@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-12-29 13:03:32 -07:00
271ae3c2fc Ignore spl downstream build (#22173) 2021-12-29 12:44:46 -07:00
a305fa0472 implements copy-on-write for vote-accounts (backport #19362) #22139 (#22139)
Bank::vote_accounts redundantly clones vote-accounts HashMap even though
an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/95c998a19/runtime/src/bank.rs#L5174-L5186

This commit implements copy-on-write semantics for vote-accounts by
wrapping the underlying HashMap in Arc<...>.

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-28 21:18:37 +00:00
262b157d21 renames solana_runtime::vote_account::VoteAccount and makes it private (backport #19153) (#22141)
* makes solana_runtime::vote_account::VoteAccount private

VoteAccount is an implementation detail, and should not be public.
Only ArcVoteAccount is the public type.

(cherry picked from commit 1403eaeefc)

# Conflicts:
#	runtime/src/vote_account.rs

* renames solana_runtime::vote_account::VoteAccount

Rename:
  VoteAccount    -> VoteAccountInner  # the private type
  ArcVoteAccount -> VoteAccount       # the public type
(cherry picked from commit 00e5e12906)

# Conflicts:
#	core/src/progress_map.rs
#	ledger/src/blockstore_processor.rs
#	ledger/src/staking_utils.rs
#	runtime/src/bank.rs
#	runtime/src/epoch_stakes.rs
#	runtime/src/serde_snapshot/tests.rs
#	runtime/src/stakes.rs
#	runtime/src/vote_account.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-28 18:26:23 +00:00
bfb02029bf adds methods to obtain data/coding shreds indices from ErasureMeta (backport #21254) (#22132)
* marks ErasureMeta.size as unused

ErasureMeta.size is unused and redundant:
https://github.com/solana-labs/solana/blob/150951333/ledger/src/blockstore_meta.rs#L57-L58

(cherry picked from commit 7e98b77b33)

* removes Default implementaion of ErasureConfig

The Default implementation of ErasureConfig is off and never useful:
https://github.com/solana-labs/solana/blob/150951333/ledger/src/erasure.rs#L49-L68

(cherry picked from commit d25d9be555)

* adds methods to obtain data/coding shreds indices from ErasureMeta

(cherry picked from commit 3fc858eb60)

# Conflicts:
#	ledger/src/blockstore_meta.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-28 01:15:09 +00:00
039244417e removes redundant args from Shredder::try_recovery (backport #21226) (#22126)
* removes redundant args from Shredder::try_recovery (#21226)

Shredder::try_recovery is already taking a Vec<Shred> as an argument. All the
other arguments are embedded in the shreds, and are so redundant.

(cherry picked from commit 5fb0ab9d00)

# Conflicts:
#	ledger/src/shred.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-27 21:02:33 +00:00
1e1f383970 rewrites Shredder::try_recovery (backport #21082) (#22125)
* rewrites Shredder::try_recovery (#21082)

For every missing data-shred, Shredder::try_recovery calls into
new_empty_data_shred which does a redundant serialization into payload
buffer which is then immediately overwritten by the erasure recovery:
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L372-L417

Additionally, the implementation is unnecessary complex hindering
upcoming changes to erasure coding generation
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L814-L938

The commit simplifies the Shredder::try_recovery implementation.

(cherry picked from commit 11a53de0e3)

# Conflicts:
#	ledger/src/erasure.rs
#	ledger/src/shred.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-27 16:42:22 +00:00
891b0a5152 optimizes and simplifies SlotMeta::completed_data_indexes ops (#21059) (#22100)
SlotMeta::completed_data_indexes is defined as a Vec<u32>:
https://github.com/solana-labs/solana/blob/a8d78e89d/ledger/src/blockstore_meta.rs#L31-L32

which results in inefficient updates:
https://github.com/solana-labs/solana/blob/a8d78e89d/ledger/src/blockstore.rs#L3245-L3326

This commit changes the type to BTreeSet<u32> for efficient and simpler
updates and lookups.

The change should be backward compatible because Vec<T> and BTreeSet<T>
are both serialized as seq:
https://github.com/serde-rs/serde/blob/ce0844b9e/serde/src/ser/impls.rs#L207-L208
https://github.com/serde-rs/serde/blob/ce0844b9e/serde/src/ser/impls.rs#L216-L217

(cherry picked from commit 484ead01ed)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-26 17:22:22 +00:00
685e40cbf2 validator: add contact-info query to admin port 2021-12-23 20:23:48 +00:00