Commit Graph

15190 Commits

Author SHA1 Message Date
cad0e7f04f cache executors on failed transactions (backport #22308) (#22327)
* cache executors on failed transactions (#22308)

(cherry picked from commit 12e160269e)

# Conflicts:
#	program-runtime/src/invoke_context.rs
#	programs/bpf_loader/src/lib.rs
#	runtime/src/bank.rs

* resolve conflicts

Co-authored-by: Jack May <jack@solana.com>
2022-01-06 15:38:39 -08:00
c90fa6643e Retain executor cache counts (#22322) (#22340)
(cherry picked from commit f1e2598baa)

Co-authored-by: Jack May <jack@solana.com>
2022-01-06 19:11:05 +00:00
54db774203 Split up local cluster tests into separate CI steps (backport #22295) (#22302)
* Split up local cluster tests into separate CI steps (#22295)

* Split up local cluster tests into separate CI steps

* Update buildkite-pipeline.sh

(cherry picked from commit 0e1afcbb26)

# Conflicts:
#	local-cluster/tests/local_cluster.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-06 15:26:56 +00:00
57b5ce5731 [ledger-tool]compare_blocks (#22229) (#22329)
* 1.made load_credentials accept credential path as a parameter. 2.partial implement bigtable comparasion function

* finding missing blocks in bigtables in a specified range

* refactor compare-blocks,add unit test for missing_blocks and fmt

* compare-block fix last block bug

* refactor compare-block and improve wording

* Update ledger-tool/src/bigtable.rs

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* update compare-block command-line description

* style:improve wording/naming/code style

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
(cherry picked from commit d9220652ad)

Co-authored-by: pieceofr <komimi.p@gmail.com>
2022-01-06 08:47:08 +00:00
e0d933f940 perf: skip updating already cached executors if unmodified (backport #22300) (#22315)
* Skip updating already cached executors if unmodified

(cherry picked from commit f2ed6f09ee)

# Conflicts:
#	program-runtime/src/invoke_context.rs
#	runtime/src/bank.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2022-01-06 02:04:55 +00:00
e2559f20df fix(rpc): recreate dead and uncleaned subscriptions (#22281) (#22293)
(cherry picked from commit c1995c647b)

Co-authored-by: Nikita <bananaelecitrus@gmail.com>
2022-01-05 10:23:34 +00:00
7b9ca3e9d9 Flip iter operations to keep associated address/header/packets together (backport #22245) (#22256)
* Flip iter operations to keep associated address/header/packets together (#22245)

Flip iter operations to keep associated address/header/packets together

Before this change, if cast_socket_addr() returned a None for any
address/header pair, the subsequent zip() would misalign the
address/header pair and packet. So, this change zips all three together,
then does filter_map() so keep things aligned.

Additionally, compute total_size inline to avoid running through packets
a second time.

(cherry picked from commit 20b61e28b6)

# Conflicts:
#	streamer/src/recvmmsg.rs

* removes backport merge conflicts

Co-authored-by: steviez <steven@solana.com>
Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-04 16:31:59 +00:00
f0de9b43be Lower vote-only-mode to 400 (#22210) (#22271)
(cherry picked from commit 2486e21ffe)

Co-authored-by: sakridge <sakridge@gmail.com>
2022-01-04 14:17:31 +00:00
6d7b64b140 Fix bug, add error specific timings (#22225) (#22251)
(cherry picked from commit 005592998d)

Co-authored-by: carllin <carl@solana.com>
2022-01-04 09:23:49 +00:00
f506851ca1 removes epoch_authorized_voters from VoteTracker (backport #22192) (#22250)
* removes epoch_authorized_voters from VoteTracker (#22192)

https://github.com/solana-labs/solana/pull/22169
verifies authorized-voter early on in vote-listener pipeline; and so
VoteTracker no longer needs to maintain and check for epoch authorized
voters.

(cherry picked from commit 69d71f8f86)

# Conflicts:
#	core/src/cluster_info_vote_listener.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-04 02:02:56 +00:00
b70113e201 Correctly set CI_COMMIT when Buildkite provides HEAD instead of a real commit 2022-01-03 17:40:35 -08:00
6093c7a218 Switch from arm64-apple-darwin to aarch64-apple-darwin to align with Rust's target names 2022-01-03 17:27:09 -08:00
b9777e10ee Add support for arm64-apple-darwin release/channel artifacts 2022-01-03 17:27:05 -08:00
d840d56565 Use experimential docker virtualization framework for arm64
(cherry picked from commit ed0b47c6f8)
2022-01-03 17:27:00 -08:00
afb8df0c62 Correctly set CI_OS_NAME for macOs buildkite agents 2022-01-03 17:26:54 -08:00
84d535b89e re-calibrate limit based on mainnet data (backport #21995) (#22211)
* re-calibrate limit based on mainnet data, see issue #21917

(cherry picked from commit d743c2917c)

# Conflicts:
#	runtime/src/block_cost_limits.rs

* set secp256k1 cost similar to sigverify

(cherry picked from commit a2a7e91ad6)

* removes backport merge conflicts

Co-authored-by: Tao Zhu <tao@solana.com>
2022-01-03 19:03:41 +00:00
de1a9f84ac Bump rbpf to v0.2.21 (#22216) (#22218)
(cherry picked from commit 9139be89b7)
2022-01-01 21:43:28 +00:00
22bbd91843 Count compute units even when transaction errors (backport #22182) (#22198)
* Resolve conflicts

* Return err

Co-authored-by: carllin <carl@solana.com>
2022-01-01 13:16:56 +00:00
1babd07faf Increment execution timings on errors as well (backport #22053) (#22071)
* Increment execution timings on errors as well (#22053)

(cherry picked from commit 37f6777ceb)

# Conflicts:
#	programs/bpf_loader/src/lib.rs

* Fix conflicts

Co-authored-by: carllin <carl@solana.com>
2022-01-01 13:04:34 +00:00
0592b5568f Bump local cluster timeout (#22214) 2022-01-01 06:02:38 -05:00
ce98feac4e Exit early on BigTable error (#22200) (#22208)
(cherry picked from commit 0b1b36f088)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2022-01-01 02:05:36 +00:00
a53dd611c0 uses enum for shred type (backport #21333) (#22147)
* uses enum for shred type

Current code is using u8 which does not have any type-safety and can
contain invalid values:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L167

Checks for invalid shred-types are scattered through the code:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/blockstore.rs#L849-L851
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L346-L348

The commit uses enum for shred type with #[repr(u8)]. Backward
compatibility is maintained by implementing Serialize and Deserialize
compatible with u8, and adding a test to assert that.

(cherry picked from commit 57057f8d39)

# Conflicts:
#	core/src/retransmit_stage.rs
#	gossip/src/cluster_info.rs
#	ledger/Cargo.toml
#	ledger/src/blockstore.rs
#	ledger/src/shred.rs

* changes Blockstore::is_shred_duplicate arg type to ShredType

(cherry picked from commit 48dfdfb4d5)

# Conflicts:
#	ledger/src/blockstore.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-31 16:47:45 +00:00
ec970f9d69 checks for authorized voter early on in the vote-listener pipeline (backport #22169) (#22188)
* checks for authorized voter early on in the vote-listener pipeline (#22169)

Before votes are verified that they are signed by the authorized voter,
they might be dropped in verified-vote-packets code. If there are
enough many spam votes from unauthorized voters, this may potentially
drop valid votes but keep the false ones.
https://github.com/solana-labs/solana/blob/57986f982/core/src/verified_vote_packets.rs#L165-L168

(cherry picked from commit c0c6038654)

# Conflicts:
#	core/src/cluster_info_vote_listener.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-30 17:52:36 +00:00
7793a11b65 Bump solana_rbpf to version v0.2.20 (#22166) 2021-12-30 17:34:11 +01:00
4deac1daa4 Update install/src/command.rs
Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
(cherry picked from commit 29edb130cc)
2021-12-30 07:30:38 -08:00
4c36a93665 Add connect timeout and change overall timeout to None
(cherry picked from commit 3c1416091e)
2021-12-30 07:30:38 -08:00
052309227f fix: Installer increase download req timeout from 30 seconds to 6 minutes
(cherry picked from commit a1912f8400)
2021-12-30 07:30:38 -08:00
3b895104f3 ClusterInfoVoteListener send only missing votes to BankingStage (backport #20873) (#22135)
* ClusterInfoVoteListener send only missing votes to BankingStage (#20873)

(cherry picked from commit b30c94ce55)

# Conflicts:
#	core/src/cluster_info_vote_listener.rs
#	core/src/verified_vote_packets.rs
#	gossip/src/cluster_info.rs
#	local-cluster/tests/local_cluster.rs
#	runtime/src/bank.rs

* Resolve conflicts

* Remove Select

* Fixup tests

Co-authored-by: carllin <carl@solana.com>
2021-12-30 06:38:49 +00:00
24b0fc8927 get_signatures_for_address does not correctly account for result sets that span local and Bigtable sources (backport #22115) (#22167)
* get_signatures_for_address does not correctly account for result sets that span local and Bigtable sources (#22115)

* get_signatures_for_address does not correctly account for result sets that span Blockstore and Bigtable.

This causes Bigtable to return `RowNotFound` until the new tx is uploaded.

Check that `before` exists in Bigtable, and if not, set it to `None` to return the full data set.

References #21442
Closes #22110

* Differentiate between before sig not found and no newer signatures

* Dedupe bigtable results to account for potential upload race

Co-authored-by: Tyera Eulberg <tyera@solana.com>
(cherry picked from commit bac6821e19)

# Conflicts:
#	ledger/src/blockstore.rs

* Fix conflicts

Co-authored-by: Omar Kilani <omar.kilani@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-12-29 13:03:32 -07:00
271ae3c2fc Ignore spl downstream build (#22173) 2021-12-29 12:44:46 -07:00
a305fa0472 implements copy-on-write for vote-accounts (backport #19362) #22139 (#22139)
Bank::vote_accounts redundantly clones vote-accounts HashMap even though
an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/95c998a19/runtime/src/bank.rs#L5174-L5186

This commit implements copy-on-write semantics for vote-accounts by
wrapping the underlying HashMap in Arc<...>.

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-28 21:18:37 +00:00
262b157d21 renames solana_runtime::vote_account::VoteAccount and makes it private (backport #19153) (#22141)
* makes solana_runtime::vote_account::VoteAccount private

VoteAccount is an implementation detail, and should not be public.
Only ArcVoteAccount is the public type.

(cherry picked from commit 1403eaeefc)

# Conflicts:
#	runtime/src/vote_account.rs

* renames solana_runtime::vote_account::VoteAccount

Rename:
  VoteAccount    -> VoteAccountInner  # the private type
  ArcVoteAccount -> VoteAccount       # the public type
(cherry picked from commit 00e5e12906)

# Conflicts:
#	core/src/progress_map.rs
#	ledger/src/blockstore_processor.rs
#	ledger/src/staking_utils.rs
#	runtime/src/bank.rs
#	runtime/src/epoch_stakes.rs
#	runtime/src/serde_snapshot/tests.rs
#	runtime/src/stakes.rs
#	runtime/src/vote_account.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-28 18:26:23 +00:00
bfb02029bf adds methods to obtain data/coding shreds indices from ErasureMeta (backport #21254) (#22132)
* marks ErasureMeta.size as unused

ErasureMeta.size is unused and redundant:
https://github.com/solana-labs/solana/blob/150951333/ledger/src/blockstore_meta.rs#L57-L58

(cherry picked from commit 7e98b77b33)

* removes Default implementaion of ErasureConfig

The Default implementation of ErasureConfig is off and never useful:
https://github.com/solana-labs/solana/blob/150951333/ledger/src/erasure.rs#L49-L68

(cherry picked from commit d25d9be555)

* adds methods to obtain data/coding shreds indices from ErasureMeta

(cherry picked from commit 3fc858eb60)

# Conflicts:
#	ledger/src/blockstore_meta.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-28 01:15:09 +00:00
039244417e removes redundant args from Shredder::try_recovery (backport #21226) (#22126)
* removes redundant args from Shredder::try_recovery (#21226)

Shredder::try_recovery is already taking a Vec<Shred> as an argument. All the
other arguments are embedded in the shreds, and are so redundant.

(cherry picked from commit 5fb0ab9d00)

# Conflicts:
#	ledger/src/shred.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-27 21:02:33 +00:00
1e1f383970 rewrites Shredder::try_recovery (backport #21082) (#22125)
* rewrites Shredder::try_recovery (#21082)

For every missing data-shred, Shredder::try_recovery calls into
new_empty_data_shred which does a redundant serialization into payload
buffer which is then immediately overwritten by the erasure recovery:
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L372-L417

Additionally, the implementation is unnecessary complex hindering
upcoming changes to erasure coding generation
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L814-L938

The commit simplifies the Shredder::try_recovery implementation.

(cherry picked from commit 11a53de0e3)

# Conflicts:
#	ledger/src/erasure.rs
#	ledger/src/shred.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-27 16:42:22 +00:00
891b0a5152 optimizes and simplifies SlotMeta::completed_data_indexes ops (#21059) (#22100)
SlotMeta::completed_data_indexes is defined as a Vec<u32>:
https://github.com/solana-labs/solana/blob/a8d78e89d/ledger/src/blockstore_meta.rs#L31-L32

which results in inefficient updates:
https://github.com/solana-labs/solana/blob/a8d78e89d/ledger/src/blockstore.rs#L3245-L3326

This commit changes the type to BTreeSet<u32> for efficient and simpler
updates and lookups.

The change should be backward compatible because Vec<T> and BTreeSet<T>
are both serialized as seq:
https://github.com/serde-rs/serde/blob/ce0844b9e/serde/src/ser/impls.rs#L207-L208
https://github.com/serde-rs/serde/blob/ce0844b9e/serde/src/ser/impls.rs#L216-L217

(cherry picked from commit 484ead01ed)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-26 17:22:22 +00:00
685e40cbf2 validator: add contact-info query to admin port 2021-12-23 20:23:48 +00:00
17d698d20a report mem stats (#21258) (#22066)
(cherry picked from commit f8dcb2f38b)

# Conflicts:
#	Cargo.lock
#	core/Cargo.toml
#	core/src/system_monitor_service.rs
#	runtime/src/bucket_map_holder_stats.rs

Co-authored-by: Jeff Washington (jwash) <75863576+jeffwashington@users.noreply.github.com>
2021-12-23 18:16:56 +00:00
cf34ae7d6f chore: bump lru from 0.7.0 to 0.7.1 (backport #22018) (#22079)
* chore: bump lru from 0.7.0 to 0.7.1 (#22018)

Bumps [lru](https://github.com/jeromefroe/lru-rs) from 0.7.0 to 0.7.1.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases)
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jeromefroe/lru-rs/compare/0.7.0...0.7.1)

---
updated-dependencies:
- dependency-name: lru
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 69d0b08dd8)

# Conflicts:
#	Cargo.lock
#	core/Cargo.toml
#	gossip/Cargo.toml

* Fix conflicts

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-12-23 02:57:40 +00:00
acd03fc29b Check file size of snapshot_version when unarchiving snapshot (backport #21925) (#21987) 2021-12-20 14:22:34 -06:00
ddef156305 new net-stats require a new table (backport #21996) (#21999)
* new net-stats require a new table (#21996)

(cherry picked from commit 3fe942ab30)

# Conflicts:
#	core/src/system_monitor_service.rs

* merge

Co-authored-by: Jeff Biseda <jbiseda@gmail.com>
2021-12-18 10:29:29 +00:00
33cd3a161e Add json support for feature sets; also print output after feature list (#21905) (#21974)
* Add json support for feature sets; also print output after feature list

* Move stringifying into Display implementation

(cherry picked from commit dcd2854829)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-12-17 06:48:18 +00:00
31a0906410 Revert "Revert "Add Accountsdb plugin documentations (#21746) (#21798)""
This reverts commit 09c68ce696.
2021-12-16 19:28:48 -07:00
a0669af872 Revert "Revert "Rename Packets to PacketBatch (backport #21794) (#21804)""
This reverts commit 13d40d6a66.
2021-12-16 19:28:48 -07:00
0f6f0545d1 Revert "Revert "resolve conflicts (#21795)""
This reverts commit b73d23d50a.
2021-12-16 19:28:48 -07:00
89524d7b61 Revert "Revert "Nits in message-processor (backport #21755) (#21761)""
This reverts commit d3e92f4250.
2021-12-16 19:28:48 -07:00
4a66832fb0 Revert "Revert "Add more reporting for invalid stake cache members and prune them (backport #21654) (#21740)""
This reverts commit 8028f218a4.
2021-12-16 19:28:48 -07:00
df40ede6ea Revert "Revert "simulateTransaction now returns the correct error code if accounts are provided as input (#21715)""
This reverts commit 6deb0a9f5d.
2021-12-16 19:28:48 -07:00
93a8fd6a2b Revert "Revert "SDK: Add stdlib.h include to pull in abort() (backport #21700) (#21704)""
This reverts commit 402d72bc48.
2021-12-16 19:28:48 -07:00
985fae2dcf Revert "Revert "Ensure we have keys to activate these features (#21669) (#21673)""
This reverts commit a0b73d5658.
2021-12-16 19:28:48 -07:00