661 Commits

Author SHA1 Message Date
mergify[bot]
aabcdcf8fb Add execute metrics (backport #22296) (#22334)
* consolidate execute timings to a common module

* Add execute metrics

* Add metrics for executor creation

* Use saturating_add_assign macro

Co-authored-by: Trent Nelson <trent@solana.com>
Co-authored-by: Carl Lin <carl@solana.com>
2022-01-18 19:18:24 +00:00
mergify[bot]
d52dd97ad1 log internals (#22493) (#22496)
(cherry picked from commit eca8d21249)

Co-authored-by: carllin <carl@solana.com>
2022-01-15 05:22:08 +00:00
mergify[bot]
61f88e04d2 adds methods to obtain shreds' erasure coded block and index (#21325) (#22508)
(cherry picked from commit 7da2df7d8c)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-14 17:43:16 +00:00
mergify[bot]
40ef11ec86 implements copy-on-write for staked-nodes (backport #19090) (#22507)
* implements copy-on-write for staked-nodes (#19090)

Bank::staked_nodes and Bank::epoch_staked_nodes redundantly clone
staked-nodes HashMap even though an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/a9014cece/runtime/src/vote_account.rs#L77

This commit implements copy-on-write semantics for staked-nodes by
wrapping the underlying HashMap in Arc<...>.

(cherry picked from commit f302774cf7)

# Conflicts:
#	runtime/src/bank.rs
#	runtime/src/stakes.rs
#	runtime/src/vote_account.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2022-01-14 15:26:11 +00:00
Trent Nelson
9aa7821277 Bump version to v1.8.13 2022-01-08 12:27:11 -07:00
mergify[bot]
7cbfab0958 remove per program timings from blockstore processor ledger replay (#22370) (#22371)
(cherry picked from commit 813006b33b)

Co-authored-by: carllin <carl@solana.com>
2022-01-08 08:33:31 +00:00
mergify[bot]
a53dd611c0 uses enum for shred type (backport #21333) (#22147)
* uses enum for shred type

Current code is using u8 which does not have any type-safety and can
contain invalid values:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L167

Checks for invalid shred-types are scattered through the code:
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/blockstore.rs#L849-L851
https://github.com/solana-labs/solana/blob/66fa062f1/ledger/src/shred.rs#L346-L348

The commit uses enum for shred type with #[repr(u8)]. Backward
compatibility is maintained by implementing Serialize and Deserialize
compatible with u8, and adding a test to assert that.

(cherry picked from commit 57057f8d39)

# Conflicts:
#	core/src/retransmit_stage.rs
#	gossip/src/cluster_info.rs
#	ledger/Cargo.toml
#	ledger/src/blockstore.rs
#	ledger/src/shred.rs

* changes Blockstore::is_shred_duplicate arg type to ShredType

(cherry picked from commit 48dfdfb4d5)

# Conflicts:
#	ledger/src/blockstore.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-31 16:47:45 +00:00
mergify[bot]
24b0fc8927 get_signatures_for_address does not correctly account for result sets that span local and Bigtable sources (backport #22115) (#22167)
* get_signatures_for_address does not correctly account for result sets that span local and Bigtable sources (#22115)

* get_signatures_for_address does not correctly account for result sets that span Blockstore and Bigtable.

This causes Bigtable to return `RowNotFound` until the new tx is uploaded.

Check that `before` exists in Bigtable, and if not, set it to `None` to return the full data set.

References #21442
Closes #22110

* Differentiate between before sig not found and no newer signatures

* Dedupe bigtable results to account for potential upload race

Co-authored-by: Tyera Eulberg <tyera@solana.com>
(cherry picked from commit bac6821e19)

# Conflicts:
#	ledger/src/blockstore.rs

* Fix conflicts

Co-authored-by: Omar Kilani <omar.kilani@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-12-29 13:03:32 -07:00
mergify[bot]
a305fa0472 implements copy-on-write for vote-accounts (backport #19362) #22139 (#22139)
Bank::vote_accounts redundantly clones vote-accounts HashMap even though
an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/95c998a19/runtime/src/bank.rs#L5174-L5186

This commit implements copy-on-write semantics for vote-accounts by
wrapping the underlying HashMap in Arc<...>.

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-28 21:18:37 +00:00
mergify[bot]
262b157d21 renames solana_runtime::vote_account::VoteAccount and makes it private (backport #19153) (#22141)
* makes solana_runtime::vote_account::VoteAccount private

VoteAccount is an implementation detail, and should not be public.
Only ArcVoteAccount is the public type.

(cherry picked from commit 1403eaeefc)

# Conflicts:
#	runtime/src/vote_account.rs

* renames solana_runtime::vote_account::VoteAccount

Rename:
  VoteAccount    -> VoteAccountInner  # the private type
  ArcVoteAccount -> VoteAccount       # the public type
(cherry picked from commit 00e5e12906)

# Conflicts:
#	core/src/progress_map.rs
#	ledger/src/blockstore_processor.rs
#	ledger/src/staking_utils.rs
#	runtime/src/bank.rs
#	runtime/src/epoch_stakes.rs
#	runtime/src/serde_snapshot/tests.rs
#	runtime/src/stakes.rs
#	runtime/src/vote_account.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-28 18:26:23 +00:00
mergify[bot]
bfb02029bf adds methods to obtain data/coding shreds indices from ErasureMeta (backport #21254) (#22132)
* marks ErasureMeta.size as unused

ErasureMeta.size is unused and redundant:
https://github.com/solana-labs/solana/blob/150951333/ledger/src/blockstore_meta.rs#L57-L58

(cherry picked from commit 7e98b77b33)

* removes Default implementaion of ErasureConfig

The Default implementation of ErasureConfig is off and never useful:
https://github.com/solana-labs/solana/blob/150951333/ledger/src/erasure.rs#L49-L68

(cherry picked from commit d25d9be555)

* adds methods to obtain data/coding shreds indices from ErasureMeta

(cherry picked from commit 3fc858eb60)

# Conflicts:
#	ledger/src/blockstore_meta.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-28 01:15:09 +00:00
mergify[bot]
039244417e removes redundant args from Shredder::try_recovery (backport #21226) (#22126)
* removes redundant args from Shredder::try_recovery (#21226)

Shredder::try_recovery is already taking a Vec<Shred> as an argument. All the
other arguments are embedded in the shreds, and are so redundant.

(cherry picked from commit 5fb0ab9d00)

# Conflicts:
#	ledger/src/shred.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-27 21:02:33 +00:00
mergify[bot]
1e1f383970 rewrites Shredder::try_recovery (backport #21082) (#22125)
* rewrites Shredder::try_recovery (#21082)

For every missing data-shred, Shredder::try_recovery calls into
new_empty_data_shred which does a redundant serialization into payload
buffer which is then immediately overwritten by the erasure recovery:
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L372-L417

Additionally, the implementation is unnecessary complex hindering
upcoming changes to erasure coding generation
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L814-L938

The commit simplifies the Shredder::try_recovery implementation.

(cherry picked from commit 11a53de0e3)

# Conflicts:
#	ledger/src/erasure.rs
#	ledger/src/shred.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-27 16:42:22 +00:00
mergify[bot]
891b0a5152 optimizes and simplifies SlotMeta::completed_data_indexes ops (#21059) (#22100)
SlotMeta::completed_data_indexes is defined as a Vec<u32>:
https://github.com/solana-labs/solana/blob/a8d78e89d/ledger/src/blockstore_meta.rs#L31-L32

which results in inefficient updates:
https://github.com/solana-labs/solana/blob/a8d78e89d/ledger/src/blockstore.rs#L3245-L3326

This commit changes the type to BTreeSet<u32> for efficient and simpler
updates and lookups.

The change should be backward compatible because Vec<T> and BTreeSet<T>
are both serialized as seq:
https://github.com/serde-rs/serde/blob/ce0844b9e/serde/src/ser/impls.rs#L207-L208
https://github.com/serde-rs/serde/blob/ce0844b9e/serde/src/ser/impls.rs#L216-L217

(cherry picked from commit 484ead01ed)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-26 17:22:22 +00:00
Tyera Eulberg
a0669af872 Revert "Revert "Rename Packets to PacketBatch (backport #21794) (#21804)""
This reverts commit 13d40d6a66.
2021-12-16 19:28:48 -07:00
Tyera Eulberg
01e932c6a4 Revert "Revert "adds back position field to coding-shred-header (backport #21600) (#21619)""
This reverts commit c901227d0f.
2021-12-16 19:28:48 -07:00
Tyera Eulberg
9f53f3455a Revert "Revert "Reformat imports to a consistent style for imports""
This reverts commit d7377d4794.
2021-12-16 19:28:48 -07:00
Tyera Eulberg
58e46e107c Revert "Revert "Update builtins.rs""
This reverts commit c60314c4a1.
2021-12-16 19:28:48 -07:00
Tyera Eulberg
3c5ac9ab27 Revert "Revert "Quash M1 build warning""
This reverts commit bcacaf78e9.
2021-12-16 19:28:48 -07:00
Tyera Eulberg
957914b3b5 Revert "Revert "Use max_complete_transaction_status_slot in BigTableUploadService (#21401) (#21411)""
This reverts commit fa83e05d2c.
2021-12-16 19:28:48 -07:00
Tyera Eulberg
c3c1b820fc Revert "Revert "Add ledger tool to remove dead slot (#20810) (#20813)""
This reverts commit 7210a883eb.
2021-12-16 19:28:48 -07:00
Michael Vines
6bc52d4d47 Bump version to v1.8.12 2021-12-15 09:25:32 -08:00
Tao Zhu
4cae58bfe9 Bump version to 1.8.11 2021-12-13 12:58:29 -08:00
Trent Nelson
16abcac802 Bump version to 1.8.10 2021-12-13 12:46:23 -06:00
Trent Nelson
a5e6a188c1 Bump version to 1.8.9 2021-12-13 12:46:23 -06:00
Trent Nelson
9f44d60a39 Bump version to 1.8.8 2021-12-13 12:46:23 -06:00
Trent Nelson
9284c6fffb Bump version to 1.8.7 2021-12-13 12:46:23 -06:00
Trent Nelson
2c49ab1ab4 Bump version to 1.8.6 2021-12-13 12:46:23 -06:00
Trent Nelson
9832ac54a4 Revert "Bump version to v1.8.6 (#21329)"
This reverts commit 336ee01aae.
2021-12-13 12:46:23 -06:00
Trent Nelson
7210a883eb Revert "Add ledger tool to remove dead slot (#20810) (#20813)"
This reverts commit d1ca16e9f8.
2021-12-13 12:46:23 -06:00
Trent Nelson
fa83e05d2c Revert "Use max_complete_transaction_status_slot in BigTableUploadService (#21401) (#21411)"
This reverts commit 73d469991f.
2021-12-13 12:46:23 -06:00
Trent Nelson
bcacaf78e9 Revert "Quash M1 build warning"
This reverts commit 713b61677e.
2021-12-13 12:46:23 -06:00
Trent Nelson
c60314c4a1 Revert "Update builtins.rs"
This reverts commit a54fa45d5a.
2021-12-13 12:46:23 -06:00
Trent Nelson
dc633e3385 Revert "Bump version to v1.8.7"
This reverts commit 008139f506.
2021-12-13 12:46:23 -06:00
Trent Nelson
d7377d4794 Revert "Reformat imports to a consistent style for imports"
This reverts commit 139d15cd84.
2021-12-13 12:46:23 -06:00
Trent Nelson
3369826bd1 Revert "Bump version to v1.8.8 (#21609)"
This reverts commit b8306a99d8.
2021-12-13 12:46:23 -06:00
Trent Nelson
c901227d0f Revert "adds back position field to coding-shred-header (backport #21600) (#21619)"
This reverts commit 6d1e1287bc.
2021-12-13 12:46:23 -06:00
Trent Nelson
c0794d6fbb Revert "Bump version to v1.8.9 (#21703)"
This reverts commit 69027e3f7e.
2021-12-13 12:46:23 -06:00
Trent Nelson
400418fe2f Revert "Bump version to 1.8.10 (#21790)"
This reverts commit 73acae9b5d.
2021-12-13 12:46:23 -06:00
Trent Nelson
13d40d6a66 Revert "Rename Packets to PacketBatch (backport #21794) (#21804)"
This reverts commit 39e27b130f.
2021-12-13 12:46:23 -06:00
mergify[bot]
39e27b130f Rename Packets to PacketBatch (backport #21794) (#21804)
* Rename Packets to PacketBatch (#21794)

(cherry picked from commit 254ef3e7b6)

# Conflicts:
#	core/src/ancestor_hashes_service.rs
#	core/src/banking_stage.rs
#	core/src/cluster_info_vote_listener.rs
#	core/src/fetch_stage.rs
#	core/src/serve_repair.rs
#	core/src/shred_fetch_stage.rs
#	core/src/sigverify_stage.rs
#	core/src/verified_vote_packets.rs
#	core/src/window_service.rs
#	gossip/src/cluster_info.rs
#	ledger/src/entry.rs
#	streamer/src/streamer.rs

* resolve conflicts

Co-authored-by: Justin Starry <justin@solana.com>
2021-12-11 17:32:17 +00:00
Tao Zhu
73acae9b5d Bump version to 1.8.10 (#21790) 2021-12-10 18:44:32 -06:00
Alexander Meißner
69027e3f7e Bump version to v1.8.9 (#21703) 2021-12-08 18:33:05 +01:00
mergify[bot]
6d1e1287bc adds back position field to coding-shred-header (backport #21600) (#21619)
* adds back position field to coding-shred-header (#21600)

https://github.com/solana-labs/solana/pull/17004
removed position field from coding-shred-header because as it stands the
field is redundant and unused.
However, with the upcoming changes to erasure coding schema this field
will no longer be redundant and needs to be populated.

(cherry picked from commit cd17f63d81)

# Conflicts:
#	core/src/window_service.rs
#	ledger/src/blockstore.rs
#	ledger/src/shred.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-12-05 17:12:12 +00:00
Trent Nelson
b8306a99d8 Bump version to v1.8.8 (#21609) 2021-12-04 10:00:35 -07:00
Michael Vines
139d15cd84 Reformat imports to a consistent style for imports
rustfmt.toml configuration:
        imports_granularity = "One"
         group_imports = "One"
2021-12-03 09:41:09 -08:00
Michael Vines
008139f506 Bump version to v1.8.7 2021-12-01 11:44:18 -08:00
Michael Vines
a54fa45d5a Update builtins.rs 2021-11-30 00:55:07 -08:00
Michael Vines
713b61677e Quash M1 build warning
(cherry picked from commit e31b469f56)

# Conflicts:
#	ledger/src/builtins.rs
2021-11-30 00:55:07 -08:00
mergify[bot]
73d469991f Use max_complete_transaction_status_slot in BigTableUploadService (#21401) (#21411)
* Use max_complete_transaction_status_slot in BigTableUploadService

* Use node root to limit BigTableUploadService

(cherry picked from commit 9e043d2055)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-11-24 02:35:42 +00:00