gaspersic/solana
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
15,974 Commits 81 Branches 357 Tags
2ec74474e97e6e438b3e9de2d57bb7ded64ee84c
Commit Graph

2587 Commits

Author SHA1 Message Date
behzad nouri
50d0e830c9 unifies cluster-nodes computation & caching across turbine stages 
Broadcast-stage is using epoch_staked_nodes based on the same slot that
shreds belong to:
https://github.com/solana-labs/solana/blob/049fb0417/core/src/broadcast_stage/standard_broadcast_run.rs#L208-L228
https://github.com/solana-labs/solana/blob/0cf52e206/core/src/broadcast_stage.rs#L342-L349

But retransmit-stage is using bank-epoch of the working-bank:
https://github.com/solana-labs/solana/blob/19bd30262/core/src/retransmit_stage.rs#L272-L289

So the two are not consistent at epoch boundaries where some nodes may
have a working bank (or similarly a root bank) lagging other nodes. As a
result the node which obtains a packet may construct turbine broadcast
tree inconsistently with its parent node in the tree and so some packets
may fail to reach all nodes in the tree.
 2021-08-05 21:47:33 +00:00
behzad nouri
aa32738dd5 uses cluster-nodes cache in broadcast-stage 
* Current caching mechanism does not update cluster-nodes when the epoch
  (and so epoch staked nodes) changes:
  https://github.com/solana-labs/solana/blob/19bd30262/core/src/broadcast_stage/standard_broadcast_run.rs#L332-L344

* Additionally, the cache update has a concurrency bug in which the
  thread which does compare_and_swap may be blocked when it tries to
  obtain the write-lock on cache, while other threads will keep running
  ahead with the outdated cache (since the atomic timestamp is already
  updated).

In the new ClusterNodesCache, entries are keyed by epoch, and so if
epoch changes cluster-nodes will be recalculated. The time-to-live
eviction policy is also encapsulated and rigidly enforced.
 2021-08-05 21:47:33 +00:00
behzad nouri
30bec3921e uses cluster-nodes cache in retransmit stage 
The new cluster-nodes cache will:
  * ensure cluster-nodes are recalculated if the epoch (and so the epoch
    staked nodes) changes.
  * encapsulate time-to-live eviction policy.
 2021-08-05 21:47:33 +00:00
behzad nouri
ecc1c7957f implements cluster-nodes cache 
Cluster nodes are cached keyed by the respective epoch from which stakes
are obtained, and so if epoch changes cluster-nodes will be recomputed.

A time-to-live eviction policy is enforced to refresh entries in case
gossip contact-infos are updated.
 2021-08-05 21:47:33 +00:00
behzad nouri
44b11154ca sends slots (instead of stakes) through broadcast flow 
Current broadcast code is computing stakes for each slot before sending
them down the channel:
https://github.com/solana-labs/solana/blob/049fb0417/core/src/broadcast_stage/standard_broadcast_run.rs#L208-L228
https://github.com/solana-labs/solana/blob/0cf52e206/core/src/broadcast_stage.rs#L342-L349

Since the stakes are a function of epoch the slot belongs to (and so
does not necessarily change from one slot to another), forwarding the
slot itself would allow better caching downstream.

In addition we need to invalidate the cache if the epoch changes (which
the current code does not do), and that requires to know which slot (and
so epoch) current broadcasted shreds belong to:
https://github.com/solana-labs/solana/blob/19bd30262/core/src/broadcast_stage/standard_broadcast_run.rs#L332-L344
 2021-08-05 21:47:33 +00:00
Jeff Washington (jwash)
e368f10973 add _for_tests to new_no_wallclock_throttle (#19086) 2021-08-05 14:50:25 -05:00
Jeff Washington (jwash)
a9014ceceb Bank::default_for_tests() (#19084) 2021-08-05 11:53:29 -05:00
behzad nouri
40914de811 updates cluster-slots with root-bank instead of root-slot + bank-forks 
ClusterSlots::update is taking both root-slot and bank-forks only to
later lookup root-bank from bank-forks, which is redundant. Also
potentially by the time bank-forks is locked to obtain root-bank,
root-slot may have already changed and so be inconsistent with the
root-slot passed in as the argument.
https://github.com/solana-labs/solana/blob/6d95d679c/core/src/cluster_slots.rs#L32-L39
https://github.com/solana-labs/solana/blob/6d95d679c/core/src/cluster_slots.rs#L122
 2021-08-05 14:43:06 +00:00
behzad nouri
2fc112edcf removes unused code from cluster-slots 2021-08-05 14:43:06 +00:00
Jeff Washington (jwash)
bf16b0517c add _for_tests to setup_bank_and_vote_pubkeys (#19060) 2021-08-05 08:43:35 -05:00
Jeff Washington (jwash)
14361906ca for all tests, bank::new -> bank::new_for_tests (#19064) 2021-08-05 08:42:38 -05:00
Jeff Washington (jwash)
bde9b4de94 Bank::new -> Bank::new_for_benches (#19063) 2021-08-04 17:30:43 -05:00
Jeff Washington (jwash)
3280ae3e9f add validator option --accounts-db-skip-shrink (#19028) 
* add validator option --accounts-db-skip-shrink

* typo
 2021-08-04 17:28:33 -05:00
Jeff Washington (jwash)
1ed12a07ab introduce Bank::new_for_tests (#19062) 2021-08-04 15:06:57 -05:00
Brooks Prumo
ca14475085 Add incremental_snapshot_archive_interval_slots to SnapshotConfig (#19026) 
This commit also renames `snapshot_interval_slots` to
`full_snapshot_archive_interval_slots`, updates the comments on the
fields, and make appropriate updates where SnapshotConfig is used.
 2021-08-04 14:40:20 -05:00
Oliver
06e08c4840 move package_snapshots to AccountsPackagePre ctors (#18997) 
This PR solves #18815. Note that I had to make the snapshot prefix
constants inside `snapshot_utils.rs` public at the crate level in order
to make this work. I'm not sure whether or not introducing this
dependency is entirely good, either way the `snapshot_utils.rs` file
needs a lot of rework so things will move around, I believe this does
the work in the meantime. Any feedback will be greatly appreciated.
 2021-08-04 09:03:03 -05:00
Trent Nelson
06a7a9e544 remove superfluous collect()s 2021-08-04 07:21:55 +00:00
carllin
03353d500f Actively manage dead slots in AncestorHashesService (#18912) 2021-08-02 14:33:28 -07:00
dependabot[bot]
c16bf02448 chore: bump serde from 1.0.126 to 1.0.127 (#19010) 
* chore: bump serde from 1.0.126 to 1.0.127

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.126 to 1.0.127.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.126...v1.0.127)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-08-02 21:16:34 +00:00
behzad nouri
049fb0417f allows sendmmsg api taking owned values (as well as references) (#18999) 
Current signature of api in sendmmsg requires a slice of inner
references:
https://github.com/solana-labs/solana/blob/fe1ee4980/streamer/src/sendmmsg.rs#L130-L152

That forces the call-site to convert owned values to references even
though doing so is redundant and adds an extra level of indirection:
https://github.com/solana-labs/solana/blob/fe1ee4980/core/src/repair_service.rs#L291

This commit expands the api using AsRef and Borrow traits to allow
calling the method with owned values (as well as references like
before).
 2021-07-30 20:58:49 +00:00
dependabot[bot]
fe1ee49807 chore: bump serde_json from 1.0.65 to 1.0.66 (#18990) 
* chore: bump serde_json from 1.0.65 to 1.0.66

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.65 to 1.0.66.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.65...v1.0.66)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-07-30 10:46:47 -06:00
Tao Zhu
5d297ccf96 Cost model uses compute_unit to replace microsecond as cost unit (#18934) 
* wip - cost_update_services to log both us and cu for each instruction to determine possible ratio

* replace microsecond with compute_unit as cost unit
 2021-07-29 22:19:36 +00:00
Brooks Prumo
b05fb87f22 Add test_bank_forks_incremental_snapshot() (#18565) 
This commit builds on PR #18504 by adding a test to core/tests/snapshot.rs for Incremental Snapshots. The test adds banks to bank forks in a loop and takes both full snapshots and incremental snapshots at intervals, and validates they are rebuild-able.

For background info about Incremental Snapshots, see #17088.

Fixes #18829 and #18972
 2021-07-29 16:46:54 -05:00
Ryo Onodera
da480bdb5f Fix unstable retransmit-num_nodes (#18970) 2021-07-29 17:32:32 +00:00
dependabot[bot]
0cf52e2060 chore: bump serde_json from 1.0.64 to 1.0.65 (#18968) 
* chore: bump serde_json from 1.0.64 to 1.0.65

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.64 to 1.0.65.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.64...v1.0.65)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-07-29 16:52:02 +00:00
behzad nouri
d06dc6c8a6 shares cluster-nodes between retransmit threads (#18947) 
cluster_nodes and last_peer_update are not shared between retransmit
threads, as each thread have its own value:
https://github.com/solana-labs/solana/blob/65ccfed86/core/src/retransmit_stage.rs#L476-L477

Additionally, with shared references, this code:
https://github.com/solana-labs/solana/blob/0167daa11/core/src/retransmit_stage.rs#L315-L328
has a concurrency bug where the thread which does compare_and_swap,
updates cluster_nodes much later after other threads have run with
outdated cluster_nodes for a while. In particular, the write-lock there
may block.
 2021-07-29 16:20:15 +00:00
dependabot[bot]
1f788781dd chore: bump raptorq from 1.6.3 to 1.6.4 (#18965) 
Bumps [raptorq](https://github.com/cberner/raptorq) from 1.6.3 to 1.6.4.
- [Release notes](https://github.com/cberner/raptorq/releases)
- [Commits](https://github.com/cberner/raptorq/compare/v1.6.3...v1.6.4)

---
updated-dependencies:
- dependency-name: raptorq
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-29 08:39:05 -06:00
dependabot[bot]
5f23d8530d chore: bump lru from 0.6.5 to 0.6.6 (#18963) 
Bumps [lru](https://github.com/jeromefroe/lru-rs) from 0.6.5 to 0.6.6.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases)
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jeromefroe/lru-rs/compare/0.6.5...0.6.6)

---
updated-dependencies:
- dependency-name: lru
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-29 08:33:41 -06:00
Trent Nelson
71f6d839f9 validator: remove disused cuda config argument 2021-07-29 03:08:52 +00:00
Trent Nelson
8ed0cd0fff validator: check target CPU features earlier 2021-07-29 03:08:52 +00:00
Trent Nelson
c435f7b3e3 validator: add avx2 runtime check 2021-07-29 03:08:52 +00:00
Trent Nelson
e641f257ef test-validator: move feature check earlier in startup 2021-07-29 03:08:52 +00:00
Trent Nelson
59641623d1 Improve check for Apple M1 silicon under Rosetta 2021-07-29 03:08:52 +00:00
Jeff Biseda
9255ae334d drop outstanding_requests lock before sending repair requests (#18893) 2021-07-28 19:30:43 -07:00
sakridge
84e78316b1 Write helper for multithread update (#18808) 2021-07-29 03:16:36 +02:00
dependabot[bot]
14f0ce850d chore: bump blake3 from 0.3.8 to 1.0.0 (#18908) 
* chore: bump blake3 from 0.3.8 to 1.0.0

Bumps [blake3](https://github.com/BLAKE3-team/BLAKE3) from 0.3.8 to 1.0.0.
- [Release notes](https://github.com/BLAKE3-team/BLAKE3/releases)
- [Commits](https://github.com/BLAKE3-team/BLAKE3/compare/0.3.8...1.0.0)

---
updated-dependencies:
- dependency-name: blake3
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Make versions consistent

* Enable new feature

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
 2021-07-27 17:55:21 -06:00
Jack May
f1b9f97aef remove avx error on macos (#18923) 2021-07-27 16:34:04 -07:00
carllin
c0704d4ec9 Plumb signal from replay to ancestor hashes service (#18880) 2021-07-26 20:59:00 -07:00
Tyera Eulberg
8596db8f53 Bump jsonrpc crates and remove old tokio (#18779) 
* Bump jsonrpc crates and replace old tokio

* Bump tokio

* getBlockTime

* getBlocks

* getBlocksWithLimit, getInflationReward

* getBlock

* getFirstAvailableBlock

* getTransaction

* getSignaturesForAddress

* getSignatureStatuses

* Remove superfluous runtime
 2021-07-26 12:32:17 -06:00
carllin
1ee64afb12 Introduce AncestorHashesService (#18812) 2021-07-23 16:54:47 -07:00
behzad nouri
d2d5f36a3c adds validator flag to allow private ip addresses (#18850) 2021-07-23 15:25:03 +00:00
Ryo Onodera
611af87fdb Really start caching by fixing swapped CAS... (#18842) 2021-07-23 10:17:19 +09:00
Brooks Prumo
d1debcd971 Add incremental snapshot utils (#18504) 
This commit adds high-level functions for creating and loading-from
incremental snapshots, plus all low-level functions required to perform
those tasks.  This commit **does not** add taking incremental snapshots
as part of a running validator, nor starting up a node with an
incremental snapshot; just laying ground work.

Additionally, `snapshot_utils` and `serde_snapshot` have been
refactored to use a common code paths for the different snapshots.

Also of note, some renaming has happened:
  1. Snapshots are now either `full_` or `incremental_` throughout the
     codebase.  If not specified, the code applies to both.
  2. Bank snapshots now are called "bank snapshots"
     (before they were called "slot snapshots", "bank snapshots", or
      just "snapshots").  The one exception is within `Bank`, where they
     are still just "snapshots", because they are already "bank
     snapshots".
  3. Snapshot archives now have `_archive` in the code.  This
     should clear up an ambiguity between bank snapshots and snapshot
     archives.
 2021-07-22 14:40:37 -05:00
behzad nouri
7d56fa8363 sends packets in batches from sigverify-stage (#18446) 
sigverify-stage is breaking batches to single-item vectors before
sending them down the channel:
https://github.com/solana-labs/solana/blob/d451363dc/core/src/sigverify_stage.rs#L88-L92

Also simplifying window-service code, reducing number of nested branches.
 2021-07-22 14:49:21 +00:00
Michael Vines
61865c0ee0 solana-validator set-identity now loads the tower file for the new identity 2021-07-21 22:22:08 -07:00
carllin
588c0464b8 Add sampling logic and DuplicateSlotRepairStatus module (#18721) 2021-07-21 11:15:08 -07:00
behzad nouri
bbd22f06f4 implements generic lookups into gossip crds table (#18765) 
This commit adds CrdsEntry trait which allows generic lookups into crds
table. For example to get ContactInfo or LowestSlot associated with a
Pubkey, the lookup code would be respectively:
   crds.get::<&ContactInfo>(pubkey)
   crds.get::<&LowestSlot>(pubkey)
 2021-07-21 12:16:26 +00:00
carllin
ce467bea20 Add frozen hashes and marking DuplicateConfirmed in blockstore to state machine (#18648) 2021-07-18 17:04:25 -07:00
behzad nouri
e316586516 excludes private ip addresses 2021-07-16 20:05:48 -06:00
Jeff Biseda
ae5ad5cf9b sendmmsg cleanup #18589 
Rationalize usage of sendmmsg(2). Skip packets which failed to send and track failures.
 2021-07-16 14:36:49 -07:00