gaspersic/solana
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
16,929 Commits 81 Branches 357 Tags
e476e17abfc42b72a2006feeb31230578670a94c
Commit Graph

2208 Commits

Author SHA1 Message Date
Brooks Prumo
d45ced0a5d Make startup aware of Incremental Snapshots (#19550) 2021-09-02 19:05:15 -05:00
Jeff Washington (jwash)
57f51352f6 refactor AccountEntry addref/unref (#19583) 2021-09-02 23:25:27 +00:00
Jeff Washington (jwash)
682daf1117 cleanup of allocate_accounts_index (#19585) 2021-09-02 22:58:07 +00:00
Jeff Washington (jwash)
78585a992b add IsCached trait bounds to AccountsIndex (#19584) 2021-09-02 22:45:35 +00:00
Jeff Washington (jwash)
2874f6b0a2 create remove_if_slot_list_empty to define accounts index behavior (#19581) 2021-09-02 16:54:37 -05:00
Jeff Washington (jwash)
0571962776 accounts index test uses # test bins (#19580) 2021-09-02 16:32:42 -05:00
Jeff Washington (jwash)
6ab8286e8d add logging for a handful of tests (#19582) 2021-09-02 16:32:19 -05:00
Jeff Washington (jwash)
f5388cfe71 tests which assert or panic in other threads now will report an error (#19532) 
* tests which assert or panic in other threads now will report an error

* rename, add assert detecting time out
 2021-09-02 11:26:57 -05:00
Jeff Washington (jwash)
7eb793a55e add debugging to test (#19563) 
* add debugging to test

* update cargo
 2021-09-02 02:52:16 +00:00
Jeff Washington (jwash)
ed2c071fe1 remove unused is_startup (#19561) 2021-09-01 18:39:00 -05:00
Lijun Wang
8378e8790f Accountsdb replication installment 2 (#19325) 
This is the 2nd installment for the AccountsDb replication.

Summary of Changes

The basic google protocol buffer protocol for replicating updated slots and accounts. tonic/tokio is used for transporting the messages.

The basic framework of the client and server for replicating slots and accounts -- the persisting of accounts in the replica-side will be done at the next PR -- right now -- the accounts are streamed to the replica-node and dumped. Replication for information about Bank is also not done in this PR -- to be addressed in the next PR to limit the change size.

Functionality used by both the client and server side are encapsulated in the replica-lib crate.

There is no impact to the existing validator by default.

Tests:

Observe the confirmed slots replicated to the replica-node.
Observe the accounts for the confirmed slot are received at the replica-node side.
 2021-09-01 14:10:16 -07:00
Jeff Washington (jwash)
a5f74d86c8 make test pass with 1 bin to remove noisy failure during testing (#19554) 2021-09-01 15:58:56 -05:00
Jeff Washington (jwash)
dd9481c403 parallelize index_read in shrink (#19506) 2021-09-01 15:40:59 -05:00
Jeff Washington (jwash)
82a6bbe068 add Debug trait to range for accounts index (#19531) 2021-09-01 08:13:56 -05:00
Jeff Washington (jwash)
12dc8749cf add Copy trait to AccountInfo for fast copies to mmapped file (#19524) 2021-09-01 08:13:08 -05:00
Tyera Eulberg
0088aefa24 Fix tests that make assumptions about tx fee rate (#19537) 2021-08-31 22:02:14 -06:00
Brooks Prumo
1d5a8ebc6a Revert "Add LastFullSnapshotSlot to SnapshotConfig (#19341)" (#19529) 
This reverts commit 4d361af976.
 2021-08-31 22:03:19 -05:00
Jeff Washington (jwash)
2df96cd81e remove type specifics to make funciton more flexible (#19536) 2021-08-31 20:03:42 -05:00
Brooks Prumo
fe9ee9134a Make background services aware of incremental snapshots (#19401) 
AccountsBackgroundService now knows about incremental snapshots.  It is
now also in charge of deciding if an AccountsPackage is destined to be a
SnapshotPackage or not (or just used by AccountsHashVerifier).

!!! New behavior changes !!!

Taking snapshots (both bank and archive) **MUST** succeed.

This is required because of how the last full snapshot slot is
calculated, which is used by AccountsBackgroundService when calling
`clean_accounts()`.

File system calls are now unwrapped and will result in a crash. As Trent told me:

>Well I think if a snapshot fails due to some IO error, it's very likely that the operator is going to have to intervene before it works.  We should exit error in this case, otherwise the validator might happily spin for several more hours, never successfully writing a complete snapshot, before something else brings it down.  This would leave the validator's last local snapshot many more slots behind than it would be had we exited outright and potentially force the operator to abandon ledger continuity in favor of a quick catchup

Other errors will set the `exit` flag to `true`, and the node will gracefully shutdown.

Fixes #19167 
Fixes #19168
 2021-08-31 18:33:27 -05:00
Brooks Prumo
718ab7c12e Revert "Add snapshot_runtime_info module (#18199)" 
This reverts commit c1b9d40a64.
 2021-08-31 17:50:18 -05:00
Jack May
5cf8d8795b Feature cleanup (#19528) 2021-08-31 14:51:26 -07:00
Jeff Washington (jwash)
e1939688c2 parallel shrink (#19507) 2021-08-31 11:57:34 -05:00
Jeff Washington (jwash)
0e94b8fd4e remove redundant cancelled_shrink (#19522) 2021-08-31 11:51:48 -05:00
Jeff Washington (jwash)
9d3afba045 Debug shrink stats (#19505) 
* add some shrink metrics

* renames
 2021-08-31 09:04:56 -05:00
Lijun Wang
09458cc802 Shrink mmap leakage (#19373) 
* Putting stores satisfying future shrink into the candidate list.

* Fixed unit tests for select_shrink_candidates

* Added metrics on stores readded back for next round

* Ehance the metrics on accountsdb to report the total bytes and alive bytes and alive ratio

* Enhance select_shrink_candidates metrics
 2021-08-31 00:55:16 -07:00
Jeff Washington (jwash)
a655b01700 some basic accounts index refactoring (#19510) 2021-08-30 18:40:10 -05:00
dependabot[bot]
6cfd44b811 chore: bump flate2 from 1.0.20 to 1.0.21 (#19500) 
* chore: bump flate2 from 1.0.20 to 1.0.21

Bumps [flate2](https://github.com/rust-lang/flate2-rs) from 1.0.20 to 1.0.21.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.0.20...1.0.21)

---
updated-dependencies:
- dependency-name: flate2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-08-30 16:48:02 -06:00
dependabot[bot]
6a53ec28e0 chore: bump serde from 1.0.129 to 1.0.130 (#19497) 
* chore: bump serde from 1.0.129 to 1.0.130

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.129 to 1.0.130.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.129...v1.0.130)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-08-30 18:19:34 +00:00
behzad nouri
8ad52fa095 implements copy-on-write for vote-accounts (#19362) 
Bank::vote_accounts redundantly clones vote-accounts HashMap even though
an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/95c998a19/runtime/src/bank.rs#L5174-L5186

This commit implements copy-on-write semantics for vote-accounts by
wrapping the underlying HashMap in Arc<...>.
 2021-08-30 15:54:01 +00:00
Justin Starry
2d7f036afd Add solana-program-runtime crate (#19438) 2021-08-27 00:30:36 +00:00
Jeff Washington (jwash)
02b050e0f5 replace AccountsIndex btree with hashmap of 8k bins (#19212) 2021-08-26 18:12:43 -05:00
Jeff Washington (jwash)
98bc694606 hash calculation adds really old slots to dirty_stores (#19434) 2021-08-26 14:32:43 -05:00
Jeff Washington (jwash)
535de3b302 add test for test_clean_nonrooted (#19409) 2021-08-26 13:56:09 -05:00
dependabot[bot]
e492638b7d chore: bump libc from 0.2.100 to 0.2.101 (#19442) 
* chore: bump libc from 0.2.100 to 0.2.101

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.100 to 0.2.101.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.100...0.2.101)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
 2021-08-26 12:38:34 -06:00
dependabot[bot]
46890ac197 chore: bump serde from 1.0.128 to 1.0.129 (#19395) 
* chore: bump serde from 1.0.128 to 1.0.129

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.128 to 1.0.129.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.128...v1.0.129)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-08-24 12:32:30 -06:00
dependabot[bot]
c0c95e88d8 chore: bump libc from 0.2.99 to 0.2.100 (#19371) 
* chore: bump libc from 0.2.99 to 0.2.100

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.99 to 0.2.100.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.99...0.2.100)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
 2021-08-23 09:34:32 -06:00
dependabot[bot]
aea9960c75 chore: bump serde from 1.0.127 to 1.0.128 (#19370) 
* chore: bump serde from 1.0.127 to 1.0.128

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.127 to 1.0.128.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.127...v1.0.128)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
 2021-08-23 09:33:34 -06:00
Brooks Prumo
1a4bede918 Remove filter_snapshot_storages_for_incremental_snapshot() (#19349) 
Now that we can get just the subset of snapshot storages we want, the extra
filtering afterwards is redundant.
 2021-08-23 09:45:29 -05:00
Brooks Prumo
6d939811e9 Name snapshots consistently (#19346) 
#### Problem

Snapshot names are overloaded, and there are multiple terms that mean the same thing. This is confusing. Here's a list of ones in the codebase that I've found:

```
- snapshot_dir
- snapshots_dir
- snapshot_path
- snapshot_output_dir
- snapshot_package_output_path
- snapshot_archives_dir
```

#### Summary of Changes

For all the ones that are about the directory where snapshot archives are stored, ensure they are `snapshot_archives_dir`. For the ones about the (bank) snapshots directory, set to `bank_snapshots_dir`.


Co-authored-by: Michael Vines <mvines@gmail.com>
 2021-08-21 15:41:03 -05:00
Brooks Prumo
234461f779 Add base_slot to get_snapshot_storages() (#19348) 2021-08-20 16:23:43 -05:00
Brooks Prumo
4d361af976 Add LastFullSnapshotSlot to SnapshotConfig (#19341) 2021-08-20 17:06:53 +00:00
jon-chuang
3e5ba8dcaa bug: sysvar::Instructions is not owned by Sysvar1111111111111111111111111111111111111 (#19242) 
* Fix instructions sysvar owner

* Update feature switch address

Co-authored-by: Justin Starry <justin@solana.com>
 2021-08-20 08:32:28 -07:00
Jon Cinque
73aa004c59 stake: Remove v2 program references (#19308) 
* stake: Remove v2 program references

* Remove stake v2 feature, along with stake rewrite
 2021-08-20 01:08:44 -04:00
Jack May
3ec33e7d02 Fail secp256k1 if the instruction data looks incorrect (#19300) 2021-08-19 13:13:54 -07:00
Jeff Washington (jwash)
89a31ff473 change untar to use unpack instead of unpack_in (#19216) 
* change untar to use unpack instead of unpack_in

* hacky, but maybe passes tests

* chore: bump tar from 0.4.35 to 0.4.37

Bumps [tar](https://github.com/alexcrichton/tar-rs) from 0.4.35 to 0.4.37.
- [Release notes](https://github.com/alexcrichton/tar-rs/releases)
- [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.35...0.4.37)

---
updated-dependencies:
- dependency-name: tar
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

* cleanup

* cleanup, add validate_inside_dst

* collapse use

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* delete comment line

* add comments

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
 2021-08-18 15:49:02 -05:00
Jeff Washington (jwash)
6327e006df CalculateHashIntermediate::new_without_slot -> new (#19274) 2021-08-18 09:07:34 -05:00
Tyera Eulberg
c167211611 Expose genesis block time via rpc (#19267) 
* Expose genesis_creation_time from Bank

* Backfill genesis_creation_time for block/block-time requests of slot 0
 2021-08-17 16:29:34 -06:00
Justin Starry
c50b01cb60 Store versioned transactions in the ledger, disabled by default (#19139) 
* Add support for versioned transactions, but disable by default

* merge conflicts

* trent's feedback

* bump Cargo.lock

* Fix transaction error encoding

* Rename legacy_transaction method

* cargo clippy

* Clean up casts, int arithmetic, and unused methods

* Check for duplicates in sanitized message conversion

* fix clippy

* fix new test

* Fix bpf conditional compilation for message module
 2021-08-17 15:17:56 -07:00
dependabot[bot]
098e2b2de3 chore: bump memmap2 from 0.3.0 to 0.3.1 (#19255) 
* chore: bump memmap2 from 0.3.0 to 0.3.1

Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.3.0 to 0.3.1.
- [Release notes](https://github.com/RazrFalcon/memmap2-rs/releases)
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v.0.3.0...v0.3.1)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
 2021-08-17 15:49:50 -06:00
Jeff Washington (jwash)
7c70f2158b accounts_index_bins to AccountsIndexConfig (#19257) 
* accounts_index_bins to AccountsIndexConfig

* rename param bins -> config

* rename BINS_FOR* to ACCOUNTS_INDEX_CONFIG_FOR*
 2021-08-17 14:50:01 -05:00