a5f74d86c8
make test pass with 1 bin to remove noisy failure during testing ( #19554 )
2021-09-01 15:58:56 -05:00
dd9481c403
parallelize index_read in shrink ( #19506 )
2021-09-01 15:40:59 -05:00
82a6bbe068
add Debug trait to range for accounts index ( #19531 )
2021-09-01 08:13:56 -05:00
12dc8749cf
add Copy trait to AccountInfo for fast copies to mmapped file ( #19524 )
2021-09-01 08:13:08 -05:00
0088aefa24
Fix tests that make assumptions about tx fee rate ( #19537 )
2021-08-31 22:02:14 -06:00
1d5a8ebc6a
Revert "Add LastFullSnapshotSlot to SnapshotConfig ( #19341 )" ( #19529 )
...
This reverts commit 4d361af976
2021-08-31 22:03:19 -05:00
2df96cd81e
remove type specifics to make funciton more flexible ( #19536 )
2021-08-31 20:03:42 -05:00
fe9ee9134a
Make background services aware of incremental snapshots ( #19401 )
...
AccountsBackgroundService now knows about incremental snapshots. It is
now also in charge of deciding if an AccountsPackage is destined to be a
SnapshotPackage or not (or just used by AccountsHashVerifier).
!!! New behavior changes !!!
Taking snapshots (both bank and archive) **MUST** succeed.
This is required because of how the last full snapshot slot is
calculated, which is used by AccountsBackgroundService when calling
`clean_accounts()`.
File system calls are now unwrapped and will result in a crash. As Trent told me:
>Well I think if a snapshot fails due to some IO error, it's very likely that the operator is going to have to intervene before it works. We should exit error in this case, otherwise the validator might happily spin for several more hours, never successfully writing a complete snapshot, before something else brings it down. This would leave the validator's last local snapshot many more slots behind than it would be had we exited outright and potentially force the operator to abandon ledger continuity in favor of a quick catchup
Other errors will set the `exit` flag to `true`, and the node will gracefully shutdown.
Fixes #19167
Fixes #19168
2021-08-31 18:33:27 -05:00
718ab7c12e
Revert "Add snapshot_runtime_info module ( #18199 )"
...
This reverts commit c1b9d40a64
2021-08-31 17:50:18 -05:00
5cf8d8795b
Feature cleanup ( #19528 )
2021-08-31 14:51:26 -07:00
e1939688c2
parallel shrink ( #19507 )
2021-08-31 11:57:34 -05:00
0e94b8fd4e
remove redundant cancelled_shrink ( #19522 )
2021-08-31 11:51:48 -05:00
9d3afba045
Debug shrink stats ( #19505 )
...
* add some shrink metrics
* renames
2021-08-31 09:04:56 -05:00
09458cc802
Shrink mmap leakage ( #19373 )
...
* Putting stores satisfying future shrink into the candidate list.
* Fixed unit tests for select_shrink_candidates
* Added metrics on stores readded back for next round
* Ehance the metrics on accountsdb to report the total bytes and alive bytes and alive ratio
* Enhance select_shrink_candidates metrics
2021-08-31 00:55:16 -07:00
a655b01700
some basic accounts index refactoring ( #19510 )
2021-08-30 18:40:10 -05:00
6cfd44b811
chore: bump flate2 from 1.0.20 to 1.0.21 ( #19500 )
...
* chore: bump flate2 from 1.0.20 to 1.0.21
Bumps [flate2](https://github.com/rust-lang/flate2-rs ) from 1.0.20 to 1.0.21.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases )
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.0.20...1.0.21 )
---
updated-dependencies:
- dependency-name: flate2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com >
2021-08-30 16:48:02 -06:00
6a53ec28e0
chore: bump serde from 1.0.129 to 1.0.130 ( #19497 )
...
* chore: bump serde from 1.0.129 to 1.0.130
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.129 to 1.0.130.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.129...v1.0.130 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com >
2021-08-30 18:19:34 +00:00
8ad52fa095
implements copy-on-write for vote-accounts ( #19362 )
...
Bank::vote_accounts redundantly clones vote-accounts HashMap even though
an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/95c998a19/runtime/src/bank.rs#L5174-L5186
This commit implements copy-on-write semantics for vote-accounts by
wrapping the underlying HashMap in Arc<...>.
2021-08-30 15:54:01 +00:00
2d7f036afd
Add solana-program-runtime crate ( #19438 )
2021-08-27 00:30:36 +00:00
02b050e0f5
replace AccountsIndex btree with hashmap of 8k bins ( #19212 )
2021-08-26 18:12:43 -05:00
98bc694606
hash calculation adds really old slots to dirty_stores ( #19434 )
2021-08-26 14:32:43 -05:00
535de3b302
add test for test_clean_nonrooted ( #19409 )
2021-08-26 13:56:09 -05:00
e492638b7d
chore: bump libc from 0.2.100 to 0.2.101 ( #19442 )
...
* chore: bump libc from 0.2.100 to 0.2.101
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.100 to 0.2.101.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.100...0.2.101 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io >
2021-08-26 12:38:34 -06:00
46890ac197
chore: bump serde from 1.0.128 to 1.0.129 ( #19395 )
...
* chore: bump serde from 1.0.128 to 1.0.129
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.128 to 1.0.129.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.128...v1.0.129 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com >
2021-08-24 12:32:30 -06:00
c0c95e88d8
chore: bump libc from 0.2.99 to 0.2.100 ( #19371 )
...
* chore: bump libc from 0.2.99 to 0.2.100
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.99 to 0.2.100.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.99...0.2.100 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com >
2021-08-23 09:34:32 -06:00
aea9960c75
chore: bump serde from 1.0.127 to 1.0.128 ( #19370 )
...
* chore: bump serde from 1.0.127 to 1.0.128
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.127 to 1.0.128.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.127...v1.0.128 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io >
2021-08-23 09:33:34 -06:00
1a4bede918
Remove filter_snapshot_storages_for_incremental_snapshot() ( #19349 )
...
Now that we can get just the subset of snapshot storages we want, the extra
filtering afterwards is redundant.
2021-08-23 09:45:29 -05:00
6d939811e9
Name snapshots consistently ( #19346 )
...
#### Problem
Snapshot names are overloaded, and there are multiple terms that mean the same thing. This is confusing. Here's a list of ones in the codebase that I've found:
```
- snapshot_dir
- snapshots_dir
- snapshot_path
- snapshot_output_dir
- snapshot_package_output_path
- snapshot_archives_dir
```
#### Summary of Changes
For all the ones that are about the directory where snapshot archives are stored, ensure they are `snapshot_archives_dir`. For the ones about the (bank) snapshots directory, set to `bank_snapshots_dir`.
Co-authored-by: Michael Vines <mvines@gmail.com >
2021-08-21 15:41:03 -05:00
234461f779
Add base_slot to get_snapshot_storages() ( #19348 )
2021-08-20 16:23:43 -05:00
4d361af976
Add LastFullSnapshotSlot to SnapshotConfig ( #19341 )
2021-08-20 17:06:53 +00:00
3e5ba8dcaa
bug: sysvar::Instructions is not owned by Sysvar1111111111111111111111111111111111111 ( #19242 )
...
* Fix instructions sysvar owner
* Update feature switch address
Co-authored-by: Justin Starry <justin@solana.com >
2021-08-20 08:32:28 -07:00
73aa004c59
stake: Remove v2 program references ( #19308 )
...
* stake: Remove v2 program references
* Remove stake v2 feature, along with stake rewrite
2021-08-20 01:08:44 -04:00
3ec33e7d02
Fail secp256k1 if the instruction data looks incorrect ( #19300 )
2021-08-19 13:13:54 -07:00
89a31ff473
change untar to use unpack instead of unpack_in ( #19216 )
...
* change untar to use unpack instead of unpack_in
* hacky, but maybe passes tests
* chore: bump tar from 0.4.35 to 0.4.37
Bumps [tar](https://github.com/alexcrichton/tar-rs ) from 0.4.35 to 0.4.37.
- [Release notes](https://github.com/alexcrichton/tar-rs/releases )
- [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.35...0.4.37 )
---
updated-dependencies:
- dependency-name: tar
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
* [auto-commit] Update all Cargo lock files
* cleanup
* cleanup, add validate_inside_dst
* collapse use
Co-authored-by: Tyera Eulberg <teulberg@gmail.com >
* delete comment line
* add comments
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com >
Co-authored-by: Tyera Eulberg <teulberg@gmail.com >
2021-08-18 15:49:02 -05:00
6327e006df
CalculateHashIntermediate::new_without_slot -> new ( #19274 )
2021-08-18 09:07:34 -05:00
c167211611
Expose genesis block time via rpc ( #19267 )
...
* Expose genesis_creation_time from Bank
* Backfill genesis_creation_time for block/block-time requests of slot 0
2021-08-17 16:29:34 -06:00
c50b01cb60
Store versioned transactions in the ledger, disabled by default ( #19139 )
...
* Add support for versioned transactions, but disable by default
* merge conflicts
* trent's feedback
* bump Cargo.lock
* Fix transaction error encoding
* Rename legacy_transaction method
* cargo clippy
* Clean up casts, int arithmetic, and unused methods
* Check for duplicates in sanitized message conversion
* fix clippy
* fix new test
* Fix bpf conditional compilation for message module
2021-08-17 15:17:56 -07:00
098e2b2de3
chore: bump memmap2 from 0.3.0 to 0.3.1 ( #19255 )
...
* chore: bump memmap2 from 0.3.0 to 0.3.1
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs ) from 0.3.0 to 0.3.1.
- [Release notes](https://github.com/RazrFalcon/memmap2-rs/releases )
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v.0.3.0...v0.3.1 )
---
updated-dependencies:
- dependency-name: memmap2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
* [auto-commit] Update all Cargo lock files
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com >
2021-08-17 15:49:50 -06:00
7c70f2158b
accounts_index_bins to AccountsIndexConfig ( #19257 )
...
* accounts_index_bins to AccountsIndexConfig
* rename param bins -> config
* rename BINS_FOR* to ACCOUNTS_INDEX_CONFIG_FOR*
2021-08-17 14:50:01 -05:00
f9986c66b8
Make SnapshotPackagerService aware of Incremental Snapshots ( #19254 )
...
Add a field to SnapshotPackage that is an enum for SnapshotType, so archive_snapshot_package() will do the right thing.
Fixes #19166
2021-08-17 13:01:59 -05:00
7481d9b66b
exhaustively_free_unused_resource uses more threads ( #19264 )
2021-08-17 12:38:41 -05:00
39a3d5b8a9
implements AsRef (instead of Borrow) for VoteAccounts ( #19252 )
...
According to the docs:
> if you want to borrow only a single field of a struct you can
> implement AsRef, but not Borrow.
https://doc.rust-lang.org/std/convert/trait.AsRef.html
2021-08-16 18:21:33 +00:00
176036aa58
Rename AccountsPacakge to SnapshotPackage and AccountsPackagePre to AccountsPackage ( #19231 )
...
Renaming these types to better communicate their usages, which will
further diverge as incremental snapshot support is added.
With the new names, AccountsPacakge now refers to the type between
AccountsBackgroundProcess and AccountsHashVerifier, and SnapshotPackage
refers to the type between AccountsHashVerifier and
SnapshotPackagerService.
2021-08-13 16:08:09 -05:00
1a3f29e22d
rework bank test to specifically use BTreeMap ( #19225 )
2021-08-13 13:30:48 -05:00
01b00bc593
Fix crossed PRs; remove deprecated fee_calculator usage ( #19228 )
2021-08-13 11:14:00 -06:00
0b50bb2b20
Deprecate FeeCalculator returning APIs ( #19120 )
2021-08-13 09:08:20 -07:00
26e963f436
remove par iter for 32k wide code path ( #18764 )
2021-08-13 10:07:47 -05:00
772341dccb
fixup! Handle cleaning of zero-lamport accounts w.r.t. Incremental Snapshots ( #18870 ) ( #19213 )
2021-08-12 17:29:58 -05:00
5fb6b341c4
Handle cleaning of zero-lamport accounts w.r.t. Incremental Snapshots ( #18870 )
...
* Handle cleaning zero-lamport accounts
Handle cleaning zero-lamport accounts in slots higher than the last full
snapshot slot. This is part of the Incremental Snapshot work.
Fixes #18825
2021-08-12 15:56:08 -05:00
414d904959
Reject blocks for costs above the max block cost ( #18994 )
...
* added realtime cost checking logic to reject block that would exceed max limit:
- defines max limits at block_cost_limits.rs
- right after each bath's execution, accumulate its cost and check again
limit, return error if limit is exceeded
* update abi that changed due to adding additional TransactionError
* To avoid counting stats mltiple times, only accumulate execute-timing when a bank is completed
* gate it by a feature
* move cost const def into block_cost_limits.rs
* redefine the cost for signature and account access, removed signer part as it is not well defined for now
* check if per_program_timings of execute_timings before sending
2021-08-12 10:48:47 -05:00
