836 Commits

Author SHA1 Message Date
c2754b8a51 chore: bump libc from 0.2.106 to 0.2.107 (#21202)
* chore: bump libc from 0.2.106 to 0.2.107

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.106 to 0.2.107.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.106...0.2.107)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-08 12:14:12 -07:00
11a53de0e3 rewrites Shredder::try_recovery (#21082)
For every missing data-shred, Shredder::try_recovery calls into
new_empty_data_shred which does a redundant serialization into payload
buffer which is then immediately overwritten by the erasure recovery:
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L372-L417

Additionally, the implementation is unnecessary complex hindering
upcoming changes to erasure coding generation
https://github.com/solana-labs/solana/blob/696501500/ledger/src/shred.rs#L814-L938

The commit simplifies the Shredder::try_recovery implementation.
2021-11-04 16:15:51 +00:00
ca51ea3e04 chore: bump libc from 0.2.105 to 0.2.106 (#21093)
* chore: bump libc from 0.2.105 to 0.2.106

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.105 to 0.2.106.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.105...0.2.106)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-01 14:17:45 -06:00
484ead01ed optimizes and simplifies SlotMeta::completed_data_indexes ops (#21059)
SlotMeta::completed_data_indexes is defined as a Vec<u32>:
https://github.com/solana-labs/solana/blob/a8d78e89d/ledger/src/blockstore_meta.rs#L31-L32

which results in inefficient updates:
https://github.com/solana-labs/solana/blob/a8d78e89d/ledger/src/blockstore.rs#L3245-L3326

This commit changes the type to BTreeSet<u32> for efficient and simpler
updates and lookups.

The change should be backward compatible because Vec<T> and BTreeSet<T>
are both serialized as seq:
https://github.com/serde-rs/serde/blob/ce0844b9e/serde/src/ser/impls.rs#L207-L208
https://github.com/serde-rs/serde/blob/ce0844b9e/serde/src/ser/impls.rs#L216-L217
2021-10-31 12:56:25 +00:00
4642a2c856 Optimize test_recovery_with_expanded_coding_shreds (#20849) 2021-10-28 11:38:08 +02:00
f0de3e9bf0 chore: bump tonic from 0.5.2 to 0.6.1 (#21035)
* Bump tonic and prost crates

* connect_lazy no longer infallible
2021-10-28 06:59:36 +00:00
036d7fcc81 Clean up sanitized tx creation for tests (#21006) 2021-10-27 18:09:16 +01:00
cb1947bea2 chore: bump libc from 0.2.104 to 0.2.105 (#20972)
* chore: bump libc from 0.2.104 to 0.2.105

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.104 to 0.2.105.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.104...0.2.105)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-10-26 00:14:58 -06:00
350bb561eb Clippy 2021-10-23 08:21:20 +00:00
66e431dad5 chore: bump libc from 0.2.103 to 0.2.104 (#20764)
* chore: bump libc from 0.2.103 to 0.2.104

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.103 to 0.2.104.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.103...0.2.104)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-10-19 22:38:02 -06:00
b5f21d5e34 Add ledger tool to remove dead slot (#20810) 2021-10-19 20:23:16 -07:00
231b58b5f1 adds more metrics to blockstore insert shreds stats (#20701) 2021-10-17 11:02:34 -04:00
70b2e5fef2 hash calc uses all threads during startup (#20735) 2021-10-15 17:41:23 -05:00
e806fa6904 Include token owners in TransactionTokenBalances (#20642)
* Cache owners in TransactionTokenBalances

* Light cleanup

* Use return struct, and remove pub

* Single-use statements

* Why not, just do the whole crate

* Add metrics

* Make datapoint_debug to prevent spam unless troubleshooting
2021-10-13 21:46:52 -06:00
09e7782d76 Refactor code to get block signatures in get_confirmed_signatures_for_address2 (#20575)
* Refactor get_confirmed_signatures_for_address2

* Move blockstore benches to ledger where they belong
2021-10-13 09:55:19 +02:00
005d6863fd - move cost tracker into bank, so each bank has its own cost tracker; (#20527)
- move related modules to runtime
2021-10-12 08:51:33 -05:00
a8e000a2a6 add filler accounts to bloat validator and predict failure (#20491)
* add filler accounts to bloat validator and predict failure

* assert no accounts match filler

* cleanup magic numbers

* panic if can't load from snapshot with filler accounts specified

* some renames

* renames

* into_par_iter

* clean filler accts, too
2021-10-11 12:46:27 -05:00
5440c1d2e1 SnapshotPackagerService pushes incremental snapshot hashes to CRDS (#20442)
Now that CRDS supports incremental snapshot hashes,
SnapshotPackagerService needs to push 'em!

This commit does two main things:

1. SnapshotPackagerService now knows about incremental snapshot hashes,
   and will push SnapshotPackage::IncrementalSnapshot hashes to CRDS.
2. At startup, when loading from a full + incremental snapshot, the
   hashes need to be passed all the way to SnapshotPackagerService so it
   can push these starting hashes to CRDS.  Those values have been piped
   through.

Fixes #20441 and #20423
2021-10-08 15:14:56 -05:00
675fa6993b - update const cost values with data collected by #19627 (#20314)
- update cost calculation to closely proposed fee schedule #16984
2021-10-08 14:48:50 -05:00
767f740305 Bump version to 1.9.0 2021-10-06 17:57:41 -07:00
7027d56064 Resolve nightly-2021-10-05 clippy complaints 2021-10-06 10:37:58 -07:00
d8e4840b49 Follow up to blockstore test cleanup (#19656) (#20143) 2021-10-04 10:35:03 -05:00
fe97cb2ddf AccountsDb plugin framework (#20047)
Summary of Changes

Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows

Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.

The code comprises 4 major parts:

accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-09-30 14:26:17 -07:00
9393654af0 chore: bump libc from 0.2.102 to 0.2.103 (#20280)
* chore: bump libc from 0.2.102 to 0.2.103

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.102 to 0.2.103.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.102...0.2.103)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2021-09-28 07:50:46 +00:00
c71fab6cb3 Add delete subcommand to ledger-tool bigtable (#19931)
* Add `delete` subcommand to `ledger-tool bigtable` command

* feedback
2021-09-16 23:37:45 +00:00
ea15c0b40b chore: bump libc from 0.2.101 to 0.2.102 (#19924)
* chore: bump libc from 0.2.101 to 0.2.102

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.101 to 0.2.102.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.101...0.2.102)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-16 09:45:01 -06:00
4fdd70c931 Properly clean temp files in Blockstore::create_new_ledger() (#19589)
Use TempDir::path() instead of TempDir::into_path(); into_path() returns
the underlying PathBuf and voids the promise to delete the directory
when TempDir goes out of scope.
2021-09-13 23:13:56 -04:00
97fc64dd0c Call exhaustively_free_unused_resource() instead (#19847) 2021-09-13 18:08:32 -05:00
ef749a2506 Flush accounts cache before calling clean_accounts() (#19839) 2021-09-13 14:56:13 -05:00
3f6eb96d6e Call Bank::update_accounts_hash() before taking bank snapshot (#19765)
This is a bugfix.

When `load_frozen_forks()` called `snapshot_bank()`, the accounts hash
was not computed.  So then when a snapshot archive was eventually
created, its hash would be all 1s, which is clearly wrong.  The fix is
to update the bank's accounts hash before taking the bank snapshot.
2021-09-13 08:27:50 -05:00
62c8bcf565 Add default() to SnapshotConfig (#19776) 2021-09-12 13:44:27 -05:00
857fce13f9 chore: bump sha2 from 0.9.6 to 0.9.8 (#19760)
* chore: bump sha2 from 0.9.6 to 0.9.8

Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.9.6 to 0.9.8.
- [Release notes](https://github.com/RustCrypto/hashes/releases)
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.9.6...sha2-v0.9.8)

---
updated-dependencies:
- dependency-name: sha2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-11 16:22:40 -06:00
c3a610374a unused deps 2021-09-11 14:33:53 -07:00
2dee098b91 Blockstore test cleanup (#19656)
* Use auto-deleting macro for ledger path
2021-09-10 07:33:08 -05:00
456bf15012 AccountsIndexConfig -> AccountsDbConfig (#19687) 2021-09-08 04:30:38 +00:00
d3f938f0cf Remove Copy from AccountsIndexConfig. Not all types will support it (#19686) 2021-09-07 20:09:40 -05:00
4a5f83d3a7 Add fn to check when to take snapshots (#19682) 2021-09-07 23:26:35 +00:00
a0552e5b46 Make startup aware of Incremental Snapshots (#19600) 2021-09-07 20:43:43 +00:00
5fa3e5744c Populate memo in blockstore signatures-for-address (#19515)
* Add TransactionMemos column family

* Traitify extract_memos

* Write TransactionMemos in TransactionStatusService

* Populate memos from column

* Dedupe and add unit test
2021-09-03 00:28:52 -06:00
170927b7c4 nit: Move builtins types to builtins file (#19597) 2021-09-02 21:29:11 -07:00
e9374d32a3 Revert "Make startup aware of Incremental Snapshots (#19550)" (#19599)
This reverts commit d45ced0a5d.
2021-09-02 19:14:41 -05:00
d45ced0a5d Make startup aware of Incremental Snapshots (#19550) 2021-09-02 19:05:15 -05:00
1d2f0f6641 Simplify process_blockstore tests (#19553)
Some of the blockstore_processor tests weren't using
`test_process_blockstore()`, but could.  I updated those tests, and also
changed to using `..` for ignoring, instead of `_`, since I'll be adding
another return value here shortly (thus reducing the need to change
these again).
2021-09-01 20:13:52 +00:00
6d9818b8e4 skips retransmit for shreds with unknown slot leader (#19472)
Shreds' signatures should be verified before they reach retransmit
stage, and if the leader is unknown they should fail signature check.
Therefore retransmit-stage can as well expect to know who the slot
leader is and otherwise just skip the shred.

Blockstore checking signature of recovered shreds before sending them to
retransmit stage:
https://github.com/solana-labs/solana/blob/4305d4b7b/ledger/src/blockstore.rs#L884-L930

Shred signature verifier:
https://github.com/solana-labs/solana/blob/4305d4b7b/core/src/sigverify_shreds.rs#L41-L57
https://github.com/solana-labs/solana/blob/4305d4b7b/ledger/src/sigverify_shreds.rs#L105
2021-09-01 15:44:26 +00:00
8e4360cf9c chore: bump futures from 0.3.16 to 0.3.17 (#19513)
* chore: bump futures from 0.3.16 to 0.3.17

Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.16 to 0.3.17.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.16...0.3.17)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-08-30 22:12:38 -06:00
dc3d7ad2bd chore: bump sha2 from 0.9.5 to 0.9.6 (#19511)
* chore: bump sha2 from 0.9.5 to 0.9.6

Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.9.5 to 0.9.6.
- [Release notes](https://github.com/RustCrypto/hashes/releases)
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.9.5...sha2-v0.9.6)

---
updated-dependencies:
- dependency-name: sha2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-30 19:58:26 -06:00
6a53ec28e0 chore: bump serde from 1.0.129 to 1.0.130 (#19497)
* chore: bump serde from 1.0.129 to 1.0.130

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.129 to 1.0.130.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.129...v1.0.130)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-30 18:19:34 +00:00
fbffdf9f3d chore: bump futures-util from 0.3.16 to 0.3.17 (#19498)
* chore: bump futures-util from 0.3.16 to 0.3.17

Bumps [futures-util](https://github.com/rust-lang/futures-rs) from 0.3.16 to 0.3.17.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.16...0.3.17)

---
updated-dependencies:
- dependency-name: futures-util
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-08-30 17:45:47 +00:00
8ad52fa095 implements copy-on-write for vote-accounts (#19362)
Bank::vote_accounts redundantly clones vote-accounts HashMap even though
an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/95c998a19/runtime/src/bank.rs#L5174-L5186

This commit implements copy-on-write semantics for vote-accounts by
wrapping the underlying HashMap in Arc<...>.
2021-08-30 15:54:01 +00:00
e492638b7d chore: bump libc from 0.2.100 to 0.2.101 (#19442)
* chore: bump libc from 0.2.100 to 0.2.101

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.100 to 0.2.101.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.100...0.2.101)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <anatoly+githubjenkins@solana.io>
2021-08-26 12:38:34 -06:00