Commit Graph

485 Commits

Author SHA1 Message Date
Michael Vines
b8837c04ec Reformat imports to a consistent style for imports
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
2021-12-03 09:19:13 -08:00
Tyera Eulberg
2b269dbe0e Ledger-tool: only require ledger dir when necessary (#21575)
* Don't canonicalize ledger_path unless ledger_path will be used

* Single use statement
2021-12-02 17:23:51 -07:00
Michael Vines
dd12d90eac Upgrade to Rust 2021 2021-11-30 20:43:46 -08:00
dependabot[bot]
a125e7b8c2 chore: bump serde_json from 1.0.71 to 1.0.72 (#21426)
* chore: bump serde_json from 1.0.71 to 1.0.72

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.71 to 1.0.72.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.71...v1.0.72)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-30 18:03:09 +00:00
Jeff Washington (jwash)
b1b334a17f ledger-tool uses jemalloc like validator (#21351) 2021-11-19 09:48:07 -06:00
Jeff Washington (jwash)
79d21d6805 add --accounts-index-scan-results-limit-mb to allow scans to abort (#21327)
* ScanConfig -> &ScanConfig

* add --accounts-index-scan-results-limit-mb to allow scans to abort

* feedback
2021-11-19 09:00:19 -06:00
dependabot[bot]
89102540b1 chore: bump serde_json from 1.0.70 to 1.0.71 (#21338)
* chore: bump serde_json from 1.0.70 to 1.0.71

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.70 to 1.0.71.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.70...v1.0.71)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-18 02:58:30 +00:00
Jeff Washington (jwash)
0f69a14247 Add ability to abort scan (#21314) 2021-11-17 13:10:29 -06:00
Jeff Biseda
d5de0c8e12 add --no-os-network-stats-reporting option (#21296) 2021-11-16 10:26:03 -08:00
dependabot[bot]
55ea672041 chore: bump serde_json from 1.0.69 to 1.0.70 (#21285)
* chore: bump serde_json from 1.0.69 to 1.0.70

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.69 to 1.0.70.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.69...v1.0.70)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-11-15 15:27:47 -07:00
Jeff Washington (jwash)
77a59ca3e3 ledger-tool verify supports --accounts-db-skip-shrink (#21284) 2021-11-15 14:42:56 -06:00
Jeff Washington (jwash)
a043b19988 ledger tool verify launches system monitor to report mem (#21265) 2021-11-15 09:29:30 -06:00
Brooks Prumo
8b2133b078 Refactor total stats calculation into shared fn (#21212) 2021-11-09 19:13:37 -06:00
dependabot[bot]
3c5f505d3e chore: bump serde_json from 1.0.68 to 1.0.69 (#21211)
* chore: bump serde_json from 1.0.68 to 1.0.69

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.68 to 1.0.69.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.68...v1.0.69)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-11-09 11:04:40 -07:00
Brooks Prumo
6f3c27f1da Add total accounts stats to ledger-tool accounts (#21187) 2021-11-08 14:45:30 -06:00
Brooks Prumo
2031522809 Add flags to not print account contents (#21178)
It takes a long time to print the contents of all the accounts.  I'll be
adding more to the "accounts" subcommand and would like to skip printing
the account contents.
2021-11-05 08:59:12 -05:00
Justin Starry
140a5f633d Simplify replay vote tracking by using packet metadata (#21112) 2021-11-03 09:02:48 +00:00
dependabot[bot]
855b3ccdc0 chore: bump signal-hook from 0.2.3 to 0.3.10 (#21081)
* chore: bump signal-hook from 0.2.3 to 0.3.10

Bumps [signal-hook](https://github.com/vorner/signal-hook) from 0.2.3 to 0.3.10.
- [Release notes](https://github.com/vorner/signal-hook/releases)
- [Changelog](https://github.com/vorner/signal-hook/blob/master/CHANGELOG.md)
- [Commits](https://github.com/vorner/signal-hook/compare/v0.2.3...v0.3.10)

---
updated-dependencies:
- dependency-name: signal-hook
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update apis

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-10-30 00:21:54 -06:00
Jeff Washington (jwash)
43ea579f63 add cli for --accounts-hash-num-passes (#20827) 2021-10-25 09:45:46 -05:00
Tao Zhu
c2bfce90b3 - cost_tracker is data member of a bank, it can report metrics when bank is frozen (#20802)
- removed cost_tracker_stats and histogram
- move stats reporting outside of bank freeze
2021-10-24 22:19:23 -05:00
Michael Vines
350bb561eb Clippy 2021-10-23 08:21:20 +00:00
Brooks Prumo
50571685d4 Take incremental snapshots with ledger-tool (#20857) 2021-10-21 21:02:12 -05:00
carllin
b5f21d5e34 Add ledger tool to remove dead slot (#20810) 2021-10-19 20:23:16 -07:00
Tao Zhu
7496b5784b - make cost_tracker a member of bank, remove shared instance from TPU; (#20627)
- decouple cost_model from cost_tracker; allowing one cost_model
  instance being shared within a validator;
- update cost_model api to calculate_cost(&self...)->transaction_cost
2021-10-19 14:37:33 -05:00
sakridge
3b40fa3f44 Add is_full to ledger-tool (#20183) 2021-10-13 12:50:18 +02:00
Tao Zhu
005d6863fd - move cost tracker into bank, so each bank has its own cost tracker; (#20527)
- move related modules to runtime
2021-10-12 08:51:33 -05:00
Jeff Washington (jwash)
a8e000a2a6 add filler accounts to bloat validator and predict failure (#20491)
* add filler accounts to bloat validator and predict failure

* assert no accounts match filler

* cleanup magic numbers

* panic if can't load from snapshot with filler accounts specified

* some renames

* renames

* into_par_iter

* clean filler accts, too
2021-10-11 12:46:27 -05:00
Jeff Washington (jwash)
6cd1a03bb5 AcctIdx: fix ledger-tool arg options (#20513) 2021-10-07 19:43:21 +00:00
Trent Nelson
767f740305 Bump version to 1.9.0 2021-10-06 17:57:41 -07:00
Michael Vines
7027d56064 Resolve nightly-2021-10-05 clippy complaints 2021-10-06 10:37:58 -07:00
Tao Zhu
03913f6661 add tx count and thread id to stats, each stat reports and resets when slot changes (#20451) 2021-10-06 00:09:19 -05:00
Justin Starry
250a8503fe Make rewards tracer async friendly (#20452) 2021-10-05 21:30:08 -04:00
Tao Zhu
6ff508c643 add transaction cost histogram metrics (#20350) 2021-10-05 08:57:39 -05:00
Lijun Wang
fe97cb2ddf AccountsDb plugin framework (#20047)
Summary of Changes

Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows

Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.

The code comprises 4 major parts:

accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-09-30 14:26:17 -07:00
Jeff Washington (jwash)
5810568c65 add --accounts-index-path to be used by disk accounts index (#19710)
* add --accounts-index-paths to be used by disk accounts index

* add multiple and default to help
2021-09-28 11:07:47 -05:00
Ryo Onodera
22f45dca80 Make bigtable delete-slots actually usable (#20037) 2021-09-20 20:00:32 +00:00
Jeff Washington (jwash)
c1d181add5 add --accounts-index-memory-limit-mb (#19269) 2021-09-19 18:00:15 -05:00
Jeff Washington (jwash)
4e038e94fd add 'drives' to AccountsIndexConfig (#19989) 2021-09-18 09:54:57 -05:00
Jeff Washington (jwash)
99f5684dc4 AcctIdx: handle future config defaults (#19967) 2021-09-17 13:12:06 -05:00
Justin Starry
c71fab6cb3 Add delete subcommand to ledger-tool bigtable (#19931)
* Add `delete` subcommand to `ledger-tool bigtable` command

* feedback
2021-09-16 23:37:45 +00:00
dependabot[bot]
68da1909de chore: bump serde_json from 1.0.67 to 1.0.68 (#19942)
* chore: bump serde_json from 1.0.67 to 1.0.68

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.67 to 1.0.68.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.67...v1.0.68)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-16 20:23:25 +00:00
Brooks Prumo
4c69229a47 Add incremental snapshot CLI args to ledger-tool (#19889)
This commit only adds args to specify the maximum full/incremental
snapshot archives to retain when purging old snapshot archives.  It
purposely does *not* add a way to generate new incremental snapshots.

Fixes #19857
2021-09-16 10:56:18 -05:00
Jeff Washington (jwash)
b57e86abf2 cache account hash info (#19426)
* cache account hash info

* ledger_path -> accounts_hash_cache_path
2021-09-13 20:39:26 -05:00
dependabot[bot]
36f870a7e9 chore: bump serde_yaml from 0.8.20 to 0.8.21 (#19820)
* chore: bump serde_yaml from 0.8.20 to 0.8.21

Bumps [serde_yaml](https://github.com/dtolnay/serde-yaml) from 0.8.20 to 0.8.21.
- [Release notes](https://github.com/dtolnay/serde-yaml/releases)
- [Commits](https://github.com/dtolnay/serde-yaml/compare/0.8.20...0.8.21)

---
updated-dependencies:
- dependency-name: serde_yaml
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-13 15:50:33 +00:00
Brooks Prumo
62c8bcf565 Add default() to SnapshotConfig (#19776) 2021-09-12 13:44:27 -05:00
Michael Vines
0f76077969 Add --destake-vote-account <VOTE_ADDRESS>... argument to create-snapshot command (#19749) 2021-09-11 14:44:37 -07:00
Giles Cope
c3a610374a unused deps 2021-09-11 14:33:53 -07:00
Jeff Washington (jwash)
456bf15012 AccountsIndexConfig -> AccountsDbConfig (#19687) 2021-09-08 04:30:38 +00:00
Brooks Prumo
a0552e5b46 Make startup aware of Incremental Snapshots (#19600) 2021-09-07 20:43:43 +00:00
Brooks Prumo
9d9482b9d8 Plumb maximum_incremental_snapshot_archives_to_retain (#19640) 2021-09-06 18:01:56 -05:00