2583 Commits

Author SHA1 Message Date
Jeff Washington (jwash)
1e064c32e2 pull append_vec_id() out of loop (#20713) 2021-10-15 10:41:23 -05:00
Jeff Washington (jwash)
00b9056004 clean metric: total_us (#20707) 2021-10-15 10:41:05 -05:00
Jeff Washington (jwash)
b80fd7566a add worker threads during startup (#20700) 2021-10-15 10:40:36 -05:00
Tao Zhu
e225ed7ab0 update config by testnet (#20699) 2021-10-15 15:32:46 +00:00
Jeff Washington (jwash)
7eead4e25f clean stats on acct idx lookup (#20632) 2021-10-14 08:37:42 -05:00
Jeff Washington (jwash)
c43b6e0248 log metric for clean_accounts.delta_insert_us (#20634) 2021-10-14 08:37:14 -05:00
Jeff Washington (jwash)
9b042fed30 AcctIdx: support unref first class for perf (#20638) 2021-10-14 08:36:53 -05:00
Jack May
da45be366a Remove blockhash from fee calculation (#20641) 2021-10-13 13:10:58 -07:00
Tao Zhu
220fd41bbc fix unstable test (#20645) 2021-10-13 11:43:18 -05:00
Jeff Washington (jwash)
aec9d8bf2f add clean stats (#20631) 2021-10-13 08:22:11 -05:00
Alexander Meißner
1d813ea078 Refactor: Make program_id always last in program chain (#20598)
* Replaces program_id field in InvokeContextStackFrame by index.

* Swaps order of program account and programdata account.

* Removes program_id parameter from InvokeContext::push().
2021-10-13 08:58:20 +02:00
Jeff Washington (jwash)
7631011d8c AcctIdx: add flush remove stat (#20625) 2021-10-12 16:13:32 -05:00
Alexander Meißner
06cad19c91 Adds missing program_id in InvokeContext::push() calls. (#20621) 2021-10-12 19:26:55 +02:00
Tao Zhu
005d6863fd - move cost tracker into bank, so each bank has its own cost tracker; (#20527)
- move related modules to runtime
2021-10-12 08:51:33 -05:00
Jack May
b0ee5d6cfa Add debug guard to catch invalid params (#20603) 2021-10-11 16:47:17 -07:00
Jeff Washington (jwash)
0da677e213 AcctIdx: write disk only when removing from mem (#20578) 2021-10-11 14:47:06 -05:00
dependabot[bot]
9d94e43839 chore: bump ouroboros from 0.12.0 to 0.13.0 (#20591)
* chore: bump ouroboros from 0.12.0 to 0.13.0

Bumps [ouroboros](https://github.com/joshua-maros/ouroboros) from 0.12.0 to 0.13.0.
- [Release notes](https://github.com/joshua-maros/ouroboros/releases)
- [Commits](https://github.com/joshua-maros/ouroboros/commits)

---
updated-dependencies:
- dependency-name: ouroboros
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-10-11 11:54:37 -06:00
Jeff Washington (jwash)
a8e000a2a6 add filler accounts to bloat validator and predict failure (#20491)
* add filler accounts to bloat validator and predict failure

* assert no accounts match filler

* cleanup magic numbers

* panic if can't load from snapshot with filler accounts specified

* some renames

* renames

* into_par_iter

* clean filler accts, too
2021-10-11 12:46:27 -05:00
Alexander Meißner
f30f3bddbb Refactor: Remove program_id from process_instruction() (#20540)
* Replaces usage of program_id parameter by invoke_context.get_caller()?.

* Removes "pubkey: &Pubkey" parameter from "process_instruction()".
2021-10-10 22:29:18 +02:00
Jeff Washington (jwash)
8cf222b8bd AcctIdx: combine if (#20545) 2021-10-10 13:48:18 -05:00
Jeff Washington (jwash)
33d8c07364 AcctIdx: combine scan and update loops (#20546) 2021-10-10 13:47:52 -05:00
Justin Starry
db9336c99e Remove feature switch handling for checking vote init (#20557) 2021-10-08 23:35:26 -04:00
Jeff Washington (jwash)
0545306f0c AcctIdx: remove stats from write lock section (#20548) 2021-10-08 15:27:38 -05:00
Brooks Prumo
5440c1d2e1 SnapshotPackagerService pushes incremental snapshot hashes to CRDS (#20442)
Now that CRDS supports incremental snapshot hashes,
SnapshotPackagerService needs to push 'em!

This commit does two main things:

1. SnapshotPackagerService now knows about incremental snapshot hashes,
   and will push SnapshotPackage::IncrementalSnapshot hashes to CRDS.
2. At startup, when loading from a full + incremental snapshot, the
   hashes need to be passed all the way to SnapshotPackagerService so it
   can push these starting hashes to CRDS.  Those values have been piped
   through.

Fixes #20441 and #20423
2021-10-08 15:14:56 -05:00
Jeff Washington (jwash)
4f6a0b2650 AcctIdx: stats for buckets on disk, add median (#20528) 2021-10-08 13:58:38 -05:00
Jeff Washington (jwash)
2c3d52b4cc AcctIdx: avoid overcounting inserts (#20544) 2021-10-08 13:57:47 -05:00
Alexander Meißner
4e65487d2f Refactor: process_instruction() (#20448)
* Adds first_instruction_account parameter to process_instruction().

* Removes InvokeContext::remove_first_keyed_account() from all BPF loaders.

* Removes InvokeContext::remove_first_keyed_account() from all builtin programs.

* Removes InvokeContext::remove_first_keyed_account() from all mock ups.

* Deprecates InvokeContext::remove_first_keyed_account().

* Documents index base of keyed_account_at_index().

* Adds dynamic offset to call sites of "keyed_account_at_index()".
2021-10-08 11:41:07 +02:00
Jeff Washington (jwash)
069586c171 AcctIdx: set startup around clean and shrink at startup to flush cache (#20526) 2021-10-07 22:57:40 +00:00
Tao Zhu
177a375479 Tpu vote 1.7 (#20187) (#20494)
* Add separate vote processing tpu port

* Add feature to send to tpu vote port

* Add vote rejecting sigverify mode

* use packet.meta.is_simple_vote_tx in place of deserialization

* consolidate code that identifies vote tx atcommon path for cpu and gpu

* new key for feature set

* banking forward tpu vote

* add tpu vote port to dockerfile and other review changes

* Simplify thread id compare

* fix a test; updated cluster_info ABI change

Co-authored-by: Tao Zhu <tao@solana.com>

Co-authored-by: sakridge <sakridge@gmail.com>
2021-10-07 09:38:23 +00:00
Jeff Washington (jwash)
df39b37cb8 improve clean stats (#20469) 2021-10-06 20:04:26 -05:00
Trent Nelson
767f740305 Bump version to 1.9.0 2021-10-06 17:57:41 -07:00
Michael Vines
3b70cdb8d8 Upgrade to Rust 1.55.0 2021-10-06 17:48:58 -07:00
Jack May
785fcb63f5 Remove support for dynamically loaded native programs (#20444) 2021-10-06 14:53:23 -07:00
Michael Vines
7027d56064 Resolve nightly-2021-10-05 clippy complaints 2021-10-06 10:37:58 -07:00
Jeff Washington (jwash)
895b46e0c3 simplify AccountsDb::new_with_config (#20473) 2021-10-06 11:41:23 -05:00
Justin Starry
129716f3f0 Optimize stakes cache and rewards at epoch boundaries (#20432)
* Optimize stakes cache and rewards at epoch boundaries

* Fetch from accounts db

* Add cli flag for disabling epoch boundary optimization
2021-10-06 00:53:26 -04:00
Justin Starry
250a8503fe Make rewards tracer async friendly (#20452) 2021-10-05 21:30:08 -04:00
Jeff Washington (jwash)
d03bf2bbfe AcctIdx: metrics fixes (#20449) 2021-10-05 22:26:29 +00:00
Jeff Washington (jwash)
4267419349 AcctIdx: sleep thread when enough threads are running to complete the remaining work (#20446) 2021-10-05 16:48:23 -05:00
Jeff Washington (jwash)
0d98a91511 AcctIdx: metric for bucket grow time (#20445) 2021-10-05 20:22:01 +00:00
Jeff Washington (jwash)
64cf354651 Add clean metrics (#20410) 2021-10-04 18:21:45 -05:00
Jeff Washington (jwash)
2d78f8ad2a parallelize unref_from_storage from clean_dead_slots_from_accounts_index (#20411) 2021-10-04 18:20:10 -05:00
Justin Starry
0ddb34a0b4 Add struct and convenience methods to track stake activation status (#20392)
* Add struct and convenience methods to track stake activation status

* fix nits

* rename
2021-10-04 18:59:11 -04:00
Jeff Washington (jwash)
b6a6d6986d lift store.slot() from loop (#20409) 2021-10-04 16:07:01 -05:00
Jeff Washington (jwash)
8da2eb980a AcctIdx: introduce BucketApi for access to a specific bucket (#20359) 2021-10-04 11:48:09 -05:00
Jeff Washington (jwash)
e31c065544 AcctIdx: when disk index insert needs to grow, drop locks, grow, then restart flush (#20333) 2021-09-30 19:27:09 -04:00
Lijun Wang
fe97cb2ddf AccountsDb plugin framework (#20047)
Summary of Changes

Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows

Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.

The code comprises 4 major parts:

accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-09-30 14:26:17 -07:00
dependabot[bot]
9ef1f06b3f chore: bump ouroboros from 0.11.1 to 0.12.0 (#20344)
* chore: bump ouroboros from 0.11.1 to 0.12.0

Bumps [ouroboros](https://github.com/joshua-maros/ouroboros) from 0.11.1 to 0.12.0.
- [Release notes](https://github.com/joshua-maros/ouroboros/releases)
- [Commits](https://github.com/joshua-maros/ouroboros/commits)

---
updated-dependencies:
- dependency-name: ouroboros
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-30 13:20:05 -06:00
Jeff Washington (jwash)
8db9586599 AcctIdx: randomly evict items from in mem (#20319) 2021-09-30 02:51:54 +00:00
Jeff Washington (jwash)
5e05f12c48 AcctIdx: bucket perf improvements (#20328) 2021-09-30 01:53:26 +00:00