Commit Graph

2578 Commits

Author SHA1 Message Date
c43b6e0248 log metric for clean_accounts.delta_insert_us (#20634) 2021-10-14 08:37:14 -05:00
9b042fed30 AcctIdx: support unref first class for perf (#20638) 2021-10-14 08:36:53 -05:00
da45be366a Remove blockhash from fee calculation (#20641) 2021-10-13 13:10:58 -07:00
220fd41bbc fix unstable test (#20645) 2021-10-13 11:43:18 -05:00
aec9d8bf2f add clean stats (#20631) 2021-10-13 08:22:11 -05:00
1d813ea078 Refactor: Make program_id always last in program chain (#20598)
* Replaces program_id field in InvokeContextStackFrame by index.

* Swaps order of program account and programdata account.

* Removes program_id parameter from InvokeContext::push().
2021-10-13 08:58:20 +02:00
7631011d8c AcctIdx: add flush remove stat (#20625) 2021-10-12 16:13:32 -05:00
06cad19c91 Adds missing program_id in InvokeContext::push() calls. (#20621) 2021-10-12 19:26:55 +02:00
005d6863fd - move cost tracker into bank, so each bank has its own cost tracker; (#20527)
- move related modules to runtime
2021-10-12 08:51:33 -05:00
b0ee5d6cfa Add debug guard to catch invalid params (#20603) 2021-10-11 16:47:17 -07:00
0da677e213 AcctIdx: write disk only when removing from mem (#20578) 2021-10-11 14:47:06 -05:00
9d94e43839 chore: bump ouroboros from 0.12.0 to 0.13.0 (#20591)
* chore: bump ouroboros from 0.12.0 to 0.13.0

Bumps [ouroboros](https://github.com/joshua-maros/ouroboros) from 0.12.0 to 0.13.0.
- [Release notes](https://github.com/joshua-maros/ouroboros/releases)
- [Commits](https://github.com/joshua-maros/ouroboros/commits)

---
updated-dependencies:
- dependency-name: ouroboros
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <you@example.com>
2021-10-11 11:54:37 -06:00
a8e000a2a6 add filler accounts to bloat validator and predict failure (#20491)
* add filler accounts to bloat validator and predict failure

* assert no accounts match filler

* cleanup magic numbers

* panic if can't load from snapshot with filler accounts specified

* some renames

* renames

* into_par_iter

* clean filler accts, too
2021-10-11 12:46:27 -05:00
f30f3bddbb Refactor: Remove program_id from process_instruction() (#20540)
* Replaces usage of program_id parameter by invoke_context.get_caller()?.

* Removes "pubkey: &Pubkey" parameter from "process_instruction()".
2021-10-10 22:29:18 +02:00
8cf222b8bd AcctIdx: combine if (#20545) 2021-10-10 13:48:18 -05:00
33d8c07364 AcctIdx: combine scan and update loops (#20546) 2021-10-10 13:47:52 -05:00
db9336c99e Remove feature switch handling for checking vote init (#20557) 2021-10-08 23:35:26 -04:00
0545306f0c AcctIdx: remove stats from write lock section (#20548) 2021-10-08 15:27:38 -05:00
5440c1d2e1 SnapshotPackagerService pushes incremental snapshot hashes to CRDS (#20442)
Now that CRDS supports incremental snapshot hashes,
SnapshotPackagerService needs to push 'em!

This commit does two main things:

1. SnapshotPackagerService now knows about incremental snapshot hashes,
   and will push SnapshotPackage::IncrementalSnapshot hashes to CRDS.
2. At startup, when loading from a full + incremental snapshot, the
   hashes need to be passed all the way to SnapshotPackagerService so it
   can push these starting hashes to CRDS.  Those values have been piped
   through.

Fixes #20441 and #20423
2021-10-08 15:14:56 -05:00
4f6a0b2650 AcctIdx: stats for buckets on disk, add median (#20528) 2021-10-08 13:58:38 -05:00
2c3d52b4cc AcctIdx: avoid overcounting inserts (#20544) 2021-10-08 13:57:47 -05:00
4e65487d2f Refactor: process_instruction() (#20448)
* Adds first_instruction_account parameter to process_instruction().

* Removes InvokeContext::remove_first_keyed_account() from all BPF loaders.

* Removes InvokeContext::remove_first_keyed_account() from all builtin programs.

* Removes InvokeContext::remove_first_keyed_account() from all mock ups.

* Deprecates InvokeContext::remove_first_keyed_account().

* Documents index base of keyed_account_at_index().

* Adds dynamic offset to call sites of "keyed_account_at_index()".
2021-10-08 11:41:07 +02:00
069586c171 AcctIdx: set startup around clean and shrink at startup to flush cache (#20526) 2021-10-07 22:57:40 +00:00
177a375479 Tpu vote 1.7 (#20187) (#20494)
* Add separate vote processing tpu port

* Add feature to send to tpu vote port

* Add vote rejecting sigverify mode

* use packet.meta.is_simple_vote_tx in place of deserialization

* consolidate code that identifies vote tx atcommon path for cpu and gpu

* new key for feature set

* banking forward tpu vote

* add tpu vote port to dockerfile and other review changes

* Simplify thread id compare

* fix a test; updated cluster_info ABI change

Co-authored-by: Tao Zhu <tao@solana.com>

Co-authored-by: sakridge <sakridge@gmail.com>
2021-10-07 09:38:23 +00:00
df39b37cb8 improve clean stats (#20469) 2021-10-06 20:04:26 -05:00
767f740305 Bump version to 1.9.0 2021-10-06 17:57:41 -07:00
3b70cdb8d8 Upgrade to Rust 1.55.0 2021-10-06 17:48:58 -07:00
785fcb63f5 Remove support for dynamically loaded native programs (#20444) 2021-10-06 14:53:23 -07:00
7027d56064 Resolve nightly-2021-10-05 clippy complaints 2021-10-06 10:37:58 -07:00
895b46e0c3 simplify AccountsDb::new_with_config (#20473) 2021-10-06 11:41:23 -05:00
129716f3f0 Optimize stakes cache and rewards at epoch boundaries (#20432)
* Optimize stakes cache and rewards at epoch boundaries

* Fetch from accounts db

* Add cli flag for disabling epoch boundary optimization
2021-10-06 00:53:26 -04:00
250a8503fe Make rewards tracer async friendly (#20452) 2021-10-05 21:30:08 -04:00
d03bf2bbfe AcctIdx: metrics fixes (#20449) 2021-10-05 22:26:29 +00:00
4267419349 AcctIdx: sleep thread when enough threads are running to complete the remaining work (#20446) 2021-10-05 16:48:23 -05:00
0d98a91511 AcctIdx: metric for bucket grow time (#20445) 2021-10-05 20:22:01 +00:00
64cf354651 Add clean metrics (#20410) 2021-10-04 18:21:45 -05:00
2d78f8ad2a parallelize unref_from_storage from clean_dead_slots_from_accounts_index (#20411) 2021-10-04 18:20:10 -05:00
0ddb34a0b4 Add struct and convenience methods to track stake activation status (#20392)
* Add struct and convenience methods to track stake activation status

* fix nits

* rename
2021-10-04 18:59:11 -04:00
b6a6d6986d lift store.slot() from loop (#20409) 2021-10-04 16:07:01 -05:00
8da2eb980a AcctIdx: introduce BucketApi for access to a specific bucket (#20359) 2021-10-04 11:48:09 -05:00
e31c065544 AcctIdx: when disk index insert needs to grow, drop locks, grow, then restart flush (#20333) 2021-09-30 19:27:09 -04:00
fe97cb2ddf AccountsDb plugin framework (#20047)
Summary of Changes

Create a plugin mechanism in the accounts update path so that accounts data can be streamed out to external data stores (be it Kafka or Postgres). The plugin mechanism allows

Data stores of connection strings/credentials to be configured,
Accounts with patterns to be streamed
PostgreSQL implementation of the streaming for different destination stores to be plugged in.

The code comprises 4 major parts:

accountsdb-plugin-intf: defines the plugin interface which concrete plugin should implement.
accountsdb-plugin-manager: manages the load/unload of plugins and provide interfaces which the validator can notify of accounts update to plugins.
accountsdb-plugin-postgres: the concrete plugin implementation for PostgreSQL
The validator integrations: updated streamed right after snapshot restore and after account update from transaction processing or other real updates.
The plugin is optionally loaded on demand by new validator CLI argument -- there is no impact if the plugin is not loaded.
2021-09-30 14:26:17 -07:00
9ef1f06b3f chore: bump ouroboros from 0.11.1 to 0.12.0 (#20344)
* chore: bump ouroboros from 0.11.1 to 0.12.0

Bumps [ouroboros](https://github.com/joshua-maros/ouroboros) from 0.11.1 to 0.12.0.
- [Release notes](https://github.com/joshua-maros/ouroboros/releases)
- [Commits](https://github.com/joshua-maros/ouroboros/commits)

---
updated-dependencies:
- dependency-name: ouroboros
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
2021-09-30 13:20:05 -06:00
8db9586599 AcctIdx: randomly evict items from in mem (#20319) 2021-09-30 02:51:54 +00:00
5e05f12c48 AcctIdx: bucket perf improvements (#20328) 2021-09-30 01:53:26 +00:00
59112adac1 AcctIdx: us stats were showing ms (#20324) 2021-09-29 22:18:07 +00:00
2ec74474e9 AcctIdx: optimize should_remove_from_mem (#20327) 2021-09-29 16:41:54 -05:00
4de5fff3ca Simplifies return_data accessors in InvokeContext. (#20290) 2021-09-29 19:11:06 +02:00
57c8abf499 Refactor: Merge message processor into invoke context (#20308)
* Inlines MessageProcessor::execute_instruction() in MessageProcessor::process_message().

* Moves MessageProcessor::create_pre_accounts() into ThisInvokeContext::push().

* Move instruction_recorders slice into InvokeContext.

* Makes account_indices optional in InvokeContext::push().

* Separates initial InvokeContext::push() from ThisInvokeContext::new().

* invoke_context.pop() the base invocation frame.

* Zip message.instructions.iter() and program_indices.iter().

* Moves ThisInvokeContext::new() to the beginning of the loop inside MessageProcessor::process_message().

* Hoists ThisInvokeContext::new() out of loop inside MessageProcessor::process_message().

* Removes unnecessary clone() from ThisInvokeContext::new() in MessageProcessor ::process_message().

* Stop ignoring errors from MessageProcessor::create_pre_accounts().

* Moves MessageProcessor::verify_account_references() and MessageProcessor::verify() into InvokeContext::verify().
2021-09-29 12:05:25 -05:00
062d8e6c5a AcctIdx: fix logic on should_remove_from_mem (#20302) 2021-09-29 13:07:40 +00:00