Commit Graph

1353 Commits

Author SHA1 Message Date
Justin Starry
0596cf5405 Remove unprocessed transactions from log notifications (#16349) 2021-04-04 23:44:59 +08:00
Jeff Washington (jwash)
3996b699dc read only account cache for executable accounts - improve replay (#16150)
* read only account cache

* tests

* clippy

* cleanup

* new file, add tests

* remove copy/paste code from test

* remove dead code

* all loads use cache

* remove stale comments

* add metrics logging for read only cache size

* report read only cache hits and misses

* consistency

* formatting

* rename, add comment

* u64

* better interaction with existing cache

* lru list saved between cleans
2021-04-01 07:16:34 -05:00
behzad nouri
3f63ed9a72 removes OrderedIterator and transaction batch iteration order (#16153)
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817

However load_and_execute_transactions is iterating over
  lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.

If iteratorion_order is [1, 2, 0], and i is 0, then:
  lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
  transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).

This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.
2021-03-31 23:59:19 +00:00
Jeff Washington (jwash)
416312b30b update timings only on outer call (#16275)
* update timings only on outer call

* test & format
2021-03-31 18:02:59 -05:00
Jeff Washington (jwash)
f374b35944 Hash stored accounts in bg (#16157)
* lazy calculate account hash

* push to bg thread

* remove deadlock

* logs

* format

* some cleanup on aisle 9

* format, fix up some metrics

* fix test, remove legacy function only there for tests

* cleanup

* remove unused store_hasher

* Switch to crossbeam

* clippy

* format

* use iter()

* rework from feedback

* hash_slot -> slot

* hash(cluster_type)

Co-authored-by: Carl Lin <carl@solana.com>
2021-03-31 15:39:34 -05:00
Jeff Washington (jwash)
a5dcee254d get rid of unnecessary data copy (#16214)
* get rid of unnecessary data copy

* preserve rent_epoch
2021-03-31 09:11:39 -05:00
sakridge
54c68ea83f Drop write lock on sysvars (#15497)
* Drop write lock on sysvars

* adds env var for demoting sysvar write lock demotion

* moves demote logic to is_writable

* feature gates sysvar write lock demotion

* adds builtins to write lock demotion

* adds system program id to builtins

* adds Feature111...

* adds an abi-freeze test

* mvines set of builtin program keys

Co-authored-by: Michael Vines <mvines@gmail.com>

* update tests

* adds bpf loader keys

* Add test sysvar

* Plumb demote_sysvar to is_writable

* more plumbing of demote_sysvar_write_locks to is_writable

* patches test_program_bpf_instruction_introspection

* hard codes demote_sysvar_write_locks to false for serialization/encoding methods

* Revert "hard codes demote_sysvar_write_locks to false for serialization/encoding methods"

This reverts commit ae3e2d2e777437bddd753933097a210dcbc1b1fc.

* change the hardcoded ones to demote_sysvar_write_locks=true

* Use data_as_mut_slice

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2021-03-30 10:05:09 -07:00
sakridge
5e5b63712b Status cache improvements (#16174) 2021-03-28 19:10:14 -07:00
Jeff Washington (jwash)
66c42f62d8 make AccountSharedData.data private to abstract storage (#16091)
* format

* 2 more tests

* use
2021-03-25 11:04:20 -05:00
Ryo Onodera
6d5c6c17c5 Simplify account.rent_epoch handling for sysvar rent (#16049)
* Add some code for special local testing

* Add comment to store_account_and_update_capitalization

* Simplify account.rent_epoch handling for sysvar rent

* Introduce *_for_test functions

* Add deprecation messages to existing api
2021-03-25 15:23:20 +09:00
sakridge
26af2d4c26 Up store limit for test_account_grow (#16099)
There can be more than 5 stores
2021-03-24 15:24:17 -07:00
Jeff Washington (jwash)
2d24d13046 AccountsSharedData: data copy on write (#15800)
* Arc<AccountData>

* try custom serializer

* adapt test from Behzad's change

* clippy

* simplify serialization

* remove abi example derive

* refactor 'take'

* remove serialization

* remove serialize calls

* remove account_data

* remove intos

* remove left over file
2021-03-23 15:19:31 -05:00
Jeff Washington (jwash)
fde43a906d revert test to use Account for serialization (#16063) 2021-03-22 22:48:09 +00:00
Jeff Washington (jwash)
4beb39f7a1 multi-pass bin scanning (#15377)
* multi-pass bin scanning

* pr feedback

* format

* fix typo

* adjust metrics for code changes

* merge errors
2021-03-18 10:32:07 -05:00
carllin
f548a04fae Allow unbounded wallclock processing time in tests (#15961) 2021-03-17 15:48:50 -07:00
DimAn
a5d144b00f Revert to removing only tmp- 2021-03-17 10:31:01 -07:00
DimAn
0209d334bd Add option for separate snapshot location
(cherry picked from commit 6126878f509c69e23480a5ec22b3271e2b16e072)
2021-03-17 10:31:01 -07:00
Michael Vines
a2eb655322 =1.7.0 2021-03-16 07:51:07 +00:00
Jeff Washington (jwash)
efee8b62d7 a few missed set_data calls (#15846)
* a few missed set_data calls

* another set data call
2021-03-15 21:57:23 -05:00
Jeff Washington (jwash)
e35d738906 rework BankClient.get_account_data (#15853) 2021-03-15 19:27:53 -05:00
carllin
c1ba265dd9 Wallclock BankingStage Throttle (#15731) 2021-03-15 17:11:15 -07:00
Michael Vines
0c9ca5522c Bump version to v1.7.0 2021-03-13 09:01:21 +00:00
Justin Starry
918d04e3f0 Add more slot update notifications (#15734)
* Add more slot update notifications

* fix merge

* Address feedback and add integration test

* switch to datapoint

* remove unused shred method

* fix clippy

* new thread for rpc completed slots

* remove extra constant

* fixes

* rely on channel closing

* fix check
2021-03-12 21:44:06 +08:00
Ryo Onodera
4bbeb9c033 Remove old feature: simple_capitalization (#15763)
* Remove old feature: simple_capitalization

* Fix another failing test in core

* Finish up test cleanup

* Further clean up a bit
2021-03-12 11:12:40 +09:00
Jeff Washington (jwash)
952c3bcbb7 AccountSharedData construction (#15790) 2021-03-11 18:09:04 -06:00
Jeff Washington (jwash)
3419a5446e AccountSharedData.set_data (#15781)
* account.set_data and resize_data

* remove data_resize
2021-03-11 16:40:45 -06:00
Michael Vines
1061d021c9 Unpack snapshot AppendVecs directly into account paths 2021-03-11 08:25:24 -08:00
Trent Nelson
24d18b3cf2 docs: add docs links for crates published to crates.io 2021-03-11 08:38:18 +00:00
Jack May
478518308d Share RO and Executable accounts within invocations (#15799) 2021-03-10 23:04:00 -08:00
Michael Vines
65dd177601 AppendVec cleanup: remove dead code, rename non-idiomatic "new" function 2021-03-10 14:50:59 -08:00
Jeff Washington (jwash)
1135ffd595 mut data refs as slice (#15782) 2021-03-10 15:28:03 -06:00
Tyera Eulberg
9c1198c0c7 Improve load_largest_accounts more (#15785)
* Add load_largest_accounts bench

* Check lamports before address filter

* Use BinaryHeap, add Accounts test

* Use pubkey reference in the min-heap

Also, flatten code with early returns

Co-authored-by: Greg Fitzgerald <greg@solana.com>
2021-03-10 11:22:02 -07:00
Jack May
369e13b111 cleanup old runtime features (#15787) 2021-03-10 09:48:41 -08:00
Tyera Eulberg
3355efbb5a Fix test to use AccountSharedData 2021-03-09 20:47:54 -07:00
Tyera Eulberg
5991cef5f5 Improve load_largest_accounts and add Bank test (#15775) 2021-03-10 00:23:41 +00:00
Jeff Washington (jwash)
52e54e1100 account.data -> data() (#15778) 2021-03-09 22:31:33 +00:00
Jeff Washington (jwash)
8a3135d17b Account->AccountSharedData (#15691) 2021-03-09 15:06:07 -06:00
Tyera Eulberg
e3e8179f2d Sdk: start to appease clippy's integer arithmetic check (#15736)
* Move to module-level clippy allowance

* Fix stake_weighted_timestamp math

* Fix genesis_config

* Fix shred_version

* Fix hard_forks

* Fix process_instruction

* Add ArithmeticOverflow ix error

* Fix nonce_keyed_account

* Update BankSlotDelta frozen abi due to new ix error
2021-03-08 18:37:57 -07:00
Ryo Onodera
beac2d6d3f Remove stale comment 2021-03-08 10:18:32 +09:00
Ryo Onodera
8b0c6db871 Remove old feature: cumulative_rent_related_fixes (#15754) 2021-03-08 09:58:50 +09:00
sakridge
c5371fdc85 More debug on shrink and flush (#15518) 2021-03-05 14:10:34 -08:00
Jeff Washington (jwash)
13b9bb6790 refactor function to store into account after data is finished (#15710) 2021-03-05 11:02:35 -06:00
Greg Fitzgerald
edd159e7d7 Fix typos (#15721) 2021-03-05 07:16:53 +00:00
Tyera Eulberg
872f7117c3 Use OrderedIterator to produce TransactionLogInfo (#15712)
* Add failing test

* Fix iteration_order issue with stored logs
2021-03-04 15:26:19 -07:00
Michael Vines
1e2f5a5f55 Permit the snapshots/status_cache file to be sparse 2021-03-04 12:36:34 -08:00
Jack May
b53cb8eb2d Deprecate Instruction::new (#15695) 2021-03-04 05:46:48 +00:00
Jeff Washington (jwash)
be35c1c1b7 add execute detail timings (#15638) 2021-03-03 17:07:45 -06:00
Greg Fitzgerald
9bbe015a05 Document and clean up AppendVec (#15640)
* Document AppendVec

* Remove the almost-duplicate state in AppendVec

AppendVec was maintaining two offsets, `current_len` and
`append_offset`. Despite the different looking names, the two
values have the same meaning, but were updated at slightly different
times. When appending a batch of accounts, `current_len` updates would
be immediately available to other threads after each append, whereas
`append_offset` would only be updated after its mutex was unlocked.
`append_offset` is redundant. By removing it, we eliminate potential
bugs and no longer need to suppress clippy warnings.

* Remove get_mut() from AppendVec design

Only the offset into the AppendVec memory is thread-safe. The memory
itself is only threadsafe because it is append-only and is otherwise
unprotected. Adding get_mut() would only be safe if the memory was
protected by a ReadWrite lock.
2021-03-03 12:51:48 -08:00
Jack May
a9c8dbfd0c Add error reporting to system program (#15644) 2021-03-02 21:44:25 -08:00
Jeff Washington (jwash)
a2c47db920 extract expected cluster type (#15615)
* extract expected cluster type

* fix clippy warning

* more clippy
2021-03-02 13:55:34 -06:00