bb24318ef0
Document shreds ( #16514 )
...
No functionality changes from this commit
2021-04-16 14:04:46 -05:00
85eb37fab0
Merge pull request from GHSA-8v47-8c53-wwrc
...
* Track transaction check time separately from account loads
* banking packet process metrics
* Remove signature clone in status cache lookup
* Reduce allocations when converting packets to transactions
* Add blake3 hash of transaction messages in status cache
* Bug fixes
* fix tests and run fmt
* Address feedback
* fix simd tx entry verification
* Fix rebase
* Feedback
* clean up
* Add tests
* Remove feature switch and fall back to signature check
* Bump programs/bpf Cargo.lock
* clippy
* nudge benches
* Bump `BankSlotDelta` frozen ABI hash`
* Add blake3 to sdk/programs/Cargo.lock
* nudge bpf tests
* short circuit status cache checks
Co-authored-by: Trent Nelson <trent@solana.com >
2021-04-13 00:28:08 -06:00
54a04bac3d
Apple M1 compatibility ( #16346 )
...
Co-authored-by: Christian Drappi <christiandrappi@Christians-MacBook-Pro.local >
2021-04-09 17:21:01 -07:00
22a18a68e3
stops consuming pinned vectors with a recycler ( #16441 )
...
If the vector is pinned and has a recycler, From<PinnedVec>
implementation of Vec should clone (instead of consuming) the underlying
vector so that the next allocation of a PinnedVec will recycle an
already pinned one.
2021-04-09 16:55:24 +00:00
b08cff9e77
Simplify some pattern-matches ( #16402 )
...
When those match an exact combinator on Option / Result.
Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust ).
2021-04-08 12:40:37 -06:00
b8b6777262
Only get Blockstore::last_root once ( #16362 )
2021-04-05 04:14:02 +00:00
1a13d22984
Fixup iterator method ( #16357 )
2021-04-04 23:32:51 +00:00
4e5ef6bce2
Add cluster state verifier logging ( #16330 )
...
* Add cluster state verifier logging
* Add duplicate-slots iterator to ledger tool
2021-04-02 21:48:44 -07:00
da27acabcc
Rpc: enable getConfirmedSignaturesForAddress2 to return confirmed (not yet finalized) data ( #16281 )
...
* Update blockstore method to allow return of unfinalized signature
* Support confirmed sigs in getConfirmedSignaturesForAddress2
* Add deprecated comments
* Update docs
* Enable confirmed transaction-history in cli
* Return real confirmation_status; fill in not-yet-finalized block time if possible
2021-04-01 04:35:57 +00:00
18bd47dbe1
Rpc: fix getConfirmedTransaction slot ( #16288 )
...
* Fix transaction blockstore apis
* Update blockstore apis in rpc
2021-03-31 21:04:00 -06:00
3f63ed9a72
removes OrderedIterator and transaction batch iteration order ( #16153 )
...
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817
However load_and_execute_transactions is iterating over
lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.
If iteratorion_order is [1, 2, 0], and i is 0, then:
lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).
This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.
2021-03-31 23:59:19 +00:00
433f1ead1c
Rpc: enable getConfirmedBlock and getConfirmedTransaction to return confirmed (not yet finalized) data ( #16142 )
...
* Add Blockstore block and tx apis that allow unrooted responses
* Add TransactionStatusMessage, and send on bank freeze; also refactor TransactionStatusSender
* Track highest slot with tx-status writes complete
* Rename and unpub fn
* Add commitment to GetConfirmed input configs
* Support confirmed blocks in getConfirmedBlock
* Support confirmed txs in getConfirmedTransaction
* Update sigs-for-addr2 comment
* Enable confirmed block in cli
* Enable confirmed transaction in cli
* Review comments
* Rename blockstore method
2021-03-26 16:47:35 -06:00
4f4cffbd03
Throttle PoH ticks by cumulative slot time ( #16139 )
...
* Throttle PoH ticks by cumulative slot time
* respond to pr feedback
* saturating sub
* updated comment
2021-03-26 18:54:16 +00:00
e7fd7d46cf
rpc: add getSlotLeaders method ( #16057 )
2021-03-23 17:48:54 +00:00
4f82b897bc
buffers data shreds to make larger erasure coded sets ( #15849 )
...
Broadcast stage batches up to 8 entries:
https://github.com/solana-labs/solana/blob/79280b304/core/src/broadcast_stage/broadcast_utils.rs#L26-L29
which will be serialized into some number of shreds and chunked into FEC
sets of at most 32 shreds each:
https://github.com/solana-labs/solana/blob/79280b304/ledger/src/shred.rs#L576-L597
So depending on the size of entries, FEC sets can be small, which may
aggravate loss rate.
For example 16 FEC sets of 2:2 data/code shreds each have higher loss
rate than one 32:32 set.
This commit broadcasts data shreds immediately, but also buffers them
until it has a batch of 32 data shreds, at which point 32 coding shreds
are generated and broadcasted.
2021-03-23 14:52:38 +00:00
d76ad33597
Handle blockstore insert dup checks ( #16051 )
2021-03-22 16:18:22 -07:00
654449ce91
Bump tokio to 1.1 ( #15926 )
2021-03-16 21:32:50 +00:00
3b85cbc504
renames is_last_in_fec_set back to is_last_data ( #15848 )
...
https://github.com/solana-labs/solana/pull/10095
renamed is_last_data to is_last_in_fec_set. However, the code shows that
this is actually meant to indicate where the serialized data is
complete:
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L599-L600
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L229-L231
There are multiple FEC sets for each `&[Entry]` serialized and this flag
does not represent shreds last in FEC sets (only the very last one by
overlap). So the name is wrong and confusing
2021-03-16 10:09:16 +00:00
918d04e3f0
Add more slot update notifications ( #15734 )
...
* Add more slot update notifications
* fix merge
* Address feedback and add integration test
* switch to datapoint
* remove unused shred method
* fix clippy
* new thread for rpc completed slots
* remove extra constant
* fixes
* rely on channel closing
* fix check
2021-03-12 21:44:06 +08:00
1135ffd595
mut data refs as slice ( #15782 )
2021-03-10 15:28:03 -06:00
8a3135d17b
Account->AccountSharedData ( #15691 )
2021-03-09 15:06:07 -06:00
9448540ea6
log timing after ledger process ( #15716 )
2021-03-05 11:01:52 -06:00
7e65289729
Convert blockstore TransactionStatus column family to protobufs ( #15733 )
...
* Prevent panic if TransactionStatus can't be deserialized
* Convert Blockstore TransactionStatus column to protobuf
* Add compatability test
2021-03-05 09:05:35 -07:00
be35c1c1b7
add execute detail timings ( #15638 )
2021-03-03 17:07:45 -06:00
f1223fb783
Lower blockstore processor error severity ( #15578 )
2021-03-01 14:57:37 -08:00
33eaa2b238
Sort forks in "ledger processed..." log message
2021-02-28 17:32:50 -08:00
e92cff3efa
create-snapshot subcommad now accepts the ROOT keyword
2021-02-26 16:40:10 -08:00
c2e8814dce
Add limit and shrink policy for recycler ( #15320 )
2021-02-24 00:15:58 -08:00
5df36aec7d
Pacify clippy
2021-02-19 20:08:41 -08:00
170cb792eb
Return blockstore error if previous_blockhash cannot be determined ( #15382 )
...
* Return blockstore error if previous_blockhash cannot be determined
* Add require_previous_blockshash flag
2021-02-18 01:04:52 +00:00
7f7370c306
Re-allow clippy::integer_arithmetic at crate-level
2021-02-17 13:55:08 -07:00
98e3e570d2
Add --force arg for bigtable upload
2021-02-16 17:24:07 -08:00
e403aeaf05
adds an inverted index to leader schedule ( #15249 )
...
next_leader_slot is doing a linear search for slots in which a pubkey is
the leader:
https://github.com/solana-labs/solana/blob/e59a24d9f/ledger/src/leader_schedule_cache.rs#L123-L157
This can be done more efficiently by adding an inverted index to leader
schedule.
2021-02-15 00:52:52 +00:00
0812931c38
Log if unsanitary transactions are read from blockstore ( #15319 )
2021-02-14 06:32:43 +00:00
da6753b8c0
Warp timestamp and extend max-allowable-drift for accommodate slow blocks ( #15204 )
...
* Remove timestamp_correction feature gating
* Remove timestamp_bounding feature gating
* Remove unused deprecated ledger code
* Remove unused deprecated unbounded-timestamp code
* Enable independent adjustment of fast/slow timestamp bounding
* Update timestamp bounds to 25% fast, 80% slow; warp timestamp
* Update bank hash test
* Add PR number to feature
Co-authored-by: Michael Vines <mvines@gmail.com >
Co-authored-by: Michael Vines <mvines@gmail.com >
2021-02-09 15:49:00 -07:00
bbae23358c
ledger-tool cleanup and additions ( #15179 )
...
* Plumb allow-dead-slots to ledger-tool verify
* ledger-tool cleanup and add some useful missing args
Print root slots and how many unrooted past last root.
2021-02-06 17:26:42 -08:00
774416a546
Ledger purge printing ( #15176 )
2021-02-06 15:45:08 -08:00
d1563f0ccd
Bump tonic, prost, tarpc, tokio ( #15013 )
...
* Update tonic & prost, and regenerate proto
* Reignore doc code
* Revert pull #14367 , but pin tokio to v0.2 for jsonrpc
* Bump backoff and goauth -> and therefore tokio
* Bump tokio in faucet, net-utils
* Bump remaining tokio, plus tarpc
2021-02-05 00:21:53 -07:00
42bf6dc2ab
Revert hard nofile limit back to 500000
2021-02-03 19:43:55 +00:00
8376781ec8
Don't squash caught errors, please ( #15046 )
...
* Don't squash caught errors, please
* Update blockstore.rs
* Update blockstore.rs
2021-02-03 14:42:34 +00:00
894b412aef
docs: bump nofiles recommendations to match maps
2021-02-02 21:59:21 -07:00
cbb8b79a60
Add validator flag to opt in to cpi and logs storage ( #14922 )
...
* Add validator flag to opt in to cpi and logs storage
* Default TestValidator to opt-in; allow using in multinode-demo
* No clone
Co-authored-by: Carl Lin <carl@solana.com >
2021-02-01 14:00:51 -07:00
cbffab7850
Upgrade to Rust v1.49.0
2021-01-23 19:16:36 -08:00
191193289f
Upgrade sha2 to 0.9.3 ( #14746 )
2021-01-22 22:25:22 -08:00
1de6d28eaf
Add block_time to getConfirmedSignaturesForAddress2 and getConfirmedTransaction ( #14572 )
...
* add block_time to get_confirmed_signatures_for_address2 and protobuf implementation for tx_by_addr
* add tests for convert
* update cargo lock
* run cargo format after rebase
* introduce legacy TransactionByAddrInfo
* move LegacyTransactionByAddrInfo back to storage-bigtable
2021-01-20 22:10:35 -08:00
907f518f6d
Add load/execute/store timings ( #14561 )
2021-01-14 14:14:16 -08:00
e95ebcf864
Don't stop to find newer cluster-confirmed roots ( #14557 )
...
* Don't stop to find newer cluster-confirmed roots
* Fix and add new tests
* nits
2021-01-13 15:47:06 +00:00
6dfad0652f
Cache account stores, flush from AccountsBackgroundService ( #13140 )
2021-01-11 17:00:23 -08:00
d8105bb7d7
Add rocskdb high priority threads ( #14515 )
...
Without them, memtable writes can stall on compactions.
2021-01-11 13:32:02 -08:00
86c81a0ba2
Humanize the 'ledger processed...' time
2021-01-08 22:56:21 -08:00
