540 Commits

Author SHA1 Message Date
Tyera Eulberg
1201ef172e Bump version to v1.6.14 (#18050) 2021-06-17 20:42:10 +00:00
Ryo Onodera
337de51088 Bump version to v1.6.13 (#17972) 2021-06-15 23:47:22 +09:00
mergify[bot]
24ee0b3934 Avoid full-range compactions with periodic filtered b.g. ones (backport #16697) (#17956)
* Avoid full-range compactions with periodic filtered b.g. ones (#16697)

* Update rocksdb to v0.16.0

* Promote the infrequent and important log to info!

* Force background compaction by ttl without manual compaction

* Fix test

* Support no compaction mode in test_ledger_cleanup_compaction

* Fix comment

* Make compaction_interval customizable

* Avoid major compaction with periodic filtering...

* Adress lazy_static, special cfs and range check

* Clean up a bit and add comment

* Add comment

* More comments...

* Config code cleanup

* Add comment

* Use .conflicts_with()

* Nullify unneeded delete_range ops for special CFs

* Some clean ups

* Clarify the locking intention

* Ensure special CFs' consistency with PurgeType::CompactionFilter

* Fix comment

* Fix bad copy paste

* Fix various types...

* Don't use tuples

* Add a unit test for compaction_filter

* Fix typo...

* Remove flag and just use new behavior always

* Fix wrong condition negation...

* Doc. about no set_last_purged_slot in purge_slots

* Write a test and fix off-by-one bug....

* Apply suggestions from code review

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>

* Follow up to github review suggestions

* Fix line-wrapping

* Fix conflict

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
(cherry picked from commit 1f97b2365f)

# Conflicts:
#	Cargo.lock
#	ledger/src/blockstore_db.rs

* Fix conflicts

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2021-06-15 08:49:13 +00:00
Tyera Eulberg
3a3454d788 Bump version to v1.6.12 (#17651) 2021-06-01 21:40:36 -06:00
mergify[bot]
0e3131f2b4 Purge expired BlockHeight data from blockstore (backport #17634) (#17640)
* Purge expired BlockHeight data from blockstore (#17634)

* Purge expired BlockHeight data from blockstore

* Also call compact_storage and add comment....

(cherry picked from commit 96cdbfdcc0)

# Conflicts:
#	ledger/src/blockstore_db.rs

* Fix conflict

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-06-02 00:04:12 +00:00
mergify[bot]
01e2d5cd35 Add block height to ConfirmedBlock structs (backport #17523) (#17534)
* Add block height to ConfirmedBlock structs (#17523)

* Add BlockHeight CF to blockstore

* Rename CacheBlockTimeService to be more general

* Cache block-height using service

* Fixup previous proto mishandling

* Add block_height to block structs

* Add block-height to solana block

* Fallback to BankForks if block time or block height are not yet written to Blockstore

* Add docs

* Review comments

(cherry picked from commit ab581dafc2)

# Conflicts:
#	core/src/replay_stage.rs
#	core/src/tvu.rs
#	core/src/validator.rs

* Fix conflicts

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-05-26 22:42:46 -07:00
mergify[bot]
f364956d15 Plumb transaction-level rewards (aka "rent debits") into the getTransaction RPC method (backport #17528) (#17532)
* Plumb transaction-level rewards (aka "rent debits") into the `getTransaction` RPC method

(cherry picked from commit 9541411c15)

# Conflicts:
#	docs/src/developing/clients/jsonrpc-api.md

* rebase

Co-authored-by: Michael Vines <mvines@gmail.com>
2021-05-26 21:33:30 -07:00
Tyera Eulberg
ae1687bc0a Bump version to v1.6.11 (#17484) 2021-05-25 15:35:50 -06:00
Tyera Eulberg
dd2d119d2b v1.6: Ensure cluster-confirmed roots are set on boot (#17442)
* Add blockstore-root-scan for api nodes on boot

* Ensure cluster-confirmed root and parents are set as root in blockstore in load_frozen_forks()

* Plumb rpc-scan-and-fix-roots validator flag
2021-05-24 20:16:37 +00:00
Tyera Eulberg
409ac4dcfa Bump version to v1.6.10 (#17250) 2021-05-15 01:47:56 +00:00
steviez
9aacd0f3c3 Zero pad data shreds on fetch from blockstore (#17147)
* Zero pad data shreds on fetch from blockstore

This is a partial backport of #16602 to allow compatibility with that change.

* Remove size check and resize shreds to consistent length
2021-05-14 16:18:00 -05:00
mergify[bot]
733ef4b0b8 type AccountSecondaryIndexes = HashSet (backport #17108) (#17149)
* type AccountSecondaryIndexes = HashSet (#17108)

(cherry picked from commit f39dda00e0)

# Conflicts:
#	runtime/benches/accounts.rs
#	runtime/src/accounts.rs
#	runtime/src/accounts_db.rs
#	runtime/src/accounts_index.rs

* resolve merge errors

Co-authored-by: Jeff Washington (jwash) <75863576+jeffwashington@users.noreply.github.com>
Co-authored-by: Jeff Washington (jwash) <wash678@gmail.com>
2021-05-10 20:55:33 +00:00
mergify[bot]
0cf83887c6 Move block-time caching earlier (#17109) (#17150)
* Require that blockstore block-time only be recognized slot, instead of root

* Move cache_block_time to after Bank freeze

* Single use statement

* Pass transaction_status_sender by reference

* Remove unnecessary slot-existence check before caching block time altogether

* Move block-time existence check into Blockstore::cache_block_time, Blockstore no longer needed in blockstore_processor helper

(cherry picked from commit 6e9deaf1bd)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-05-10 20:31:56 +00:00
Michael Vines
65e1b881f9 Bump version to v1.6.9 2021-05-08 06:28:08 +00:00
Michael Vines
524b380a71 Bump version to 1.6.8 2021-05-04 12:46:57 -07:00
mergify[bot]
5a3bf5c90e limits to data_header.size when combining shreds' payloads (backport #16708) (#16870)
* limits to data_header.size when combining shreds' payloads (#16708)

Shredder::deshred is ignoring data_header.size when combining shreds' payloads:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/shred.rs#L940-L961

Also adding more sanity checks on the alignment of data shreds indices.

(cherry picked from commit 0f3ac51cf1)

# Conflicts:
#	ledger/src/shred.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-04-27 14:44:58 +00:00
mergify[bot]
786fa4f22e removes first_coding_index from erasure recovery code (#16646) (#16770)
first_coding_index is the same as the set_index and is so redundant:
https://github.com/solana-labs/solana/blob/37b8587d4/ledger/src/blockstore_meta.rs#L49-L60

(cherry picked from commit 03194145c0)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-04-23 13:21:27 +00:00
mergify[bot]
ae605f8f02 expands number of erasure coding shreds in the last batch in slots (backport #16484) (#16707)
* expands number of erasure coding shreds in the last batch in slots (#16484)

Number of parity coding shreds is always less than the number of data
shreds in FEC blocks:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L719

Data shreds are batched in chunks of 32 shreds each:
https://github.com/solana-labs/solana/blob/6907a2366/ledger/src/shred.rs#L714

However the very last batch of data shreds in a slot can be small, in
which case the loss rate can be exacerbated.

This commit expands the number of coding shreds in the last FEC block in
slots to: 64 - number of data shreds; so that FEC blocks are always 64
data and parity coding shreds each.

As a consequence of this, the last FEC block has more parity coding
shreds than data shreds. So for some shred indices we will have a coding
shred but no data shreds. This should not cause any kind of overlapping
FEC blocks as in:
https://github.com/solana-labs/solana/pull/10095
since this is done only for the very last batch in a slot, and the next
slot will reset the shred index.

(cherry picked from commit 37b8587d4e)

# Conflicts:
#	core/benches/shredder.rs
#	ledger/src/shred.rs

* removes backport merge conflicts

* ignore the flaky test for now

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-04-21 15:25:26 +00:00
Michael Vines
f59d4f29d9 clippy 2021-04-19 04:12:16 +00:00
Trent Nelson
4e94446fc3 Bump version to v1.6.7 2021-04-16 23:31:30 +00:00
Michael Vines
4ef3a679a4 Bump version to v1.6.6 2021-04-14 10:27:02 -07:00
Justin Starry
579065443a v1.6: Use blake3 message hash in status cache (#16507) 2021-04-13 16:57:20 +08:00
mergify[bot]
723e7f11b9 Simplify some pattern-matches (#16402) (#16446)
When those match an exact combinator on Option / Result.

Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust).

(cherry picked from commit b08cff9e77)

Co-authored-by: François Garillot <4142+huitseeker@users.noreply.github.com>
2021-04-08 20:45:01 +00:00
mergify[bot]
443f132de5 Add cluster state verifier logging (#16330) (#16336)
* Add cluster state verifier logging

* Add duplicate-slots iterator to ledger tool

(cherry picked from commit 4e5ef6bce2)

Co-authored-by: carllin <carl@solana.com>
2021-04-06 01:25:12 +00:00
Michael Vines
270749185c Adjust tokio version to just "1"
(cherry picked from commit 43feef7362)

# Conflicts:
#	faucet/Cargo.toml
#	net-utils/Cargo.toml
2021-04-05 14:02:34 -07:00
mergify[bot]
c8bb13b3f7 Fixup AncestorIterator method (bp #16357) (#16359)
* Fixup iterator method (#16357)

(cherry picked from commit 1a13d22984)

* Only get Blockstore::last_root once (#16362)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-04-05 05:27:58 +00:00
Tyera Eulberg
5da83c1491 Bump version to v1.6.5 (#16361) 2021-04-04 22:00:40 -06:00
sakridge
a788021181 Bump version to v1.6.4 (#16345) 2021-04-04 13:31:35 -07:00
mergify[bot]
f7905d369a Throttle PoH ticks by cumulative slot time (#16139) (#16315)
* Throttle PoH ticks by cumulative slot time

    * respond to pr feedback

    * saturating sub

    * updated comment

    (cherry picked from commit 4f4cffbd03)

    # Conflicts:
    #       core/src/poh_recorder.rs

Co-authored-by: Jeff Washington (jwash) <wash678@gmail.com>
2021-04-03 23:40:46 +00:00
mergify[bot]
a0c4b4e5fc Rpc: enable getConfirmedSignaturesForAddress2 to return confirmed (not yet finalized) data (#16281) (#16293)
* Update blockstore method to allow return of unfinalized signature

* Support confirmed sigs in getConfirmedSignaturesForAddress2

* Add deprecated comments

* Update docs

* Enable confirmed transaction-history in cli

* Return real confirmation_status; fill in not-yet-finalized block time if possible

(cherry picked from commit da27acabcc)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-04-01 06:30:36 +00:00
mergify[bot]
282315a721 Rpc: fix getConfirmedTransaction slot (#16288) (#16290)
* Fix transaction blockstore apis

* Update blockstore apis in rpc

(cherry picked from commit 18bd47dbe1)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-04-01 04:58:53 +00:00
mergify[bot]
b8198f8cc5 removes OrderedIterator and transaction batch iteration order (#16153) (#16285)
In TransactionBatch,
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/transaction_batch.rs#L4-L11
lock_results[i] is aligned with transactions[iteration_order[i]]:
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2414-L2424
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/accounts.rs#L788-L817

However load_and_execute_transactions is iterating over
  lock_results[iteration_order[i]]
https://github.com/solana-labs/solana/blob/e50f59844/runtime/src/bank.rs#L2878-L2889
and then returning i as for the index of the retryable transaction.

If iteratorion_order is [1, 2, 0], and i is 0, then:
  lock_results[iteration_order[i]] = lock_results[1]
which corresponds to
  transactions[iteration_order[1]] = transactions[2]
so neither i = 0, nor iteration_order[i] = 1 gives the correct index for the
corresponding transaction (which is 2).

This commit removes OrderedIterator and transaction batch iteration order
entirely. There is only one place in blockstore processor which the
iteration order is not ordinal:
https://github.com/solana-labs/solana/blob/e50f59844/ledger/src/blockstore_processor.rs#L269-L271
It seems like, instead of using an iteration order, that can shuffle entry
transactions in-place.

(cherry picked from commit 3f63ed9a72)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-04-01 01:28:01 +00:00
Trent Nelson
634959b3ab Bump version to v1.6.3 2021-03-30 16:17:47 +00:00
Trent Nelson
03b21f2e9d Bump version to v1.6.2 2021-03-30 00:06:01 -06:00
mergify[bot]
625773e5b8 Rpc: enable getConfirmedBlock and getConfirmedTransaction to return confirmed (not yet finalized) data (bp #16142) (#16160)
* Rpc: enable getConfirmedBlock and getConfirmedTransaction to return confirmed (not yet finalized) data (#16142)

* Add Blockstore block and tx apis that allow unrooted responses

* Add TransactionStatusMessage, and send on bank freeze; also refactor TransactionStatusSender

* Track highest slot with tx-status writes complete

* Rename and unpub fn

* Add commitment to GetConfirmed input configs

* Support confirmed blocks in getConfirmedBlock

* Support confirmed txs in getConfirmedTransaction

* Update sigs-for-addr2 comment

* Enable confirmed block in cli

* Enable confirmed transaction in cli

* Review comments

* Rename blockstore method

(cherry picked from commit 433f1ead1c)

# Conflicts:
#	core/src/replay_stage.rs

* Fix conflict

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-03-27 04:51:53 +00:00
mergify[bot]
c3c4991c44 rpc: add getSlotLeaders method (#16057) (#16079)
(cherry picked from commit e7fd7d46cf)

Co-authored-by: Justin Starry <justin@solana.com>
2021-03-23 19:27:18 +00:00
mergify[bot]
9d37a33dcd buffers data shreds to make larger erasure coded sets (bp #15849) (#16074)
* buffers data shreds to make larger erasure coded sets (#15849)

Broadcast stage batches up to 8 entries:
https://github.com/solana-labs/solana/blob/79280b304/core/src/broadcast_stage/broadcast_utils.rs#L26-L29
which will be serialized into some number of shreds and chunked into FEC
sets of at most 32 shreds each:
https://github.com/solana-labs/solana/blob/79280b304/ledger/src/shred.rs#L576-L597
So depending on the size of entries, FEC sets can be small, which may
aggravate loss rate.
For example 16 FEC sets of 2:2 data/code shreds each have higher loss
rate than one 32:32 set.

This commit broadcasts data shreds immediately, but also buffers them
until it has a batch of 32 data shreds, at which point 32 coding shreds
are generated and broadcasted.

(cherry picked from commit 4f82b897bc)

# Conflicts:
#	ledger/src/shred.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-03-23 18:23:09 +00:00
mergify[bot]
a04ca03fee renames is_last_in_fec_set back to is_last_data (#15848) (#16075)
https://github.com/solana-labs/solana/pull/10095
renamed is_last_data to is_last_in_fec_set. However, the code shows that
this is actually meant to indicate where the serialized data is
complete:
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L599-L600
https://github.com/solana-labs/solana/blob/420174d3d/ledger/src/shred.rs#L229-L231

There are multiple FEC sets for each `&[Entry]` serialized and this flag
does not represent shreds last in FEC sets (only the very last one by
overlap). So the name is wrong and confusing

(cherry picked from commit 3b85cbc504)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2021-03-23 16:59:47 +00:00
mergify[bot]
7ac3c9ec76 Handle blockstore insert dup checks (#16051) (#16066)
(cherry picked from commit d76ad33597)

Co-authored-by: carllin <carl@solana.com>
2021-03-23 00:49:10 +00:00
mergify[bot]
07df827411 Bump tokio to 1.1 (#15926) (#15928)
(cherry picked from commit 654449ce91)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2021-03-16 23:29:55 +00:00
Michael Vines
b1a90c3580 =1.6.1 2021-03-16 08:41:28 +00:00
Michael Vines
be00246fb5 Bump version to v1.6.1 2021-03-15 14:47:58 -06:00
Justin Starry
918d04e3f0 Add more slot update notifications (#15734)
* Add more slot update notifications

* fix merge

* Address feedback and add integration test

* switch to datapoint

* remove unused shred method

* fix clippy

* new thread for rpc completed slots

* remove extra constant

* fixes

* rely on channel closing

* fix check
2021-03-12 21:44:06 +08:00
Trent Nelson
24d18b3cf2 docs: add docs links for crates published to crates.io 2021-03-11 08:38:18 +00:00
Jeff Washington (jwash)
1135ffd595 mut data refs as slice (#15782) 2021-03-10 15:28:03 -06:00
Jeff Washington (jwash)
8a3135d17b Account->AccountSharedData (#15691) 2021-03-09 15:06:07 -06:00
Jeff Washington (jwash)
9448540ea6 log timing after ledger process (#15716) 2021-03-05 11:01:52 -06:00
Tyera Eulberg
7e65289729 Convert blockstore TransactionStatus column family to protobufs (#15733)
* Prevent panic if TransactionStatus can't be deserialized

* Convert Blockstore TransactionStatus column to protobuf

* Add compatability test
2021-03-05 09:05:35 -07:00
Jeff Washington (jwash)
be35c1c1b7 add execute detail timings (#15638) 2021-03-03 17:07:45 -06:00
sakridge
f1223fb783 Lower blockstore processor error severity (#15578) 2021-03-01 14:57:37 -08:00