348 Commits

Author SHA1 Message Date
mergify[bot]
b0523dc236 Fix assertion failure (#13626) (#13630)
Co-authored-by: Carl Lin <carl@solana.com>
(cherry picked from commit afc1b59475)

Co-authored-by: carllin <wumu727@gmail.com>
2020-11-17 09:33:02 +00:00
mergify[bot]
5a61827702 Fix overflow in entry tick verification (bp #13572) (#13580)
* Fix overflow in entry hash count verification

(cherry picked from commit d611337394)

* clippy

(cherry picked from commit 01a4889b53)

Co-authored-by: Justin Starry <justin@solana.com>
2020-11-13 16:33:03 +00:00
mergify[bot]
b34895db37 Fix slow/stuck unstaking due to toggling in epoch (bp #13501) (#13534)
* Fix slow/stuck unstaking due to toggling in epoch (#13501)

* Fix slow/stuck unstaking due to toggling in epoch

* nits

* nits

* Add stake_program_v2 feature status check to cli

Co-authored-by: Tyera Eulberg <tyera@solana.com>
(cherry picked from commit 89b474e192)

* Fix conflict

* PartialEq<Vec<T>> is not impl for &[T] on rust v1.45.1

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-11-12 01:20:52 +00:00
mergify[bot]
ef776c0a0e Allow feature builtins to overwrite existing builtins (bp #13403) (#13419)
* Allow feature builtins to overwrite existing builtins (#13403)

* Allow feature builtins to overwrite existing builtins

* Add feature_builtin ActivationType

* Correctly retain idempotent for replacing case

* Fix test

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
(cherry picked from commit bc62313c66)

# Conflicts:
#	ledger/src/builtins.rs
#	runtime/src/bank.rs

* Fix conflicts

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-11-05 17:53:03 +00:00
mergify[bot]
65ad72cd64 improves threads' utilization in processing gossip packets (#12962) (#13251)
ClusterInfo::process_packets handles incoming packets in a thread_pool:
https://github.com/solana-labs/solana/blob/87311cce7/core/src/cluster_info.rs#L2118-L2134

However, profiling runtime shows that threads are not well utilized and
a lot of the processing is done sequentially.

This commit redistributes the work done in parallel. Testing on a gce
cluster shows 20%+ improvement in processing gossip packets with much
smaller variations.

(cherry picked from commit 75d62ca095)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-10-29 13:59:12 +00:00
Michael Vines
bff820d549 Remove unused pubkey::Pubkey imports 2020-10-21 20:27:57 -07:00
Michael Vines
5ad0ccdfe1 cargo fmt 2020-10-21 20:27:57 -07:00
Michael Vines
422bb3c526 Run codemod --extensions rs Pubkey::new_rand solana_sdk::pubkey::new_rand 2020-10-21 20:27:57 -07:00
Ryo Onodera
57d8d0ab6e Add ledger-tool dead-slots and improve purge a lot (#13070) 2020-10-21 18:38:37 +00:00
mergify[bot]
9f82208c29 Store program logs in blockstore / bigtable (TransactionWithStatusMeta) (bp #12678) (#12734)
* Store program logs in blockstore / bigtable (TransactionWithStatusMeta) (#12678)

* introduce store program logs in blockstore / bigtable

* fix test, transaction logs created for successful transactions

* fix test for legacy bincode implementation around log_messages

* only api nodes should record logs

* truncate transaction logs to 100KB

* refactor log truncate for improved coverage

(cherry picked from commit 8f5431551e)

# Conflicts:
#	runtime/src/bank.rs

* Resolve merge conflicts in bank.rs

* rerun cargo fmt

Co-authored-by: Josh <josh.hundley@gmail.com>
2020-10-10 08:55:41 +00:00
Trent Nelson
41b99b96c0 Ryos compat fixes 1.3 (#12762)
* Fix various ledger-tool error due to no builtins

* Add missing file...

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-10-09 11:49:15 -06:00
mergify[bot]
af7f48a2fd Track inserted repair shreds (#12455) (#12563)
(cherry picked from commit ce98088457)

Co-authored-by: sakridge <sakridge@gmail.com>
2020-09-29 22:57:58 +00:00
mergify[bot]
4cf69365b2 Port BPFLoader2 activation to FeatureSet (bp #12490) (#12530)
* Cargo.lock

(cherry picked from commit 6071d0d206)

# Conflicts:
#	Cargo.lock

* Port BPFLoader2 activation to FeatureSet and rework built-in program activation

(cherry picked from commit 31696a1d72)

# Conflicts:
#	core/Cargo.toml
#	genesis-programs/Cargo.toml
#	genesis/Cargo.toml
#	ledger/Cargo.toml
#	local-cluster/Cargo.toml
#	runtime/src/bank.rs

* Add Builtin AbiExample

(cherry picked from commit 833ad20b01)

* Rebase

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-28 23:12:05 +00:00
mergify[bot]
bc3aa53e02 Runtime feature activation framework (bp #12376) (#12497)
* Runtime feature activation framework

(cherry picked from commit 93259f0bae)

# Conflicts:
#	runtime/src/bank.rs

* Add feature set identifier to gossiped version information

(cherry picked from commit 35f5f9fc7b)

# Conflicts:
#	Cargo.lock
#	version/Cargo.toml

* Port instructions sysvar and secp256k1 program activation to FeatureSet

(cherry picked from commit c10da16d7b)

# Conflicts:
#	runtime/src/bank.rs
#	runtime/src/message_processor.rs

* Add feature management commands

(cherry picked from commit 93ed0ab2bb)

# Conflicts:
#	Cargo.lock
#	cli/Cargo.toml

* Make test_process_rest_api less fragile

(cherry picked from commit 7526bb96f3)

* Remove id field

(cherry picked from commit cc6ba1e131)

* FeatureSet test

(cherry picked from commit 92406cf9a0)

* cargo fmt

(cherry picked from commit 199940d683)

* cli review feedback

(cherry picked from commit 3a2b8c5e5b)

* Rename active() to is_active()

(cherry picked from commit e39fac9f01)

* Resolve merge conflicts

* Remove continues from compute_active_feature_set()

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-26 17:49:53 +00:00
mergify[bot]
e85c792f70 Add RPC notify and banking keys debug (bp #12396) (#12452)
* Add RPC notify and banking keys debug (#12396)

(cherry picked from commit 68e5a2ef56)

# Conflicts:
#	core/src/validator.rs

* Rebase

Co-authored-by: sakridge <sakridge@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-25 21:05:21 +00:00
mergify[bot]
7212bb12ea Record and store invoked instructions in transaction meta (#12311) (#12449)
* Record invoked instructions and store in transaction meta

* Enable cpi recording if transaction sender is some

* Rename invoked to innerInstructions

(cherry picked from commit 6601ec8f26)

Co-authored-by: Justin Starry <justin@solana.com>
2020-09-24 15:42:34 +00:00
mergify[bot]
9ff2378948 Remove transaction encoding from storage layer (bp #12404) (#12440)
* Remove transaction encoding from storage layer (#12404)

(cherry picked from commit 731a943239)

* Bump

Co-authored-by: Justin Starry <justin@solana.com>
2020-09-24 10:11:27 +00:00
Tyera Eulberg
35a1ab981c v1.3: Backport block time updates (#12423)
* Submit a vote timestamp every vote (#10630)

* Submit a timestamp for every vote

* Submit at most one vote timestamp per second

* Submit a timestamp for every new vote

Co-authored-by: Tyera Eulberg <tyera@solana.com>

* Timestamp first vote (#11856)

* Cache block time in Blockstore (#11955)

* Add blockstore column to cache block times

* Add method to cache block time

* Add service to cache block time

* Update rpc getBlockTime to use new method, and refactor blockstore slightly

* Return block_time with confirmed block, if available

* Add measure and warning to cache-block-time

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-23 13:54:49 -06:00
mergify[bot]
65e4aac306 Add blockstore column to store performance sampling data (bp #12251) (#12393)
* Add blockstore column to store performance sampling data (#12251)

* Add blockstore column to store performance sampling data

* introduce timer and write performance metrics to blockstore

* introduce getRecentPerformanceSamples rpc

* only run on rpc nodes enabled with transaction history

* add unit tests for get_recent_performance_samples

* remove RpcResponse from rpc call

* refactor to use Instant::now and elapsed for timer

* switch to root bank and ensure not negative subraction

* Add PerfSamples to purge/compaction

* refactor to use Instant::now and elapsed for timer

* switch to root bank and ensure not negative subraction

* remove duplicate constants

Co-authored-by: Tyera Eulberg <tyera@solana.com>
(cherry picked from commit 65a6bfad09)

# Conflicts:
#	core/src/validator.rs
#	ledger/src/blockstore.rs

* merge cherry pick of 65a6bfad0

Co-authored-by: Josh <josh.hundley@gmail.com>
2020-09-22 22:00:51 +00:00
mergify[bot]
8af90c9c08 Add keccak-secp256k1 instruction (#11839) (#12368)
* Implement keccak-secp256k1 instruction

Verifies eth addreses with ecrecover function

* Move secp256k1 test

Co-authored-by: sakridge <sakridge@gmail.com>
2020-09-21 18:09:05 +00:00
Tyera Eulberg
95ed3641c6 Add blocktime column to blockstore (#12336) 2020-09-18 21:42:45 +00:00
mergify[bot]
7656034bd4 Fix blockstore processor squash (#12319) (#12323)
(cherry picked from commit 3533e11786)

Co-authored-by: carllin <wumu727@gmail.com>
2020-09-18 05:34:37 +00:00
mergify[bot]
640bf7015f Check bank capitalization (bp #11927) (#12184)
* Check bank capitalization (#11927)

* Check bank capitalization

* Simplify and unify capitalization calculation

* Improve and add tests

* Avoid overflow and inhibit automatic restart

* Fix test

* Tweak checked sum for cap. and add tests

* Fix broken build after merge conflicts..

* Rename to ClusterType

* Rename confusing method

* Clarify comment

* Verify cap. in rent and inflation tests

Co-authored-by: Stephen Akridge <sakridge@gmail.com>
(cherry picked from commit de4a613610)

# Conflicts:
#	Cargo.lock
#	accounts-bench/Cargo.toml

* Fix conflict 1/2

* Fix conflict 2/2

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-09-11 18:04:10 +00:00
mergify[bot]
ed237af4d8 Prevent unbound memory growth by blockstore_processor (#12110) (#12122)
* Prevent unbound memory growth by blockstore_processor

* Promote log to info! considering infrequency

* Exclude the time of freeing from interval...

* Skip not-shrinkable slots even if forced

* Add comment

(cherry picked from commit c274e26eb8)

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-09-09 04:09:27 +00:00
Ryo Onodera
2341394e8b Rename to ClusterType and restore devnet compat. (manual bp) (#12069)
* Rename to ClusterType and restore devnet compat.

* De-duplicate parse code and add comments

* Adjust default Devnet genesis & reduce it in tests
2020-09-08 23:54:54 +09:00
mergify[bot]
6a74bc1297 validator: Add --enable-bigtable-ledger-upload flag (bp #12040) (#12057)
* Add --enable-bigtable-ledger-upload flag

(cherry picked from commit d8e2038dda)

* Relocate BigTable uploader to ledger/ crate

(cherry picked from commit 91a56caed2)

# Conflicts:
#	ledger/Cargo.toml

* Cargo.lock

(cherry picked from commit 2b8a521562)

* Add BigTableUploadService

(cherry picked from commit bc7731b969)

* Add BigTableUploadService

(cherry picked from commit bafdcf24f5)

* Add exit flag for bigtable upload operations

(cherry picked from commit d3611f74c8)

* Remove dead code

(cherry picked from commit cd3c134b58)

* Request correct access

(cherry picked from commit 4ba43c29ce)

* Add LARGEST_CONFIRMED_ROOT_UPLOAD_DELAY

(cherry picked from commit b64fb295a1)

* Resolve merge conflicts

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-05 17:09:32 +00:00
mergify[bot]
a5832366a7 Detect and notify when deserializable shreds are available (bp #11816) (#11988)
* Detect and notify when deserializable shreds are available (#11816)

* Add logic to check for complete data ranges

* Add RPC signature notification

Co-authored-by: Carl <carl@solana.com>
(cherry picked from commit 1c1a3f979d)

# Conflicts:
#	accounts-bench/Cargo.toml
#	core/src/rpc_pubsub.rs

* Fix conflicts

Co-authored-by: carllin <wumu727@gmail.com>
Co-authored-by: Carl <carl@solana.com>
2020-09-02 06:38:10 +00:00
mergify[bot]
99001f7f2e Bump MacOS nofile recommendation message (#11835)
(cherry picked from commit 8841c3398c)

Co-authored-by: Trent Nelson <trent@solana.com>
2020-08-25 19:57:08 +00:00
mergify[bot]
d5d1a344c3 Switch programs activation to whole-set based gating (bp #11750) (#11837)
* Switch programs activation to whole-set based gating (#11750)

* Implement Debug for MessageProcessor

* Switch from delta-based gating to whole-set gating

* Remove dbg!

* Fix clippy

* Clippy

* Add test

* add loader to stable operating mode at proper epoch

* refresh_programs_and_inflation after ancestor setup

* Callback via snapshot; avoid account re-add; Debug

* Fix test

* Fix test and fix the past history

* Make callback management stricter and cleaner

* Fix test

* Test overwrite and frozen for native programs

* Test epoch callback with genesis-programs

* Add assertions for parent bank

* Add tests and some minor cleaning

* Remove unsteady assertion...

* Fix test...

* Fix DOS

* Skip ensuring account by dual (whole/delta) gating

* Fix frozen abi implementation...

* Move compute budget constatnt init back into bank

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
(cherry picked from commit db4bbb3569)

# Conflicts:
#	genesis-programs/src/lib.rs

* Fix conflicts

Co-authored-by: Jack May <jack@solana.com>
Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-08-25 19:00:41 +00:00
mergify[bot]
62e3c084d3 Update system tuning and docs (bp #11680) (#11830)
* Sync FD limit and max maps to 500k

(cherry picked from commit 11951eb009)

* Expand system tuning docs

(cherry picked from commit 5354df8c1c)

Co-authored-by: Trent Nelson <trent@solana.com>
2020-08-25 17:13:27 +00:00
mergify[bot]
6540d3c63e Make BPF Loader static (bp #11516) (#11790)
* Make BPF Loader static (#11516)

(cherry picked from commit 7c736f71fe)

# Conflicts:
#	Cargo.lock
#	core/Cargo.toml
#	core/src/lib.rs
#	core/src/validator.rs
#	genesis-programs/src/lib.rs
#	programs/bpf_loader/src/deprecated.rs
#	programs/bpf_loader/src/lib.rs
#	sdk/src/entrypoint_native.rs
#	sdk/src/lib.rs

* resolve conflicts

Co-authored-by: Jack May <jack@solana.com>
2020-08-22 01:54:50 +00:00
mergify[bot]
4be9d5030d Squash supermajority root on blockstore replay at startup (#11727) (#11765)
(cherry picked from commit f7adb68599)

Co-authored-by: carllin <wumu727@gmail.com>
2020-08-21 08:20:09 +00:00
carllin
a9f914da8e Cleanup test utilities (#11766)
* Add voting utility

* Add blockstore utility

Co-authored-by: Carl <carl@solana.com>
2020-08-21 06:52:01 +00:00
mergify[bot]
e41004f185 rpc: rework binary encoding. BREAKING CHANGE (bp #11646) (#11674)
* Add base64 (binary64) encoding for getConfirmedTransaction/getConfirmedBlock

(cherry picked from commit b5f3ced860)

* decode-transaction now supports binary64

(cherry picked from commit 2ebc68a9e2)

# Conflicts:
#	cli/src/cli.rs

* Rework UiAccountData encode/decode such that it works from Rust

(cherry picked from commit 757e147b3b)

# Conflicts:
#	account-decoder/src/lib.rs
#	cli/src/cli.rs

* Rename Binary64 to Base64.  Establish Base58 encoding

(cherry picked from commit adc984a225)

# Conflicts:
#	account-decoder/src/lib.rs

* Remove "binary" encoding. Document "encoding" as required

(cherry picked from commit e5281157fa)

* resolve conflicts

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-08-18 02:53:10 +00:00
mergify[bot]
9015e47cc5 Rpc: Add until parameter for getConfirmedSignaturesForAddress2 (#11644) (#11648)
* Refactor bigtable apis to accept start and end keys

* Make helper fn to deserialize cell data

* Refactor get_confirmed_signatures_for_address to use get_row_data range

* Add until param to get_confirmed_signatures_for_address

* Add until param to blockstore api

* Plumb until through client/cli

* Simplify client params

(cherry picked from commit 6c5b8f324a)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-08-15 17:59:35 +00:00
mergify[bot]
4f7bfbdbe9 RPC: getConfirmedSignaturesForAddress2 only returns confirmed signatures (#11615) (#11618)
* Add failing test case

* Limit to only rooted slots

(cherry picked from commit 99fb36fe45)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-08-13 18:19:54 +00:00
mergify[bot]
520453e1f3 Blockstore address signatures: handle slots that cross primary indexes, and refactor get_confirmed_signatures_for_address2 (#11497) (#11508)
* Freeze address-signature index in the middle of slot to show failure case

* Secondary filter on signature

* Use AddressSignatures iterator instead of manually decrementing slots

* Remove unused method

* Add metrics

* Add transaction-status-index doccumentation

(cherry picked from commit de5fb3ba0e)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-08-10 17:43:00 +00:00
mergify[bot]
f9d6fb48a4 Send votes from banking stage to vote listener (#11434) (#11454)
*  Send votes from banking stage to vote listener

Co-authored-by: Carl <carl@solana.com>
(cherry picked from commit 7e25130529)

Co-authored-by: carllin <wumu727@gmail.com>
2020-08-07 19:45:40 +00:00
mergify[bot]
8be23a2bb2 Add address-based lower bound to get_confirmed_signatures_for_address2 loop (#11426) (#11432)
(cherry picked from commit 5530ee4c95)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-08-07 00:06:44 +00:00
mergify[bot]
4ff9a6910d Fix blockstore empty panic (#11423) (#11430)
* Add panicking test

* Add failing test: fresh transaction-status column shouldn't point at valid root 0

* Prevent transaction status match outside of primary-index bounds

* Initialize transaction-status and address-signature primer entries with Slot::MAX

* Revert "Add failing test: fresh transaction-status column shouldn't point at valid root 0"

This reverts commit cbad2a9fae.

* Revert "Initialize transaction-status and address-signature primer entries with Slot::MAX"

This reverts commit ffaeac0669.

(cherry picked from commit 1061b50665)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-08-06 23:39:01 +00:00
mergify[bot]
5841e4d665 Long-term ledger storage with BigTable (bp #11222) (#11392)
* ledger-storage-bigtable boilerplate

(cherry picked from commit 9d2293bb32)

* $ wget https://pki.goog/roots.pem -O pki-goog-roots.pem

(cherry picked from commit 1617a025ce)

* Add access_token module

(cherry picked from commit 59d266a111)

* Add root_ca_certificate

(cherry picked from commit faa016e4b7)

* Add build-proto

(cherry picked from commit c31e1f5bf0)

* UiTransactionEncoding is now copy

(cherry picked from commit 494968be66)

* Increase timeout

(cherry picked from commit 57dfebc5ba)

* Add build-proto/build.sh output

(cherry picked from commit 54dae6ba2c)

* Supress doctest errors

(cherry picked from commit 019c75797d)

* Add compression

(cherry picked from commit 243e05d59f)

* Add bigtable

(cherry picked from commit 6e0353965a)

* Add configuration info

(cherry picked from commit 98cca1e774)

* Add ledger-tool bigtable subcommands

(cherry picked from commit f9049d6ee4)

# Conflicts:
#	ledger-tool/Cargo.toml

* Make room for tokio 0.2

(cherry picked from commit b876fb84ba)

# Conflicts:
#	core/Cargo.toml

* Setup a tokio 0.2 runtime for RPC usage

(cherry picked from commit 0e02740565)

# Conflicts:
#	core/Cargo.toml

* Plumb Bigtable ledger storage into the RPC subsystem

(cherry picked from commit dfae9a9864)

# Conflicts:
#	core/Cargo.toml

* Add RPC transaction history design

(cherry picked from commit e56ea138c7)

* Simplify access token refreshing

(cherry picked from commit 1f7af14386)

* Report block status more frequently

(cherry picked from commit 22c46ebf96)

* after -> before

(cherry picked from commit 227ea934ff)

* Rebase

* Cargo.lock

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-08-06 04:06:44 +00:00
mergify[bot]
888f3522d8 Add getConfirmedSignaturesForAddress2 RPC method (bp #11259) (#11394)
* Add getConfirmedSignaturesForAddress2 RPC method

(cherry picked from commit 1b2276520b)

* Reimplement transaction-history command with getConfirmedSignaturesForAddress2

(cherry picked from commit 087fd32ce3)

* Rework get_confirmed_signatures_for_address2

(cherry picked from commit a11f137810)

* Rename startAfter to before

(cherry picked from commit 02c0981ecf)

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-08-05 22:47:50 +00:00
Michael Vines
eefcf484cb clippy 2020-08-03 18:35:15 +00:00
carllin
bf18524368 Add hook for getting vote transactions on replay (#11264)
* Add hook for getting vote transactions on replay

Co-authored-by: Carl <carl@solana.com>
2020-07-29 23:17:40 -07:00
Tyera Eulberg
6c38369042 Use OrderedIterator in TransactionStatusService (#11149)
* Split out get-first-err for unit testing

* Add failing test

* Add missing ordering
2020-07-21 20:06:49 +00:00
Tyera Eulberg
9a80e31bae Expose tss to the other blockstore_processor path (#11070) 2020-07-14 21:14:48 -06:00
Michael Vines
491f5ae61a Add block time placeholder to getConfirmedBlock 2020-07-10 14:47:11 +00:00
dependabot-preview[bot]
841ecfd927 chore(deps): bump bincode from 1.2.1 to 1.3.1 (#10867)
* chore(deps): bump bincode from 1.2.1 to 1.3.1

Bumps [bincode](https://github.com/servo/bincode) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/servo/bincode/releases)
- [Commits](https://github.com/servo/bincode/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* [auto-commit] Update all Cargo lock files

* Switch from deprecated method

* Add options to maintain behavior with bincode::options()

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <dependabot-buildkite@noreply.solana.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-07-09 00:08:05 +00:00
carllin
3f6042d8b3 Add RepairWeight to track votes seen in gossip for weighted repair (#10903)
* Add RepairWeight

Co-authored-by: Carl <carl@solana.com>
2020-07-06 22:49:40 -07:00
sakridge
58a475b789 Add db recovery methods (#10838) 2020-07-06 12:43:45 -07:00