Commit Graph

1867 Commits

Author SHA1 Message Date
mergify[bot]
925973ee60 RPC: Transaction deser can be quite slow (bp #12683) (#12702)
* RPC: Check encoded transaction size before decoding

(cherry picked from commit 7f67d36777)

* RPC: Support base64 encoded transactions

Defaults to base58

(cherry picked from commit e35889542b)

Co-authored-by: Trent Nelson <trent@solana.com>
2020-10-07 05:50:23 +00:00
mergify[bot]
34591d2f2f Show commit in --version and ledger-tool's log (#12636) (#12662)
* Show commit in `--version` and ledger-tool's log

* Another handy hidden env var

* Fix test

* Rename to semver!

* Fix syntax error...

(cherry picked from commit 026e7de819)

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-10-03 16:45:27 +00:00
Michael Vines
bc1b95d6d0 Bump version to v1.3.16 2020-10-03 04:24:03 +00:00
mergify[bot]
ffa0ee69ca Weight push peers by how long we haven't pushed to them (#12620) (#12651)
(cherry picked from commit 71c469c72b)

Co-authored-by: sakridge <sakridge@gmail.com>
2020-10-02 22:27:35 +00:00
mergify[bot]
efb665071c limits number of threads in core/tests/crds_gossip.rs (#12615) (#12641)
crds_gossip tests start large networks, which with large thread-pools
will exhaust system resources, causing failures in ci tests:
https://buildkite.com/solana-labs/solana/builds/31953

The commit limits size of thread-pools in the test.

(cherry picked from commit 2c669f65f1)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-10-02 20:01:49 +00:00
Michael Vines
e69ee1ec64 Add GetConfirmedBlocksWithLimit RPC method
(cherry picked from commit 75b621160e)
2020-10-02 08:21:08 -07:00
mergify[bot]
71526923a6 Expose validator cli arguments for pubsub buffer tuning (#12622)
(cherry picked from commit f41a73d76a)

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-10-02 04:43:44 +00:00
mergify[bot]
17de653ce0 Move nonce utils from runtime to sdk (bp #12577) (#12583)
* runtime: Move prepare_if_nonce_account into accounts

(cherry picked from commit caec631344)

* Move nonced tx helpers to SDK

(cherry picked from commit 65b868f4eb)

* Move remaining nonce utils from runtime to SDK

(cherry picked from commit 3c7b9c2938)

# Conflicts:
#	runtime/src/bank.rs

* Fix conflict

Co-authored-by: Trent Nelson <trent@solana.com>
Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-10-01 06:25:25 +00:00
mergify[bot]
fbe5a89e74 retains hash value of outdated responses received from pull requests (#12513) (#12603)
pull_response_fail_inserts has been increasing:
https://cdn.discordapp.com/attachments/478692221441409024/759096187587657778/pull_response_fail_insert.png
but for outdated values which fail to insert:
https://github.com/solana-labs/solana/blob/a5c3fc14b3/core/src/crds_gossip_pull.rs#L332-L344
https://github.com/solana-labs/solana/blob/a5c3fc14b3/core/src/crds.rs#L104-L108
are not recorded anywhere, and so the next pull request may obtain the
same redundant payload again, unnecessary taking bandwidth.

This commit holds on to the hashes of failed-inserts for a while, similar
to purged_values:
https://github.com/solana-labs/solana/blob/a5c3fc14b3/core/src/crds_gossip_pull.rs#L380
and filter them out for the next pull request:
https://github.com/solana-labs/solana/blob/a5c3fc14b3/core/src/crds_gossip_pull.rs#L204

(cherry picked from commit 1866521df6)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-10-01 01:47:20 +00:00
mergify[bot]
afbdcf3068 Include post balance information for rewards (#12598) (#12602)
* Include post balance information for rewards

* Add post-balance to stored Reward struct

* Handle extended Reward in bigtable

Co-authored-by: Michael Vines <mvines@gmail.com>
(cherry picked from commit c31a34fbcb)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-10-01 01:05:37 +00:00
mergify[bot]
55b0e9e9c7 builds crds filters in parallel (bp #12360) (#12571)
* builds crds filters in parallel (#12360)

Based on run-time profiles, the majority time of new_pull_requests is
spent building bloom filters, in hashing and bit-vec ops.

This commit builds crds filters in parallel using rayon constructs. The
added benchmark shows ~5x speedup (4-core machine, 8 threads).

(cherry picked from commit 537bbde22e)

# Conflicts:
#	core/Cargo.toml

* resolves mergify merge conflict

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-09-30 01:08:16 +00:00
mergify[bot]
d19ed8816e Query BigTable for block time if does not exist in blockstore (#12560) (#12565)
(cherry picked from commit 96a7d4dbd8)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-09-29 23:26:03 +00:00
mergify[bot]
af7f48a2fd Track inserted repair shreds (#12455) (#12563)
(cherry picked from commit ce98088457)

Co-authored-by: sakridge <sakridge@gmail.com>
2020-09-29 22:57:58 +00:00
Trent Nelson
24c60cf3db Bump version to v1.3.15 2020-09-29 20:57:08 +00:00
mergify[bot]
836ed842d6 Increase rpc pubsub max payload to unblock large account notifications (#12548) (#12551)
(cherry picked from commit 36d55c0667)

Co-authored-by: Justin Starry <justin@solana.com>
2020-09-29 17:20:31 +00:00
mergify[bot]
a07e90516b separates out ClusterInfo::{gossip,listen} thread-pools (#12535) (#12547)
https://github.com/solana-labs/solana/pull/12402
moved gossip-work threads:
https://github.com/solana-labs/solana/blob/afd9bfc45/core/src/cluster_info.rs#L2330-L2334
to ClusterInfo::new as a new field in the ClusterInfo struct:
https://github.com/solana-labs/solana/blob/35208c5ee/core/src/cluster_info.rs#L249
So that they can be shared between listen and gossip threads:
https://github.com/solana-labs/solana/blob/afd9bfc45/core/src/gossip_service.rs#L54-L67

However, in testing https://github.com/solana-labs/solana/pull/12360
it turned out this will cause breakage:
https://buildkite.com/solana-labs/solana/builds/31646
https://buildkite.com/solana-labs/solana/builds/31651
https://buildkite.com/solana-labs/solana/builds/31655
Whereas with separate thread pools all is good. It might be the case
that one thread is slowing down the other by exhausting the thread-pool
whereas with separate thread-pools we get fair scheduling guarantees
from the os.

This commit reverts https://github.com/solana-labs/solana/pull/12402
and instead adds separate thread-pools for listen and gossip threads:
https://github.com/solana-labs/solana/blob/afd9bfc45/core/src/gossip_service.rs#L54-L67

(cherry picked from commit 0d5258b6d3)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-09-29 11:27:47 +00:00
mergify[bot]
bd2e09d55a patches bug in Crds::find_old_labels with pubkey specific timeout (#12528) (#12546)
Current code only returns values which are expired based on the default
timeout. Example from the added unit test:
  - value inserted at time 0
  - pubkey specific timeout = 1
  - default timeout = 3
Then at now = 2, the value is expired, but the function fails to return
the value because it compares with the default timeout.

(cherry picked from commit 57ed4e4657)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-09-29 10:13:13 +00:00
mergify[bot]
5b322a995f Rpc -> proper optimistic confirmation (#12514) (#12537)
* Add service to track the most recent optimistically confirmed bank

* Plumb service into ClusterInfoVoteListener and ReplayStage

* Clean up test

* Use OptimisticallyConfirmedBank in RPC

* Remove superfluous notifications from RpcSubscriptions

* Use crossbeam to avoid mpsc recv_timeout panic

* Review comments

* Remove superfluous last_checked_slots, but pass in OptimisticallyConfirmedBank for complete correctness

(cherry picked from commit 89621adca7)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-09-29 03:49:18 +00:00
mergify[bot]
63d9f32bb4 purges old pending push messages more efficiently (#12522) (#12533)
(cherry picked from commit c94fe9236f)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-09-29 01:34:58 +00:00
mergify[bot]
4cf69365b2 Port BPFLoader2 activation to FeatureSet (bp #12490) (#12530)
* Cargo.lock

(cherry picked from commit 6071d0d206)

# Conflicts:
#	Cargo.lock

* Port BPFLoader2 activation to FeatureSet and rework built-in program activation

(cherry picked from commit 31696a1d72)

# Conflicts:
#	core/Cargo.toml
#	genesis-programs/Cargo.toml
#	genesis/Cargo.toml
#	ledger/Cargo.toml
#	local-cluster/Cargo.toml
#	runtime/src/bank.rs

* Add Builtin AbiExample

(cherry picked from commit 833ad20b01)

* Rebase

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-28 23:12:05 +00:00
mergify[bot]
72c082f55a Add precompile verification to preflight (#12486) (#12516)
(cherry picked from commit 6583c8cffe)

Co-authored-by: sakridge <sakridge@gmail.com>
2020-09-28 06:37:52 +00:00
mergify[bot]
bc3aa53e02 Runtime feature activation framework (bp #12376) (#12497)
* Runtime feature activation framework

(cherry picked from commit 93259f0bae)

# Conflicts:
#	runtime/src/bank.rs

* Add feature set identifier to gossiped version information

(cherry picked from commit 35f5f9fc7b)

# Conflicts:
#	Cargo.lock
#	version/Cargo.toml

* Port instructions sysvar and secp256k1 program activation to FeatureSet

(cherry picked from commit c10da16d7b)

# Conflicts:
#	runtime/src/bank.rs
#	runtime/src/message_processor.rs

* Add feature management commands

(cherry picked from commit 93ed0ab2bb)

# Conflicts:
#	Cargo.lock
#	cli/Cargo.toml

* Make test_process_rest_api less fragile

(cherry picked from commit 7526bb96f3)

* Remove id field

(cherry picked from commit cc6ba1e131)

* FeatureSet test

(cherry picked from commit 92406cf9a0)

* cargo fmt

(cherry picked from commit 199940d683)

* cli review feedback

(cherry picked from commit 3a2b8c5e5b)

* Rename active() to is_active()

(cherry picked from commit e39fac9f01)

* Resolve merge conflicts

* Remove continues from compute_active_feature_set()

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-26 17:49:53 +00:00
mergify[bot]
e85c792f70 Add RPC notify and banking keys debug (bp #12396) (#12452)
* Add RPC notify and banking keys debug (#12396)

(cherry picked from commit 68e5a2ef56)

# Conflicts:
#	core/src/validator.rs

* Rebase

Co-authored-by: sakridge <sakridge@gmail.com>
Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-25 21:05:21 +00:00
Trent Nelson
b65a764593 Bump jsonrpc crates to 15.0.0 (bp #12491) 2020-09-25 19:49:10 +00:00
mergify[bot]
495ea7cd2f introduce RpcPerfSample and modify getPerformanceSamples output (#12434) (#12464)
* introduce RpcPerfSample and modify getPerformanceSamples output

* camelCase test results

(cherry picked from commit 1d04c1db94)

Co-authored-by: Josh <josh.hundley@gmail.com>
2020-09-24 22:45:05 +00:00
mergify[bot]
72365bb9d2 moves gossip-work thread pool cons to ClusterInfo::new (#12402) (#12458)
(cherry picked from commit 42f1ef8acb)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-09-24 20:01:03 +00:00
mergify[bot]
4e9e05f311 shards crds values based on their hash prefix (bp #12187) (#12312)
* shards crds values based on their hash prefix (#12187)

filter_crds_values checks every crds filter against every hash value:
https://github.com/solana-labs/solana/blob/ee646aa7/core/src/crds_gossip_pull.rs#L432
which can be inefficient if the filter's bit-mask only matches small
portion of the entire crds table.

This commit shards crds values into separate tables based on shard_bits
first bits of their hash prefix. Given a (mask, mask_bits) filter,
filtering crds can be done by inspecting only relevant shards.

If CrdsFilter.mask_bits <= shard_bits, then precisely only the crds
values which match (mask, mask_bits) bit pattern are traversed.
If CrdsFilter.mask_bits > shard_bits, then approximately only
1/2^shard_bits of crds values are inspected.

Benchmarking on a gce cluster of 20 nodes, I see ~10% improvement in
generate_pull_responses metric, but with larger clusters, crds table and
2^mask_bits are both larger, so the impact should be more significant.

(cherry picked from commit 9b866d79fb)

* bumps indexmap to 1.6.0

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-09-24 16:10:53 +00:00
mergify[bot]
7212bb12ea Record and store invoked instructions in transaction meta (#12311) (#12449)
* Record invoked instructions and store in transaction meta

* Enable cpi recording if transaction sender is some

* Rename invoked to innerInstructions

(cherry picked from commit 6601ec8f26)

Co-authored-by: Justin Starry <justin@solana.com>
2020-09-24 15:42:34 +00:00
mergify[bot]
9ff2378948 Remove transaction encoding from storage layer (bp #12404) (#12440)
* Remove transaction encoding from storage layer (#12404)

(cherry picked from commit 731a943239)

* Bump

Co-authored-by: Justin Starry <justin@solana.com>
2020-09-24 10:11:27 +00:00
Tyera Eulberg
ec4938a9f3 Bump version to 1.3.14 (#12444) 2020-09-24 07:42:54 +00:00
Tyera Eulberg
35a1ab981c v1.3: Backport block time updates (#12423)
* Submit a vote timestamp every vote (#10630)

* Submit a timestamp for every vote

* Submit at most one vote timestamp per second

* Submit a timestamp for every new vote

Co-authored-by: Tyera Eulberg <tyera@solana.com>

* Timestamp first vote (#11856)

* Cache block time in Blockstore (#11955)

* Add blockstore column to cache block times

* Add method to cache block time

* Add service to cache block time

* Update rpc getBlockTime to use new method, and refactor blockstore slightly

* Return block_time with confirmed block, if available

* Add measure and warning to cache-block-time

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-23 13:54:49 -06:00
mergify[bot]
e545bdcb51 Bump spl-token (bp #12395) (#12400)
* Bump spl-token (#12395)

(cherry picked from commit e1a212fb79)

# Conflicts:
#	Cargo.lock
#	account-decoder/Cargo.toml
#	core/Cargo.toml

* Fix conflicts

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2020-09-23 01:08:20 +00:00
mergify[bot]
65e4aac306 Add blockstore column to store performance sampling data (bp #12251) (#12393)
* Add blockstore column to store performance sampling data (#12251)

* Add blockstore column to store performance sampling data

* introduce timer and write performance metrics to blockstore

* introduce getRecentPerformanceSamples rpc

* only run on rpc nodes enabled with transaction history

* add unit tests for get_recent_performance_samples

* remove RpcResponse from rpc call

* refactor to use Instant::now and elapsed for timer

* switch to root bank and ensure not negative subraction

* Add PerfSamples to purge/compaction

* refactor to use Instant::now and elapsed for timer

* switch to root bank and ensure not negative subraction

* remove duplicate constants

Co-authored-by: Tyera Eulberg <tyera@solana.com>
(cherry picked from commit 65a6bfad09)

# Conflicts:
#	core/src/validator.rs
#	ledger/src/blockstore.rs

* merge cherry pick of 65a6bfad0

Co-authored-by: Josh <josh.hundley@gmail.com>
2020-09-22 22:00:51 +00:00
mergify[bot]
8af90c9c08 Add keccak-secp256k1 instruction (#11839) (#12368)
* Implement keccak-secp256k1 instruction

Verifies eth addreses with ecrecover function

* Move secp256k1 test

Co-authored-by: sakridge <sakridge@gmail.com>
2020-09-21 18:09:05 +00:00
Michael Vines
bc774e3ea6 Bump version to v1.3.13 2020-09-20 05:40:32 +00:00
mergify[bot]
a26e1f62bb validator/ cleanup (bp #12340) (#12352)
* validator/ cleanup

(cherry picked from commit 1a03afccb1)

# Conflicts:
#	core/src/validator.rs

* Move TestValidator into its own module

(cherry picked from commit 208dd1de3a)

# Conflicts:
#	core/src/validator.rs
#	tokens/tests/commands.rs

* Rebase

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-19 22:45:09 +00:00
mergify[bot]
9fbbb7b044 SendTransactionServices now exit their thread on channel drop instead of by a flag (bp #12333) (#12335)
* Give the duplicate send_transaction_service a different thread name

(cherry picked from commit 75c3690ccd)

* SendTransactionServices now exit their thread on channel drop instead of by a flag

(cherry picked from commit c4913e3c9e)

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-18 18:32:06 +00:00
Justin Starry
37175d0cdf Fix off-by-one max payload checks
(cherry picked from commit f6cda2579f)
2020-09-16 17:05:09 -07:00
mergify[bot]
f00c504555 RPC: Limit request payload size to 50kB (#12287)
(cherry picked from commit 32dcce0ac1)

Co-authored-by: Trent Nelson <trent@solana.com>
2020-09-16 21:26:31 +00:00
mergify[bot]
d35dc79ee7 RPC sendTransaction now returns transaction logs on simulation failure (bp #12267) (#12276)
* RPC sendTransaction now returns transaction logs on simulation failure

(cherry picked from commit 749208fa32)

* Remove stale comment

(cherry picked from commit c6eea94edc)

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-16 16:45:09 +00:00
Ryo Onodera
7267257073 Bump version to v1.3.12 (#12249) 2020-09-15 20:15:03 +00:00
Ryo Onodera
52ee9155b5 Bump version to v1.3.11 (#12238) 2020-09-15 10:25:22 +00:00
mergify[bot]
b5c7ad3a9b Add new validator options for running in more restrictive environments (bp #12191) (#12218)
* Add --restricted-repair-only-mode flag

(cherry picked from commit 63a67f415e)

* Add --gossip-validator argument

(cherry picked from commit daae638781)

* Documenet how to reduce validator port exposure

(cherry picked from commit c8f03c7f6d)

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-15 04:26:23 +00:00
mergify[bot]
a5bf59b92a patches default impl for crds filter (#12199) (#12200)
In CrdsFilter.mask all bits after mask_bits are set to 1:
https://github.com/solana-labs/solana/blob/555252f4/core/src/crds_gossip_pull.rs#L65
However the default implementation, sets both mask and mask_bits to zero
which is inconsistent with CrdsFilter::compute_mask for a mask_bits of
zero.

This commit changes the default implementation by setting mask to
`!0u64` (i.e all bits set to one). As a result, for the default crds
filter, `test_mask` will always return true, whereas previously it was
always returning false.
https://github.com/solana-labs/solana/blob/555252f4/core/src/crds_gossip_pull.rs#L85

This is only used in tests and benchmarks, but causes some benchmarks to
be misleading by short circuiting in this line:
https://github.com/solana-labs/solana/blob/555252f4/core/src/crds_gossip_pull.rs#L429

(cherry picked from commit d6ec03f13c)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-09-13 14:10:42 +00:00
mergify[bot]
640bf7015f Check bank capitalization (bp #11927) (#12184)
* Check bank capitalization (#11927)

* Check bank capitalization

* Simplify and unify capitalization calculation

* Improve and add tests

* Avoid overflow and inhibit automatic restart

* Fix test

* Tweak checked sum for cap. and add tests

* Fix broken build after merge conflicts..

* Rename to ClusterType

* Rename confusing method

* Clarify comment

* Verify cap. in rent and inflation tests

Co-authored-by: Stephen Akridge <sakridge@gmail.com>
(cherry picked from commit de4a613610)

# Conflicts:
#	Cargo.lock
#	accounts-bench/Cargo.toml

* Fix conflict 1/2

* Fix conflict 2/2

Co-authored-by: Ryo Onodera <ryoqun@gmail.com>
2020-09-11 18:04:10 +00:00
mergify[bot]
db1f57162a Fix propagation on startup from snapshot (#12177) (#12182)
(cherry picked from commit 9c490e06b0)

Co-authored-by: carllin <wumu727@gmail.com>
2020-09-11 10:04:55 +00:00
mergify[bot]
bbddffa805 solana-validator --rpc-bind-address argument now works as expected (bp #12168) (#12174)
* `solana-validator --rpc-bind-address` argument now works as expected

(cherry picked from commit 6f325d4594)

* Update bootstrap-validator.sh

Co-authored-by: Michael Vines <mvines@gmail.com>
2020-09-10 22:02:50 +00:00
mergify[bot]
398f12dcc5 Program subscriptions now properly check results len and token program id (#12139) (#12141)
(cherry picked from commit 4431080066)

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-09-09 22:23:11 +00:00
mergify[bot]
796624adf9 uses rust intrinsics to convert hashes to u64 (#12097) (#12133)
(cherry picked from commit 28f2fa3fd5)

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
2020-09-09 16:33:15 +00:00
Michael Vines
415c80c204 Bump version to v1.3.10 2020-09-09 01:29:39 +00:00