Don't advertise the snapshot that the node was loaded from

snapshot_packager_service will remove this snapshot hash from gossip
when it starts

.gitbook.yaml

@@ -1,4 +1,4 @@
-root: ./book/src
+root: ./docs/src
 
 structure:
     readme: introduction.md

.gitignore

@@ -1,6 +1,6 @@
-/book/html/
-/book/src/tests.ok
-/book/src/.gitbook/assets/*.svg
+/docs/html/
+/docs/src/tests.ok
+/docs/src/.gitbook/assets/*.svg
 /farf/
 /solana-release/
 /solana-release.tar.bz2

.mergify.yml

@@ -19,14 +19,6 @@ pull_request_rules:
       label:
         add:
           - automerge
-  - name: v0.22 backport
-    conditions:
-      - base=master
-      - label=v0.22
-    actions:
-      backport:
-        branches:
-          - v0.22
   - name: v0.23 backport
     conditions:
       - base=master
@@ -35,11 +27,27 @@ pull_request_rules:
       backport:
         branches:
           - v0.23
-  - name: v0.24 backport
+  - name: v1.0 backport
     conditions:
       - base=master
-      - label=v0.24
+      - label=v1.0
     actions:
       backport:
         branches:
-          - v0.24
+          - v1.0
+  - name: v1.1 backport
+    conditions:
+      - base=master
+      - label=v1.1
+    actions:
+      backport:
+        branches:
+          - v1.1
+  - name: v1.2 backport
+    conditions:
+      - base=master
+      - label=v1.2
+    actions:
+      backport:
+        branches:
+          - v1.2

CONTRIBUTING.md

@@ -45,7 +45,7 @@ $ git pull --rebase upstream master
 
 If there are no functional changes, PRs can be very large and that's no
 problem. If, however, your changes are making meaningful changes or additions,
-then about 1,000 lines of changes is about the most you should ask a Solana
+then about 1.0.2 lines of changes is about the most you should ask a Solana
 maintainer to review.
 
 ### Should I send small PRs as I develop large, new components?
@@ -224,21 +224,20 @@ Inventing new terms is allowed, but should only be done when the term is widely
 used and understood. Avoid introducing new 3-letter terms, which can be
 confused with 3-letter acronyms.
 
-[Terms currently in use](book/src/terminology.md)
+[Terms currently in use](docs/src/terminology.md)
 
 
 ## Design Proposals
 
-Solana's architecture is described by a book generated from markdown files in
-the `book/src/` directory, maintained by an *editor* (currently @garious). To
-add a design proposal, you'll need to at least propose a change the content
-under the [Accepted Design
-Proposals](https://docs.solana.com/book/v/master/proposals) chapter.  Here's
-the full process:
+Solana's architecture is described by docs generated from markdown files in
+the `docs/src/` directory, maintained by an *editor* (currently @garious). To
+add a design proposal, you'll need to include it in the
+[Accepted Design Proposals](https://docs.solana.com/proposals)
+section of the Solana docs.  Here's the full process:
 
 1. Propose a design by creating a PR that adds a markdown document to the
-   directory `book/src/` and references it from the [table of
-   contents](book/src/SUMMARY.md). Add any relevant *maintainers* to the PR
+   `docs/src/proposals` directory and references it from the [table of
+   contents](docs/src/SUMMARY.md). Add any relevant *maintainers* to the PR
    review.
 2. The PR being merged indicates your proposed change was accepted and that the
    maintainers support your plan of attack.

Cargo.toml

@@ -7,6 +7,7 @@ members = [
     "chacha",
     "chacha-cuda",
     "chacha-sys",
+    "cli-config",
     "client",
     "core",
     "faucet",
@@ -42,6 +43,7 @@ members = [
     "archiver",
     "archiver-lib",
     "archiver-utils",
+    "remote-wallet",
     "runtime",
     "sdk",
     "sdk-c",

README.md

@@ -23,12 +23,12 @@ It's possible for a centralized database to process 710,000 transactions per sec
 
 Furthermore, and much to our surprise, it can be implemented using a mechanism that has existed in Bitcoin since day one. The Bitcoin feature is called nLocktime and it can be used to postdate transactions using block height instead of a timestamp. As a Bitcoin client, you'd use block height instead of a timestamp if you don't trust the network. Block height turns out to be an instance of what's being called a Verifiable Delay Function in cryptography circles. It's a cryptographically secure way to say time has passed. In Solana, we use a far more granular verifiable delay function, a SHA 256 hash chain, to checkpoint the ledger and coordinate consensus. With it, we implement Optimistic Concurrency Control and are now well en route towards that theoretical limit of 710,000 transactions per second.
 
-Architecture
+Documentation
 ===
 
-Before you jump into the code, review the online book [Solana: Blockchain Rebuilt for Scale](https://docs.solana.com/book/).
+Before you jump into the code, review the documentation [Solana: Blockchain Rebuilt for Scale](https://docs.solana.com).
 
-(The _latest_ development version of the online book is also [available here](https://docs.solana.com/book/v/master/).)
+(The _latest_ development version of the docs is [available here](https://docs.solana.com/v/master).)
 
 Release Binaries
 ===
@@ -87,7 +87,8 @@ $ rustup update
 On Linux systems you may need to install libssl-dev, pkg-config, zlib1g-dev, etc.  On Ubuntu:
 
 ```bash
-$ sudo apt-get install libssl-dev pkg-config zlib1g-dev llvm clang
+$ sudo apt-get update
+$ sudo apt-get install libssl-dev libudev-dev pkg-config zlib1g-dev llvm clang
 ```
 
 Download the source code:
@@ -120,16 +121,13 @@ $ cargo test
 Local Testnet
 ---
 
-Start your own testnet locally, instructions are in the book [Solana: Blockchain Rebuild for Scale: Getting Started](https://docs.solana.com/book/getting-started).
+Start your own testnet locally, instructions are in the online docs [Solana: Blockchain Rebuild for Scale: Getting Started](https://docs.solana.com/building-from-source).
 
 Remote Testnets
 ---
 
-We maintain several testnets:
+* `testnet` - public stable testnet accessible via devnet.solana.com. Runs 24/7
 
-* `testnet` - public stable testnet accessible via testnet.solana.com. Runs 24/7
-* `testnet-beta` - public beta channel testnet accessible via beta.testnet.solana.com. Runs 24/7
-* `testnet-edge` - public edge channel testnet accessible via edge.testnet.solana.com. Runs 24/7
 
 ## Deploy process
 

RELEASE.md

@@ -138,11 +138,11 @@ There are three release channels that map to branches as follows:
 ### Update documentation
 TODO: Documentation update procedure is WIP as we move to gitbook
 
-Document the new recommended version by updating `book/src/running-archiver.md` and `book/src/validator-testnet.md` on the release (beta) branch to point at the `solana-install` for the upcoming release version.
+Document the new recommended version by updating `docs/src/running-archiver.md` and `docs/src/validator-testnet.md` on the release (beta) branch to point at the `solana-install` for the upcoming release version.
 
-### Update software on testnet.solana.com
+### Update software on devnet.solana.com
 
-The testnet running on testnet.solana.com is set to use a fixed release tag
+The testnet running on devnet.solana.com is set to use a fixed release tag
 which is set in the Buildkite testnet-management pipeline.
 This tag needs to be updated and the testnet restarted after a new release
 tag is created.
@@ -182,4 +182,4 @@ TESTNET_OP=create-and-start
 ### Alert the community
 
 Notify Discord users on #validator-support that a new release for
-testnet.solana.com is available
+devnet.solana.com is available

archiver-lib/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "solana-archiver-lib"
-version = "0.23.0"
+version = "1.0.2"
 description = "Solana Archiver Library"
 authors = ["Solana Maintainers <maintainers@solana.com>"]
 repository = "https://github.com/solana-labs/solana"
@@ -15,22 +15,22 @@ ed25519-dalek = "=1.0.0-pre.1"
 log = "0.4.8"
 rand = "0.6.5"
 rand_chacha = "0.1.1"
-solana-client = { path = "../client", version = "0.23.0" }
-solana-storage-program = { path = "../programs/storage", version = "0.23.0" }
+solana-client = { path = "../client", version = "1.0.2" }
+solana-storage-program = { path = "../programs/storage", version = "1.0.2" }
 thiserror = "1.0"
 serde = "1.0.104"
-serde_json = "1.0.44"
+serde_json = "1.0.46"
 serde_derive = "1.0.103"
-solana-net-utils = { path = "../net-utils", version = "0.23.0" }
-solana-chacha = { path = "../chacha", version = "0.23.0" }
-solana-chacha-sys = { path = "../chacha-sys", version = "0.23.0" }
-solana-ledger = { path = "../ledger", version = "0.23.0" }
-solana-logger = { path = "../logger", version = "0.23.0" }
-solana-perf = { path = "../perf", version = "0.23.0" }
-solana-sdk = { path = "../sdk", version = "0.23.0" }
-solana-core = { path = "../core", version = "0.23.0" }
-solana-archiver-utils = { path = "../archiver-utils", version = "0.23.0" }
-solana-metrics = { path = "../metrics", version = "0.23.0" }
+solana-net-utils = { path = "../net-utils", version = "1.0.2" }
+solana-chacha = { path = "../chacha", version = "1.0.2" }
+solana-chacha-sys = { path = "../chacha-sys", version = "1.0.2" }
+solana-ledger = { path = "../ledger", version = "1.0.2" }
+solana-logger = { path = "../logger", version = "1.0.2" }
+solana-perf = { path = "../perf", version = "1.0.2" }
+solana-sdk = { path = "../sdk", version = "1.0.2" }
+solana-core = { path = "../core", version = "1.0.2" }
+solana-archiver-utils = { path = "../archiver-utils", version = "1.0.2" }
+solana-metrics = { path = "../metrics", version = "1.0.2" }
 
 [dev-dependencies]
 hex = "0.4.0"

archiver-lib/src/archiver.rs

@@ -1,6 +1,5 @@
 use crate::result::ArchiverError;
 use crossbeam_channel::unbounded;
-use ed25519_dalek;
 use rand::{thread_rng, Rng, SeedableRng};
 use rand_chacha::ChaChaRng;
 use solana_archiver_utils::sample_file;
@@ -16,6 +15,7 @@ use solana_core::{
     packet::{limited_deserialize, PACKET_DATA_SIZE},
     repair_service,
     repair_service::{RepairService, RepairSlotRange, RepairStrategy},
+    serve_repair::ServeRepair,
     shred_fetch_stage::ShredFetchStage,
     sigverify_stage::{DisabledSigVerifier, SigVerifyStage},
     storage_stage::NUM_STORAGE_SAMPLES,
@@ -36,7 +36,7 @@ use solana_sdk::{
     commitment_config::CommitmentConfig,
     hash::Hash,
     message::Message,
-    signature::{Keypair, KeypairUtil, Signature},
+    signature::{Keypair, Signature, Signer},
     timing::timestamp,
     transaction::Transaction,
     transport::TransportError,
@@ -87,11 +87,11 @@ struct ArchiverMeta {
 }
 
 fn get_slot_from_signature(
-    signature: &ed25519_dalek::Signature,
+    signature: &Signature,
     storage_turn: u64,
     slots_per_segment: u64,
 ) -> u64 {
-    let signature_vec = signature.to_bytes();
+    let signature_vec = signature.as_ref();
     let mut segment_index = u64::from(signature_vec[0])
         | (u64::from(signature_vec[1]) << 8)
         | (u64::from(signature_vec[1]) << 16)
@@ -195,13 +195,7 @@ impl Archiver {
             Blockstore::open(ledger_path).expect("Expected to be able to open database ledger"),
         );
 
-        let gossip_service = GossipService::new(
-            &cluster_info,
-            Some(blockstore.clone()),
-            None,
-            node.sockets.gossip,
-            &exit,
-        );
+        let gossip_service = GossipService::new(&cluster_info, None, node.sockets.gossip, &exit);
 
         info!("Connecting to the cluster via {:?}", cluster_entrypoint);
         let (nodes, _) =
@@ -390,7 +384,7 @@ impl Archiver {
                     );
                     let message =
                         Message::new_with_payer(vec![ix], Some(&archiver_keypair.pubkey()));
-                    if let Err(e) = client.send_message(&[&archiver_keypair], message) {
+                    if let Err(e) = client.send_message(&[archiver_keypair.as_ref()], message) {
                         error!("unable to redeem reward, tx failed: {:?}", e);
                     } else {
                         info!(
@@ -443,13 +437,13 @@ impl Archiver {
                 return Err(e);
             }
         };
-        let signature = storage_keypair.sign(segment_blockhash.as_ref());
+        let signature = storage_keypair.sign_message(segment_blockhash.as_ref());
         let slot = get_slot_from_signature(&signature, segment_slot, slots_per_segment);
         info!("replicating slot: {}", slot);
         slot_sender.send(slot)?;
         meta.slot = slot;
         meta.slots_per_segment = slots_per_segment;
-        meta.signature = Signature::new(&signature.to_bytes());
+        meta.signature = signature;
         meta.blockhash = segment_blockhash;
 
         let mut repair_slot_range = RepairSlotRange::default();
@@ -522,6 +516,8 @@ impl Archiver {
         let mut contact_info = node_info.clone();
         contact_info.tvu = "0.0.0.0:0".parse().unwrap();
         contact_info.wallclock = timestamp();
+        // copy over the adopted shred_version from the entrypoint
+        contact_info.shred_version = cluster_info.read().unwrap().my_data().shred_version;
         {
             let mut cluster_info_w = cluster_info.write().unwrap();
             cluster_info_w.insert_self(contact_info);
@@ -675,7 +671,7 @@ impl Archiver {
             blockhash,
         );
         if let Err(err) = client.send_and_confirm_transaction(
-            &[&archiver_keypair, &storage_keypair],
+            &[archiver_keypair.as_ref(), storage_keypair.as_ref()],
             &mut transaction,
             10,
             0,
@@ -701,7 +697,7 @@ impl Archiver {
     ) -> Result<u64> {
         let rpc_peers = {
             let cluster_info = cluster_info.read().unwrap();
-            cluster_info.rpc_peers()
+            cluster_info.all_rpc_peers()
         };
         debug!("rpc peers: {:?}", rpc_peers);
         if !rpc_peers.is_empty() {
@@ -757,7 +753,7 @@ impl Archiver {
         loop {
             let rpc_peers = {
                 let cluster_info = cluster_info.read().unwrap();
-                cluster_info.rpc_peers()
+                cluster_info.all_rpc_peers()
             };
             debug!("rpc peers: {:?}", rpc_peers);
             if !rpc_peers.is_empty() {
@@ -812,7 +808,7 @@ impl Archiver {
     /// It is recommended to use a temporary blockstore for this since the download will not verify
     /// shreds received and might impact the chaining of shreds across slots
     pub fn download_from_archiver(
-        cluster_info: &Arc<RwLock<ClusterInfo>>,
+        serve_repair: &ServeRepair,
         archiver_info: &ContactInfo,
         blockstore: &Arc<Blockstore>,
         slots_per_segment: u64,
@@ -832,10 +828,10 @@ impl Archiver {
             Recycler::default(),
             "archiver_reeciver",
         );
-        let id = cluster_info.read().unwrap().id();
+        let id = serve_repair.keypair().pubkey();
         info!(
             "Sending repair requests from: {} to: {}",
-            cluster_info.read().unwrap().my_data().id,
+            serve_repair.my_info().id,
             archiver_info.gossip
         );
         let repair_slot_range = RepairSlotRange {
@@ -855,9 +851,7 @@ impl Archiver {
                 let reqs: Vec<_> = repairs
                     .into_iter()
                     .filter_map(|repair_request| {
-                        cluster_info
-                            .read()
-                            .unwrap()
+                        serve_repair
                             .map_repair_request(&repair_request)
                             .map(|result| ((archiver_info.gossip, result), repair_request))
                             .ok()

archiver-utils/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "solana-archiver-utils"
-version = "0.23.0"
+version = "1.0.2"
 description = "Solana Archiver Utils"
 authors = ["Solana Maintainers <maintainers@solana.com>"]
 repository = "https://github.com/solana-labs/solana"
@@ -11,13 +11,12 @@ edition = "2018"
 [dependencies]
 log = "0.4.8"
 rand = "0.6.5"
-rand_chacha = "0.1.1"
-solana-chacha = { path = "../chacha", version = "0.23.0" }
-solana-chacha-sys = { path = "../chacha-sys", version = "0.23.0" }
-solana-ledger = { path = "../ledger", version = "0.23.0" }
-solana-logger = { path = "../logger", version = "0.23.0" }
-solana-perf = { path = "../perf", version = "0.23.0" }
-solana-sdk = { path = "../sdk", version = "0.23.0" }
+solana-chacha = { path = "../chacha", version = "1.0.2" }
+solana-chacha-sys = { path = "../chacha-sys", version = "1.0.2" }
+solana-ledger = { path = "../ledger", version = "1.0.2" }
+solana-logger = { path = "../logger", version = "1.0.2" }
+solana-perf = { path = "../perf", version = "1.0.2" }
+solana-sdk = { path = "../sdk", version = "1.0.2" }
 
 [dev-dependencies]
 hex = "0.4.0"

archiver/Cargo.toml

@@ -2,19 +2,19 @@
 authors = ["Solana Maintainers <maintainers@solana.com>"]
 edition = "2018"
 name = "solana-archiver"
-version = "0.23.0"
+version = "1.0.2"
 repository = "https://github.com/solana-labs/solana"
 license = "Apache-2.0"
 homepage = "https://solana.com/"
 
 [dependencies]
 clap = "2.33.0"
-console = "0.9.1"
-solana-clap-utils = { path = "../clap-utils", version = "0.23.0" }
-solana-core = { path = "../core", version = "0.23.0" }
-solana-logger = { path = "../logger", version = "0.23.0" }
-solana-metrics = { path = "../metrics", version = "0.23.0" }
-solana-archiver-lib = { path = "../archiver-lib", version = "0.23.0" }
-solana-net-utils = { path = "../net-utils", version = "0.23.0" }
-solana-sdk = { path = "../sdk", version = "0.23.0" }
+console = "0.9.2"
+solana-clap-utils = { path = "../clap-utils", version = "1.0.2" }
+solana-core = { path = "../core", version = "1.0.2" }
+solana-logger = { path = "../logger", version = "1.0.2" }
+solana-metrics = { path = "../metrics", version = "1.0.2" }
+solana-archiver-lib = { path = "../archiver-lib", version = "1.0.2" }
+solana-net-utils = { path = "../net-utils", version = "1.0.2" }
+solana-sdk = { path = "../sdk", version = "1.0.2" }
 

archiver/src/main.rs

@@ -12,7 +12,7 @@ use solana_core::{
     cluster_info::{Node, VALIDATOR_PORT_RANGE},
     contact_info::ContactInfo,
 };
-use solana_sdk::{commitment_config::CommitmentConfig, signature::KeypairUtil};
+use solana_sdk::{commitment_config::CommitmentConfig, signature::Signer};
 use std::{net::SocketAddr, path::PathBuf, process::exit, sync::Arc};
 
 fn main() {

banking-bench/Cargo.toml

@@ -2,7 +2,7 @@
 authors = ["Solana Maintainers <maintainers@solana.com>"]
 edition = "2018"
 name = "solana-banking-bench"
-version = "0.23.0"
+version = "1.0.2"
 repository = "https://github.com/solana-labs/solana"
 license = "Apache-2.0"
 homepage = "https://solana.com/"
@@ -10,11 +10,11 @@ homepage = "https://solana.com/"
 [dependencies]
 log = "0.4.6"
 rayon = "1.2.0"
-solana-core = { path = "../core", version = "0.23.0" }
-solana-ledger = { path = "../ledger", version = "0.23.0" }
-solana-logger = { path = "../logger", version = "0.23.0" }
-solana-runtime = { path = "../runtime", version = "0.23.0" }
-solana-measure = { path = "../measure", version = "0.23.0" }
-solana-sdk = { path = "../sdk", version = "0.23.0" }
+solana-core = { path = "../core", version = "1.0.2" }
+solana-ledger = { path = "../ledger", version = "1.0.2" }
+solana-logger = { path = "../logger", version = "1.0.2" }
+solana-runtime = { path = "../runtime", version = "1.0.2" }
+solana-measure = { path = "../measure", version = "1.0.2" }
+solana-sdk = { path = "../sdk", version = "1.0.2" }
 rand = "0.6.5"
 crossbeam-channel = "0.3"

bench-exchange/Cargo.toml

@@ -2,40 +2,33 @@
 authors = ["Solana Maintainers <maintainers@solana.com>"]
 edition = "2018"
 name = "solana-bench-exchange"
-version = "0.23.0"
+version = "1.0.2"
 repository = "https://github.com/solana-labs/solana"
 license = "Apache-2.0"
 homepage = "https://solana.com/"
 publish = false
 
 [dependencies]
-bincode = "1.2.1"
-bs58 = "0.3.0"
 clap = "2.32.0"
-env_logger = "0.7.1"
 itertools = "0.8.2"
 log = "0.4.8"
 num-derive = "0.3"
 num-traits = "0.2"
 rand = "0.6.5"
 rayon = "1.2.0"
-serde = "1.0.104"
-serde_derive = "1.0.103"
-serde_json = "1.0.44"
+serde_json = "1.0.46"
 serde_yaml = "0.8.11"
-solana-clap-utils = { path = "../clap-utils", version = "0.23.0" }
-solana-core = { path = "../core", version = "0.23.0" }
-solana-genesis = { path = "../genesis", version = "0.23.0" }
-solana-client = { path = "../client", version = "0.23.0" }
-solana-faucet = { path = "../faucet", version = "0.23.0" }
-solana-exchange-program = { path = "../programs/exchange", version = "0.23.0" }
-solana-logger = { path = "../logger", version = "0.23.0" }
-solana-metrics = { path = "../metrics", version = "0.23.0" }
-solana-net-utils = { path = "../net-utils", version = "0.23.0" }
-solana-runtime = { path = "../runtime", version = "0.23.0" }
-solana-sdk = { path = "../sdk", version = "0.23.0" }
-untrusted = "0.7.0"
-ws = "0.9.1"
+solana-clap-utils = { path = "../clap-utils", version = "1.0.2" }
+solana-core = { path = "../core", version = "1.0.2" }
+solana-genesis = { path = "../genesis", version = "1.0.2" }
+solana-client = { path = "../client", version = "1.0.2" }
+solana-faucet = { path = "../faucet", version = "1.0.2" }
+solana-exchange-program = { path = "../programs/exchange", version = "1.0.2" }
+solana-logger = { path = "../logger", version = "1.0.2" }
+solana-metrics = { path = "../metrics", version = "1.0.2" }
+solana-net-utils = { path = "../net-utils", version = "1.0.2" }
+solana-runtime = { path = "../runtime", version = "1.0.2" }
+solana-sdk = { path = "../sdk", version = "1.0.2" }
 
 [dev-dependencies]
-solana-local-cluster = { path = "../local-cluster", version = "0.23.0" }
+solana-local-cluster = { path = "../local-cluster", version = "1.0.2" }

bench-exchange/src/bench.rs

@@ -15,7 +15,7 @@ use solana_sdk::{
     client::{Client, SyncClient},
     commitment_config::CommitmentConfig,
     pubkey::Pubkey,
-    signature::{Keypair, KeypairUtil},
+    signature::{Keypair, Signer},
     timing::{duration_as_ms, duration_as_s},
     transaction::Transaction,
     {system_instruction, system_program},
@@ -701,7 +701,7 @@ fn verify_funding_transfer<T: SyncClient + ?Sized>(
     false
 }
 
-pub fn fund_keys(client: &dyn Client, source: &Keypair, dests: &[Arc<Keypair>], lamports: u64) {
+pub fn fund_keys<T: Client>(client: &T, source: &Keypair, dests: &[Arc<Keypair>], lamports: u64) {
     let total = lamports * (dests.len() as u64 + 1);
     let mut funded: Vec<(&Keypair, u64)> = vec![(source, total)];
     let mut notfunded: Vec<&Arc<Keypair>> = dests.iter().collect();
@@ -824,7 +824,11 @@ pub fn fund_keys(client: &dyn Client, source: &Keypair, dests: &[Arc<Keypair>],
     }
 }
 
-pub fn create_token_accounts(client: &dyn Client, signers: &[Arc<Keypair>], accounts: &[Keypair]) {
+pub fn create_token_accounts<T: Client>(
+    client: &T,
+    signers: &[Arc<Keypair>],
+    accounts: &[Keypair],
+) {
     let mut notfunded: Vec<(&Arc<Keypair>, &Keypair)> = signers.iter().zip(accounts).collect();
 
     while !notfunded.is_empty() {
@@ -968,7 +972,12 @@ fn generate_keypairs(num: u64) -> Vec<Keypair> {
     rnd.gen_n_keypairs(num)
 }
 
-pub fn airdrop_lamports(client: &dyn Client, faucet_addr: &SocketAddr, id: &Keypair, amount: u64) {
+pub fn airdrop_lamports<T: Client>(
+    client: &T,
+    faucet_addr: &SocketAddr,
+    id: &Keypair,
+    amount: u64,
+) {
     let balance = client.get_balance_with_commitment(&id.pubkey(), CommitmentConfig::recent());
     let balance = balance.unwrap_or(0);
     if balance >= amount {

bench-exchange/src/cli.rs

@@ -1,7 +1,7 @@
 use clap::{crate_description, crate_name, value_t, App, Arg, ArgMatches};
 use solana_core::gen_keys::GenKeys;
 use solana_faucet::faucet::FAUCET_PORT;
-use solana_sdk::signature::{read_keypair_file, Keypair, KeypairUtil};
+use solana_sdk::signature::{read_keypair_file, Keypair};
 use std::net::SocketAddr;
 use std::process::exit;
 use std::time::Duration;

bench-exchange/src/main.rs

@@ -5,7 +5,7 @@ pub mod order_book;
 use crate::bench::{airdrop_lamports, create_client_accounts_file, do_bench_exchange, Config};
 use log::*;
 use solana_core::gossip_service::{discover_cluster, get_multi_client};
-use solana_sdk::signature::KeypairUtil;
+use solana_sdk::signature::Signer;
 
 fn main() {
     solana_logger::setup();

bench-exchange/tests/bench_exchange.rs

@@ -10,12 +10,13 @@ use solana_local_cluster::local_cluster::{ClusterConfig, LocalCluster};
 use solana_runtime::bank::Bank;
 use solana_runtime::bank_client::BankClient;
 use solana_sdk::genesis_config::create_genesis_config;
-use solana_sdk::signature::{Keypair, KeypairUtil};
+use solana_sdk::signature::{Keypair, Signer};
 use std::process::exit;
 use std::sync::mpsc::channel;
 use std::time::Duration;
 
 #[test]
+#[ignore]
 fn test_exchange_local_cluster() {
     solana_logger::setup();
 

bench-streamer/Cargo.toml

@@ -2,14 +2,14 @@
 authors = ["Solana Maintainers <maintainers@solana.com>"]
 edition = "2018"
 name = "solana-bench-streamer"
-version = "0.23.0"
+version = "1.0.2"
 repository = "https://github.com/solana-labs/solana"
 license = "Apache-2.0"
 homepage = "https://solana.com/"
 
 [dependencies]
 clap = "2.33.0"
-solana-clap-utils = { path = "../clap-utils", version = "0.23.0" }
-solana-core = { path = "../core", version = "0.23.0" }
-solana-logger = { path = "../logger", version = "0.23.0" }
-solana-net-utils = { path = "../net-utils", version = "0.23.0" }
+solana-clap-utils = { path = "../clap-utils", version = "1.0.2" }
+solana-core = { path = "../core", version = "1.0.2" }
+solana-logger = { path = "../logger", version = "1.0.2" }
+solana-net-utils = { path = "../net-utils", version = "1.0.2" }

bench-tps/Cargo.toml

@@ -2,7 +2,7 @@
 authors = ["Solana Maintainers <maintainers@solana.com>"]
 edition = "2018"
 name = "solana-bench-tps"
-version = "0.23.0"
+version = "1.0.2"
 repository = "https://github.com/solana-labs/solana"
 license = "Apache-2.0"
 homepage = "https://solana.com/"
@@ -12,28 +12,26 @@ bincode = "1.2.1"
 clap = "2.33.0"
 log = "0.4.8"
 rayon = "1.2.0"
-serde = "1.0.104"
-serde_derive = "1.0.103"
-serde_json = "1.0.44"
+serde_json = "1.0.46"
 serde_yaml = "0.8.11"
-solana-clap-utils = { path = "../clap-utils", version = "0.23.0" }
-solana-core = { path = "../core", version = "0.23.0" }
-solana-genesis = { path = "../genesis", version = "0.23.0" }
-solana-client = { path = "../client", version = "0.23.0" }
-solana-faucet = { path = "../faucet", version = "0.23.0" }
-solana-librapay = { path = "../programs/librapay", version = "0.23.0", optional = true }
-solana-logger = { path = "../logger", version = "0.23.0" }
-solana-metrics = { path = "../metrics", version = "0.23.0" }
-solana-measure = { path = "../measure", version = "0.23.0" }
-solana-net-utils = { path = "../net-utils", version = "0.23.0" }
-solana-runtime = { path = "../runtime", version = "0.23.0" }
-solana-sdk = { path = "../sdk", version = "0.23.0" }
-solana-move-loader-program = { path = "../programs/move_loader", version = "0.23.0", optional = true }
+solana-clap-utils = { path = "../clap-utils", version = "1.0.2" }
+solana-core = { path = "../core", version = "1.0.2" }
+solana-genesis = { path = "../genesis", version = "1.0.2" }
+solana-client = { path = "../client", version = "1.0.2" }
+solana-faucet = { path = "../faucet", version = "1.0.2" }
+solana-librapay = { path = "../programs/librapay", version = "1.0.2", optional = true }
+solana-logger = { path = "../logger", version = "1.0.2" }
+solana-metrics = { path = "../metrics", version = "1.0.2" }
+solana-measure = { path = "../measure", version = "1.0.2" }
+solana-net-utils = { path = "../net-utils", version = "1.0.2" }
+solana-runtime = { path = "../runtime", version = "1.0.2" }
+solana-sdk = { path = "../sdk", version = "1.0.2" }
+solana-move-loader-program = { path = "../programs/move_loader", version = "1.0.2", optional = true }
 
 [dev-dependencies]
 serial_test = "0.3.2"
-serial_test_derive = "0.3.1"
-solana-local-cluster = { path = "../local-cluster", version = "0.23.0" }
+serial_test_derive = "0.4.0"
+solana-local-cluster = { path = "../local-cluster", version = "1.0.2" }
 
 [features]
 move = ["solana-librapay", "solana-move-loader-program"]

bench-tps/src/bench.rs

@@ -7,7 +7,7 @@ use solana_faucet::faucet::request_airdrop_transaction;
 #[cfg(feature = "move")]
 use solana_librapay::{create_genesis, upload_mint_script, upload_payment_script};
 use solana_measure::measure::Measure;
-use solana_metrics::{self, datapoint_debug};
+use solana_metrics::{self, datapoint_info};
 use solana_sdk::{
     client::Client,
     clock::{DEFAULT_TICKS_PER_SECOND, DEFAULT_TICKS_PER_SLOT, MAX_PROCESSING_AGE},
@@ -15,7 +15,7 @@ use solana_sdk::{
     fee_calculator::FeeCalculator,
     hash::Hash,
     pubkey::Pubkey,
-    signature::{Keypair, KeypairUtil},
+    signature::{Keypair, Signer},
     system_instruction, system_transaction,
     timing::{duration_as_ms, duration_as_s, duration_as_us, timestamp},
     transaction::Transaction,
@@ -244,7 +244,7 @@ where
 
 fn metrics_submit_lamport_balance(lamport_balance: u64) {
     info!("Token balance: {}", lamport_balance);
-    datapoint_debug!(
+    datapoint_info!(
         "bench-tps-lamport_balance",
         ("balance", lamport_balance, i64)
     );
@@ -375,7 +375,7 @@ fn generate_txs(
         duration_as_ms(&duration),
         blockhash,
     );
-    datapoint_debug!(
+    datapoint_info!(
         "bench-tps-generate_txs",
         ("duration", duration_as_us(&duration), i64)
     );
@@ -481,7 +481,7 @@ fn do_tx_transfers<T: Client>(
                 duration_as_ms(&transfer_start.elapsed()),
                 tx_len as f32 / duration_as_s(&transfer_start.elapsed()),
             );
-            datapoint_debug!(
+            datapoint_info!(
                 "bench-tps-do_tx_transfers",
                 ("duration", duration_as_us(&transfer_start.elapsed()), i64),
                 ("count", tx_len, i64)

bench-tps/src/cli.rs

@@ -1,7 +1,7 @@
 use clap::{crate_description, crate_name, App, Arg, ArgMatches};
 use solana_faucet::faucet::FAUCET_PORT;
 use solana_sdk::fee_calculator::FeeCalculator;
-use solana_sdk::signature::{read_keypair_file, Keypair, KeypairUtil};
+use solana_sdk::signature::{read_keypair_file, Keypair};
 use std::{net::SocketAddr, process::exit, time::Duration};
 
 const NUM_LAMPORTS_PER_ACCOUNT_DEFAULT: u64 = solana_sdk::native_token::LAMPORTS_PER_SOL;

bench-tps/src/main.rs

@@ -4,7 +4,7 @@ use solana_bench_tps::cli;
 use solana_core::gossip_service::{discover_cluster, get_client, get_multi_client};
 use solana_genesis::Base64Account;
 use solana_sdk::fee_calculator::FeeCalculator;
-use solana_sdk::signature::{Keypair, KeypairUtil};
+use solana_sdk::signature::{Keypair, Signer};
 use solana_sdk::system_program;
 use std::{collections::HashMap, fs::File, io::prelude::*, path::Path, process::exit, sync::Arc};
 

bench-tps/tests/bench_tps.rs

@@ -8,7 +8,7 @@ use solana_faucet::faucet::run_local_faucet;
 use solana_local_cluster::local_cluster::{ClusterConfig, LocalCluster};
 #[cfg(feature = "move")]
 use solana_sdk::move_loader::solana_move_loader_program;
-use solana_sdk::signature::{Keypair, KeypairUtil};
+use solana_sdk::signature::{Keypair, Signer};
 use std::sync::{mpsc::channel, Arc};
 use std::time::Duration;
 

book/src/implemented-proposals/ed_overview/ed_validation_client_economics/ed_vce_replication_validation_transaction_fees.md

@@ -1,11 +0,0 @@
-# Replication-validation Transaction Fees
-
-**Subject to change.**
-
-As previously mentioned, validator-clients will also be responsible for validating PoReps submitted into the PoH stream by archiver-clients. In this case, validators are providing compute \(CPU/GPU\) and light storage resources to confirm that these replication proofs could only be generated by a client that is storing the referenced PoH leger block.
-
-While replication-clients are incentivized and rewarded through protocol-based rewards schedule \(see [Replication-client Economics](../ed_replication_client_economics/)\), validator-clients will be incentivized to include and validate PoReps in PoH through collection of transaction fees associated with the submitted PoReps and distribution of protocol rewards proportional to the validated PoReps. As will be described in detail in the Section 3.1, replication-client rewards are protocol-based and designed to reward based on a global data redundancy factor. I.e. the protocol will incentivize replication-client participation through rewards based on a target ledger redundancy \(e.g. 10x data redundancy\).
-
-The validation of PoReps by validation-clients is computationally more expensive than state-validation \(detail in the [Economic Sustainability](../ed_economic_sustainability.md) chapter\), thus the transaction fees are expected to be proportionally higher.
-
-There are various attack vectors available for colluding validation and replication clients, also described in detail below in [Economic Sustainability](../ed_economic_sustainability/README.md). To protect against various collusion attack vectors, for a given epoch, validator rewards are distributed across participating validation-clients in proportion to the number of validated PoReps in the epoch less the number of PoReps that mismatch the archivers challenge. The PoRep challenge game is described in [Ledger Replication](https://github.com/solana-labs/solana/blob/master/book/src/ledger-replication.md#the-porep-game). This design rewards validators proportional to the number of PoReps they process and validate, while providing negative pressure for validation-clients to submit lazy or malicious invalid votes on submitted PoReps \(note that it is computationally prohibitive to determine whether a validator-client has marked a valid PoRep as invalid\).

book/src/implemented-proposals/repair-service.md

@@ -1,49 +0,0 @@
-# Repair Service
-
-## Repair Service
-
-The RepairService is in charge of retrieving missing shreds that failed to be delivered by primary communication protocols like Avalanche. It is in charge of managing the protocols described below in the `Repair Protocols` section below.
-
-## Challenges:
-
-1\) Validators can fail to receive particular shreds due to network failures
-
-2\) Consider a scenario where blockstore contains the set of slots {1, 3, 5}. Then Blockstore receives shreds for some slot 7, where for each of the shreds b, b.parent == 6, so then the parent-child relation 6 -> 7 is stored in blockstore. However, there is no way to chain these slots to any of the existing banks in Blockstore, and thus the `Shred Repair` protocol will not repair these slots. If these slots happen to be part of the main chain, this will halt replay progress on this node.
-
-3\) Validators that find themselves behind the cluster by an entire epoch struggle/fail to catch up because they do not have a leader schedule for future epochs. If nodes were to blindly accept repair shreds in these future epochs, this exposes nodes to spam.
-
-## Repair Protocols
-
-The repair protocol makes best attempts to progress the forking structure of Blockstore.
-
-The different protocol strategies to address the above challenges:
-
-1. Shred Repair \(Addresses Challenge \#1\): This is the most basic repair protocol, with the purpose of detecting and filling "holes" in the ledger. Blockstore tracks the latest root slot. RepairService will then periodically iterate every fork in blockstore starting from the root slot, sending repair requests to validators for any missing shreds. It will send at most some `N` repair reqeusts per iteration.
-
-   Note: Validators will only accept shreds within the current verifiable epoch \(epoch the validator has a leader schedule for\).
-
-2. Preemptive Slot Repair \(Addresses Challenge \#2\): The goal of this protocol is to discover the chaining relationship of "orphan" slots that do not currently chain to any known fork.
-   * Blockstore will track the set of "orphan" slots in a separate column family.
-   * RepairService will periodically make `RequestOrphan` requests for each of the orphans in blockstore.
-
-     `RequestOrphan(orphan)` request - `orphan` is the orphan slot that the requestor wants to know the parents of `RequestOrphan(orphan)` response - The highest shreds for each of the first `N` parents of the requested `orphan`
-
-     On receiving the responses `p`, where `p` is some shred in a parent slot, validators will:
-
-     * Insert an empty `SlotMeta` in blockstore for `p.slot` if it doesn't already exist.
-     * If `p.slot` does exist, update the parent of `p` based on `parents`
-
-     Note: that once these empty slots are added to blockstore, the `Shred Repair` protocol should attempt to fill those slots.
-
-     Note: Validators will only accept responses containing shreds within the current verifiable epoch \(epoch the validator has a leader schedule for\).
-3. Repairmen \(Addresses Challenge \#3\): This part of the repair protocol is the primary mechanism by which new nodes joining the cluster catch up after loading a snapshot. This protocol works in a "forward" fashion, so validators can verify every shred that they receive against a known leader schedule.
-
-   Each validator advertises in gossip:
-
-   * Current root
-   * The set of all completed slots in the confirmed epochs \(an epoch that was calculated based on a bank <= current root\) past the current root
-
-     Observers of this gossip message with higher epochs \(repairmen\) send shreds to catch the lagging node up with the rest of the cluster. The repairmen are responsible for sending the slots within the epochs that are confrimed by the advertised `root` in gossip. The repairmen divide the responsibility of sending each of the missing slots in these epochs based on a random seed \(simple shred.index iteration by N, seeded with the repairman's node\_pubkey\). Ideally, each repairman in an N node cluster \(N nodes whose epochs are higher than that of the repairee\) sends 1/N of the missing shreds. Both data and coding shreds for missing slots are sent. Repairmen do not send shreds again to the same validator until they see the message in gossip updated, at which point they perform another iteration of this protocol.
-
-     Gossip messages are updated every time a validator receives a complete slot within the epoch. Completed slots are detected by blockstore and sent over a channel to RepairService. It is important to note that we know that by the time a slot X is complete, the epoch schedule must exist for the epoch that contains slot X because WindowService will reject shreds for unconfirmed epochs. When a newly completed slot is detected, we also update the current root if it has changed since the last update. The root is made available to RepairService through Blockstore, which holds the latest root.
-

chacha-cuda/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "solana-chacha-cuda"
-version = "0.23.0"
+version = "1.0.2"
 description = "Solana Chacha Cuda APIs"
 authors = ["Solana Maintainers <maintainers@solana.com>"]
 repository = "https://github.com/solana-labs/solana"
@@ -10,12 +10,12 @@ edition = "2018"
 
 [dependencies]
 log = "0.4.8"
-solana-archiver-utils = { path = "../archiver-utils", version = "0.23.0" }
-solana-chacha = { path = "../chacha", version = "0.23.0" }
-solana-ledger = { path = "../ledger", version = "0.23.0" }
-solana-logger = { path = "../logger", version = "0.23.0" }
-solana-perf = { path = "../perf", version = "0.23.0" }
-solana-sdk = { path = "../sdk", version = "0.23.0" }
+solana-archiver-utils = { path = "../archiver-utils", version = "1.0.2" }
+solana-chacha = { path = "../chacha", version = "1.0.2" }
+solana-ledger = { path = "../ledger", version = "1.0.2" }
+solana-logger = { path = "../logger", version = "1.0.2" }
+solana-perf = { path = "../perf", version = "1.0.2" }
+solana-sdk = { path = "../sdk", version = "1.0.2" }
 
 [dev-dependencies]
 hex-literal = "0.2.1"

chacha-cuda/src/chacha_cuda.rs

@@ -118,7 +118,7 @@ mod tests {
     use solana_ledger::entry::create_ticks;
     use solana_ledger::get_tmp_ledger_path;
     use solana_sdk::clock::DEFAULT_SLOTS_PER_SEGMENT;
-    use solana_sdk::signature::{Keypair, KeypairUtil};
+    use solana_sdk::signature::Keypair;
     use std::fs::{remove_dir_all, remove_file};
     use std::path::Path;
 

chacha-sys/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "solana-chacha-sys"
-version = "0.23.0"
+version = "1.0.2"
 description = "Solana chacha-sys"
 authors = ["Solana Maintainers <maintainers@solana.com>"]
 repository = "https://github.com/solana-labs/solana"

chacha/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "solana-chacha"
-version = "0.23.0"
+version = "1.0.2"
 description = "Solana Chacha APIs"
 authors = ["Solana Maintainers <maintainers@solana.com>"]
 repository = "https://github.com/solana-labs/solana"
@@ -12,11 +12,11 @@ edition = "2018"
 log = "0.4.8"
 rand = "0.6.5"
 rand_chacha = "0.1.1"
-solana-chacha-sys = { path = "../chacha-sys", version = "0.23.0" }
-solana-ledger = { path = "../ledger", version = "0.23.0" }
-solana-logger = { path = "../logger", version = "0.23.0" }
-solana-perf = { path = "../perf", version = "0.23.0" }
-solana-sdk = { path = "../sdk", version = "0.23.0" }
+solana-chacha-sys = { path = "../chacha-sys", version = "1.0.2" }
+solana-ledger = { path = "../ledger", version = "1.0.2" }
+solana-logger = { path = "../logger", version = "1.0.2" }
+solana-perf = { path = "../perf", version = "1.0.2" }
+solana-sdk = { path = "../sdk", version = "1.0.2" }
 
 [dev-dependencies]
 hex-literal = "0.2.1"

chacha/src/chacha.rs

@@ -81,7 +81,7 @@ mod tests {
     use solana_ledger::get_tmp_ledger_path;
     use solana_sdk::hash::{hash, Hash, Hasher};
     use solana_sdk::pubkey::Pubkey;
-    use solana_sdk::signature::{Keypair, KeypairUtil};
+    use solana_sdk::signature::{Keypair, Signer};
     use solana_sdk::system_transaction;
     use std::fs::remove_file;
     use std::fs::File;

ci/affects-files.sh

@@ -9,7 +9,7 @@
 #     ./affects-files.sh ^snap/  -- anything under the snap/ subdirectory
 #     ./affects-files.sh snap/   -- also matches foo/snap/
 # Any pattern starting with the ! character will be negated:
-#     ./affects-files.sh !^book/  -- anything *not* under the book/ subdirectory
+#     ./affects-files.sh !^docs/  -- anything *not* under the docs/ subdirectory
 #
 set -e
 cd "$(dirname "$0")"/..

ci/buildkite-secondary.yml

@@ -6,7 +6,7 @@ steps:
     timeout_in_minutes: 60
     name: "publish docker"
   - command: "ci/publish-crate.sh"
-    timeout_in_minutes: 120
+    timeout_in_minutes: 240
     name: "publish crate"
     branches: "!master"
   - command: "ci/publish-bpf-sdk.sh"
@@ -15,6 +15,6 @@ steps:
   - command: "ci/publish-tarball.sh"
     timeout_in_minutes: 60
     name: "publish tarball"
-  - command: "ci/publish-book.sh"
+  - command: "ci/publish-docs.sh"
     timeout_in_minutes: 15
-    name: "publish book"
+    name: "publish docs"

ci/docker-run.sh

@@ -78,7 +78,9 @@ ARGS+=(
 # Also propagate environment variables needed for codecov
 # https://docs.codecov.io/docs/testing-with-docker#section-codecov-inside-docker
 # We normalize CI to `1`; but codecov expects it to be `true` to detect Buildkite...
-CODECOV_ENVS=$(CI=true bash <(curl -s https://codecov.io/env))
+# Unfortunately, codecov.io fails sometimes:
+#   curl: (7) Failed to connect to codecov.io port 443: Connection timed out
+CODECOV_ENVS=$(CI=true bash <(while ! curl -sS --retry 5 --retry-delay 2 --retry-connrefused https://codecov.io/env; do sleep 10; done))
 
 if $INTERACTIVE; then
   if [[ -n $1 ]]; then

ci/docker-rust-nightly/Dockerfile

@@ -1,4 +1,4 @@
-FROM solanalabs/rust:1.40.0
+FROM solanalabs/rust:1.41.1
 ARG date
 
 RUN set -x \

ci/docker-rust/Dockerfile

@@ -1,6 +1,6 @@
 # Note: when the rust version is changed also modify
 # ci/rust-version.sh to pick up the new image tag
-FROM rust:1.40.0
+FROM rust:1.41.1
 
 # Add Google Protocol Buffers for Libra's metrics library.
 ENV PROTOC_VERSION 3.8.0
@@ -17,6 +17,7 @@ RUN set -x \
       clang-7 \
       cmake \
       lcov \
+      libudev-dev \
       libclang-common-7-dev \
       mscgen \
       net-tools \

ci/nits.sh

@@ -18,20 +18,22 @@ declare prints=(
 
 # Parts of the tree that are expected to be print free
 declare print_free_tree=(
-  'core/src'
-  'faucet/src'
-  'ledger/src'
-  'metrics/src'
-  'net-utils/src'
-  'runtime/src'
-  'sdk/bpf/rust/rust-utils'
-  'sdk/src'
-  'programs/bpf/rust'
-  'programs/stake/src'
-  'programs/vote/src'
+  ':core/src/**.rs'
+  ':faucet/src/**.rs'
+  ':ledger/src/**.rs'
+  ':metrics/src/**.rs'
+  ':net-utils/src/**.rs'
+  ':runtime/src/**.rs'
+  ':sdk/bpf/rust/rust-utils/**.rs'
+  ':sdk/**.rs'
+  ':programs/**.rs'
+  ':^**bin**.rs'
+  ':^**bench**.rs'
+  ':^**test**.rs'
+  ':^**/build.rs'
 )
 
-if _ git --no-pager grep -n --max-depth=0 "${prints[@]/#/-e }" -- "${print_free_tree[@]}"; then
+if _ git --no-pager grep -n "${prints[@]/#/-e}" -- "${print_free_tree[@]}"; then
     exit 1
 fi
 

ci/publish-docs.sh

@@ -11,10 +11,10 @@ if [[ -n $CI_BRANCH ]]; then
     set -x
     (
       . ci/rust-version.sh stable
-      ci/docker-run.sh "$rust_stable_docker_image" make -Cbook -B svg
+      ci/docker-run.sh "$rust_stable_docker_image" make -C docs -B svg
     )
     # make a local commit for the svgs
-    git add -A -f book/src/.gitbook/assets/.
+    git add -A -f docs/src/.gitbook/assets/.
     if ! git diff-index --quiet HEAD; then
       git config user.email maintainers@solana.com
       git config user.name "$me"

ci/rust-version.sh

@@ -16,13 +16,13 @@
 if [[ -n $RUST_STABLE_VERSION ]]; then
   stable_version="$RUST_STABLE_VERSION"
 else
-  stable_version=1.40.0
+  stable_version=1.41.1
 fi
 
 if [[ -n $RUST_NIGHTLY_VERSION ]]; then
   nightly_version="$RUST_NIGHTLY_VERSION"
 else
-  nightly_version=2019-12-19
+  nightly_version=2020-02-27
 fi
 
 

ci/test-checks.sh

@@ -25,7 +25,7 @@ _ cargo +"$rust_stable" audit --version
 _ cargo +"$rust_stable" audit --ignore RUSTSEC-2020-0002
 _ ci/nits.sh
 _ ci/order-crates-for-publishing.py
-_ book/build.sh
+_ docs/build.sh
 _ ci/check-ssh-keys.sh
 
 {

ci/test-coverage.sh

@@ -42,7 +42,9 @@ if [[ -z "$CODECOV_TOKEN" ]]; then
   echo CODECOV_TOKEN undefined, codecov.io upload skipped
 else
   # We normalize CI to `1`; but codecov expects it to be `true` to detect Buildkite...
-  CI=true bash <(curl -s https://codecov.io/bash) -X gcov -f target/cov/lcov.info
+  # Unfortunately, codecov.io fails sometimes:
+  #   curl: (7) Failed to connect to codecov.io port 443: Connection timed out
+  CI=true bash <(while ! curl -sS --retry 5 --retry-delay 2 --retry-connrefused https://codecov.io/bash; do sleep 10; done) -Z -X gcov -f target/cov/lcov.info
 
   annotate --style success --context codecov.io \
     "CodeCov report: https://codecov.io/github/solana-labs/solana/commit/${CI_COMMIT:0:9}"

ci/test-stable.sh

@@ -13,12 +13,12 @@ annotate() {
 # Run the appropriate test based on entrypoint
 testName=$(basename "$0" .sh)
 
-# Skip if only the book has been modified
+# Skip if only the docs have been modified
 ci/affects-files.sh \
-  \!^book/ \
+  \!^docs/ \
 || {
   annotate --style info \
-    "Skipped $testName as only book files were modified"
+    "Skipped $testName as only docs/ files were modified"
   exit 0
 }
 

ci/testnet-manager.sh

@@ -32,20 +32,10 @@ steps:
         options:
           - label: "testnet"
             value: "testnet"
-          - label: "testnet-perf"
-            value: "testnet-perf"
           - label: "testnet-edge"
             value: "testnet-edge"
-          - label: "testnet-edge-perf"
-            value: "testnet-edge-perf"
           - label: "testnet-beta"
             value: "testnet-beta"
-          - label: "testnet-beta-perf"
-            value: "testnet-beta-perf"
-          - label: "testnet-demo"
-            value: "testnet-demo"
-          - label: "tds"
-            value: "tds"
       - select: "Operation"
         key: "testnet-operation"
         default: "sanity-or-restart"
@@ -131,11 +121,11 @@ GCE_LOW_QUOTA_ZONES=(
 )
 
 case $TESTNET in
-testnet-edge|testnet-edge-perf)
+testnet-edge)
   CHANNEL_OR_TAG=edge
   CHANNEL_BRANCH=$EDGE_CHANNEL
   ;;
-testnet-beta|testnet-beta-perf)
+testnet-beta)
   CHANNEL_OR_TAG=beta
   CHANNEL_BRANCH=$BETA_CHANNEL
   ;;
@@ -144,22 +134,6 @@ testnet)
   CHANNEL_BRANCH=$STABLE_CHANNEL
   export CLOUDSDK_CORE_PROJECT=testnet-solana-com
   ;;
-testnet-perf)
-  CHANNEL_OR_TAG=$STABLE_CHANNEL_LATEST_TAG
-  CHANNEL_BRANCH=$STABLE_CHANNEL
-  ;;
-testnet-demo)
-  CHANNEL_OR_TAG=beta
-  CHANNEL_BRANCH=$BETA_CHANNEL
-  : "${GCE_NODE_COUNT:=150}"
-  : "${GCE_LOW_QUOTA_NODE_COUNT:=70}"
-  ;;
-tds)
-  : "${TDS_CHANNEL_OR_TAG:=edge}"
-  CHANNEL_OR_TAG="$TDS_CHANNEL_OR_TAG"
-  CHANNEL_BRANCH="$CI_BRANCH"
-  export CLOUDSDK_CORE_PROJECT=tour-de-sol
-  ;;
 *)
   echo "Error: Invalid TESTNET=$TESTNET"
   exit 1
@@ -234,64 +208,22 @@ sanity() {
     (
       set -x
       NO_INSTALL_CHECK=1 \
-        ci/testnet-sanity.sh edge-testnet-solana-com gce -P us-west1-b
+        ci/testnet-sanity.sh edge-devnet-solana-com gce -P us-west1-b
       maybe_deploy_software
     )
     ;;
-  testnet-edge-perf)
-    (
-      set -x
-      REJECT_EXTRA_NODES=1 \
-        ci/testnet-sanity.sh edge-perf-testnet-solana-com ec2 us-west-2b
-    )
-    ;;
   testnet-beta)
     (
       set -x
       NO_INSTALL_CHECK=1 \
-        ci/testnet-sanity.sh beta-testnet-solana-com gce -P us-west1-b
+        ci/testnet-sanity.sh beta-devnet-solana-com gce -P us-west1-b
       maybe_deploy_software --deploy-if-newer
     )
     ;;
-  testnet-beta-perf)
-    (
-      set -x
-      REJECT_EXTRA_NODES=1 \
-        ci/testnet-sanity.sh beta-perf-testnet-solana-com ec2 us-west-2b
-    )
-    ;;
   testnet)
     (
       set -x
-      ci/testnet-sanity.sh testnet-solana-com gce -P us-west1-b
-    )
-    ;;
-  testnet-perf)
-    (
-      set -x
-      REJECT_EXTRA_NODES=1 \
-        ci/testnet-sanity.sh perf-testnet-solana-com gce us-west1-b
-      #ci/testnet-sanity.sh perf-testnet-solana-com ec2 us-east-1a
-    )
-    ;;
-  testnet-demo)
-    (
-      set -x
-
-      ok=true
-      if [[ -n $GCE_NODE_COUNT ]]; then
-        ci/testnet-sanity.sh demo-testnet-solana-com gce "${GCE_ZONES[0]}" -f || ok=false
-      else
-        echo "Error: no GCE nodes"
-        ok=false
-      fi
-      $ok
-    )
-    ;;
-  tds)
-    (
-      set -x
-      ci/testnet-sanity.sh tds-solana-com gce "${GCE_ZONES[0]}" -f
+      ci/testnet-sanity.sh devnet-solana-com gce -P us-west1-b
     )
     ;;
   *)
@@ -327,9 +259,9 @@ deploy() {
   testnet-edge)
     (
       set -x
-      ci/testnet-deploy.sh -p edge-testnet-solana-com -C gce -z us-west1-b \
+      ci/testnet-deploy.sh -p edge-devnet-solana-com -C gce -z us-west1-b \
         -t "$CHANNEL_OR_TAG" -n 3 -c 0 -u -P \
-        -a edge-testnet-solana-com --letsencrypt edge.testnet.solana.com \
+        -a edge-devnet-solana-com --letsencrypt edge.devnet.solana.com \
         --limit-ledger-size \
         ${skipCreate:+-e} \
         ${skipStart:+-s} \
@@ -337,24 +269,12 @@ deploy() {
         ${maybeDelete:+-D}
     )
     ;;
-  testnet-edge-perf)
-    (
-      set -x
-      RUST_LOG=solana=warn \
-        ci/testnet-deploy.sh -p edge-perf-testnet-solana-com -C ec2 -z us-west-2b \
-          -g -t "$CHANNEL_OR_TAG" -c 2 \
-          ${skipCreate:+-e} \
-          ${skipStart:+-s} \
-          ${maybeStop:+-S} \
-          ${maybeDelete:+-D}
-    )
-    ;;
   testnet-beta)
     (
       set -x
-      ci/testnet-deploy.sh -p beta-testnet-solana-com -C gce -z us-west1-b \
+      ci/testnet-deploy.sh -p beta-devnet-solana-com -C gce -z us-west1-b \
         -t "$CHANNEL_OR_TAG" -n 3 -c 0 -u -P \
-        -a beta-testnet-solana-com --letsencrypt beta.testnet.solana.com \
+        -a beta-devnet-solana-com --letsencrypt beta.devnet.solana.com \
         --limit-ledger-size \
         ${skipCreate:+-e} \
         ${skipStart:+-s} \
@@ -362,24 +282,12 @@ deploy() {
         ${maybeDelete:+-D}
     )
     ;;
-  testnet-beta-perf)
-    (
-      set -x
-      RUST_LOG=solana=warn \
-        ci/testnet-deploy.sh -p beta-perf-testnet-solana-com -C ec2 -z us-west-2b \
-          -g -t "$CHANNEL_OR_TAG" -c 2 \
-          ${skipCreate:+-e} \
-          ${skipStart:+-s} \
-          ${maybeStop:+-S} \
-          ${maybeDelete:+-D}
-    )
-    ;;
   testnet)
     (
       set -x
-      ci/testnet-deploy.sh -p testnet-solana-com -C gce -z us-west1-b \
-        -t "$CHANNEL_OR_TAG" -n 1 -c 0 -u -P \
-        -a testnet-solana-com --letsencrypt testnet.solana.com \
+      ci/testnet-deploy.sh -p devnet-solana-com -C gce -z us-west1-b \
+        -t "$CHANNEL_OR_TAG" -n 0 -c 0 -u -P \
+        -a testnet-solana-com --letsencrypt devnet.solana.com \
         --limit-ledger-size \
         ${skipCreate:+-e} \
         ${skipStart:+-s} \
@@ -389,159 +297,7 @@ deploy() {
     (
       echo "--- net.sh update"
       set -x
-      time net/net.sh update -t "$CHANNEL_OR_TAG" --platform linux --platform osx --platform windows
-    )
-    ;;
-  testnet-perf)
-    (
-      set -x
-      RUST_LOG=solana=warn \
-        ci/testnet-deploy.sh -p perf-testnet-solana-com -C gce -z us-west1-b \
-          -G "--machine-type n1-standard-16 --accelerator count=2,type=nvidia-tesla-v100" \
-          -t "$CHANNEL_OR_TAG" -c 2 \
-          -d pd-ssd \
-          ${skipCreate:+-e} \
-          ${skipStart:+-s} \
-          ${maybeStop:+-S} \
-          ${maybeDelete:+-D}
-    )
-    ;;
-  testnet-demo)
-    (
-      set -x
-
-      if [[ -n $GCE_LOW_QUOTA_NODE_COUNT ]] || [[ -n $skipStart ]]; then
-        maybeSkipStart="skip"
-      fi
-
-      # shellcheck disable=SC2068
-      ci/testnet-deploy.sh -p demo-testnet-solana-com -C gce ${GCE_ZONE_ARGS[@]} \
-        -t "$CHANNEL_OR_TAG" -n "$GCE_NODE_COUNT" -c 0 -P -u --allow-boot-failures \
-        --skip-remote-log-retrieval \
-        -a demo-testnet-solana-com \
-        ${skipCreate:+-e} \
-        ${maybeSkipStart:+-s} \
-        ${maybeStop:+-S} \
-        ${maybeDelete:+-D}
-
-      if [[ -n $GCE_LOW_QUOTA_NODE_COUNT ]]; then
-        # shellcheck disable=SC2068
-        ci/testnet-deploy.sh -p demo-testnet-solana-com2 -C gce ${GCE_LOW_QUOTA_ZONE_ARGS[@]} \
-          -t "$CHANNEL_OR_TAG" -n "$GCE_LOW_QUOTA_NODE_COUNT" -c 0 -P --allow-boot-failures -x \
-          --skip-remote-log-retrieval \
-          ${skipCreate:+-e} \
-          ${skipStart:+-s} \
-          ${maybeStop:+-S} \
-          ${maybeDelete:+-D}
-      fi
-    )
-    ;;
-  tds)
-    (
-      set -x
-
-      # Allow cluster configuration to be overridden from env vars
-
-      if [[ -z $TDS_ZONES ]]; then
-        TDS_ZONES="us-west1-a,us-central1-a,europe-west4-a"
-      fi
-      GCE_CLOUD_ZONES=(); while read -r -d, ; do GCE_CLOUD_ZONES+=( "$REPLY" ); done <<< "${TDS_ZONES},"
-
-      if [[ -z $TDS_NODE_COUNT ]]; then
-        TDS_NODE_COUNT="3"
-      fi
-
-      if [[ -z $TDS_CLIENT_COUNT ]]; then
-        TDS_CLIENT_COUNT="1"
-      fi
-
-      if [[ -z $ENABLE_GPU ]]; then
-        maybeGpu=(-G "--machine-type n1-standard-16 --accelerator count=2,type=nvidia-tesla-v100")
-      elif [[ $ENABLE_GPU == skip ]]; then
-        maybeGpu=()
-      else
-        maybeGpu=(-G "${ENABLE_GPU}")
-      fi
-
-      if [[ -z $HASHES_PER_TICK ]]; then
-        maybeHashesPerTick="--hashes-per-tick auto"
-      elif [[ $HASHES_PER_TICK == skip ]]; then
-        maybeHashesPerTick=""
-      else
-        maybeHashesPerTick="--hashes-per-tick ${HASHES_PER_TICK}"
-      fi
-
-      if [[ -z $DISABLE_AIRDROPS ]]; then
-        DISABLE_AIRDROPS="true"
-      fi
-
-      if [[ $DISABLE_AIRDROPS == true ]] ; then
-        maybeDisableAirdrops="--no-airdrop"
-      else
-        maybeDisableAirdrops=""
-      fi
-
-      if [[ -z $INTERNAL_NODES_STAKE_LAMPORTS ]]; then
-        maybeInternalNodesStakeLamports="--internal-nodes-stake-lamports 1000000000" # 1 SOL
-      elif [[ $INTERNAL_NODES_STAKE_LAMPORTS == skip ]]; then
-        maybeInternalNodesStakeLamports=""
-      else
-        maybeInternalNodesStakeLamports="--internal-nodes-stake-lamports ${INTERNAL_NODES_STAKE_LAMPORTS}"
-      fi
-
-      if [[ -z $INTERNAL_NODES_LAMPORTS ]]; then
-        maybeInternalNodesLamports="--internal-nodes-lamports 500000000000" # 500 SOL
-      elif [[ $INTERNAL_NODES_LAMPORTS == skip ]]; then
-        maybeInternalNodesLamports=""
-      else
-        maybeInternalNodesLamports="--internal-nodes-lamports ${INTERNAL_NODES_LAMPORTS}"
-      fi
-
-      EXTERNAL_ACCOUNTS_FILE=/tmp/validator.yml
-      if [[ -z $EXTERNAL_ACCOUNTS_FILE_URL ]]; then
-        EXTERNAL_ACCOUNTS_FILE_URL=https://raw.githubusercontent.com/solana-labs/tour-de-sol/master/validators/all.yml
-        wget ${EXTERNAL_ACCOUNTS_FILE_URL} -O ${EXTERNAL_ACCOUNTS_FILE}
-        maybeExternalAccountsFile="--external-accounts-file ${EXTERNAL_ACCOUNTS_FILE}"
-      elif [[ $EXTERNAL_ACCOUNTS_FILE_URL == skip ]]; then
-        maybeExternalAccountsFile=""
-      else
-        wget ${EXTERNAL_ACCOUNTS_FILE_URL} -O ${EXTERNAL_ACCOUNTS_FILE}
-        maybeExternalAccountsFile="--external-accounts-file ${EXTERNAL_ACCOUNTS_FILE}"
-      fi
-
-      if [[ -z $ADDITIONAL_DISK_SIZE_GB ]]; then
-        maybeAdditionalDisk="--validator-additional-disk-size-gb 32000"
-      elif [[ $ADDITIONAL_DISK_SIZE_GB == skip ]]; then
-        maybeAdditionalDisk=""
-      else
-        maybeAdditionalDisk="--validator-additional-disk-size-gb ${ADDITIONAL_DISK_SIZE_GB}"
-      fi
-
-      # Multiple V100 GPUs are available in us-west1, us-central1 and europe-west4
-      # shellcheck disable=SC2068
-      # shellcheck disable=SC2086
-      ci/testnet-deploy.sh -p tds-solana-com -C gce \
-        "${maybeGpu[@]}" \
-        -d pd-ssd \
-        ${GCE_CLOUD_ZONES[@]/#/-z } \
-        -t "$CHANNEL_OR_TAG" \
-        -n ${TDS_NODE_COUNT} \
-        -c ${TDS_CLIENT_COUNT} \
-        --idle-clients \
-        -P -u \
-        -a tds-solana-com --letsencrypt tds.solana.com \
-        ${maybeHashesPerTick} \
-        ${skipCreate:+-e} \
-        ${skipStart:+-s} \
-        ${maybeStop:+-S} \
-        ${maybeDelete:+-D} \
-        ${maybeDisableAirdrops} \
-        ${maybeInternalNodesStakeLamports} \
-        ${maybeInternalNodesLamports} \
-        ${maybeExternalAccountsFile} \
-        --target-lamports-per-signature 0 \
-        --slots-per-epoch 4096 \
-        ${maybeAdditionalDisk}
+      time net/net.sh update -t "$CHANNEL_OR_TAG" --platform linux --platform osx #--platform windows
     )
     ;;
   *)

clap-utils/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "solana-clap-utils"
-version = "0.23.0"
+version = "1.0.2"
 description = "Solana utilities for the clap"
 authors = ["Solana Maintainers <maintainers@solana.com>"]
 repository = "https://github.com/solana-labs/solana"
@@ -11,8 +11,8 @@ edition = "2018"
 [dependencies]
 clap = "2.33.0"
 rpassword = "4.0"
-semver = "0.9.0"
-solana-sdk = { path = "../sdk", version = "0.23.0" }
+solana-remote-wallet = { path = "../remote-wallet", version = "1.0.2" }
+solana-sdk = { path = "../sdk", version = "1.0.2" }
 tiny-bip39 = "0.7.0"
 url = "2.1.0"
 chrono = "0.4"

clap-utils/src/input_parsers.rs

@@ -1,13 +1,16 @@
-use crate::keypair::{keypair_from_seed_phrase, ASK_KEYWORD, SKIP_SEED_PHRASE_VALIDATION_ARG};
+use crate::keypair::{
+    keypair_from_seed_phrase, signer_from_path, ASK_KEYWORD, SKIP_SEED_PHRASE_VALIDATION_ARG,
+};
 use chrono::DateTime;
 use clap::ArgMatches;
+use solana_remote_wallet::remote_wallet::{DerivationPath, RemoteWalletManager};
 use solana_sdk::{
     clock::UnixTimestamp,
     native_token::sol_to_lamports,
     pubkey::Pubkey,
-    signature::{read_keypair_file, Keypair, KeypairUtil, Signature},
+    signature::{read_keypair_file, Keypair, Signature, Signer},
 };
-use std::str::FromStr;
+use std::{str::FromStr, sync::Arc};
 
 // Return parsed values from matches at `name`
 pub fn values_of<T>(matches: &ArgMatches<'_>, name: &str) -> Option<Vec<T>>
@@ -92,14 +95,36 @@ pub fn pubkeys_sigs_of(matches: &ArgMatches<'_>, name: &str) -> Option<Vec<(Pubk
     })
 }
 
-pub fn amount_of(matches: &ArgMatches<'_>, name: &str, unit: &str) -> Option<u64> {
-    if matches.value_of(unit) == Some("lamports") {
-        value_of(matches, name)
+// Return a signer from matches at `name`
+#[allow(clippy::type_complexity)]
+pub fn signer_of(
+    matches: &ArgMatches<'_>,
+    name: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
+) -> Result<(Option<Box<dyn Signer>>, Option<Pubkey>), Box<dyn std::error::Error>> {
+    if let Some(location) = matches.value_of(name) {
+        let signer = signer_from_path(matches, location, name, wallet_manager)?;
+        let signer_pubkey = signer.pubkey();
+        Ok((Some(signer), Some(signer_pubkey)))
     } else {
-        value_of(matches, name).map(sol_to_lamports)
+        Ok((None, None))
     }
 }
 
+pub fn lamports_of_sol(matches: &ArgMatches<'_>, name: &str) -> Option<u64> {
+    value_of(matches, name).map(sol_to_lamports)
+}
+
+pub fn derivation_of(matches: &ArgMatches<'_>, name: &str) -> Option<DerivationPath> {
+    matches.value_of(name).map(|derivation_str| {
+        let derivation_str = derivation_str.replace("'", "");
+        let mut parts = derivation_str.split('/');
+        let account = parts.next().map(|account| account.parse::<u32>().unwrap());
+        let change = parts.next().map(|change| change.parse::<u32>().unwrap());
+        DerivationPath { account, change }
+    })
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -258,23 +283,56 @@ mod tests {
     }
 
     #[test]
-    fn test_amount_of() {
+    fn test_lamports_of_sol() {
         let matches = app()
             .clone()
-            .get_matches_from(vec!["test", "--single", "50", "--unit", "lamports"]);
-        assert_eq!(amount_of(&matches, "single", "unit"), Some(50));
-        assert_eq!(amount_of(&matches, "multiple", "unit"), None);
+            .get_matches_from(vec!["test", "--single", "50"]);
+        assert_eq!(lamports_of_sol(&matches, "single"), Some(50000000000));
+        assert_eq!(lamports_of_sol(&matches, "multiple"), None);
         let matches = app()
             .clone()
-            .get_matches_from(vec!["test", "--single", "50", "--unit", "SOL"]);
-        assert_eq!(amount_of(&matches, "single", "unit"), Some(50000000000));
+            .get_matches_from(vec!["test", "--single", "1.5"]);
+        assert_eq!(lamports_of_sol(&matches, "single"), Some(1500000000));
+        assert_eq!(lamports_of_sol(&matches, "multiple"), None);
         let matches = app()
             .clone()
-            .get_matches_from(vec!["test", "--single", "1.5", "--unit", "SOL"]);
-        assert_eq!(amount_of(&matches, "single", "unit"), Some(1500000000));
+            .get_matches_from(vec!["test", "--single", "0.03"]);
+        assert_eq!(lamports_of_sol(&matches, "single"), Some(30000000));
+    }
+
+    #[test]
+    fn test_derivation_of() {
         let matches = app()
             .clone()
-            .get_matches_from(vec!["test", "--single", "1.5", "--unit", "lamports"]);
-        assert_eq!(amount_of(&matches, "single", "unit"), None);
+            .get_matches_from(vec!["test", "--single", "2/3"]);
+        assert_eq!(
+            derivation_of(&matches, "single"),
+            Some(DerivationPath {
+                account: Some(2),
+                change: Some(3)
+            })
+        );
+        assert_eq!(derivation_of(&matches, "another"), None);
+        let matches = app()
+            .clone()
+            .get_matches_from(vec!["test", "--single", "2"]);
+        assert_eq!(
+            derivation_of(&matches, "single"),
+            Some(DerivationPath {
+                account: Some(2),
+                change: None
+            })
+        );
+        assert_eq!(derivation_of(&matches, "another"), None);
+        let matches = app()
+            .clone()
+            .get_matches_from(vec!["test", "--single", "2'/3'"]);
+        assert_eq!(
+            derivation_of(&matches, "single"),
+            Some(DerivationPath {
+                account: Some(2),
+                change: Some(3)
+            })
+        );
     }
 }

clap-utils/src/input_validators.rs

@@ -1,8 +1,11 @@
-use crate::keypair::ASK_KEYWORD;
+use crate::keypair::{parse_keypair_path, KeypairUrl, ASK_KEYWORD};
 use chrono::DateTime;
-use solana_sdk::hash::Hash;
-use solana_sdk::pubkey::Pubkey;
-use solana_sdk::signature::{read_keypair_file, Signature};
+use solana_sdk::{
+    clock::Slot,
+    hash::Hash,
+    pubkey::Pubkey,
+    signature::{read_keypair_file, Signature},
+};
 use std::str::FromStr;
 
 // Return an error if a pubkey cannot be parsed.
@@ -48,6 +51,13 @@ pub fn is_pubkey_or_keypair_or_ask_keyword(string: String) -> Result<(), String>
     is_pubkey(string.clone()).or_else(|_| is_keypair_or_ask_keyword(string))
 }
 
+pub fn is_valid_signer(string: String) -> Result<(), String> {
+    match parse_keypair_path(&string) {
+        KeypairUrl::Filepath(path) => is_keypair(path),
+        _ => Ok(()),
+    }
+}
+
 // Return an error if string cannot be parsed as pubkey=signature string
 pub fn is_pubkey_sig(string: String) -> Result<(), String> {
     let mut signer = string.split('=');
@@ -84,18 +94,10 @@ pub fn is_url(string: String) -> Result<(), String> {
     }
 }
 
-pub fn is_semver(semver: &str) -> Result<(), String> {
-    match semver::Version::parse(&semver) {
-        Ok(_) => Ok(()),
-        Err(err) => Err(format!("{:?}", err)),
-    }
-}
-
-pub fn is_release_channel(channel: &str) -> Result<(), String> {
-    match channel {
-        "edge" | "beta" | "stable" => Ok(()),
-        _ => Err(format!("Invalid release channel {}", channel)),
-    }
+pub fn is_slot(slot: String) -> Result<(), String> {
+    slot.parse::<Slot>()
+        .map(|_| ())
+        .map_err(|e| format!("{:?}", e))
 }
 
 pub fn is_port(port: String) -> Result<(), String> {
@@ -141,3 +143,48 @@ pub fn is_rfc3339_datetime(value: String) -> Result<(), String> {
         .map(|_| ())
         .map_err(|e| format!("{:?}", e))
 }
+
+pub fn is_derivation(value: String) -> Result<(), String> {
+    let value = value.replace("'", "");
+    let mut parts = value.split('/');
+    let account = parts.next().unwrap();
+    account
+        .parse::<u32>()
+        .map_err(|e| {
+            format!(
+                "Unable to parse derivation, provided: {}, err: {:?}",
+                account, e
+            )
+        })
+        .and_then(|_| {
+            if let Some(change) = parts.next() {
+                change.parse::<u32>().map_err(|e| {
+                    format!(
+                        "Unable to parse derivation, provided: {}, err: {:?}",
+                        change, e
+                    )
+                })
+            } else {
+                Ok(0)
+            }
+        })
+        .map(|_| ())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_is_derivation() {
+        assert_eq!(is_derivation("2".to_string()), Ok(()));
+        assert_eq!(is_derivation("0".to_string()), Ok(()));
+        assert_eq!(is_derivation("65537".to_string()), Ok(()));
+        assert_eq!(is_derivation("0/2".to_string()), Ok(()));
+        assert_eq!(is_derivation("0'/2'".to_string()), Ok(()));
+        assert!(is_derivation("a".to_string()).is_err());
+        assert!(is_derivation("4294967296".to_string()).is_err());
+        assert!(is_derivation("a/b".to_string()).is_err());
+        assert!(is_derivation("0/4294967296".to_string()).is_err());
+    }
+}

clap-utils/src/keypair.rs

@@ -1,20 +1,114 @@
-use crate::ArgConstant;
+use crate::{
+    input_parsers::{derivation_of, pubkeys_sigs_of},
+    offline::SIGNER_ARG,
+    ArgConstant,
+};
 use bip39::{Language, Mnemonic, Seed};
-use clap::values_t;
+use clap::{values_t, ArgMatches, Error, ErrorKind};
 use rpassword::prompt_password_stderr;
+use solana_remote_wallet::{
+    remote_keypair::generate_remote_keypair,
+    remote_wallet::{RemoteWalletError, RemoteWalletManager},
+};
 use solana_sdk::{
     pubkey::Pubkey,
     signature::{
-        keypair_from_seed, keypair_from_seed_phrase_and_passphrase, read_keypair_file, Keypair,
-        KeypairUtil,
+        keypair_from_seed, keypair_from_seed_phrase_and_passphrase, read_keypair,
+        read_keypair_file, Keypair, Presigner, Signature, Signer,
     },
 };
 use std::{
     error,
     io::{stdin, stdout, Write},
     process::exit,
+    str::FromStr,
+    sync::Arc,
 };
 
+pub enum KeypairUrl {
+    Ask,
+    Filepath(String),
+    Usb(String),
+    Stdin,
+    Pubkey(Pubkey),
+}
+
+pub fn parse_keypair_path(path: &str) -> KeypairUrl {
+    if path == "-" {
+        KeypairUrl::Stdin
+    } else if path == ASK_KEYWORD {
+        KeypairUrl::Ask
+    } else if path.starts_with("usb://") {
+        KeypairUrl::Usb(path.to_string())
+    } else if let Ok(pubkey) = Pubkey::from_str(path) {
+        KeypairUrl::Pubkey(pubkey)
+    } else {
+        KeypairUrl::Filepath(path.to_string())
+    }
+}
+
+pub fn presigner_from_pubkey_sigs(
+    pubkey: &Pubkey,
+    signers: &[(Pubkey, Signature)],
+) -> Option<Presigner> {
+    signers.iter().find_map(|(signer, sig)| {
+        if *signer == *pubkey {
+            Some(Presigner::new(signer, sig))
+        } else {
+            None
+        }
+    })
+}
+
+pub fn signer_from_path(
+    matches: &ArgMatches,
+    path: &str,
+    keypair_name: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
+) -> Result<Box<dyn Signer>, Box<dyn error::Error>> {
+    match parse_keypair_path(path) {
+        KeypairUrl::Ask => {
+            let skip_validation = matches.is_present(SKIP_SEED_PHRASE_VALIDATION_ARG.name);
+            Ok(Box::new(keypair_from_seed_phrase(
+                keypair_name,
+                skip_validation,
+                false,
+            )?))
+        }
+        KeypairUrl::Filepath(path) => Ok(Box::new(read_keypair_file(&path)?)),
+        KeypairUrl::Stdin => {
+            let mut stdin = std::io::stdin();
+            Ok(Box::new(read_keypair(&mut stdin)?))
+        }
+        KeypairUrl::Usb(path) => {
+            if let Some(wallet_manager) = wallet_manager {
+                Ok(Box::new(generate_remote_keypair(
+                    path,
+                    derivation_of(matches, "derivation_path"),
+                    wallet_manager,
+                    matches.is_present("confirm_key"),
+                )?))
+            } else {
+                Err(RemoteWalletError::NoDeviceFound.into())
+            }
+        }
+        KeypairUrl::Pubkey(pubkey) => {
+            let presigner = pubkeys_sigs_of(matches, SIGNER_ARG.name)
+                .as_ref()
+                .and_then(|presigners| presigner_from_pubkey_sigs(&pubkey, presigners));
+            if let Some(presigner) = presigner {
+                Ok(Box::new(presigner))
+            } else {
+                Err(Error::with_description(
+                    "Missing signature for supplied pubkey",
+                    ErrorKind::MissingRequiredArgument,
+                )
+                .into())
+            }
+        }
+    }
+}
+
 // Keyword used to indicate that the user should be asked for a keypair seed phrase
 pub const ASK_KEYWORD: &str = "ASK";
 
@@ -32,8 +126,8 @@ pub const SKIP_SEED_PHRASE_VALIDATION_ARG: ArgConstant<'static> = ArgConstant {
 
 #[derive(Debug, PartialEq)]
 pub enum Source {
-    File,
     Generated,
+    Path,
     SeedPhrase,
 }
 
@@ -87,7 +181,7 @@ pub fn keypair_from_seed_phrase(
     };
 
     if confirm_pubkey {
-        let pubkey = Pubkey::new(keypair.public.as_ref());
+        let pubkey = keypair.pubkey();
         print!("Recovered pubkey `{:?}`. Continue? (y/n): ", pubkey);
         let _ignored = stdout().flush();
         let mut input = String::new();
@@ -131,7 +225,12 @@ pub fn keypair_input(
         keypair_from_seed_phrase(keypair_name, skip_validation, true)
             .map(|keypair| KeypairWithSource::new(keypair, Source::SeedPhrase))
     } else if let Some(keypair_file) = matches.value_of(keypair_match_name) {
-        read_keypair_file(keypair_file).map(|keypair| KeypairWithSource::new(keypair, Source::File))
+        if keypair_file.starts_with("usb://") {
+            Ok(KeypairWithSource::new(Keypair::new(), Source::Path))
+        } else {
+            read_keypair_file(keypair_file)
+                .map(|keypair| KeypairWithSource::new(keypair, Source::Path))
+        }
     } else {
         Ok(KeypairWithSource::new(Keypair::new(), Source::Generated))
     }

clap-utils/src/lib.rs

@@ -26,3 +26,4 @@ pub struct ArgConstant<'a> {
 pub mod input_parsers;
 pub mod input_validators;
 pub mod keypair;
+pub mod offline;

clap-utils/src/offline.rs

@@ -0,0 +1,19 @@
+use crate::ArgConstant;
+
+pub const BLOCKHASH_ARG: ArgConstant<'static> = ArgConstant {
+    name: "blockhash",
+    long: "blockhash",
+    help: "Use the supplied blockhash",
+};
+
+pub const SIGN_ONLY_ARG: ArgConstant<'static> = ArgConstant {
+    name: "sign_only",
+    long: "sign-only",
+    help: "Sign the transaction offline",
+};
+
+pub const SIGNER_ARG: ArgConstant<'static> = ArgConstant {
+    name: "signer",
+    long: "signer",
+    help: "Provide a public-key/signature pair for the transaction",
+};

cli-config/Cargo.toml

@@ -0,0 +1,16 @@
+[package]
+authors = ["Solana Maintainers <maintainers@solana.com>"]
+edition = "2018"
+name = "solana-cli-config"
+description = "Blockchain, Rebuilt for Scale"
+version = "1.0.2"
+repository = "https://github.com/solana-labs/solana"
+license = "Apache-2.0"
+homepage = "https://solana.com/"
+
+[dependencies]
+dirs = "2.0.2"
+lazy_static = "1.4.0"
+serde = "1.0.104"
+serde_derive = "1.0.103"
+serde_yaml = "0.8.11"

cli-config/src/config.rs

@@ -1,8 +1,10 @@
 // Wallet settings that can be configured for long-term use
 use serde_derive::{Deserialize, Serialize};
-use std::fs::{create_dir_all, File};
-use std::io::{self, Write};
-use std::path::Path;
+use std::{
+    fs::{create_dir_all, File},
+    io::{self, Write},
+    path::Path,
+};
 
 lazy_static! {
     pub static ref CONFIG_FILE: Option<String> = {
@@ -16,13 +18,15 @@ lazy_static! {
 #[derive(Serialize, Deserialize, Default, Debug, PartialEq)]
 pub struct Config {
     pub url: String,
+    pub websocket_url: String,
     pub keypair_path: String,
 }
 
 impl Config {
-    pub fn new(url: &str, keypair_path: &str) -> Self {
+    pub fn new(url: &str, websocket_url: &str, keypair_path: &str) -> Self {
         Self {
             url: url.to_string(),
+            websocket_url: websocket_url.to_string(),
             keypair_path: keypair_path.to_string(),
         }
     }

cli-config/src/lib.rs

@@ -0,0 +1,4 @@
+#[macro_use]
+extern crate lazy_static;
+
+pub mod config;

cli/Cargo.toml

@@ -3,7 +3,7 @@ authors = ["Solana Maintainers <maintainers@solana.com>"]
 edition = "2018"
 name = "solana-cli"
 description = "Blockchain, Rebuilt for Scale"
-version = "0.23.0"
+version = "1.0.2"
 repository = "https://github.com/solana-labs/solana"
 license = "Apache-2.0"
 homepage = "https://solana.com/"
@@ -14,38 +14,39 @@ bs58 = "0.3.0"
 chrono = { version = "0.4.10", features = ["serde"] }
 clap = "2.33.0"
 criterion-stats = "0.3.0"
-ctrlc = { version = "3.1.3", features = ["termination"] }
-console = "0.9.1"
+ctrlc = { version = "3.1.4", features = ["termination"] }
+console = "0.9.2"
 dirs = "2.0.2"
-lazy_static = "1.4.0"
 log = "0.4.8"
-indicatif = "0.13.0"
+indicatif = "0.14.0"
 humantime = "2.0.0"
 num-traits = "0.2"
 pretty-hex = "0.1.1"
 reqwest = { version = "0.10.1", default-features = false, features = ["blocking", "rustls-tls"] }
 serde = "1.0.104"
 serde_derive = "1.0.103"
-serde_json = "1.0.44"
-serde_yaml = "0.8.11"
-solana-budget-program = { path = "../programs/budget", version = "0.23.0" }
-solana-clap-utils = { path = "../clap-utils", version = "0.23.0" }
-solana-client = { path = "../client", version = "0.23.0" }
-solana-config-program = { path = "../programs/config", version = "0.23.0" }
-solana-faucet = { path = "../faucet", version = "0.23.0" }
-solana-logger = { path = "../logger", version = "0.23.0" }
-solana-net-utils = { path = "../net-utils", version = "0.23.0" }
-solana-runtime = { path = "../runtime", version = "0.23.0" }
-solana-sdk = { path = "../sdk", version = "0.23.0" }
-solana-stake-program = { path = "../programs/stake", version = "0.23.0" }
-solana-storage-program = { path = "../programs/storage", version = "0.23.0" }
-solana-vote-program = { path = "../programs/vote", version = "0.23.0" }
-solana-vote-signer = { path = "../vote-signer", version = "0.23.0" }
+serde_json = "1.0.46"
+solana-budget-program = { path = "../programs/budget", version = "1.0.2" }
+solana-clap-utils = { path = "../clap-utils", version = "1.0.2" }
+solana-cli-config = { path = "../cli-config", version = "1.0.2" }
+solana-client = { path = "../client", version = "1.0.2" }
+solana-config-program = { path = "../programs/config", version = "1.0.2" }
+solana-faucet = { path = "../faucet", version = "1.0.2" }
+solana-logger = { path = "../logger", version = "1.0.2" }
+solana-net-utils = { path = "../net-utils", version = "1.0.2" }
+solana-remote-wallet = { path = "../remote-wallet", version = "1.0.2" }
+solana-runtime = { path = "../runtime", version = "1.0.2" }
+solana-sdk = { path = "../sdk", version = "1.0.2" }
+solana-stake-program = { path = "../programs/stake", version = "1.0.2" }
+solana-storage-program = { path = "../programs/storage", version = "1.0.2" }
+solana-vote-program = { path = "../programs/vote", version = "1.0.2" }
+solana-vote-signer = { path = "../vote-signer", version = "1.0.2" }
+titlecase = "1.1.0"
 url = "2.1.1"
 
 [dev-dependencies]
-solana-core = { path = "../core", version = "0.23.0" }
-solana-budget-program = { path = "../programs/budget", version = "0.23.0" }
+solana-core = { path = "../core", version = "1.0.2" }
+solana-budget-program = { path = "../programs/budget", version = "1.0.2" }
 tempfile = "3.1.0"
 
 [[bin]]

cli/src/cluster_query.rs

@@ -8,21 +8,32 @@ use crate::{
 use clap::{value_t, value_t_or_exit, App, Arg, ArgMatches, SubCommand};
 use console::{style, Emoji};
 use indicatif::{ProgressBar, ProgressStyle};
-use solana_clap_utils::{input_parsers::*, input_validators::*};
-use solana_client::{rpc_client::RpcClient, rpc_response::RpcVoteAccountInfo};
+use solana_clap_utils::{input_parsers::*, input_validators::*, keypair::signer_from_path};
+use solana_client::{
+    pubsub_client::{PubsubClient, SlotInfoMessage},
+    rpc_client::RpcClient,
+    rpc_response::RpcVoteAccountInfo,
+};
+use solana_remote_wallet::remote_wallet::RemoteWalletManager;
 use solana_sdk::{
     account_utils::StateMut,
     clock::{self, Slot},
     commitment_config::CommitmentConfig,
-    epoch_schedule::{Epoch, EpochSchedule},
+    epoch_schedule::Epoch,
     hash::Hash,
+    message::Message,
     pubkey::Pubkey,
-    signature::{Keypair, KeypairUtil},
-    system_transaction,
+    signature::{Keypair, Signer},
+    system_instruction,
+    transaction::Transaction,
 };
 use std::{
     collections::{HashMap, VecDeque},
     net::SocketAddr,
+    sync::{
+        atomic::{AtomicBool, Ordering},
+        Arc,
+    },
     thread::sleep,
     time::{Duration, Instant},
 };
@@ -67,6 +78,7 @@ impl ClusterQuerySubCommands for App<'_, '_> {
                     .help("Slot number of the block to query")
             )
         )
+        .subcommand(SubCommand::with_name("leader-schedule").about("Display leader schedule"))
         .subcommand(
             SubCommand::with_name("epoch-info")
             .about("Get information about the current epoch")
@@ -156,6 +168,10 @@ impl ClusterQuerySubCommands for App<'_, '_> {
                         ),
                 ),
         )
+        .subcommand(
+            SubCommand::with_name("live-slots")
+                .about("Show information about the current slot progression"),
+        )
         .subcommand(
             SubCommand::with_name("block-production")
                 .about("Show information about block production")
@@ -215,11 +231,15 @@ pub fn parse_catchup(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, CliErro
     let node_pubkey = pubkey_of(matches, "node_pubkey").unwrap();
     Ok(CliCommandInfo {
         command: CliCommand::Catchup { node_pubkey },
-        require_keypair: false,
+        signers: vec![],
     })
 }
 
-pub fn parse_cluster_ping(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, CliError> {
+pub fn parse_cluster_ping(
+    matches: &ArgMatches<'_>,
+    default_signer_path: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
+) -> Result<CliCommandInfo, CliError> {
     let lamports = value_t_or_exit!(matches, "lamports", u64);
     let interval = Duration::from_secs(value_t_or_exit!(matches, "interval", u64));
     let count = if matches.is_present("count") {
@@ -241,7 +261,12 @@ pub fn parse_cluster_ping(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, Cl
             timeout,
             commitment_config,
         },
-        require_keypair: true,
+        signers: vec![signer_from_path(
+            matches,
+            default_signer_path,
+            "keypair",
+            wallet_manager,
+        )?],
     })
 }
 
@@ -249,7 +274,7 @@ pub fn parse_get_block_time(matches: &ArgMatches<'_>) -> Result<CliCommandInfo,
     let slot = value_t_or_exit!(matches, "slot", u64);
     Ok(CliCommandInfo {
         command: CliCommand::GetBlockTime { slot },
-        require_keypair: false,
+        signers: vec![],
     })
 }
 
@@ -261,7 +286,7 @@ pub fn parse_get_epoch_info(matches: &ArgMatches<'_>) -> Result<CliCommandInfo,
     };
     Ok(CliCommandInfo {
         command: CliCommand::GetEpochInfo { commitment_config },
-        require_keypair: false,
+        signers: vec![],
     })
 }
 
@@ -273,7 +298,7 @@ pub fn parse_get_slot(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, CliErr
     };
     Ok(CliCommandInfo {
         command: CliCommand::GetSlot { commitment_config },
-        require_keypair: false,
+        signers: vec![],
     })
 }
 
@@ -285,7 +310,7 @@ pub fn parse_get_transaction_count(matches: &ArgMatches<'_>) -> Result<CliComman
     };
     Ok(CliCommandInfo {
         command: CliCommand::GetTransactionCount { commitment_config },
-        require_keypair: false,
+        signers: vec![],
     })
 }
 
@@ -298,7 +323,7 @@ pub fn parse_show_stakes(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, Cli
             use_lamports_unit,
             vote_account_pubkeys,
         },
-        require_keypair: false,
+        signers: vec![],
     })
 }
 
@@ -307,7 +332,7 @@ pub fn parse_show_validators(matches: &ArgMatches<'_>) -> Result<CliCommandInfo,
 
     Ok(CliCommandInfo {
         command: CliCommand::ShowValidators { use_lamports_unit },
-        require_keypair: false,
+        signers: vec![],
     })
 }
 
@@ -320,20 +345,6 @@ fn new_spinner_progress_bar() -> ProgressBar {
     progress_bar
 }
 
-/// Aggregate epoch credit stats and return (total credits, total slots, total epochs)
-pub fn aggregate_epoch_credits(
-    epoch_credits: &[(Epoch, u64, u64)],
-    epoch_schedule: &EpochSchedule,
-) -> (u64, u64, u64) {
-    epoch_credits
-        .iter()
-        .fold((0, 0, 0), |acc, (epoch, credits, prev_credits)| {
-            let credits_earned = credits - prev_credits;
-            let slots_in_epoch = epoch_schedule.get_slots_in_epoch(*epoch);
-            (acc.0 + credits_earned, acc.1 + slots_in_epoch, acc.2 + 1)
-        })
-}
-
 pub fn process_catchup(rpc_client: &RpcClient, node_pubkey: &Pubkey) -> ProcessResult {
     let cluster_nodes = rpc_client.get_cluster_nodes()?;
 
@@ -406,6 +417,41 @@ pub fn process_fees(rpc_client: &RpcClient) -> ProcessResult {
     ))
 }
 
+pub fn process_leader_schedule(rpc_client: &RpcClient) -> ProcessResult {
+    let epoch_info = rpc_client.get_epoch_info()?;
+    let first_slot_in_epoch = epoch_info.absolute_slot - epoch_info.slot_index;
+
+    let leader_schedule = rpc_client.get_leader_schedule(Some(first_slot_in_epoch))?;
+    if leader_schedule.is_none() {
+        return Err(format!(
+            "Unable to fetch leader schedule for slot {}",
+            first_slot_in_epoch
+        )
+        .into());
+    }
+    let leader_schedule = leader_schedule.unwrap();
+
+    let mut leader_per_slot_index = Vec::new();
+    for (pubkey, leader_slots) in leader_schedule.iter() {
+        for slot_index in leader_slots.iter() {
+            if *slot_index >= leader_per_slot_index.len() {
+                leader_per_slot_index.resize(*slot_index + 1, "?");
+            }
+            leader_per_slot_index[*slot_index] = pubkey;
+        }
+    }
+
+    for (slot_index, leader) in leader_per_slot_index.iter().enumerate() {
+        println!(
+            "  {:<15} {:<44}",
+            first_slot_in_epoch + slot_index as u64,
+            leader
+        );
+    }
+
+    Ok("".to_string())
+}
+
 pub fn process_get_block_time(rpc_client: &RpcClient, slot: Slot) -> ProcessResult {
     let timestamp = rpc_client.get_block_time(slot)?;
     Ok(timestamp.to_string())
@@ -429,11 +475,11 @@ pub fn process_get_epoch_info(
     let start_slot = epoch_info.absolute_slot - epoch_info.slot_index;
     let end_slot = start_slot + epoch_info.slots_in_epoch;
     println_name_value(
-        "Epoch slot range:",
+        "Epoch Slot Range:",
         &format!("[{}..{})", start_slot, end_slot),
     );
     println_name_value(
-        "Epoch completed percent:",
+        "Epoch Completed Percent:",
         &format!(
             "{:>3.3}%",
             epoch_info.slot_index as f64 / epoch_info.slots_in_epoch as f64 * 100_f64
@@ -441,14 +487,14 @@ pub fn process_get_epoch_info(
     );
     let remaining_slots_in_epoch = epoch_info.slots_in_epoch - epoch_info.slot_index;
     println_name_value(
-        "Epoch completed slots:",
+        "Epoch Completed Slots:",
         &format!(
             "{}/{} ({} remaining)",
             epoch_info.slot_index, epoch_info.slots_in_epoch, remaining_slots_in_epoch
         ),
     );
     println_name_value(
-        "Epoch completed time:",
+        "Epoch Completed Time:",
         &format!(
             "{}/{} ({} remaining)",
             slot_to_human_time(epoch_info.slot_index),
@@ -478,7 +524,7 @@ pub fn parse_show_block_production(matches: &ArgMatches<'_>) -> Result<CliComman
 
     Ok(CliCommandInfo {
         command: CliCommand::ShowBlockProduction { epoch, slot_limit },
-        require_keypair: false,
+        signers: vec![],
     })
 }
 
@@ -673,8 +719,8 @@ pub fn process_ping(
 ) -> ProcessResult {
     let to = Keypair::new().pubkey();
 
-    println_name_value("Source account:", &config.keypair.pubkey().to_string());
-    println_name_value("Destination account:", &to.to_string());
+    println_name_value("Source Account:", &config.signers[0].pubkey().to_string());
+    println_name_value("Destination Account:", &to.to_string());
     println!();
 
     let (signal_sender, signal_receiver) = std::sync::mpsc::channel();
@@ -692,11 +738,13 @@ pub fn process_ping(
         let (recent_blockhash, fee_calculator) = rpc_client.get_new_blockhash(&last_blockhash)?;
         last_blockhash = recent_blockhash;
 
-        let transaction =
-            system_transaction::transfer(&config.keypair, &to, lamports, recent_blockhash);
+        let ix = system_instruction::transfer(&config.signers[0].pubkey(), &to, lamports);
+        let message = Message::new(vec![ix]);
+        let mut transaction = Transaction::new_unsigned(message);
+        transaction.try_sign(&config.signers, recent_blockhash)?;
         check_account_for_fee(
             rpc_client,
-            &config.keypair.pubkey(),
+            &config.signers[0].pubkey(),
             &fee_calculator,
             &transaction.message,
         )?;
@@ -788,6 +836,101 @@ pub fn process_ping(
     Ok("".to_string())
 }
 
+pub fn process_live_slots(url: &str) -> ProcessResult {
+    let exit = Arc::new(AtomicBool::new(false));
+
+    // Disable Ctrl+C handler as sometimes the PubsubClient shutdown can stall.  Also it doesn't
+    // really matter that the shutdown is clean because the process is terminating.
+    /*
+    let exit_clone = exit.clone();
+    ctrlc::set_handler(move || {
+        exit_clone.store(true, Ordering::Relaxed);
+    })?;
+    */
+
+    let mut current: Option<SlotInfoMessage> = None;
+    let mut message = "".to_string();
+
+    let slot_progress = new_spinner_progress_bar();
+    slot_progress.set_message("Connecting...");
+    let (mut client, receiver) = PubsubClient::slot_subscribe(url)?;
+    slot_progress.set_message("Connected.");
+
+    let spacer = "|";
+    slot_progress.println(spacer);
+
+    let mut last_root = std::u64::MAX;
+    let mut last_root_update = Instant::now();
+    let mut slots_per_second = std::f64::NAN;
+    loop {
+        if exit.load(Ordering::Relaxed) {
+            eprintln!("{}", message);
+            client.shutdown().unwrap();
+            break;
+        }
+
+        match receiver.recv() {
+            Ok(new_info) => {
+                if last_root == std::u64::MAX {
+                    last_root = new_info.root;
+                    last_root_update = Instant::now();
+                }
+                if last_root_update.elapsed().as_secs() >= 5 {
+                    let root = new_info.root;
+                    slots_per_second =
+                        (root - last_root) as f64 / last_root_update.elapsed().as_secs() as f64;
+                    last_root_update = Instant::now();
+                    last_root = root;
+                }
+
+                message = if slots_per_second.is_nan() {
+                    format!("{:?}", new_info)
+                } else {
+                    format!(
+                        "{:?} | root slot advancing at {:.2} slots/second",
+                        new_info, slots_per_second
+                    )
+                }
+                .to_owned();
+                slot_progress.set_message(&message);
+
+                if let Some(previous) = current {
+                    let slot_delta: i64 = new_info.slot as i64 - previous.slot as i64;
+                    let root_delta: i64 = new_info.root as i64 - previous.root as i64;
+
+                    //
+                    // if slot has advanced out of step with the root, we detect
+                    // a mismatch and output the slot information
+                    //
+                    if slot_delta != root_delta {
+                        let prev_root = format!(
+                            "|<--- {} <- … <- {} <- {}   (prev)",
+                            previous.root, previous.parent, previous.slot
+                        );
+                        slot_progress.println(&prev_root);
+
+                        let new_root = format!(
+                            "|  '- {} <- … <- {} <- {}   (next)",
+                            new_info.root, new_info.parent, new_info.slot
+                        );
+
+                        slot_progress.println(prev_root);
+                        slot_progress.println(new_root);
+                        slot_progress.println(spacer);
+                    }
+                }
+                current = Some(new_info);
+            }
+            Err(err) => {
+                eprintln!("disconnected: {:?}", err);
+                break;
+            }
+        }
+    }
+
+    Ok("".to_string())
+}
+
 pub fn process_show_gossip(rpc_client: &RpcClient) -> ProcessResult {
     let cluster_nodes = rpc_client.get_cluster_nodes()?;
 
@@ -864,7 +1007,7 @@ pub fn process_show_stakes(
 }
 
 pub fn process_show_validators(rpc_client: &RpcClient, use_lamports_unit: bool) -> ProcessResult {
-    let epoch_schedule = rpc_client.get_epoch_schedule()?;
+    let epoch_info = rpc_client.get_epoch_info()?;
     let vote_accounts = rpc_client.get_vote_accounts()?;
     let total_active_stake = vote_accounts
         .current
@@ -913,7 +1056,7 @@ pub fn process_show_validators(rpc_client: &RpcClient, use_lamports_unit: bool)
             "Commission",
             "Last Vote",
             "Root Block",
-            "Uptime",
+            "Credits",
             "Active Stake",
         ))
         .bold()
@@ -921,7 +1064,7 @@ pub fn process_show_validators(rpc_client: &RpcClient, use_lamports_unit: bool)
 
     fn print_vote_account(
         vote_account: RpcVoteAccountInfo,
-        epoch_schedule: &EpochSchedule,
+        current_epoch: Epoch,
         total_active_stake: f64,
         use_lamports_unit: bool,
         delinquent: bool,
@@ -934,17 +1077,6 @@ pub fn process_show_validators(rpc_client: &RpcClient, use_lamports_unit: bool)
             }
         }
 
-        fn uptime(epoch_credits: Vec<(Epoch, u64, u64)>, epoch_schedule: &EpochSchedule) -> String {
-            let (total_credits, total_slots, _) =
-                aggregate_epoch_credits(&epoch_credits, &epoch_schedule);
-            if total_slots > 0 {
-                let total_uptime = 100_f64 * total_credits as f64 / total_slots as f64;
-                format!("{:.2}%", total_uptime)
-            } else {
-                "-".into()
-            }
-        }
-
         println!(
             "{} {:<44}  {:<44}  {:>9}%   {:>8}  {:>10}  {:>7}  {}",
             if delinquent {
@@ -957,7 +1089,15 @@ pub fn process_show_validators(rpc_client: &RpcClient, use_lamports_unit: bool)
             vote_account.commission,
             non_zero_or_dash(vote_account.last_vote),
             non_zero_or_dash(vote_account.root_slot),
-            uptime(vote_account.epoch_credits, epoch_schedule),
+            vote_account
+                .epoch_credits
+                .iter()
+                .find_map(|(epoch, credits, _)| if *epoch == current_epoch {
+                    Some(*credits)
+                } else {
+                    None
+                })
+                .unwrap_or(0),
             if vote_account.activated_stake > 0 {
                 format!(
                     "{} ({:.2}%)",
@@ -973,7 +1113,7 @@ pub fn process_show_validators(rpc_client: &RpcClient, use_lamports_unit: bool)
     for vote_account in vote_accounts.current.into_iter() {
         print_vote_account(
             vote_account,
-            &epoch_schedule,
+            epoch_info.epoch,
             total_active_stake,
             use_lamports_unit,
             false,
@@ -982,7 +1122,7 @@ pub fn process_show_validators(rpc_client: &RpcClient, use_lamports_unit: bool)
     for vote_account in vote_accounts.delinquent.into_iter() {
         print_vote_account(
             vote_account,
-            &epoch_schedule,
+            epoch_info.epoch,
             total_active_stake,
             use_lamports_unit,
             true,
@@ -996,28 +1136,38 @@ pub fn process_show_validators(rpc_client: &RpcClient, use_lamports_unit: bool)
 mod tests {
     use super::*;
     use crate::cli::{app, parse_command};
+    use solana_sdk::signature::{write_keypair, Keypair};
+    use tempfile::NamedTempFile;
+
+    fn make_tmp_file() -> (String, NamedTempFile) {
+        let tmp_file = NamedTempFile::new().unwrap();
+        (String::from(tmp_file.path().to_str().unwrap()), tmp_file)
+    }
 
     #[test]
     fn test_parse_command() {
         let test_commands = app("test", "desc", "version");
+        let default_keypair = Keypair::new();
+        let (default_keypair_file, mut tmp_file) = make_tmp_file();
+        write_keypair(&default_keypair, tmp_file.as_file_mut()).unwrap();
 
         let test_cluster_version = test_commands
             .clone()
             .get_matches_from(vec!["test", "cluster-version"]);
         assert_eq!(
-            parse_command(&test_cluster_version).unwrap(),
+            parse_command(&test_cluster_version, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::ClusterVersion,
-                require_keypair: false
+                signers: vec![],
             }
         );
 
         let test_fees = test_commands.clone().get_matches_from(vec!["test", "fees"]);
         assert_eq!(
-            parse_command(&test_fees).unwrap(),
+            parse_command(&test_fees, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::Fees,
-                require_keypair: false
+                signers: vec![],
             }
         );
 
@@ -1027,10 +1177,10 @@ mod tests {
                 .clone()
                 .get_matches_from(vec!["test", "block-time", &slot.to_string()]);
         assert_eq!(
-            parse_command(&test_get_block_time).unwrap(),
+            parse_command(&test_get_block_time, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::GetBlockTime { slot },
-                require_keypair: false
+                signers: vec![],
             }
         );
 
@@ -1038,12 +1188,12 @@ mod tests {
             .clone()
             .get_matches_from(vec!["test", "epoch-info"]);
         assert_eq!(
-            parse_command(&test_get_epoch_info).unwrap(),
+            parse_command(&test_get_epoch_info, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::GetEpochInfo {
                     commitment_config: CommitmentConfig::recent(),
                 },
-                require_keypair: false
+                signers: vec![],
             }
         );
 
@@ -1051,21 +1201,21 @@ mod tests {
             .clone()
             .get_matches_from(vec!["test", "genesis-hash"]);
         assert_eq!(
-            parse_command(&test_get_genesis_hash).unwrap(),
+            parse_command(&test_get_genesis_hash, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::GetGenesisHash,
-                require_keypair: false
+                signers: vec![],
             }
         );
 
         let test_get_slot = test_commands.clone().get_matches_from(vec!["test", "slot"]);
         assert_eq!(
-            parse_command(&test_get_slot).unwrap(),
+            parse_command(&test_get_slot, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::GetSlot {
                     commitment_config: CommitmentConfig::recent(),
                 },
-                require_keypair: false
+                signers: vec![],
             }
         );
 
@@ -1073,12 +1223,12 @@ mod tests {
             .clone()
             .get_matches_from(vec!["test", "transaction-count"]);
         assert_eq!(
-            parse_command(&test_transaction_count).unwrap(),
+            parse_command(&test_transaction_count, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::GetTransactionCount {
                     commitment_config: CommitmentConfig::recent(),
                 },
-                require_keypair: false
+                signers: vec![],
             }
         );
 
@@ -1094,7 +1244,7 @@ mod tests {
             "--confirmed",
         ]);
         assert_eq!(
-            parse_command(&test_ping).unwrap(),
+            parse_command(&test_ping, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::Ping {
                     lamports: 1,
@@ -1103,7 +1253,7 @@ mod tests {
                     timeout: Duration::from_secs(3),
                     commitment_config: CommitmentConfig::default(),
                 },
-                require_keypair: true
+                signers: vec![default_keypair.into()],
             }
         );
     }

cli/src/display.rs

@@ -1,3 +1,4 @@
+use crate::cli::SettingType;
 use console::style;
 use solana_sdk::transaction::Transaction;
 
@@ -11,17 +12,19 @@ pub fn println_name_value(name: &str, value: &str) {
     println!("{} {}", style(name).bold(), styled_value);
 }
 
-pub fn println_name_value_or(name: &str, value: &str, default_value: &str) {
-    if value == "" {
-        println!(
-            "{} {} {}",
-            style(name).bold(),
-            style(default_value),
-            style("(default)").italic()
-        );
-    } else {
-        println!("{} {}", style(name).bold(), style(value));
+pub fn println_name_value_or(name: &str, value: &str, setting_type: SettingType) {
+    let description = match setting_type {
+        SettingType::Explicit => "",
+        SettingType::Computed => "(computed)",
+        SettingType::SystemDefault => "(default)",
     };
+
+    println!(
+        "{} {} {}",
+        style(name).bold(),
+        style(value),
+        style(description).italic(),
+    );
 }
 
 pub fn println_signers(tx: &Transaction) {

cli/src/lib.rs

@@ -1,11 +1,8 @@
-#[macro_use]
-extern crate lazy_static;
-
 pub mod cli;
 pub mod cluster_query;
-pub mod config;
 pub mod display;
 pub mod nonce;
+pub mod offline;
 pub mod stake;
 pub mod storage;
 pub mod validator_info;

cli/src/main.rs

@@ -2,20 +2,17 @@ use clap::{crate_description, crate_name, AppSettings, Arg, ArgGroup, ArgMatches
 use console::style;
 
 use solana_clap_utils::{
-    input_validators::is_url,
-    keypair::{
-        self, keypair_input, KeypairWithSource, ASK_SEED_PHRASE_ARG,
-        SKIP_SEED_PHRASE_VALIDATION_ARG,
-    },
+    input_parsers::derivation_of,
+    input_validators::{is_derivation, is_url},
+    keypair::SKIP_SEED_PHRASE_VALIDATION_ARG,
 };
 use solana_cli::{
-    cli::{app, parse_command, process_command, CliCommandInfo, CliConfig, CliError},
-    config::{self, Config},
+    cli::{app, parse_command, process_command, CliCommandInfo, CliConfig, CliSigners},
     display::{println_name_value, println_name_value_or},
 };
-use solana_sdk::signature::read_keypair_file;
-
-use std::error;
+use solana_cli_config::config::{Config, CONFIG_FILE};
+use solana_remote_wallet::remote_wallet::{maybe_wallet_manager, RemoteWalletManager};
+use std::{error, sync::Arc};
 
 fn parse_settings(matches: &ArgMatches<'_>) -> Result<bool, Box<dyn error::Error>> {
     let parse_args = match matches.subcommand() {
@@ -23,25 +20,31 @@ fn parse_settings(matches: &ArgMatches<'_>) -> Result<bool, Box<dyn error::Error
             ("get", Some(subcommand_matches)) => {
                 if let Some(config_file) = matches.value_of("config_file") {
                     let config = Config::load(config_file).unwrap_or_default();
+
+                    let (url_setting_type, json_rpc_url) =
+                        CliConfig::compute_json_rpc_url_setting("", &config.url);
+                    let (ws_setting_type, websocket_url) = CliConfig::compute_websocket_url_setting(
+                        "",
+                        &config.websocket_url,
+                        "",
+                        &config.url,
+                    );
+                    let (keypair_setting_type, keypair_path) =
+                        CliConfig::compute_keypair_path_setting("", &config.keypair_path);
+
                     if let Some(field) = subcommand_matches.value_of("specific_setting") {
-                        let (value, default_value) = match field {
-                            "url" => (config.url, CliConfig::default_json_rpc_url()),
-                            "keypair" => (config.keypair_path, CliConfig::default_keypair_path()),
+                        let (field_name, value, setting_type) = match field {
+                            "json_rpc_url" => ("RPC URL", json_rpc_url, url_setting_type),
+                            "websocket_url" => ("WS URL", websocket_url, ws_setting_type),
+                            "keypair" => ("Key Path", keypair_path, keypair_setting_type),
                             _ => unreachable!(),
                         };
-                        println_name_value_or(&format!("* {}:", field), &value, &default_value);
+                        println_name_value_or(&format!("{}:", field_name), &value, setting_type);
                     } else {
-                        println_name_value("Wallet Config:", config_file);
-                        println_name_value_or(
-                            "* url:",
-                            &config.url,
-                            &CliConfig::default_json_rpc_url(),
-                        );
-                        println_name_value_or(
-                            "* keypair:",
-                            &config.keypair_path,
-                            &CliConfig::default_keypair_path(),
-                        );
+                        println_name_value("Config File:", config_file);
+                        println_name_value_or("RPC URL:", &json_rpc_url, url_setting_type);
+                        println_name_value_or("WS URL:", &websocket_url, ws_setting_type);
+                        println_name_value_or("Keypair Path:", &keypair_path, keypair_setting_type);
                     }
                 } else {
                     println!(
@@ -57,13 +60,29 @@ fn parse_settings(matches: &ArgMatches<'_>) -> Result<bool, Box<dyn error::Error
                     if let Some(url) = subcommand_matches.value_of("json_rpc_url") {
                         config.url = url.to_string();
                     }
+                    if let Some(url) = subcommand_matches.value_of("websocket_url") {
+                        config.websocket_url = url.to_string();
+                    }
                     if let Some(keypair) = subcommand_matches.value_of("keypair") {
                         config.keypair_path = keypair.to_string();
                     }
                     config.save(config_file)?;
-                    println_name_value("Wallet Config Updated:", config_file);
-                    println_name_value("* url:", &config.url);
-                    println_name_value("* keypair:", &config.keypair_path);
+
+                    let (url_setting_type, json_rpc_url) =
+                        CliConfig::compute_json_rpc_url_setting("", &config.url);
+                    let (ws_setting_type, websocket_url) = CliConfig::compute_websocket_url_setting(
+                        "",
+                        &config.websocket_url,
+                        "",
+                        &config.url,
+                    );
+                    let (keypair_setting_type, keypair_path) =
+                        CliConfig::compute_keypair_path_setting("", &config.keypair_path);
+
+                    println_name_value("Config File:", config_file);
+                    println_name_value_or("RPC URL:", &json_rpc_url, url_setting_type);
+                    println_name_value_or("WS URL:", &websocket_url, ws_setting_type);
+                    println_name_value_or("Keypair Path:", &keypair_path, keypair_setting_type);
                 } else {
                     println!(
                         "{} Either provide the `--config` arg or ensure home directory exists to use the default config location",
@@ -79,72 +98,46 @@ fn parse_settings(matches: &ArgMatches<'_>) -> Result<bool, Box<dyn error::Error
     Ok(parse_args)
 }
 
-pub fn parse_args(matches: &ArgMatches<'_>) -> Result<CliConfig, Box<dyn error::Error>> {
+pub fn parse_args<'a>(
+    matches: &ArgMatches<'_>,
+    wallet_manager: Option<Arc<RemoteWalletManager>>,
+) -> Result<(CliConfig<'a>, CliSigners), Box<dyn error::Error>> {
     let config = if let Some(config_file) = matches.value_of("config_file") {
         Config::load(config_file).unwrap_or_default()
     } else {
         Config::default()
     };
-    let json_rpc_url = if let Some(url) = matches.value_of("json_rpc_url") {
-        url.to_string()
-    } else if config.url != "" {
-        config.url
-    } else {
-        let default = CliConfig::default();
-        default.json_rpc_url
-    };
+    let (_, json_rpc_url) = CliConfig::compute_json_rpc_url_setting(
+        matches.value_of("json_rpc_url").unwrap_or(""),
+        &config.url,
+    );
+    let (_, websocket_url) = CliConfig::compute_websocket_url_setting(
+        matches.value_of("websocket_url").unwrap_or(""),
+        &config.websocket_url,
+        matches.value_of("json_rpc_url").unwrap_or(""),
+        &config.url,
+    );
+    let (_, default_signer_path) = CliConfig::compute_keypair_path_setting(
+        matches.value_of("keypair").unwrap_or(""),
+        &config.keypair_path,
+    );
 
-    let CliCommandInfo {
-        command,
-        require_keypair,
-    } = parse_command(&matches)?;
+    let CliCommandInfo { command, signers } =
+        parse_command(&matches, &default_signer_path, wallet_manager.as_ref())?;
 
-    let (keypair, keypair_path) = if require_keypair {
-        let KeypairWithSource { keypair, source } = keypair_input(&matches, "keypair")?;
-        match source {
-            keypair::Source::File => (
-                keypair,
-                Some(matches.value_of("keypair").unwrap().to_string()),
-            ),
-            keypair::Source::SeedPhrase => (keypair, None),
-            keypair::Source::Generated => {
-                let keypair_path = if config.keypair_path != "" {
-                    config.keypair_path
-                } else {
-                    let default_keypair_path = CliConfig::default_keypair_path();
-                    if !std::path::Path::new(&default_keypair_path).exists() {
-                        return Err(CliError::KeypairFileNotFound(format!(
-                            "Generate a new keypair at {} with `solana-keygen new`",
-                            default_keypair_path
-                        ))
-                        .into());
-                    }
-                    default_keypair_path
-                };
-
-                let keypair = read_keypair_file(&keypair_path).or_else(|err| {
-                    Err(CliError::BadParameter(format!(
-                        "{}: Unable to open keypair file: {}",
-                        err, keypair_path
-                    )))
-                })?;
-
-                (keypair, Some(keypair_path))
-            }
-        }
-    } else {
-        let default = CliConfig::default();
-        (default.keypair, None)
-    };
-
-    Ok(CliConfig {
-        command,
-        json_rpc_url,
-        keypair,
-        keypair_path,
-        rpc_client: None,
-        verbose: matches.is_present("verbose"),
-    })
+    Ok((
+        CliConfig {
+            command,
+            json_rpc_url,
+            websocket_url,
+            signers: vec![],
+            keypair_path: default_signer_path,
+            derivation_path: derivation_of(matches, "derivation_path"),
+            rpc_client: None,
+            verbose: matches.is_present("verbose"),
+        },
+        signers,
+    ))
 }
 
 fn main() -> Result<(), Box<dyn error::Error>> {
@@ -162,7 +155,7 @@ fn main() -> Result<(), Box<dyn error::Error>> {
             .takes_value(true)
             .global(true)
             .help("Configuration file to use");
-        if let Some(ref config_file) = *config::CONFIG_FILE {
+        if let Some(ref config_file) = *CONFIG_FILE {
             arg.default_value(&config_file)
         } else {
             arg
@@ -178,6 +171,15 @@ fn main() -> Result<(), Box<dyn error::Error>> {
             .validator(is_url)
             .help("JSON RPC URL for the solana cluster"),
     )
+    .arg(
+        Arg::with_name("websocket_url")
+            .long("ws")
+            .value_name("URL")
+            .takes_value(true)
+            .global(true)
+            .validator(is_url)
+            .help("WebSocket URL for the solana cluster"),
+    )
     .arg(
         Arg::with_name("keypair")
             .short("k")
@@ -185,7 +187,16 @@ fn main() -> Result<(), Box<dyn error::Error>> {
             .value_name("PATH")
             .global(true)
             .takes_value(true)
-            .help("/path/to/id.json"),
+            .help("/path/to/id.json or usb://remote/wallet/path"),
+    )
+    .arg(
+        Arg::with_name("derivation_path")
+            .long("derivation-path")
+            .value_name("ACCOUNT or ACCOUNT/CHANGE")
+            .global(true)
+            .takes_value(true)
+            .validator(is_derivation)
+            .help("Derivation path to use: m/44'/501'/ACCOUNT'/CHANGE'; default key is device base pubkey: m/44'/501'/0'")
     )
     .arg(
         Arg::with_name("verbose")
@@ -194,15 +205,6 @@ fn main() -> Result<(), Box<dyn error::Error>> {
             .global(true)
             .help("Show extra information header"),
     )
-    .arg(
-        Arg::with_name(ASK_SEED_PHRASE_ARG.name)
-            .long(ASK_SEED_PHRASE_ARG.long)
-            .value_name("KEYPAIR NAME")
-            .global(true)
-            .takes_value(true)
-            .possible_values(&["keypair"])
-            .help(ASK_SEED_PHRASE_ARG.help),
-    )
     .arg(
         Arg::with_name(SKIP_SEED_PHRASE_VALIDATION_ARG.name)
             .long(SKIP_SEED_PHRASE_VALIDATION_ARG.long)
@@ -222,7 +224,7 @@ fn main() -> Result<(), Box<dyn error::Error>> {
                             .index(1)
                             .value_name("CONFIG_FIELD")
                             .takes_value(true)
-                            .possible_values(&["url", "keypair"])
+                            .possible_values(&["json_rpc_url", "websocket_url", "keypair"])
                             .help("Return a specific config setting"),
                     ),
             )
@@ -231,7 +233,7 @@ fn main() -> Result<(), Box<dyn error::Error>> {
                     .about("Set a config setting")
                     .group(
                         ArgGroup::with_name("config_settings")
-                            .args(&["json_rpc_url", "keypair"])
+                            .args(&["json_rpc_url", "websocket_url", "keypair"])
                             .multiple(true)
                             .required(true),
                     ),
@@ -240,7 +242,10 @@ fn main() -> Result<(), Box<dyn error::Error>> {
     .get_matches();
 
     if parse_settings(&matches)? {
-        let config = parse_args(&matches)?;
+        let wallet_manager = maybe_wallet_manager()?;
+
+        let (mut config, signers) = parse_args(&matches, wallet_manager)?;
+        config.signers = signers.iter().map(|s| s.as_ref()).collect();
         let result = process_command(&config)?;
         println!("{}", result);
     }

cli/src/nonce.rs

@@ -1,18 +1,21 @@
 use crate::cli::{
-    build_balance_message, check_account_for_fee, check_unique_pubkeys,
-    log_instruction_custom_error, required_lamports_from, CliCommand, CliCommandInfo, CliConfig,
-    CliError, ProcessResult, SigningAuthority,
+    build_balance_message, check_account_for_fee, check_unique_pubkeys, generate_unique_signers,
+    log_instruction_custom_error, CliCommand, CliCommandInfo, CliConfig, CliError, ProcessResult,
+    SignerIndex,
 };
 use clap::{App, Arg, ArgMatches, SubCommand};
-use solana_clap_utils::{input_parsers::*, input_validators::*, ArgConstant};
+use solana_clap_utils::{
+    input_parsers::*, input_validators::*, offline::BLOCKHASH_ARG, ArgConstant,
+};
 use solana_client::rpc_client::RpcClient;
+use solana_remote_wallet::remote_wallet::RemoteWalletManager;
 use solana_sdk::{
     account::Account,
     account_utils::StateMut,
     hash::Hash,
+    message::Message,
     nonce_state::{Meta, NonceState},
     pubkey::Pubkey,
-    signature::{Keypair, KeypairUtil},
     system_instruction::{
         advance_nonce_account, authorize_nonce_account, create_address_with_seed,
         create_nonce_account, create_nonce_account_with_seed, withdraw_nonce_account, NonceError,
@@ -21,6 +24,7 @@ use solana_sdk::{
     system_program,
     transaction::Transaction,
 };
+use std::sync::Arc;
 
 #[derive(Debug, Clone, PartialEq)]
 pub enum CliNonceError {
@@ -55,7 +59,7 @@ pub fn nonce_arg<'a, 'b>() -> Arg<'a, 'b> {
         .long(NONCE_ARG.long)
         .takes_value(true)
         .value_name("PUBKEY")
-        .requires("blockhash")
+        .requires(BLOCKHASH_ARG.name)
         .validator(is_pubkey)
         .help(NONCE_ARG.help)
 }
@@ -64,8 +68,8 @@ pub fn nonce_authority_arg<'a, 'b>() -> Arg<'a, 'b> {
     Arg::with_name(NONCE_AUTHORITY_ARG.name)
         .long(NONCE_AUTHORITY_ARG.long)
         .takes_value(true)
-        .value_name("KEYPAIR or PUBKEY")
-        .validator(is_pubkey_or_keypair_or_ask_keyword)
+        .value_name("KEYPAIR or PUBKEY or REMOTE WALLET PATH")
+        .validator(is_valid_signer)
         .help(NONCE_AUTHORITY_ARG.help)
 }
 
@@ -120,15 +124,7 @@ impl NonceSubCommands for App<'_, '_> {
                         .takes_value(true)
                         .required(true)
                         .validator(is_amount)
-                        .help("The amount to load the nonce account with (default unit SOL)"),
-                )
-                .arg(
-                    Arg::with_name("unit")
-                        .index(3)
-                        .value_name("UNIT")
-                        .takes_value(true)
-                        .possible_values(&["SOL", "lamports"])
-                        .help("Specify unit to use for request"),
+                        .help("The amount to load the nonce account with, in SOL"),
                 )
                 .arg(
                     Arg::with_name(NONCE_AUTHORITY_ARG.name)
@@ -215,58 +211,68 @@ impl NonceSubCommands for App<'_, '_> {
                         .takes_value(true)
                         .required(true)
                         .validator(is_amount)
-                        .help("The amount to withdraw from the nonce account (default unit SOL)"),
-                )
-                .arg(
-                    Arg::with_name("unit")
-                        .index(4)
-                        .value_name("UNIT")
-                        .takes_value(true)
-                        .possible_values(&["SOL", "lamports"])
-                        .help("Specify unit to use for request"),
+                        .help("The amount to withdraw from the nonce account, in SOL"),
                 )
                 .arg(nonce_authority_arg()),
         )
     }
 }
 
-pub fn parse_authorize_nonce_account(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, CliError> {
+pub fn parse_authorize_nonce_account(
+    matches: &ArgMatches<'_>,
+    default_signer_path: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
+) -> Result<CliCommandInfo, CliError> {
     let nonce_account = pubkey_of(matches, "nonce_account_keypair").unwrap();
     let new_authority = pubkey_of(matches, "new_authority").unwrap();
-    let nonce_authority = if matches.is_present(NONCE_AUTHORITY_ARG.name) {
-        Some(SigningAuthority::new_from_matches(
-            &matches,
-            NONCE_AUTHORITY_ARG.name,
-            None,
-        )?)
-    } else {
-        None
-    };
+    let (nonce_authority, nonce_authority_pubkey) =
+        signer_of(matches, NONCE_AUTHORITY_ARG.name, wallet_manager)?;
+
+    let payer_provided = None;
+    let signer_info = generate_unique_signers(
+        vec![payer_provided, nonce_authority],
+        matches,
+        default_signer_path,
+        wallet_manager,
+    )?;
 
     Ok(CliCommandInfo {
         command: CliCommand::AuthorizeNonceAccount {
             nonce_account,
-            nonce_authority,
+            nonce_authority: signer_info.index_of(nonce_authority_pubkey).unwrap(),
             new_authority,
         },
-        require_keypair: true,
+        signers: signer_info.signers,
     })
 }
 
-pub fn parse_nonce_create_account(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, CliError> {
-    let nonce_account = keypair_of(matches, "nonce_account_keypair").unwrap();
+pub fn parse_nonce_create_account(
+    matches: &ArgMatches<'_>,
+    default_signer_path: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
+) -> Result<CliCommandInfo, CliError> {
+    let (nonce_account, nonce_account_pubkey) =
+        signer_of(matches, "nonce_account_keypair", wallet_manager)?;
     let seed = matches.value_of("seed").map(|s| s.to_string());
-    let lamports = required_lamports_from(matches, "amount", "unit")?;
+    let lamports = lamports_of_sol(matches, "amount").unwrap();
     let nonce_authority = pubkey_of(matches, NONCE_AUTHORITY_ARG.name);
 
+    let payer_provided = None;
+    let signer_info = generate_unique_signers(
+        vec![payer_provided, nonce_account],
+        matches,
+        default_signer_path,
+        wallet_manager,
+    )?;
+
     Ok(CliCommandInfo {
         command: CliCommand::CreateNonceAccount {
-            nonce_account: nonce_account.into(),
+            nonce_account: signer_info.index_of(nonce_account_pubkey).unwrap(),
             seed,
             nonce_authority,
             lamports,
         },
-        require_keypair: true,
+        signers: signer_info.signers,
     })
 }
 
@@ -275,28 +281,33 @@ pub fn parse_get_nonce(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, CliEr
 
     Ok(CliCommandInfo {
         command: CliCommand::GetNonce(nonce_account_pubkey),
-        require_keypair: false,
+        signers: vec![],
     })
 }
 
-pub fn parse_new_nonce(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, CliError> {
+pub fn parse_new_nonce(
+    matches: &ArgMatches<'_>,
+    default_signer_path: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
+) -> Result<CliCommandInfo, CliError> {
     let nonce_account = pubkey_of(matches, "nonce_account_keypair").unwrap();
-    let nonce_authority = if matches.is_present(NONCE_AUTHORITY_ARG.name) {
-        Some(SigningAuthority::new_from_matches(
-            &matches,
-            NONCE_AUTHORITY_ARG.name,
-            None,
-        )?)
-    } else {
-        None
-    };
+    let (nonce_authority, nonce_authority_pubkey) =
+        signer_of(matches, NONCE_AUTHORITY_ARG.name, wallet_manager)?;
+
+    let payer_provided = None;
+    let signer_info = generate_unique_signers(
+        vec![payer_provided, nonce_authority],
+        matches,
+        default_signer_path,
+        wallet_manager,
+    )?;
 
     Ok(CliCommandInfo {
         command: CliCommand::NewNonce {
             nonce_account,
-            nonce_authority,
+            nonce_authority: signer_info.index_of(nonce_authority_pubkey).unwrap(),
         },
-        require_keypair: true,
+        signers: signer_info.signers,
     })
 }
 
@@ -309,34 +320,37 @@ pub fn parse_show_nonce_account(matches: &ArgMatches<'_>) -> Result<CliCommandIn
             nonce_account_pubkey,
             use_lamports_unit,
         },
-        require_keypair: false,
+        signers: vec![],
     })
 }
 
 pub fn parse_withdraw_from_nonce_account(
     matches: &ArgMatches<'_>,
+    default_signer_path: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
 ) -> Result<CliCommandInfo, CliError> {
     let nonce_account = pubkey_of(matches, "nonce_account_keypair").unwrap();
     let destination_account_pubkey = pubkey_of(matches, "destination_account_pubkey").unwrap();
-    let lamports = required_lamports_from(matches, "amount", "unit")?;
-    let nonce_authority = if matches.is_present(NONCE_AUTHORITY_ARG.name) {
-        Some(SigningAuthority::new_from_matches(
-            &matches,
-            NONCE_AUTHORITY_ARG.name,
-            None,
-        )?)
-    } else {
-        None
-    };
+    let lamports = lamports_of_sol(matches, "amount").unwrap();
+    let (nonce_authority, nonce_authority_pubkey) =
+        signer_of(matches, NONCE_AUTHORITY_ARG.name, wallet_manager)?;
+
+    let payer_provided = None;
+    let signer_info = generate_unique_signers(
+        vec![payer_provided, nonce_authority],
+        matches,
+        default_signer_path,
+        wallet_manager,
+    )?;
 
     Ok(CliCommandInfo {
         command: CliCommand::WithdrawFromNonceAccount {
             nonce_account,
-            nonce_authority,
+            nonce_authority: signer_info.index_of(nonce_authority_pubkey).unwrap(),
             destination_account_pubkey,
             lamports,
         },
-        require_keypair: true,
+        signers: signer_info.signers,
     })
 }
 
@@ -372,41 +386,36 @@ pub fn process_authorize_nonce_account(
     rpc_client: &RpcClient,
     config: &CliConfig,
     nonce_account: &Pubkey,
-    nonce_authority: Option<&SigningAuthority>,
+    nonce_authority: SignerIndex,
     new_authority: &Pubkey,
 ) -> ProcessResult {
     let (recent_blockhash, fee_calculator) = rpc_client.get_recent_blockhash()?;
 
-    let nonce_authority = nonce_authority
-        .map(|a| a.keypair())
-        .unwrap_or(&config.keypair);
+    let nonce_authority = config.signers[nonce_authority];
     let ix = authorize_nonce_account(nonce_account, &nonce_authority.pubkey(), new_authority);
-    let mut tx = Transaction::new_signed_with_payer(
-        vec![ix],
-        Some(&config.keypair.pubkey()),
-        &[&config.keypair, nonce_authority],
-        recent_blockhash,
-    );
+    let message = Message::new_with_payer(vec![ix], Some(&config.signers[0].pubkey()));
+    let mut tx = Transaction::new_unsigned(message);
+    tx.try_sign(&config.signers, recent_blockhash)?;
+
     check_account_for_fee(
         rpc_client,
-        &config.keypair.pubkey(),
+        &config.signers[0].pubkey(),
         &fee_calculator,
         &tx.message,
     )?;
-    let result =
-        rpc_client.send_and_confirm_transaction(&mut tx, &[&config.keypair, nonce_authority]);
+    let result = rpc_client.send_and_confirm_transaction(&mut tx, &config.signers);
     log_instruction_custom_error::<NonceError>(result)
 }
 
 pub fn process_create_nonce_account(
     rpc_client: &RpcClient,
     config: &CliConfig,
-    nonce_account: &Keypair,
+    nonce_account: SignerIndex,
     seed: Option<String>,
     nonce_authority: Option<Pubkey>,
     lamports: u64,
 ) -> ProcessResult {
-    let nonce_account_pubkey = nonce_account.pubkey();
+    let nonce_account_pubkey = config.signers[nonce_account].pubkey();
     let nonce_account_address = if let Some(seed) = seed.clone() {
         create_address_with_seed(&nonce_account_pubkey, &seed, &system_program::id())?
     } else {
@@ -414,7 +423,7 @@ pub fn process_create_nonce_account(
     };
 
     check_unique_pubkeys(
-        (&config.keypair.pubkey(), "cli keypair".to_string()),
+        (&config.signers[0].pubkey(), "cli keypair".to_string()),
         (&nonce_account_address, "nonce_account".to_string()),
     )?;
 
@@ -441,20 +450,20 @@ pub fn process_create_nonce_account(
         .into());
     }
 
-    let nonce_authority = nonce_authority.unwrap_or_else(|| config.keypair.pubkey());
+    let nonce_authority = nonce_authority.unwrap_or_else(|| config.signers[0].pubkey());
 
     let ixs = if let Some(seed) = seed {
         create_nonce_account_with_seed(
-            &config.keypair.pubkey(), // from
-            &nonce_account_address,   // to
-            &nonce_account_pubkey,    // base
-            &seed,                    // seed
+            &config.signers[0].pubkey(), // from
+            &nonce_account_address,      // to
+            &nonce_account_pubkey,       // base
+            &seed,                       // seed
             &nonce_authority,
             lamports,
         )
     } else {
         create_nonce_account(
-            &config.keypair.pubkey(),
+            &config.signers[0].pubkey(),
             &nonce_account_pubkey,
             &nonce_authority,
             lamports,
@@ -463,25 +472,17 @@ pub fn process_create_nonce_account(
 
     let (recent_blockhash, fee_calculator) = rpc_client.get_recent_blockhash()?;
 
-    let signers = if nonce_account_pubkey != config.keypair.pubkey() {
-        vec![&config.keypair, nonce_account] // both must sign if `from` and `to` differ
-    } else {
-        vec![&config.keypair] // when stake_account == config.keypair and there's a seed, we only need one signature
-    };
+    let message = Message::new_with_payer(ixs, Some(&config.signers[0].pubkey()));
+    let mut tx = Transaction::new_unsigned(message);
+    tx.try_sign(&config.signers, recent_blockhash)?;
 
-    let mut tx = Transaction::new_signed_with_payer(
-        ixs,
-        Some(&config.keypair.pubkey()),
-        &signers,
-        recent_blockhash,
-    );
     check_account_for_fee(
         rpc_client,
-        &config.keypair.pubkey(),
+        &config.signers[0].pubkey(),
         &fee_calculator,
         &tx.message,
     )?;
-    let result = rpc_client.send_and_confirm_transaction(&mut tx, &signers);
+    let result = rpc_client.send_and_confirm_transaction(&mut tx, &config.signers);
     log_instruction_custom_error::<SystemError>(result)
 }
 
@@ -509,10 +510,10 @@ pub fn process_new_nonce(
     rpc_client: &RpcClient,
     config: &CliConfig,
     nonce_account: &Pubkey,
-    nonce_authority: Option<&SigningAuthority>,
+    nonce_authority: SignerIndex,
 ) -> ProcessResult {
     check_unique_pubkeys(
-        (&config.keypair.pubkey(), "cli keypair".to_string()),
+        (&config.signers[0].pubkey(), "cli keypair".to_string()),
         (&nonce_account, "nonce_account_pubkey".to_string()),
     )?;
 
@@ -523,25 +524,20 @@ pub fn process_new_nonce(
         .into());
     }
 
-    let nonce_authority = nonce_authority
-        .map(|a| a.keypair())
-        .unwrap_or(&config.keypair);
+    let nonce_authority = config.signers[nonce_authority];
     let ix = advance_nonce_account(&nonce_account, &nonce_authority.pubkey());
     let (recent_blockhash, fee_calculator) = rpc_client.get_recent_blockhash()?;
-    let mut tx = Transaction::new_signed_with_payer(
-        vec![ix],
-        Some(&config.keypair.pubkey()),
-        &[&config.keypair, nonce_authority],
-        recent_blockhash,
-    );
+    let message = Message::new_with_payer(vec![ix], Some(&config.signers[0].pubkey()));
+    let mut tx = Transaction::new_unsigned(message);
+    tx.try_sign(&config.signers, recent_blockhash)?;
     check_account_for_fee(
         rpc_client,
-        &config.keypair.pubkey(),
+        &config.signers[0].pubkey(),
         &fee_calculator,
         &tx.message,
     )?;
     let result =
-        rpc_client.send_and_confirm_transaction(&mut tx, &[&config.keypair, nonce_authority]);
+        rpc_client.send_and_confirm_transaction(&mut tx, &[config.signers[0], nonce_authority]);
     log_instruction_custom_error::<SystemError>(result)
 }
 
@@ -560,11 +556,11 @@ pub fn process_show_nonce_account(
     }
     let print_account = |data: Option<(Meta, Hash)>| {
         println!(
-            "balance: {}",
+            "Balance: {}",
             build_balance_message(nonce_account.lamports, use_lamports_unit, true)
         );
         println!(
-            "minimum balance required: {}",
+            "Minimum Balance Required: {}",
             build_balance_message(
                 rpc_client.get_minimum_balance_for_rent_exemption(NonceState::size())?,
                 use_lamports_unit,
@@ -573,12 +569,12 @@ pub fn process_show_nonce_account(
         );
         match data {
             Some((meta, hash)) => {
-                println!("nonce: {}", hash);
-                println!("authority: {}", meta.nonce_authority);
+                println!("Nonce: {}", hash);
+                println!("Authority: {}", meta.nonce_authority);
             }
             None => {
-                println!("nonce: uninitialized");
-                println!("authority: uninitialized");
+                println!("Nonce: uninitialized");
+                println!("Authority: uninitialized");
             }
         }
         Ok("".to_string())
@@ -598,35 +594,29 @@ pub fn process_withdraw_from_nonce_account(
     rpc_client: &RpcClient,
     config: &CliConfig,
     nonce_account: &Pubkey,
-    nonce_authority: Option<&SigningAuthority>,
+    nonce_authority: SignerIndex,
     destination_account_pubkey: &Pubkey,
     lamports: u64,
 ) -> ProcessResult {
     let (recent_blockhash, fee_calculator) = rpc_client.get_recent_blockhash()?;
 
-    let nonce_authority = nonce_authority
-        .map(|a| a.keypair())
-        .unwrap_or(&config.keypair);
+    let nonce_authority = config.signers[nonce_authority];
     let ix = withdraw_nonce_account(
         nonce_account,
         &nonce_authority.pubkey(),
         destination_account_pubkey,
         lamports,
     );
-    let mut tx = Transaction::new_signed_with_payer(
-        vec![ix],
-        Some(&config.keypair.pubkey()),
-        &[&config.keypair, nonce_authority],
-        recent_blockhash,
-    );
+    let message = Message::new_with_payer(vec![ix], Some(&config.signers[0].pubkey()));
+    let mut tx = Transaction::new_unsigned(message);
+    tx.try_sign(&config.signers, recent_blockhash)?;
     check_account_for_fee(
         rpc_client,
-        &config.keypair.pubkey(),
+        &config.signers[0].pubkey(),
         &fee_calculator,
         &tx.message,
     )?;
-    let result =
-        rpc_client.send_and_confirm_transaction(&mut tx, &[&config.keypair, nonce_authority]);
+    let result = rpc_client.send_and_confirm_transaction(&mut tx, &config.signers);
     log_instruction_custom_error::<NonceError>(result)
 }
 
@@ -638,7 +628,7 @@ mod tests {
         account::Account,
         hash::hash,
         nonce_state::{Meta as NonceMeta, NonceState},
-        signature::{read_keypair_file, write_keypair},
+        signature::{read_keypair_file, write_keypair, Keypair, Signer},
         system_program,
     };
     use tempfile::NamedTempFile;
@@ -651,6 +641,9 @@ mod tests {
     #[test]
     fn test_parse_command() {
         let test_commands = app("test", "desc", "version");
+        let default_keypair = Keypair::new();
+        let (default_keypair_file, mut tmp_file) = make_tmp_file();
+        write_keypair(&default_keypair, tmp_file.as_file_mut()).unwrap();
         let (keypair_file, mut tmp_file) = make_tmp_file();
         let nonce_account_keypair = Keypair::new();
         write_keypair(&nonce_account_keypair, tmp_file.as_file_mut()).unwrap();
@@ -669,14 +662,14 @@ mod tests {
             &Pubkey::default().to_string(),
         ]);
         assert_eq!(
-            parse_command(&test_authorize_nonce_account).unwrap(),
+            parse_command(&test_authorize_nonce_account, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::AuthorizeNonceAccount {
                     nonce_account: nonce_account_pubkey,
-                    nonce_authority: None,
+                    nonce_authority: 0,
                     new_authority: Pubkey::default(),
                 },
-                require_keypair: true,
+                signers: vec![read_keypair_file(&default_keypair_file).unwrap().into()],
             }
         );
 
@@ -690,16 +683,17 @@ mod tests {
             &authority_keypair_file,
         ]);
         assert_eq!(
-            parse_command(&test_authorize_nonce_account).unwrap(),
+            parse_command(&test_authorize_nonce_account, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::AuthorizeNonceAccount {
                     nonce_account: read_keypair_file(&keypair_file).unwrap().pubkey(),
-                    nonce_authority: Some(
-                        read_keypair_file(&authority_keypair_file).unwrap().into()
-                    ),
+                    nonce_authority: 1,
                     new_authority: Pubkey::default(),
                 },
-                require_keypair: true,
+                signers: vec![
+                    read_keypair_file(&default_keypair_file).unwrap().into(),
+                    read_keypair_file(&authority_keypair_file).unwrap().into()
+                ],
             }
         );
 
@@ -709,18 +703,20 @@ mod tests {
             "create-nonce-account",
             &keypair_file,
             "50",
-            "lamports",
         ]);
         assert_eq!(
-            parse_command(&test_create_nonce_account).unwrap(),
+            parse_command(&test_create_nonce_account, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::CreateNonceAccount {
-                    nonce_account: read_keypair_file(&keypair_file).unwrap().into(),
+                    nonce_account: 1,
                     seed: None,
                     nonce_authority: None,
-                    lamports: 50,
+                    lamports: 50_000_000_000,
                 },
-                require_keypair: true
+                signers: vec![
+                    read_keypair_file(&default_keypair_file).unwrap().into(),
+                    read_keypair_file(&keypair_file).unwrap().into()
+                ],
             }
         );
 
@@ -730,22 +726,22 @@ mod tests {
             "create-nonce-account",
             &keypair_file,
             "50",
-            "lamports",
             "--nonce-authority",
             &authority_keypair_file,
         ]);
         assert_eq!(
-            parse_command(&test_create_nonce_account).unwrap(),
+            parse_command(&test_create_nonce_account, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::CreateNonceAccount {
-                    nonce_account: read_keypair_file(&keypair_file).unwrap().into(),
+                    nonce_account: 1,
                     seed: None,
-                    nonce_authority: Some(
-                        read_keypair_file(&authority_keypair_file).unwrap().pubkey()
-                    ),
-                    lamports: 50,
+                    nonce_authority: Some(nonce_authority_keypair.pubkey()),
+                    lamports: 50_000_000_000,
                 },
-                require_keypair: true
+                signers: vec![
+                    read_keypair_file(&default_keypair_file).unwrap().into(),
+                    read_keypair_file(&keypair_file).unwrap().into()
+                ],
             }
         );
 
@@ -756,10 +752,10 @@ mod tests {
             &nonce_account_string,
         ]);
         assert_eq!(
-            parse_command(&test_get_nonce).unwrap(),
+            parse_command(&test_get_nonce, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
-                command: CliCommand::GetNonce(nonce_account_keypair.pubkey(),),
-                require_keypair: false
+                command: CliCommand::GetNonce(nonce_account_keypair.pubkey()),
+                signers: vec![],
             }
         );
 
@@ -770,13 +766,13 @@ mod tests {
                 .get_matches_from(vec!["test", "new-nonce", &keypair_file]);
         let nonce_account = read_keypair_file(&keypair_file).unwrap();
         assert_eq!(
-            parse_command(&test_new_nonce).unwrap(),
+            parse_command(&test_new_nonce, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::NewNonce {
                     nonce_account: nonce_account.pubkey(),
-                    nonce_authority: None,
+                    nonce_authority: 0,
                 },
-                require_keypair: true
+                signers: vec![read_keypair_file(&default_keypair_file).unwrap().into()],
             }
         );
 
@@ -790,15 +786,16 @@ mod tests {
         ]);
         let nonce_account = read_keypair_file(&keypair_file).unwrap();
         assert_eq!(
-            parse_command(&test_new_nonce).unwrap(),
+            parse_command(&test_new_nonce, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::NewNonce {
                     nonce_account: nonce_account.pubkey(),
-                    nonce_authority: Some(
-                        read_keypair_file(&authority_keypair_file).unwrap().into()
-                    ),
+                    nonce_authority: 1,
                 },
-                require_keypair: true
+                signers: vec![
+                    read_keypair_file(&default_keypair_file).unwrap().into(),
+                    read_keypair_file(&authority_keypair_file).unwrap().into()
+                ],
             }
         );
 
@@ -809,13 +806,13 @@ mod tests {
             &nonce_account_string,
         ]);
         assert_eq!(
-            parse_command(&test_show_nonce_account).unwrap(),
+            parse_command(&test_show_nonce_account, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::ShowNonceAccount {
                     nonce_account_pubkey: nonce_account_keypair.pubkey(),
                     use_lamports_unit: false,
                 },
-                require_keypair: false
+                signers: vec![],
             }
         );
 
@@ -826,18 +823,22 @@ mod tests {
             &keypair_file,
             &nonce_account_string,
             "42",
-            "lamports",
         ]);
         assert_eq!(
-            parse_command(&test_withdraw_from_nonce_account).unwrap(),
+            parse_command(
+                &test_withdraw_from_nonce_account,
+                &default_keypair_file,
+                None
+            )
+            .unwrap(),
             CliCommandInfo {
                 command: CliCommand::WithdrawFromNonceAccount {
                     nonce_account: read_keypair_file(&keypair_file).unwrap().pubkey(),
-                    nonce_authority: None,
+                    nonce_authority: 0,
                     destination_account_pubkey: nonce_account_pubkey,
-                    lamports: 42
+                    lamports: 42_000_000_000
                 },
-                require_keypair: true
+                signers: vec![read_keypair_file(&default_keypair_file).unwrap().into()],
             }
         );
 
@@ -847,18 +848,22 @@ mod tests {
             &keypair_file,
             &nonce_account_string,
             "42",
-            "SOL",
         ]);
         assert_eq!(
-            parse_command(&test_withdraw_from_nonce_account).unwrap(),
+            parse_command(
+                &test_withdraw_from_nonce_account,
+                &default_keypair_file,
+                None
+            )
+            .unwrap(),
             CliCommandInfo {
                 command: CliCommand::WithdrawFromNonceAccount {
                     nonce_account: read_keypair_file(&keypair_file).unwrap().pubkey(),
-                    nonce_authority: None,
+                    nonce_authority: 0,
                     destination_account_pubkey: nonce_account_pubkey,
                     lamports: 42000000000
                 },
-                require_keypair: true
+                signers: vec![read_keypair_file(&default_keypair_file).unwrap().into()],
             }
         );
 
@@ -869,22 +874,27 @@ mod tests {
             &keypair_file,
             &nonce_account_string,
             "42",
-            "lamports",
             "--nonce-authority",
             &authority_keypair_file,
         ]);
         assert_eq!(
-            parse_command(&test_withdraw_from_nonce_account).unwrap(),
+            parse_command(
+                &test_withdraw_from_nonce_account,
+                &default_keypair_file,
+                None
+            )
+            .unwrap(),
             CliCommandInfo {
                 command: CliCommand::WithdrawFromNonceAccount {
                     nonce_account: read_keypair_file(&keypair_file).unwrap().pubkey(),
-                    nonce_authority: Some(
-                        read_keypair_file(&authority_keypair_file).unwrap().into()
-                    ),
+                    nonce_authority: 1,
                     destination_account_pubkey: nonce_account_pubkey,
-                    lamports: 42
+                    lamports: 42_000_000_000
                 },
-                require_keypair: true
+                signers: vec![
+                    read_keypair_file(&default_keypair_file).unwrap().into(),
+                    read_keypair_file(&authority_keypair_file).unwrap().into()
+                ],
             }
         );
     }

cli/src/offline.rs

@@ -0,0 +1,253 @@
+use clap::{App, Arg, ArgMatches};
+use serde_json::Value;
+use solana_clap_utils::{
+    input_parsers::value_of,
+    input_validators::{is_hash, is_pubkey_sig},
+    offline::{BLOCKHASH_ARG, SIGNER_ARG, SIGN_ONLY_ARG},
+};
+use solana_client::rpc_client::RpcClient;
+use solana_sdk::{fee_calculator::FeeCalculator, hash::Hash, pubkey::Pubkey, signature::Signature};
+use std::str::FromStr;
+
+#[derive(Clone, Debug, PartialEq)]
+pub enum BlockhashQuery {
+    None(Hash, FeeCalculator),
+    FeeCalculator(Hash),
+    All,
+}
+
+impl BlockhashQuery {
+    pub fn new(blockhash: Option<Hash>, sign_only: bool) -> Self {
+        match blockhash {
+            Some(hash) if sign_only => Self::None(hash, FeeCalculator::default()),
+            Some(hash) if !sign_only => Self::FeeCalculator(hash),
+            None if !sign_only => Self::All,
+            _ => panic!("Cannot resolve blockhash"),
+        }
+    }
+
+    pub fn new_from_matches(matches: &ArgMatches<'_>) -> Self {
+        let blockhash = value_of(matches, BLOCKHASH_ARG.name);
+        let sign_only = matches.is_present(SIGN_ONLY_ARG.name);
+        BlockhashQuery::new(blockhash, sign_only)
+    }
+
+    pub fn get_blockhash_fee_calculator(
+        &self,
+        rpc_client: &RpcClient,
+    ) -> Result<(Hash, FeeCalculator), Box<dyn std::error::Error>> {
+        let (hash, fee_calc) = match self {
+            BlockhashQuery::None(hash, fee_calc) => (Some(hash), Some(fee_calc)),
+            BlockhashQuery::FeeCalculator(hash) => (Some(hash), None),
+            BlockhashQuery::All => (None, None),
+        };
+        if None == fee_calc {
+            let (cluster_hash, fee_calc) = rpc_client.get_recent_blockhash()?;
+            Ok((*hash.unwrap_or(&cluster_hash), fee_calc))
+        } else {
+            Ok((*hash.unwrap(), fee_calc.unwrap().clone()))
+        }
+    }
+}
+
+impl Default for BlockhashQuery {
+    fn default() -> Self {
+        BlockhashQuery::All
+    }
+}
+
+fn blockhash_arg<'a, 'b>() -> Arg<'a, 'b> {
+    Arg::with_name(BLOCKHASH_ARG.name)
+        .long(BLOCKHASH_ARG.long)
+        .takes_value(true)
+        .value_name("BLOCKHASH")
+        .validator(is_hash)
+        .help(BLOCKHASH_ARG.help)
+}
+
+fn sign_only_arg<'a, 'b>() -> Arg<'a, 'b> {
+    Arg::with_name(SIGN_ONLY_ARG.name)
+        .long(SIGN_ONLY_ARG.long)
+        .takes_value(false)
+        .requires(BLOCKHASH_ARG.name)
+        .help(SIGN_ONLY_ARG.help)
+}
+
+fn signer_arg<'a, 'b>() -> Arg<'a, 'b> {
+    Arg::with_name(SIGNER_ARG.name)
+        .long(SIGNER_ARG.long)
+        .takes_value(true)
+        .value_name("BASE58_PUBKEY=BASE58_SIG")
+        .validator(is_pubkey_sig)
+        .requires(BLOCKHASH_ARG.name)
+        .multiple(true)
+        .help(SIGNER_ARG.help)
+}
+
+pub trait OfflineArgs {
+    fn offline_args(self) -> Self;
+}
+
+impl OfflineArgs for App<'_, '_> {
+    fn offline_args(self) -> Self {
+        self.arg(blockhash_arg())
+            .arg(sign_only_arg())
+            .arg(signer_arg())
+    }
+}
+
+pub fn parse_sign_only_reply_string(reply: &str) -> (Hash, Vec<(Pubkey, Signature)>) {
+    let object: Value = serde_json::from_str(&reply).unwrap();
+    let blockhash_str = object.get("blockhash").unwrap().as_str().unwrap();
+    let blockhash = blockhash_str.parse::<Hash>().unwrap();
+    let signer_strings = object.get("signers").unwrap().as_array().unwrap();
+    let signers = signer_strings
+        .iter()
+        .map(|signer_string| {
+            let mut signer = signer_string.as_str().unwrap().split('=');
+            let key = Pubkey::from_str(signer.next().unwrap()).unwrap();
+            let sig = Signature::from_str(signer.next().unwrap()).unwrap();
+            (key, sig)
+        })
+        .collect();
+    (blockhash, signers)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use clap::App;
+    use serde_json::{self, json, Value};
+    use solana_client::{
+        rpc_request::RpcRequest,
+        rpc_response::{Response, RpcResponseContext},
+    };
+    use solana_sdk::{fee_calculator::FeeCalculator, hash::hash};
+    use std::collections::HashMap;
+
+    #[test]
+    fn test_blockhashspec_new_ok() {
+        let blockhash = hash(&[1u8]);
+
+        assert_eq!(
+            BlockhashQuery::new(Some(blockhash), true),
+            BlockhashQuery::None(blockhash, FeeCalculator::default()),
+        );
+        assert_eq!(
+            BlockhashQuery::new(Some(blockhash), false),
+            BlockhashQuery::FeeCalculator(blockhash),
+        );
+        assert_eq!(BlockhashQuery::new(None, false), BlockhashQuery::All,);
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_blockhashspec_new_fail() {
+        BlockhashQuery::new(None, true);
+    }
+
+    #[test]
+    fn test_blockhashspec_new_from_matches_ok() {
+        let test_commands = App::new("blockhashspec_test").offline_args();
+        let blockhash = hash(&[1u8]);
+        let blockhash_string = blockhash.to_string();
+
+        let matches = test_commands.clone().get_matches_from(vec![
+            "blockhashspec_test",
+            "--blockhash",
+            &blockhash_string,
+            "--sign-only",
+        ]);
+        assert_eq!(
+            BlockhashQuery::new_from_matches(&matches),
+            BlockhashQuery::None(blockhash, FeeCalculator::default()),
+        );
+
+        let matches = test_commands.clone().get_matches_from(vec![
+            "blockhashspec_test",
+            "--blockhash",
+            &blockhash_string,
+        ]);
+        assert_eq!(
+            BlockhashQuery::new_from_matches(&matches),
+            BlockhashQuery::FeeCalculator(blockhash),
+        );
+
+        let matches = test_commands
+            .clone()
+            .get_matches_from(vec!["blockhashspec_test"]);
+        assert_eq!(
+            BlockhashQuery::new_from_matches(&matches),
+            BlockhashQuery::All,
+        );
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_blockhashspec_new_from_matches_fail() {
+        let test_commands = App::new("blockhashspec_test")
+            .arg(blockhash_arg())
+            // We can really only hit this case unless the arg requirements
+            // are broken, so unset the requires() to recreate that condition
+            .arg(sign_only_arg().requires(""));
+
+        let matches = test_commands
+            .clone()
+            .get_matches_from(vec!["blockhashspec_test", "--sign-only"]);
+        BlockhashQuery::new_from_matches(&matches);
+    }
+
+    #[test]
+    fn test_blockhashspec_get_blockhash_fee_calc() {
+        let test_blockhash = hash(&[0u8]);
+        let rpc_blockhash = hash(&[1u8]);
+        let rpc_fee_calc = FeeCalculator::new(42, 42);
+        let get_recent_blockhash_response = json!(Response {
+            context: RpcResponseContext { slot: 1 },
+            value: json!((
+                Value::String(rpc_blockhash.to_string()),
+                serde_json::to_value(rpc_fee_calc.clone()).unwrap()
+            )),
+        });
+        let mut mocks = HashMap::new();
+        mocks.insert(
+            RpcRequest::GetRecentBlockhash,
+            get_recent_blockhash_response.clone(),
+        );
+        let rpc_client = RpcClient::new_mock_with_mocks("".to_string(), mocks);
+        assert_eq!(
+            BlockhashQuery::All
+                .get_blockhash_fee_calculator(&rpc_client)
+                .unwrap(),
+            (rpc_blockhash, rpc_fee_calc.clone()),
+        );
+        let mut mocks = HashMap::new();
+        mocks.insert(
+            RpcRequest::GetRecentBlockhash,
+            get_recent_blockhash_response.clone(),
+        );
+        let rpc_client = RpcClient::new_mock_with_mocks("".to_string(), mocks);
+        assert_eq!(
+            BlockhashQuery::FeeCalculator(test_blockhash)
+                .get_blockhash_fee_calculator(&rpc_client)
+                .unwrap(),
+            (test_blockhash, rpc_fee_calc.clone()),
+        );
+        let mut mocks = HashMap::new();
+        mocks.insert(
+            RpcRequest::GetRecentBlockhash,
+            get_recent_blockhash_response.clone(),
+        );
+        let rpc_client = RpcClient::new_mock_with_mocks("".to_string(), mocks);
+        assert_eq!(
+            BlockhashQuery::None(test_blockhash, FeeCalculator::default())
+                .get_blockhash_fee_calculator(&rpc_client)
+                .unwrap(),
+            (test_blockhash, FeeCalculator::default()),
+        );
+        let rpc_client = RpcClient::new_mock("fails".to_string());
+        assert!(BlockhashQuery::All
+            .get_blockhash_fee_calculator(&rpc_client)
+            .is_err());
+    }
+}

cli/src/storage.rs

@@ -1,16 +1,17 @@
 use crate::cli::{
     check_account_for_fee, check_unique_pubkeys, log_instruction_custom_error, CliCommand,
-    CliCommandInfo, CliConfig, CliError, ProcessResult,
+    CliCommandInfo, CliConfig, CliError, ProcessResult, SignerIndex,
 };
 use clap::{App, Arg, ArgMatches, SubCommand};
-use solana_clap_utils::{input_parsers::*, input_validators::*};
+use solana_clap_utils::{input_parsers::*, input_validators::*, keypair::signer_from_path};
 use solana_client::rpc_client::RpcClient;
-use solana_sdk::signature::Keypair;
+use solana_remote_wallet::remote_wallet::RemoteWalletManager;
 use solana_sdk::{
-    account_utils::StateMut, message::Message, pubkey::Pubkey, signature::KeypairUtil,
-    system_instruction::SystemError, transaction::Transaction,
+    account_utils::StateMut, message::Message, pubkey::Pubkey, system_instruction::SystemError,
+    transaction::Transaction,
 };
 use solana_storage_program::storage_instruction::{self, StorageAccountType};
+use std::sync::Arc;
 
 pub trait StorageSubCommands {
     fn storage_subcommands(self) -> Self;
@@ -99,35 +100,49 @@ impl StorageSubCommands for App<'_, '_> {
 
 pub fn parse_storage_create_archiver_account(
     matches: &ArgMatches<'_>,
+    default_signer_path: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
 ) -> Result<CliCommandInfo, CliError> {
     let account_owner = pubkey_of(matches, "storage_account_owner").unwrap();
     let storage_account = keypair_of(matches, "storage_account").unwrap();
     Ok(CliCommandInfo {
         command: CliCommand::CreateStorageAccount {
             account_owner,
-            storage_account: storage_account.into(),
+            storage_account: 1,
             account_type: StorageAccountType::Archiver,
         },
-        require_keypair: true,
+        signers: vec![
+            signer_from_path(matches, default_signer_path, "keypair", wallet_manager)?,
+            storage_account.into(),
+        ],
     })
 }
 
 pub fn parse_storage_create_validator_account(
     matches: &ArgMatches<'_>,
+    default_signer_path: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
 ) -> Result<CliCommandInfo, CliError> {
     let account_owner = pubkey_of(matches, "storage_account_owner").unwrap();
     let storage_account = keypair_of(matches, "storage_account").unwrap();
     Ok(CliCommandInfo {
         command: CliCommand::CreateStorageAccount {
             account_owner,
-            storage_account: storage_account.into(),
+            storage_account: 1,
             account_type: StorageAccountType::Validator,
         },
-        require_keypair: true,
+        signers: vec![
+            signer_from_path(matches, default_signer_path, "keypair", wallet_manager)?,
+            storage_account.into(),
+        ],
     })
 }
 
-pub fn parse_storage_claim_reward(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, CliError> {
+pub fn parse_storage_claim_reward(
+    matches: &ArgMatches<'_>,
+    default_signer_path: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
+) -> Result<CliCommandInfo, CliError> {
     let node_account_pubkey = pubkey_of(matches, "node_account_pubkey").unwrap();
     let storage_account_pubkey = pubkey_of(matches, "storage_account_pubkey").unwrap();
     Ok(CliCommandInfo {
@@ -135,7 +150,12 @@ pub fn parse_storage_claim_reward(matches: &ArgMatches<'_>) -> Result<CliCommand
             node_account_pubkey,
             storage_account_pubkey,
         },
-        require_keypair: true,
+        signers: vec![signer_from_path(
+            matches,
+            default_signer_path,
+            "keypair",
+            wallet_manager,
+        )?],
     })
 }
 
@@ -145,20 +165,21 @@ pub fn parse_storage_get_account_command(
     let storage_account_pubkey = pubkey_of(matches, "storage_account_pubkey").unwrap();
     Ok(CliCommandInfo {
         command: CliCommand::ShowStorageAccount(storage_account_pubkey),
-        require_keypair: false,
+        signers: vec![],
     })
 }
 
 pub fn process_create_storage_account(
     rpc_client: &RpcClient,
     config: &CliConfig,
+    storage_account: SignerIndex,
     account_owner: &Pubkey,
-    storage_account: &Keypair,
     account_type: StorageAccountType,
 ) -> ProcessResult {
+    let storage_account = config.signers[storage_account];
     let storage_account_pubkey = storage_account.pubkey();
     check_unique_pubkeys(
-        (&config.keypair.pubkey(), "cli keypair".to_string()),
+        (&config.signers[0].pubkey(), "cli keypair".to_string()),
         (
             &storage_account_pubkey,
             "storage_account_pubkey".to_string(),
@@ -183,7 +204,7 @@ pub fn process_create_storage_account(
         .max(1);
 
     let ixs = storage_instruction::create_storage_account(
-        &config.keypair.pubkey(),
+        &config.signers[0].pubkey(),
         &account_owner,
         &storage_account_pubkey,
         required_balance,
@@ -191,19 +212,16 @@ pub fn process_create_storage_account(
     );
     let (recent_blockhash, fee_calculator) = rpc_client.get_recent_blockhash()?;
 
-    let mut tx = Transaction::new_signed_instructions(
-        &[&config.keypair, &storage_account],
-        ixs,
-        recent_blockhash,
-    );
+    let message = Message::new(ixs);
+    let mut tx = Transaction::new_unsigned(message);
+    tx.try_sign(&config.signers, recent_blockhash)?;
     check_account_for_fee(
         rpc_client,
-        &config.keypair.pubkey(),
+        &config.signers[0].pubkey(),
         &fee_calculator,
         &tx.message,
     )?;
-    let result =
-        rpc_client.send_and_confirm_transaction(&mut tx, &[&config.keypair, &storage_account]);
+    let result = rpc_client.send_and_confirm_transaction(&mut tx, &config.signers);
     log_instruction_custom_error::<SystemError>(result)
 }
 
@@ -217,13 +235,13 @@ pub fn process_claim_storage_reward(
 
     let instruction =
         storage_instruction::claim_reward(node_account_pubkey, storage_account_pubkey);
-    let signers = [&config.keypair];
+    let signers = [config.signers[0]];
     let message = Message::new_with_payer(vec![instruction], Some(&signers[0].pubkey()));
-
-    let mut tx = Transaction::new(&signers, message, recent_blockhash);
+    let mut tx = Transaction::new_unsigned(message);
+    tx.try_sign(&signers, recent_blockhash)?;
     check_account_for_fee(
         rpc_client,
-        &config.keypair.pubkey(),
+        &config.signers[0].pubkey(),
         &fee_calculator,
         &tx.message,
     )?;
@@ -251,7 +269,7 @@ pub fn process_show_storage_account(
         CliError::RpcRequestError(format!("Unable to deserialize storage account: {:?}", err))
     })?;
     println!("{:#?}", storage_contract);
-    println!("account lamports: {}", account.lamports);
+    println!("Account Lamports: {}", account.lamports);
     Ok("".to_string())
 }
 
@@ -259,7 +277,7 @@ pub fn process_show_storage_account(
 mod tests {
     use super::*;
     use crate::cli::{app, parse_command};
-    use solana_sdk::signature::write_keypair;
+    use solana_sdk::signature::{read_keypair_file, write_keypair, Keypair, Signer};
     use tempfile::NamedTempFile;
 
     fn make_tmp_file() -> (String, NamedTempFile) {
@@ -273,6 +291,10 @@ mod tests {
         let pubkey = Pubkey::new_rand();
         let pubkey_string = pubkey.to_string();
 
+        let default_keypair = Keypair::new();
+        let (default_keypair_file, mut tmp_file) = make_tmp_file();
+        write_keypair(&default_keypair, tmp_file.as_file_mut()).unwrap();
+
         let (keypair_file, mut tmp_file) = make_tmp_file();
         let storage_account_keypair = Keypair::new();
         write_keypair(&storage_account_keypair, tmp_file.as_file_mut()).unwrap();
@@ -284,14 +306,22 @@ mod tests {
             &keypair_file,
         ]);
         assert_eq!(
-            parse_command(&test_create_archiver_storage_account).unwrap(),
+            parse_command(
+                &test_create_archiver_storage_account,
+                &default_keypair_file,
+                None
+            )
+            .unwrap(),
             CliCommandInfo {
                 command: CliCommand::CreateStorageAccount {
                     account_owner: pubkey,
-                    storage_account: storage_account_keypair.into(),
+                    storage_account: 1,
                     account_type: StorageAccountType::Archiver,
                 },
-                require_keypair: true
+                signers: vec![
+                    read_keypair_file(&default_keypair_file).unwrap().into(),
+                    storage_account_keypair.into()
+                ],
             }
         );
 
@@ -308,14 +338,22 @@ mod tests {
             &keypair_file,
         ]);
         assert_eq!(
-            parse_command(&test_create_validator_storage_account).unwrap(),
+            parse_command(
+                &test_create_validator_storage_account,
+                &default_keypair_file,
+                None
+            )
+            .unwrap(),
             CliCommandInfo {
                 command: CliCommand::CreateStorageAccount {
                     account_owner: pubkey,
-                    storage_account: storage_account_keypair.into(),
+                    storage_account: 1,
                     account_type: StorageAccountType::Validator,
                 },
-                require_keypair: true
+                signers: vec![
+                    read_keypair_file(&default_keypair_file).unwrap().into(),
+                    storage_account_keypair.into()
+                ],
             }
         );
 
@@ -326,13 +364,13 @@ mod tests {
             &storage_account_string,
         ]);
         assert_eq!(
-            parse_command(&test_claim_storage_reward).unwrap(),
+            parse_command(&test_claim_storage_reward, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::ClaimStorageReward {
                     node_account_pubkey: pubkey,
                     storage_account_pubkey,
                 },
-                require_keypair: true
+                signers: vec![read_keypair_file(&default_keypair_file).unwrap().into()],
             }
         );
     }

cli/src/validator_info.rs

@@ -11,19 +11,21 @@ use serde_json::{Map, Value};
 use solana_clap_utils::{
     input_parsers::pubkey_of,
     input_validators::{is_pubkey, is_url},
+    keypair::signer_from_path,
 };
 use solana_client::rpc_client::RpcClient;
 use solana_config_program::{config_instruction, get_config_data, ConfigKeys, ConfigState};
+use solana_remote_wallet::remote_wallet::RemoteWalletManager;
 use solana_sdk::{
     account::Account,
     commitment_config::CommitmentConfig,
     message::Message,
     pubkey::Pubkey,
-    signature::{Keypair, KeypairUtil},
+    signature::{Keypair, Signer},
     transaction::Transaction,
 };
-
-use std::error;
+use std::{error, sync::Arc};
+use titlecase::titlecase;
 
 pub const MAX_SHORT_FIELD_LENGTH: usize = 70;
 pub const MAX_LONG_FIELD_LENGTH: usize = 300;
@@ -224,17 +226,26 @@ impl ValidatorInfoSubCommands for App<'_, '_> {
     }
 }
 
-pub fn parse_validator_info_command(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, CliError> {
+pub fn parse_validator_info_command(
+    matches: &ArgMatches<'_>,
+    default_signer_path: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
+) -> Result<CliCommandInfo, CliError> {
     let info_pubkey = pubkey_of(matches, "info_pubkey");
     // Prepare validator info
-    let validator_info = parse_args(&matches);
+    let validator_info = parse_args(matches);
     Ok(CliCommandInfo {
         command: CliCommand::SetValidatorInfo {
             validator_info,
             force_keybase: matches.is_present("force"),
             info_pubkey,
         },
-        require_keypair: true,
+        signers: vec![signer_from_path(
+            matches,
+            default_signer_path,
+            "keypair",
+            wallet_manager,
+        )?],
     })
 }
 
@@ -244,7 +255,7 @@ pub fn parse_get_validator_info_command(
     let info_pubkey = pubkey_of(matches, "info_pubkey");
     Ok(CliCommandInfo {
         command: CliCommand::GetValidatorInfo(info_pubkey),
-        require_keypair: false,
+        signers: vec![],
     })
 }
 
@@ -257,7 +268,7 @@ pub fn process_set_validator_info(
 ) -> ProcessResult {
     // Validate keybase username
     if let Some(string) = validator_info.get("keybaseUsername") {
-        let result = verify_keybase(&config.keypair.pubkey(), &string);
+        let result = verify_keybase(&config.signers[0].pubkey(), &string);
         if result.is_err() {
             if force_keybase {
                 println!("--force supplied, ignoring: {:?}", result);
@@ -282,7 +293,7 @@ pub fn process_set_validator_info(
         })
         .find(|(pubkey, account)| {
             let (validator_pubkey, _) = parse_validator_info(&pubkey, &account).unwrap();
-            validator_pubkey == config.keypair.pubkey()
+            validator_pubkey == config.signers[0].pubkey()
         });
 
     // Create validator-info keypair to use if info_pubkey not provided or does not exist
@@ -300,8 +311,8 @@ pub fn process_set_validator_info(
         .poll_get_balance_with_commitment(&info_pubkey, CommitmentConfig::default())
         .unwrap_or(0);
 
-    let keys = vec![(id(), false), (config.keypair.pubkey(), true)];
-    let (message, signers): (Message, Vec<&Keypair>) = if balance == 0 {
+    let keys = vec![(id(), false), (config.signers[0].pubkey(), true)];
+    let (message, signers): (Message, Vec<&dyn Signer>) = if balance == 0 {
         if info_pubkey != info_keypair.pubkey() {
             println!(
                 "Account {:?} does not exist. Generating new keypair...",
@@ -311,12 +322,12 @@ pub fn process_set_validator_info(
         }
         println!(
             "Publishing info for Validator {:?}",
-            config.keypair.pubkey()
+            config.signers[0].pubkey()
         );
         let lamports = rpc_client
             .get_minimum_balance_for_rent_exemption(ValidatorInfo::max_space() as usize)?;
         let mut instructions = config_instruction::create_account::<ValidatorInfo>(
-            &config.keypair.pubkey(),
+            &config.signers[0].pubkey(),
             &info_keypair.pubkey(),
             lamports,
             keys.clone(),
@@ -327,13 +338,13 @@ pub fn process_set_validator_info(
             keys,
             &validator_info,
         )]);
-        let signers = vec![&config.keypair, &info_keypair];
+        let signers = vec![config.signers[0], &info_keypair];
         let message = Message::new(instructions);
         (message, signers)
     } else {
         println!(
             "Updating Validator {:?} info at: {:?}",
-            config.keypair.pubkey(),
+            config.signers[0].pubkey(),
             info_pubkey
         );
         let instructions = vec![config_instruction::store(
@@ -342,17 +353,18 @@ pub fn process_set_validator_info(
             keys,
             &validator_info,
         )];
-        let message = Message::new_with_payer(instructions, Some(&config.keypair.pubkey()));
-        let signers = vec![&config.keypair];
+        let message = Message::new_with_payer(instructions, Some(&config.signers[0].pubkey()));
+        let signers = vec![config.signers[0]];
         (message, signers)
     };
 
     // Submit transaction
     let (recent_blockhash, fee_calculator) = rpc_client.get_recent_blockhash()?;
-    let mut tx = Transaction::new(&signers, message, recent_blockhash);
+    let mut tx = Transaction::new_unsigned(message);
+    tx.try_sign(&signers, recent_blockhash)?;
     check_account_for_fee(
         rpc_client,
-        &config.keypair.pubkey(),
+        &config.signers[0].pubkey(),
         &fee_calculator,
         &tx.message,
     )?;
@@ -390,9 +402,12 @@ pub fn process_get_validator_info(rpc_client: &RpcClient, pubkey: Option<Pubkey>
             parse_validator_info(&validator_info_pubkey, &validator_info_account)?;
         println!();
         println_name_value("Validator Identity Pubkey:", &validator_pubkey.to_string());
-        println_name_value("  info pubkey:", &validator_info_pubkey.to_string());
+        println_name_value("  Info Pubkey:", &validator_info_pubkey.to_string());
         for (key, value) in validator_info.iter() {
-            println_name_value(&format!("  {}:", key), &value.as_str().unwrap_or("?"));
+            println_name_value(
+                &format!("  {}:", titlecase(key)),
+                &value.as_str().unwrap_or("?"),
+            );
         }
     }
 

cli/src/vote.rs

@@ -1,19 +1,16 @@
-use crate::{
-    cli::{
-        build_balance_message, check_account_for_fee, check_unique_pubkeys,
-        log_instruction_custom_error, CliCommand, CliCommandInfo, CliConfig, CliError,
-        ProcessResult,
-    },
-    cluster_query::aggregate_epoch_credits,
+use crate::cli::{
+    build_balance_message, check_account_for_fee, check_unique_pubkeys, generate_unique_signers,
+    log_instruction_custom_error, CliCommand, CliCommandInfo, CliConfig, CliError, CliSignerInfo,
+    ProcessResult,
 };
 use clap::{value_t_or_exit, App, Arg, ArgMatches, SubCommand};
 use solana_clap_utils::{input_parsers::*, input_validators::*};
 use solana_client::rpc_client::RpcClient;
+use solana_remote_wallet::remote_wallet::RemoteWalletManager;
 use solana_sdk::{
     account::Account,
+    message::Message,
     pubkey::Pubkey,
-    signature::Keypair,
-    signature::KeypairUtil,
     system_instruction::{create_address_with_seed, SystemError},
     transaction::Transaction,
 };
@@ -21,6 +18,7 @@ use solana_vote_program::{
     vote_instruction::{self, VoteError},
     vote_state::{VoteAuthorize, VoteInit, VoteState},
 };
+use std::sync::Arc;
 
 pub trait VoteSubCommands {
     fn vote_subcommands(self) -> Self;
@@ -176,84 +174,91 @@ impl VoteSubCommands for App<'_, '_> {
                         .help("Display balance in lamports instead of SOL"),
                 ),
         )
-        .subcommand(
-            SubCommand::with_name("uptime")
-                .about("Show the uptime of a validator, based on epoch voting history")
-                .arg(
-                    Arg::with_name("vote_account_pubkey")
-                        .index(1)
-                        .value_name("VOTE ACCOUNT PUBKEY")
-                        .takes_value(true)
-                        .required(true)
-                        .validator(is_pubkey_or_keypair)
-                        .help("Vote account pubkey"),
-                )
-                .arg(
-                    Arg::with_name("span")
-                        .long("span")
-                        .value_name("NUM OF EPOCHS")
-                        .takes_value(true)
-                        .help("Number of recent epochs to examine"),
-                )
-                .arg(
-                    Arg::with_name("aggregate")
-                        .long("aggregate")
-                        .help("Aggregate uptime data across span"),
-                ),
-        )
     }
 }
 
-pub fn parse_vote_create_account(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, CliError> {
-    let vote_account = keypair_of(matches, "vote_account").unwrap();
+pub fn parse_vote_create_account(
+    matches: &ArgMatches<'_>,
+    default_signer_path: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
+) -> Result<CliCommandInfo, CliError> {
+    let (vote_account, _) = signer_of(matches, "vote_account", wallet_manager)?;
     let seed = matches.value_of("seed").map(|s| s.to_string());
     let identity_pubkey = pubkey_of(matches, "identity_pubkey").unwrap();
     let commission = value_t_or_exit!(matches, "commission", u8);
     let authorized_voter = pubkey_of(matches, "authorized_voter");
     let authorized_withdrawer = pubkey_of(matches, "authorized_withdrawer");
 
+    let payer_provided = None;
+    let CliSignerInfo { signers } = generate_unique_signers(
+        vec![payer_provided, vote_account],
+        matches,
+        default_signer_path,
+        wallet_manager,
+    )?;
+
     Ok(CliCommandInfo {
         command: CliCommand::CreateVoteAccount {
-            vote_account: vote_account.into(),
             seed,
             node_pubkey: identity_pubkey,
             authorized_voter,
             authorized_withdrawer,
             commission,
         },
-        require_keypair: true,
+        signers,
     })
 }
 
 pub fn parse_vote_authorize(
     matches: &ArgMatches<'_>,
+    default_signer_path: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
     vote_authorize: VoteAuthorize,
 ) -> Result<CliCommandInfo, CliError> {
     let vote_account_pubkey = pubkey_of(matches, "vote_account_pubkey").unwrap();
     let new_authorized_pubkey = pubkey_of(matches, "new_authorized_pubkey").unwrap();
 
+    let authorized_voter_provided = None;
+    let CliSignerInfo { signers } = generate_unique_signers(
+        vec![authorized_voter_provided],
+        matches,
+        default_signer_path,
+        wallet_manager,
+    )?;
+
     Ok(CliCommandInfo {
         command: CliCommand::VoteAuthorize {
             vote_account_pubkey,
             new_authorized_pubkey,
             vote_authorize,
         },
-        require_keypair: true,
+        signers,
     })
 }
 
-pub fn parse_vote_update_validator(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, CliError> {
+pub fn parse_vote_update_validator(
+    matches: &ArgMatches<'_>,
+    default_signer_path: &str,
+    wallet_manager: Option<&Arc<RemoteWalletManager>>,
+) -> Result<CliCommandInfo, CliError> {
     let vote_account_pubkey = pubkey_of(matches, "vote_account_pubkey").unwrap();
     let new_identity_pubkey = pubkey_of(matches, "new_identity_pubkey").unwrap();
-    let authorized_voter = keypair_of(matches, "authorized_voter").unwrap();
+    let (authorized_voter, _) = signer_of(matches, "authorized_voter", wallet_manager)?;
+
+    let payer_provided = None;
+    let CliSignerInfo { signers } = generate_unique_signers(
+        vec![payer_provided, authorized_voter],
+        matches,
+        default_signer_path,
+        wallet_manager,
+    )?;
 
     Ok(CliCommandInfo {
         command: CliCommand::VoteUpdateValidator {
             vote_account_pubkey,
             new_identity_pubkey,
-            authorized_voter: authorized_voter.into(),
         },
-        require_keypair: true,
+        signers,
     })
 }
 
@@ -267,38 +272,20 @@ pub fn parse_vote_get_account_command(
             pubkey: vote_account_pubkey,
             use_lamports_unit,
         },
-        require_keypair: false,
-    })
-}
-
-pub fn parse_vote_uptime_command(matches: &ArgMatches<'_>) -> Result<CliCommandInfo, CliError> {
-    let vote_account_pubkey = pubkey_of(matches, "vote_account_pubkey").unwrap();
-    let aggregate = matches.is_present("aggregate");
-    let span = if matches.is_present("span") {
-        Some(value_t_or_exit!(matches, "span", u64))
-    } else {
-        None
-    };
-    Ok(CliCommandInfo {
-        command: CliCommand::Uptime {
-            pubkey: vote_account_pubkey,
-            aggregate,
-            span,
-        },
-        require_keypair: false,
+        signers: vec![],
     })
 }
 
 pub fn process_create_vote_account(
     rpc_client: &RpcClient,
     config: &CliConfig,
-    vote_account: &Keypair,
     seed: &Option<String>,
     identity_pubkey: &Pubkey,
     authorized_voter: &Option<Pubkey>,
     authorized_withdrawer: &Option<Pubkey>,
     commission: u8,
 ) -> ProcessResult {
+    let vote_account = config.signers[1];
     let vote_account_pubkey = vote_account.pubkey();
     let vote_account_address = if let Some(seed) = seed {
         create_address_with_seed(&vote_account_pubkey, &seed, &solana_vote_program::id())?
@@ -306,7 +293,7 @@ pub fn process_create_vote_account(
         vote_account_pubkey
     };
     check_unique_pubkeys(
-        (&config.keypair.pubkey(), "cli keypair".to_string()),
+        (&config.signers[0].pubkey(), "cli keypair".to_string()),
         (&vote_account_address, "vote_account".to_string()),
     )?;
 
@@ -340,16 +327,16 @@ pub fn process_create_vote_account(
 
     let ixs = if let Some(seed) = seed {
         vote_instruction::create_account_with_seed(
-            &config.keypair.pubkey(), // from
-            &vote_account_address,    // to
-            &vote_account_pubkey,     // base
-            seed,                     // seed
+            &config.signers[0].pubkey(), // from
+            &vote_account_address,       // to
+            &vote_account_pubkey,        // base
+            seed,                        // seed
             &vote_init,
             required_balance,
         )
     } else {
         vote_instruction::create_account(
-            &config.keypair.pubkey(),
+            &config.signers[0].pubkey(),
             &vote_account_pubkey,
             &vote_init,
             required_balance,
@@ -357,20 +344,16 @@ pub fn process_create_vote_account(
     };
     let (recent_blockhash, fee_calculator) = rpc_client.get_recent_blockhash()?;
 
-    let signers = if vote_account_pubkey != config.keypair.pubkey() {
-        vec![&config.keypair, vote_account] // both must sign if `from` and `to` differ
-    } else {
-        vec![&config.keypair] // when stake_account == config.keypair and there's a seed, we only need one signature
-    };
-
-    let mut tx = Transaction::new_signed_instructions(&signers, ixs, recent_blockhash);
+    let message = Message::new(ixs);
+    let mut tx = Transaction::new_unsigned(message);
+    tx.try_sign(&config.signers, recent_blockhash)?;
     check_account_for_fee(
         rpc_client,
-        &config.keypair.pubkey(),
+        &config.signers[0].pubkey(),
         &fee_calculator,
         &tx.message,
     )?;
-    let result = rpc_client.send_and_confirm_transaction(&mut tx, &signers);
+    let result = rpc_client.send_and_confirm_transaction(&mut tx, &config.signers);
     log_instruction_custom_error::<SystemError>(result)
 }
 
@@ -387,25 +370,22 @@ pub fn process_vote_authorize(
     )?;
     let (recent_blockhash, fee_calculator) = rpc_client.get_recent_blockhash()?;
     let ixs = vec![vote_instruction::authorize(
-        vote_account_pubkey,      // vote account to update
-        &config.keypair.pubkey(), // current authorized voter
-        new_authorized_pubkey,    // new vote signer/withdrawer
-        vote_authorize,           // vote or withdraw
+        vote_account_pubkey,         // vote account to update
+        &config.signers[0].pubkey(), // current authorized voter
+        new_authorized_pubkey,       // new vote signer/withdrawer
+        vote_authorize,              // vote or withdraw
     )];
 
-    let mut tx = Transaction::new_signed_with_payer(
-        ixs,
-        Some(&config.keypair.pubkey()),
-        &[&config.keypair],
-        recent_blockhash,
-    );
+    let message = Message::new_with_payer(ixs, Some(&config.signers[0].pubkey()));
+    let mut tx = Transaction::new_unsigned(message);
+    tx.try_sign(&config.signers, recent_blockhash)?;
     check_account_for_fee(
         rpc_client,
-        &config.keypair.pubkey(),
+        &config.signers[0].pubkey(),
         &fee_calculator,
         &tx.message,
     )?;
-    let result = rpc_client.send_and_confirm_transaction(&mut tx, &[&config.keypair]);
+    let result = rpc_client.send_and_confirm_transaction(&mut tx, &[config.signers[0]]);
     log_instruction_custom_error::<VoteError>(result)
 }
 
@@ -414,8 +394,8 @@ pub fn process_vote_update_validator(
     config: &CliConfig,
     vote_account_pubkey: &Pubkey,
     new_identity_pubkey: &Pubkey,
-    authorized_voter: &Keypair,
 ) -> ProcessResult {
+    let authorized_voter = config.signers[1];
     check_unique_pubkeys(
         (vote_account_pubkey, "vote_account_pubkey".to_string()),
         (new_identity_pubkey, "new_identity_pubkey".to_string()),
@@ -427,19 +407,16 @@ pub fn process_vote_update_validator(
         new_identity_pubkey,
     )];
 
-    let mut tx = Transaction::new_signed_with_payer(
-        ixs,
-        Some(&config.keypair.pubkey()),
-        &[&config.keypair, authorized_voter],
-        recent_blockhash,
-    );
+    let message = Message::new_with_payer(ixs, Some(&config.signers[0].pubkey()));
+    let mut tx = Transaction::new_unsigned(message);
+    tx.try_sign(&config.signers, recent_blockhash)?;
     check_account_for_fee(
         rpc_client,
-        &config.keypair.pubkey(),
+        &config.signers[0].pubkey(),
         &fee_calculator,
         &tx.message,
     )?;
-    let result = rpc_client.send_and_confirm_transaction(&mut tx, &[&config.keypair]);
+    let result = rpc_client.send_and_confirm_transaction(&mut tx, &config.signers);
     log_instruction_custom_error::<VoteError>(result)
 }
 
@@ -476,25 +453,25 @@ pub fn process_show_vote_account(
     let epoch_schedule = rpc_client.get_epoch_schedule()?;
 
     println!(
-        "account balance: {}",
+        "Account Balance: {}",
         build_balance_message(vote_account.lamports, use_lamports_unit, true)
     );
-    println!("validator identity: {}", vote_state.node_pubkey);
-    println!("authorized voter: {}", vote_state.authorized_voter);
+    println!("Validator Identity: {}", vote_state.node_pubkey);
+    println!("Authorized Voter: {:?}", vote_state.authorized_voters());
     println!(
-        "authorized withdrawer: {}",
+        "Authorized Withdrawer: {}",
         vote_state.authorized_withdrawer
     );
-    println!("credits: {}", vote_state.credits());
-    println!("commission: {}%", vote_state.commission);
+    println!("Credits: {}", vote_state.credits());
+    println!("Commission: {}%", vote_state.commission);
     println!(
-        "root slot: {}",
+        "Root Slot: {}",
         match vote_state.root_slot {
             Some(slot) => slot.to_string(),
             None => "~".to_string(),
         }
     );
-    println!("recent timestamp: {:?}", vote_state.last_timestamp);
+    println!("Recent Timestamp: {:?}", vote_state.last_timestamp);
     if !vote_state.votes.is_empty() {
         println!("recent votes:");
         for vote in &vote_state.votes {
@@ -504,7 +481,7 @@ pub fn process_show_vote_account(
             );
         }
 
-        println!("epoch voting history:");
+        println!("Epoch Voting History:");
         for (epoch, credits, prev_credits) in vote_state.epoch_credits() {
             let credits_earned = credits - prev_credits;
             let slots_in_epoch = epoch_schedule.get_slots_in_epoch(*epoch);
@@ -517,65 +494,11 @@ pub fn process_show_vote_account(
     Ok("".to_string())
 }
 
-pub fn process_uptime(
-    rpc_client: &RpcClient,
-    _config: &CliConfig,
-    vote_account_pubkey: &Pubkey,
-    aggregate: bool,
-    span: Option<u64>,
-) -> ProcessResult {
-    let (_vote_account, vote_state) = get_vote_account(rpc_client, vote_account_pubkey)?;
-
-    let epoch_schedule = rpc_client.get_epoch_schedule()?;
-
-    println!("validator identity: {}", vote_state.node_pubkey);
-    println!("authorized voter: {}", vote_state.authorized_voter);
-    if !vote_state.votes.is_empty() {
-        println!("uptime:");
-
-        let epoch_credits: Vec<(u64, u64, u64)> = if let Some(x) = span {
-            vote_state
-                .epoch_credits()
-                .iter()
-                .rev()
-                .take(x as usize)
-                .cloned()
-                .collect()
-        } else {
-            vote_state.epoch_credits().iter().rev().cloned().collect()
-        };
-
-        if aggregate {
-            let (total_credits, total_slots, epochs) =
-                aggregate_epoch_credits(&epoch_credits, &epoch_schedule);
-            if total_slots > 0 {
-                let total_uptime = 100_f64 * total_credits as f64 / total_slots as f64;
-                println!("{:.2}% over {} epochs", total_uptime, epochs);
-            } else {
-                println!("Insufficient voting history available");
-            }
-        } else {
-            for (epoch, credits, prev_credits) in epoch_credits {
-                let credits_earned = credits - prev_credits;
-                let slots_in_epoch = epoch_schedule.get_slots_in_epoch(epoch);
-                let uptime = credits_earned as f64 / slots_in_epoch as f64;
-                println!("- epoch: {} {:.2}% uptime", epoch, uptime * 100_f64,);
-            }
-        }
-        if let Some(x) = span {
-            if x > vote_state.epoch_credits().len() as u64 {
-                println!("(span longer than available epochs)");
-            }
-        }
-    }
-    Ok("".to_string())
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
     use crate::cli::{app, parse_command};
-    use solana_sdk::signature::write_keypair;
+    use solana_sdk::signature::{read_keypair_file, write_keypair, Keypair, Signer};
     use tempfile::NamedTempFile;
 
     fn make_tmp_file() -> (String, NamedTempFile) {
@@ -593,6 +516,10 @@ mod tests {
         let pubkey2 = keypair2.pubkey();
         let pubkey2_string = pubkey2.to_string();
 
+        let default_keypair = Keypair::new();
+        let (default_keypair_file, mut tmp_file) = make_tmp_file();
+        write_keypair(&default_keypair, tmp_file.as_file_mut()).unwrap();
+
         let test_authorize_voter = test_commands.clone().get_matches_from(vec![
             "test",
             "vote-authorize-voter",
@@ -600,14 +527,14 @@ mod tests {
             &pubkey2_string,
         ]);
         assert_eq!(
-            parse_command(&test_authorize_voter).unwrap(),
+            parse_command(&test_authorize_voter, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::VoteAuthorize {
                     vote_account_pubkey: pubkey,
                     new_authorized_pubkey: pubkey2,
                     vote_authorize: VoteAuthorize::Voter
                 },
-                require_keypair: true
+                signers: vec![read_keypair_file(&default_keypair_file).unwrap().into()],
             }
         );
 
@@ -626,17 +553,19 @@ mod tests {
             "10",
         ]);
         assert_eq!(
-            parse_command(&test_create_vote_account).unwrap(),
+            parse_command(&test_create_vote_account, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::CreateVoteAccount {
-                    vote_account: keypair.into(),
                     seed: None,
                     node_pubkey,
                     authorized_voter: None,
                     authorized_withdrawer: None,
                     commission: 10,
                 },
-                require_keypair: true
+                signers: vec![
+                    read_keypair_file(&default_keypair_file).unwrap().into(),
+                    Box::new(keypair)
+                ],
             }
         );
 
@@ -651,17 +580,19 @@ mod tests {
             &node_pubkey_string,
         ]);
         assert_eq!(
-            parse_command(&test_create_vote_account2).unwrap(),
+            parse_command(&test_create_vote_account2, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::CreateVoteAccount {
-                    vote_account: keypair.into(),
                     seed: None,
                     node_pubkey,
                     authorized_voter: None,
                     authorized_withdrawer: None,
                     commission: 100,
                 },
-                require_keypair: true
+                signers: vec![
+                    read_keypair_file(&default_keypair_file).unwrap().into(),
+                    Box::new(keypair)
+                ],
             }
         );
 
@@ -680,17 +611,19 @@ mod tests {
             &authed.to_string(),
         ]);
         assert_eq!(
-            parse_command(&test_create_vote_account3).unwrap(),
+            parse_command(&test_create_vote_account3, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::CreateVoteAccount {
-                    vote_account: keypair.into(),
                     seed: None,
                     node_pubkey,
                     authorized_voter: Some(authed),
                     authorized_withdrawer: None,
                     commission: 100
                 },
-                require_keypair: true
+                signers: vec![
+                    read_keypair_file(&default_keypair_file).unwrap().into(),
+                    Box::new(keypair)
+                ],
             }
         );
 
@@ -707,17 +640,19 @@ mod tests {
             &authed.to_string(),
         ]);
         assert_eq!(
-            parse_command(&test_create_vote_account4).unwrap(),
+            parse_command(&test_create_vote_account4, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::CreateVoteAccount {
-                    vote_account: keypair.into(),
                     seed: None,
                     node_pubkey,
                     authorized_voter: None,
                     authorized_withdrawer: Some(authed),
                     commission: 100
                 },
-                require_keypair: true
+                signers: vec![
+                    read_keypair_file(&default_keypair_file).unwrap().into(),
+                    Box::new(keypair)
+                ],
             }
         );
 
@@ -729,38 +664,16 @@ mod tests {
             &keypair_file,
         ]);
         assert_eq!(
-            parse_command(&test_update_validator).unwrap(),
+            parse_command(&test_update_validator, &default_keypair_file, None).unwrap(),
             CliCommandInfo {
                 command: CliCommand::VoteUpdateValidator {
                     vote_account_pubkey: pubkey,
                     new_identity_pubkey: pubkey2,
-                    authorized_voter: solana_sdk::signature::read_keypair_file(&keypair_file)
-                        .unwrap()
-                        .into(),
                 },
-                require_keypair: true
-            }
-        );
-
-        // Test Uptime Subcommand
-        let pubkey = Pubkey::new_rand();
-        let matches = test_commands.clone().get_matches_from(vec![
-            "test",
-            "uptime",
-            &pubkey.to_string(),
-            "--span",
-            "4",
-            "--aggregate",
-        ]);
-        assert_eq!(
-            parse_command(&matches).unwrap(),
-            CliCommandInfo {
-                command: CliCommand::Uptime {
-                    pubkey,
-                    aggregate: true,
-                    span: Some(4)
-                },
-                require_keypair: false
+                signers: vec![
+                    read_keypair_file(&default_keypair_file).unwrap().into(),
+                    Box::new(read_keypair_file(&keypair_file).unwrap())
+                ],
             }
         );
     }

cli/tests/deploy.rs

@@ -1,9 +1,9 @@
 use serde_json::Value;
 use solana_cli::cli::{process_command, CliCommand, CliConfig};
 use solana_client::rpc_client::RpcClient;
-use solana_core::validator::new_validator_for_tests;
+use solana_core::validator::TestValidator;
 use solana_faucet::faucet::run_local_faucet;
-use solana_sdk::{bpf_loader, pubkey::Pubkey};
+use solana_sdk::{bpf_loader, pubkey::Pubkey, signature::Keypair};
 use std::{
     fs::{remove_dir_all, File},
     io::Read,
@@ -22,7 +22,13 @@ fn test_cli_deploy_program() {
     pathbuf.push("noop");
     pathbuf.set_extension("so");
 
-    let (server, leader_data, alice, ledger_path) = new_validator_for_tests();
+    let TestValidator {
+        server,
+        leader_data,
+        alice,
+        ledger_path,
+        ..
+    } = TestValidator::run();
 
     let (sender, receiver) = channel();
     run_local_faucet(alice, sender, None);
@@ -38,13 +44,15 @@ fn test_cli_deploy_program() {
         .unwrap();
 
     let mut config = CliConfig::default();
+    let keypair = Keypair::new();
     config.json_rpc_url = format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
     config.command = CliCommand::Airdrop {
         faucet_host: None,
         faucet_port: faucet_addr.port(),
+        pubkey: None,
         lamports: minimum_balance_for_rent_exemption + 1, // min balance for rent exemption + leftover for tx processing
-        use_lamports_unit: true,
     };
+    config.signers = vec![&keypair];
     process_command(&config).unwrap();
 
     config.command = CliCommand::Deploy(pathbuf.to_str().unwrap().to_string());

cli/tests/nonce.rs

@@ -1,29 +1,15 @@
-use solana_cli::cli::{
-    process_command, request_and_confirm_airdrop, CliCommand, CliConfig, SigningAuthority,
-};
+use solana_cli::cli::{process_command, request_and_confirm_airdrop, CliCommand, CliConfig};
 use solana_client::rpc_client::RpcClient;
+use solana_core::validator::TestValidator;
 use solana_faucet::faucet::run_local_faucet;
 use solana_sdk::{
     hash::Hash,
     pubkey::Pubkey,
-    signature::{read_keypair_file, write_keypair, Keypair, KeypairUtil},
+    signature::{keypair_from_seed, Keypair, Signer},
     system_instruction::create_address_with_seed,
     system_program,
 };
-use std::fs::remove_dir_all;
-use std::sync::mpsc::channel;
-
-#[cfg(test)]
-use solana_core::validator::new_validator_for_tests;
-use std::thread::sleep;
-use std::time::Duration;
-
-use tempfile::NamedTempFile;
-
-fn make_tmp_file() -> (String, NamedTempFile) {
-    let tmp_file = NamedTempFile::new().unwrap();
-    (String::from(tmp_file.path().to_str().unwrap()), tmp_file)
-}
+use std::{fs::remove_dir_all, sync::mpsc::channel, thread::sleep, time::Duration};
 
 fn check_balance(expected_balance: u64, client: &RpcClient, pubkey: &Pubkey) {
     (0..5).for_each(|tries| {
@@ -40,32 +26,21 @@ fn check_balance(expected_balance: u64, client: &RpcClient, pubkey: &Pubkey) {
 
 #[test]
 fn test_nonce() {
-    let (server, leader_data, alice, ledger_path) = new_validator_for_tests();
+    let TestValidator {
+        server,
+        leader_data,
+        alice,
+        ledger_path,
+        ..
+    } = TestValidator::run();
     let (sender, receiver) = channel();
     run_local_faucet(alice, sender, None);
     let faucet_addr = receiver.recv().unwrap();
 
     let rpc_client = RpcClient::new_socket(leader_data.rpc);
+    let json_rpc_url = format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
 
-    let mut config_payer = CliConfig::default();
-    config_payer.json_rpc_url =
-        format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
-
-    let mut config_nonce = CliConfig::default();
-    config_nonce.json_rpc_url =
-        format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
-    let (keypair_file, mut tmp_file) = make_tmp_file();
-    write_keypair(&config_nonce.keypair, tmp_file.as_file_mut()).unwrap();
-
-    full_battery_tests(
-        &rpc_client,
-        &faucet_addr,
-        &mut config_payer,
-        &mut config_nonce,
-        &keypair_file,
-        None,
-        None,
-    );
+    full_battery_tests(&rpc_client, &faucet_addr, json_rpc_url, None, false);
 
     server.close().unwrap();
     remove_dir_all(ledger_path).unwrap();
@@ -73,31 +48,26 @@ fn test_nonce() {
 
 #[test]
 fn test_nonce_with_seed() {
-    let (server, leader_data, alice, ledger_path) = new_validator_for_tests();
+    let TestValidator {
+        server,
+        leader_data,
+        alice,
+        ledger_path,
+        ..
+    } = TestValidator::run();
     let (sender, receiver) = channel();
     run_local_faucet(alice, sender, None);
     let faucet_addr = receiver.recv().unwrap();
 
     let rpc_client = RpcClient::new_socket(leader_data.rpc);
-
-    let mut config_payer = CliConfig::default();
-    config_payer.json_rpc_url =
-        format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
-
-    let mut config_nonce = CliConfig::default();
-    config_nonce.json_rpc_url =
-        format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
-    let (keypair_file, mut tmp_file) = make_tmp_file();
-    write_keypair(&config_nonce.keypair, tmp_file.as_file_mut()).unwrap();
+    let json_rpc_url = format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
 
     full_battery_tests(
         &rpc_client,
         &faucet_addr,
-        &mut config_payer,
-        &mut config_nonce,
-        &keypair_file,
+        json_rpc_url,
         Some(String::from("seed")),
-        None,
+        false,
     );
 
     server.close().unwrap();
@@ -106,84 +76,85 @@ fn test_nonce_with_seed() {
 
 #[test]
 fn test_nonce_with_authority() {
-    let (server, leader_data, alice, ledger_path) = new_validator_for_tests();
+    let TestValidator {
+        server,
+        leader_data,
+        alice,
+        ledger_path,
+        ..
+    } = TestValidator::run();
     let (sender, receiver) = channel();
     run_local_faucet(alice, sender, None);
     let faucet_addr = receiver.recv().unwrap();
 
     let rpc_client = RpcClient::new_socket(leader_data.rpc);
+    let json_rpc_url = format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
 
-    let mut config_payer = CliConfig::default();
-    config_payer.json_rpc_url =
-        format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
-
-    let mut config_nonce = CliConfig::default();
-    config_nonce.json_rpc_url =
-        format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
-    let (nonce_keypair_file, mut tmp_file) = make_tmp_file();
-    write_keypair(&config_nonce.keypair, tmp_file.as_file_mut()).unwrap();
-
-    let nonce_authority = Keypair::new();
-    let (authority_keypair_file, mut tmp_file2) = make_tmp_file();
-    write_keypair(&nonce_authority, tmp_file2.as_file_mut()).unwrap();
-
-    full_battery_tests(
-        &rpc_client,
-        &faucet_addr,
-        &mut config_payer,
-        &mut config_nonce,
-        &nonce_keypair_file,
-        None,
-        Some(&authority_keypair_file),
-    );
+    full_battery_tests(&rpc_client, &faucet_addr, json_rpc_url, None, true);
 
     server.close().unwrap();
     remove_dir_all(ledger_path).unwrap();
 }
 
-fn read_keypair_from_option(keypair_file: &Option<&str>) -> Option<SigningAuthority> {
-    keypair_file.map(|akf| read_keypair_file(&akf).unwrap().into())
-}
-
 fn full_battery_tests(
     rpc_client: &RpcClient,
     faucet_addr: &std::net::SocketAddr,
-    config_payer: &mut CliConfig,
-    config_nonce: &mut CliConfig,
-    nonce_keypair_file: &str,
+    json_rpc_url: String,
     seed: Option<String>,
-    authority_keypair_file: Option<&str>,
+    use_nonce_authority: bool,
 ) {
+    let mut config_payer = CliConfig::default();
+    config_payer.json_rpc_url = json_rpc_url.clone();
+    let payer = Keypair::new();
+    config_payer.signers = vec![&payer];
+
     request_and_confirm_airdrop(
         &rpc_client,
         &faucet_addr,
-        &config_payer.keypair.pubkey(),
+        &config_payer.signers[0].pubkey(),
         2000,
     )
     .unwrap();
-    check_balance(2000, &rpc_client, &config_payer.keypair.pubkey());
+    check_balance(2000, &rpc_client, &config_payer.signers[0].pubkey());
+
+    let mut config_nonce = CliConfig::default();
+    config_nonce.json_rpc_url = json_rpc_url;
+    let nonce_keypair = keypair_from_seed(&[0u8; 32]).unwrap();
+    config_nonce.signers = vec![&nonce_keypair];
 
     let nonce_account = if let Some(seed) = seed.as_ref() {
-        create_address_with_seed(&config_nonce.keypair.pubkey(), seed, &system_program::id())
-            .unwrap()
+        create_address_with_seed(
+            &config_nonce.signers[0].pubkey(),
+            seed,
+            &system_program::id(),
+        )
+        .unwrap()
     } else {
-        config_nonce.keypair.pubkey()
+        nonce_keypair.pubkey()
+    };
+
+    let nonce_authority = Keypair::new();
+    let optional_authority = if use_nonce_authority {
+        Some(nonce_authority.pubkey())
+    } else {
+        None
     };
 
     // Create nonce account
+    config_payer.signers.push(&nonce_keypair);
     config_payer.command = CliCommand::CreateNonceAccount {
-        nonce_account: read_keypair_file(&nonce_keypair_file).unwrap().into(),
+        nonce_account: 1,
         seed,
-        nonce_authority: read_keypair_from_option(&authority_keypair_file)
-            .map(|na: SigningAuthority| na.pubkey()),
+        nonce_authority: optional_authority,
         lamports: 1000,
     };
 
     process_command(&config_payer).unwrap();
-    check_balance(1000, &rpc_client, &config_payer.keypair.pubkey());
+    check_balance(1000, &rpc_client, &config_payer.signers[0].pubkey());
     check_balance(1000, &rpc_client, &nonce_account);
 
     // Get nonce
+    config_payer.signers.pop();
     config_payer.command = CliCommand::GetNonce(nonce_account);
     let first_nonce_string = process_command(&config_payer).unwrap();
     let first_nonce = first_nonce_string.parse::<Hash>().unwrap();
@@ -195,14 +166,24 @@ fn full_battery_tests(
 
     assert_eq!(first_nonce, second_nonce);
 
+    let mut authorized_signers: Vec<&dyn Signer> = vec![&payer];
+    let index = if use_nonce_authority {
+        authorized_signers.push(&nonce_authority);
+        1
+    } else {
+        0
+    };
+
     // New nonce
+    config_payer.signers = authorized_signers.clone();
     config_payer.command = CliCommand::NewNonce {
         nonce_account,
-        nonce_authority: read_keypair_from_option(&authority_keypair_file),
+        nonce_authority: index,
     };
     process_command(&config_payer).unwrap();
 
     // Get nonce
+    config_payer.signers = vec![&payer];
     config_payer.command = CliCommand::GetNonce(nonce_account);
     let third_nonce_string = process_command(&config_payer).unwrap();
     let third_nonce = third_nonce_string.parse::<Hash>().unwrap();
@@ -211,14 +192,15 @@ fn full_battery_tests(
 
     // Withdraw from nonce account
     let payee_pubkey = Pubkey::new_rand();
+    config_payer.signers = authorized_signers;
     config_payer.command = CliCommand::WithdrawFromNonceAccount {
         nonce_account,
-        nonce_authority: read_keypair_from_option(&authority_keypair_file),
+        nonce_authority: index,
         destination_account_pubkey: payee_pubkey,
         lamports: 100,
     };
     process_command(&config_payer).unwrap();
-    check_balance(1000, &rpc_client, &config_payer.keypair.pubkey());
+    check_balance(1000, &rpc_client, &config_payer.signers[0].pubkey());
     check_balance(900, &rpc_client, &nonce_account);
     check_balance(100, &rpc_client, &payee_pubkey);
 
@@ -231,48 +213,37 @@ fn full_battery_tests(
 
     // Set new authority
     let new_authority = Keypair::new();
-    let (new_authority_keypair_file, mut tmp_file) = make_tmp_file();
-    write_keypair(&new_authority, tmp_file.as_file_mut()).unwrap();
     config_payer.command = CliCommand::AuthorizeNonceAccount {
         nonce_account,
-        nonce_authority: read_keypair_from_option(&authority_keypair_file),
-        new_authority: read_keypair_file(&new_authority_keypair_file)
-            .unwrap()
-            .pubkey(),
+        nonce_authority: index,
+        new_authority: new_authority.pubkey(),
     };
     process_command(&config_payer).unwrap();
 
     // Old authority fails now
     config_payer.command = CliCommand::NewNonce {
         nonce_account,
-        nonce_authority: read_keypair_from_option(&authority_keypair_file),
+        nonce_authority: index,
     };
     process_command(&config_payer).unwrap_err();
 
     // New authority can advance nonce
+    config_payer.signers = vec![&payer, &new_authority];
     config_payer.command = CliCommand::NewNonce {
         nonce_account,
-        nonce_authority: Some(
-            read_keypair_file(&new_authority_keypair_file)
-                .unwrap()
-                .into(),
-        ),
+        nonce_authority: 1,
     };
     process_command(&config_payer).unwrap();
 
     // New authority can withdraw from nonce account
     config_payer.command = CliCommand::WithdrawFromNonceAccount {
         nonce_account,
-        nonce_authority: Some(
-            read_keypair_file(&new_authority_keypair_file)
-                .unwrap()
-                .into(),
-        ),
+        nonce_authority: 1,
         destination_account_pubkey: payee_pubkey,
         lamports: 100,
     };
     process_command(&config_payer).unwrap();
-    check_balance(1000, &rpc_client, &config_payer.keypair.pubkey());
+    check_balance(1000, &rpc_client, &config_payer.signers[0].pubkey());
     check_balance(800, &rpc_client, &nonce_account);
     check_balance(200, &rpc_client, &payee_pubkey);
 }

cli/tests/pay.rs

@@ -1,31 +1,21 @@
 use chrono::prelude::*;
 use serde_json::Value;
-use solana_cli::cli::{
-    process_command, request_and_confirm_airdrop, CliCommand, CliConfig, PayCommand,
+use solana_clap_utils::keypair::presigner_from_pubkey_sigs;
+use solana_cli::{
+    cli::{process_command, request_and_confirm_airdrop, CliCommand, CliConfig, PayCommand},
+    offline::{parse_sign_only_reply_string, BlockhashQuery},
 };
 use solana_client::rpc_client::RpcClient;
+use solana_core::validator::TestValidator;
 use solana_faucet::faucet::run_local_faucet;
 use solana_sdk::{
     account_utils::StateMut,
-    hash::Hash,
+    fee_calculator::FeeCalculator,
     nonce_state::NonceState,
     pubkey::Pubkey,
-    signature::{read_keypair_file, write_keypair, Keypair, KeypairUtil, Signature},
+    signature::{Keypair, Signer},
 };
-use std::fs::remove_dir_all;
-use std::str::FromStr;
-use std::sync::mpsc::channel;
-
-#[cfg(test)]
-use solana_core::validator::new_validator_for_tests;
-use std::thread::sleep;
-use std::time::Duration;
-use tempfile::NamedTempFile;
-
-fn make_tmp_file() -> (String, NamedTempFile) {
-    let tmp_file = NamedTempFile::new().unwrap();
-    (String::from(tmp_file.path().to_str().unwrap()), tmp_file)
-}
+use std::{fs::remove_dir_all, sync::mpsc::channel, thread::sleep, time::Duration};
 
 fn check_balance(expected_balance: u64, client: &RpcClient, pubkey: &Pubkey) {
     (0..5).for_each(|tries| {
@@ -42,7 +32,13 @@ fn check_balance(expected_balance: u64, client: &RpcClient, pubkey: &Pubkey) {
 
 #[test]
 fn test_cli_timestamp_tx() {
-    let (server, leader_data, alice, ledger_path) = new_validator_for_tests();
+    let TestValidator {
+        server,
+        leader_data,
+        alice,
+        ledger_path,
+        ..
+    } = TestValidator::run();
     let bob_pubkey = Pubkey::new_rand();
 
     let (sender, receiver) = channel();
@@ -50,32 +46,36 @@ fn test_cli_timestamp_tx() {
     let faucet_addr = receiver.recv().unwrap();
 
     let rpc_client = RpcClient::new_socket(leader_data.rpc);
+    let default_signer0 = Keypair::new();
+    let default_signer1 = Keypair::new();
 
     let mut config_payer = CliConfig::default();
     config_payer.json_rpc_url =
         format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
+    config_payer.signers = vec![&default_signer0];
 
     let mut config_witness = CliConfig::default();
     config_witness.json_rpc_url = config_payer.json_rpc_url.clone();
+    config_witness.signers = vec![&default_signer1];
 
     assert_ne!(
-        config_payer.keypair.pubkey(),
-        config_witness.keypair.pubkey()
+        config_payer.signers[0].pubkey(),
+        config_witness.signers[0].pubkey()
     );
 
     request_and_confirm_airdrop(
         &rpc_client,
         &faucet_addr,
-        &config_payer.keypair.pubkey(),
+        &config_payer.signers[0].pubkey(),
         50,
     )
     .unwrap();
-    check_balance(50, &rpc_client, &config_payer.keypair.pubkey());
+    check_balance(50, &rpc_client, &config_payer.signers[0].pubkey());
 
     request_and_confirm_airdrop(
         &rpc_client,
         &faucet_addr,
-        &config_witness.keypair.pubkey(),
+        &config_witness.signers[0].pubkey(),
         1,
     )
     .unwrap();
@@ -87,7 +87,7 @@ fn test_cli_timestamp_tx() {
         lamports: 10,
         to: bob_pubkey,
         timestamp: Some(dt),
-        timestamp_pubkey: Some(config_witness.keypair.pubkey()),
+        timestamp_pubkey: Some(config_witness.signers[0].pubkey()),
         ..PayCommand::default()
     });
     let sig_response = process_command(&config_payer);
@@ -99,7 +99,7 @@ fn test_cli_timestamp_tx() {
         .expect("base58-encoded public key");
     let process_id = Pubkey::new(&process_id_vec);
 
-    check_balance(40, &rpc_client, &config_payer.keypair.pubkey()); // config_payer balance
+    check_balance(40, &rpc_client, &config_payer.signers[0].pubkey()); // config_payer balance
     check_balance(10, &rpc_client, &process_id); // contract balance
     check_balance(0, &rpc_client, &bob_pubkey); // recipient balance
 
@@ -107,7 +107,7 @@ fn test_cli_timestamp_tx() {
     config_witness.command = CliCommand::TimeElapsed(bob_pubkey, process_id, dt);
     process_command(&config_witness).unwrap();
 
-    check_balance(40, &rpc_client, &config_payer.keypair.pubkey()); // config_payer balance
+    check_balance(40, &rpc_client, &config_payer.signers[0].pubkey()); // config_payer balance
     check_balance(0, &rpc_client, &process_id); // contract balance
     check_balance(10, &rpc_client, &bob_pubkey); // recipient balance
 
@@ -117,7 +117,13 @@ fn test_cli_timestamp_tx() {
 
 #[test]
 fn test_cli_witness_tx() {
-    let (server, leader_data, alice, ledger_path) = new_validator_for_tests();
+    let TestValidator {
+        server,
+        leader_data,
+        alice,
+        ledger_path,
+        ..
+    } = TestValidator::run();
     let bob_pubkey = Pubkey::new_rand();
 
     let (sender, receiver) = channel();
@@ -125,30 +131,34 @@ fn test_cli_witness_tx() {
     let faucet_addr = receiver.recv().unwrap();
 
     let rpc_client = RpcClient::new_socket(leader_data.rpc);
+    let default_signer0 = Keypair::new();
+    let default_signer1 = Keypair::new();
 
     let mut config_payer = CliConfig::default();
     config_payer.json_rpc_url =
         format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
+    config_payer.signers = vec![&default_signer0];
 
     let mut config_witness = CliConfig::default();
     config_witness.json_rpc_url = config_payer.json_rpc_url.clone();
+    config_witness.signers = vec![&default_signer1];
 
     assert_ne!(
-        config_payer.keypair.pubkey(),
-        config_witness.keypair.pubkey()
+        config_payer.signers[0].pubkey(),
+        config_witness.signers[0].pubkey()
     );
 
     request_and_confirm_airdrop(
         &rpc_client,
         &faucet_addr,
-        &config_payer.keypair.pubkey(),
+        &config_payer.signers[0].pubkey(),
         50,
     )
     .unwrap();
     request_and_confirm_airdrop(
         &rpc_client,
         &faucet_addr,
-        &config_witness.keypair.pubkey(),
+        &config_witness.signers[0].pubkey(),
         1,
     )
     .unwrap();
@@ -157,7 +167,7 @@ fn test_cli_witness_tx() {
     config_payer.command = CliCommand::Pay(PayCommand {
         lamports: 10,
         to: bob_pubkey,
-        witnesses: Some(vec![config_witness.keypair.pubkey()]),
+        witnesses: Some(vec![config_witness.signers[0].pubkey()]),
         ..PayCommand::default()
     });
     let sig_response = process_command(&config_payer);
@@ -169,7 +179,7 @@ fn test_cli_witness_tx() {
         .expect("base58-encoded public key");
     let process_id = Pubkey::new(&process_id_vec);
 
-    check_balance(40, &rpc_client, &config_payer.keypair.pubkey()); // config_payer balance
+    check_balance(40, &rpc_client, &config_payer.signers[0].pubkey()); // config_payer balance
     check_balance(10, &rpc_client, &process_id); // contract balance
     check_balance(0, &rpc_client, &bob_pubkey); // recipient balance
 
@@ -177,7 +187,7 @@ fn test_cli_witness_tx() {
     config_witness.command = CliCommand::Witness(bob_pubkey, process_id);
     process_command(&config_witness).unwrap();
 
-    check_balance(40, &rpc_client, &config_payer.keypair.pubkey()); // config_payer balance
+    check_balance(40, &rpc_client, &config_payer.signers[0].pubkey()); // config_payer balance
     check_balance(0, &rpc_client, &process_id); // contract balance
     check_balance(10, &rpc_client, &bob_pubkey); // recipient balance
 
@@ -187,7 +197,13 @@ fn test_cli_witness_tx() {
 
 #[test]
 fn test_cli_cancel_tx() {
-    let (server, leader_data, alice, ledger_path) = new_validator_for_tests();
+    let TestValidator {
+        server,
+        leader_data,
+        alice,
+        ledger_path,
+        ..
+    } = TestValidator::run();
     let bob_pubkey = Pubkey::new_rand();
 
     let (sender, receiver) = channel();
@@ -195,23 +211,27 @@ fn test_cli_cancel_tx() {
     let faucet_addr = receiver.recv().unwrap();
 
     let rpc_client = RpcClient::new_socket(leader_data.rpc);
+    let default_signer0 = Keypair::new();
+    let default_signer1 = Keypair::new();
 
     let mut config_payer = CliConfig::default();
     config_payer.json_rpc_url =
         format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
+    config_payer.signers = vec![&default_signer0];
 
     let mut config_witness = CliConfig::default();
     config_witness.json_rpc_url = config_payer.json_rpc_url.clone();
+    config_witness.signers = vec![&default_signer1];
 
     assert_ne!(
-        config_payer.keypair.pubkey(),
-        config_witness.keypair.pubkey()
+        config_payer.signers[0].pubkey(),
+        config_witness.signers[0].pubkey()
     );
 
     request_and_confirm_airdrop(
         &rpc_client,
         &faucet_addr,
-        &config_payer.keypair.pubkey(),
+        &config_payer.signers[0].pubkey(),
         50,
     )
     .unwrap();
@@ -220,7 +240,7 @@ fn test_cli_cancel_tx() {
     config_payer.command = CliCommand::Pay(PayCommand {
         lamports: 10,
         to: bob_pubkey,
-        witnesses: Some(vec![config_witness.keypair.pubkey()]),
+        witnesses: Some(vec![config_witness.signers[0].pubkey()]),
         cancelable: true,
         ..PayCommand::default()
     });
@@ -233,7 +253,7 @@ fn test_cli_cancel_tx() {
         .expect("base58-encoded public key");
     let process_id = Pubkey::new(&process_id_vec);
 
-    check_balance(40, &rpc_client, &config_payer.keypair.pubkey()); // config_payer balance
+    check_balance(40, &rpc_client, &config_payer.signers[0].pubkey()); // config_payer balance
     check_balance(10, &rpc_client, &process_id); // contract balance
     check_balance(0, &rpc_client, &bob_pubkey); // recipient balance
 
@@ -241,7 +261,7 @@ fn test_cli_cancel_tx() {
     config_payer.command = CliCommand::Cancel(process_id);
     process_command(&config_payer).unwrap();
 
-    check_balance(50, &rpc_client, &config_payer.keypair.pubkey()); // config_payer balance
+    check_balance(50, &rpc_client, &config_payer.signers[0].pubkey()); // config_payer balance
     check_balance(0, &rpc_client, &process_id); // contract balance
     check_balance(0, &rpc_client, &bob_pubkey); // recipient balance
 
@@ -251,7 +271,13 @@ fn test_cli_cancel_tx() {
 
 #[test]
 fn test_offline_pay_tx() {
-    let (server, leader_data, alice, ledger_path) = new_validator_for_tests();
+    let TestValidator {
+        server,
+        leader_data,
+        alice,
+        ledger_path,
+        ..
+    } = TestValidator::run();
     let bob_pubkey = Pubkey::new_rand();
 
     let (sender, receiver) = channel();
@@ -259,22 +285,26 @@ fn test_offline_pay_tx() {
     let faucet_addr = receiver.recv().unwrap();
 
     let rpc_client = RpcClient::new_socket(leader_data.rpc);
+    let default_signer = Keypair::new();
+    let default_offline_signer = Keypair::new();
 
     let mut config_offline = CliConfig::default();
     config_offline.json_rpc_url =
         format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
+    config_offline.signers = vec![&default_offline_signer];
     let mut config_online = CliConfig::default();
     config_online.json_rpc_url =
         format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
+    config_online.signers = vec![&default_signer];
     assert_ne!(
-        config_offline.keypair.pubkey(),
-        config_online.keypair.pubkey()
+        config_offline.signers[0].pubkey(),
+        config_online.signers[0].pubkey()
     );
 
     request_and_confirm_airdrop(
         &rpc_client,
         &faucet_addr,
-        &config_offline.keypair.pubkey(),
+        &config_offline.signers[0].pubkey(),
         50,
     )
     .unwrap();
@@ -282,49 +312,42 @@ fn test_offline_pay_tx() {
     request_and_confirm_airdrop(
         &rpc_client,
         &faucet_addr,
-        &config_online.keypair.pubkey(),
+        &config_online.signers[0].pubkey(),
         50,
     )
     .unwrap();
-    check_balance(50, &rpc_client, &config_offline.keypair.pubkey());
-    check_balance(50, &rpc_client, &config_online.keypair.pubkey());
+    check_balance(50, &rpc_client, &config_offline.signers[0].pubkey());
+    check_balance(50, &rpc_client, &config_online.signers[0].pubkey());
 
+    let (blockhash, _) = rpc_client.get_recent_blockhash().unwrap();
     config_offline.command = CliCommand::Pay(PayCommand {
         lamports: 10,
         to: bob_pubkey,
+        blockhash_query: BlockhashQuery::None(blockhash, FeeCalculator::default()),
         sign_only: true,
         ..PayCommand::default()
     });
     let sig_response = process_command(&config_offline).unwrap();
 
-    check_balance(50, &rpc_client, &config_offline.keypair.pubkey());
-    check_balance(50, &rpc_client, &config_online.keypair.pubkey());
+    check_balance(50, &rpc_client, &config_offline.signers[0].pubkey());
+    check_balance(50, &rpc_client, &config_online.signers[0].pubkey());
     check_balance(0, &rpc_client, &bob_pubkey);
 
-    let object: Value = serde_json::from_str(&sig_response).unwrap();
-    let blockhash_str = object.get("blockhash").unwrap().as_str().unwrap();
-    let signer_strings = object.get("signers").unwrap().as_array().unwrap();
-    let signers: Vec<_> = signer_strings
-        .iter()
-        .map(|signer_string| {
-            let mut signer = signer_string.as_str().unwrap().split('=');
-            let key = Pubkey::from_str(signer.next().unwrap()).unwrap();
-            let sig = Signature::from_str(signer.next().unwrap()).unwrap();
-            (key, sig)
-        })
-        .collect();
-
+    let (blockhash, signers) = parse_sign_only_reply_string(&sig_response);
+    let offline_presigner =
+        presigner_from_pubkey_sigs(&config_offline.signers[0].pubkey(), &signers).unwrap();
+    let online_pubkey = config_online.signers[0].pubkey();
+    config_online.signers = vec![&offline_presigner];
     config_online.command = CliCommand::Pay(PayCommand {
         lamports: 10,
         to: bob_pubkey,
-        signers: Some(signers),
-        blockhash: Some(blockhash_str.parse::<Hash>().unwrap()),
+        blockhash_query: BlockhashQuery::FeeCalculator(blockhash),
         ..PayCommand::default()
     });
     process_command(&config_online).unwrap();
 
-    check_balance(40, &rpc_client, &config_offline.keypair.pubkey());
-    check_balance(50, &rpc_client, &config_online.keypair.pubkey());
+    check_balance(40, &rpc_client, &config_offline.signers[0].pubkey());
+    check_balance(50, &rpc_client, &online_pubkey);
     check_balance(10, &rpc_client, &bob_pubkey);
 
     server.close().unwrap();
@@ -335,15 +358,23 @@ fn test_offline_pay_tx() {
 fn test_nonced_pay_tx() {
     solana_logger::setup();
 
-    let (server, leader_data, alice, ledger_path) = new_validator_for_tests();
+    let TestValidator {
+        server,
+        leader_data,
+        alice,
+        ledger_path,
+        ..
+    } = TestValidator::run();
     let (sender, receiver) = channel();
     run_local_faucet(alice, sender, None);
     let faucet_addr = receiver.recv().unwrap();
 
     let rpc_client = RpcClient::new_socket(leader_data.rpc);
+    let default_signer = Keypair::new();
 
     let mut config = CliConfig::default();
     config.json_rpc_url = format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
+    config.signers = vec![&default_signer];
 
     let minimum_nonce_balance = rpc_client
         .get_minimum_balance_for_rent_exemption(NonceState::size())
@@ -352,29 +383,28 @@ fn test_nonced_pay_tx() {
     request_and_confirm_airdrop(
         &rpc_client,
         &faucet_addr,
-        &config.keypair.pubkey(),
+        &config.signers[0].pubkey(),
         50 + minimum_nonce_balance,
     )
     .unwrap();
     check_balance(
         50 + minimum_nonce_balance,
         &rpc_client,
-        &config.keypair.pubkey(),
+        &config.signers[0].pubkey(),
     );
 
     // Create nonce account
     let nonce_account = Keypair::new();
-    let (nonce_keypair_file, mut tmp_file) = make_tmp_file();
-    write_keypair(&nonce_account, tmp_file.as_file_mut()).unwrap();
     config.command = CliCommand::CreateNonceAccount {
-        nonce_account: read_keypair_file(&nonce_keypair_file).unwrap().into(),
+        nonce_account: 1,
         seed: None,
-        nonce_authority: Some(config.keypair.pubkey()),
+        nonce_authority: Some(config.signers[0].pubkey()),
         lamports: minimum_nonce_balance,
     };
+    config.signers.push(&nonce_account);
     process_command(&config).unwrap();
 
-    check_balance(50, &rpc_client, &config.keypair.pubkey());
+    check_balance(50, &rpc_client, &config.signers[0].pubkey());
     check_balance(minimum_nonce_balance, &rpc_client, &nonce_account.pubkey());
 
     // Fetch nonce hash
@@ -386,16 +416,17 @@ fn test_nonced_pay_tx() {
     };
 
     let bob_pubkey = Pubkey::new_rand();
+    config.signers = vec![&default_signer];
     config.command = CliCommand::Pay(PayCommand {
         lamports: 10,
         to: bob_pubkey,
-        blockhash: Some(nonce_hash),
+        blockhash_query: BlockhashQuery::FeeCalculator(nonce_hash),
         nonce_account: Some(nonce_account.pubkey()),
         ..PayCommand::default()
     });
     process_command(&config).expect("failed to process pay command");
 
-    check_balance(40, &rpc_client, &config.keypair.pubkey());
+    check_balance(40, &rpc_client, &config.signers[0].pubkey());
     check_balance(10, &rpc_client, &bob_pubkey);
 
     // Verify that nonce has been used

cli/tests/request_airdrop.rs

@@ -1,14 +1,19 @@
 use solana_cli::cli::{process_command, CliCommand, CliConfig};
 use solana_client::rpc_client::RpcClient;
-use solana_core::validator::new_validator_for_tests;
+use solana_core::validator::TestValidator;
 use solana_faucet::faucet::run_local_faucet;
-use solana_sdk::signature::KeypairUtil;
-use std::fs::remove_dir_all;
-use std::sync::mpsc::channel;
+use solana_sdk::signature::Keypair;
+use std::{fs::remove_dir_all, sync::mpsc::channel};
 
 #[test]
 fn test_cli_request_airdrop() {
-    let (server, leader_data, alice, ledger_path) = new_validator_for_tests();
+    let TestValidator {
+        server,
+        leader_data,
+        alice,
+        ledger_path,
+        ..
+    } = TestValidator::run();
     let (sender, receiver) = channel();
     run_local_faucet(alice, sender, None);
     let faucet_addr = receiver.recv().unwrap();
@@ -18,9 +23,11 @@ fn test_cli_request_airdrop() {
     bob_config.command = CliCommand::Airdrop {
         faucet_host: None,
         faucet_port: faucet_addr.port(),
+        pubkey: None,
         lamports: 50,
-        use_lamports_unit: true,
     };
+    let keypair = Keypair::new();
+    bob_config.signers = vec![&keypair];
 
     let sig_response = process_command(&bob_config);
     sig_response.unwrap();
@@ -28,7 +35,7 @@ fn test_cli_request_airdrop() {
     let rpc_client = RpcClient::new_socket(leader_data.rpc);
 
     let balance = rpc_client
-        .retry_get_balance(&bob_config.keypair.pubkey(), 1)
+        .retry_get_balance(&bob_config.signers[0].pubkey(), 1)
         .unwrap()
         .unwrap();
     assert_eq!(balance, 50);

cli/tests/transfer.rs

@@ -0,0 +1,216 @@
+use solana_clap_utils::keypair::presigner_from_pubkey_sigs;
+use solana_cli::{
+    cli::{process_command, request_and_confirm_airdrop, CliCommand, CliConfig},
+    offline::{parse_sign_only_reply_string, BlockhashQuery},
+};
+use solana_client::rpc_client::RpcClient;
+use solana_core::validator::{TestValidator, TestValidatorOptions};
+use solana_faucet::faucet::run_local_faucet;
+use solana_sdk::{
+    account_utils::StateMut,
+    fee_calculator::FeeCalculator,
+    nonce_state::NonceState,
+    pubkey::Pubkey,
+    signature::{keypair_from_seed, Keypair, Signer},
+};
+use std::{fs::remove_dir_all, sync::mpsc::channel, thread::sleep, time::Duration};
+
+fn check_balance(expected_balance: u64, client: &RpcClient, pubkey: &Pubkey) {
+    (0..5).for_each(|tries| {
+        let balance = client.retry_get_balance(pubkey, 1).unwrap().unwrap();
+        if balance == expected_balance {
+            return;
+        }
+        if tries == 4 {
+            assert_eq!(balance, expected_balance);
+        }
+        sleep(Duration::from_millis(500));
+    });
+}
+
+#[test]
+fn test_transfer() {
+    let TestValidator {
+        server,
+        leader_data,
+        alice: mint_keypair,
+        ledger_path,
+        ..
+    } = TestValidator::run_with_options(TestValidatorOptions {
+        fees: 1,
+        bootstrap_validator_lamports: 42_000,
+    });
+
+    let (sender, receiver) = channel();
+    run_local_faucet(mint_keypair, sender, None);
+    let faucet_addr = receiver.recv().unwrap();
+
+    let rpc_client = RpcClient::new_socket(leader_data.rpc);
+
+    let default_signer = Keypair::new();
+    let default_offline_signer = Keypair::new();
+
+    let mut config = CliConfig::default();
+    config.json_rpc_url = format!("http://{}:{}", leader_data.rpc.ip(), leader_data.rpc.port());
+    config.signers = vec![&default_signer];
+
+    let sender_pubkey = config.signers[0].pubkey();
+    let recipient_pubkey = Pubkey::new(&[1u8; 32]);
+
+    request_and_confirm_airdrop(&rpc_client, &faucet_addr, &sender_pubkey, 50_000).unwrap();
+    check_balance(50_000, &rpc_client, &sender_pubkey);
+    check_balance(0, &rpc_client, &recipient_pubkey);
+
+    // Plain ole transfer
+    config.command = CliCommand::Transfer {
+        lamports: 10,
+        to: recipient_pubkey,
+        from: 0,
+        sign_only: false,
+        blockhash_query: BlockhashQuery::All,
+        nonce_account: None,
+        nonce_authority: 0,
+        fee_payer: 0,
+    };
+    process_command(&config).unwrap();
+    check_balance(49_989, &rpc_client, &sender_pubkey);
+    check_balance(10, &rpc_client, &recipient_pubkey);
+
+    let mut offline = CliConfig::default();
+    offline.json_rpc_url = String::default();
+    offline.signers = vec![&default_offline_signer];
+    // Verify we cannot contact the cluster
+    offline.command = CliCommand::ClusterVersion;
+    process_command(&offline).unwrap_err();
+
+    let offline_pubkey = offline.signers[0].pubkey();
+    request_and_confirm_airdrop(&rpc_client, &faucet_addr, &offline_pubkey, 50).unwrap();
+    check_balance(50, &rpc_client, &offline_pubkey);
+
+    // Offline transfer
+    let (blockhash, _) = rpc_client.get_recent_blockhash().unwrap();
+    offline.command = CliCommand::Transfer {
+        lamports: 10,
+        to: recipient_pubkey,
+        from: 0,
+        sign_only: true,
+        blockhash_query: BlockhashQuery::None(blockhash, FeeCalculator::default()),
+        nonce_account: None,
+        nonce_authority: 0,
+        fee_payer: 0,
+    };
+    let sign_only_reply = process_command(&offline).unwrap();
+    let (blockhash, signers) = parse_sign_only_reply_string(&sign_only_reply);
+    let offline_presigner = presigner_from_pubkey_sigs(&offline_pubkey, &signers).unwrap();
+    config.signers = vec![&offline_presigner];
+    config.command = CliCommand::Transfer {
+        lamports: 10,
+        to: recipient_pubkey,
+        from: 0,
+        sign_only: false,
+        blockhash_query: BlockhashQuery::FeeCalculator(blockhash),
+        nonce_account: None,
+        nonce_authority: 0,
+        fee_payer: 0,
+    };
+    process_command(&config).unwrap();
+    check_balance(39, &rpc_client, &offline_pubkey);
+    check_balance(20, &rpc_client, &recipient_pubkey);
+
+    // Create nonce account
+    let nonce_account = keypair_from_seed(&[3u8; 32]).unwrap();
+    let minimum_nonce_balance = rpc_client
+        .get_minimum_balance_for_rent_exemption(NonceState::size())
+        .unwrap();
+    config.signers = vec![&default_signer, &nonce_account];
+    config.command = CliCommand::CreateNonceAccount {
+        nonce_account: 1,
+        seed: None,
+        nonce_authority: None,
+        lamports: minimum_nonce_balance,
+    };
+    process_command(&config).unwrap();
+    check_balance(49_987 - minimum_nonce_balance, &rpc_client, &sender_pubkey);
+
+    // Fetch nonce hash
+    let account = rpc_client.get_account(&nonce_account.pubkey()).unwrap();
+    let nonce_state: NonceState = account.state().unwrap();
+    let nonce_hash = match nonce_state {
+        NonceState::Initialized(_meta, hash) => hash,
+        _ => panic!("Nonce is not initialized"),
+    };
+
+    // Nonced transfer
+    config.signers = vec![&default_signer];
+    config.command = CliCommand::Transfer {
+        lamports: 10,
+        to: recipient_pubkey,
+        from: 0,
+        sign_only: false,
+        blockhash_query: BlockhashQuery::FeeCalculator(nonce_hash),
+        nonce_account: Some(nonce_account.pubkey()),
+        nonce_authority: 0,
+        fee_payer: 0,
+    };
+    process_command(&config).unwrap();
+    check_balance(49_976 - minimum_nonce_balance, &rpc_client, &sender_pubkey);
+    check_balance(30, &rpc_client, &recipient_pubkey);
+    let account = rpc_client.get_account(&nonce_account.pubkey()).unwrap();
+    let nonce_state: NonceState = account.state().unwrap();
+    let new_nonce_hash = match nonce_state {
+        NonceState::Initialized(_meta, hash) => hash,
+        _ => panic!("Nonce is not initialized"),
+    };
+    assert_ne!(nonce_hash, new_nonce_hash);
+
+    // Assign nonce authority to offline
+    config.signers = vec![&default_signer];
+    config.command = CliCommand::AuthorizeNonceAccount {
+        nonce_account: nonce_account.pubkey(),
+        nonce_authority: 0,
+        new_authority: offline_pubkey,
+    };
+    process_command(&config).unwrap();
+    check_balance(49_975 - minimum_nonce_balance, &rpc_client, &sender_pubkey);
+
+    // Fetch nonce hash
+    let account = rpc_client.get_account(&nonce_account.pubkey()).unwrap();
+    let nonce_state: NonceState = account.state().unwrap();
+    let nonce_hash = match nonce_state {
+        NonceState::Initialized(_meta, hash) => hash,
+        _ => panic!("Nonce is not initialized"),
+    };
+
+    // Offline, nonced transfer
+    offline.signers = vec![&default_offline_signer];
+    offline.command = CliCommand::Transfer {
+        lamports: 10,
+        to: recipient_pubkey,
+        from: 0,
+        sign_only: true,
+        blockhash_query: BlockhashQuery::None(nonce_hash, FeeCalculator::default()),
+        nonce_account: Some(nonce_account.pubkey()),
+        nonce_authority: 0,
+        fee_payer: 0,
+    };
+    let sign_only_reply = process_command(&offline).unwrap();
+    let (blockhash, signers) = parse_sign_only_reply_string(&sign_only_reply);
+    let offline_presigner = presigner_from_pubkey_sigs(&offline_pubkey, &signers).unwrap();
+    config.signers = vec![&offline_presigner];
+    config.command = CliCommand::Transfer {
+        lamports: 10,
+        to: recipient_pubkey,
+        from: 0,
+        sign_only: false,
+        blockhash_query: BlockhashQuery::FeeCalculator(blockhash),
+        nonce_account: Some(nonce_account.pubkey()),
+        nonce_authority: 0,
+        fee_payer: 0,
+    };
+    process_command(&config).unwrap();
+    check_balance(28, &rpc_client, &offline_pubkey);
+    check_balance(40, &rpc_client, &recipient_pubkey);
+
+    server.close().unwrap();
+    remove_dir_all(ledger_path).unwrap();
+}

client/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "solana-client"
-version = "0.23.0"
+version = "1.0.2"
 description = "Solana Client"
 authors = ["Solana Maintainers <maintainers@solana.com>"]
 repository = "https://github.com/solana-labs/solana"
@@ -13,17 +13,19 @@ bincode = "1.2.1"
 bs58 = "0.3.0"
 jsonrpc-core = "14.0.5"
 log = "0.4.8"
-rand = "0.6.5"
 rayon = "1.2.0"
 reqwest = { version = "0.10.1", default-features = false, features = ["blocking", "rustls-tls"] }
 serde = "1.0.104"
 serde_derive = "1.0.103"
-serde_json = "1.0.44"
-solana-net-utils = { path = "../net-utils", version = "0.23.0" }
-solana-sdk = { path = "../sdk", version = "0.23.0" }
+serde_json = "1.0.46"
+solana-net-utils = { path = "../net-utils", version = "1.0.2" }
+solana-sdk = { path = "../sdk", version = "1.0.2" }
+thiserror = "1.0"
+tungstenite = "0.10.1"
+url = "2.1.1"
 
 [dev-dependencies]
 assert_matches = "1.3.0"
 jsonrpc-core = "14.0.5"
-jsonrpc-http-server = "14.0.5"
-solana-logger = { path = "../logger", version = "0.23.0" }
+jsonrpc-http-server = "14.0.6"
+solana-logger = { path = "../logger", version = "1.0.2" }

client/src/client_error.rs

@@ -1,14 +1,16 @@
 use crate::rpc_request;
-use solana_sdk::transaction::TransactionError;
+use solana_sdk::{signature::SignerError, transaction::TransactionError};
 use std::{fmt, io};
+use thiserror::Error;
 
-#[derive(Debug)]
+#[derive(Error, Debug)]
 pub enum ClientError {
-    Io(io::Error),
-    Reqwest(reqwest::Error),
-    RpcError(rpc_request::RpcError),
-    SerdeJson(serde_json::error::Error),
-    TransactionError(TransactionError),
+    Io(#[from] io::Error),
+    Reqwest(#[from] reqwest::Error),
+    RpcError(#[from] rpc_request::RpcError),
+    SerdeJson(#[from] serde_json::error::Error),
+    SigningError(#[from] SignerError),
+    TransactionError(#[from] TransactionError),
 }
 
 impl fmt::Display for ClientError {
@@ -16,35 +18,3 @@ impl fmt::Display for ClientError {
         write!(f, "solana client error")
     }
 }
-
-impl std::error::Error for ClientError {}
-
-impl From<io::Error> for ClientError {
-    fn from(err: io::Error) -> ClientError {
-        ClientError::Io(err)
-    }
-}
-
-impl From<reqwest::Error> for ClientError {
-    fn from(err: reqwest::Error) -> ClientError {
-        ClientError::Reqwest(err)
-    }
-}
-
-impl From<rpc_request::RpcError> for ClientError {
-    fn from(err: rpc_request::RpcError) -> ClientError {
-        ClientError::RpcError(err)
-    }
-}
-
-impl From<serde_json::error::Error> for ClientError {
-    fn from(err: serde_json::error::Error) -> ClientError {
-        ClientError::SerdeJson(err)
-    }
-}
-
-impl From<TransactionError> for ClientError {
-    fn from(err: TransactionError) -> ClientError {
-        ClientError::TransactionError(err)
-    }
-}

client/src/lib.rs

@@ -5,6 +5,7 @@ pub mod client_error;
 mod generic_rpc_client_request;
 pub mod mock_rpc_client_request;
 pub mod perf_utils;
+pub mod pubsub_client;
 pub mod rpc_client;
 pub mod rpc_client_request;
 pub mod rpc_request;

client/src/mock_rpc_client_request.rs

@@ -60,13 +60,10 @@ impl GenericRpcClientRequest for MockRpcClientRequest {
                     Value::Null
                 }
             }
-            RpcRequest::GetBalance => {
-                let n = if self.url == "airdrop" { 0 } else { 50 };
-                serde_json::to_value(Response {
-                    context: RpcResponseContext { slot: 1 },
-                    value: Value::Number(Number::from(n)),
-                })?
-            }
+            RpcRequest::GetBalance => serde_json::to_value(Response {
+                context: RpcResponseContext { slot: 1 },
+                value: Value::Number(Number::from(50)),
+            })?,
             RpcRequest::GetRecentBlockhash => serde_json::to_value(Response {
                 context: RpcResponseContext { slot: 1 },
                 value: (

client/src/pubsub_client.rs

@@ -0,0 +1,220 @@
+use log::*;
+use serde::{de::DeserializeOwned, Deserialize};
+use serde_json::{
+    json,
+    value::Value::{Number, Object},
+    Map, Value,
+};
+use std::{
+    marker::PhantomData,
+    sync::{
+        atomic::{AtomicBool, Ordering},
+        mpsc::{channel, Receiver},
+        Arc, RwLock,
+    },
+    thread::JoinHandle,
+};
+use thiserror::Error;
+use tungstenite::{client::AutoStream, connect, Message, WebSocket};
+use url::{ParseError, Url};
+
+#[derive(Debug, Error)]
+pub enum PubsubClientError {
+    #[error("url parse error")]
+    UrlParseError(#[from] ParseError),
+
+    #[error("unable to connect to server")]
+    ConnectionError(#[from] tungstenite::Error),
+
+    #[error("json parse error")]
+    JsonParseError(#[from] serde_json::error::Error),
+
+    #[error("unexpected message format")]
+    UnexpectedMessageError,
+}
+
+#[derive(Serialize, Deserialize, PartialEq, Clone, Debug)]
+pub struct SlotInfoMessage {
+    pub parent: u64,
+    pub root: u64,
+    pub slot: u64,
+}
+
+pub struct PubsubClientSubscription<T>
+where
+    T: DeserializeOwned,
+{
+    message_type: PhantomData<T>,
+    operation: &'static str,
+    socket: Arc<RwLock<WebSocket<AutoStream>>>,
+    subscription_id: u64,
+    t_cleanup: Option<JoinHandle<()>>,
+    exit: Arc<AtomicBool>,
+}
+
+impl<T> Drop for PubsubClientSubscription<T>
+where
+    T: DeserializeOwned,
+{
+    fn drop(&mut self) {
+        self.send_unsubscribe()
+            .unwrap_or_else(|_| warn!("unable to unsubscribe from websocket"));
+        self.socket
+            .write()
+            .unwrap()
+            .close(None)
+            .unwrap_or_else(|_| warn!("unable to close websocket"));
+    }
+}
+
+impl<T> PubsubClientSubscription<T>
+where
+    T: DeserializeOwned,
+{
+    fn send_subscribe(
+        writable_socket: &Arc<RwLock<WebSocket<AutoStream>>>,
+        operation: &str,
+    ) -> Result<u64, PubsubClientError> {
+        let method = format!("{}Subscribe", operation);
+        writable_socket
+            .write()
+            .unwrap()
+            .write_message(Message::Text(
+                json!({
+                "jsonrpc":"2.0","id":1,"method":method,"params":[]
+                })
+                .to_string(),
+            ))?;
+        let message = writable_socket.write().unwrap().read_message()?;
+        Self::extract_subscription_id(message)
+    }
+
+    fn extract_subscription_id(message: Message) -> Result<u64, PubsubClientError> {
+        let message_text = &message.into_text()?;
+        let json_msg: Map<String, Value> = serde_json::from_str(message_text)?;
+
+        if let Some(Number(x)) = json_msg.get("result") {
+            if let Some(x) = x.as_u64() {
+                return Ok(x);
+            }
+        }
+
+        Err(PubsubClientError::UnexpectedMessageError)
+    }
+
+    pub fn send_unsubscribe(&self) -> Result<(), PubsubClientError> {
+        let method = format!("{}Unubscribe", self.operation);
+        self.socket
+            .write()
+            .unwrap()
+            .write_message(Message::Text(
+                json!({
+                "jsonrpc":"2.0","id":1,"method":method,"params":[self.subscription_id]
+                })
+                .to_string(),
+            ))
+            .map_err(|err| err.into())
+    }
+
+    fn read_message(
+        writable_socket: &Arc<RwLock<WebSocket<AutoStream>>>,
+    ) -> Result<T, PubsubClientError> {
+        let message = writable_socket.write().unwrap().read_message()?;
+        let message_text = &message.into_text().unwrap();
+        let json_msg: Map<String, Value> = serde_json::from_str(message_text)?;
+
+        if let Some(Object(value_1)) = json_msg.get("params") {
+            if let Some(value_2) = value_1.get("result") {
+                let x: T = serde_json::from_value::<T>(value_2.clone()).unwrap();
+                return Ok(x);
+            }
+        }
+
+        Err(PubsubClientError::UnexpectedMessageError)
+    }
+
+    pub fn shutdown(&mut self) -> std::thread::Result<()> {
+        if self.t_cleanup.is_some() {
+            info!("websocket thread - shutting down");
+            self.exit.store(true, Ordering::Relaxed);
+            let x = self.t_cleanup.take().unwrap().join();
+            info!("websocket thread - shut down.");
+            x
+        } else {
+            warn!("websocket thread - already shut down.");
+            Ok(())
+        }
+    }
+}
+
+const SLOT_OPERATION: &str = "slot";
+
+pub struct PubsubClient {}
+
+impl PubsubClient {
+    pub fn slot_subscribe(
+        url: &str,
+    ) -> Result<
+        (
+            PubsubClientSubscription<SlotInfoMessage>,
+            Receiver<SlotInfoMessage>,
+        ),
+        PubsubClientError,
+    > {
+        let url = Url::parse(url)?;
+        let (socket, _response) = connect(url)?;
+        let (sender, receiver) = channel::<SlotInfoMessage>();
+
+        let socket = Arc::new(RwLock::new(socket));
+        let socket_clone = socket.clone();
+        let exit = Arc::new(AtomicBool::new(false));
+        let exit_clone = exit.clone();
+        let subscription_id = PubsubClientSubscription::<SlotInfoMessage>::send_subscribe(
+            &socket_clone,
+            SLOT_OPERATION,
+        )
+        .unwrap();
+
+        let t_cleanup = std::thread::spawn(move || {
+            loop {
+                if exit_clone.load(Ordering::Relaxed) {
+                    break;
+                }
+
+                let message: Result<SlotInfoMessage, PubsubClientError> =
+                    PubsubClientSubscription::read_message(&socket_clone);
+
+                if let Ok(msg) = message {
+                    match sender.send(msg.clone()) {
+                        Ok(_) => (),
+                        Err(err) => {
+                            info!("receive error: {:?}", err);
+                            break;
+                        }
+                    }
+                } else {
+                    info!("receive error: {:?}", message);
+                    break;
+                }
+            }
+
+            info!("websocket - exited receive loop");
+        });
+
+        let result: PubsubClientSubscription<SlotInfoMessage> = PubsubClientSubscription {
+            message_type: PhantomData,
+            operation: SLOT_OPERATION,
+            socket,
+            subscription_id,
+            t_cleanup: Some(t_cleanup),
+            exit,
+        };
+
+        Ok((result, receiver))
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    // see core/tests/client.rs#test_slot_subscription()
+}

client/src/rpc_client.rs

@@ -22,7 +22,8 @@ use solana_sdk::{
     hash::Hash,
     inflation::Inflation,
     pubkey::Pubkey,
-    signature::{KeypairUtil, Signature},
+    signature::Signature,
+    signers::Signers,
     transaction::{self, Transaction, TransactionError},
 };
 use std::{
@@ -177,7 +178,7 @@ impl RpcClient {
         serde_json::from_value(response).map_err(|err| {
             io::Error::new(
                 io::ErrorKind::Other,
-                format!("GetVoteAccounts parse failure: {}", err),
+                format!("GetVoteAccounts parse failure: {:?}", err),
             )
         })
     }
@@ -196,7 +197,7 @@ impl RpcClient {
         serde_json::from_value(response).map_err(|err| {
             io::Error::new(
                 io::ErrorKind::Other,
-                format!("GetClusterNodes parse failure: {}", err),
+                format!("GetClusterNodes parse failure: {:?}", err),
             )
         })
     }
@@ -215,7 +216,7 @@ impl RpcClient {
         serde_json::from_value(response).map_err(|err| {
             io::Error::new(
                 io::ErrorKind::Other,
-                format!("GetConfirmedBlock parse failure: {}", err),
+                format!("GetConfirmedBlock parse failure: {:?}", err),
             )
         })
     }
@@ -242,7 +243,7 @@ impl RpcClient {
         serde_json::from_value(response).map_err(|err| {
             io::Error::new(
                 io::ErrorKind::Other,
-                format!("GetConfirmedBlocks parse failure: {}", err),
+                format!("GetConfirmedBlocks parse failure: {:?}", err),
             )
         })
     }
@@ -293,7 +294,7 @@ impl RpcClient {
         serde_json::from_value(response).map_err(|err| {
             io::Error::new(
                 io::ErrorKind::Other,
-                format!("GetEpochInfo parse failure: {}", err),
+                format!("GetEpochInfo parse failure: {:?}", err),
             )
         })
     }
@@ -324,7 +325,7 @@ impl RpcClient {
         serde_json::from_value(response).map_err(|err| {
             io::Error::new(
                 io::ErrorKind::Other,
-                format!("GetLeaderSchedule failure: {}", err),
+                format!("GetLeaderSchedule failure: {:?}", err),
             )
         })
     }
@@ -343,7 +344,7 @@ impl RpcClient {
         serde_json::from_value(response).map_err(|err| {
             io::Error::new(
                 io::ErrorKind::Other,
-                format!("GetEpochSchedule parse failure: {}", err),
+                format!("GetEpochSchedule parse failure: {:?}", err),
             )
         })
     }
@@ -381,7 +382,7 @@ impl RpcClient {
         serde_json::from_value(response).map_err(|err| {
             io::Error::new(
                 io::ErrorKind::Other,
-                format!("GetVersion parse failure: {}", err),
+                format!("GetVersion parse failure: {:?}", err),
             )
         })
     }
@@ -400,15 +401,15 @@ impl RpcClient {
         serde_json::from_value(response).map_err(|err| {
             io::Error::new(
                 io::ErrorKind::Other,
-                format!("MinimumLedgerSlot parse failure: {}", err),
+                format!("MinimumLedgerSlot parse failure: {:?}", err),
             )
         })
     }
 
-    pub fn send_and_confirm_transaction<T: KeypairUtil>(
+    pub fn send_and_confirm_transaction<T: Signers>(
         &self,
         transaction: &mut Transaction,
-        signer_keys: &[&T],
+        signer_keys: &T,
     ) -> Result<String, ClientError> {
         let mut send_retries = 20;
         loop {
@@ -456,10 +457,10 @@ impl RpcClient {
         }
     }
 
-    pub fn send_and_confirm_transactions<T: KeypairUtil>(
+    pub fn send_and_confirm_transactions<T: Signers>(
         &self,
         mut transactions: Vec<Transaction>,
-        signer_keys: &[&T],
+        signer_keys: &T,
     ) -> Result<(), Box<dyn error::Error>> {
         let mut send_retries = 5;
         loop {
@@ -516,24 +517,22 @@ impl RpcClient {
             // Re-sign any failed transactions with a new blockhash and retry
             let (blockhash, _fee_calculator) =
                 self.get_new_blockhash(&transactions_signatures[0].0.message().recent_blockhash)?;
-            transactions = transactions_signatures
-                .into_iter()
-                .map(|(mut transaction, _)| {
-                    transaction.sign(signer_keys, blockhash);
-                    transaction
-                })
-                .collect();
+            transactions = vec![];
+            for (mut transaction, _) in transactions_signatures.into_iter() {
+                transaction.try_sign(signer_keys, blockhash)?;
+                transactions.push(transaction);
+            }
         }
     }
 
-    pub fn resign_transaction<T: KeypairUtil>(
+    pub fn resign_transaction<T: Signers>(
         &self,
         tx: &mut Transaction,
-        signer_keys: &[&T],
+        signer_keys: &T,
     ) -> Result<(), ClientError> {
         let (blockhash, _fee_calculator) =
             self.get_new_blockhash(&tx.message().recent_blockhash)?;
-        tx.sign(signer_keys, blockhash);
+        tx.try_sign(signer_keys, blockhash)?;
         Ok(())
     }
 
@@ -612,7 +611,7 @@ impl RpcClient {
             .map_err(|err| {
                 io::Error::new(
                     io::ErrorKind::Other,
-                    format!("AccountNotFound: pubkey={}: {}", pubkey, err),
+                    format!("AccountNotFound: pubkey={}: {:?}", pubkey, err),
                 )
             })?
     }
@@ -698,7 +697,7 @@ impl RpcClient {
             .map_err(|err| {
                 io::Error::new(
                     io::ErrorKind::Other,
-                    format!("AccountNotFound: pubkey={}: {}", pubkey, err),
+                    format!("AccountNotFound: pubkey={}: {:?}", pubkey, err),
                 )
             })?;
 
@@ -749,7 +748,7 @@ impl RpcClient {
         serde_json::from_value(response).map_err(|err| {
             io::Error::new(
                 io::ErrorKind::Other,
-                format!("GetTransactionCount parse failure: {}", err),
+                format!("GetTransactionCount parse failure: {:?}", err),
             )
         })
     }
@@ -1129,9 +1128,7 @@ mod tests {
     use serde_json::Number;
     use solana_logger;
     use solana_sdk::{
-        instruction::InstructionError,
-        signature::{Keypair, KeypairUtil},
-        system_transaction,
+        instruction::InstructionError, signature::Keypair, system_transaction,
         transaction::TransactionError,
     };
     use std::{sync::mpsc::channel, thread};

client/src/rpc_client_request.rs

@@ -15,10 +15,7 @@ pub struct RpcClientRequest {
 
 impl RpcClientRequest {
     pub fn new(url: String) -> Self {
-        Self {
-            client: reqwest::blocking::Client::new(),
-            url,
-        }
+        Self::new_with_timeout(url, Duration::from_secs(20))
     }
 
     pub fn new_with_timeout(url: String, timeout: Duration) -> Self {

client/src/rpc_response.rs

@@ -32,6 +32,14 @@ pub struct RpcBlockCommitment<T> {
     pub total_stake: u64,
 }
 
+#[derive(Debug, PartialEq, Serialize, Deserialize)]
+pub struct RpcReward {
+    pub pubkey: String,
+    pub lamports: i64,
+}
+
+pub type RpcRewards = Vec<RpcReward>;
+
 #[derive(Debug, PartialEq, Serialize, Deserialize)]
 #[serde(rename_all = "camelCase")]
 pub struct RpcConfirmedBlock {
@@ -39,6 +47,7 @@ pub struct RpcConfirmedBlock {
     pub blockhash: String,
     pub parent_slot: Slot,
     pub transactions: Vec<RpcTransactionWithStatusMeta>,
+    pub rewards: RpcRewards,
 }
 
 #[derive(Debug, PartialEq, Serialize, Deserialize)]

client/src/thin_client.rs

@@ -17,7 +17,8 @@ use solana_sdk::{
     message::Message,
     packet::PACKET_DATA_SIZE,
     pubkey::Pubkey,
-    signature::{Keypair, KeypairUtil, Signature},
+    signature::{Keypair, Signature, Signer},
+    signers::Signers,
     system_instruction,
     timing::duration_as_ms,
     transaction::{self, Transaction},
@@ -202,9 +203,9 @@ impl ThinClient {
     }
 
     /// Retry sending a signed Transaction to the server for processing
-    pub fn send_and_confirm_transaction(
+    pub fn send_and_confirm_transaction<T: Signers>(
         &self,
-        keypairs: &[&Keypair],
+        keypairs: &T,
         transaction: &mut Transaction,
         tries: usize,
         pending_confirmations: usize,
@@ -351,9 +352,13 @@ impl Client for ThinClient {
 }
 
 impl SyncClient for ThinClient {
-    fn send_message(&self, keypairs: &[&Keypair], message: Message) -> TransportResult<Signature> {
+    fn send_message<T: Signers>(
+        &self,
+        keypairs: &T,
+        message: Message,
+    ) -> TransportResult<Signature> {
         let (blockhash, _fee_calculator) = self.get_recent_blockhash()?;
-        let mut transaction = Transaction::new(&keypairs, message, blockhash);
+        let mut transaction = Transaction::new(keypairs, message, blockhash);
         let signature = self.send_and_confirm_transaction(keypairs, &mut transaction, 5, 0)?;
         Ok(signature)
     }
@@ -561,13 +566,13 @@ impl AsyncClient for ThinClient {
             .send_to(&buf[..], &self.tpu_addr())?;
         Ok(transaction.signatures[0])
     }
-    fn async_send_message(
+    fn async_send_message<T: Signers>(
         &self,
-        keypairs: &[&Keypair],
+        keypairs: &T,
         message: Message,
         recent_blockhash: Hash,
     ) -> io::Result<Signature> {
-        let transaction = Transaction::new(&keypairs, message, recent_blockhash);
+        let transaction = Transaction::new(keypairs, message, recent_blockhash);
         self.async_send_transaction(transaction)
     }
     fn async_send_instruction(

core/Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 name = "solana-core"
 description = "Blockchain, Rebuilt for Scale"
-version = "0.23.0"
+version = "1.0.2"
 documentation = "https://docs.rs/solana"
 homepage = "https://solana.com/"
 readme = "../README.md"
@@ -18,65 +18,64 @@ bincode = "1.2.1"
 bs58 = "0.3.0"
 byteorder = "1.3.2"
 chrono = { version = "0.4.10", features = ["serde"] }
+compression = "0.1.5"
 core_affinity = "0.5.10"
-crc = { version = "1.8.1", optional = true }
 crossbeam-channel = "0.3"
 fs_extra = "1.1.0"
 indexmap = "1.3"
 itertools = "0.8.2"
 jsonrpc-core = "14.0.5"
+jsonrpc-core-client = { version = "14.0.5", features = ["ws"] }
 jsonrpc-derive = "14.0.5"
-jsonrpc-http-server = "14.0.5"
-jsonrpc-pubsub = "14.0.5"
-jsonrpc-ws-server = "14.0.5"
+jsonrpc-http-server = "14.0.6"
+jsonrpc-pubsub = "14.0.6"
+jsonrpc-ws-server = "14.0.6"
 libc = "0.2.66"
 log = "0.4.8"
-memmap = { version = "0.7.0", optional = true }
-nix = "0.16.1"
+nix = "0.17.0"
 num-traits = "0.2"
 rand = "0.6.5"
 rand_chacha = "0.1.1"
 rayon = "1.2.0"
+regex = "1.3.4"
 serde = "1.0.104"
 serde_derive = "1.0.103"
-serde_json = "1.0.44"
-solana-budget-program = { path = "../programs/budget", version = "0.23.0" }
-solana-clap-utils = { path = "../clap-utils", version = "0.23.0" }
-solana-client = { path = "../client", version = "0.23.0" }
-solana-faucet = { path = "../faucet", version = "0.23.0" }
+serde_json = "1.0.46"
+solana-budget-program = { path = "../programs/budget", version = "1.0.2" }
+solana-clap-utils = { path = "../clap-utils", version = "1.0.2" }
+solana-client = { path = "../client", version = "1.0.2" }
+solana-faucet = { path = "../faucet", version = "1.0.2" }
 ed25519-dalek = "=1.0.0-pre.1"
-solana-ledger = { path = "../ledger", version = "0.23.0" }
-solana-logger = { path = "../logger", version = "0.23.0" }
-solana-merkle-tree = { path = "../merkle-tree", version = "0.23.0" }
-solana-metrics = { path = "../metrics", version = "0.23.0" }
-solana-measure = { path = "../measure", version = "0.23.0" }
-solana-net-utils = { path = "../net-utils", version = "0.23.0" }
-solana-chacha-cuda = { path = "../chacha-cuda", version = "0.23.0" }
-solana-perf = { path = "../perf", version = "0.23.0" }
-solana-runtime = { path = "../runtime", version = "0.23.0" }
-solana-sdk = { path = "../sdk", version = "0.23.0" }
-solana-stake-program = { path = "../programs/stake", version = "0.23.0" }
-solana-storage-program = { path = "../programs/storage", version = "0.23.0" }
-solana-vote-program = { path = "../programs/vote", version = "0.23.0" }
-solana-vote-signer = { path = "../vote-signer", version = "0.23.0" }
-solana-sys-tuner = { path = "../sys-tuner", version = "0.23.0" }
-symlink = "0.1.0"
-sys-info = "0.5.8"
+solana-ledger = { path = "../ledger", version = "1.0.2" }
+solana-logger = { path = "../logger", version = "1.0.2" }
+solana-merkle-tree = { path = "../merkle-tree", version = "1.0.2" }
+solana-metrics = { path = "../metrics", version = "1.0.2" }
+solana-measure = { path = "../measure", version = "1.0.2" }
+solana-net-utils = { path = "../net-utils", version = "1.0.2" }
+solana-chacha-cuda = { path = "../chacha-cuda", version = "1.0.2" }
+solana-perf = { path = "../perf", version = "1.0.2" }
+solana-runtime = { path = "../runtime", version = "1.0.2" }
+solana-sdk = { path = "../sdk", version = "1.0.2" }
+solana-stake-program = { path = "../programs/stake", version = "1.0.2" }
+solana-storage-program = { path = "../programs/storage", version = "1.0.2" }
+solana-vote-program = { path = "../programs/vote", version = "1.0.2" }
+solana-vote-signer = { path = "../vote-signer", version = "1.0.2" }
+solana-sys-tuner = { path = "../sys-tuner", version = "1.0.2" }
+sys-info = "0.5.9"
 tempfile = "3.1.0"
 thiserror = "1.0"
 tokio = "0.1"
 tokio-codec = "0.1"
 tokio-fs = "0.1"
 tokio-io = "0.1"
-untrusted = "0.7.0"
-solana-rayon-threadlimit = { path = "../rayon-threadlimit", version = "0.23.0" }
-reed-solomon-erasure = { package = "solana-reed-solomon-erasure", version = "4.0.1-3", features = ["simd-accel"] }
+solana-rayon-threadlimit = { path = "../rayon-threadlimit", version = "1.0.2" }
+trees = "0.2.1"
 
 [dev-dependencies]
 matches = "0.1.6"
 reqwest = { version = "0.10.1", default-features = false, features = ["blocking", "rustls-tls"] }
 serial_test = "0.3.2"
-serial_test_derive = "0.3.1"
+serial_test_derive = "0.4.0"
 systemstat = "0.1.5"
 
 [[bench]]

core/benches/banking_stage.rs

@@ -21,8 +21,8 @@ use solana_sdk::genesis_config::GenesisConfig;
 use solana_sdk::hash::Hash;
 use solana_sdk::pubkey::Pubkey;
 use solana_sdk::signature::Keypair;
-use solana_sdk::signature::KeypairUtil;
 use solana_sdk::signature::Signature;
+use solana_sdk::signature::Signer;
 use solana_sdk::system_instruction;
 use solana_sdk::system_transaction;
 use solana_sdk::timing::{duration_as_us, timestamp};

core/benches/poh_verify.rs

@@ -3,7 +3,7 @@ extern crate test;
 
 use solana_ledger::entry::{next_entry_mut, Entry, EntrySlice};
 use solana_sdk::hash::{hash, Hash};
-use solana_sdk::signature::{Keypair, KeypairUtil};
+use solana_sdk::signature::{Keypair, Signer};
 use solana_sdk::system_transaction;
 use test::Bencher;
 

core/benches/shredder.rs

@@ -9,7 +9,7 @@ use solana_ledger::shred::{
 };
 use solana_perf::test_tx;
 use solana_sdk::hash::Hash;
-use solana_sdk::signature::{Keypair, KeypairUtil};
+use solana_sdk::signature::{Keypair, Signer};
 use std::sync::Arc;
 use test::Bencher;
 

core/benches/sigverify_stage.rs

@@ -11,7 +11,7 @@ use solana_core::sigverify::TransactionSigVerifier;
 use solana_core::sigverify_stage::SigVerifyStage;
 use solana_perf::test_tx::test_tx;
 use solana_sdk::hash::Hash;
-use solana_sdk::signature::{Keypair, KeypairUtil};
+use solana_sdk::signature::{Keypair, Signer};
 use solana_sdk::system_transaction;
 use solana_sdk::timing::duration_as_ms;
 use std::sync::mpsc::channel;

core/src/banking_stage.rs

@@ -404,7 +404,7 @@ impl BankingStage {
                     if unprocessed_packets.is_empty() {
                         continue;
                     }
-                    let num = unprocessed_packets
+                    let num: usize = unprocessed_packets
                         .iter()
                         .map(|(_, unprocessed)| unprocessed.len())
                         .sum();
@@ -660,10 +660,7 @@ impl BankingStage {
         transactions
             .into_iter()
             .zip(indexes)
-            .filter_map(|(tx, index)| match tx {
-                None => None,
-                Some(tx) => Some((tx, index)),
-            })
+            .filter_map(|(tx, index)| tx.map(|tx| (tx, index)))
             .unzip()
     }
 
@@ -1029,7 +1026,7 @@ mod tests {
     use solana_runtime::bank::HashAgeKind;
     use solana_sdk::{
         instruction::InstructionError,
-        signature::{Keypair, KeypairUtil},
+        signature::{Keypair, Signer},
         system_transaction,
         transaction::TransactionError,
     };

core/src/blockstream.rs

@@ -180,7 +180,7 @@ mod test {
     use chrono::{DateTime, FixedOffset};
     use serde_json::Value;
     use solana_sdk::hash::Hash;
-    use solana_sdk::signature::{Keypair, KeypairUtil};
+    use solana_sdk::signature::{Keypair, Signer};
     use solana_sdk::system_transaction;
     use std::collections::HashSet;
     use std::path::PathBuf;

core/src/blockstream_service.rs

@@ -58,31 +58,35 @@ impl BlockstreamService {
         let timeout = Duration::new(1, 0);
         let (slot, slot_leader) = slot_full_receiver.recv_timeout(timeout)?;
 
-        let entries = blockstore.get_slot_entries(slot, 0, None).unwrap();
-        let blockstore_meta = blockstore.meta(slot).unwrap().unwrap();
-        let _parent_slot = if slot == 0 {
-            None
-        } else {
-            Some(blockstore_meta.parent_slot)
-        };
-        let ticks_per_slot = entries.iter().filter(|entry| entry.is_tick()).count() as u64;
-        let mut tick_height = ticks_per_slot * slot;
+        // Slot might not exist due to LedgerCleanupService, check first
+        let blockstore_meta = blockstore.meta(slot).unwrap();
+        if let Some(blockstore_meta) = blockstore_meta {
+            // Return error to main loop. Thread won't exit, will just log the error
+            let entries = blockstore.get_slot_entries(slot, 0, None)?;
+            let _parent_slot = if slot == 0 {
+                None
+            } else {
+                Some(blockstore_meta.parent_slot)
+            };
+            let ticks_per_slot = entries.iter().filter(|entry| entry.is_tick()).count() as u64;
+            let mut tick_height = ticks_per_slot * slot;
 
-        for (i, entry) in entries.iter().enumerate() {
-            if entry.is_tick() {
-                tick_height += 1;
-            }
-            blockstream
-                .emit_entry_event(slot, tick_height, &slot_leader, &entry)
-                .unwrap_or_else(|e| {
-                    debug!("Blockstream error: {:?}, {:?}", e, blockstream.output);
-                });
-            if i == entries.len() - 1 {
+            for (i, entry) in entries.iter().enumerate() {
+                if entry.is_tick() {
+                    tick_height += 1;
+                }
                 blockstream
-                    .emit_block_event(slot, tick_height, &slot_leader, entry.hash)
+                    .emit_entry_event(slot, tick_height, &slot_leader, &entry)
                     .unwrap_or_else(|e| {
                         debug!("Blockstream error: {:?}, {:?}", e, blockstream.output);
                     });
+                if i == entries.len() - 1 {
+                    blockstream
+                        .emit_block_event(slot, tick_height, &slot_leader, entry.hash)
+                        .unwrap_or_else(|e| {
+                            debug!("Blockstream error: {:?}, {:?}", e, blockstream.output);
+                        });
+                }
             }
         }
         Ok(())
@@ -103,7 +107,7 @@ mod test {
     use solana_ledger::create_new_tmp_ledger;
     use solana_ledger::entry::{create_ticks, Entry};
     use solana_sdk::hash::Hash;
-    use solana_sdk::signature::{Keypair, KeypairUtil};
+    use solana_sdk::signature::{Keypair, Signer};
     use solana_sdk::system_transaction;
     use std::path::PathBuf;
     use std::sync::mpsc::channel;

core/src/broadcast_stage.rs

@@ -259,7 +259,7 @@ mod test {
     use solana_runtime::bank::Bank;
     use solana_sdk::hash::Hash;
     use solana_sdk::pubkey::Pubkey;
-    use solana_sdk::signature::{Keypair, KeypairUtil};
+    use solana_sdk::signature::{Keypair, Signer};
     use std::path::Path;
     use std::sync::atomic::AtomicBool;
     use std::sync::mpsc::channel;

core/src/broadcast_stage/fail_entry_verification_broadcast_run.rs

@@ -82,7 +82,7 @@ impl BroadcastRun for FailEntryVerificationBroadcastRun {
         // Broadcast data
         let all_shred_bufs: Vec<Vec<u8>> = shreds.to_vec().into_iter().map(|s| s.payload).collect();
         cluster_info
-            .read()
+            .write()
             .unwrap()
             .broadcast_shreds(sock, all_shred_bufs, &all_seeds, stakes)?;
         Ok(())

core/src/broadcast_stage/standard_broadcast_run.rs

@@ -264,7 +264,7 @@ impl StandardBroadcastRun {
         trace!("Broadcasting {:?} shreds", shred_bufs.len());
 
         cluster_info
-            .read()
+            .write()
             .unwrap()
             .broadcast_shreds(sock, shred_bufs, &seeds, stakes)?;
 
@@ -362,7 +362,7 @@ mod test {
     use solana_sdk::{
         clock::Slot,
         genesis_config::GenesisConfig,
-        signature::{Keypair, KeypairUtil},
+        signature::{Keypair, Signer},
     };
     use std::sync::{Arc, RwLock};
     use std::time::Duration;

core/src/cluster_info_vote_listener.rs

@@ -84,7 +84,7 @@ impl ClusterInfoVoteListener {
 mod tests {
     use crate::packet;
     use solana_sdk::hash::Hash;
-    use solana_sdk::signature::{Keypair, KeypairUtil};
+    use solana_sdk::signature::{Keypair, Signer};
     use solana_sdk::transaction::Transaction;
     use solana_vote_program::vote_instruction;
     use solana_vote_program::vote_state::Vote;

core/src/commitment.rs

@@ -233,7 +233,7 @@ mod tests {
     use crate::genesis_utils::{create_genesis_config, GenesisConfigInfo};
     use solana_sdk::pubkey::Pubkey;
     use solana_stake_program::stake_state;
-    use solana_vote_program::vote_state;
+    use solana_vote_program::vote_state::{self, VoteStateVersions};
 
     #[test]
     fn test_block_commitment() {
@@ -446,13 +446,15 @@ mod tests {
         let mut vote_state1 = VoteState::from(&vote_account1).unwrap();
         vote_state1.process_slot_vote_unchecked(3);
         vote_state1.process_slot_vote_unchecked(5);
-        vote_state1.to(&mut vote_account1).unwrap();
+        let versioned = VoteStateVersions::Current(Box::new(vote_state1));
+        VoteState::to(&versioned, &mut vote_account1).unwrap();
         bank.store_account(&pk1, &vote_account1);
 
         let mut vote_state2 = VoteState::from(&vote_account2).unwrap();
         vote_state2.process_slot_vote_unchecked(9);
         vote_state2.process_slot_vote_unchecked(10);
-        vote_state2.to(&mut vote_account2).unwrap();
+        let versioned = VoteStateVersions::Current(Box::new(vote_state2));
+        VoteState::to(&versioned, &mut vote_account2).unwrap();
         bank.store_account(&pk2, &vote_account2);
 
         let commitment = AggregateCommitmentService::aggregate_commitment(&ancestors, &bank);

core/src/consensus.rs

@@ -1,6 +1,5 @@
 use chrono::prelude::*;
 use solana_ledger::bank_forks::BankForks;
-use solana_metrics::datapoint_debug;
 use solana_runtime::bank::Bank;
 use solana_sdk::{
     account::Account,
@@ -37,7 +36,6 @@ impl StakeLockout {
     }
 }
 
-#[derive(Default)]
 pub struct Tower {
     node_pubkey: Pubkey,
     threshold_depth: usize,
@@ -47,22 +45,35 @@ pub struct Tower {
     last_timestamp: BlockTimestamp,
 }
 
-impl Tower {
-    pub fn new(node_pubkey: &Pubkey, vote_account_pubkey: &Pubkey, bank_forks: &BankForks) -> Self {
-        let mut tower = Self {
-            node_pubkey: *node_pubkey,
+impl Default for Tower {
+    fn default() -> Self {
+        Self {
+            node_pubkey: Pubkey::default(),
             threshold_depth: VOTE_THRESHOLD_DEPTH,
             threshold_size: VOTE_THRESHOLD_SIZE,
             lockouts: VoteState::default(),
             last_vote: Vote::default(),
             last_timestamp: BlockTimestamp::default(),
-        };
+        }
+    }
+}
+
+impl Tower {
+    pub fn new(node_pubkey: &Pubkey, vote_account_pubkey: &Pubkey, bank_forks: &BankForks) -> Self {
+        let mut tower = Self::new_with_key(node_pubkey);
 
         tower.initialize_lockouts_from_bank_forks(&bank_forks, vote_account_pubkey);
 
         tower
     }
 
+    pub fn new_with_key(node_pubkey: &Pubkey) -> Self {
+        Self {
+            node_pubkey: *node_pubkey,
+            ..Tower::default()
+        }
+    }
+
     #[cfg(test)]
     pub fn new_for_tests(threshold_depth: usize, threshold_size: f64) -> Self {
         Self {
@@ -110,7 +121,7 @@ impl Tower {
                     vote_state.nth_recent_vote(0).map(|v| v.slot).unwrap_or(0) as i64
                 );
                 debug!("observed root {}", vote_state.root_slot.unwrap_or(0) as i64);
-                datapoint_debug!(
+                datapoint_info!(
                     "tower-observed",
                     (
                         "slot",
@@ -229,7 +240,7 @@ impl Tower {
         self.lockouts.process_vote_unchecked(&vote);
         self.last_vote = vote;
 
-        datapoint_debug!(
+        datapoint_info!(
             "tower-vote",
             ("latest", slot, i64),
             ("root", self.lockouts.root_slot.unwrap_or(0), i64)
@@ -284,7 +295,6 @@ impl Tower {
         assert!(ancestors.contains_key(&slot));
 
         if !self.is_recent(slot) {
-            trace!("slot is not recent: {}", slot);
             return true;
         }
 
@@ -321,12 +331,21 @@ impl Tower {
             if let Some(fork_stake) = stake_lockouts.get(&vote.slot) {
                 let lockout = fork_stake.stake as f64 / total_staked as f64;
                 trace!(
-                    "fork_stake {} {} {} {}",
+                    "fork_stake slot: {} lockout: {} fork_stake: {} total_stake: {}",
                     slot,
                     lockout,
                     fork_stake.stake,
                     total_staked
                 );
+                if vote.confirmation_count as usize > self.threshold_depth {
+                    for old_vote in &self.lockouts.votes {
+                        if old_vote.slot == vote.slot
+                            && old_vote.confirmation_count == vote.confirmation_count
+                        {
+                            return true;
+                        }
+                    }
+                }
                 lockout > self.threshold_size
             } else {
                 false
@@ -447,9 +466,239 @@ impl Tower {
 }
 
 #[cfg(test)]
-mod test {
+pub mod test {
     use super::*;
+    use crate::replay_stage::{ForkProgress, ReplayStage};
+    use solana_ledger::bank_forks::BankForks;
+    use solana_runtime::{
+        bank::Bank,
+        genesis_utils::{
+            create_genesis_config_with_vote_accounts, GenesisConfigInfo, ValidatorVoteKeypairs,
+        },
+    };
+    use solana_sdk::{
+        clock::Slot,
+        hash::Hash,
+        pubkey::Pubkey,
+        signature::{Keypair, Signer},
+        transaction::Transaction,
+    };
+    use solana_vote_program::{
+        vote_instruction,
+        vote_state::{Vote, VoteStateVersions},
+    };
+    use std::collections::{HashMap, VecDeque};
+    use std::sync::RwLock;
     use std::{thread::sleep, time::Duration};
+    use trees::{tr, Node, Tree};
+
+    pub(crate) struct VoteSimulator<'a> {
+        searchable_nodes: HashMap<u64, &'a Node<u64>>,
+    }
+
+    impl<'a> VoteSimulator<'a> {
+        pub(crate) fn new(forks: &'a Tree<u64>) -> Self {
+            let mut searchable_nodes = HashMap::new();
+            let root = forks.root();
+            searchable_nodes.insert(root.data, root);
+            Self { searchable_nodes }
+        }
+
+        pub(crate) fn simulate_vote(
+            &mut self,
+            vote_slot: Slot,
+            bank_forks: &RwLock<BankForks>,
+            cluster_votes: &mut HashMap<Pubkey, Vec<u64>>,
+            validator_keypairs: &HashMap<Pubkey, ValidatorVoteKeypairs>,
+            my_keypairs: &ValidatorVoteKeypairs,
+            progress: &mut HashMap<u64, ForkProgress>,
+            tower: &mut Tower,
+        ) -> Vec<VoteFailures> {
+            let node = self
+                .find_node_and_update_simulation(vote_slot)
+                .expect("Vote to simulate must be for a slot in the tree");
+
+            let mut missing_nodes = VecDeque::new();
+            let mut current = node;
+            loop {
+                let current_slot = current.data;
+                if bank_forks.read().unwrap().get(current_slot).is_some()
+                    || tower.root().map(|r| current_slot < r).unwrap_or(false)
+                {
+                    break;
+                } else {
+                    missing_nodes.push_front(current);
+                }
+
+                if let Some(parent) = current.parent() {
+                    current = parent;
+                } else {
+                    break;
+                }
+            }
+
+            // Create any missing banks along the path
+            for missing_node in missing_nodes {
+                let missing_slot = missing_node.data;
+                let parent = missing_node.parent().unwrap().data;
+                let parent_bank = bank_forks
+                    .read()
+                    .unwrap()
+                    .get(parent)
+                    .expect("parent bank must exist")
+                    .clone();
+                info!("parent of {} is {}", missing_slot, parent_bank.slot(),);
+                progress
+                    .entry(missing_slot)
+                    .or_insert_with(|| ForkProgress::new(parent_bank.last_blockhash()));
+
+                // Create the missing bank
+                let new_bank =
+                    Bank::new_from_parent(&parent_bank, &Pubkey::default(), missing_slot);
+
+                // Simulate ingesting the cluster's votes for the parent into this bank
+                for (pubkey, vote) in cluster_votes.iter() {
+                    if vote.contains(&parent_bank.slot()) {
+                        let keypairs = validator_keypairs.get(pubkey).unwrap();
+                        let node_pubkey = keypairs.node_keypair.pubkey();
+                        let vote_pubkey = keypairs.vote_keypair.pubkey();
+                        let last_blockhash = parent_bank.last_blockhash();
+                        let votes = Vote::new(vec![parent_bank.slot()], parent_bank.hash());
+                        info!("voting {} {}", parent_bank.slot(), parent_bank.hash());
+                        let vote_ix = vote_instruction::vote(&vote_pubkey, &vote_pubkey, votes);
+                        let mut vote_tx =
+                            Transaction::new_with_payer(vec![vote_ix], Some(&node_pubkey));
+                        vote_tx.partial_sign(&[&keypairs.node_keypair], last_blockhash);
+                        vote_tx.partial_sign(&[&keypairs.vote_keypair], last_blockhash);
+                        new_bank.process_transaction(&vote_tx).unwrap();
+                    }
+                }
+                new_bank.freeze();
+                bank_forks.write().unwrap().insert(new_bank);
+            }
+
+            // Now try to simulate the vote
+            let my_pubkey = my_keypairs.node_keypair.pubkey();
+            let my_vote_pubkey = my_keypairs.vote_keypair.pubkey();
+            let ancestors = bank_forks.read().unwrap().ancestors();
+            let mut frozen_banks: Vec<_> = bank_forks
+                .read()
+                .unwrap()
+                .frozen_banks()
+                .values()
+                .cloned()
+                .collect();
+
+            ReplayStage::compute_bank_stats(
+                &my_pubkey,
+                &ancestors,
+                &mut frozen_banks,
+                tower,
+                progress,
+            );
+
+            let bank = bank_forks
+                .read()
+                .unwrap()
+                .get(vote_slot)
+                .expect("Bank must have been created before vote simulation")
+                .clone();
+            // Make sure this slot isn't locked out or failing threshold
+            let fork_progress = progress
+                .get(&vote_slot)
+                .expect("Slot for vote must exist in progress map");
+            info!("Checking vote: {}", vote_slot);
+            info!("lockouts: {:?}", fork_progress.fork_stats.stake_lockouts);
+            let mut failures = vec![];
+            if fork_progress.fork_stats.is_locked_out {
+                failures.push(VoteFailures::LockedOut(vote_slot));
+            }
+            if !fork_progress.fork_stats.vote_threshold {
+                failures.push(VoteFailures::FailedThreshold(vote_slot));
+            }
+            if !failures.is_empty() {
+                return failures;
+            }
+            let vote = tower.new_vote_from_bank(&bank, &my_vote_pubkey).0;
+            if let Some(new_root) = tower.record_bank_vote(vote) {
+                ReplayStage::handle_new_root(new_root, bank_forks, progress, &None);
+            }
+
+            // Mark the vote for this bank under this node's pubkey so it will be
+            // integrated into any future child banks
+            cluster_votes.entry(my_pubkey).or_default().push(vote_slot);
+            vec![]
+        }
+
+        // Find a node representing the given slot
+        fn find_node_and_update_simulation(&mut self, slot: u64) -> Option<&'a Node<u64>> {
+            let mut successful_search_node: Option<&'a Node<u64>> = None;
+            let mut found_node = None;
+            for search_node in self.searchable_nodes.values() {
+                if let Some((target, new_searchable_nodes)) = Self::find_node(search_node, slot) {
+                    successful_search_node = Some(search_node);
+                    found_node = Some(target);
+                    for node in new_searchable_nodes {
+                        self.searchable_nodes.insert(node.data, node);
+                    }
+                    break;
+                }
+            }
+            successful_search_node.map(|node| {
+                self.searchable_nodes.remove(&node.data);
+            });
+            found_node
+        }
+
+        fn find_node(
+            node: &'a Node<u64>,
+            slot: u64,
+        ) -> Option<(&'a Node<u64>, Vec<&'a Node<u64>>)> {
+            if node.data == slot {
+                Some((node, node.iter().collect()))
+            } else {
+                let mut search_result: Option<(&'a Node<u64>, Vec<&'a Node<u64>>)> = None;
+                for child in node.iter() {
+                    if let Some((_, ref mut new_searchable_nodes)) = search_result {
+                        new_searchable_nodes.push(child);
+                        continue;
+                    }
+                    search_result = Self::find_node(child, slot);
+                }
+
+                search_result
+            }
+        }
+    }
+
+    #[derive(PartialEq, Debug)]
+    pub(crate) enum VoteFailures {
+        LockedOut(u64),
+        FailedThreshold(u64),
+    }
+
+    // Setup BankForks with bank 0 and all the validator accounts
+    pub(crate) fn initialize_state(
+        validator_keypairs_map: &HashMap<Pubkey, ValidatorVoteKeypairs>,
+    ) -> (BankForks, HashMap<u64, ForkProgress>) {
+        let validator_keypairs: Vec<_> = validator_keypairs_map.values().collect();
+        let GenesisConfigInfo {
+            genesis_config,
+            mint_keypair,
+            voting_keypair: _,
+        } = create_genesis_config_with_vote_accounts(1_000_000_000, &validator_keypairs);
+
+        let bank0 = Bank::new(&genesis_config);
+
+        for pubkey in validator_keypairs_map.keys() {
+            bank0.transfer(10_000, &mint_keypair, pubkey).unwrap();
+        }
+
+        bank0.freeze();
+        let mut progress = HashMap::new();
+        progress.insert(0, ForkProgress::new(bank0.last_blockhash()));
+        (BankForks::new(0, bank0), progress)
+    }
 
     fn gen_stakes(stake_votes: &[(u64, &[u64])]) -> Vec<(Pubkey, (u64, Account))> {
         let mut stakes = vec![];
@@ -461,14 +710,207 @@ mod test {
             for slot in *votes {
                 vote_state.process_slot_vote_unchecked(*slot);
             }
-            vote_state
-                .serialize(&mut account.data)
-                .expect("serialize state");
+            VoteState::serialize(
+                &VoteStateVersions::Current(Box::new(vote_state)),
+                &mut account.data,
+            )
+            .expect("serialize state");
             stakes.push((Pubkey::new_rand(), (*lamports, account)));
         }
         stakes
     }
 
+    fn can_progress_on_fork(
+        my_pubkey: &Pubkey,
+        tower: &mut Tower,
+        start_slot: u64,
+        num_slots: u64,
+        bank_forks: &RwLock<BankForks>,
+        cluster_votes: &mut HashMap<Pubkey, Vec<u64>>,
+        keypairs: &HashMap<Pubkey, ValidatorVoteKeypairs>,
+        progress: &mut HashMap<u64, ForkProgress>,
+    ) -> bool {
+        // Check that within some reasonable time, validator can make a new
+        // root on this fork
+        let old_root = tower.root();
+        let mut main_fork = tr(start_slot);
+        let mut tip = main_fork.root_mut();
+
+        for i in 1..num_slots {
+            tip.push_front(tr(start_slot + i));
+            tip = tip.first_mut().unwrap();
+        }
+        let mut voting_simulator = VoteSimulator::new(&main_fork);
+        for i in 1..num_slots {
+            voting_simulator.simulate_vote(
+                i + start_slot,
+                &bank_forks,
+                cluster_votes,
+                &keypairs,
+                keypairs.get(&my_pubkey).unwrap(),
+                progress,
+                tower,
+            );
+            if old_root != tower.root() {
+                return true;
+            }
+        }
+
+        false
+    }
+
+    #[test]
+    fn test_simple_votes() {
+        let node_keypair = Keypair::new();
+        let vote_keypair = Keypair::new();
+        let stake_keypair = Keypair::new();
+        let node_pubkey = node_keypair.pubkey();
+
+        let mut keypairs = HashMap::new();
+        keypairs.insert(
+            node_pubkey,
+            ValidatorVoteKeypairs::new(node_keypair, vote_keypair, stake_keypair),
+        );
+
+        // Initialize BankForks
+        let (bank_forks, mut progress) = initialize_state(&keypairs);
+        let bank_forks = RwLock::new(bank_forks);
+
+        // Create the tree of banks
+        let forks = tr(0) / (tr(1) / (tr(2) / (tr(3) / (tr(4) / tr(5)))));
+
+        // Set the voting behavior
+        let mut voting_simulator = VoteSimulator::new(&forks);
+        let votes = vec![0, 1, 2, 3, 4, 5];
+
+        // Simulate the votes
+        let mut tower = Tower::new_with_key(&node_pubkey);
+
+        let mut cluster_votes = HashMap::new();
+        for vote in votes {
+            assert!(voting_simulator
+                .simulate_vote(
+                    vote,
+                    &bank_forks,
+                    &mut cluster_votes,
+                    &keypairs,
+                    keypairs.get(&node_pubkey).unwrap(),
+                    &mut progress,
+                    &mut tower,
+                )
+                .is_empty());
+        }
+
+        for i in 0..5 {
+            assert_eq!(tower.lockouts.votes[i].slot as usize, i);
+            assert_eq!(tower.lockouts.votes[i].confirmation_count as usize, 6 - i);
+        }
+    }
+
+    #[test]
+    fn test_double_partition() {
+        solana_logger::setup();
+        let node_keypair = Keypair::new();
+        let vote_keypair = Keypair::new();
+        let stake_keypair = Keypair::new();
+        let node_pubkey = node_keypair.pubkey();
+        let vote_pubkey = vote_keypair.pubkey();
+
+        let mut keypairs = HashMap::new();
+        info!("my_pubkey: {}", node_pubkey);
+        keypairs.insert(
+            node_pubkey,
+            ValidatorVoteKeypairs::new(node_keypair, vote_keypair, stake_keypair),
+        );
+
+        // Create the tree of banks in a BankForks object
+        let forks = tr(0)
+            / (tr(1)
+                / (tr(2)
+                    / (tr(3)
+                        / (tr(4)
+                            / (tr(5)
+                                / (tr(6)
+                                    / (tr(7)
+                                        / (tr(8)
+                                            / (tr(9)
+                                                // Minor fork 1
+                                                / (tr(10) / (tr(11) / (tr(12) / (tr(13) / (tr(14))))))
+                                                / (tr(43)
+                                                    / (tr(44)
+                                                        // Minor fork 2
+                                                        / (tr(45) / (tr(46) / (tr(47) / (tr(48) / (tr(49) / (tr(50)))))))
+                                                        / (tr(110)))))))))))));
+
+        // Set the voting behavior
+        let mut voting_simulator = VoteSimulator::new(&forks);
+        let mut votes: Vec<Slot> = vec![];
+        // Vote on the first minor fork
+        votes.extend((0..=14).into_iter());
+        // Come back to the main fork
+        votes.extend((43..=44).into_iter());
+        // Vote on the second minor fork
+        votes.extend((45..=50).into_iter());
+
+        let mut cluster_votes: HashMap<Pubkey, Vec<Slot>> = HashMap::new();
+        let (bank_forks, mut progress) = initialize_state(&keypairs);
+        let bank_forks = RwLock::new(bank_forks);
+
+        // Simulate the votes. Should fail on trying to come back to the main fork
+        // at 106 exclusively due to threshold failure
+        let mut tower = Tower::new_with_key(&node_pubkey);
+        for vote in &votes {
+            // All these votes should be ok
+            assert!(voting_simulator
+                .simulate_vote(
+                    *vote,
+                    &bank_forks,
+                    &mut cluster_votes,
+                    &keypairs,
+                    keypairs.get(&node_pubkey).unwrap(),
+                    &mut progress,
+                    &mut tower,
+                )
+                .is_empty());
+        }
+
+        // Try to come back to main fork
+        let next_unlocked_slot = 110;
+        assert!(voting_simulator
+            .simulate_vote(
+                next_unlocked_slot,
+                &bank_forks,
+                &mut cluster_votes,
+                &keypairs,
+                keypairs.get(&node_pubkey).unwrap(),
+                &mut progress,
+                &mut tower,
+            )
+            .is_empty());
+
+        info!("local tower: {:#?}", tower.lockouts.votes);
+        let vote_accounts = bank_forks
+            .read()
+            .unwrap()
+            .get(next_unlocked_slot)
+            .unwrap()
+            .vote_accounts();
+        let observed = vote_accounts.get(&vote_pubkey).unwrap();
+        let state = VoteState::from(&observed.1).unwrap();
+        info!("observed tower: {:#?}", state.votes);
+
+        assert!(can_progress_on_fork(
+            &node_pubkey,
+            &mut tower,
+            next_unlocked_slot,
+            200,
+            &bank_forks,
+            &mut cluster_votes,
+            &keypairs,
+            &mut progress
+        ));
+    }
+
     #[test]
     fn test_collect_vote_lockouts_sums() {
         //two accounts voting for slot 0 with 1 token staked
@@ -542,6 +984,24 @@ mod test {
         assert!(tower.check_vote_stake_threshold(0, &stakes, 2));
     }
 
+    #[test]
+    fn test_check_vote_threshold_no_skip_lockout_with_new_root() {
+        solana_logger::setup();
+        let mut tower = Tower::new_for_tests(4, 0.67);
+        let mut stakes = HashMap::new();
+        for i in 0..(MAX_LOCKOUT_HISTORY as u64 + 1) {
+            stakes.insert(
+                i,
+                StakeLockout {
+                    stake: 1,
+                    lockout: 8,
+                },
+            );
+            tower.record_vote(i, Hash::default());
+        }
+        assert!(!tower.check_vote_stake_threshold(MAX_LOCKOUT_HISTORY as u64 + 1, &stakes, 2));
+    }
+
     #[test]
     fn test_is_slot_confirmed_not_enough_stake_failure() {
         let tower = Tower::new_for_tests(1, 0.67);
@@ -742,6 +1202,34 @@ mod test {
         assert!(!tower.check_vote_stake_threshold(1, &stakes, 2));
     }
 
+    #[test]
+    fn test_check_vote_threshold_lockouts_not_updated() {
+        solana_logger::setup();
+        let mut tower = Tower::new_for_tests(1, 0.67);
+        let stakes = vec![
+            (
+                0,
+                StakeLockout {
+                    stake: 1,
+                    lockout: 8,
+                },
+            ),
+            (
+                1,
+                StakeLockout {
+                    stake: 2,
+                    lockout: 8,
+                },
+            ),
+        ]
+        .into_iter()
+        .collect();
+        tower.record_vote(0, Hash::default());
+        tower.record_vote(1, Hash::default());
+        tower.record_vote(2, Hash::default());
+        assert!(tower.check_vote_stake_threshold(6, &stakes, 2));
+    }
+
     #[test]
     fn test_lockout_is_updated_for_entire_branch() {
         let mut stake_lockouts = HashMap::new();

core/src/contact_info.rs

@@ -2,7 +2,7 @@ use solana_sdk::pubkey::Pubkey;
 #[cfg(test)]
 use solana_sdk::rpc_port;
 #[cfg(test)]
-use solana_sdk::signature::{Keypair, KeypairUtil};
+use solana_sdk::signature::{Keypair, Signer};
 use solana_sdk::timing::timestamp;
 use std::cmp::{Ord, Ordering, PartialEq, PartialOrd};
 use std::net::{IpAddr, SocketAddr};
@@ -17,7 +17,7 @@ pub struct ContactInfo {
     pub tvu: SocketAddr,
     /// address to forward shreds to
     pub tvu_forwards: SocketAddr,
-    /// address to send repairs to
+    /// address to send repair responses to
     pub repair: SocketAddr,
     /// transactions address
     pub tpu: SocketAddr,
@@ -29,6 +29,8 @@ pub struct ContactInfo {
     pub rpc: SocketAddr,
     /// websocket for JSON-RPC push notifications
     pub rpc_pubsub: SocketAddr,
+    /// address to send repair requests to
+    pub serve_repair: SocketAddr,
     /// latest wallclock picked
     pub wallclock: u64,
     /// node shred version
@@ -85,6 +87,7 @@ impl Default for ContactInfo {
             storage_addr: socketaddr_any!(),
             rpc: socketaddr_any!(),
             rpc_pubsub: socketaddr_any!(),
+            serve_repair: socketaddr_any!(),
             wallclock: 0,
             shred_version: 0,
         }
@@ -92,70 +95,44 @@ impl Default for ContactInfo {
 }
 
 impl ContactInfo {
-    #[allow(clippy::too_many_arguments)]
-    pub fn new(
-        id: &Pubkey,
-        gossip: SocketAddr,
-        tvu: SocketAddr,
-        tvu_forwards: SocketAddr,
-        repair: SocketAddr,
-        tpu: SocketAddr,
-        tpu_forwards: SocketAddr,
-        storage_addr: SocketAddr,
-        rpc: SocketAddr,
-        rpc_pubsub: SocketAddr,
-        now: u64,
-    ) -> Self {
+    pub fn new_localhost(id: &Pubkey, now: u64) -> Self {
         Self {
             id: *id,
-            gossip,
-            tvu,
-            tvu_forwards,
-            repair,
-            tpu,
-            tpu_forwards,
-            storage_addr,
-            rpc,
-            rpc_pubsub,
+            gossip: socketaddr!("127.0.0.1:1234"),
+            tvu: socketaddr!("127.0.0.1:1235"),
+            tvu_forwards: socketaddr!("127.0.0.1:1236"),
+            repair: socketaddr!("127.0.0.1:1237"),
+            tpu: socketaddr!("127.0.0.1:1238"),
+            tpu_forwards: socketaddr!("127.0.0.1:1239"),
+            storage_addr: socketaddr!("127.0.0.1:1240"),
+            rpc: socketaddr!("127.0.0.1:1241"),
+            rpc_pubsub: socketaddr!("127.0.0.1:1242"),
+            serve_repair: socketaddr!("127.0.0.1:1243"),
             wallclock: now,
             shred_version: 0,
         }
     }
 
-    pub fn new_localhost(id: &Pubkey, now: u64) -> Self {
-        Self::new(
-            id,
-            socketaddr!("127.0.0.1:1234"),
-            socketaddr!("127.0.0.1:1235"),
-            socketaddr!("127.0.0.1:1236"),
-            socketaddr!("127.0.0.1:1237"),
-            socketaddr!("127.0.0.1:1238"),
-            socketaddr!("127.0.0.1:1239"),
-            socketaddr!("127.0.0.1:1240"),
-            socketaddr!("127.0.0.1:1241"),
-            socketaddr!("127.0.0.1:1242"),
-            now,
-        )
-    }
-
     #[cfg(test)]
     /// ContactInfo with multicast addresses for adversarial testing.
     pub fn new_multicast() -> Self {
         let addr = socketaddr!("224.0.1.255:1000");
         assert!(addr.ip().is_multicast());
-        Self::new(
-            &Pubkey::new_rand(),
-            addr,
-            addr,
-            addr,
-            addr,
-            addr,
-            addr,
-            addr,
-            addr,
-            addr,
-            0,
-        )
+        Self {
+            id: Pubkey::new_rand(),
+            gossip: addr,
+            tvu: addr,
+            tvu_forwards: addr,
+            repair: addr,
+            tpu: addr,
+            tpu_forwards: addr,
+            storage_addr: addr,
+            rpc: addr,
+            rpc_pubsub: addr,
+            serve_repair: addr,
+            wallclock: 0,
+            shred_version: 0,
+        }
     }
 
     #[cfg(test)]
@@ -166,27 +143,30 @@ impl ContactInfo {
             nxt_addr
         }
 
-        let tpu_addr = *bind_addr;
-        let gossip_addr = next_port(&bind_addr, 1);
-        let tvu_addr = next_port(&bind_addr, 2);
-        let tpu_forwards_addr = next_port(&bind_addr, 3);
-        let tvu_forwards_addr = next_port(&bind_addr, 4);
+        let tpu = *bind_addr;
+        let gossip = next_port(&bind_addr, 1);
+        let tvu = next_port(&bind_addr, 2);
+        let tpu_forwards = next_port(&bind_addr, 3);
+        let tvu_forwards = next_port(&bind_addr, 4);
         let repair = next_port(&bind_addr, 5);
-        let rpc_addr = SocketAddr::new(bind_addr.ip(), rpc_port::DEFAULT_RPC_PORT);
-        let rpc_pubsub_addr = SocketAddr::new(bind_addr.ip(), rpc_port::DEFAULT_RPC_PUBSUB_PORT);
-        Self::new(
-            pubkey,
-            gossip_addr,
-            tvu_addr,
-            tvu_forwards_addr,
+        let rpc = SocketAddr::new(bind_addr.ip(), rpc_port::DEFAULT_RPC_PORT);
+        let rpc_pubsub = SocketAddr::new(bind_addr.ip(), rpc_port::DEFAULT_RPC_PUBSUB_PORT);
+        let serve_repair = next_port(&bind_addr, 6);
+        Self {
+            id: *pubkey,
+            gossip,
+            tvu,
+            tvu_forwards,
             repair,
-            tpu_addr,
-            tpu_forwards_addr,
-            "0.0.0.0:0".parse().unwrap(),
-            rpc_addr,
-            rpc_pubsub_addr,
-            timestamp(),
-        )
+            tpu,
+            tpu_forwards,
+            storage_addr: "0.0.0.0:0".parse().unwrap(),
+            rpc,
+            rpc_pubsub,
+            serve_repair,
+            wallclock: timestamp(),
+            shred_version: 0,
+        }
     }
 
     #[cfg(test)]
@@ -197,20 +177,12 @@ impl ContactInfo {
 
     // Construct a ContactInfo that's only usable for gossip
     pub fn new_gossip_entry_point(gossip_addr: &SocketAddr) -> Self {
-        let daddr: SocketAddr = socketaddr!("0.0.0.0:0");
-        Self::new(
-            &Pubkey::default(),
-            *gossip_addr,
-            daddr,
-            daddr,
-            daddr,
-            daddr,
-            daddr,
-            daddr,
-            daddr,
-            daddr,
-            timestamp(),
-        )
+        Self {
+            id: Pubkey::default(),
+            gossip: *gossip_addr,
+            wallclock: timestamp(),
+            ..ContactInfo::default()
+        }
     }
 
     fn is_valid_ip(addr: IpAddr) -> bool {
@@ -267,6 +239,7 @@ mod tests {
         assert!(ci.rpc_pubsub.ip().is_unspecified());
         assert!(ci.tpu.ip().is_unspecified());
         assert!(ci.storage_addr.ip().is_unspecified());
+        assert!(ci.serve_repair.ip().is_unspecified());
     }
     #[test]
     fn test_multicast() {
@@ -278,6 +251,7 @@ mod tests {
         assert!(ci.rpc_pubsub.ip().is_multicast());
         assert!(ci.tpu.ip().is_multicast());
         assert!(ci.storage_addr.ip().is_multicast());
+        assert!(ci.serve_repair.ip().is_multicast());
     }
     #[test]
     fn test_entry_point() {
@@ -290,6 +264,7 @@ mod tests {
         assert!(ci.rpc_pubsub.ip().is_unspecified());
         assert!(ci.tpu.ip().is_unspecified());
         assert!(ci.storage_addr.ip().is_unspecified());
+        assert!(ci.serve_repair.ip().is_unspecified());
     }
     #[test]
     fn test_socketaddr() {
@@ -299,10 +274,12 @@ mod tests {
         assert_eq!(ci.gossip.port(), 11);
         assert_eq!(ci.tvu.port(), 12);
         assert_eq!(ci.tpu_forwards.port(), 13);
-        assert_eq!(ci.rpc.port(), 8899);
-        assert_eq!(ci.rpc_pubsub.port(), 8900);
+        assert_eq!(ci.rpc.port(), rpc_port::DEFAULT_RPC_PORT);
+        assert_eq!(ci.rpc_pubsub.port(), rpc_port::DEFAULT_RPC_PUBSUB_PORT);
         assert!(ci.storage_addr.ip().is_unspecified());
+        assert_eq!(ci.serve_repair.port(), 16);
     }
+
     #[test]
     fn replayed_data_new_with_socketaddr_with_pubkey() {
         let keypair = Keypair::new();
@@ -315,8 +292,17 @@ mod tests {
         assert_eq!(d1.tvu, socketaddr!("127.0.0.1:1236"));
         assert_eq!(d1.tpu_forwards, socketaddr!("127.0.0.1:1237"));
         assert_eq!(d1.tpu, socketaddr!("127.0.0.1:1234"));
-        assert_eq!(d1.rpc, socketaddr!("127.0.0.1:8899"));
-        assert_eq!(d1.rpc_pubsub, socketaddr!("127.0.0.1:8900"));
+        assert_eq!(
+            d1.rpc,
+            socketaddr!(format!("127.0.0.1:{}", rpc_port::DEFAULT_RPC_PORT))
+        );
+        assert_eq!(
+            d1.rpc_pubsub,
+            socketaddr!(format!("127.0.0.1:{}", rpc_port::DEFAULT_RPC_PUBSUB_PORT))
+        );
+        assert_eq!(d1.tvu_forwards, socketaddr!("127.0.0.1:1238"));
+        assert_eq!(d1.repair, socketaddr!("127.0.0.1:1239"));
+        assert_eq!(d1.serve_repair, socketaddr!("127.0.0.1:1240"));
     }
 
     #[test]

core/src/crds_gossip.rs

@@ -3,11 +3,13 @@
 //! designed to run with a simulator or over a UDP network connection with messages up to a
 //! packet::PACKET_DATA_SIZE size.
 
-use crate::crds::{Crds, VersionedCrdsValue};
-use crate::crds_gossip_error::CrdsGossipError;
-use crate::crds_gossip_pull::{CrdsFilter, CrdsGossipPull};
-use crate::crds_gossip_push::{CrdsGossipPush, CRDS_GOSSIP_NUM_ACTIVE};
-use crate::crds_value::{CrdsValue, CrdsValueLabel};
+use crate::{
+    crds::{Crds, VersionedCrdsValue},
+    crds_gossip_error::CrdsGossipError,
+    crds_gossip_pull::{CrdsFilter, CrdsGossipPull},
+    crds_gossip_push::{CrdsGossipPush, CRDS_GOSSIP_NUM_ACTIVE},
+    crds_value::{CrdsValue, CrdsValueLabel},
+};
 use solana_sdk::pubkey::Pubkey;
 use std::collections::{HashMap, HashSet};
 
@@ -156,11 +158,12 @@ impl CrdsGossip {
     pub fn process_pull_response(
         &mut self,
         from: &Pubkey,
+        timeouts: &HashMap<Pubkey, u64>,
         response: Vec<CrdsValue>,
         now: u64,
     ) -> usize {
         self.pull
-            .process_pull_response(&mut self.crds, from, response, now)
+            .process_pull_response(&mut self.crds, from, timeouts, response, now)
     }
 
     pub fn make_timeouts_test(&self) -> HashMap<Pubkey, u64> {

core/src/crds_gossip_pull.rs

@@ -25,6 +25,8 @@ use std::collections::HashMap;
 use std::collections::VecDeque;
 
 pub const CRDS_GOSSIP_PULL_CRDS_TIMEOUT_MS: u64 = 15000;
+// The maximum age of a value received over pull responses
+pub const CRDS_GOSSIP_PULL_MSG_TIMEOUT_MS: u64 = 60000;
 pub const FALSE_RATE: f64 = 0.1f64;
 pub const KEYS: f64 = 8f64;
 
@@ -117,6 +119,7 @@ pub struct CrdsGossipPull {
     /// hash and insert time
     purged_values: VecDeque<(Hash, u64)>,
     pub crds_timeout: u64,
+    pub msg_timeout: u64,
 }
 
 impl Default for CrdsGossipPull {
@@ -125,6 +128,7 @@ impl Default for CrdsGossipPull {
             purged_values: VecDeque::new(),
             pull_request_time: HashMap::new(),
             crds_timeout: CRDS_GOSSIP_PULL_CRDS_TIMEOUT_MS,
+            msg_timeout: CRDS_GOSSIP_PULL_MSG_TIMEOUT_MS,
         }
     }
 }
@@ -210,12 +214,56 @@ impl CrdsGossipPull {
         &mut self,
         crds: &mut Crds,
         from: &Pubkey,
+        timeouts: &HashMap<Pubkey, u64>,
         response: Vec<CrdsValue>,
         now: u64,
     ) -> usize {
         let mut failed = 0;
         for r in response {
             let owner = r.label().pubkey();
+            // Check if the crds value is older than the msg_timeout
+            if now
+                > r.wallclock()
+                    .checked_add(self.msg_timeout)
+                    .unwrap_or_else(|| 0)
+                || now + self.msg_timeout < r.wallclock()
+            {
+                match &r.label() {
+                    CrdsValueLabel::ContactInfo(_) => {
+                        // Check if this ContactInfo is actually too old, it's possible that it has
+                        // stake and so might have a longer effective timeout
+                        let timeout = *timeouts
+                            .get(&owner)
+                            .unwrap_or_else(|| timeouts.get(&Pubkey::default()).unwrap());
+                        if now > r.wallclock().checked_add(timeout).unwrap_or_else(|| 0)
+                            || now + timeout < r.wallclock()
+                        {
+                            inc_new_counter_warn!(
+                                "cluster_info-gossip_pull_response_value_timeout",
+                                1
+                            );
+                            failed += 1;
+                            continue;
+                        }
+                    }
+                    _ => {
+                        // Before discarding this value, check if a ContactInfo for the owner
+                        // exists in the table. If it doesn't, that implies that this value can be discarded
+                        if crds.lookup(&CrdsValueLabel::ContactInfo(owner)).is_none() {
+                            inc_new_counter_warn!(
+                                "cluster_info-gossip_pull_response_value_timeout",
+                                1
+                            );
+                            failed += 1;
+                            continue;
+                        } else {
+                            // Silently insert this old value without bumping record timestamps
+                            failed += crds.insert(r, now).is_err() as usize;
+                            continue;
+                        }
+                    }
+                }
+            }
             let old = crds.insert(r, now);
             failed += old.is_err() as usize;
             old.ok().map(|opt| {
@@ -322,8 +370,9 @@ impl CrdsGossipPull {
 mod test {
     use super::*;
     use crate::contact_info::ContactInfo;
-    use crate::crds_value::CrdsData;
+    use crate::crds_value::{CrdsData, Vote};
     use itertools::Itertools;
+    use solana_perf::test_tx::test_tx;
     use solana_sdk::hash::hash;
     use solana_sdk::packet::PACKET_DATA_SIZE;
 
@@ -534,8 +583,13 @@ mod test {
                 continue;
             }
             assert_eq!(rsp.len(), 1);
-            let failed =
-                node.process_pull_response(&mut node_crds, &node_pubkey, rsp.pop().unwrap(), 1);
+            let failed = node.process_pull_response(
+                &mut node_crds,
+                &node_pubkey,
+                &node.make_timeouts_def(&node_pubkey, &HashMap::new(), 0, 1),
+                rsp.pop().unwrap(),
+                1,
+            );
             assert_eq!(failed, 0);
             assert_eq!(
                 node_crds
@@ -675,4 +729,87 @@ mod test {
             .collect();
         assert_eq!(masks.len(), 2u64.pow(mask_bits) as usize)
     }
+
+    #[test]
+    fn test_process_pull_response() {
+        let mut node_crds = Crds::default();
+        let mut node = CrdsGossipPull::default();
+
+        let peer_pubkey = Pubkey::new_rand();
+        let peer_entry = CrdsValue::new_unsigned(CrdsData::ContactInfo(
+            ContactInfo::new_localhost(&peer_pubkey, 0),
+        ));
+        let mut timeouts = HashMap::new();
+        timeouts.insert(Pubkey::default(), node.crds_timeout);
+        timeouts.insert(peer_pubkey, node.msg_timeout + 1);
+        // inserting a fresh value should be fine.
+        assert_eq!(
+            node.process_pull_response(
+                &mut node_crds,
+                &peer_pubkey,
+                &timeouts,
+                vec![peer_entry.clone()],
+                1,
+            ),
+            0
+        );
+
+        let mut node_crds = Crds::default();
+        let unstaked_peer_entry = CrdsValue::new_unsigned(CrdsData::ContactInfo(
+            ContactInfo::new_localhost(&peer_pubkey, 0),
+        ));
+        // check that old contact infos fail if they are too old, regardless of "timeouts"
+        assert_eq!(
+            node.process_pull_response(
+                &mut node_crds,
+                &peer_pubkey,
+                &timeouts,
+                vec![peer_entry.clone(), unstaked_peer_entry],
+                node.msg_timeout + 100,
+            ),
+            2
+        );
+
+        let mut node_crds = Crds::default();
+        // check that old contact infos can still land as long as they have a "timeouts" entry
+        assert_eq!(
+            node.process_pull_response(
+                &mut node_crds,
+                &peer_pubkey,
+                &timeouts,
+                vec![peer_entry.clone()],
+                node.msg_timeout + 1,
+            ),
+            0
+        );
+
+        // construct something that's not a contact info
+        let peer_vote =
+            CrdsValue::new_unsigned(CrdsData::Vote(0, Vote::new(&peer_pubkey, test_tx(), 0)));
+        // check that older CrdsValues (non-ContactInfos) infos pass even if are too old,
+        // but a recent contact info (inserted above) exists
+        assert_eq!(
+            node.process_pull_response(
+                &mut node_crds,
+                &peer_pubkey,
+                &timeouts,
+                vec![peer_vote.clone()],
+                node.msg_timeout + 1,
+            ),
+            0
+        );
+
+        let mut node_crds = Crds::default();
+        // without a contact info, inserting an old value should fail
+        assert_eq!(
+            node.process_pull_response(
+                &mut node_crds,
+                &peer_pubkey,
+                &timeouts,
+                vec![peer_vote.clone()],
+                node.msg_timeout + 1,
+            ),
+            1
+        );
+    }
 }

core/src/crds_gossip_push.rs

@@ -8,29 +8,31 @@
 //!    the local nodes wallclock window they are drooped silently.
 //! 2. The prune set is stored in a Bloom filter.
 
-use crate::contact_info::ContactInfo;
-use crate::crds::{Crds, VersionedCrdsValue};
-use crate::crds_gossip::{get_stake, get_weight, CRDS_GOSSIP_DEFAULT_BLOOM_ITEMS};
-use crate::crds_gossip_error::CrdsGossipError;
-use crate::crds_value::{CrdsValue, CrdsValueLabel};
-use crate::weighted_shuffle::weighted_shuffle;
+use crate::{
+    contact_info::ContactInfo,
+    crds::{Crds, VersionedCrdsValue},
+    crds_gossip::{get_stake, get_weight, CRDS_GOSSIP_DEFAULT_BLOOM_ITEMS},
+    crds_gossip_error::CrdsGossipError,
+    crds_value::{CrdsValue, CrdsValueLabel},
+    weighted_shuffle::weighted_shuffle,
+};
 use bincode::serialized_size;
 use indexmap::map::IndexMap;
 use itertools::Itertools;
-use rand;
-use rand::seq::SliceRandom;
-use rand::{thread_rng, RngCore};
+use rand::{self, seq::SliceRandom, thread_rng, RngCore};
 use solana_runtime::bloom::Bloom;
-use solana_sdk::hash::Hash;
-use solana_sdk::packet::PACKET_DATA_SIZE;
-use solana_sdk::pubkey::Pubkey;
-use solana_sdk::timing::timestamp;
-use std::cmp;
-use std::collections::{HashMap, HashSet};
+use solana_sdk::{hash::Hash, packet::PACKET_DATA_SIZE, pubkey::Pubkey, timing::timestamp};
+use std::{
+    cmp,
+    collections::{HashMap, HashSet},
+};
 
 pub const CRDS_GOSSIP_NUM_ACTIVE: usize = 30;
 pub const CRDS_GOSSIP_PUSH_FANOUT: usize = 6;
-pub const CRDS_GOSSIP_PUSH_MSG_TIMEOUT_MS: u64 = 5000;
+// With a fanout of 6, a 1000 node cluster should only take ~4 hops to converge.
+// However since pushes are stake weighed, some trailing nodes
+// might need more time to receive values. 30 seconds should be plenty.
+pub const CRDS_GOSSIP_PUSH_MSG_TIMEOUT_MS: u64 = 30000;
 pub const CRDS_GOSSIP_PRUNE_MSG_TIMEOUT_MS: u64 = 500;
 pub const CRDS_GOSSIP_PRUNE_STAKE_THRESHOLD_PCT: f64 = 0.15;
 
@@ -135,7 +137,12 @@ impl CrdsGossipPush {
         value: CrdsValue,
         now: u64,
     ) -> Result<Option<VersionedCrdsValue>, CrdsGossipError> {
-        if now > value.wallclock() + self.msg_timeout {
+        if now
+            > value
+                .wallclock()
+                .checked_add(self.msg_timeout)
+                .unwrap_or_else(|| 0)
+        {
             return Err(CrdsGossipError::PushMessageTimeout);
         }
         if now + self.msg_timeout < value.wallclock() {