Cherry pick fix for freeze (#4459)

* check freeze before updating slot_hashes (#4448)

* check freeze before updating slot_hashes

* fixup

* add more information to dropped vote warning (#4449)

* add more information to dropped vote warning

* fixup

book/makefile

@@ -1,7 +1,8 @@
 BOB_SRCS=$(wildcard art/*.bob)
+MSC_SRCS=$(wildcard art/*.msc)
 MD_SRCS=$(wildcard src/*.md)
 
-SVG_IMGS=$(BOB_SRCS:art/%.bob=src/img/%.svg)
+SVG_IMGS=$(BOB_SRCS:art/%.bob=src/img/%.svg) $(MSC_SRCS:art/%.msc=src/img/%.svg)
 
 all: html/index.html
 
@@ -17,6 +18,10 @@ src/img/%.svg: art/%.bob
 	@mkdir -p $(@D)
 	svgbob < $< > $@
 
+ src/img/%.svg: art/%.msc
+	@mkdir -p $(@D)
+	mscgen -T svg -i $< -o $@
+
 src/%.md: %.md
 	@mkdir -p $(@D)
 	@cp $< $@

book/src/SUMMARY.md

@@ -19,7 +19,7 @@
   - [Data Plane Fanout](data-plane-fanout.md)
   - [Ledger Replication](ledger-replication.md)
   - [Secure Vote Signing](vote-signing.md)
-  - [Staking Delegation and Rewards](stake-delegation-and-rewards.md)
+  - [Stake Delegation and Rewards](stake-delegation-and-rewards.md)
   - [Performance Metrics](performance-metrics.md)
 
 - [Anatomy of a Validator](validator.md)
@@ -39,7 +39,6 @@
   - [Ledger Replication](ledger-replication-to-implement.md)
   - [Secure Vote Signing](vote-signing-to-implement.md)
   - [Staking Rewards](staking-rewards.md)
-  - [Passive Stake Delegation and Rewards](passive-stake-delegation-and-rewards.md)
   - [Cluster Economics](ed_overview.md)
     - [Validation-client Economics](ed_validation_client_economics.md)
       - [State-validation Protocol-based Rewards](ed_vce_state_validation_protocol_based_rewards.md)
@@ -67,3 +66,4 @@
   - [Reliable Vote Transmission](reliable-vote-transmission.md)
   - [Persistent Account Storage](persistent-account-storage.md)
   - [Cluster Software Installation and Updates](installer.md)
+  - [Passive Stake Delegation and Rewards](passive-stake-delegation-and-rewards.md)

book/src/stake-delegation-and-rewards.md

@@ -1,68 +1,195 @@
 # Stake Delegation and Rewards
 
-Stakers are rewarded for helping validate the ledger. They do it by delegating
-their stake to fullnodes. Those fullnodes do the legwork and send votes to the
-stakers' staking accounts. The rest of the cluster uses those stake-weighted
-votes to select a block when forks arise. Both the fullnode and staker need
-some economic incentive to play their part. The fullnode needs to be
-compensated for its hardware and the staker needs to be compensated for risking
-getting its stake slashed.  The economics are covered in [staking
+Stakers are rewarded for helping to validate the ledger. They do this by
+delegating their stake to validator nodes. Those validators do the legwork of
+replaying the ledger and send votes to a per-node vote account to which stakers
+can delegate their stakes.  The rest of the cluster uses those stake-weighted
+votes to select a block when forks arise. Both the validator and staker need
+some economic incentive to play their part. The validator needs to be
+compensated for its hardware and the staker needs to be compensated for the risk
+of getting its stake slashed.  The economics are covered in [staking
 rewards](staking-rewards.md).  This chapter, on the other hand, describes the
 underlying mechanics of its implementation.
 
-## Vote and Rewards accounts
+## Basic Besign
 
-The rewards process is split into two on-chain programs. The Vote program
-solves the problem of making stakes slashable. The Rewards account acts as
-custodian of the rewards pool. It is responsible for paying out each staker
-once the staker proves to the Rewards program that it participated in
-validating the ledger.
+The general idea is that the validator owns a Vote account. The Vote account
+tracks validator votes, counts validator generated credits, and provides any
+additional validator specific state.  The Vote account is not aware of any
+stakes delegated to it and has no staking weight.
 
-The Vote account contains the following state information:
+A separate Stake account (created by a staker) names a Vote account to which the
+stake is delegated.  Rewards generated are proportional to the amount of
+lamports staked.  The Stake account is owned by the staker only.  Lamports
+stored in this account are the stake.
 
-* votes - The submitted votes.
+## Passive Delegation
 
-* `delegate_pubkey` - An identity that may operate with the weight of this
-  account's stake. It is typically the identity of a fullnode, but may be any
-identity involved in stake-weighted computations.
+Any number of Stake accounts can delegate to a single
+Vote account without an interactive action from the identity controlling
+the Vote account or submitting votes to the account.
 
-* `authorized_voter_pubkey` - Only this identity is authorized to submit votes.
+The total stake allocated to a Vote account can be calculated by the sum of
+all the Stake accounts that have the Vote account pubkey as the
+`StakeState::Delegate::voter_pubkey`.
 
-* `credits` - The amount of unclaimed rewards.
+## Vote and Stake accounts
 
-* `root_slot` - The last slot to reach the full lockout commitment necessary
-  for rewards.
+The rewards process is split into two on-chain programs. The Vote program solves
+the problem of making stakes slashable. The Stake account acts as custodian of
+the rewards pool, and provides passive delegation. The Stake program is
+responsible for paying out each staker once the staker proves to the Stake
+program that its delegate has participated in validating the ledger.
 
-The Rewards program is stateless and pays out reward when a staker submits its
-Vote account to the program. Claiming a reward requires a transaction that
-includes the following instructions:
+### VoteState
 
-1. `RewardsInstruction::RedeemVoteCredits`
-2. `VoteInstruction::ClearCredits`
+VoteState is the current state of all the votes the validator has submitted to
+the network. VoteState contains the following state information:
 
-The Rewards program transfers lamports from the Rewards account to the Vote
-account's public key. The Rewards program also ensures that the `ClearCredits`
-instruction follows the `RedeemVoteCredits` instruction, such that a staker may
-not claim rewards for the same work more than once.
+* votes - The submitted votes data structure.
+
+* credits - The total number of rewards this vote program has generated over its
+lifetime.
+
+* root\_slot - The last slot to reach the full lockout commitment necessary for
+rewards.
+
+* commission - The commission taken by this VoteState for any rewards claimed by
+staker's Stake accounts.  This is the percentage ceiling of the reward.
+
+* Account::lamports - The accumulated lamports from the commission.  These do not
+count as stakes.
+
+* `authorized_vote_signer` - Only this identity is authorized to submit votes. This field can only modified by this identity.
+
+### VoteInstruction::Initialize
+
+* `account[0]` - RW - The VoteState
+  `VoteState::authorized_vote_signer` is initialized to `account[0]`
+   other VoteState members defaulted
+
+### VoteInstruction::AuthorizeVoteSigner(Pubkey)
+
+* `account[0]` - RW - The VoteState
+  `VoteState::authorized_vote_signer` is set to to `Pubkey`, instruction must by
+   signed by Pubkey
+
+### VoteInstruction::Vote(Vec<Vote>)
+
+* `account[0]` - RW - The VoteState
+  `VoteState::lockouts` and `VoteState::credits` are updated according to voting lockout rules see [Fork Selection](fork-selection.md)
 
 
-### Delegating Stake
+* `account[1]` - RO - A list of some N most recent slots and their hashes for the vote to be verified against.
 
-`VoteInstruction::DelegateStake` allows the staker to choose a fullnode to
-validate the ledger on its behalf. By being a delegate, the fullnode is
-entitled to collect transaction fees when its is leader. The larger the stake,
-the more often the fullnode will be able to collect those fees.
 
-### Authorizing a Vote Signer
+### StakeState
+
+A StakeState takes one of two forms, StakeState::Delegate and StakeState::MiningPool.
+
+### StakeState::Delegate
+
+StakeState is the current delegation preference of the **staker**. StakeState
+contains the following state information:
+
+* Account::lamports - The staked lamports.
+
+* `voter_pubkey` - The pubkey of the VoteState instance the lamports are
+delegated to.
+
+* `credits_observed` - The total credits claimed over the lifetime of the
+program.
+
+### StakeState::MiningPool
+
+There are two approaches to the mining pool.  The bank could allow the
+StakeState program to bypass the token balance check, or a program representing
+the mining pool could run on the network.  To avoid a single network wide lock,
+the pool can be split into several mining pools.  This design focuses on using
+StakeState::MiningPool instances as the cluster wide mining pools.
+
+* 256 StakeState::MiningPool are initialized, each with 1/256 number of mining pool
+tokens stored as `Account::lamports`.
+
+The stakes and the MiningPool are accounts that are owned by the same `Stake`
+program.
+
+### StakeInstruction::Initialize
+
+* `account[0]` - RW - The StakeState::Delegate instance.
+  `StakeState::Delegate::credits_observed` is initialized to `VoteState::credits`.
+  `StakeState::Delegate::voter_pubkey` is initialized to `account[1]`
+
+* `account[1]` - R - The VoteState instance.
+
+### StakeInstruction::RedeemVoteCredits
+
+The Staker or the owner of the Stake account sends a transaction with this
+instruction to claim rewards.
+
+The Vote account and the Stake account pair maintain a lifetime counter
+of total rewards generated and claimed.  When claiming rewards, the total lamports
+deposited into the Stake account and as validator commission is proportional to
+`VoteState::credits - StakeState::credits_observed`.
+
+
+* `account[0]` - RW - The StakeState::MiningPool instance that will fulfill the
+reward.
+* `account[1]` - RW - The StakeState::Delegate instance that is redeeming votes
+credits.
+* `account[2]` - R - The VoteState instance, must be the same as
+`StakeState::voter_pubkey`
+
+Reward is paid out for the difference between `VoteState::credits` to
+`StakeState::Delgate.credits_observed`, and `credits_observed` is updated to
+`VoteState::credits`.  The commission is deposited into the Vote account token
+balance, and the reward is deposited to the Stake account token balance.
+
+The total lamports paid is a percentage-rate of the lamports staked muiltplied by
+the ratio of rewards being redeemed to rewards that could have been generated
+during the rate period.
+
+Any random MiningPool can be used to redeem the credits.
+
+```rust,ignore
+let credits_to_claim = vote_state.credits - stake_state.credits_observed;
+stake_state.credits_observed = vote_state.credits;
+```
+
+`credits_to_claim` is used to compute the reward and commission, and
+`StakeState::Delegate::credits_observed` is updated to the latest
+`VoteState::credits` value.
+
+## Collecting network fees into the MiningPool
+
+At the end of the block, before the bank is frozen, but after it processed all
+the transactions for the block, a virtual instruction is executed to collect
+the transaction fees.
+
+* A portion of the fees are deposited into the leader's account.
+* A portion of the fees are deposited into the smallest StakeState::MiningPool
+account.
+
+## Authorizing a Vote Signer
 
 `VoteInstruction::AuthorizeVoter` allows a staker to choose a signing service
 for its votes. That service is responsible for ensuring the vote won't cause
 the staker to be slashed.
 
-## Limitations
+## Benefits of the design
 
-Many stakers may delegate their stakes to the same fullnode. The fullnode must
-send a separate vote to each staking account. If there are far more stakers
-than fullnodes, that's a lot of network traffic. An alternative design might
-have fullnodes submit each vote to just one account and then have each staker
-submit that account along with their own to collect its reward.
+* Single vote for all the stakers.
+
+* Clearing of the credit variable is not necessary for claiming rewards.
+
+* Each delegated stake can claim its rewards independently.
+
+* Commission for the work is deposited when a reward is claimed by the delegated
+stake.
+
+This proposal would benefit from the `read-only` accounts proposal to allow for
+many rewards to be claimed concurrently.
+
+## Example Callflow
+
+<img alt="Passive Staking Callflow" src="img/passive-staking-callflow.svg" class="center"/>

book/src/staking-rewards.md

@@ -1,8 +1,8 @@
 # Staking Rewards
 
-Initial Proof of Stake (PoS) (i.e. using in-protocol asset, SOL, to provide
-secure consensus) design ideas outlined here. Solana will implement a proof of
-stake reward/security scheme for node validators in the cluster. The purpose is
+A Proof of Stake (PoS), (i.e. using in-protocol asset, SOL, to provide
+secure consensus) design is outlined here. Solana implements a proof of
+stake reward/security scheme for validator nodes in the cluster. The purpose is
 threefold:
 
 - Align validator incentives with that of the greater cluster through
@@ -64,7 +64,7 @@ capital-at-risk to prevent a logical/optimal strategy of multiple chain voting.
 We intend to implement slashing rules which, if broken, result some amount of
 the offending validator's deposited stake to be removed from circulation. Given
 the ordering properties of the PoH data structure, we believe we can simplify
-our slashing rules to the level of a voting lockout time assigned per vote.  
+our slashing rules to the level of a voting lockout time assigned per vote.
 
 I.e. Each vote has an associated lockout time (PoH duration) that represents a
 duration by any additional vote from that validator must be in a PoH that
@@ -110,7 +110,7 @@ in a slashable amount as a function of either:
 1. the fraction of validators, out of the total validator pool, that were also
    slashed during the same time period (ala Casper)
 2. the amount of time since the vote was cast (e.g. a linearly increasing % of
-   total deposited as slashable amount over time), or both.  
+   total deposited as slashable amount over time), or both.
 
 This is an area currently under exploration
 

ci/testnet-manager.sh

@@ -132,19 +132,16 @@ case $TESTNET in
 testnet-edge|testnet-edge-perf)
   CHANNEL_OR_TAG=edge
   CHANNEL_BRANCH=$EDGE_CHANNEL
-  : "${TESTNET_DB_HOST:=https://clocktower-f1d56615.influxcloud.net:8086}"
   ;;
 testnet-beta|testnet-beta-perf)
   CHANNEL_OR_TAG=beta
   CHANNEL_BRANCH=$BETA_CHANNEL
-  : "${TESTNET_DB_HOST:=https://clocktower-f1d56615.influxcloud.net:8086}"
   ;;
 testnet)
   CHANNEL_OR_TAG=$STABLE_CHANNEL_LATEST_TAG
   CHANNEL_BRANCH=$STABLE_CHANNEL
   : "${EC2_NODE_COUNT:=10}"
   : "${GCE_NODE_COUNT:=}"
-  : "${TESTNET_DB_HOST:=https://clocktower-f1d56615.influxcloud.net:8086}"
   ;;
 testnet-perf)
   CHANNEL_OR_TAG=$STABLE_CHANNEL_LATEST_TAG
@@ -155,7 +152,6 @@ testnet-demo)
   CHANNEL_BRANCH=$BETA_CHANNEL
   : "${GCE_NODE_COUNT:=150}"
   : "${GCE_LOW_QUOTA_NODE_COUNT:=70}"
-  : "${TESTNET_DB_HOST:=https://clocktower-f1d56615.influxcloud.net:8086}"
   ;;
 *)
   echo "Error: Invalid TESTNET=$TESTNET"

core/src/blocktree_processor.rs

@@ -3,7 +3,7 @@ use crate::blocktree::Blocktree;
 use crate::entry::{Entry, EntrySlice};
 use crate::leader_schedule_cache::LeaderScheduleCache;
 use rayon::prelude::*;
-use solana_metrics::{datapoint, inc_new_counter_debug};
+use solana_metrics::{datapoint, datapoint_error, inc_new_counter_debug};
 use solana_runtime::bank::Bank;
 use solana_runtime::locked_accounts_results::LockedAccountsResults;
 use solana_sdk::genesis_block::GenesisBlock;
@@ -45,7 +45,7 @@ fn par_execute_entries(
                     }
                     if !Bank::can_commit(&r) {
                         warn!("Unexpected validator error: {:?}, tx: {:?}", e, tx);
-                        datapoint!(
+                        datapoint_error!(
                             "validator_process_entry_error",
                             ("error", format!("error: {:?}, tx: {:?}", e, tx), String)
                         );

install/src/main.rs

@@ -58,7 +58,7 @@ fn main() -> Result<(), String> {
                 .arg({
                     let arg = Arg::with_name("data_dir")
                         .short("d")
-                        .long("data_dir")
+                        .long("data-dir")
                         .value_name("PATH")
                         .takes_value(true)
                         .required(true)

net/init-metrics.sh

@@ -17,7 +17,6 @@ usage: $0 [-e] [-d] [-c] [username]
 Creates a testnet dev metrics database
 
   username        InfluxDB user with access to create a new database
-  -c              Use Influx Cloud instance
   -d              Delete the database instead of creating it
   -e              Assume database already exists and SOLANA_METRICS_CONFIG is
                   defined in the environment already
@@ -31,15 +30,12 @@ loadConfigFile
 useEnv=false
 delete=false
 host="https://metrics.solana.com:8086"
-while getopts "hcde" opt; do
+while getopts "hde" opt; do
   case $opt in
   h|\?)
     usage
     exit 0
     ;;
-  c)
-    host="https://clocktower-f1d56615.influxcloud.net:8086"
-    ;;
   d)
     delete=true
     ;;

net/remote/remote-node.sh

@@ -89,7 +89,8 @@ local|tar)
       --gossip-port "$entrypointIp":8001
     )
 
-    ./multinode-demo/validator.sh --bootstrap-leader "${args[@]}" > fullnode.log 2>&1 &
+    nohup ./multinode-demo/validator.sh --bootstrap-leader "${args[@]}" > fullnode.log 2>&1 &
+    sleep 1
     ;;
   validator|blockstreamer)
     net/scripts/rsync-retry.sh -vPrc "$entrypointIp":~/.cargo/bin/ ~/.cargo/bin/
@@ -129,7 +130,7 @@ local|tar)
       ./multinode-demo/drone.sh > drone.log 2>&1 &
 
       export BLOCKEXPLORER_GEOIP_WHITELIST=$PWD/net/config/geoip.yml
-      npm install @solana/blockexplorer@1
+      npm install @solana/blockexplorer@1.8.12
       npx solana-blockexplorer > blockexplorer.log 2>&1 &
 
       # Confirm the blockexplorer is accessible
@@ -144,7 +145,8 @@ local|tar)
       curl --head "$(curl ifconfig.io)"
     fi
 
-    ./multinode-demo/validator.sh "${args[@]}" > fullnode.log 2>&1 &
+    nohup ./multinode-demo/validator.sh "${args[@]}" > fullnode.log 2>&1 &
+    sleep 1
     ;;
   *)
     echo "Error: unknown node type: $nodeType"

programs/vote_api/src/vote_state.rs

@@ -145,10 +145,22 @@ impl VoteState {
             .iter()
             .any(|(slot, hash)| vote.slot == *slot && vote.hash == *hash)
         {
-            warn!(
-                "dropping vote {:?}, no matching slot/hash combination",
-                vote
-            );
+            if log_enabled!(log::Level::Warn) {
+                for (slot, hash) in slot_hashes {
+                    if vote.slot == *slot {
+                        warn!(
+                            "dropped vote {:?} matched slot {}, but not hash {:?}",
+                            vote, *slot, *hash
+                        );
+                    }
+                    if vote.hash == *hash {
+                        warn!(
+                            "dropped vote {:?} matched hash {:?}, but not slot {}",
+                            vote, *hash, *slot,
+                        );
+                    }
+                }
+            }
             return;
         }
 

runtime/src/bank.rs

@@ -215,18 +215,22 @@ impl Bank {
         self.store(&slot_hashes::id(), &account);
     }
 
-    fn set_hash(&self) {
+    fn set_hash(&self) -> bool {
         let mut hash = self.hash.write().unwrap();
 
         if *hash == Hash::default() {
             //  freeze is a one-way trip, idempotent
             *hash = self.hash_internal_state();
+            true
+        } else {
+            false
         }
     }
 
     pub fn freeze(&self) {
-        self.set_hash();
-        self.update_slot_hashes();
+        if self.set_hash() {
+            self.update_slot_hashes();
+        }
     }
 
     pub fn epoch_schedule(&self) -> &EpochSchedule {
@@ -1347,6 +1351,36 @@ mod tests {
         assert_eq!(bank.transaction_count(), 1);
     }
 
+    #[test]
+    fn test_need_credit_only_accounts() {
+        let (genesis_block, mint_keypair) = create_genesis_block(10);
+        let bank = Bank::new(&genesis_block);
+        let payer0 = Keypair::new();
+        let payer1 = Keypair::new();
+        let recipient = Pubkey::new_rand();
+        // Fund additional payers
+        bank.transfer(3, &mint_keypair, &payer0.pubkey()).unwrap();
+        bank.transfer(3, &mint_keypair, &payer1.pubkey()).unwrap();
+        let tx0 = system_transaction::transfer(&mint_keypair, &recipient, 1, genesis_block.hash());
+        let tx1 = system_transaction::transfer(&payer0, &recipient, 1, genesis_block.hash());
+        let tx2 = system_transaction::transfer(&payer1, &recipient, 1, genesis_block.hash());
+        let txs = vec![tx0, tx1, tx2];
+        let results = bank.process_transactions(&txs);
+
+        // If multiple transactions attempt to deposit into the same account, only the first will
+        // succeed, even though such atomic adds are safe. A System Transfer `To` account should be
+        // given credit-only handling
+
+        assert_eq!(results[0], Ok(()));
+        assert_eq!(results[1], Err(TransactionError::AccountInUse));
+        assert_eq!(results[2], Err(TransactionError::AccountInUse));
+
+        // After credit-only account handling is implemented, the following checks should pass instead:
+        // assert_eq!(results[0], Ok(()));
+        // assert_eq!(results[1], Ok(()));
+        // assert_eq!(results[2], Ok(()));
+    }
+
     #[test]
     fn test_interleaving_locks() {
         let (genesis_block, mint_keypair) = create_genesis_block(3);

sdk/bpf/scripts/install.sh

@@ -19,9 +19,6 @@ download() {
     "--retry-connrefused"
     "--read-timeout=30"
   )
-  # Github release URLs 302 to AWS S3.  Sometimes that S3 URL returns 403
-  args+=("--retry-on-http-error=403")
-
   wget "${args[@]}"
 }
 
@@ -36,7 +33,7 @@ if [[ ! -r criterion-$machine-$version.md ]]; then
     mkdir criterion
     cd criterion
 
-    base=https://github.com/Snaipe/Criterion/releases/
+    base=https://github.com/Snaipe/Criterion/releases
     download $base/download/$version/$filename $filename mega
     tar --strip-components 1 -jxf $filename
     rm -rf $filename