gaspersic/solana
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Revert "GitBook: [master] 156 pages and 8 assets modified"

Browse Source 
This reverts commit 60320e6b6e.
This commit is contained in:
Michael Vines
2019-09-21 22:11:59 -07:00
parent 60320e6b6e
commit ae5a6a06bb
144 changed files with 5461 additions and 4122 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
book/src/proposals/README.md
View File

@@ -1,4 +0,0 @@
# Accepted Design Proposals
The following architectural proposals have been accepted by the Solana team, but are not yet fully implemented. The proposals may be implemented as described, implemented differently as issues in the designs become evident, or not implemented at all. If implemented, the descriptions will be moved from this section to earlier chapters in a future version of this book.

102
book/src/proposals/cluster-test-framework.md
View File

@@ -1,102 +0,0 @@
# Cluster Test Framework
This document proposes the Cluster Test Framework \(CTF\). CTF is a test harness that allows tests to execute against a local, in-process cluster or a deployed cluster.
## Motivation
The goal of CTF is to provide a framework for writing tests independent of where and how the cluster is deployed. Regressions can be captured in these tests and the tests can be run against deployed clusters to verify the deployment. The focus of these tests should be on cluster stability, consensus, fault tolerance, API stability.
Tests should verify a single bug or scenario, and should be written with the least amount of internal plumbing exposed to the test.
## Design Overview
Tests are provided an entry point, which is a `contact_info::ContactInfo` structure, and a keypair that has already been funded.
Each node in the cluster is configured with a `fullnode::ValidatorConfig` at boot time. At boot time this configuration specifies any extra cluster configuration required for the test. The cluster should boot with the configuration when it is run in-process or in a data center.
Once booted, the test will discover the cluster through a gossip entry point and configure any runtime behaviors via fullnode RPC.
## Test Interface
Each CTF test starts with an opaque entry point and a funded keypair. The test should not depend on how the cluster is deployed, and should be able to exercise all the cluster functionality through the publicly available interfaces.
```text
use crate::contact_info::ContactInfo;
use solana_sdk::signature::{Keypair, KeypairUtil};
pub fn test_this_behavior(
entry_point_info: &ContactInfo,
funding_keypair: &Keypair,
num_nodes: usize,
)
```
## Cluster Discovery
At test start, the cluster has already been established and is fully connected. The test can discover most of the available nodes over a few second.
```text
use crate::gossip_service::discover_nodes;
// Discover the cluster over a few seconds.
let cluster_nodes = discover_nodes(&entry_point_info, num_nodes);
```
## Cluster Configuration
To enable specific scenarios, the cluster needs to be booted with special configurations. These configurations can be captured in `fullnode::ValidatorConfig`.
For example:
```text
let mut validator_config = ValidatorConfig::default();
validator_config.rpc_config.enable_fullnode_exit = true;
let local = LocalCluster::new_with_config(
num_nodes,
10_000,
100,
&validator_config
);
```
## How to design a new test
For example, there is a bug that shows that the cluster fails when it is flooded with invalid advertised gossip nodes. Our gossip library and protocol may change, but the cluster still needs to stay resilient to floods of invalid advertised gossip nodes.
Configure the RPC service:
```text
let mut validator_config = ValidatorConfig::default();
validator_config.rpc_config.enable_rpc_gossip_push = true;
validator_config.rpc_config.enable_rpc_gossip_refresh_active_set = true;
```
Wire the RPCs and write a new test:
```text
pub fn test_large_invalid_gossip_nodes(
entry_point_info: &ContactInfo,
funding_keypair: &Keypair,
num_nodes: usize,
) {
let cluster = discover_nodes(&entry_point_info, num_nodes);
// Poison the cluster.
let client = create_client(entry_point_info.client_facing_addr(), FULLNODE_PORT_RANGE);
for _ in 0..(num_nodes * 100) {
client.gossip_push(
cluster_info::invalid_contact_info()
);
}
sleep(Durration::from_millis(1000));
// Force refresh of the active set.
for node in &cluster {
let client = create_client(node.client_facing_addr(), FULLNODE_PORT_RANGE);
client.gossip_refresh_active_set();
}
// Verify that spends still work.
verify_spends(&cluster);
}
```

71
book/src/proposals/cross-program-invocation.md
View File

@@ -1,71 +0,0 @@
# Cross-Program Invocation
## Problem
In today's implementation a client can create a transaction that modifies two accounts, each owned by a separate on-chain program:
```text
let message = Message::new(vec![
token_instruction::pay(&alice_pubkey),
acme_instruction::launch_missiles(&bob_pubkey),
]);
client.send_message(&[&alice_keypair, &bob_keypair], &message);
```
The current implementation does not, however, allow the `acme` program to conveniently invoke `token` instructions on the client's behalf:
```text
let message = Message::new(vec![
acme_instruction::pay_and_launch_missiles(&alice_pubkey, &bob_pubkey),
]);
client.send_message(&[&alice_keypair, &bob_keypair], &message);
```
Currently, there is no way to create instruction `pay_and_launch_missiles` that executes `token_instruction::pay` from the `acme` program. The workaround is to extend the `acme` program with the implementation of the `token` program, and create `token` accounts with `ACME_PROGRAM_ID`, which the `acme` program is permitted to modify. With that workaround, `acme` can modify token-like accounts created by the `acme` program, but not token accounts created by the `token` program.
## Proposed Solution
The goal of this design is to modify Solana's runtime such that an on-chain program can invoke an instruction from another program.
Given two on-chain programs `token` and `acme`, each implementing instructions `pay()` and `launch_missiles()` respectively, we would ideally like to implement the `acme` module with a call to a function defined in the `token` module:
```text
use token;
fn launch_missiles(keyed_accounts: &[KeyedAccount]) -> Result<()> {
...
}
fn pay_and_launch_missiles(keyed_accounts: &[KeyedAccount]) -> Result<()> {
token::pay(&keyed_accounts[1..])?;
launch_missiles(keyed_accounts)?;
}
```
The above code would require that the `token` crate be dynamically linked, so that a custom linker could intercept calls and validate accesses to `keyed_accounts`. That is, even though the client intends to modify both `token` and `acme` accounts, only `token` program is permitted to modify the `token` account, and only the `acme` program is permitted to modify the `acme` account.
Backing off from that ideal cross-program call, a slightly more verbose solution is to expose token's existing `process_instruction()` entrypoint to the acme program:
```text
use token_instruction;
fn launch_missiles(keyed_accounts: &[KeyedAccount]) -> Result<()> {
...
}
fn pay_and_launch_missiles(keyed_accounts: &[KeyedAccount]) -> Result<()> {
let alice_pubkey = keyed_accounts[1].key;
let instruction = token_instruction::pay(&alice_pubkey);
process_instruction(&instruction)?;
launch_missiles(keyed_accounts)?;
}
```
where `process_instruction()` is built into Solana's runtime and responsible for routing the given instruction to the `token` program via the instruction's `program_id` field. Before invoking `pay()`, the runtime must also ensure that `acme` didn't modify any accounts owned by `token`. It does this by calling `runtime::verify_instruction()` and then afterward updating all the `pre_*` variables to tentatively commit `acme`'s account modifications. After `pay()` completes, the runtime must again ensure that `token` didn't modify any accounts owned by `acme`. It should call `verify_instruction()` again, but this time with the `token` program ID. Lastly, after `pay_and_launch_missiles()` completes, the runtime must call `verify_instruction()` one more time, where it normally would, but using all updated `pre_*` variables. If executing `pay_and_launch_missiles()` up to `pay()` made no invalid account changes, `pay()` made no invalid changes, and executing from `pay()` until `pay_and_launch_missiles()` returns made no invalid changes, then the runtime can transitively assume `pay_and_launch_missiles()` as whole made no invalid account changes, and therefore commit all account modifications.
### Setting `KeyedAccount.is_signer`
When `process_instruction()` is invoked, the runtime must create a new `KeyedAccounts` parameter using the signatures from the _original_ transaction data. Since the `token` program is immutable and existed on-chain prior to the `acme` program, the runtime can safely treat the transaction signature as a signature of a transaction with a `token` instruction. When the runtime sees the given instruction references `alice_pubkey`, it looks up the key in the transaction to see if that key corresponds to a transaction signature. In this case it does and so sets `KeyedAccount.is_signer`, thereby authorizing the `token` program to modify Alice's account.

104
book/src/proposals/interchain-transaction-verification.md
View File

@@ -1,104 +0,0 @@
# Inter-chain Transaction Verification
## Problem
Inter-chain applications are not new to the digital asset ecosystem; in fact, even the smaller centralized exchanges still categorically dwarf all single chain dapps put together in terms of users and volume. They command massive valuations and have spent years effectively optimizing their core products for a broad range of end users. However, their basic operations center around mechanisms that require their users to unilaterally trust them, typically with little to no recourse or protection from accidental loss. This has led to the broader digital asset ecosystem being fractured along network lines because interoperability solutions typically:
* Are technically complex to fully implement
* Create unstable network scale incentive structures
* Require consistent and high level cooperation between stakeholders
## Proposed Solution
Simple Payment Verification \(SPV\) is a generic term for a range of different methodologies used by light clients on most major blockchain networks to verify aspects of the network state without the burden of fully storing and maintaining the chain itself. In most cases, this means relying on a form of hash tree to supply a proof of the presence of a given transaction in a certain block by comparing against a root hash in that block’s header or equivalent. This allows a light client or wallet to reach a probabilistic level of certainty about on-chain events by itself with a minimum of trust required with regard to network nodes.
Traditionally the process of assembling and validating these proofs is carried out off chain by nodes, wallets, or other clients, but it also offers a potential mechanism for inter-chain state verification. However, by moving the capability to validate SPV proofs on-chain as a smart contract while leveraging the archival properties inherent to the blockchain, it is possible to construct a system for programmatically detecting and verifying transactions on other networks without the involvement of any type of trusted oracle or complex multi-stage consensus mechanism. This concept is broadly generalisable to any network with an SPV mechanism and can even be operated bilaterally on other smart contract platforms, opening up the possibility of cheap, fast, inter-chain transfer of value without relying on collateral, hashlocks, or trusted intermediaries.
Opting to take advantage of well established and developmentally stable mechanisms already common to all major blockchains allows SPV based interoperability solutions to be dramatically simpler than orchestrated multi-stage approaches. As part of this, they dispense with the need for widely agreed upon cross chain communication standards and the large multi-party organizations that write them in favor of a set of discrete contract-based services that can be easily utilized by caller contracts through a common abstraction format. This will set the groundwork for a broad range of dapps and contracts able to interoperate across the variegated and every growing platform ecosystem.
## Terminology
SPV Program - Client-facing interface for the inter-chain SPV system, manages participant roles. SPV Engine - Validates transaction proofs, subset of the SPV Program. Client - The caller to the SPV Program, typically another solana contract. Prover - Party who generates proofs for transactions and submits them to the SPV Program. Transaction Proof - Created by Provers, contains a merkle proof, transaction, and blockheader reference. Merkle Proof - Basic SPV proof that validates the presence of a transaction in a certain block. Block Header - Represents the basic parameters and relative position of a given block. Proof Request - An order placed by a client for verification of transaction\(s\) by provers. Header Store - A data structure for storing and referencing ranges of block headers in proofs. Client Request - Transaction from the client to the SPV Program to trigger creation of a Proof Request. Sub-account - A Solana account owned by another contract account, without its own private key.
## Service
SPV Programs run as contracts deployed on the Solana network and maintain a type of public marketplace for SPV proofs that allows any party to submit both requests for proofs as well as proofs themselves for verification in response to requests. There will be multiple SPV Program instances active at any given time, at least one for each connected external network and potentially multiple instances per network. SPV program instances will be relatively consistent in their high level API and feature sets with some variation between currency platforms \(Bitcoin, Litecoin\) and smart contract platforms owing to the potential for verification of network state changes beyond simply transactions. In every case regardless of network, the SPV Program relies on an internal component called an SPV engine to provide stateless verification of the actual SPV proofs upon which the higher level client facing features and api are built. The SPV engine requires a network specific implementation, but allows easy extension of the larger inter-chain ecosystem by any team who chooses to carry out that implementation and drop it into the standard SPV program for deployment.
For purposes of Proof Requests, the requester is referred to as the program client, which in most if not all cases will be another Solana Contract. The client can choose to submit a request pertaining to a specific transaction or to include a broader filter that can apply to any of a range of parameters of a transaction including its inputs, outputs, and amount. For example, A client could submit a request for any transaction sent from a given address A to address B with the amount X after a certain time. This structure can be used in a range of applications, such as verifying a specific intended payment in the case of an atomic swap or detecting the movement of collateral assets for a loan.
Following submission of a Client Request, assuming that it is successfully validated, a proof request account is created by the SPV program to track the progress of the request. Provers use the account to specify the request they intend to fill in the proofs they submit for validation, at which point the SPV program validates those proofs and if successful, saves them to the account data of the request account. Clients can monitor the status of their requests and see any applicable transactions alongside their proofs by querying the account data of the request account. In future iterations when supported by Solana, this process will be simplified by contracts publishing events rather than requiring a polling style process as described.
## Implementation
The Solana Inter-chain SPV mechanism consists of the following components and participants:
### SPV engine
A contract deployed on Solana which statelessly verifies SPV proofs for the caller. It takes as arguments for validation:
* An SPV proof in the correct format of the blockchain associated with the program
* Reference\(s\) to the relevant block headers to compare that proof against
* The necessary parameters of the transaction to verify
If the proof in question is successfully validated, the SPV program saves proof
of that verification to the request account, which can be saved by the caller to
its account data or otherwise handled as necessary. SPV programs also expose
utilities and structs used for representation and validation of headers,
transactions, hashes, etc. on a chain by chain basis.
### SPV program
A contract deployed on Solana which coordinates and intermediates the interaction between Clients and Provers and manages the validation of requests, headers, proofs, etc. It is the primary point of access for Client contracts to access the inter-chain. SPV mechanism. It offers the following core features:
* Submit Proof Request - allows client to place a request for a specific proof or set of proofs
* Cancel Proof Request - allows client to invalidate a pending request
* Fill Proof Request - used by Provers to submit for validation a proof corresponding to a given Proof Request
The SPV program maintains a publicly available listing of valid pending Proof
Requests in its account data for the benefit of the Provers, who monitor it and
enclose references to target requests with their submitted proofs.
### Proof Request
A message sent by the Client to the SPV engine denoting a request for a proof of a specific transaction or set of transactions. Proof Requests can either manually specify a certain transaction by its hash or can elect to submit a filter that matches multiple transactions or classes of transactions. For example, a filter matching “any transaction from address xxx to address yyy” could be used to detect payment of a debt or settlement of an inter-chain swap. Likewise, a filter matching “any transaction from address xxx” could be used by a lending or synthetic token minting contract to monitor and react to changes in collateralization. Proof Requests are sent with a fee, which is disbursed by the SPV engine contract to the appropriate Prover once a proof matching that request is validated.
### Request Book
The public listing of valid, open Proof Requests available to provers to fill or for clients to cancel. Roughly analogous to an orderbook in an exchange, but with a single type of listing rather than two separate sides. It is stored in the account data of the SPV program.
### Proof
A proof of the presence of a given transaction in the blockchain in question. Proofs encompass both the actual merkle proof and reference\(s\) to a chain of valid sequential block headers. They are constructed and submitted by Provers in accordance with the specifications of the publicly available Proof Requests hosted on the request book by the SPV program. Upon Validation, they are saved to the account data of the relevant Proof Request, which can be used by the Client to monitor the state of the request.
### Client
The originator of a request for a transaction proof. Clients will most often be other contracts as parts of dapps or specific financial products like loans, swaps, escrow, etc. The client in any given verification process cycle initially submits a ClientRequest which communicates the parameters and fee and if successfully validated, results in the creation of a Proof Request account by the SPV program. The Client may also submit a CancelRequest referencing an active Proof Request in order to denote it as invalid for purposes of proof submission.
### Prover
The submitter of a proof that fills a Proof Request. Provers monitor the request book of the SPV program for outstanding Proof Requests and generate matching proofs, which they submit to the SPV program for validation. If the proof is accepted, the fee associated with the Proof Request in question is disbursed to the Prover. Provers typically operate as Solana Blockstreamer nodes that also have access to a Bitcoin node, which they use for purposes of constructing proofs and accessing block headers.
### Header Store
An account-based data structure used to maintain block headers for the purpose of inclusion in submitted proofs by reference to the header store account. header stores can be maintained by independent entities, since header chain validation is a component of the SPV program proof validation mechanism. Fees that are paid out by Proof Requests to Provers are split between the submitter of the merkle proof itself and the header store that is referenced in the submitted proof. Due to the current inability to grow already allocated account data capacity, the use case necessitates a data structure that can grow indefinitely without rebalancing. Sub-accounts are accounts owned by the SPV program without their own private keys that are used for storage by allocating blockheaders to their account data. Multiple potential approaches to the implementation of the header store system are feasible:
Store Headers in program sub-accounts indexed by Public address:
* Each sub-account holds one header and has a public key matching the blockhash
* Requires same number of account data lookups as confirmations per verification
* Limit on number of confirmations \(15-20\) via max transaction data ceiling
* No network-wide duplication of individual headers
Linked List of multiple sub-accounts storing headers:
* Maintain sequential index of storage accounts, many headers per storage account
* Max 2 account data lookups for &gt;99.9% of verifications \(1 for most\)
* Compact sequential data address format allows any number of confirmations and fast lookups
* Facilitates network-wide header duplication inefficiencies

137
book/src/proposals/ledger-replication-to-implement.md
View File

@@ -1,137 +0,0 @@
# Ledger Replication
Replication behavior yet to be implemented.
## Storage epoch
The storage epoch should be the number of slots which results in around 100GB-1TB of ledger to be generated for replicators to store. Replicators will start storing ledger when a given fork has a high probability of not being rolled back.
## Validator behavior
1. Every NUM\_KEY\_ROTATION\_TICKS it also validates samples received from
replicators. It signs the PoH hash at that point and uses the following
algorithm with the signature as the input:
* The low 5 bits of the first byte of the signature creates an index into
another starting byte of the signature.
* The validator then looks at the set of storage proofs where the byte of
the proof's sha state vector starting from the low byte matches exactly
with the chosen byte\(s\) of the signature.
* If the set of proofs is larger than the validator can handle, then it
increases to matching 2 bytes in the signature.
* Validator continues to increase the number of matching bytes until a
workable set is found.
* It then creates a mask of valid proofs and fake proofs and sends it to
the leader. This is a storage proof confirmation transaction.
2. After a lockout period of NUM\_SECONDS\_STORAGE\_LOCKOUT seconds, the
validator then submits a storage proof claim transaction which then causes the
distribution of the storage reward if no challenges were seen for the proof to
the validators and replicators party to the proofs.
## Replicator behavior
1. The replicator then generates another set of offsets which it submits a fake
proof with an incorrect sha state. It can be proven to be fake by providing the
seed for the hash result.
* A fake proof should consist of a replicator hash of a signature of a PoH
value. That way when the replicator reveals the fake proof, it can be
verified on chain.
2. The replicator monitors the ledger, if it sees a fake proof integrated, it
creates a challenge transaction and submits it to the current leader. The
transacation proves the validator incorrectly validated a fake storage proof.
The replicator is rewarded and the validator's staking balance is slashed or
frozen.
## Storage proof contract logic
Each replicator and validator will have their own storage account. The validator's account would be separate from their gossip id similiar to their vote account. These should be implemented as two programs one which handles the validator as the keysigner and one for the replicator. In that way when the programs reference other accounts, they can check the program id to ensure it is a validator or replicator account they are referencing.
### SubmitMiningProof
```text
SubmitMiningProof {
slot: u64,
sha_state: Hash,
signature: Signature,
};
keys = [replicator_keypair]
```
Replicators create these after mining their stored ledger data for a certain hash value. The slot is the end slot of the segment of ledger they are storing, the sha\_state the result of the replicator using the hash function to sample their encrypted ledger segment. The signature is the signature that was created when they signed a PoH value for the current storage epoch. The list of proofs from the current storage epoch should be saved in the account state, and then transfered to a list of proofs for the previous epoch when the epoch passes. In a given storage epoch a given replicator should only submit proofs for one segment.
The program should have a list of slots which are valid storage mining slots. This list should be maintained by keeping track of slots which are rooted slots in which a significant portion of the network has voted on with a high lockout value, maybe 32-votes old. Every SLOTS\_PER\_SEGMENT number of slots would be added to this set. The program should check that the slot is in this set. The set can be maintained by receiving a AdvertiseStorageRecentBlockHash and checking with its bank/Tower BFT state.
The program should do a signature verify check on the signature, public key from the transaction submitter and the message of the previous storage epoch PoH value.
### ProofValidation
```text
ProofValidation {
proof_mask: Vec<ProofStatus>,
}
keys = [validator_keypair, replicator_keypair(s) (unsigned)]
```
A validator will submit this transaction to indicate that a set of proofs for a given segment are valid/not-valid or skipped where the validator did not look at it. The keypairs for the replicators that it looked at should be referenced in the keys so the program logic can go to those accounts and see that the proofs are generated in the previous epoch. The sampling of the storage proofs should be verified ensuring that the correct proofs are skipped by the validator according to the logic outlined in the validator behavior of sampling.
The included replicator keys will indicate the the storage samples which are being referenced; the length of the proof\_mask should be verified against the set of storage proofs in the referenced replicator account\(s\), and should match with the number of proofs submitted in the previous storage epoch in the state of said replicator account.
### ClaimStorageReward
```text
ClaimStorageReward {
}
keys = [validator_keypair or replicator_keypair, validator/replicator_keypairs (unsigned)]
```
Replicators and validators will use this transaction to get paid tokens from a program state where SubmitStorageProof, ProofValidation and ChallengeProofValidations are in a state where proofs have been submitted and validated and there are no ChallengeProofValidations referencing those proofs. For a validator, it should reference the replicator keypairs to which it has validated proofs in the relevant epoch. And for a replicator it should reference validator keypairs for which it has validated and wants to be rewarded.
### ChallengeProofValidation
```text
ChallengeProofValidation {
proof_index: u64,
hash_seed_value: Vec<u8>,
}
keys = [replicator_keypair, validator_keypair]
```
This transaction is for catching lazy validators who are not doing the work to validate proofs. A replicator will submit this transaction when it sees a validator has approved a fake SubmitMiningProof transaction. Since the replicator is a light client not looking at the full chain, it will have to ask a validator or some set of validators for this information maybe via RPC call to obtain all ProofValidations for a certain segment in the previous storage epoch. The program will look in the validator account state see that a ProofValidation is submitted in the previous storage epoch and hash the hash\_seed\_value and see that the hash matches the SubmitMiningProof transaction and that the validator marked it as valid. If so, then it will save the challenge to the list of challenges that it has in its state.
### AdvertiseStorageRecentBlockhash
```text
AdvertiseStorageRecentBlockhash {
hash: Hash,
slot: u64,
}
```
Validators and replicators will submit this to indicate that a new storage epoch has passed and that the storage proofs which are current proofs should now be for the previous epoch. Other transactions should check to see that the epoch that they are referencing is accurate according to current chain state.

52
book/src/proposals/rent.md
View File

@@ -1,52 +0,0 @@
# Rent
Accounts on Solana may have owner-controlled state \(`Account::data`\) that's separate from the account's balance \(`Account::lamports`\). Since validators on the network need to maintain a working copy of this state in memory, the network charges a time-and-space based fee for this resource consumption, also known as Rent.
## Two-tiered rent regime
Accounts which maintain a minimum balance equivalent to 2 years of rent payments are exempt. Accounts whose balance falls below this threshold are charged rent at a rate specified in genesis, in lamports per kilobyte-year. The network charges rent on a per-epoch basis, in credit for the next epoch \(but in arrears when necessary\), and `Account::rent_epoch` keeps track of the next time rent should be collected from the account.
## Collecting rent
Rent is due at account creation time for one epoch's worth of time, and the new account has `Account::rent_epoch` of `current_epoch + 1`. After that, the bank deducts rent from accounts during normal transaction processing as part of the load phase.
If the account is in the exempt regime, `Account::rent_epoch` is simply pushed to `current_epoch + 1`.
If the account is non-exempt, the difference between the next epoch and `Account::rent_epoch` is used to calculate the amount of rent owed by this account \(via `Rent::due()`\). Any fractional lamports of the calculation are truncated. Rent due is deducted from `Account::lamports` and `Account::rent_epoch` is updated to the next epoch. If the amount of rent due is less than one lamport, no changes are made to the account.
Accounts whose balance is insufficient to satisfy the rent that would be due simply fail to load.
A percentage of the rent collected is destroyed. The rest is distributed to validator accounts by stake weight, a la transaction fees, at the end of every slot.
## Credit only
Credit only accounts are treated as a special case. They are loaded as if rent were due, but updates to their state may be delayed until the end of the slot, when credits are paid.
## Design considerations, others considered
Under this design, it is possible to have accounts that linger, never get touched, and never have to pay rent. `Noop` instructions that name these accounts can be used to "garbage collect", but it'd also be possible for accounts that never get touched to migrate out of a validator's working set, thereby reducing memory consumption and obviating the need to charge rent.
### Ad-hoc collection
Collecting rent on an as-needed basis \(i.e. whenever accounts were loaded/accessed\) was considered. The issues with such an approach are:
* accounts loaded as "credit only" for a transaction could very reasonably be expected to have rent due,
but would not be debitable during any such transaction
* a mechanism to "beat the bushes" \(i.e. go find accounts that need to pay rent\) is desirable,
lest accounts that are loaded infrequently get a free ride
### System instruction for collecting rent
Collecting rent via a system instruction was considered, as it would naturally have distributed rent to active and stake-weighted nodes and could have been done incrementally. However:
* it would have adversely affected network throughput
* it would require special-casing by the runtime, as accounts with non-SystemProgram owners may be debited by this instruction
* someone would have to issue the transactions
### Account scans on every epoch
Scanning the entire Bank for accounts that owe rent at the beginning of each epoch was considered. This would have been an expensive operation, and would require that the entire current state of the network be present on every validator at the beginning of each epoch.

110
book/src/proposals/simple-payment-and-state-verification.md
View File

@@ -1,110 +0,0 @@
# Simple Payment and State Verification
It is often useful to allow low resourced clients to participate in a Solana cluster. Be this participation economic or contract execution, verification that a client's activity has been accepted by the network is typically expensive. This proposal lays out a mechanism for such clients to confirm that their actions have been committed to the ledger state with minimal resource expenditure and third-party trust.
## A Naive Approach
Validators store the signatures of recently confirmed transactions for a short period of time to ensure that they are not processed more than once. Validators provide a JSON RPC endpoint, which clients can use to query the cluster if a transaction has been recently processed. Validators also provide a PubSub notification, whereby a client registers to be notified when a given signature is observed by the validator. While these two mechanisms allow a client to verify a payment, they are not a proof and rely on completely trusting a fullnode.
We will describe a way to minimize this trust using Merkle Proofs to anchor the fullnode's response in the ledger, allowing the client to confirm on their own that a sufficient number of their preferred validators have confirmed a transaction. Requiring multiple validator attestations further reduces trust in the fullnode, as it increases both the technical and economic difficulty of compromising several other network participants.
## Light Clients
A 'light client' is a cluster participant that does not itself run a fullnode. This light client would provide a level of security greater than trusting a remote fullnode, without requiring the light client to spend a lot of resources verifying the ledger.
Rather than providing transaction signatures directly to a light client, the fullnode instead generates a Merkle Proof from the transaction of interest to the root of a Merkle Tree of all transactions in the including block. This Merkle Root is stored in a ledger entry which is voted on by validators, providing it consensus legitimacy. The additional level of security for a light client depends on an initial canonical set of validators the light client considers to be the stakeholders of the cluster. As that set is changed, the client can update its internal set of known validators with [receipts](simple-payment-and-state-verification.md#receipts). This may become challenging with a large number of delegated stakes.
Fullnodes themselves may want to use light client APIs for performance reasons. For example, during the initial launch of a fullnode, the fullnode may use a cluster provided checkpoint of the state and verify it with a receipt.
## Receipts
A receipt is a minimal proof that; a transaction has been included in a block, that the block has been voted on by the client's preferred set of validators and that the votes have reached the desired confirmation depth.
The receipts for both state and payments start with a Merkle Path from the value into a Bank-Merkle that has been voted on and included in the ledger. A chain of PoH Entries containing subsequent validator votes, deriving from the Bank-Merkle, is the confirmation proof.
Clients can examine this ledger data and compute the finality using Solana's fork selection rules.
### Payment Merkle Path
A payment receipt is a data structure that contains a Merkle Path from a transaction to the required set of validator votes.
An Entry-Merkle is a Merkle Root including all transactions in the entry, sorted by signature.
![Block Merkle Diagram](https://github.com/solana-labs/solana/tree/a4e72ac0375e9521305f992b89bba841ad152eda/book/src/img/spv-block-merkle.svg)
A Block-Merkle is a Merkle root of all the Entry-Merkles sequenced in the block. Transaction status is necessary for the receipt because the state receipt is constructed for the block. Two transactions over the same state can appear in the block, and therefore, there is no way to infer from just the state whether a transaction that is committed to the ledger has succeeded or failed in modifying the intended state. It may not be necessary to encode the full status code, but a single status bit to indicate the transaction's success.
### State Merkle Path
A state receipt provides a confirmation that a specific state is committed at the end of the block. Inter-block state transitions do not generate a receipt.
For example:
* A sends 5 Lamports to B
* B spends 5 Lamports
* C sends 5 Lamports to A
At the end of the block, A and B are in the exact same starting state, and any state receipt would point to the same value for A or B.
The Bank-Merkle is computed from the Merkle Tree of the new state changes, along with the Previous Bank-Merkle, and the Block-Merkle.
![Bank Merkle Diagram](https://github.com/solana-labs/solana/tree/a4e72ac0375e9521305f992b89bba841ad152eda/book/src/img/spv-bank-merkle.svg)
A state receipt contains only the state changes occurring in the block. A direct Merkle Path to the current Bank-Merkle guarantees the state value at that bank hash, but it cannot be used to generate a “current” receipt to the latest state if the state modification occurred in some previous block. There is no guarantee that the path provided by the validator is the latest one available out of all the previous Bank-Merkles.
Clients that want to query the chain for a receipt of the "latest" state would need to create a transaction that would update the Merkle Path for that account, such as a credit of 0 Lamports.
### Validator Votes
Leaders should coalesce the validator votes by stake weight into a single entry. This will reduce the number of entries necessary to create a receipt.
### Chain of Entries
A receipt has a PoH link from the payment or state Merkle Path root to a list of consecutive validation votes.
It contains the following:
* State -&gt; Bank-Merkle
or
* Transaction -&gt; Entry-Merkle -&gt; Block-Merkle -&gt; Bank-Merkle
And a vector of PoH entries:
* Validator vote entries
* Ticks
* Light entries
```text
/// This Entry definition skips over the transactions and only contains the
/// hash of the transactions used to modify PoH.
LightEntry {
/// The number of hashes since the previous Entry ID.
pub num_hashes: u64,
/// The SHA-256 hash `num_hashes` after the previous Entry ID.
hash: Hash,
/// The Merkle Root of the transactions encoded into the Entry.
entry_hash: Hash,
}
```
The light entries are reconstructed from Entries and simply show the entry Merkle Root that was mixed in to the PoH hash, instead of the full transaction set.
Clients do not need the starting vote state. The [fork selection](https://github.com/solana-labs/solana/tree/a4e72ac0375e9521305f992b89bba841ad152eda/book/src/book/src/fork-selection.md) algorithm is defined such that only votes that appear after the transaction provide finality for the transaction, and finality is independent of the starting state.
### Verification
A light client that is aware of the supermajority set validators can verify a receipt by following the Merkle Path to the PoH chain. The Bank-Merkle is the Merkle Root and will appear in votes included in an Entry. The light client can simulate [fork selection](https://github.com/solana-labs/solana/tree/a4e72ac0375e9521305f992b89bba841ad152eda/book/src/book/src/fork-selection.md) for the consecutive votes and verify that the receipt is confirmed at the desired lockout threshold.
### Synthetic State
Synthetic state should be computed into the Bank-Merkle along with the bank generated state.
For example:
* Epoch validator accounts and their stakes and weights.
* Computed fee rates
These values should have an entry in the Bank-Merkle. They should live under known accounts, and therefore have an exact address in the Merkle Path.

45
book/src/proposals/snapshot-verification.md
View File

@@ -1,45 +0,0 @@
# Snapshot Verification
## Problem
When a validator boots up from a snapshot, it needs a way to verify the account set matches what the rest of the network sees quickly. A potential attacker could give the validator an incorrect state, and then try to convince it to accept a transaction that would otherwise be rejected.
## Solution
Currently the bank hash is derived from hashing the delta state of the accounts in a slot which is then combined with the previous bank hash value. The problem with this is that the list of hashes will grow on the order of the number of slots processed by the chain and become a burden to both transmit and verify successfully.
Another naive method could be to create a merkle tree of the account state. This has the downside that with each account update which removes an account state from the tree, the merkle tree would have to be recomputed from the entire account state of all live accounts in the system.
To verify the snapshot, we propose the following:
On account store hash the following data:
* Account owner
* Account data
* Account pubkey
* Account lamports balance
* Fork the account is stored on
Use this resulting hash value as input to an expansion function which expands the hash value into an image value. The function will create a 440 byte block of data where the first 32 bytes are the hash value, and the next 440 - 32 bytes are generated from a Chacha RNG with the hash as the seed.
The account images are then combined with xor. The previous account value will be xored into the state and the new account value also xored into the state.
Voting and sysvar hash values occur with the hash of the resulting full image value.
On validator boot, when it loads from a snapshot, it would verify the hash value with the accounts set. It would then use SPV to display the percentage of the network that voted for the hash value given.
The resulting value can be verified by a validator to be the result of xoring all current account states together.
An attack on the xor state could be made to influence its value:
Thus the 440 byte image size comes from this paper, avoiding xor collision with 0 \(or thus any other given bit pattern\): \[[https://link.springer.com/content/pdf/10.1007%2F3-540-45708-9\_19.pdf](https://link.springer.com/content/pdf/10.1007%2F3-540-45708-9_19.pdf)\]
The math provides 128 bit security in this case:
```text
O(k * 2^(n/(1+lg(k)))
k=2^40 accounts
n=440
2^(40) * 2^(448 * 8 / 41) ~= O(2^(128))
```

95
book/src/proposals/staking-rewards.md
View File

@@ -1,95 +0,0 @@
# Staking Rewards
A Proof of Stake \(PoS\), \(i.e. using in-protocol asset, SOL, to provide secure consensus\) design is outlined here. Solana implements a proof of stake reward/security scheme for validator nodes in the cluster. The purpose is threefold:
* Align validator incentives with that of the greater cluster through
skin-in-the-game deposits at risk
* Avoid 'nothing at stake' fork voting issues by implementing slashing rules
aimed at promoting fork convergence
* Provide an avenue for validator rewards provided as a function of validator
participation in the cluster.
While many of the details of the specific implementation are currently under consideration and are expected to come into focus through specific modeling studies and parameter exploration on the Solana testnet, we outline here our current thinking on the main components of the PoS system. Much of this thinking is based on the current status of Casper FFG, with optimizations and specific attributes to be modified as is allowed by Solana's Proof of History \(PoH\) blockchain data structure.
## General Overview
Solana's ledger validation design is based on a rotating, stake-weighted selected leader broadcasting transactions in a PoH data structure to validating nodes. These nodes, upon receiving the leader's broadcast, have the opportunity to vote on the current state and PoH height by signing a transaction into the PoH stream.
To become a Solana validator, a fullnode must deposit/lock-up some amount of SOL in a contract. This SOL will not be accessible for a specific time period. The precise duration of the staking lockup period has not been determined. However we can consider three phases of this time for which specific parameters will be necessary:
* _Warm-up period_: which SOL is deposited and inaccessible to the node,
however PoH transaction validation has not begun. Most likely on the order of
days to weeks
* _Validation period_: a minimum duration for which the deposited SOL will be
inaccessible, at risk of slashing \(see slashing rules below\) and earning
rewards for the validator participation. Likely duration of months to a
year.
* _Cool-down period_: a duration of time following the submission of a
'withdrawal' transaction. During this period validation responsibilities have
been removed and the funds continue to be inaccessible. Accumulated rewards
should be delivered at the end of this period, along with the return of the
initial deposit.
Solana's trustless sense of time and ordering provided by its PoH data structure, along with its [turbine](https://www.youtube.com/watch?v=qt_gDRXHrHQ&t=1s) data broadcast and transmission design, should provide sub-second transaction confirmation times that scale with the log of the number of nodes in the cluster. This means we shouldn't have to restrict the number of validating nodes with a prohibitive 'minimum deposits' and expect nodes to be able to become validators with nominal amounts of SOL staked. At the same time, Solana's focus on high-throughput should create incentive for validation clients to provide high-performant and reliable hardware. Combined with potential a minimum network speed threshold to join as a validation-client, we expect a healthy validation delegation market to emerge. To this end, Solana's testnet will lead into a "Tour de SOL" validation-client competition, focusing on throughput and uptime to rank and reward testnet validators.
## Slashing rules
Unlike Proof of Work \(PoW\) where off-chain capital expenses are already deployed at the time of block construction/voting, PoS systems require capital-at-risk to prevent a logical/optimal strategy of multiple chain voting. We intend to implement slashing rules which, if broken, result some amount of the offending validator's deposited stake to be removed from circulation. Given the ordering properties of the PoH data structure, we believe we can simplify our slashing rules to the level of a voting lockout time assigned per vote.
I.e. Each vote has an associated lockout time \(PoH duration\) that represents a duration by any additional vote from that validator must be in a PoH that contains the original vote, or a portion of that validator's stake is slashable. This duration time is a function of the initial vote PoH count and all additional vote PoH counts. It will likely take the form:
Lockouti\(PoHi, PoHj\) = PoHj + K \* exp\(\(PoHj - PoHi\) / K\)
Where PoHi is the height of the vote that the lockout is to be applied to and PoHj is the height of the current vote on the same fork. If the validator submits a vote on a different PoH fork on any PoHk where k &gt; j &gt; i and PoHk &lt; Lockout\(PoHi, PoHj\), then a portion of that validator's stake is at risk of being slashed.
In addition to the functional form lockout described above, early implementation may be a numerical approximation based on a First In, First Out \(FIFO\) data structure and the following logic:
* FIFO queue holding 32 votes per active validator
* new votes are pushed on top of queue \(`push_front`\)
* expired votes are popped off top \(`pop_front`\)
* as votes are pushed into the queue, the lockout of each queued vote doubles
* votes are removed from back of queue if `queue.len() > 32`
* the earliest and latest height that has been removed from the back of the
queue should be stored
It is likely that a reward will be offered as a % of the slashed amount to any node that submits proof of this slashing condition being violated to the PoH.
### Partial Slashing
In the schema described so far, when a validator votes on a given PoH stream, they are committing themselves to that fork for a time determined by the vote lockout. An open question is whether validators will be hesitant to begin voting on an available fork if the penalties are perceived too harsh for an honest mistake or flipped bit.
One way to address this concern would be a partial slashing design that results in a slashable amount as a function of either:
1. the fraction of validators, out of the total validator pool, that were also
slashed during the same time period \(ala Casper\)
2. the amount of time since the vote was cast \(e.g. a linearly increasing % of
total deposited as slashable amount over time\), or both.
This is an area currently under exploration
## Penalties
As discussed in the [Economic Design](../implemented-proposals/ed_overview/) section, annual validator interest rates are to be specified as a function of total percentage of circulating supply that has been staked. The cluster rewards validators who are online and actively participating in the validation process throughout the entirety of their _validation period_. For validators that go offline/fail to validate transactions during this period, their annual reward is effectively reduced.
Similarly, we may consider an algorithmic reduction in a validator's active amount staked amount in the case that they are offline. I.e. if a validator is inactive for some amount of time, either due to a partition or otherwise, the amount of their stake that is considered ‘active’ \(eligible to earn rewards\) may be reduced. This design would be structured to help long-lived partitions to eventually reach finality on their respective chains as the % of non-voting total stake is reduced over time until a super-majority can be achieved by the active validators in each partition. Similarly, upon re-engaging, the ‘active’ amount staked will come back online at some defined rate. Different rates of stake reduction may be considered depending on the size of the partition/active set.

36
book/src/proposals/validator-proposal.md
View File

@@ -1,36 +0,0 @@
# Validator
## History
When we first started Solana, the goal was to de-risk our TPS claims. We knew that between optimistic concurrency control and sufficiently long leader slots, that PoS consensus was not the biggest risk to TPS. It was GPU-based signature verification, software pipelining and concurrent banking. Thus, the TPU was born. After topping 100k TPS, we split the team into one group working toward 710k TPS and another to flesh out the validator pipeline. Hence, the TVU was born. The current architecture is a consequence of incremental development with that ordering and project priorities. It is not a reflection of what we ever believed was the most technically elegant cross-section of those technologies. In the context of leader rotation, the strong distinction between leading and validating is blurred.
## Difference between validating and leading
The fundamental difference between the pipelines is when the PoH is present. In a leader, we process transactions, removing bad ones, and then tag the result with a PoH hash. In the validator, we verify that hash, peel it off, and process the transactions in exactly the same way. The only difference is that if a validator sees a bad transaction, it can't simply remove it like the leader does, because that would cause the PoH hash to change. Instead, it rejects the whole block. The other difference between the pipelines is what happens _after_ banking. The leader broadcasts entries to downstream validators whereas the validator will have already done that in RetransmitStage, which is a confirmation time optimization. The validation pipeline, on the other hand, has one last step. Any time it finishes processing a block, it needs to weigh any forks it's observing, possibly cast a vote, and if so, reset its PoH hash to the block hash it just voted on.
## Proposed Design
We unwrap the many abstraction layers and build a single pipeline that can toggle leader mode on whenever the validator's ID shows up in the leader schedule.
![Validator block diagram](https://github.com/solana-labs/solana/tree/a4e72ac0375e9521305f992b89bba841ad152eda/book/src/img/validator-proposal.svg)
## Notable changes
* No threads are shut down to switch out of leader mode. Instead, FetchStage
should forward transactions to the next leader.
* Hoist FetchStage and BroadcastStage out of TPU
* Blocktree renamed to Blockstore
* BankForks renamed to Banktree
* TPU moves to new socket-free crate called solana-tpu.
* TPU's BankingStage absorbs ReplayStage
* TVU goes away
* New RepairStage absorbs Blob Fetch Stage and repair requests
* JSON RPC Service is optional - used for debugging. It should instead be part
of a separate `solana-blockstreamer` executable.
* New MulticastStage absorbs retransmit part of RetransmitStage
* MulticastStage downstream of Blockstore

111
book/src/proposals/vote-signing-to-implement.md
View File

@@ -1,111 +0,0 @@
# Secure Vote Signing
## Secure Vote Signing
This design describes additional vote signing behavior that will make the process more secure.
Currently, Solana implements a vote-signing service that evaluates each vote to ensure it does not violate a slashing condition. The service could potentially have different variations, depending on the hardware platform capabilities. In particular, it could be used in conjunction with a secure enclave \(such as SGX\). The enclave could generate an asymmetric key, exposing an API for user \(untrusted\) code to sign the vote transactions, while keeping the vote-signing private key in its protected memory.
The following sections outline how this architecture would work:
### Message Flow
1. The node initializes the enclave at startup
* The enclave generates an asymmetric key and returns the public key to the
node
* The keypair is ephemeral. A new keypair is generated on node bootup. A
new keypair might also be generated at runtime based on some TBD
criteria.
* The enclave returns its attestation report to the node
2. The node performs attestation of the enclave \(e.g using Intel's IAS APIs\)
* The node ensures that the Secure Enclave is running on a TPM and is
signed by a trusted party
3. The stakeholder of the node grants ephemeral key permission to use its stake.
This process is TBD.
4. The node's untrusted, non-enclave software calls trusted enclave software
using its interface to sign transactions and other data.
* In case of vote signing, the node needs to verify the PoH. The PoH
verification is an integral part of signing. The enclave would be
presented with some verifiable data to check before signing the vote.
* The process of generating the verifiable data in untrusted space is TBD
### PoH Verification
1. When the node votes on an en entry `X`, there's a lockout period `N`, for
which it cannot vote on a fork that does not contain `X` in its history.
2. Every time the node votes on the derivative of `X`, say `X+y`, the lockout
period for `X` increases by a factor `F` \(i.e. the duration node cannot vote on
a fork that does not contain `X` increases\).
* The lockout period for `X+y` is still `N` until the node votes again.
3. The lockout period increment is capped \(e.g. factor `F` applies maximum 32
times\).
4. The signing enclave must not sign a vote that violates this policy. This
means
* Enclave is initialized with `N`, `F` and `Factor cap`
* Enclave stores `Factor cap` number of entry IDs on which the node had
previously voted
* The sign request contains the entry ID for the new vote
* Enclave verifies that new vote's entry ID is on the correct fork
\(following the rules \#1 and \#2 above\)
### Ancestor Verification
This is alternate, albeit, less certain approach to verifying voting fork. 1. The validator maintains an active set of nodes in the cluster 2. It observes the votes from the active set in the last voting period 3. It stores the ancestor/last\_tick at which each node voted 4. It sends new vote request to vote-signing service
* It includes previous votes from nodes in the active set, and their
corresponding ancestors
1. The signer checks if the previous votes contains a vote from the validator,
and the vote ancestor matches with majority of the nodes
* It signs the new vote if the check is successful
* It asserts \(raises an alarm of some sort\) if the check is unsuccessful
The premise is that the validator can be spoofed at most once to vote on incorrect data. If someone hijacks the validator and submits a vote request for bogus data, that vote will not be included in the PoH \(as it'll be rejected by the cluster\). The next time the validator sends a request to sign the vote, the signing service will detect that validator's last vote is missing \(as part of
## 5 above\).
### Fork determination
Due to the fact that the enclave cannot process PoH, it has no direct knowledge of fork history of a submitted validator vote. Each enclave should be initiated with the current _active set_ of public keys. A validator should submit its current vote along with the votes of the active set \(including itself\) that it observed in the slot of its previous vote. In this way, the enclave can surmise the votes accompanying the validator's previous vote and thus the fork being voted on. This is not possible for the validator's initial submitted vote, as it will not have a 'previous' slot to reference. To account for this, a short voting freeze should apply until the second vote is submitted containing the votes within the active set, along with it's own vote, at the height of the initial vote.
### Enclave configuration
A staking client should be configurable to prevent voting on inactive forks. This mechanism should use the client's known active set `N_active` along with a threshold vote `N_vote` and a threshold depth `N_depth` to determine whether or not to continue voting on a submitted fork. This configuration should take the form of a rule such that the client will only vote on a fork if it observes more than `N_vote` at `N_depth`. Practically, this represents the client from confirming that it has observed some probability of economic finality of the submitted fork at a depth where an additional vote would create a lockout for an undesirable amount of time if that fork turns out not to be live.
### Challenges
1. Generation of verifiable data in untrusted space for PoH verification in the
enclave.
2. Need infrastructure for granting stake to an ephemeral key.