Move stragglers into the book
The stuff added between the time we switched to proposals/ and the git-revert
This commit is contained in:
Greg Fitzgerald
Grimes
30
book/art/passive-staking-callflow.msc
Normal file
30
book/art/passive-staking-callflow.msc
Normal file
@@ -0,0 +1,30 @@
|
||||
msc {
|
||||
hscale="2.2";
|
||||
VoteSigner,
|
||||
Validator,
|
||||
Cluster,
|
||||
StakerX,
|
||||
StakerY;
|
||||
|
||||
|||;
|
||||
Validator box Validator [label="boot.."];
|
||||
|
||||
VoteSigner <:> Validator [label="register\n\n(optional)"];
|
||||
Validator => Cluster [label="VoteState::Initialize(VoteSigner)"];
|
||||
StakerX => Cluster [label="StakeState::Delegate(Validator)"];
|
||||
StakerY => Cluster [label="StakeState::Delegate(Validator)"];
|
||||
|
||||
|||;
|
||||
Validator box Cluster [label="\nvalidate\n"];
|
||||
Validator => VoteSigner [label="sign(vote)"];
|
||||
VoteSigner >> Validator [label="signed vote"];
|
||||
|
||||
Validator => Cluster [label="gossip(vote)"];
|
||||
...;
|
||||
... ;
|
||||
Validator abox Validator [label="\nmax\nlockout\n"];
|
||||
|||;
|
||||
StakerX => Cluster [label="StakeState::RedeemCredits()"];
|
||||
StakerY => Cluster [label="StakeState::RedeemCredits()"] ;
|
||||
|
||||
}
|
||||
@@ -56,6 +56,9 @@
|
||||
- [References](ed_references.md)
|
||||
- [Cluster Test Framework](cluster-test-framework.md)
|
||||
- [Testing Programs](testing-programs.md)
|
||||
- [Credit-only Accounts](credit-only-credit-debit-accounts.md)
|
||||
- [Solana Installer](installer.md)
|
||||
- [Testnet Participation](testnet-participation.md)
|
||||
|
||||
- [Implemented Design Proposals](implemented-proposals.md)
|
||||
- [Leader-to-Leader Transition](leader-leader-transition.md)
|
||||
|
||||
141
book/src/credit-only-credit-debit-accounts.md
Normal file
141
book/src/credit-only-credit-debit-accounts.md
Normal file
@@ -0,0 +1,141 @@
|
||||
# Credit-Only Accounts
|
||||
|
||||
This design covers the handling of credit-only and credit-debit accounts in the
|
||||
[runtime](runtime.md). Accounts already distinguish themselves as credit-only or
|
||||
credit-debit based on the program ID specified by the transaction's instruction.
|
||||
Programs must treat accounts that are not owned by them as credit-only.
|
||||
|
||||
To identify credit-only accounts by program id would require the account to be
|
||||
fetched and loaded from disk. This operation is expensive, and while it is
|
||||
occurring, the runtime would have to reject any transactions referencing the same
|
||||
account.
|
||||
|
||||
The proposal introduces a `num_readonly_accounts` field to the transaction
|
||||
structure, and removes the `program_ids` dedicated vector for program accounts.
|
||||
|
||||
This design doesn't change the runtime transaction processing rules.
|
||||
Programs still can't write or spend accounts that they do not own, but it
|
||||
allows the runtime to optimistically take the correct lock for each account
|
||||
specified in the transaction before loading the accounts from storage.
|
||||
|
||||
Accounts selected as credit-debit by the transaction can still be treated as
|
||||
credit-only by the instructions.
|
||||
|
||||
## Runtime handling
|
||||
|
||||
credit-only accounts have the following properties:
|
||||
|
||||
* Can be deposited into: Deposits can be implemented as a simple `atomic_add`.
|
||||
|
||||
* credit-only access to account data.
|
||||
|
||||
Instructions that debit or modify the credit-only account data will fail.
|
||||
|
||||
## Account Lock Optimizations
|
||||
|
||||
The Accounts module keeps track of current locked accounts in the runtime,
|
||||
which separates credit-only accounts from the credit-debit accounts. The credit-only
|
||||
accounts can be cached in memory and shared between all the threads executing
|
||||
transactions.
|
||||
|
||||
The current runtime can't predict whether an account is credit-only or credit-debit when
|
||||
the transaction account keys are locked at the start of the transaction
|
||||
processing pipeline. Accounts referenced by the transaction have not been
|
||||
loaded from the disk yet.
|
||||
|
||||
An ideal design would cache the credit-only accounts while they are referenced by
|
||||
any transaction moving through the runtime, and release the cache when the last
|
||||
transaction exits the runtime.
|
||||
|
||||
## Credit-only accounts and read-only account data
|
||||
|
||||
Credit-only account data can be treated as read-only. Credit-debit
|
||||
account data is treated as read-write.
|
||||
|
||||
## Transaction changes
|
||||
|
||||
To enable the possibility of caching accounts only while they are in the
|
||||
runtime, the Transaction structure should be changed in the following way:
|
||||
|
||||
* `program_ids: Vec<Pubkey>` - This vector is removed. Program keys can be
|
||||
placed at the end of the `account_keys` vector within the `num_readonly_accounts`
|
||||
number set to the number of programs.
|
||||
|
||||
* `num_readonly_accounts: u8` - The number of keys from the **end** of the
|
||||
transaction's `account_keys` array that is credit-only.
|
||||
|
||||
The following possible accounts are present in an transaction:
|
||||
|
||||
* paying account
|
||||
* RW accounts
|
||||
* R accounts
|
||||
* Program IDs
|
||||
|
||||
The paying account must be credit-debit, and program IDs must be credit-only. The
|
||||
first account in the `account_keys` array is always the account that pays for
|
||||
the transaction fee, therefore it cannot be credit-only. For these reasons the
|
||||
credit-only accounts are all grouped together at the end of the `account_keys`
|
||||
vector. Counting credit-only accounts from the end allow for the default `0`
|
||||
value to still be functionally correct, since a transaction will succeed with
|
||||
all credit-debit accounts.
|
||||
|
||||
Since accounts can only appear once in the transaction's `account_keys` array,
|
||||
an account can only be credit-only or credit-debit in a single transaction, not
|
||||
both. The runtime treats a transaction as one atomic unit of execution. If any
|
||||
instruction needs credit-debit access to an account, a copy needs to be made. The
|
||||
write lock is held for the entire time the transaction is being processed by
|
||||
the runtime.
|
||||
|
||||
## Starvation
|
||||
|
||||
Read locks for credit-only accounts can keep the runtime from executing
|
||||
transactions requesting a write lock to a credit-debit account.
|
||||
|
||||
When a request for a write lock is made while a read lock is open, the
|
||||
transaction requesting the write lock should be cached. Upon closing the read
|
||||
lock, the pending transactions can be pushed through the runtime.
|
||||
|
||||
While a pending write transaction exists, any additional read lock requests for
|
||||
that account should fail. It follows that any other write lock requests will also
|
||||
fail. Currently, clients must retransmit when a transaction fails because of
|
||||
a pending transaction. This approach would mimic that behavior as closely as
|
||||
possible while preventing write starvation.
|
||||
|
||||
## Program execution with credit-only accounts
|
||||
|
||||
Before handing off the accounts to program execution, the runtime can mark each
|
||||
account in each instruction as a credit-only account. The credit-only accounts can
|
||||
be passed as references without an extra copy. The transaction will abort on a
|
||||
write to credit-only.
|
||||
|
||||
An alternative is to detect writes to credit-only accounts and fail the
|
||||
transactions before commit.
|
||||
|
||||
## Alternative design
|
||||
|
||||
This design attempts to cache a credit-only account after loading without the use
|
||||
of a transaction-specified credit-only accounts list. Instead, the credit-only
|
||||
accounts are held in a reference-counted table inside the runtime as the
|
||||
transactions are processed.
|
||||
|
||||
1. Transaction accounts are locked.
|
||||
a. If the account is present in the ‘credit-only' table, the TX does not fail.
|
||||
The pending state for this TX is marked NeedReadLock.
|
||||
2. Transaction accounts are loaded.
|
||||
a. Transaction accounts that are credit-only increase their reference
|
||||
count in the `credit-only` table.
|
||||
b. Transaction accounts that need a write lock and are present in the
|
||||
`credit-only` table fail.
|
||||
3. Transaction accounts are unlocked.
|
||||
a. Decrement the `credit-only` lock table reference count; remove if its 0
|
||||
b. Remove from the `lock` set if the account is not in the `credit-only`
|
||||
table.
|
||||
|
||||
The downside with this approach is that if the `lock` set mutex is released
|
||||
between lock and load to allow better pipelining of transactions, a request for
|
||||
a credit-only account may fail. Therefore, this approach is not suitable for
|
||||
treating programs as credit-only accounts.
|
||||
|
||||
Holding the accounts lock mutex while fetching the account from disk would
|
||||
potentially have a significant performance hit on the runtime. Fetching from
|
||||
disk is expected to be slow, but can be parallelized between multiple disks.
|
||||
213
book/src/installer.md
Normal file
213
book/src/installer.md
Normal file
@@ -0,0 +1,213 @@
|
||||
## Cluster Software Installation and Updates
|
||||
Currently users are required to build the solana cluster software themselves
|
||||
from the git repository and manually update it, which is error prone and
|
||||
inconvenient.
|
||||
|
||||
This document proposes an easy to use software install and updater that can be
|
||||
used to deploy pre-built binaries for supported platforms. Users may elect to
|
||||
use binaries supplied by Solana or any other party they trust. Deployment of
|
||||
updates is managed using an on-chain update manifest program.
|
||||
|
||||
### Motivating Examples
|
||||
#### Fetch and run a pre-built installer using a bootstrap curl/shell script
|
||||
The easiest install method for supported platforms:
|
||||
```bash
|
||||
$ curl -sSf https://raw.githubusercontent.com/solana-labs/solana/v0.13.0/install/solana-install-init.sh | sh
|
||||
```
|
||||
|
||||
This script will check github for the latest tagged release and download and run the
|
||||
`solana-install` binary from there.
|
||||
|
||||
|
||||
If additional arguments need to be specified during the installation, the
|
||||
following shell syntax is used:
|
||||
```bash
|
||||
$ init_args=.... # arguments for `solana-installer init ...`
|
||||
$ curl -sSf https://raw.githubusercontent.com/solana-labs/solana/v0.13.0/install/solana-install-init.sh | sh -s - ${init_args}
|
||||
```
|
||||
|
||||
#### Fetch and run a pre-built installer from a Github release
|
||||
With a well-known release URL, a pre-built binary can be obtained for supported
|
||||
platforms:
|
||||
|
||||
```bash
|
||||
$ curl -o solana-install https://github.com/solana-labs/solana/releases/download/v0.13.0/solana-install-x86_64-apple-darwin
|
||||
$ chmod +x ./solana-install
|
||||
$ ./solana-install --help
|
||||
```
|
||||
|
||||
#### Build and run the installer from source
|
||||
If a pre-built binary is not available for a given platform, building the
|
||||
installer from source is always an option:
|
||||
```bash
|
||||
$ git clone https://github.com/solana-labs/solana.git
|
||||
$ cd solana/install
|
||||
$ cargo run -- --help
|
||||
```
|
||||
|
||||
#### Deploy a new update to a cluster
|
||||
Given a solana release tarball (as created by `ci/publish-tarball.sh`) that has already been uploaded to a publicly accessible URL,
|
||||
the following commands will deploy the update:
|
||||
```bash
|
||||
$ solana-keygen -o update-manifest.json # <-- only generated once, the public key is shared with users
|
||||
$ solana-install deploy http://example.com/path/to/solana-release.tar.bz2 update-manifest.json
|
||||
```
|
||||
|
||||
#### Run a validator node that auto updates itself
|
||||
```bash
|
||||
$ solana-install init --pubkey 92DMonmBYXwEMHJ99c9ceRSpAmk9v6i3RdvDdXaVcrfj # <-- pubkey is obtained from whoever is deploying the updates
|
||||
$ export PATH=~/.local/share/solana-install/bin:$PATH
|
||||
$ solana-keygen ... # <-- runs the latest solana-keygen
|
||||
$ solana-install run solana-fullnode ... # <-- runs a fullnode, restarting it as necesary when an update is applied
|
||||
```
|
||||
|
||||
### On-chain Update Manifest
|
||||
An update manifest is used to advertise the deployment of new release tarballs
|
||||
on a solana cluster. The update manifest is stored using the `config` program,
|
||||
and each update manifest account describes a logical update channel for a given
|
||||
target triple (eg, `x86_64-apple-darwin`). The account public key is well-known
|
||||
between the entity deploying new updates and users consuming those updates.
|
||||
|
||||
The update tarball itself is hosted elsewhere, off-chain and can be fetched from
|
||||
the specified `download_url`.
|
||||
|
||||
```rust,ignore
|
||||
use solana_sdk::signature::Signature;
|
||||
|
||||
/// Information required to download and apply a given update
|
||||
pub struct UpdateManifest {
|
||||
pub timestamp_secs: u64, // When the release was deployed in seconds since UNIX EPOCH
|
||||
pub download_url: String, // Download URL to the release tar.bz2
|
||||
pub download_sha256: String, // SHA256 digest of the release tar.bz2 file
|
||||
}
|
||||
|
||||
/// Userdata of an Update Manifest program Account.
|
||||
#[derive(Serialize, Deserialize, Default, Debug, PartialEq)]
|
||||
pub struct SignedUpdateManifest {
|
||||
pub manifest: UpdateManifest,
|
||||
pub manifest_signature: Signature,
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
Note that the `manifest` field itself contains a corresponding signature
|
||||
(`manifest_signature`) to guard against man-in-the-middle attacks between the
|
||||
`solana-install` tool and the solana cluster RPC API.
|
||||
|
||||
To guard against rollback attacks, `solana-install` will refuse to install an
|
||||
update with an older `timestamp_secs` than what is currently installed.
|
||||
|
||||
### Release Archive Contents
|
||||
A release archive is expected to be a tar file compressed with
|
||||
bzip2 with the following internal structure:
|
||||
|
||||
* `/version.yml` - a simple YAML file containing the field `"target"` - the
|
||||
target tuple. Any additional fields are ignored.
|
||||
* `/bin/` -- directory containing available programs in the release.
|
||||
`solana-install` will symlink this directory to
|
||||
`~/.local/share/solana-install/bin` for use by the `PATH` environment
|
||||
variable.
|
||||
* `...` -- any additional files and directories are permitted
|
||||
|
||||
### solana-install Tool
|
||||
The `solana-install` tool is used by the user to install and update their cluster software.
|
||||
|
||||
It manages the following files and directories in the user's home directory:
|
||||
* `~/.config/solana/install/config.yml` - user configuration and information about currently installed software version
|
||||
* `~/.local/share/solana/install/bin` - a symlink to the current release. eg, `~/.local/share/solana-update/<update-pubkey>-<manifest_signature>/bin`
|
||||
* `~/.local/share/solana/install/releases/<download_sha256>/` - contents of a release
|
||||
|
||||
#### Command-line Interface
|
||||
```manpage
|
||||
solana-install 0.13.0
|
||||
The solana cluster software installer
|
||||
|
||||
USAGE:
|
||||
solana-install [OPTIONS] <SUBCOMMAND>
|
||||
|
||||
FLAGS:
|
||||
-h, --help Prints help information
|
||||
-V, --version Prints version information
|
||||
|
||||
OPTIONS:
|
||||
-c, --config <PATH> Configuration file to use [default: /Users/mvines/Library/Preferences/solana/install.yml]
|
||||
|
||||
SUBCOMMANDS:
|
||||
deploy deploys a new update
|
||||
help Prints this message or the help of the given subcommand(s)
|
||||
info displays information about the current installation
|
||||
init initializes a new installation
|
||||
run Runs a program while periodically checking and applying software updates
|
||||
update checks for an update, and if available downloads and applies it
|
||||
```
|
||||
|
||||
```manpage
|
||||
solana-install-init
|
||||
initializes a new installation
|
||||
|
||||
USAGE:
|
||||
solana-install init [OPTIONS]
|
||||
|
||||
FLAGS:
|
||||
-h, --help Prints help information
|
||||
|
||||
OPTIONS:
|
||||
-d, --data_dir <PATH> Directory to store install data [default: /Users/mvines/Library/Application Support/solana]
|
||||
-u, --url <URL> JSON RPC URL for the solana cluster [default: https://api.testnet.solana.com/]
|
||||
-p, --pubkey <PUBKEY> Public key of the update manifest [default: 9XX329sPuskWhH4DQh6k16c87dHKhXLBZTL3Gxmve8Gp]
|
||||
```
|
||||
|
||||
```manpage
|
||||
solana-install-info
|
||||
displays information about the current installation
|
||||
|
||||
USAGE:
|
||||
solana-install info [FLAGS]
|
||||
|
||||
FLAGS:
|
||||
-h, --help Prints help information
|
||||
-l, --local only display local information, don't check the cluster for new updates
|
||||
```
|
||||
|
||||
```manpage
|
||||
solana-install-deploy
|
||||
deploys a new update
|
||||
|
||||
USAGE:
|
||||
solana-install deploy <download_url> <update_manifest_keypair>
|
||||
|
||||
FLAGS:
|
||||
-h, --help Prints help information
|
||||
|
||||
ARGS:
|
||||
<download_url> URL to the solana release archive
|
||||
<update_manifest_keypair> Keypair file for the update manifest (/path/to/keypair.json)
|
||||
```
|
||||
|
||||
```manpage
|
||||
solana-install-update
|
||||
checks for an update, and if available downloads and applies it
|
||||
|
||||
USAGE:
|
||||
solana-install update
|
||||
|
||||
FLAGS:
|
||||
-h, --help Prints help information
|
||||
```
|
||||
|
||||
```manpage
|
||||
solana-install-run
|
||||
Runs a program while periodically checking and applying software updates
|
||||
|
||||
USAGE:
|
||||
solana-install run <program_name> [program_arguments]...
|
||||
|
||||
FLAGS:
|
||||
-h, --help Prints help information
|
||||
|
||||
ARGS:
|
||||
<program_name> program to run
|
||||
<program_arguments>... arguments to supply to the program
|
||||
|
||||
The program will be restarted upon a successful software update
|
||||
```
|
||||
102
book/src/testnet-participation.md
Normal file
102
book/src/testnet-participation.md
Normal file
@@ -0,0 +1,102 @@
|
||||
## Testnet Participation
|
||||
This document describes how to participate in a public testnet as a
|
||||
validator node using the *Beacons v0.12* release.
|
||||
|
||||
Please note some of the information and instructions described here may change
|
||||
in future releases.
|
||||
|
||||
### Beta Testnet Overview
|
||||
The beta testnet features a validator running at beta.testnet.solana.com, which
|
||||
serves as the entrypoint to the cluster for your validator.
|
||||
|
||||
Additionally there is a blockexplorer available at http://beta.testnet.solana.com/.
|
||||
|
||||
The beta testnet is configured to reset the ledger every 24hours, or sooner
|
||||
should an hourly automated sanity test fail.
|
||||
|
||||
### Machine Requirements
|
||||
Since the beta testnet is not intended for stress testing of max transaction
|
||||
throughput, a higher-end machine with a GPU is not necessary to participate.
|
||||
|
||||
However ensure the machine used is not behind a residential NAT to avoid NAT
|
||||
traversal issues. A cloud-hosted machine works best. Ensure that IP ports
|
||||
8000 through 10000 are not blocked for Internet traffic.
|
||||
|
||||
Prebuilt binaries are available for Linux x86_64 (Ubuntu 18.04 recommended).
|
||||
MacOS or WSL users may build from source.
|
||||
|
||||
### Validator Setup
|
||||
The shell commands in this section assume the following environment variables are
|
||||
set:
|
||||
```bash
|
||||
$ export release=0.12.1
|
||||
$ export ip=$(dig +short beta.testnet.solana.com)
|
||||
$ export USE_INSTALL=1
|
||||
```
|
||||
|
||||
#### Obtaining The Software
|
||||
Prebuilt binaries are available for Linux x86_64 systems. Download and install by running:
|
||||
```bash
|
||||
$ wget https://github.com/solana-labs/solana/releases/download/v${release:?}/solana-release-x86_64-unknown-linux-gnu.tar.bz2 -O solana-release.tar.bz2
|
||||
$ tar jxf solana-release.tar.bz2
|
||||
$ cd solana-release/
|
||||
$ export PATH=$PWD/bin:$PATH
|
||||
```
|
||||
|
||||
If you are unable to use the prebuilt binaries or prefer to build it yourself from source:
|
||||
```bash
|
||||
$ wget https://github.com/solana-labs/solana/archive/v${release:?}.tar.gz -O solana-release.tar.gz
|
||||
$ tar zxf solana-release.tar.gz
|
||||
$ cd solana-${release:?}
|
||||
$ ./scripts/cargo-install-all.sh .
|
||||
$ export PATH=$PWD/bin:$PATH
|
||||
```
|
||||
|
||||
#### Confirm The Testnet Is Reachable
|
||||
Before attaching a validator node, sanity check that the cluster is accessible
|
||||
to your machine by running some simple wallet commands. If any of these
|
||||
commands fail, please retry 5-10 minutes later to confirm the testnet is not
|
||||
just restarting itself before debugging further.
|
||||
|
||||
Receive an airdrop of lamports from the testnet drone:
|
||||
```bash
|
||||
$ solana-wallet -n ${ip:?} airdrop 123
|
||||
$ solana-wallet -n ${ip:?} balance
|
||||
```
|
||||
|
||||
Fetch the current testnet transaction count over JSON RPC:
|
||||
```bash
|
||||
$ curl -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":1, "method":"getTransactionCount"}' http://beta.testnet.solana.com:8899
|
||||
```
|
||||
|
||||
Inspect the blockexplorer at http://beta.testnet.solana.com/ for activity.
|
||||
|
||||
Run the following command to join the gossip network and view all the other nodes in the cluster:
|
||||
```bash
|
||||
$ RUST_LOG=info solana-bench-tps --converge-only --num-nodes 100000 --network ${ip:?}:8001
|
||||
```
|
||||
|
||||
#### Starting The Validator
|
||||
The following command will start a new validator node:
|
||||
```bash
|
||||
$ RUST_LOG=warn ./multinode-demo/fullnode-x.sh --public-address --poll-for-new-genesis-block ${ip:?}
|
||||
```
|
||||
|
||||
Then from another console, confirm the IP address if your node is now visible in
|
||||
the gossip network by running:
|
||||
```bash
|
||||
$ RUST_LOG=info solana-bench-tps --converge-only --num-nodes 100000 --network ${ip:?}:8001
|
||||
```
|
||||
|
||||
Congratulations, you're now participating in the testnet cluster!
|
||||
|
||||
#### Sharing Metrics From Your Validator
|
||||
If you'd like to share metrics perform the following steps before starting the
|
||||
validator node:
|
||||
```bash
|
||||
export u="username obtained from the Solana maintainers"
|
||||
export p="password obtained from the Solana maintainers"
|
||||
export SOLANA_METRICS_CONFIG="db=testnet-beta,u=${u:?},p=${p:?}"
|
||||
source scripts/configure-metrics.sh
|
||||
```
|
||||
Inspect https://metrics.solana.com:3000/d/testnet-beta/testnet-monitor-beta?refresh=60s&orgId=2&var-testnet=testnet-beta&var-hostid=All for your contributions to our metrics dashboard.
|
||||
Reference in New Issue
Block a user