Files
solana/watchtower
Trent Nelson 4a42cfc42a Cli error cleanup 1.0 (#8834)
* Don't use move semantics if not needed (#8793)

* SDK: Deboilerplate `TransportError` with thiserror

* Enable interchange between `TransportError` and `ClientError`

* SDK: Retval consistency between `Client` and `AsyncClient` traits

* Client: Introduce/use `Result` type

* Client: Remove unused `RpcResponseIn` type

* Client: Rename `RpcResponse` to more appropriate `RpcResult`

* Client: Death to `io::Result` return types

* Client: Struct-ify `ClientError`

* Client: Add optional `command` parameter to `ClientError`

* RpcClient: Stop abusing `io::Error` (low-fruit)

* ClientError: Use `thiserror`'s `Display` impl

* Extend `RpcError`'s utility

* RpcClient: Stop abusing `io::Error` (the rest)

* CLI: Shim `main()` so we can `Display` format errors

* claputils: format input validator errors with `Display`

They are intended to be displayed to users

* SDK: `thiserror` for hash and sig parse erros

* Keygen: Shim main to format errors with `Display`

* SDK: `thiserror` for `InstructionError`

* CLI: `thiserror` for `CliError`

* CLI: Format user messages with `Display`

* Client: Tweak `Display` for `ClientError`

* RpcClient: Improve messaging when TX cannot be confirmed

* fu death io res retval

* CLI/Keygen - fix shell return value on error

* Tweak `InstructionError` `Display` messages as per review

* Cleanup hackjob return code fix

* Embrace that which you hate most

* Too much...

Co-authored-by: Jack May <jack@solana.com>
2020-03-13 07:42:25 -06:00
..
2020-03-13 07:42:25 -06:00
2019-12-12 12:03:13 -07:00
2020-03-09 22:38:57 -07:00

The solana-watchtower program is used to monitor the health of a cluster. It periodically polls the cluster over an RPC API to confirm that the transaction count is advancing, new blockhashes are available, and no validators are delinquent. Results are reported as InfluxDB metrics, with an optional push notification on sanity failure.

If you only care about the health of one specific validator, the --validator-identity command-line argument can be used to restrict failure notifications to issues only affecting that validator.

If you do not want duplicate notifications, for example if you have elected to recieve notifications by SMS the --no-duplicate-notifications command-line argument will suppress identical failure notifications.

Metrics

watchtower-sanity

On every iteration this data point will be emitted indicating the overall result using a boolean ok field.

watchtower-sanity-failure

On failure this data point contains details about the specific test that failed via the following fields:

  • test: name of the sanity test that failed
  • err: exact sanity failure message

Sanity failure push notification

To receive a Slack, Discord and/or Telegram notification on sanity failure, define environment variables before running solana-watchtower:

export SLACK_WEBHOOK=...
export DISCORD_WEBHOOK=...

Telegram requires the following two variables:

export TELEGRAM_BOT_TOKEN=...
export TELEGRAM_CHAT_ID=...

To receive a Twilio SMS notification on failure, having a Twilio account, and a sending number owned by that account, define environment variable before running solana-watchtower:

export TWILIO_CONFIG='ACCOUNT=<account>,TOKEN=<securityToken>,TO=<receivingNumber>,FROM=<sendingNumber>'