123 Commits

Author SHA1 Message Date
carllin
0bd1412562
Switch leader scheduler to use PoH ticks instead of Entry height (#1519)
* Add PoH height to process_ledger()

* Moved broadcast_stage Leader Scheduling logic to use Poh height instead of entry_height

* Moved LeaderScheduler logic to PoH in ReplicateStage

* Fix Leader scheduling tests to use PoH instead of entry height

* Change is_leader detection in repair() to use PoH instead of entry height

* Add tests to LeaderScheduler for new functionality

* fix Entry::new and genesis block PoH counts

* Moved LeaderScheduler to PoH ticks

* Cleanup to resolve PR comments
2018-10-18 22:57:48 -07:00
Rob Walker
d0f43e9934 consolidate tmp ledgers 2018-10-18 08:45:31 -06:00
Rob Walker
912eb5e8e9
remove bank.is_leader, dead code (#1516) 2018-10-16 15:26:44 -07:00
Tyera Eulberg
1b2e9122d5 Pubsub listen on random open port when rpc does (quiet some test errors) 2018-10-16 00:11:26 -06:00
Michael Vines
537436bd5e RPC PubSub now uses a well-known socket 2018-10-16 00:11:26 -06:00
Tyera Eulberg
1515bba9c6
Use cluster_info in rpc to get current leader addresses (#1480) 2018-10-12 14:25:56 -06:00
carllin
47f69f2d24
1) Switch broken tests to generate an empty tick in their ledgers to use as last_id, 2) Fix bug where PoH generator in BankingStage did not referenced the last tick instead of the last entry on startup, causing ledger verification to fail on the new tick added by the PoH generator (#1479) 2018-10-12 00:39:10 -07:00
Stephen Akridge
9dd4dc2088 Mark failing tests as ignore 2018-10-11 15:32:36 -07:00
carllin
9931ac9780
Leader scheduler plumbing (#1440)
* Added LeaderScheduler module and tests

* plumbing for LeaderScheduler in Fullnode + tests. Add vote processing for active set to ReplicateStage and WriteStage

* Add LeaderScheduler plumbing for Tvu, window, and tests

* Fix bank and switch tests to use new LeaderScheduler

* move leader rotation check from window service to replicate stage

* Add replicate_stage leader rotation exit test

* removed leader scheduler from the window service and associated modules/tests

* Corrected is_leader calculation in repair() function in window.rs

* Integrate LeaderScheduler with write_stage for leader to validator transitions

* Integrated LeaderScheduler with BroadcastStage

* Removed gossip leader rotation from crdt

* Add multi validator, leader test

* Comments and cleanup

* Remove unneeded checks from broadcast stage

* Fix case where a validator/leader need to immediately transition on startup after reading ledger and seeing they are not in the correct role

* Set new leader in validator -> validator transitions

* Clean up for PR comments, refactor LeaderScheduler from process_entry/process_ledger_tail

* Cleaned out LeaderScheduler options, implemented LeaderScheduler strategy that only picks the bootstrap leader to support existing tests, drone/airdrops

* Ignore test_full_leader_validator_network test due to bug where the next leader in line fails to get the last entry before rotation (b/c it hasn't started up yet). Added a test test_dropped_handoff_recovery go track this bug
2018-10-10 16:49:41 -07:00
Greg Fitzgerald
95701114e3 Crdt -> ClusterInfo 2018-10-09 03:49:39 -06:00
Carl
5491422b12 Fix validator_to_leader_transition test to not start up tpu after shutting down tvu, as the tpu now outputs ticks that will mess up the verification check 2018-10-03 21:58:29 -07:00
Pankaj Garg
e10574c64d Remove recycler and it's usage
- The memory usage due to recycler was high, and incrementing with
  time.
2018-09-27 10:42:37 -06:00
jackcmay
9c47e022dc
break dependency of programs on solana core (#1371)
* break dependency of programs on Solana core
2018-09-27 07:49:26 -07:00
Greg Fitzgerald
b7ae5b712a Move Pubkey into its own module 2018-09-26 20:40:40 -06:00
Anatoly Yakovenko
93c4f6c9b8 Synchronize PoH, bank last_id queue and ledger entry channel.
PoH, bank's last_id queue and the Entry channel need to have a synchronized order of ids.
2018-09-26 16:19:03 -07:00
carllin
e7383a7e66
Validator to leader (#1303)
* Add check in window_service to exit in checks for leader rotation, and propagate that service exit up to fullnode

* Added logic to shutdown Tvu once ReplicateStage finishes

* Added test for successfully shutting down validator and starting up leader

* Add test for leader validator interaction

* fix streamer to check for exit signal before checking socket again to prevent busy leaders from never returning

* PR comments - Rewrite make_consecutive_blobs() function, revert genesis function change
2018-09-25 15:41:29 -07:00
Greg Fitzgerald
95677a81c5 Pacify clippy 2018-09-24 13:36:31 -06:00
Jack May
a1f01fb8f8 revert is_some to not is_none, causes test failure 2018-09-23 17:09:18 -06:00
Jack May
b2be0e2e5e fix clippy warning 2018-09-23 17:09:18 -06:00
Jack May
1a45587c08 fix clippy warnings 2018-09-23 17:09:18 -06:00
Tyera Eulberg
ad4fef4f09 Doc for rpc_port configuration 2018-09-21 18:06:20 -06:00
Tyera Eulberg
e3b3701e13 Add RPC port option to fullnode 2018-09-21 18:06:20 -06:00
Carl
55126f5fb6 Marked Tvu functionality in Fullnode as unused for now 2018-09-19 16:05:31 -07:00
Anatoly Yakovenko
431692d9d0 Use a Drop trait to keep track of lifetimes for recycled objects.
* Move recycler instances to the point of allocation
* sinks no longer need to call `recycle`
* Remove the recycler arguments from all the apis that no longer need them
2018-09-19 16:59:42 -06:00
Carl
cd2d50e06c Changed transition to restart Rpu rather than modify bank to prevent lock contention 2018-09-19 10:48:05 -06:00
Carl
b10de40506 Made LEADER_ROTATION_INTERVAL settable so that integration tests don't time out 2018-09-19 10:48:05 -06:00
Carl
2030dfa435 Implement PR comments, tidy up 2018-09-19 10:48:05 -06:00
Carl
bfe64f5f6e Added integration test for transitioning leader to validator to see that tpu pipeline can exit and restart a tvu. Fixed Tpu and broadcast stage so that exiting later stages in the pipeline also causes earlier stages to exit. 2018-09-19 10:48:05 -06:00
Carl
6d27751365 give fullnode ownership of state needed to dynamically start up a tpu or tvu for role transition 2018-09-19 10:48:05 -06:00
Carl
1fb1c0a681 added jointypes to the stages in the tpu involved in leader rotation 2018-09-19 10:48:05 -06:00
Carl
062f654fe0 formatted code 2018-09-19 10:48:05 -06:00
Carl
98b47d2540 Added check in broadcast stage to exit after transmitting last blob before leader rotation. Also added tests 2018-09-19 10:48:05 -06:00
Carl
f28ba3937b Added check in write stage to exit when scheduled entry_height for leader rotation is detected 2018-09-19 10:48:05 -06:00
Carl
91cf14e641 Rewrote service trait join() method to allow thread join handles to return values other than () 2018-09-19 10:48:05 -06:00
Pankaj Garg
5310b6e5a2
Move entry->blob creation out of write stage (#1257)
- The write stage will output vector of entries
- Broadcast stage will create blobs out of the entries
- Helps reduce MIPS requirements for write stage
2018-09-18 13:49:10 -07:00
Anatoly Yakovenko
fd36954477 clippy 2018-09-15 05:12:53 -06:00
Michael Vines
4196cf43e8 cargo fmt 2018-09-14 16:37:49 -07:00
sakridge
4dc30ea104
Add recycler stats (#1187) 2018-09-13 14:49:48 -07:00
carllin
8706774ea7
Rewrote service trait join() method to allow thread join handles to return values other than () (#1213) 2018-09-13 14:00:17 -07:00
Rob Walker
b52230097e groom Fullnode's new_with_bank() to match new() more 2018-09-12 09:24:42 -07:00
Rob Walker
a8fdb8a5a7 use a single BlobRecycler per fullnode 2018-09-11 16:56:54 -07:00
Rob Walker
b313b7f6f9 Revert "move rpc_server to drop() semantics instead of having its own thread"
This reverts commit 40aa0654fa012c835c2de2e33c29b288b4f89aea.
2018-09-10 22:48:33 -07:00
Rob Walker
40aa0654fa move rpc_server to drop() semantics instead of having its own thread 2018-09-10 20:25:53 -07:00
Greg Fitzgerald
fc64e1853c Initialize Window, not SharedWindow
Wrap with Arc<RwLock>> when/if needed, no earlier.
2018-09-10 11:40:26 -06:00
sakridge
d82ec2634c
Fix is_leader boolean (#1115)
A node is the leader if the address is none
2018-09-04 13:38:24 -07:00
Greg Fitzgerald
c9a1ac9b8c Don't propogate errors we'll never handle 2018-09-04 06:01:32 -10:00
Rob Walker
176e806d94 rework of netwrk rendezvous
* rename NodeInfo field of Node from "data" to "info"
      (touches a lot of files)

  * update client to use gossip to find leader, a la drone

  * rework multinode scripts
      * move more stuff into rust
      * added usage to all
      * no more rsync unless you're a validator (TODO: whack that, too)
  * fullnode doesn't bail if drone isn't up yet, just keeps trying
  * drone doesn't bail if network isn't up yet, just keeps trying
2018-08-31 23:21:07 +09:00
Rob Walker
1af4cee63b fix #1079
* move gossip/NCP off assuming anything about its address
  * use a single socket to send and receive gossip
  * remove --addr/-a from CLIs
  * rearrange networking utility code
  * use Arc<UdpSocket> to share the Sync-safe UdpSocket among threads
  * rename TestNode to Node

TODO:

  * re-enable 127.0.0.1 as a valid address in crdt
  * change repair request/response to a similar, single socket
  * pick cloned sockets or Arc<UdpSocket> for all these (rpu uses tryclone())
  * update contact_info with network truthiness instead of what the node
      says?
2018-08-31 23:21:07 +09:00
Greg Fitzgerald
4ab58f069a Add back JsonRpcService changes 2018-08-23 13:42:54 -07:00
Greg Fitzgerald
85f96d926a Pacify clippy 2018-08-23 13:42:54 -07:00