192 Commits

Author SHA1 Message Date
behzad nouri
d0b850cdd9 removes turbine peers shuffle patch feature 2022-04-05 12:04:12 +00:00
Stephen Akridge
976b138e76 Add tx weighting stage 2022-03-17 19:31:28 -05:00
Tao Zhu
8590911b0a Replace type alias with newtype for UnprocesedPacketBatches 2022-03-14 13:14:27 -05:00
Tao Zhu
35d1235ed0 - move unprocessed_packet_batches from BankingStage to its own (#23508)
module
- deserialize packets during receving and buffering
2022-03-10 18:47:46 +00:00
Jeff Biseda
c69e3b73ff bench get_retransmit_peers (#23292) 2022-03-01 19:10:29 -08:00
buffalu
70ebab2c82 Add rustfmt.toml and cargo fmt (#23238)
* fmt

* formatted

Co-authored-by: Lucas B <buffalu@jito.network>
2022-02-19 13:32:29 +08:00
anatoly yakovenko
83d31c9e65 shrink batches when over 80% of the space is wasted (#23066)
* shrink batches when over 80% of the space is wasted
2022-02-16 08:18:17 -08:00
carllin
2f9e30a1f7 Introduce slot-specific packet metrics (#22906) 2022-02-11 03:07:45 -05:00
Tao Zhu
e52e48076e bench should update leader schedule cache (#22991) 2022-02-08 02:28:28 +00:00
anatoly yakovenko
d343713f61 Optimize packet dedup (#22571)
* Use bloom filter to dedup packets

* dedup first

* Update bloom/src/bloom.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* fixup

* fixup

* fixup

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2022-01-19 13:58:20 -08:00
behzad nouri
dcf44d2523 improves sigverify discard_excess_packets performance (#22577)
As shown by the added benchmark, current code does worse if there is a
spam address plus a lot of unique addresses.

on current master:
test bench_packet_discard_many_senders  ... bench:   1,997,960 ns/iter (+/- 103,715)
test bench_packet_discard_mixed_senders ... bench:  14,256,116 ns/iter (+/- 534,865)
test bench_packet_discard_single_sender ... bench:   1,306,809 ns/iter (+/- 61,992)

with this commit:
test bench_packet_discard_many_senders  ... bench:   1,644,025 ns/iter (+/- 83,715)
test bench_packet_discard_mixed_senders ... bench:   1,089,789 ns/iter (+/- 86,324)
test bench_packet_discard_single_sender ... bench:     955,234 ns/iter (+/- 55,953)
2022-01-19 18:10:02 +00:00
sakridge
49443406fd Use VecDeque instead of Vec in sigverify stage (#22538)
avoid bad performance of remove(0) for a single sender
2022-01-17 18:37:05 +01:00
Tao Zhu
9c9f2dd5bd port counting vote CUs to block cost (#22477) 2022-01-14 10:50:29 -06:00
Eric Warehime
8161cee70f Remove unnecessary var in banking_stage bench (#22408) 2022-01-11 22:25:21 -06:00
Jeff Biseda
8b66625c95 convert std::sync::mpsc to crossbeam_channel (#22264) 2022-01-11 02:44:46 -08:00
behzad nouri
01a096adc8 adds bitflags to Packet.Meta
Instead of a separate bool type for each flag, all the flags can be
encoded in a type-safe bitflags encoded in a single u8:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L19-L31
2022-01-04 13:53:40 +00:00
behzad nouri
73a7741c49 uses std::net::IpAddr type for Packet.Meta.addr 2022-01-04 13:53:40 +00:00
Justin Starry
93c776ce19 Refactor packet deduplication and harden bench test (#22080) 2021-12-22 23:05:10 -06:00
Tao Zhu
dd80a525ef Leader QoS service metrics (#21708)
* - qos_service metrics tagged with leader thread ids to separate gossip/tpu votes and transactions;
- qos_service metrics is reported with bank slot;
- replaced timer-based reporting with signal via channel; removed async report test as qos_service now lives within a thread

* - add tpu live packets (eg, not buffered packets) states to qos metrics reporting
2021-12-22 21:39:59 +00:00
behzad nouri
65d59f4ef0 tracks erasure coding shreds' indices explicitly (#21822)
The indices for erasure coding shreds are tied to data shreds:
https://github.com/solana-labs/solana/blob/90f41fd9b/ledger/src/shred.rs#L921

However with the upcoming changes to erasure schema, there will be more
erasure coding shreds than data shreds and we can no longer infer coding
shreds indices from data shreds.

The commit adds constructs to track coding shreds indices explicitly.
2021-12-19 22:37:55 +00:00
behzad nouri
89d66c3210 removes next_shred_index from return value of entries to shreds api (#21961)
next-shred-index is already readily available from returned data shreds.
The commit simplifies the api for upcoming changes to erasure coding
schema which will require explicit tracking of indices for coding shreds
as well as data shreds.
2021-12-17 15:01:55 +00:00
Justin Starry
254ef3e7b6 Rename Packets to PacketBatch (#21794) 2021-12-11 09:44:15 -05:00
Michael Vines
b8837c04ec Reformat imports to a consistent style for imports
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
2021-12-03 09:19:13 -08:00
Tao Zhu
0ca255220e - Encapsulate QoS Service metrics reporting within QosServioce, so client (#21191)
code (eg banking_stage) doesn't need to worry about it.
- Remove dead cost_* stats from banking_stage, clean up call path.
2021-11-18 15:35:30 -06:00
Justin Starry
66fa062f13 rename process_entries to indicate it's only for tests (#21321) 2021-11-17 20:53:40 +01:00
behzad nouri
5fb0ab9d00 removes redundant args from Shredder::try_recovery (#21226)
Shredder::try_recovery is already taking a Vec<Shred> as an argument. All the
other arguments are embedded in the shreds, and are so redundant.
2021-11-10 21:19:03 +00:00
Tao Zhu
c2bfce90b3 - cost_tracker is data member of a bank, it can report metrics when bank is frozen (#20802)
- removed cost_tracker_stats and histogram
- move stats reporting outside of bank freeze
2021-10-24 22:19:23 -05:00
Tao Zhu
7496b5784b - make cost_tracker a member of bank, remove shared instance from TPU; (#20627)
- decouple cost_model from cost_tracker; allowing one cost_model
  instance being shared within a validator;
- update cost_model api to calculate_cost(&self...)->transaction_cost
2021-10-19 14:37:33 -05:00
sakridge
09e7782d76 Refactor code to get block signatures in get_confirmed_signatures_for_address2 (#20575)
* Refactor get_confirmed_signatures_for_address2

* Move blockstore benches to ledger where they belong
2021-10-13 09:55:19 +02:00
Tao Zhu
005d6863fd - move cost tracker into bank, so each bank has its own cost tracker; (#20527)
- move related modules to runtime
2021-10-12 08:51:33 -05:00
Tao Zhu
177a375479 Tpu vote 1.7 (#20187) (#20494)
* Add separate vote processing tpu port

* Add feature to send to tpu vote port

* Add vote rejecting sigverify mode

* use packet.meta.is_simple_vote_tx in place of deserialization

* consolidate code that identifies vote tx atcommon path for cpu and gpu

* new key for feature set

* banking forward tpu vote

* add tpu vote port to dockerfile and other review changes

* Simplify thread id compare

* fix a test; updated cluster_info ABI change

Co-authored-by: Tao Zhu <tao@solana.com>

Co-authored-by: sakridge <sakridge@gmail.com>
2021-10-07 09:38:23 +00:00
Tao Zhu
03913f6661 add tx count and thread id to stats, each stat reports and resets when slot changes (#20451) 2021-10-06 00:09:19 -05:00
Tao Zhu
6ff508c643 add transaction cost histogram metrics (#20350) 2021-10-05 08:57:39 -05:00
sakridge
94668c95c2 Prune sigverify queue (#20331) 2021-09-30 05:41:05 +02:00
Jon Cinque
567f30aa1a windows: Make solana-test-validator work (#20099)
* windows: Make solana-test-validator work

The important changes to get this going on Windows:

* ledger lock needs to be done on a file instead of the directory
* IPC service needs to use the Windows pipe naming scheme
* always disable the JIT
* file logging not possible yet because we can't redirect stderr,
but this will change once env_logger fixes the pipe output target!

* Integrate review feedback
2021-09-22 23:10:35 +02:00
behzad nouri
01a7ec8198 uses rayon thread-pool for retransmit-stage parallelization (#19486) 2021-09-07 15:15:01 +00:00
behzad nouri
1deb4add81 removes Slot from TransmitShreds (#19327)
An earlier version of the code was funneling through stakes along with
shreds to broadcast:
https://github.com/solana-labs/solana/blob/b67ffab37/core/src/broadcast_stage.rs#L127

This was changed to only slots as stakes computation was pushed further
down the pipeline in:
https://github.com/solana-labs/solana/pull/18971

However shreds themselves embody which slot they belong to. So pairing
them with slot is redundant and adds rooms for bugs should they become
inconsistent.
2021-08-20 13:48:33 +00:00
Justin Starry
c50b01cb60 Store versioned transactions in the ledger, disabled by default (#19139)
* Add support for versioned transactions, but disable by default

* merge conflicts

* trent's feedback

* bump Cargo.lock

* Fix transaction error encoding

* Rename legacy_transaction method

* cargo clippy

* Clean up casts, int arithmetic, and unused methods

* Check for duplicates in sanitized message conversion

* fix clippy

* fix new test

* Fix bpf conditional compilation for message module
2021-08-17 15:17:56 -07:00
behzad nouri
3efccbffab sends shreds (instead of packets) to retransmit stage
Working towards channelling through shreds recovered from erasure codes
to retransmit stage.
2021-08-17 13:44:10 +00:00
behzad nouri
6e413331b5 removes erroneous uses of Arc<...> from retransmit stage 2021-08-17 13:44:10 +00:00
Michael Vines
e9722474eb Move tower storage into its own module 2021-08-11 00:20:46 -07:00
Michael Vines
397801a2d8 Extract tower storage details from Tower struct 2021-08-06 10:04:37 -07:00
Jeff Washington (jwash)
a9014ceceb Bank::default_for_tests() (#19084) 2021-08-05 11:53:29 -05:00
Jeff Washington (jwash)
bde9b4de94 Bank::new -> Bank::new_for_benches (#19063) 2021-08-04 17:30:43 -05:00
carllin
03353d500f Actively manage dead slots in AncestorHashesService (#18912) 2021-08-02 14:33:28 -07:00
behzad nouri
d06dc6c8a6 shares cluster-nodes between retransmit threads (#18947)
cluster_nodes and last_peer_update are not shared between retransmit
threads, as each thread have its own value:
https://github.com/solana-labs/solana/blob/65ccfed86/core/src/retransmit_stage.rs#L476-L477

Additionally, with shared references, this code:
https://github.com/solana-labs/solana/blob/0167daa11/core/src/retransmit_stage.rs#L315-L328
has a concurrency bug where the thread which does compare_and_swap,
updates cluster_nodes much later after other threads have run with
outdated cluster_nodes for a while. In particular, the write-lock there
may block.
2021-07-29 16:20:15 +00:00
sakridge
84e78316b1 Write helper for multithread update (#18808) 2021-07-29 03:16:36 +02:00
behzad nouri
d2d5f36a3c adds validator flag to allow private ip addresses (#18850) 2021-07-23 15:25:03 +00:00
Justin Starry
d166b9856a Move transaction sanitization earlier in the pipeline (#18655)
* Move transaction sanitization earlier in the pipeline

* Renamed HashedTransaction to SanitizedTransaction

* Implement deref for sanitized transaction

* bring back process_transactions test method

* Use sanitized transactions for cost model calculation
2021-07-15 22:51:27 -05:00
sakridge
7f2254225e Move entry/poh to own crate to speed up poh bench build (#18225) 2021-07-14 14:16:29 +02:00