192 Commits

Author SHA1 Message Date
a6ded6a5ed removes turbine legacy code and already activated features (backport #24080) (#24117) 2022-04-06 01:12:18 +00:00
976b138e76 Add tx weighting stage 2022-03-17 19:31:28 -05:00
8590911b0a Replace type alias with newtype for UnprocesedPacketBatches 2022-03-14 13:14:27 -05:00
35d1235ed0 - move unprocessed_packet_batches from BankingStage to its own (#23508)
module
- deserialize packets during receving and buffering
2022-03-10 18:47:46 +00:00
c69e3b73ff bench get_retransmit_peers (#23292) 2022-03-01 19:10:29 -08:00
70ebab2c82 Add rustfmt.toml and cargo fmt (#23238)
* fmt

* formatted

Co-authored-by: Lucas B <buffalu@jito.network>
2022-02-19 13:32:29 +08:00
83d31c9e65 shrink batches when over 80% of the space is wasted (#23066)
* shrink batches when over 80% of the space is wasted
2022-02-16 08:18:17 -08:00
2f9e30a1f7 Introduce slot-specific packet metrics (#22906) 2022-02-11 03:07:45 -05:00
e52e48076e bench should update leader schedule cache (#22991) 2022-02-08 02:28:28 +00:00
d343713f61 Optimize packet dedup (#22571)
* Use bloom filter to dedup packets

* dedup first

* Update bloom/src/bloom.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* Update core/src/sigverify_stage.rs

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>

* fixup

* fixup

* fixup

Co-authored-by: Trent Nelson <trent.a.b.nelson@gmail.com>
2022-01-19 13:58:20 -08:00
dcf44d2523 improves sigverify discard_excess_packets performance (#22577)
As shown by the added benchmark, current code does worse if there is a
spam address plus a lot of unique addresses.

on current master:
test bench_packet_discard_many_senders  ... bench:   1,997,960 ns/iter (+/- 103,715)
test bench_packet_discard_mixed_senders ... bench:  14,256,116 ns/iter (+/- 534,865)
test bench_packet_discard_single_sender ... bench:   1,306,809 ns/iter (+/- 61,992)

with this commit:
test bench_packet_discard_many_senders  ... bench:   1,644,025 ns/iter (+/- 83,715)
test bench_packet_discard_mixed_senders ... bench:   1,089,789 ns/iter (+/- 86,324)
test bench_packet_discard_single_sender ... bench:     955,234 ns/iter (+/- 55,953)
2022-01-19 18:10:02 +00:00
49443406fd Use VecDeque instead of Vec in sigverify stage (#22538)
avoid bad performance of remove(0) for a single sender
2022-01-17 18:37:05 +01:00
9c9f2dd5bd port counting vote CUs to block cost (#22477) 2022-01-14 10:50:29 -06:00
8161cee70f Remove unnecessary var in banking_stage bench (#22408) 2022-01-11 22:25:21 -06:00
8b66625c95 convert std::sync::mpsc to crossbeam_channel (#22264) 2022-01-11 02:44:46 -08:00
01a096adc8 adds bitflags to Packet.Meta
Instead of a separate bool type for each flag, all the flags can be
encoded in a type-safe bitflags encoded in a single u8:
https://github.com/solana-labs/solana/blob/d6ec103be/sdk/src/packet.rs#L19-L31
2022-01-04 13:53:40 +00:00
73a7741c49 uses std::net::IpAddr type for Packet.Meta.addr 2022-01-04 13:53:40 +00:00
93c776ce19 Refactor packet deduplication and harden bench test (#22080) 2021-12-22 23:05:10 -06:00
dd80a525ef Leader QoS service metrics (#21708)
* - qos_service metrics tagged with leader thread ids to separate gossip/tpu votes and transactions;
- qos_service metrics is reported with bank slot;
- replaced timer-based reporting with signal via channel; removed async report test as qos_service now lives within a thread

* - add tpu live packets (eg, not buffered packets) states to qos metrics reporting
2021-12-22 21:39:59 +00:00
65d59f4ef0 tracks erasure coding shreds' indices explicitly (#21822)
The indices for erasure coding shreds are tied to data shreds:
https://github.com/solana-labs/solana/blob/90f41fd9b/ledger/src/shred.rs#L921

However with the upcoming changes to erasure schema, there will be more
erasure coding shreds than data shreds and we can no longer infer coding
shreds indices from data shreds.

The commit adds constructs to track coding shreds indices explicitly.
2021-12-19 22:37:55 +00:00
89d66c3210 removes next_shred_index from return value of entries to shreds api (#21961)
next-shred-index is already readily available from returned data shreds.
The commit simplifies the api for upcoming changes to erasure coding
schema which will require explicit tracking of indices for coding shreds
as well as data shreds.
2021-12-17 15:01:55 +00:00
254ef3e7b6 Rename Packets to PacketBatch (#21794) 2021-12-11 09:44:15 -05:00
b8837c04ec Reformat imports to a consistent style for imports
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
2021-12-03 09:19:13 -08:00
0ca255220e - Encapsulate QoS Service metrics reporting within QosServioce, so client (#21191)
code (eg banking_stage) doesn't need to worry about it.
- Remove dead cost_* stats from banking_stage, clean up call path.
2021-11-18 15:35:30 -06:00
66fa062f13 rename process_entries to indicate it's only for tests (#21321) 2021-11-17 20:53:40 +01:00
5fb0ab9d00 removes redundant args from Shredder::try_recovery (#21226)
Shredder::try_recovery is already taking a Vec<Shred> as an argument. All the
other arguments are embedded in the shreds, and are so redundant.
2021-11-10 21:19:03 +00:00
c2bfce90b3 - cost_tracker is data member of a bank, it can report metrics when bank is frozen (#20802)
- removed cost_tracker_stats and histogram
- move stats reporting outside of bank freeze
2021-10-24 22:19:23 -05:00
7496b5784b - make cost_tracker a member of bank, remove shared instance from TPU; (#20627)
- decouple cost_model from cost_tracker; allowing one cost_model
  instance being shared within a validator;
- update cost_model api to calculate_cost(&self...)->transaction_cost
2021-10-19 14:37:33 -05:00
09e7782d76 Refactor code to get block signatures in get_confirmed_signatures_for_address2 (#20575)
* Refactor get_confirmed_signatures_for_address2

* Move blockstore benches to ledger where they belong
2021-10-13 09:55:19 +02:00
005d6863fd - move cost tracker into bank, so each bank has its own cost tracker; (#20527)
- move related modules to runtime
2021-10-12 08:51:33 -05:00
177a375479 Tpu vote 1.7 (#20187) (#20494)
* Add separate vote processing tpu port

* Add feature to send to tpu vote port

* Add vote rejecting sigverify mode

* use packet.meta.is_simple_vote_tx in place of deserialization

* consolidate code that identifies vote tx atcommon path for cpu and gpu

* new key for feature set

* banking forward tpu vote

* add tpu vote port to dockerfile and other review changes

* Simplify thread id compare

* fix a test; updated cluster_info ABI change

Co-authored-by: Tao Zhu <tao@solana.com>

Co-authored-by: sakridge <sakridge@gmail.com>
2021-10-07 09:38:23 +00:00
03913f6661 add tx count and thread id to stats, each stat reports and resets when slot changes (#20451) 2021-10-06 00:09:19 -05:00
6ff508c643 add transaction cost histogram metrics (#20350) 2021-10-05 08:57:39 -05:00
94668c95c2 Prune sigverify queue (#20331) 2021-09-30 05:41:05 +02:00
567f30aa1a windows: Make solana-test-validator work (#20099)
* windows: Make solana-test-validator work

The important changes to get this going on Windows:

* ledger lock needs to be done on a file instead of the directory
* IPC service needs to use the Windows pipe naming scheme
* always disable the JIT
* file logging not possible yet because we can't redirect stderr,
but this will change once env_logger fixes the pipe output target!

* Integrate review feedback
2021-09-22 23:10:35 +02:00
01a7ec8198 uses rayon thread-pool for retransmit-stage parallelization (#19486) 2021-09-07 15:15:01 +00:00
1deb4add81 removes Slot from TransmitShreds (#19327)
An earlier version of the code was funneling through stakes along with
shreds to broadcast:
https://github.com/solana-labs/solana/blob/b67ffab37/core/src/broadcast_stage.rs#L127

This was changed to only slots as stakes computation was pushed further
down the pipeline in:
https://github.com/solana-labs/solana/pull/18971

However shreds themselves embody which slot they belong to. So pairing
them with slot is redundant and adds rooms for bugs should they become
inconsistent.
2021-08-20 13:48:33 +00:00
c50b01cb60 Store versioned transactions in the ledger, disabled by default (#19139)
* Add support for versioned transactions, but disable by default

* merge conflicts

* trent's feedback

* bump Cargo.lock

* Fix transaction error encoding

* Rename legacy_transaction method

* cargo clippy

* Clean up casts, int arithmetic, and unused methods

* Check for duplicates in sanitized message conversion

* fix clippy

* fix new test

* Fix bpf conditional compilation for message module
2021-08-17 15:17:56 -07:00
3efccbffab sends shreds (instead of packets) to retransmit stage
Working towards channelling through shreds recovered from erasure codes
to retransmit stage.
2021-08-17 13:44:10 +00:00
6e413331b5 removes erroneous uses of Arc<...> from retransmit stage 2021-08-17 13:44:10 +00:00
e9722474eb Move tower storage into its own module 2021-08-11 00:20:46 -07:00
397801a2d8 Extract tower storage details from Tower struct 2021-08-06 10:04:37 -07:00
a9014ceceb Bank::default_for_tests() (#19084) 2021-08-05 11:53:29 -05:00
bde9b4de94 Bank::new -> Bank::new_for_benches (#19063) 2021-08-04 17:30:43 -05:00
03353d500f Actively manage dead slots in AncestorHashesService (#18912) 2021-08-02 14:33:28 -07:00
d06dc6c8a6 shares cluster-nodes between retransmit threads (#18947)
cluster_nodes and last_peer_update are not shared between retransmit
threads, as each thread have its own value:
https://github.com/solana-labs/solana/blob/65ccfed86/core/src/retransmit_stage.rs#L476-L477

Additionally, with shared references, this code:
https://github.com/solana-labs/solana/blob/0167daa11/core/src/retransmit_stage.rs#L315-L328
has a concurrency bug where the thread which does compare_and_swap,
updates cluster_nodes much later after other threads have run with
outdated cluster_nodes for a while. In particular, the write-lock there
may block.
2021-07-29 16:20:15 +00:00
84e78316b1 Write helper for multithread update (#18808) 2021-07-29 03:16:36 +02:00
d2d5f36a3c adds validator flag to allow private ip addresses (#18850) 2021-07-23 15:25:03 +00:00
d166b9856a Move transaction sanitization earlier in the pipeline (#18655)
* Move transaction sanitization earlier in the pipeline

* Renamed HashedTransaction to SanitizedTransaction

* Implement deref for sanitized transaction

* bring back process_transactions test method

* Use sanitized transactions for cost model calculation
2021-07-15 22:51:27 -05:00
7f2254225e Move entry/poh to own crate to speed up poh bench build (#18225) 2021-07-14 14:16:29 +02:00