Commit Graph

10518 Commits

Author SHA1 Message Date
80a2a35bc3 trie: there's no point in retrieving the metaroot 2019-04-08 12:00:42 +03:00
fca5f9fd6f common/fdlimit: fix macos file descriptors for Go 1.12 2019-04-02 13:14:21 +03:00
38c30f8dd8 light, params: update CHTs, integrate CHT for Goerli too 2019-04-02 12:10:06 +03:00
c942700427 Merge pull request #19029 from holiman/update1.8
Update1.8
v1.8.23
2019-02-20 10:48:12 +02:00
cde35439e0 params, swarm: release Geth v1.8.23, Swarm v0.3.11 2019-02-20 10:42:02 +02:00
4f908db69e cmd/utils: allow for multiple influxdb tags (#18520)
This PR is replacing the metrics.influxdb.host.tag cmd-line flag with metrics.influxdb.tags - a comma-separated key/value tags, that are passed to the InfluxDB reporter, so that we can index measurements with multiple tags, and not just one host tag.

This will be useful for Swarm, where we want to index measurements not just with the host tag, but also with bzzkey and git commit version (for long-running deployments).

(cherry picked from commit 21acf0bc8d)
2019-02-19 17:34:48 +01:00
320d132925 swarm/metrics: Send the accounting registry to InfluxDB (#18470)
(cherry picked from commit f28da4f602)
2019-02-19 17:34:42 +01:00
7ae2a7bd84 swarm: Reinstate Pss Protocol add call through swarm service (#19117)
* swarm: Reinstate Pss Protocol add call through swarm service

* swarm: Even less self

(cherry picked from commit d88c6ce6b0)
2019-02-19 13:18:10 +01:00
fd34bf594c contracts/*: golint updates for this or self warning
(cherry picked from commit 53b823afc8)
2019-02-19 13:18:02 +01:00
996230174c cmd/swarm/swarm-smoke: Trigger chunk debug on timeout (#19101)
* cmd/swarm/swarm-smoke: first version trigger has-chunks on timeout

* cmd/swarm/swarm-smoke: finalize trigger to chunk debug

* cmd/swarm/swarm-smoke: fixed httpEndpoint for trigger

* cmd/swarm/swarm-smoke: port

* cmd/swarm/swarm-smoke: ws not rpc

* cmd/swarm/swarm-smoke: added debug output

* cmd/swarm/swarm-smoke: addressed PR comments

* cmd/swarm/swarm-smoke: renamed track-timeout and track-chunks

(cherry picked from commit 62d7688d0a)
2019-02-19 13:11:53 +01:00
8857707606 p2p, swarm: fix node up races by granular locking (#18976)
* swarm/network: DRY out repeated giga comment

I not necessarily agree with the way we wait for event propagation.
But I truly disagree with having duplicated giga comments.

* p2p/simulations: encapsulate Node.Up field so we avoid data races

The Node.Up field was accessed concurrently without "proper" locking.
There was a lock on Network and that was used sometimes to access
the  field. Other times the locking was missed and we had
a data race.

For example: https://github.com/ethereum/go-ethereum/pull/18464
The case above was solved, but there were still intermittent/hard to
reproduce races. So let's solve the issue permanently.

resolves: ethersphere/go-ethereum#1146

* p2p/simulations: fix unmarshal of simulations.Node

Making Node.Up field private in 13292ee897
broke TestHTTPNetwork and TestHTTPSnapshot. Because the default
UnmarshalJSON does not handle unexported fields.

Important: The fix is partial and not proper to my taste. But I cut
scope as I think the fix may require a change to the current
serialization format. New ticket:
https://github.com/ethersphere/go-ethereum/issues/1177

* p2p/simulations: Add a sanity test case for Node.Config UnmarshalJSON

* p2p/simulations: revert back to defer Unlock() pattern for Network

It's a good patten to call `defer Unlock()` right after `Lock()` so
(new) error cases won't miss to unlock. Let's get back to that pattern.

The patten was abandoned in 85a79b3ad3,
while fixing a data race. That data race does not exist anymore,
since the Node.Up field got hidden behind its own lock.

* p2p/simulations: consistent naming for test providers Node.UnmarshalJSON

* p2p/simulations: remove JSON annotation from private fields of Node

As unexported fields are not serialized.

* p2p/simulations: fix deadlock in Network.GetRandomDownNode()

Problem: GetRandomDownNode() locks -> getDownNodeIDs() ->
GetNodes() tries to lock -> deadlock

On Network type, unexported functions must assume that `net.lock`
is already acquired and should not call exported functions which
might try to lock again.

* p2p/simulations: ensure method conformity for Network

Connect* methods were moved to p2p/simulations.Network from
swarm/network/simulation. However these new methods did not follow
the pattern of Network methods, i.e., all exported method locks
the whole Network either for read or write.

* p2p/simulations: fix deadlock during network shutdown

`TestDiscoveryPersistenceSimulationSimAdapter` often got into deadlock.
The execution was stuck on two locks, i.e, `Kademlia.lock` and
`p2p/simulations.Network.lock`. Usually the test got stuck once in each
20 executions with high confidence.

`Kademlia` was stuck in `Kademlia.EachAddr()` and `Network` in
`Network.Stop()`.

Solution: in `Network.Stop()` `net.lock` must be released before
calling `node.Stop()` as stopping a node (somehow - I did not find
the exact code path) causes `Network.InitConn()` to be called from
`Kademlia.SuggestPeer()` and that blocks on `net.lock`.

Related ticket: https://github.com/ethersphere/go-ethereum/issues/1223

* swarm/state: simplify if statement in DBStore.Put()

* p2p/simulations: remove faulty godoc from private function

The comment started with the wrong method name.

The method is simple and self explanatory. Also, it's private.
=> Let's just remove the comment.

(cherry picked from commit 50b872bf05)
2019-02-19 13:11:52 +01:00
d6c1fcbe04 swarm/pss: refactoring (#19110)
* swarm/pss: split pss and keystore

* swarm/pss: moved whisper to keystore

* swarm/pss: goimports fixed

(cherry picked from commit 12ca3b172a)
2019-02-19 13:11:52 +01:00
79cac793c0 swarm/storage/netstore: add fetcher cancellation on shutdown (#19049)
swarm/network/stream: remove netstore internal wg
swarm/network/stream: run individual tests with t.Run

(cherry picked from commit 3ee09ba035)
2019-02-19 13:11:52 +01:00
5de6b6b529 swarm/network: Saturation check for healthy networks (#19071)
* swarm/network: new saturation for  implementation

* swarm/network: re-added saturation func in Kademlia as it is used elsewhere

* swarm/network: saturation with higher MinBinSize

* swarm/network: PeersPerBin with depth check

* swarm/network: edited tests to pass new saturated check

* swarm/network: minor fix saturated check

* swarm/network/simulations/discovery: fixed renamed RPC call

* swarm/network: renamed to isSaturated and returns bool

* swarm/network: early depth check

(cherry picked from commit 2af24724dd)
2019-02-19 13:11:52 +01:00
3d2bedf8d0 swarm/storage: fix influxdb gc metrics report (#19102)
(cherry picked from commit 5b8ae7885e)
2019-02-19 13:11:52 +01:00
8ea3d8ad90 swarm: fix network/stream data races (#19051)
* swarm/network/stream: newStreamerTester cleanup only if err is nil

* swarm/network/stream: raise newStreamerTester waitForPeers timeout

* swarm/network/stream: fix data races in GetPeerSubscriptions

* swarm/storage: prevent data race on LDBStore.batchesC

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461775049

* swarm/network/stream: fix TestGetSubscriptionsRPC data race

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461768477

* swarm/network/stream: correctly use Simulation.Run callback

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461783804

* swarm/network: protect addrCountC in Kademlia.AddrCountC function

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-462273444

* p2p/simulations: fix a deadlock calling getRandomNode with lock

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-462317407

* swarm/network/stream: terminate disconnect goruotines in tests

* swarm/network/stream: reduce memory consumption when testing data races

* swarm/network/stream: add watchDisconnections helper function

* swarm/network/stream: add concurrent counter for tests

* swarm/network/stream: rename race/norace test files and use const

* swarm/network/stream: remove watchSim and its panic

* swarm/network/stream: pass context in watchDisconnections

* swarm/network/stream: add concurrent safe bool for watchDisconnections

* swarm/storage: fix LDBStore.batchesC data race by not closing it

(cherry picked from commit 3fd6db2bf6)
2019-02-19 13:11:52 +01:00
a0127019c3 swarm: fix uptime gauge update goroutine leak by introducing cleanup functions (#19040)
(cherry picked from commit d596bea2d5)
2019-02-19 13:11:51 +01:00
7a333e4104 swarm/storage: fix HashExplore concurrency bug ethersphere#1211 (#19028)
* swarm/storage: fix HashExplore concurrency bug ethersphere#1211

*  swarm/storage: lock as value not pointer

* swarm/storage: wait for  to complete

* swarm/storage: fix linter problems

* swarm/storage: append to nil slice

(cherry picked from commit 3d22a46c94)
2019-02-19 13:11:51 +01:00
799fe99537 swarm/pss: mutex lifecycle fixed (#19045)
(cherry picked from commit b30109df3c)
2019-02-19 13:11:51 +01:00
3b02b0ba4b swarm/docker: add global-store and split docker images (#19038)
(cherry picked from commit 6cb7d52a29)
2019-02-19 13:11:51 +01:00
85217b08bd cmd/swarm/global-store: global store cmd (#19014)
(cherry picked from commit 33d0a0efa6)
2019-02-19 13:11:51 +01:00
dcff622d43 swarm: CI race detector test adjustments (#19017)
(cherry picked from commit 27e3f96819)
2019-02-19 13:11:51 +01:00
a3db00f270 swarm/network: refactor simulation tests bootstrap (#18975)
(cherry picked from commit 597597e8b2)
2019-02-19 13:11:50 +01:00
769e43e334 swarm: GetPeerSubscriptions RPC (#18972)
(cherry picked from commit 43e1b7b124)
2019-02-19 13:11:50 +01:00
8d8ddea1a3 swarm/pss: transition to whisper v6 (#19023)
(cherry picked from commit cde02e017e)
2019-02-19 13:09:10 +01:00
068725c5b0 swarm/network, swarm/storage: Preserve opentracing contexts (#19022)
(cherry picked from commit 0c10d37606)
2019-02-19 13:09:09 +01:00
710775f435 swarm/network: fix data race in fetcher_test.go (#18469)
(cherry picked from commit 19bfcbf911)
2019-02-19 13:09:09 +01:00
0fd0108507 swarm/pss: Remove pss service leak in test (#18992)
(cherry picked from commit 7c60d0a6a2)
2019-02-19 13:06:14 +01:00
3c62cc6bba swarm/storage: fix test timeout with -race by increasing mget timeout
(cherry picked from commit 1c3aa8d9b1)
2019-02-19 13:06:14 +01:00
333b1bfb6c swarm/storage/localstore: new localstore package (#19015)
(cherry picked from commit 4f3d22f06c)
2019-02-19 13:06:14 +01:00
d1ace4f344 swarm: Debug API and HasChunks() API endpoint (#18980)
(cherry picked from commit 41597c2856)
2019-02-19 13:06:13 +01:00
637a75d61a cmd/swarm/swarm-smoke: refactor generateEndpoints (#19006)
(cherry picked from commit d212535ddd)
2019-02-19 13:05:55 +01:00
355d55bd34 cmd/swarm/swarm-smoke: remove wrong metrics (#18970)
(cherry picked from commit c5c9cef5c0)
2019-02-19 13:05:37 +01:00
7038b5734c cmd/swarm/swarm-smoke: sliding window test (#18967)
(cherry picked from commit b91bf08876)
2019-02-19 13:05:26 +01:00
1ecf2860cf cmd/swarm: hashes command (#19008)
(cherry picked from commit 7f55b0cbd8)
2019-02-19 12:57:53 +01:00
034f65e9e8 swarm/storage: Get all chunk references for a given file (#19002)
(cherry picked from commit 3eff652a7b)
2019-02-19 12:57:53 +01:00
607a1968e6 swarm/network: Remove extra random peer, connect test sanity, comments (#18964)
(cherry picked from commit f9401ae011)
2019-02-19 12:57:53 +01:00
3f54994db0 swarm: fix flaky delivery tests (#18971)
(cherry picked from commit 592bf6a59c)
2019-02-19 12:57:53 +01:00
2695aa9e0d p2p/testing, swarm: remove unused testing.T in protocol tester (#18500)
(cherry picked from commit 2abeb35d54)
2019-02-19 12:56:31 +01:00
e247dcc141 swarm/version: commit version added (#18510)
(cherry picked from commit ad13d2d407)
2019-02-19 12:56:31 +01:00
b774d0a507 swarm: fix a data race on startTime (#18511)
(cherry picked from commit fa34429a26)
2019-02-19 12:56:30 +01:00
4976fcc91a swarm: bootnode-mode, new bootnodes and no p2p package discovery (#18498)
(cherry picked from commit bbd120354a)
2019-02-19 12:56:30 +01:00
878aa58ec6 cmd/swarm: use resetting timer to measure fetch time (#18474)
(cherry picked from commit a0b0db6305)
2019-02-19 12:56:30 +01:00
475a0664c5 p2p/simulations: fix data race on swarm/network/simulations (#18464)
(cherry picked from commit 85a79b3ad3)
2019-02-19 12:56:30 +01:00
4625b1257f cmd/swarm/swarm-smoke: use ResettingTimer instead of Counters for times (#18479)
(cherry picked from commit 560957799a)
2019-02-19 12:56:30 +01:00
21d54bcaac cmd/swarm/swarm-snapshot: disable tests on windows (#18478)
(cherry picked from commit 632135ce4c)
2019-02-19 12:56:30 +01:00
7383db4dac Upload speed (#18442)
(cherry picked from commit 257bfff316)
2019-02-19 12:56:30 +01:00
afb65f6ace swarm/network: fix data race warning on TestBzzHandshakeLightNode (#18459)
(cherry picked from commit 81e26d5a48)
2019-02-19 12:55:18 +01:00
1f1c751b6e swarm/network: rewrite of peer suggestion engine, fix skipped tests (#18404)
* swarm/network: fix skipped tests related to suggestPeer

* swarm/network: rename depth to radius

* swarm/network: uncomment assertHealth and improve comments

* swarm/network: remove commented code

* swarm/network: kademlia suggestPeer algo correction

* swarm/network: kademlia suggest peer

 * simplify suggest Peer code
 * improve peer suggestion algo
 * add comments
 * kademlia testing improvements
   * assertHealth -> checkHealth (test helper)
   * testSuggestPeer -> checkSuggestPeer (test helper)
   * remove testSuggestPeerBug and TestKademliaCase

* swarm/network: kademlia suggestPeer cleanup, improved comments

* swarm/network: minor comment, discovery test default arg

(cherry picked from commit bcb2594151)
2019-02-19 12:55:07 +01:00
a3f31f51f3 cmd/swarm/swarm-snapshot: swarm snapshot generator (#18453)
* cmd/swarm/swarm-snapshot: add binary to create network snapshots

* cmd/swarm/swarm-snapshot: refactor and extend tests

* p2p/simulations: remove unused triggerChecks func and fix linter

* internal/cmdtest: raise the timeout for killing TestCmd

* cmd/swarm/swarm-snapshot: add more comments and other minor adjustments

* cmd/swarm/swarm-snapshot: remove redundant check in createSnapshot

* cmd/swarm/swarm-snapshot: change comment wording

* p2p/simulations: revert Simulation.Run from master

https://github.com/ethersphere/go-ethereum/pull/1077/files#r247078904

* cmd/swarm/swarm-snapshot: address pr comments

* swarm/network/simulations/discovery: removed snapshot write to file

* cmd/swarm/swarm-snapshot, swarm/network/simulations: removed redundant connection event check, fixed lint error

(cherry picked from commit 34f11e752f)
2019-02-19 12:54:56 +01:00