314 Commits

Author SHA1 Message Date
Péter Szilágyi
009d2fe2d6 params, swarm: release Geth v1.8.24 (noop Swarm 0.3.12) 2019-04-08 16:06:59 +03:00
Péter Szilágyi
cde35439e0 params, swarm: release Geth v1.8.23, Swarm v0.3.11 2019-02-20 10:42:02 +02:00
Anton Evangelatov
4f908db69e cmd/utils: allow for multiple influxdb tags (#18520)
This PR is replacing the metrics.influxdb.host.tag cmd-line flag with metrics.influxdb.tags - a comma-separated key/value tags, that are passed to the InfluxDB reporter, so that we can index measurements with multiple tags, and not just one host tag.

This will be useful for Swarm, where we want to index measurements not just with the host tag, but also with bzzkey and git commit version (for long-running deployments).

(cherry picked from commit 21acf0bc8d)
2019-02-19 17:34:48 +01:00
Jerzy Lasyk
320d132925 swarm/metrics: Send the accounting registry to InfluxDB (#18470)
(cherry picked from commit f28da4f602)
2019-02-19 17:34:42 +01:00
lash
7ae2a7bd84 swarm: Reinstate Pss Protocol add call through swarm service (#19117)
* swarm: Reinstate Pss Protocol add call through swarm service

* swarm: Even less self

(cherry picked from commit d88c6ce6b0)
2019-02-19 13:18:10 +01:00
Kiel barry
fd34bf594c contracts/*: golint updates for this or self warning
(cherry picked from commit 53b823afc8)
2019-02-19 13:18:02 +01:00
Ferenc Szabo
8857707606 p2p, swarm: fix node up races by granular locking (#18976)
* swarm/network: DRY out repeated giga comment

I not necessarily agree with the way we wait for event propagation.
But I truly disagree with having duplicated giga comments.

* p2p/simulations: encapsulate Node.Up field so we avoid data races

The Node.Up field was accessed concurrently without "proper" locking.
There was a lock on Network and that was used sometimes to access
the  field. Other times the locking was missed and we had
a data race.

For example: https://github.com/ethereum/go-ethereum/pull/18464
The case above was solved, but there were still intermittent/hard to
reproduce races. So let's solve the issue permanently.

resolves: ethersphere/go-ethereum#1146

* p2p/simulations: fix unmarshal of simulations.Node

Making Node.Up field private in 13292ee897
broke TestHTTPNetwork and TestHTTPSnapshot. Because the default
UnmarshalJSON does not handle unexported fields.

Important: The fix is partial and not proper to my taste. But I cut
scope as I think the fix may require a change to the current
serialization format. New ticket:
https://github.com/ethersphere/go-ethereum/issues/1177

* p2p/simulations: Add a sanity test case for Node.Config UnmarshalJSON

* p2p/simulations: revert back to defer Unlock() pattern for Network

It's a good patten to call `defer Unlock()` right after `Lock()` so
(new) error cases won't miss to unlock. Let's get back to that pattern.

The patten was abandoned in 85a79b3ad3,
while fixing a data race. That data race does not exist anymore,
since the Node.Up field got hidden behind its own lock.

* p2p/simulations: consistent naming for test providers Node.UnmarshalJSON

* p2p/simulations: remove JSON annotation from private fields of Node

As unexported fields are not serialized.

* p2p/simulations: fix deadlock in Network.GetRandomDownNode()

Problem: GetRandomDownNode() locks -> getDownNodeIDs() ->
GetNodes() tries to lock -> deadlock

On Network type, unexported functions must assume that `net.lock`
is already acquired and should not call exported functions which
might try to lock again.

* p2p/simulations: ensure method conformity for Network

Connect* methods were moved to p2p/simulations.Network from
swarm/network/simulation. However these new methods did not follow
the pattern of Network methods, i.e., all exported method locks
the whole Network either for read or write.

* p2p/simulations: fix deadlock during network shutdown

`TestDiscoveryPersistenceSimulationSimAdapter` often got into deadlock.
The execution was stuck on two locks, i.e, `Kademlia.lock` and
`p2p/simulations.Network.lock`. Usually the test got stuck once in each
20 executions with high confidence.

`Kademlia` was stuck in `Kademlia.EachAddr()` and `Network` in
`Network.Stop()`.

Solution: in `Network.Stop()` `net.lock` must be released before
calling `node.Stop()` as stopping a node (somehow - I did not find
the exact code path) causes `Network.InitConn()` to be called from
`Kademlia.SuggestPeer()` and that blocks on `net.lock`.

Related ticket: https://github.com/ethersphere/go-ethereum/issues/1223

* swarm/state: simplify if statement in DBStore.Put()

* p2p/simulations: remove faulty godoc from private function

The comment started with the wrong method name.

The method is simple and self explanatory. Also, it's private.
=> Let's just remove the comment.

(cherry picked from commit 50b872bf05)
2019-02-19 13:11:52 +01:00
gluk256
d6c1fcbe04 swarm/pss: refactoring (#19110)
* swarm/pss: split pss and keystore

* swarm/pss: moved whisper to keystore

* swarm/pss: goimports fixed

(cherry picked from commit 12ca3b172a)
2019-02-19 13:11:52 +01:00
Elad
79cac793c0 swarm/storage/netstore: add fetcher cancellation on shutdown (#19049)
swarm/network/stream: remove netstore internal wg
swarm/network/stream: run individual tests with t.Run

(cherry picked from commit 3ee09ba035)
2019-02-19 13:11:52 +01:00
holisticode
5de6b6b529 swarm/network: Saturation check for healthy networks (#19071)
* swarm/network: new saturation for  implementation

* swarm/network: re-added saturation func in Kademlia as it is used elsewhere

* swarm/network: saturation with higher MinBinSize

* swarm/network: PeersPerBin with depth check

* swarm/network: edited tests to pass new saturated check

* swarm/network: minor fix saturated check

* swarm/network/simulations/discovery: fixed renamed RPC call

* swarm/network: renamed to isSaturated and returns bool

* swarm/network: early depth check

(cherry picked from commit 2af24724dd)
2019-02-19 13:11:52 +01:00
Elad
3d2bedf8d0 swarm/storage: fix influxdb gc metrics report (#19102)
(cherry picked from commit 5b8ae7885e)
2019-02-19 13:11:52 +01:00
Janoš Guljaš
8ea3d8ad90 swarm: fix network/stream data races (#19051)
* swarm/network/stream: newStreamerTester cleanup only if err is nil

* swarm/network/stream: raise newStreamerTester waitForPeers timeout

* swarm/network/stream: fix data races in GetPeerSubscriptions

* swarm/storage: prevent data race on LDBStore.batchesC

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461775049

* swarm/network/stream: fix TestGetSubscriptionsRPC data race

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461768477

* swarm/network/stream: correctly use Simulation.Run callback

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461783804

* swarm/network: protect addrCountC in Kademlia.AddrCountC function

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-462273444

* p2p/simulations: fix a deadlock calling getRandomNode with lock

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-462317407

* swarm/network/stream: terminate disconnect goruotines in tests

* swarm/network/stream: reduce memory consumption when testing data races

* swarm/network/stream: add watchDisconnections helper function

* swarm/network/stream: add concurrent counter for tests

* swarm/network/stream: rename race/norace test files and use const

* swarm/network/stream: remove watchSim and its panic

* swarm/network/stream: pass context in watchDisconnections

* swarm/network/stream: add concurrent safe bool for watchDisconnections

* swarm/storage: fix LDBStore.batchesC data race by not closing it

(cherry picked from commit 3fd6db2bf6)
2019-02-19 13:11:52 +01:00
Elad
a0127019c3 swarm: fix uptime gauge update goroutine leak by introducing cleanup functions (#19040)
(cherry picked from commit d596bea2d5)
2019-02-19 13:11:51 +01:00
holisticode
7a333e4104 swarm/storage: fix HashExplore concurrency bug ethersphere#1211 (#19028)
* swarm/storage: fix HashExplore concurrency bug ethersphere#1211

*  swarm/storage: lock as value not pointer

* swarm/storage: wait for  to complete

* swarm/storage: fix linter problems

* swarm/storage: append to nil slice

(cherry picked from commit 3d22a46c94)
2019-02-19 13:11:51 +01:00
gluk256
799fe99537 swarm/pss: mutex lifecycle fixed (#19045)
(cherry picked from commit b30109df3c)
2019-02-19 13:11:51 +01:00
Rafael Matias
3b02b0ba4b swarm/docker: add global-store and split docker images (#19038)
(cherry picked from commit 6cb7d52a29)
2019-02-19 13:11:51 +01:00
Janoš Guljaš
85217b08bd cmd/swarm/global-store: global store cmd (#19014)
(cherry picked from commit 33d0a0efa6)
2019-02-19 13:11:51 +01:00
Ferenc Szabo
dcff622d43 swarm: CI race detector test adjustments (#19017)
(cherry picked from commit 27e3f96819)
2019-02-19 13:11:51 +01:00
Anton Evangelatov
a3db00f270 swarm/network: refactor simulation tests bootstrap (#18975)
(cherry picked from commit 597597e8b2)
2019-02-19 13:11:50 +01:00
holisticode
769e43e334 swarm: GetPeerSubscriptions RPC (#18972)
(cherry picked from commit 43e1b7b124)
2019-02-19 13:11:50 +01:00
gluk256
8d8ddea1a3 swarm/pss: transition to whisper v6 (#19023)
(cherry picked from commit cde02e017e)
2019-02-19 13:09:10 +01:00
lash
068725c5b0 swarm/network, swarm/storage: Preserve opentracing contexts (#19022)
(cherry picked from commit 0c10d37606)
2019-02-19 13:09:09 +01:00
Ferenc Szabo
710775f435 swarm/network: fix data race in fetcher_test.go (#18469)
(cherry picked from commit 19bfcbf911)
2019-02-19 13:09:09 +01:00
lash
0fd0108507 swarm/pss: Remove pss service leak in test (#18992)
(cherry picked from commit 7c60d0a6a2)
2019-02-19 13:06:14 +01:00
Ferenc Szabo
3c62cc6bba swarm/storage: fix test timeout with -race by increasing mget timeout
(cherry picked from commit 1c3aa8d9b1)
2019-02-19 13:06:14 +01:00
Janoš Guljaš
333b1bfb6c swarm/storage/localstore: new localstore package (#19015)
(cherry picked from commit 4f3d22f06c)
2019-02-19 13:06:14 +01:00
holisticode
d1ace4f344 swarm: Debug API and HasChunks() API endpoint (#18980)
(cherry picked from commit 41597c2856)
2019-02-19 13:06:13 +01:00
holisticode
1ecf2860cf cmd/swarm: hashes command (#19008)
(cherry picked from commit 7f55b0cbd8)
2019-02-19 12:57:53 +01:00
holisticode
034f65e9e8 swarm/storage: Get all chunk references for a given file (#19002)
(cherry picked from commit 3eff652a7b)
2019-02-19 12:57:53 +01:00
lash
607a1968e6 swarm/network: Remove extra random peer, connect test sanity, comments (#18964)
(cherry picked from commit f9401ae011)
2019-02-19 12:57:53 +01:00
Janoš Guljaš
3f54994db0 swarm: fix flaky delivery tests (#18971)
(cherry picked from commit 592bf6a59c)
2019-02-19 12:57:53 +01:00
Elad
2695aa9e0d p2p/testing, swarm: remove unused testing.T in protocol tester (#18500)
(cherry picked from commit 2abeb35d54)
2019-02-19 12:56:31 +01:00
gluk256
e247dcc141 swarm/version: commit version added (#18510)
(cherry picked from commit ad13d2d407)
2019-02-19 12:56:31 +01:00
Janoš Guljaš
b774d0a507 swarm: fix a data race on startTime (#18511)
(cherry picked from commit fa34429a26)
2019-02-19 12:56:30 +01:00
Anton Evangelatov
4976fcc91a swarm: bootnode-mode, new bootnodes and no p2p package discovery (#18498)
(cherry picked from commit bbd120354a)
2019-02-19 12:56:30 +01:00
Elad
afb65f6ace swarm/network: fix data race warning on TestBzzHandshakeLightNode (#18459)
(cherry picked from commit 81e26d5a48)
2019-02-19 12:55:18 +01:00
Viktor Trón
1f1c751b6e swarm/network: rewrite of peer suggestion engine, fix skipped tests (#18404)
* swarm/network: fix skipped tests related to suggestPeer

* swarm/network: rename depth to radius

* swarm/network: uncomment assertHealth and improve comments

* swarm/network: remove commented code

* swarm/network: kademlia suggestPeer algo correction

* swarm/network: kademlia suggest peer

 * simplify suggest Peer code
 * improve peer suggestion algo
 * add comments
 * kademlia testing improvements
   * assertHealth -> checkHealth (test helper)
   * testSuggestPeer -> checkSuggestPeer (test helper)
   * remove testSuggestPeerBug and TestKademliaCase

* swarm/network: kademlia suggestPeer cleanup, improved comments

* swarm/network: minor comment, discovery test default arg

(cherry picked from commit bcb2594151)
2019-02-19 12:55:07 +01:00
Elad
a3f31f51f3 cmd/swarm/swarm-snapshot: swarm snapshot generator (#18453)
* cmd/swarm/swarm-snapshot: add binary to create network snapshots

* cmd/swarm/swarm-snapshot: refactor and extend tests

* p2p/simulations: remove unused triggerChecks func and fix linter

* internal/cmdtest: raise the timeout for killing TestCmd

* cmd/swarm/swarm-snapshot: add more comments and other minor adjustments

* cmd/swarm/swarm-snapshot: remove redundant check in createSnapshot

* cmd/swarm/swarm-snapshot: change comment wording

* p2p/simulations: revert Simulation.Run from master

https://github.com/ethersphere/go-ethereum/pull/1077/files#r247078904

* cmd/swarm/swarm-snapshot: address pr comments

* swarm/network/simulations/discovery: removed snapshot write to file

* cmd/swarm/swarm-snapshot, swarm/network/simulations: removed redundant connection event check, fixed lint error

(cherry picked from commit 34f11e752f)
2019-02-19 12:54:56 +01:00
Janoš Guljaš
e63995b3f3 swarm/network: fix data race in TestNetworkID test (#18460)
(cherry picked from commit 96c7c18b18)
2019-02-19 12:54:10 +01:00
Janoš Guljaš
dd3e894747 swarm/storage: fix mockNetFetcher data races (#18462)
fixes: ethersphere/go-ethereum#1117
(cherry picked from commit f728837ee6)
2019-02-19 12:54:10 +01:00
Felix Lange
7fa3509e2e params, swarm/version: Geth 1.8.22-stable, Swarm 0.3.10-stable 2019-01-31 11:52:18 +01:00
Péter Szilágyi
9dc5d1a915 params, swarm: release Geth v1.8.21 and Swarm v0.3.9 2019-01-15 22:51:31 +02:00
gluk256
4aeeecfded swarm/pot: each() functions refactored (#18452) 2019-01-15 11:51:33 +01:00
gluk256
1636d9574b swarm/pot: pot.remove fixed (#18431)
* swarm/pot: refactored pot.remove(), updated comments

* swarm/pot: comments updated
2019-01-11 20:42:33 +01:00
holisticode
88168ff5c5 Stream subscriptions (#18355)
* swarm/network: eachBin now starts at kaddepth for nn

* swarm/network: fix Kademlia.EachBin

* swarm/network: fix kademlia.EachBin

* swarm/network: correct EachBin implementation according to requirements

* swarm/network: less addresses simplified tests

* swarm: calc kad depth outside loop in EachBin test

* swarm/network: removed printResults

* swarm/network: cleanup imports

* swarm/network: remove kademlia.EachBin; fix RequestSubscriptions and add unit test

* swarm/network/stream: address PR comments

* swarm/network/stream: package-wide subscriptionFunc

* swarm/network/stream: refactor to kad.EachConn
2019-01-11 15:08:09 +01:00
Ferenc Szabo
2eb838ed97 p2p/simulations: eliminate concept of pivot (#18426) 2019-01-11 10:23:45 +01:00
lash
7240f4d800 swarm/network: Rename minproxbinsize, add as member of simulation (#18408)
* swarm/network: Rename minproxbinsize, add as member of simulation

* swarm/network: Deactivate WaitTillHealthy, unreliable pending suggestpeer
2019-01-10 12:33:51 +01:00
Viktor Trón
6df3e4eeb0 swarm/network: remove isproxbin bool from kad.Each* iterfunc (#18239)
* swarm/network, swarm/pss: remove isproxbin bool from kad.Each* iterfunc

* swarm/network: restore comment and unskip snapshot sync tests
2019-01-10 03:36:19 +01:00
Janoš Guljaš
d70c4faf20 swarm: Fix T.Fatal inside a goroutine in tests (#18409)
* swarm/storage: fix T.Fatal inside a goroutine

* swarm/network/simulation: fix T.Fatal inside a goroutine

* swarm/network/stream: fix T.Fatal inside a goroutine

* swarm/network/simulation: consistent failures in TestPeerEventsTimeout

* swarm/network/simulation: rename sendRunSignal to triggerSimulationRun
2019-01-09 07:05:55 +01:00
holisticode
ae857e74bf swarm, p2p/protocols: Stream accounting (#18337)
* swarm: completed 1st phase of swap accounting

* swarm, p2p/protocols: added stream pricing

* swarm/network/stream: gofmt simplify stream.go

* swarm: fixed review comments

* swarm: used snapshots for swap tests

* swarm: custom retrieve for swap (less cascaded requests at any one time)

* swarm: addressed PR comments

* swarm: log output formatting

* swarm: removed parallelism in swap tests

* swarm: swap tests simplification

* swarm: removed swap_test.go

* swarm/network/stream: added prefix space for comments

* swarm/network/stream: unit test for prices

* swarm/network/stream: don't hardcode price

* swarm/network/stream: fixed invalid price check
2019-01-08 00:59:00 +01:00