This resolves a minor issue where neighbors responses containing less
than 16 nodes would bump the failure counter, removing the node. One
situation where this can happen is a private deployment where the total
number of extant nodes is less than 16.
Issue found by @jsying.
This PR is replacing the metrics.influxdb.host.tag cmd-line flag with metrics.influxdb.tags - a comma-separated key/value tags, that are passed to the InfluxDB reporter, so that we can index measurements with multiple tags, and not just one host tag.
This will be useful for Swarm, where we want to index measurements not just with the host tag, but also with bzzkey and git commit version (for long-running deployments).
(cherry picked from commit 21acf0bc8d)
* swarm/network: DRY out repeated giga comment
I not necessarily agree with the way we wait for event propagation.
But I truly disagree with having duplicated giga comments.
* p2p/simulations: encapsulate Node.Up field so we avoid data races
The Node.Up field was accessed concurrently without "proper" locking.
There was a lock on Network and that was used sometimes to access
the field. Other times the locking was missed and we had
a data race.
For example: https://github.com/ethereum/go-ethereum/pull/18464
The case above was solved, but there were still intermittent/hard to
reproduce races. So let's solve the issue permanently.
resolves: ethersphere/go-ethereum#1146
* p2p/simulations: fix unmarshal of simulations.Node
Making Node.Up field private in 13292ee897
broke TestHTTPNetwork and TestHTTPSnapshot. Because the default
UnmarshalJSON does not handle unexported fields.
Important: The fix is partial and not proper to my taste. But I cut
scope as I think the fix may require a change to the current
serialization format. New ticket:
https://github.com/ethersphere/go-ethereum/issues/1177
* p2p/simulations: Add a sanity test case for Node.Config UnmarshalJSON
* p2p/simulations: revert back to defer Unlock() pattern for Network
It's a good patten to call `defer Unlock()` right after `Lock()` so
(new) error cases won't miss to unlock. Let's get back to that pattern.
The patten was abandoned in 85a79b3ad3,
while fixing a data race. That data race does not exist anymore,
since the Node.Up field got hidden behind its own lock.
* p2p/simulations: consistent naming for test providers Node.UnmarshalJSON
* p2p/simulations: remove JSON annotation from private fields of Node
As unexported fields are not serialized.
* p2p/simulations: fix deadlock in Network.GetRandomDownNode()
Problem: GetRandomDownNode() locks -> getDownNodeIDs() ->
GetNodes() tries to lock -> deadlock
On Network type, unexported functions must assume that `net.lock`
is already acquired and should not call exported functions which
might try to lock again.
* p2p/simulations: ensure method conformity for Network
Connect* methods were moved to p2p/simulations.Network from
swarm/network/simulation. However these new methods did not follow
the pattern of Network methods, i.e., all exported method locks
the whole Network either for read or write.
* p2p/simulations: fix deadlock during network shutdown
`TestDiscoveryPersistenceSimulationSimAdapter` often got into deadlock.
The execution was stuck on two locks, i.e, `Kademlia.lock` and
`p2p/simulations.Network.lock`. Usually the test got stuck once in each
20 executions with high confidence.
`Kademlia` was stuck in `Kademlia.EachAddr()` and `Network` in
`Network.Stop()`.
Solution: in `Network.Stop()` `net.lock` must be released before
calling `node.Stop()` as stopping a node (somehow - I did not find
the exact code path) causes `Network.InitConn()` to be called from
`Kademlia.SuggestPeer()` and that blocks on `net.lock`.
Related ticket: https://github.com/ethersphere/go-ethereum/issues/1223
* swarm/state: simplify if statement in DBStore.Put()
* p2p/simulations: remove faulty godoc from private function
The comment started with the wrong method name.
The method is simple and self explanatory. Also, it's private.
=> Let's just remove the comment.
(cherry picked from commit 50b872bf05)
* swarm/network: new saturation for implementation
* swarm/network: re-added saturation func in Kademlia as it is used elsewhere
* swarm/network: saturation with higher MinBinSize
* swarm/network: PeersPerBin with depth check
* swarm/network: edited tests to pass new saturated check
* swarm/network: minor fix saturated check
* swarm/network/simulations/discovery: fixed renamed RPC call
* swarm/network: renamed to isSaturated and returns bool
* swarm/network: early depth check
(cherry picked from commit 2af24724dd)
* common/fdlimit: cap on MacOS file limits, fixes#18994
* common/fdlimit: fix Maximum-check to respect OPEN_MAX
* common/fdlimit: return error if OPEN_MAX is exceeded in Raise()
* common/fdlimit: goimports
* common/fdlimit: check value after setting fdlimit
* common/fdlimit: make comment a bit more descriptive
* cmd/utils: make fdlimit happy path a bit cleaner
(cherry picked from commit f48da43bae)
This change clears up confusion around the two ways in which nodes
can be added to the table.
When a neighbors packet is received as a reply to findnode, the nodes
contained in the reply are added as 'seen' entries if sufficient space
is available.
When a ping is received and the endpoint verification has taken place,
the remote node is added as a 'verified' entry or moved to the front of
the bucket if present. This also updates the node's IP address and port
if they have changed.
This change resolves multiple issues around handling of endpoint proofs.
The proof is now done separately for each IP and completing the proof
requires a matching ping hash.
Also remove waitping because it's equivalent to sleep. waitping was
slightly more efficient, but that may cause issues with findnode if
packets are reordered and the remote end sees findnode before pong.
Logging of received packets was hitherto done after handling the packet,
which meant that sent replies were logged before the packet that
generated them. This change splits up packet handling into 'preverify'
and 'handle'. The error from 'preverify' is logged, but 'handle' happens
after the message is logged. This fixes the order. Packet logs now
contain the node ID.
dput --passive should make repo pushes from Travis work again.
dput --no-upload-log works around an issue I had while uploading locally.
debuild -d says that debuild shouldn't check for build dependencies when
creating the source package. This option is needed to make builds work
in environments where the installed Go version doesn't match the
declared dependency in the source package.
This PR adds a new fork which disables EIP-1283. Internally it's called Petersburg,
but the genesis/config field is ConstantinopleFix.
The block numbers are:
7280000 for Constantinople on Mainnet
7280000 for ConstantinopleFix on Mainnet
4939394 for ConstantinopleFix on Ropsten
9999999 for ConstantinopleFix on Rinkeby (real number decided later)
This PR also defaults to using the same ConstantinopleFix number as whatever
Constantinople is set to. That is, it will default to mainnet behaviour if ConstantinopleFix
is not set.This means that for private networks which have already transitioned
to Constantinople, this PR will break the network unless ConstantinopleFix is
explicitly set!
* accounts/abi: fix name styling when unpacking abi fields w/ underscores
ABI fields with underscores that are being unpacked
into structs expect structs with following form:
int_one -> Int_one
whereas in abigen the generated structs are camelcased
int_one -> IntOne
so updated the unpack method to expect camelcased structs as well.
* swarm/network: Revised depth calculation with tests
* swarm/network: WIP remove redundant "full" function
* swarm/network: WIP peerpot refactor
* swarm/network: Make test methods submethod of peerpot and embed kad
* swarm/network: Remove commented out code
* swarm/network: Rename health test functions
* swarm/network: Too many n's
* swarm/network: Change hive Healthy func to accept addresses
* swarm/network: Add Healthy proxy method for api in hive
* swarm/network: Skip failing test out of scope for PR
* swarm/network: Skip all tests dependent on SuggestPeers
* swarm/network: Remove commented code and useless kad Pof member
* swarm/network: Remove more unused code, add counter on depth test errors
* swarm/network: WIP Create Healthy assertion tests
* swarm/network: Roll back health related methods receiver change
* swarm/network: Hardwire network minproxbinsize in swarm sim
* swarm/network: Rework Health test to strict
Pending add test for saturation
And add test for as many as possible up to saturation
* swarm/network: Skip discovery tests (dependent on SuggestPeer)
* swarm/network: Remove useless minProxBinSize in stream
* swarm/network: Remove unnecessary testing.T param to assert health
* swarm/network: Implement t.Helper() in checkHealth
* swarm/network: Rename check back to assert now that we have helper magic
* swarm/network: Revert WaitTillHealthy change (deferred to nxt PR)
* swarm/network: Kademlia tests GotNN => ConnectNN
* swarm/network: Renames and comments
* swarm/network: Add comments
* p2p/simulation: WIP minimal snapshot test
* p2p/simulation: Add snapshot create, load and verify to snapshot test
* build: add test tag for tests
* p2p/simulations, build: Revert travis change, build test sym always
* p2p/simulations: Add comments, timeout check on additional events
* p2p/simulation: Add benchmark template for minimal peer protocol init
* p2p/simulations: Remove unused code
* p2p/simulation: Correct timer reset
* p2p/simulations: Put snapshot check events in buffer and call blocking
* p2p/simulations: TestSnapshot fail if Load function returns early
* p2p/simulations: TestSnapshot wait for all connections before returning
* p2p/simulation: Revert to before wait for snap load (5e75594)
* p2p/simulations: add "conns after load" subtest to TestSnapshot
and nudge
* swarm/shed: add metrics to each shed db
* swarm/shed: push metrics prefix up
* swarm/shed: rename prefix to metricsPrefix
* swarm/shed: unexport Meter, remove Mutex for quit channel
* geth/core/eth: implement constantinople override flag
* les: implemnent constantinople override flag for les clients
* cmd/geth, eth, les: fix typo, move flag to experimentals
* Rejects peers that respond with a different hash for any of the passed in block numbers.
* Meant for emergency situations when the network forks unexpectedly.
* swarm/network: Hive - do not notify peer if discovery is disabled
* p2p/simulations: validate all connections on loading a snapshot
* p2p/simulations: track all connections in on snapshot loading
* p2p/simulations: add snapshotLoadTimeout variable
* p2p/simulations: ignore control events in snapshot load
* p2p/simulations: simplify event loop synchronization
* p2p/simulations: return already connected error from Load function
* p2p/simulations: log warning on snapshot loading disconnection
Until this commit, when sending an RPC request that called `NewEVM`, a blank `vm.Config`
would be taken so as to set some options, based on the default configuration. If some extra
configuration switches were passed to the blockchain, those would be ignored.
This PR adds a function to get the config from the blockchain, and this is what is now used
for RPC calls.
Some subsequent changes need to be made, see https://github.com/ethereum/go-ethereum/pull/17955#pullrequestreview-182237244
for the details of the discussion.
* swarm/network/stream/: added stream protocol version match tests
* Increase BZZ version due to streamer version change; version tests
* swarm/network: increased hive and test protocol version
TryUpdate does not call t.trie.TryUpdate(key, value) and calls t.trie.TryDelete
instead. The update operation simply deletes the corresponding entry, though
it could retrieve later by odr. However, it adds further network overhead.
- Have `${DataDir}/bzzd.ipc` as IPC path default.
- Respect the `--datadir` flag.
- Keep only the global `--ipcpath` flag and drop the local `--ipcpath` flag
as flags might overwrite each other. (Note: before global `--ipcpath`
was ignored even if it was set)
fixes ethersphere#795
- Replace "crypto/rand" to "math/rand" for files content generation
- Remove swarm/network_test.go.Shuffle and swarm/btm/btm_test.go.Shuffle - because go1.9 support dropped (see https://github.com/ethereum/go-ethereum/pull/17807 and comments to swarm/network_test.go.Shuffle)
On file access LDBStore's tryAccessIdx() function created a faulty
GC Index Data entry, because not indexing the ikey correctly.
That caused the chunk addresses/hashes to start with '00' and the last
two digits were dropped. => Incorrect chunk address.
Besides the fix, the commit also contains a schema change which will
run the CleanGCIndex() function to clean the GC index from erroneous
entries.
Note: CleanGCIndex() rebuilds the index from scratch which can take
a really-really long time with a huge DB (possibly an hour).
Access count was not incremented when chunk was retrieved
from cache. So the garbage collector might have deleted the most
frequently accessed chunk from disk.
Co-authored-by: Ferenc Szabo <ferenc.szabo@ethereum.org>
ethereum/go-ethereum#16734 introduced BlockHash to the FilterQuery
struct. However, ethclient was not updated to include BlockHash in the actual
RPC request.
* RequestFromPeers does not use peers marked as lightnode
* fix warning about variable name
* write tests for RequestFromPeers
* lightnodes should be omitted from the addressbook
* resolve pr comments regarding logging, formatting and comments
* resolve pr comments regarding comments and added a missing newline
* add assertions to check peers in live connections
* core: speed up GenerateChain
Use a mock implementation of ChainReader instead of creating
and destroying a BlockChain object for each generated block.
* eth/downloader: speed up tests by generating chain only once
This change reworks the downloader tests so they share a common test
blockchain instead of generating a chain in every test. The tests are
roughly twice as fast now.
* swarm: clean up unused private types and functions
Those that were identified by code inspection tool.
* swarm/storage: move/add Proximity GoDoc from deleted private function
The mentioned proximity() private function was deleted in:
1ca8fc1e6f
* cmd/clef: replace password arg with prompt (#17829)
Entering passwords on the command line is not secure as it is easy to recover from bash_history or the process table.
1. The clef command addpw was renamed to setpw to better describe the functionality
2. The <password> argument was removed and replaced with an interactive prompt
* cmd/clef: remove undeclared variable
This adds the global accumulated refund counter to the standard
json output as a numeric json value. Previously this was not very
interesting since it was not used much, but with the new sstore
gas changes the value is a lot more interesting from a consensus
investigation perspective.
* swarm/network/stream: disambiguate chunk delivery messages (retrieval vs syncing)
* swarm/network/stream: addressed PR comments
* swarm/network/stream: stream protocol version change due to new message types in this PR
* first impl of eth_getProof
* fixed docu
* added comments and refactored based on comments from holiman
* created structs
* handle errors correctly
* change Value to *hexutil.Big in order to have the same output as parity
* use ProofList as return type
This change extends the peer metrics collection:
- traces the life-cycle of the peers
- meters the peer traffic separately for every peer
- creates event feed for the peer events
- emits the peer events
* swarm/network/stream: generalize SetNextBatch and add Server SessionIndex
* swarm/network/stream: fix a typo in comment
* swarm/network/stream: remove live argument from NewSwarmSyncerServer
This PR adds enode.LocalNode and integrates it into the p2p
subsystem. This new object is the keeper of the local node
record. For now, a new version of the record is produced every
time the client restarts. We'll make it smarter to avoid that in
the future.
There are a couple of other changes in this commit: discovery now
waits for all of its goroutines at shutdown and the p2p server
now closes the node database after discovery has shut down. This
fixes a leveldb crash in tests. p2p server startup is faster
because it doesn't need to wait for the external IP query
anymore.
This fixes a rare deadlock with the inproc adapter:
- A node is stopped, which acquires Network.lock.
- The protocol code being simulated (swarm/network in my case)
waits for its goroutines to shut down.
- One of those goroutines calls into the simulation to add a peer,
which waits for Network.lock.
The fix for the deadlock is really simple, just release the lock
before stopping the simulation node.
Other changes in this PR clean up the exec adapter so it reports
node startup errors better and remove the docker adapter because
it just adds overhead.
In the exec adapter, node information is now posted to a one-shot
server. This avoids log parsing and allows reporting startup
errors to the simulation host.
A small change in package node was needed because simulation
nodes use port zero. Node.{HTTP,WS}Endpoint now return the live
endpoints after startup by checking the TCP listener.
Notifier tracks whether subscription are 'active'. A subscription
becomes active when the subscription ID has been sent to the client. If
the client sends notifications in the request handler before the
subscription becomes active they are dropped. The tests tried to work
around this problem by always waiting 5s before sending the first
notification.
Fix it by buffering notifications until the subscription becomes active.
This speeds up all subscription tests.
Also fix TestSubscriptionMultipleNamespaces to wait for three messages
per subscription instead of six. The test now finishes just after all
notifications have been received and doesn't hit the 30s timeout anymore.
* travis: exclude non-test jobs for PRs
We don't usually look at these builders and not starting them
removes ~15min of build time.
* build: don't run vet before tests
Recent versions of Go run vet during 'go test' and we have
a dedicated lint job.
* build: use -timeout 5m for tests
Tests sometimes hang on Travis. CI runs are aborted after 10min with no
output. Adding the timeout means we get to see the stack trace for
timeouts.
- Mime types generator (Standard "mime" package rely on system-settings, see mime.osInitMime)
- Changed swarm/api.Upload:
- simplify I/O throttling by semaphore primitive and use file name where possible
- f.Close() must be called in Defer - otherwise panic or future added early return will cause leak of file descriptors
- one error was suppressed
When CLI tests were spanning new nodes, the log level verbosity was
hard coded as 6. So the Swarm process was always polluting the test
output with TRACE level logs.
Now `go test -v ./cmd/swarm -loglevel 0` works as expected.
* swarm/storage/mru: Adaptive Frequency
swarm/storage/mru/lookup: fixed getBaseTime
Added NewEpoch constructor
swarm/api/client: better error handling in GetResource()
swarm/storage/mru: Renamed structures.
Renamed ResourceMetadata to ResourceID.
Renamed ResourceID.Name to ResourceID.Topic
swarm/storage/mru: Added binarySerializer interface and test tools
swarm/storage/mru/lookup: Changed base time to time and + marshallers
swarm/storage/mru: Added ResourceID (former resourceMetadata)
swarm/storage/mru: Added ResourceViewId and serialization tests
swarm/storage/mru/lookup: fixed epoch unmarshaller. Added Epoch Equals
swarm/storage/mru: Fixes as per review comments
cmd/swarm: reworded resource create/update help text regarding topic
swarm/storage/mru: Added UpdateLookup and serializer tests
swarm/storage/mru: Added UpdateHeader, serializers and tests
swarm/storage/mru: changed UpdateAddr / epoch to Base()
swarm/storage/mru: Added resourceUpdate serializer and tests
swarm/storage/mru: Added SignedResourceUpdate tests and serializers
swarm/storage/mru/lookup: fixed GetFirstEpoch bug
swarm/storage/mru: refactor, comments, cleanup
Also added tests for Topic
swarm/storage/mru: handler tests pass
swarm/storage/mru: all resource package tests pass
swarm/storage/mru: resource test pass after adding
timestamp checking support
swarm/storage/mru: Added JSON serializers to ResourceIDView structures
swarm/storage/mru: Sever, client, API test pass
swarm/storage/mru: server test pass
swarm/storage/mru: Added topic length check
swarm/storage/mru: removed some literals,
improved "previous lookup" test case
swarm/storage/mru: some fixes and comments as per review
swarm/storage/mru: first working version without metadata chunk
swarm/storage/mru: Various fixes as per review
swarm/storage/mru: client test pass
swarm/storage/mru: resource query strings and manifest-less queries
swarm/storage/mru: simplify naming
swarm/storage/mru: first autofreq working version
swarm/storage/mru: renamed ToValues to AppendValues
swarm/resource/mru: Added ToValues / FromValues for URL query strings
swarm/storage/mru: Changed POST resource to work with query strings.
No more JSON.
swarm/storage/mru: removed resourceid
swarm/storage/mru: Opened up structures
swarm/storage/mru: Merged Request and SignedResourceUpdate
swarm/storage/mru: removed initial data from CLI resource create
swarm/storage/mru: Refactor Topic as a direct fixed-length array
swarm/storage/mru/lookup: Comprehensive GetNextLevel tests
swarm/storage/mru: Added comments
Added length checks in Topic
swarm/storage/mru: fixes in tests and some code comments
swarm/storage/mru/lookup: new optimized lookup algorithm
swarm/api: moved getResourceView to api out of server
swarm/storage/mru: Lookup algorithm working
swarm/storage/mru: comments and renamed NewLookupParams
Deleted commented code
swarm/storage/mru/lookup: renamed Epoch.LaterThan to After
swarm/storage/mru/lookup: Comments and tidying naming
swarm/storage/mru: fix lookup algorithm
swarm/storage/mru: exposed lookup hint
removed updateheader
swarm/storage/mru/lookup: changed GetNextEpoch for initial values
swarm/storage/mru: resource tests pass
swarm/storage/mru: valueSerializer interface and tests
swarm/storage/mru/lookup: Comments, improvements, fixes, more tests
swarm/storage/mru: renamed UpdateLookup to ID, LookupParams to Query
swarm/storage/mru: renamed query receiver var
swarm/cmd: MRU CLI tests
* cmd/swarm: remove rogue fmt
* swarm/storage/mru: Add version / header for future use
* swarm/storage/mru: Fixes/comments as per review
cmd/swarm: remove rogue fmt
swarm/storage/mru: Add version / header for future use-
* swarm/storage/mru: fix linter errors
* cmd/swarm: Speeded up TestCLIResourceUpdate
* signer: remove local path disclosure from extapi
* signer: show more data in cli ui
* rpc: make http server forward UA and Origin via Context
* signer, clef/core: ui changes + display UA and Origin
* signer: cliui - indicate less trust in remote headers, see https://github.com/ethereum/go-ethereum/issues/17637
* signer: prevent possibility swap KV-entries in aes_gcm storage, fixes#17635
* signer: remove ecrecover from external API
* signer,clef: default reject instead of warn + valideate new passwords. fixes#17632 and #17631
* signer: check calldata length even if no ABI signature is present
* signer: fix failing testcase
* clef: remove account import from external api
* signer: allow space in passwords, improve error messsage
* signer/storage: fix typos
The contributing instructions in the README are not in the GitHub contributing
guide, which means that people coming from the GitHub issues are less likely to
see them.
Package p2p/enode provides a generalized representation of p2p nodes
which can contain arbitrary information in key/value pairs. It is also
the new home for the node database. The "v4" identity scheme is also
moved here from p2p/enr to remove the dependency on Ethereum crypto from
that package.
Record signature handling is changed significantly. The identity scheme
registry is removed and acceptable schemes must be passed to any method
that needs identity. This means records must now be validated explicitly
after decoding.
The enode API is designed to make signature handling easy and safe: most
APIs around the codebase work with enode.Node, which is a wrapper around
a valid record. Going from enr.Record to enode.Node requires a valid
signature.
* p2p/discover: port to p2p/enode
This ports the discovery code to the new node representation in
p2p/enode. The wire protocol is unchanged, this can be considered a
refactoring change. The Kademlia table can now deal with nodes using an
arbitrary identity scheme. This requires a few incompatible API changes:
- Table.Lookup is not available anymore. It used to take a public key
as argument because v4 protocol requires one. Its replacement is
LookupRandom.
- Table.Resolve takes *enode.Node instead of NodeID. This is also for
v4 protocol compatibility because nodes cannot be looked up by ID
alone.
- Types Node and NodeID are gone. Further commits in the series will be
fixes all over the the codebase to deal with those removals.
* p2p: port to p2p/enode and discovery changes
This adapts package p2p to the changes in p2p/discover. All uses of
discover.Node and discover.NodeID are replaced by their equivalents from
p2p/enode.
New API is added to retrieve the enode.Node instance of a peer. The
behavior of Server.Self with discovery disabled is improved. It now
tries much harder to report a working IP address, falling back to
127.0.0.1 if no suitable address can be determined through other means.
These changes were needed for tests of other packages later in the
series.
* p2p/simulations, p2p/testing: port to p2p/enode
No surprises here, mostly replacements of discover.Node, discover.NodeID
with their new equivalents. The 'interesting' API changes are:
- testing.ProtocolSession tracks complete nodes, not just their IDs.
- adapters.NodeConfig has a new method to create a complete node.
These changes were needed to make swarm tests work.
Note that the NodeID change makes the code incompatible with old
simulation snapshots.
* whisper/whisperv5, whisper/whisperv6: port to p2p/enode
This port was easy because whisper uses []byte for node IDs and
URL strings in the API.
* eth: port to p2p/enode
Again, easy to port because eth uses strings for node IDs and doesn't
care about node information in any way.
* les: port to p2p/enode
Apart from replacing discover.NodeID with enode.ID, most changes are in
the server pool code. It now deals with complete nodes instead
of (Pubkey, IP, Port) triples. The database format is unchanged for now,
but we should probably change it to use the node database later.
* node: port to p2p/enode
This change simply replaces discover.Node and discover.NodeID with their
new equivalents.
* swarm/network: port to p2p/enode
Swarm has its own node address representation, BzzAddr, containing both
an overlay address (the hash of a secp256k1 public key) and an underlay
address (enode:// URL).
There are no changes to the BzzAddr format in this commit, but certain
operations such as creating a BzzAddr from a node ID are now impossible
because node IDs aren't public keys anymore.
Most swarm-related changes in the series remove uses of
NewAddrFromNodeID, replacing it with NewAddr which takes a complete node
as argument. ToOverlayAddr is removed because we can just use the node
ID directly.
Interpreter initialization is left to the PRs implementing them.
Options for external interpreters are passed after a colon in the
`--vm.ewasm` and `--vm.evm` switches.
* ethdb: unified code comment style.
* puppeth: it is unnecessary to alloc pre-funded to 256 addresses
* Revert "puppeth: it is unnecessary to alloc pre-funded to 256 addresses"
This reverts commit 5e04fbccf0.
* puppeth: fix comment typo
* Revert "ethdb: unified code comment style."
This reverts commit a581efb3f0.
* swarm/network: simplify kademlia/hive; rid interfaces
* swarm, swarm/network/stream, swarm/netork/simulations,, swarm/pss: adapt to new Kad API
* swarm/network: minor changes re review; add missing lock to NeighbourhoodDepthC
* swarm/storage/encryption: async segmentwise encryption/decryption
* swarm/storage: adapt hasherstore to encryption API change
* swarm/api: adapt RefEncryption for AC to new Encryption API
* swarm/storage/encryption: address review comments
Makes Interface interface a bit more stateless and abstract.
Obviously this change is dictated by EVMC design. The EVMC tries to keep the responsibility for EVM features totally inside the VMs, if feasible. This makes VM "stateless" because VM does not need to pass any information between executions, all information is included in parameters of the execute function.
This commit does a few things at once:
- Updates the tests to contain the latest data from ethereum/tests repo.
- Enables Constantinople state tests. This is needed to be able to
fuzz-test the evm with constantinople rules.
- Fixes the error in opSAR that we've known about for some time. I was
kind of saving it to see if we hit upon it with the random test
generator, but it's difficult to both enable the tests and have the
bug there -- we don't want to forget about it, so maybe it's better
to just fix it.
* miner: commit state which is relative with sealing result
* consensus, core, miner, mobile: introduce sealHash interface
* miner: evict pending task with threshold
* miner: go fmt
* les: fix crasher in NodeInfo when running as server
The ProtocolManager computes CHT and Bloom trie roots by asking the
indexers for their current head. It tried to get the indexers from
LesOdr, but no LesOdr instance is created in server mode.
Attempt to fix this by moving the indexers, protocol creation and
NodeInfo to a new lesCommons struct which is embedded into both server
and client.
All this setup code should really be cleaned up, but this is just a
hotfix so we have to do that some other time.
* les: fix commons protocol maker
This PR enables the indexers to work in light client mode by
downloading a part of these tries (the Merkle proofs of the last
values of the last known section) in order to be able to add new
values and recalculate subsequent hashes. It also adds CHT data to
NodeInfo.
* backends: increase gaslimit in order to allow tests of large contracts
* backends: increase gaslimit in order to allow tests of large contracts
* backends: increase gaslimit in order to allow tests of large contracts
This PR implements les.freeClientPool. It also adds a simulated clock
in common/mclock, which enables time-sensitive tests to run quickly
and still produce accurate results, and package common/prque which is
a generalised variant of prque that enables removing elements other
than the top one from the queue.
les.freeClientPool implements a client database that limits the
connection time of each client and manages accepting/rejecting
incoming connections and even kicking out some connected clients. The
pool calculates recent usage time for each known client (a value that
increases linearly when the client is connected and decreases
exponentially when not connected). Clients with lower recent usage are
preferred, unknown nodes have the highest priority. Already connected
nodes receive a small bias in their favor in order to avoid accepting
and instantly kicking out clients.
Note: the pool can use any string for client identification. Using
signature keys for that purpose would not make sense when being known
has a negative value for the client. Currently the LES protocol
manager uses IP addresses (without port address) to identify clients.
* swarm: Added lightnode flag
Added --lightnode command line parameter
Added LightNode to Handshake message
* swarm/config: Fixed variable naming
* cmd/swarm: Changed BoolTFlag to BoolFlag for SwarmLightNodeEnabled
* swarm/network: Changed logging
* swarm/network: Changed protocol version testing
* swarm/network: Renamed DefaultNetworkID variable to TestProtocolNetworkID
* swarm/network: Bumped protocol version
* swarm/network: Changed LightNode handhsake test to table driven
* swarm/network: Changed back TestProtocolVersion to 5 for now
* swarm/network: Moved the test configuration inside the test function scope
* swarm/network/simulation: increase the sleep duration for TestRun
* cmd/swarm, swarm: fix failing tests on mac
* cmd/swarm: update TestCLISwarmFs skip comment
* swarm/network/simulation: adjust disconnections on simulation close
* swarm/network/simulation: call cleanups after net shutdown
* consensus/ethash: start remote ggoroutine to handle remote mining
* consensus/ethash: expose remote miner api
* consensus/ethash: expose submitHashrate api
* miner, ethash: push empty block to sealer without waiting execution
* consensus, internal: add getHashrate API for ethash
* consensus: add three method for consensus interface
* miner: expose consensus engine running status to miner
* eth, miner: specify etherbase when miner created
* miner: commit new work when consensus engine is started
* consensus, miner: fix some logics
* all: delete useless interfaces
* consensus: polish a bit
- Update benchmarks to use a pool of int pools.
Unless benchmarks are aborted with segmentation fault.
Signed-off-by: Hyung-Kyu Choi <hqueue@users.noreply.github.com>
* rpc: Make HTTP server timeout values configurable
* rpc: Remove flags for setting HTTP Timeouts, configuring via .toml is sufficient.
* rpc: Replace separate constants with a single default struct.
* rpc: Update HTTP Server Read and Write Timeouts to 30s.
* rpc: Remove redundant NewDefaultHTTPTimeouts function.
* rpc: document HTTPTimeouts.
* rpc: sanitize timeout values for library use
* build: add support for different package and binary names
* build: bump up copyright date
* build: change default PackageName to empty string
* build, internal, swarm: enhance build/release process
* build: hack ethereum-swarm as a "depends" in deb package
* build/ci: remove redundant variables
* build, cmd, mobile, params, swarm: remove VERSION file; rename Version to VersionMeta;
* internal: remove VERSION() method which reads VERSION file
* build: fix VersionFilePath to Version
* Makefile: remove clean_go_build_cache.sh until it works
* Makefile: revert removal of clean_go_build_cache.sh
- Define an Interpreter interface
- One contract can call contracts from other interpreter types.
- Pass the interpreter to the operands instead of the evm.
This is meant to prevent type assertions in operands.
Our original wrapper code had two parts. One taken from a third
party repository (who took it from upstream Go) licensed under
BSD-3. The second written by Jeff, Felix and Gustav, licensed
under LGPL. This made this package problematic to use from the
outside.
With the agreement of the original copyright holders, this commit
changes the license of the LGPL portions of the code to BSD-3:
---
I agree changing from LGPL to a BSD style license.
Jeff
---
Hey guys,
My preference would be to relicense to GNUBL, but I'm also OK with BSD.
Cheers,
Gustav
---
Felix Lange (fjl):
I would approve anything that makes our licensing less complicated
---
* swarm/storage/mru: Add embedded publickey and remove ENS dep
This commit breaks swarm, swarm/api...
but tests in swarm/storage/mru pass
* swarm: Refactor swarm, swarm/api to mru changes, make tests pass
* swarm/storage/mru: Remove self from recv, remove test ens vldtr
* swarm/storage/mru: Remove redundant test, expose ResourceHash mthd
* swarm/storage/mru: Make HeaderGetter mandatory + godoc fixes
* swarm/storage: Remove validator prefix for metadata chunk
* swarm/storage/mru: Use Address instead of PublicKey
* swarm/storage/mru: Change index from name to metadata chunk addr
* swarm/storage/mru: Refactor swarm/api/... to MRU index changes
* swarm/storage/mru: Refactor cleanup
* swarm/storage/mru: Rebase cleanup
* swarm: Use constructor for GenericSigner MRU in swarm.go
* swarm/storage: Change to BMTHash for MRU hashing
* swarm/storage: Reduce loglevel on chunk validator logs
* swarm/storage/mru: Delint
* swarm: MRU Rebase cleanup
* swarm/storage/mru: client-side mru signatures
Rebase to PR #668 and fix all conflicts
* swarm/storage/mru: refactor and documentation
* swarm/resource/mru: error-checking tests for parseUpdate/newUpdateChunk
* swarm/storage/mru: Added resourcemetadata tests
* swarm/storage/mru: Added tests for UpdateRequest
* swarm/storage/mru: more test coverage for UpdateRequest and comments
* swarm/storage/mru: Avoid fake chunks in parseUpdate()
* swarm/storage/mru: Documented resource.go extensively
moved some functions where they make most sense
* swarm/storage/mru: increase test coverage for UpdateRequest and
variable name changes throughout to increase consistency
* swarm/storage/mru: moved default timestamp to NewCreateRequest-
* swarm/storage/mru: lookup refactor
* swarm/storage/mru: added comments and renamed raw flag to rawmru
* swarm/storage/mru: fix receiver typo
* swarm/storage/mru: refactored update chunk new/create
* swarm/storage/mru: refactored signature digest to avoid malleability
* swarm/storage/mru: optimize update data serialization
* swarm/storage/mru: refactor and cleanup
* swarm/storage/mru: add timestamp struct and serialization
* swarm/storage/mru: fix lint error and mark some old code for deletion
* swarm/storage/mru: remove unnecessary variable
* swarm/storage/mru: Added more comments throughout
* swarm/storage/mru: Refactored metadata chunk layout + extensive error...
* swarm/storage/mru: refactor cli parser
Changed resource info output to JSON
* swarm/storage/mru: refactor serialization for extensibility
refactored error messages to NewErrorf
* swarm/storage/mru: Moved Signature to resource_sign.
Check Sign errors in server tests
* swarm/storage/mru: Remove isSafeName() checks
* swarm/storage/mru: scrubbed off all references to "block" for time
* swarm/storage/mru: removed superfluous isSynced() call.
* swarm/storage/mru: remove isMultihash() and ToSafeName functions
* swarm/storage/mru: various fixes and comments
* swarm/storage/mru: decoupled cli for independent create/update
* Made resource name optional
* Removed unused LookupPrevious
* swarm/storage/mru: Decoupled resource create / update & refactor
* swarm/storage/mru: Fixed some comments as per issues raised in PR #743
* swarm/storage/mru: Cosmetic changes as per #743 comments
* swarm/storage/mru: refct request encoder/decoder > marshal/unmarshal
* swarm/storage/mru: Cosmetic changes as per review in #748
* swarm/storage/mru: removed timestamp proof placeholder
* swarm/storage/mru: cosmetic/doc/fixes changes as per comments in #704
* swarm/storage/mru: removed unnecessary check in Handler.update
* swarm/storage/mru: Implemented Marshaler/Unmarshaler iface in Request
* swarm/storage/mru: Fixed linter error
* swarm/storage/mru: removed redundant address in signature digest
* swarm/storage/mru: fixed bug: LookupLatestVersionInPeriod not working
* swarm/storage/mru: Unfold Request creation API for create or update+create
set common time source for mru package
* swarm/api/http: fix HandleGetResource error variable shadowed
when requesting a resource that does not exist
* swarm/storage/mru: Add simple check to detect duplicate updates
* swarm/storage/mru: moved Multihash() to the right place.
* cmd/swarm: remove unneeded clientaccountmanager.go
* swarm/storage/mru: Changed some comments as per reviews in #784
* swarm/storage/mru: Made SignedResourceUpdate.GetDigest() public
* swarm/storage/mru: cosmetic changes as per comments in #784
* cmd/swarm: Inverted --multihash flag default
* swarm/storage/mru: removed Verify from SignedResourceUpdate.fromChunk
* swarm/storage/mru: Moved validation code out of serializer
Cosmetic / comment changes
* swarm/storage/mru: Added unit tests for UpdateLookup
* swarm/storage/mru: Increased coverage of metadata serialization
* swarm/storage/mru: Increased test coverage of updateHeader serializers
* swarm/storage/mru: Add resourceUpdate serializer test
* cmd/swarm: minor cli flag text adjustments
* cmd/swarm, swarm/storage, swarm: fix mingw on windows test issues
* cmd/swarm: support for smoke tests on the production swarm cluster
* cmd/swarm/swarm-smoke: simplify cluster logic as per suggestion
* changed colour of landing page
* landing page reacts to enter keypress
* swarm/api/http: sticky footer for swarm landing page using flex
* swarm/api/http: sticky footer for error pages and fix for multiple choices
* swarm: propagate ctx to internal apis (#754)
* swarm/simnet: add basic node/service functions
* swarm/netsim: add buckets for global state and kademlia health check
* swarm/netsim: Use sync.Map as bucket and provide cleanup function for...
* swarm, swarm/netsim: adjust SwarmNetworkTest
* swarm/netsim: fix tests
* swarm: added visualization option to sim net redesign
* swarm/netsim: support multiple services per node
* swarm/netsim: remove redundant return statement
* swarm/netsim: add comments
* swarm: shutdown HTTP in Simulation.Close
* swarm: sim HTTP server timeout
* swarm/netsim: add more simulation methods and peer events examples
* swarm/netsim: add WaitKademlia example
* swarm/netsim: fix comments
* swarm/netsim: terminate peer events goroutines on simulation done
* swarm, swarm/netsim: naming updates
* swarm/netsim: return not healthy kademlias on WaitTillHealthy
* swarm: fix WaitTillHealthy call in testSwarmNetwork
* swarm/netsim: allow bucket to have any type for a key
* swarm: Added snapshots to new netsim
* swarm/netsim: add more tests for bucket
* swarm/netsim: move http related things into separate files
* swarm/netsim: add AddNodeWithService option
* swarm/netsim: add more tests and Start* methods
* swarm/netsim: add peer events and kademlia tests
* swarm/netsim: fix some tests flakiness
* swarm/netsim: improve random nodes selection, fix TestStartStop* tests
* swarm/netsim: remove time measurement from TestClose to avoid flakiness
* swarm/netsim: builder pattern for netsim HTTP server (#773)
* swarm/netsim: add connect related tests
* swarm/netsim: add comment for TestPeerEvents
* swarm: rename netsim package to network/simulation
- AsyncHasher implements AsyncWriter interface
- add extra level for zerohashes in pool to lookup empty data hash
- remove unused segment, hash and depth fields from Tree
- Hash pkg function -> syncHash moved to test
- add asyncHash helper func to tests using shuffle
- add TestAsyncCorrectness to tests
- add BenchmarkBMTAsync to tests
- refactor benchmarks using subbenchmarks
- improved comments
- preinitialise base hashers on the nodes
* cmd/swarm: minor cli flag text adjustments
* swarm/api/http: sticky footer for swarm landing page using flex
* swarm/api/http: sticky footer for error pages and fix for multiple choices
* cmd/swarm, swarm/storage, swarm: fix mingw on windows test issues
* cmd/swarm: update description of swarm cmd
* swarm: added network ID test
* cmd/swarm: support for smoke tests on the production swarm cluster
* cmd/swarm/swarm-smoke: simplify cluster logic as per suggestion
* swarm: propagate ctx to internal apis (#754)
* swarm/metrics: collect disk measurements
* swarm/bmt: fix io.Writer interface
* Write now tolerates arbitrary variable buffers
* added variable buffer tests
* Write loop and finalise optimisation
* refactor / rename
* add tests for empty input
* swarm/pss: (UPDATE) Generic notifications package (#744)
swarm/pss: Generic package for creating pss notification svcs
* swarm: Adding context to more functions
* swarm/api: change colour of landing page in templates
* swarm/api: change landing page to react to enter keypress
* core: fix func TxDifference
fix a typo in func comment;
change named return to unnamed as there's explicit return in the body
* fix another typo in TxDifference
* p2p/discover: move bond logic from table to transport
This commit moves node endpoint verification (bonding) from the table to
the UDP transport implementation. Previously, adding a node to the table
entailed pinging the node if needed. With this change, the ping-back
logic is embedded in the packet handler at a lower level.
It is easy to verify that the basic protocol is unchanged: we still
require a valid pong reply from the node before findnode is accepted.
The node database tracked the time of last ping sent to the node and
time of last valid pong received from the node. Node endpoints are
considered verified when a valid pong is received and the time of last
pong was called 'bond time'. The time of last ping sent was unused. In
this commit, the last ping database entry is repurposed to mean last
ping _received_. This entry is now used to track whether the node needs
to be pinged back.
The other big change is how nodes are added to the table. We used to add
nodes in Table.bond, which ran when a remote node pinged us or when we
encountered the node in a neighbors reply. The transport now adds to the
table directly after the endpoint is verified through ping. To ensure
that the Table can't be filled just by pinging the node repeatedly, we
retain the isInitDone check. During init, only nodes from neighbors
replies are added.
* p2p/discover: reduce findnode failure counter on success
* p2p/discover: remove unused parameter of loadSeedNodes
* p2p/discover: improve ping-back check and comments
* p2p/discover: add neighbors reply nodes always, not just during init
These RPC calls are analogous to Parity's parity_addReservedPeer and
parity_removeReservedPeer.
They are useful for adjusting the trusted peer set during runtime,
without requiring restarting the server.
The current trie memory database/cache that we do pruning on stores
trie nodes as binary rlp encoded blobs, and also stores the node
relationships/references for GC purposes. However, most of the trie
nodes (everything apart from a value node) is in essence just a
collection of references.
This PR switches out the RLP encoded trie blobs with the
collapsed-but-not-serialized trie nodes. This permits most of the
references to be recovered from within the node data structure,
avoiding the need to track them a second time (expensive memory wise).
* debug: Use pprof goroutine writer in debug.Stacks() to ensure all goroutines are captured.
* Up to 64MB limit, previous code only captured first 1MB of goroutines.
* internal/debug: simplify stacks handler
* fix typo
* fix pointer receiver
This package was meant to hold an improved 256 bit integer library, but
the effort was abandoned in 2015. AFAIK nothing ever used this package.
Time to say goodbye.
Prior to this change, when geth was started with `geth -dev -rpc`,
it would report a network id of `1` in response to the `net_version` RPC
request. But the actual network id it used to verify transactions
was `1337`.
This change causes geth instead respond with `1337` to the `net_version`
RPC when geth is started with `geth -dev -rpc`.
* vm/test: add tests+benchmarks for mstore
* core/vm: less alloc and copying for mstore
* core/vm: less allocs in sload
* vm: check for errors more correctly
This commit adds all changes needed for the merge of swarm-network-rewrite.
The changes:
- build: increase linter timeout
- contracts/ens: export ensNode
- log: add Output method and enable fractional seconds in format
- metrics: relax test timeout
- p2p: reduced some log levels, updates to simulation packages
- rpc: increased maxClientSubscriptionBuffer to 20000
Improves test portability by resolving 127.0.0.1:0
to get a random free port instead of the hard coded one. Now
the test works if you have a running node on the same
interface already.
Fixes#15685
This PR fixes a retriever logic bug. When a peer had a soft timeout
and then a response arrived, it always assumed it was the same peer
even though it could have been a later requested one that did not time
out at all yet. In this case the logic went to an illegal state and
deadlocked, causing a goroutine leak.
Fixes#16243 and replaces #16359.
Thanks to @riceke for finding the bug in the logic.
ToECDSAPub was unsafe because it returned a non-nil key with nil X, Y in
case of invalid input. This change replaces ToECDSAPub with
UnmarshalPubkey across the codebase.
The error produced when using a Parity RPC was the following:
ERROR: transaction did not get mined: failed to get tx for txid 0xbdeb094b3278019383c8da148ff1cb5b5dbd61bf8731bc2310ac1b8ed0235226: json: cannot unmarshal non-string into Go struct field txExtraInfo.blockHash of type common.Hash
Allow the --abi flag to be given - to indicate that it should read the
ABI information from standard input. It expects to read the solc output
with the --combined-json flag providing bin, abi, userdoc, devdoc, and
metadata, and works very similarly to the internal invocation of solc,
except it allows external invocation of solc.
This facilitates integration with more complex solc invocations, such
as invocations that require path remapping or --allow-paths tweaks.
Simple usage example:
solc --combined-json bin,abi,userdoc,devdoc,metadata *.sol | abigen --abi -
* metrics: expvar support for ResettingTimer
* metrics: use integers for percentiles; remove Overall
* metrics: fix edge-case panic for index-out-of-range
This removes a golint warning: type name will be used as trie.TrieSync by
other packages, and that stutters; consider calling this Sync.
In hexToKeybytes len(hex) is even and (even+1)/2 == even/2, remove the +1.
* core: use a wrapped `map` and `sync.RWMutex` for `TxPool.all` to remove contention in `TxPool.Get`.
* core: Remove redundant `txLookup.Find` and improve comments on txLookup methods.
This applies spec changes from ethereum/EIPs#1049 and adds support for
pluggable identity schemes.
Some care has been taken to make the "v4" scheme standalone. It uses
public APIs only and could be moved out of package enr at any time.
A couple of minor changes were needed to make identity schemes work:
- The sequence number is now updated in Set instead of when signing.
- Record is now copy-safe, i.e. calling Set on a shallow copy doesn't
modify the record it was copied from.
Feed keeps active subscription channels in a slice called 'f.sendCases'.
The Send method tracks the active cases in a local variable 'cases'
whose value is f.sendCases initially. 'cases' shrinks to a shorter
prefix of f.sendCases every time a send succeeds, moving the successful
case out of range of the active case list.
This can be confusing because the two slices share a backing array. Add
more comments to document what is going on. Also add a test for removing
a case that is in 'f.sentCases' but not 'cases'.
* whisper/mailserver: pass init error to the caller
* whisper/mailserver: add returns to fmt.Errorf
* whisper/mailserver: check err in mailserver init test
This commit is meant to allow ecosystem projects such as ethersphere
to minimize CI build times by specifying an environment variable with
the packages to run tests on.
If the environment variable isn't defined the build script will test
all packages so this shouldn't affect the main go-ethereum repository.
The 'from' and 'to' methods on StateTransitions are reader methods and
shouldn't have inadvertent side effects on state.
It is safe to remove the check in 'from' because account existence is
implicitly checked by the nonce and balance checks. If the account has
non-zero balance or nonce, it must exist. Even if the sender account has
nonce zero at the start of the state transition or no balance, the nonce
is incremented before execution and the account will be created at that
time.
It is safe to remove the check in 'to' because the EVM creates the
account if necessary.
Fixes#15119
* common: delete StringToAddress, StringToHash
These functions are confusing because they don't parse hex, but use the
bytes of the string. This change removes them, replacing all uses of
StringToAddress(s) by BytesToAddress([]byte(s)).
* eth/filters: remove incorrect use of common.BytesToAddress
Most of these methods did not contain all the relevant information
inside the object and were not using a similar formatting type.
Moreover, the existence of a suboptimal String method breaks usage
with more advanced data dumping tools like go-spew.
- Fixes#16271. What was appeneded was a pointer to
an object that changes during the iteration.
- The topic is allocated as a 4-byte array, fill partial topics
with 0s. Partial topics are currently disabled, but would
crash as they rely on the presence of byte number 3.
* crypto/bn256: full switchover to cloudflare's code
* crypto/bn256: only use cloudflare for optimized architectures
* crypto/bn256: upstream fallback for non-optimized code
* .travis, build: drop support for Go 1.8 (need type aliases)
* crypto/bn256/cloudflare: enable curve mul lattice optimization
* core/vm, crypto/bn256: switch over to cloudflare library
* crypto/bn256: unmarshal constraint + start pure go impl
* crypto/bn256: combo cloudflare and google lib
* travis: drop 386 test job
whisper: refactoring go-routines workflow
Move the call mailServer.Init() down (to the bottom of the function) because if the function initialize() completes successfully, then it will be followed by mailServer.Close() in shutdown(). The workflow of the corresponding goroutines is clearer now.
* accounts/abi/bind: support for multi-dim arrays
Also:
- reduce usage of regexes a bit.
- fix minor Java syntax problems
Fixes#15648
* accounts/abi/bind: Add some more documentation
* accounts/abi/bind: Improve code readability
* accounts/abi: bugfix for unpacking nested arrays
The code previously assumed the arrays/slices were always 1 level
deep. While the packing supports nested arrays (!!!).
The current code for unpacking doesn't return the "consumed" length, so
this fix had to work around that by calculating it (i.e. packing and
getting resulting length) after the unpacking of the array element.
It's far from ideal, but unpacking behaviour is fixed now.
* accounts/abi: Fix unpacking of nested arrays
Removed the temporary workaround of packing to calculate size, which was
incorrect for slice-like types anyway.
Full size of nested arrays is used now.
* accounts/abi: deeply nested array unpack test
Test unpacking of an array nested more than one level.
* accounts/abi: Add deeply nested array pack test
Same as the deep nested array unpack test, but the other way around.
* accounts/abi/bind: deeply nested arrays bind test
Test the usage of bindings that were generated
for methods with multi-dimensional (and not
just a single extra dimension, like foo[2][3])
array arguments and returns.
edit: trigger rebuild, CI failed to fetch linter module.
* accounts/abi/bind: improve array binding
wrapArray uses a regex now, and arrayBindingJava is improved.
* accounts/abi: Improve naming of element size func
The full step size for unpacking an array
is now retrieved with "getFullElemSize".
* accounts/abi: support nested nested array args
Previously, the code only considered the outer-size of the array,
ignoring the size of the contents. This was fine for most types,
but nested arrays are packed directly into it, and count towards
the total size. This resulted in arguments following a nested
array to replicate some of the binary contents of the array.
The fix: for arrays, calculate their complete contents size:
count the arg.Type.Elem.Size when Elem is an Array, and
repeat when their child is an array too, etc.
The count is the number of 32 byte elements, similar to how it
previously counted, but nested.
* accounts/abi: Test deep nested arr multi-arguments
Arguments with a deeply nested array should not cause the next arguments
to be read from the wrong position.
The diagnostic tool was saving the unencrypted version of the messages, which is an obvious
security flaw. As of this commit:
* encrypted messages saved instead of plain text.
* all messages are stored, even that created by the user of wnode.
whisper: mailserver no longer supports the signature validation
Mailserver is provided as an example, but client validation belongs to the upper layer protocol and
needs not be covered in this example. The check that was previously available hinders the switch
to libp2p so we agreed not to include that check in that example code anymore.
- added a case error struct that contains information about certain error cases
in which we would like to output more information to the client
- added a validation method that iterates and adds the information that is
stored in the error cases
* swarm: began work on GetHandleFile method re: issue #155
* swarm: now able to serve landing page template
* swarm: added landing page template
* swarm: landing page has working input
* swarm: fixed CSS issue in template
* swarm: deleted extra lines
* swarm: deleted time header and made redirect a relative path
* swarm: removed code mistakenly left
This change removes a peer information from dialing history
when peer is removed from static list. It allows to force a
server to re-dial concrete peer if it is needed.
In our case we are running geth node on mobile devices, and
it is common for a network connection to flap on mobile.
Almost every time it flaps or network connection is changed
from cellular to wifi peers are disconnected with read
timeout. And usually it takes 30 seconds (default expiration
timeout) to recover connection with static peers after
connectivity is restored.
This change allows us to reconnect with peers almost
immediately and it seems harmless enough.
I forgot to change the check in udp.go when I changed Table.bond to be
based on lastPong instead of node presence in db. Rename lastPong to
bondTime and add hasBond so it's clearer what this DB key is used for
now.
* swarm: added script to HTML header to create favicon addresses #153
* swarm: moved data blob direclty into link tag, removed script
* swarm: added favicon info to other html templates
* swarm: fixing test errors
* swarm: fixing favicon test
* swarm: fixing travis tests
* swarm: added script to HTML header to create favicon addresses #153
* swarm: moved data blob direclty into link tag, removed script
* swarm: added favicon info to other html templates
* swarm: fixing test errors
* swarm: fixing favicon test
* swarm: fixing travis tests
This is in preparation for the switch to libp2p: the ID generated
will be from a private key created with the help of libp2p's crypto
library, while Whisper will still use Go's default crypto libraries
for encrypting its messages. This change removes a conflict.
It shouldn't have any impact as the person receiving emails is
the user, not the node.
- according to implementation of `IntrinsicGas`
we can continue execution since problem will be detected
later. However, early return is future-proof for changes.
Talk about "state" instead of "trie timing", "trie memory" and remove
the overzealous warning when the limit is just reached. Since the time
limit is always reached on slow machines, move the message to info level
so users don't freak out about internal details.
* cmd,node,rpc: add allowedHosts to prevent dns rebinding attacks
* p2p,node: Fix bug with dumpconfig introduced in r54aeb8e4c0bb9f0e7a6c67258af67df3b266af3d
* rpc: add wildcard support for rpcallowedhosts + go fmt
* cmd/geth, cmd/utils, node, rpc: ignore direct ip(v4/6) addresses in rpc virtual hostnames check
* http, rpc, utils: make vhosts into map, address review concerns
* node: change log messages to use geth standard (not sprintf)
* rpc: fix spelling
* p2p: add DialRatio for configuration of inbound vs. dialed connections
* p2p: add connection flags to PeerInfo
* p2p/netutil: add SameNet, DistinctNetSet
* p2p/discover: improve revalidation and seeding
This changes node revalidation to be periodic instead of on-demand. This
should prevent issues where dead nodes get stuck in closer buckets
because no other node will ever come along to replace them.
Every 5 seconds (on average), the last node in a random bucket is
checked and moved to the front of the bucket if it is still responding.
If revalidation fails, the last node is replaced by an entry of the
'replacement list' containing recently-seen nodes.
Most close buckets are removed because it's very unlikely we'll ever
encounter a node that would fall into any of those buckets.
Table seeding is also improved: we now require a few minutes of table
membership before considering a node as a potential seed node. This
should make it less likely to store short-lived nodes as potential
seeds.
* p2p/discover: fix nits in UDP transport
We would skip sending neighbors replies if there were fewer than
maxNeighbors results and CheckRelayIP returned an error for the last
one. While here, also resolve a TODO about pong reply tokens.
* les, light: fix CHT trie retrievals
* les, light: minor polishes, test remote CHT retrievals
* les, light: deterministic nodeset rlp, bloombits test skeleton
* les: add an event emission to the les bloombits test
* les: drop dead tester code
The bulk of the issue was to adapt to the new requirement
that a v6 filter has to either contain a symmertric key or
an asymmetric one.
This commits revert one of the fixes that I made to remove
a linter warning: unexporting NewSentMessage. This is not
really a problem as I have a cleanup in the pipe that will
solve this issue.
* Remove --fast flag and clarify default
`--fast` is no longer a flag it's `--syncmode "fast"` and that is the default
* Remove --cache flag
--cache=512 is no longer required as of 1.8 as the default has been increased
* README: Minor cache amount fix, mention Rinkeby
* whisper: fixes warnings from the code linter
* whisper: more non-API-breaking changes
The remaining lint errors are because of auto-generated
files and one is because an exported function has a non-
exported return type. Changing this would break the API,
and will be part of another commit for easier reversal.
* whisper: un-export NewSentMessage to please the linter
This is an API change, which is why it's in its own commit.
This change was initiated after the linter complained that
the returned type wasn't exported. I chose to un-export
the function instead of exporting the type, because that
type is an implementation detail that I would like to
change in the near future to make the code more
readable and with an increased coverage.
* whisper: update gencodec output after upgrading it to new lint standards
* rpc: Support specifying HTTP client in RPC dialing
Adds a minimal interface that captures http.Client and adds a new method
rpc.DialHTTPClient that takes a client using that interface. The existing
rpc.DialHTTP method is then alternatively implemented by using the new
rpc.DialHTTPClient method provided with a standard *http.Client.
* rpc: fix minor doc typos
* dashboard: footer, deep state update
* dashboard: resolve asset path
* dashboard: prevent state update on every reconnection
* dashboard: fix linter issue
* dashboard, cmd: minor UI fix, include commit hash
* dashboard: gitCommit renamed to commit
* dashboard: move the geth version to the right, make commit optional
* dashboard: memory, traffic and CPU on footer
* dashboard: fix merge
* dashboard: CPU, diskIO on footer
* dashboard: rename variables, use group declaration
* dashboard: docs
* consensus/ethash: add maxEpoch constant
* consensus/ethash: improve cache/dataset handling
There are two fixes in this commit:
Unmap the memory through a finalizer like the libethash wrapper did. The
release logic was incorrect and freed the memory while it was being
used, leading to crashes like in #14495 or #14943.
Track caches and datasets using simplelru instead of reinventing LRU
logic. This should make it easier to see whether it's correct.
* consensus/ethash: restore 'future item' logic in lru
* consensus/ethash: use mmap even in test mode
This makes it possible to shorten the time taken for TestCacheFileEvict.
* consensus/ethash: shuffle func calc*Size comments around
* consensus/ethash: ensure future cache/dataset is in the lru cache
* consensus/ethash: add issue link to the new test
* consensus/ethash: fix vet
* consensus/ethash: fix test
* consensus: tiny issue + nitpick fixes
This commit affects p2p/discv5 "topic discovery" by running it on
the same UDP port where the old discovery works. This is realized
by giving an "unhandled" packet channel to the old v4 discovery
packet handler where all invalid packets are sent. These packets
are then processed by v5. v5 packets are always invalid when
interpreted by v4 and vice versa. This is ensured by adding one
to the first byte of the packet hash in v5 packets.
DiscoveryV5Bootnodes is also changed to point to new bootnodes
that are implementing the changed packet format with modified
hash. Existing and new v5 bootnodes are both running on different
ports ATM.
This commit:
- Adds a --msgfile option to read the message to sign from a file
instead of command line argument.
- Adds a unit test for signing subcommands.
- Removes some weird whitespace in the code.
* dashboard: footer, deep state update
* dashboard: resolve asset path
* dashboard: remove bundle.js
* dashboard: prevent state update on every reconnection
* dashboard: fix linter issue
* dashboard, cmd: minor UI fix, include commit hash
* remove geth binary
* dashboard: gitCommit renamed to commit
* dashboard: move the geth version to the right, make commit optional
* dashboard: commit limited to 7 characters
* dashboard: limit commit length on client side
* dashboard: run go generate
* common/fdlimit: Move fdlimit files to separate package
When go-ethereum is used as a library the calling program need to set
the FD limit.
This commit extract fdlimit files to a separate package so it can be
used outside of go-ethereum.
* common/fdlimit: Remove FdLimit from functions signature
* common/fdlimit: Rename fdlimit functions
The first should address a long term issue where we recommend a gas
price that is greater than that required for 50% of transactions in
recent blocks, which can lead to gas price inflation as people take
this figure and add a margin to it, resulting in a positive feedback
loop.
* core/types, core/vm, eth, tests: regenerate gencodec files
* Makefile: update devtools target
Install protoc-gen-go and print reminders about npm, solc and protoc.
Also switch to github.com/kevinburke/go-bindata because it's more
maintained.
* contracts/ens: update contracts and regenerate with solidity v0.4.19
The newer upstream version of the FIFSRegistrar contract doesn't set the
resolver anymore. The resolver is now deployed separately.
* contracts/release: regenerate with solidity v0.4.19
* contracts/chequebook: fix fallback and regenerate with solidity v0.4.19
The contract didn't have a fallback function, payments would be rejected
when compiled with newer solidity. References to 'mortal' and 'owned'
use the local file system so we can compile without network access.
* p2p/discv5: regenerate with recent stringer
* cmd/faucet: regenerate
* dashboard: regenerate
* eth/tracers: regenerate
* internal/jsre/deps: regenerate
* dashboard: avoid sed -i because it's not portable
* accounts/usbwallet/internal/trezor: fix go generate warnings
* go-ethereum/dashboard: update assets.go
Use current rsc/go-bindata instead of jteeuwen/go-bindata, to get
balanced tree in very long string concatenations.
This works around problems in current Go distributions.
For golang/go#23222.
* dashboard: run last two go:generate steps for linter
+ The event slice unpacker doesn't correctly extract element from the
slice. The indexed arguments are not ignored as they should be
(the data offset should not include the indexed arguments).
+ The `Elem()` call in the slice unpack doesn't work.
The Slice related tests fails because of that.
+ the check in the loop are suboptimal and have been extracted
out of the loop.
+ extracted common code from event and method tupleUnpack
* cmd/utils: Add check on hard limit, skip test if below target
* cmd/utils: Cross platform compatible fd limit test
* cmd/utils: Remove syscall.Rlimit in test
* cmd/utils: comment fd utility method
* swarm/api: url scheme bzz-hash to get hashes of swarm content (#15238)
Update URI to support bzz-hash scheme and handle such HTTP requests by
responding with hash of the content as a text/plain response.
* swarm/api: return hash of the content for bzz-hash:// requests
* swarm/api: revert "return hash of the content for bzz-hash:// requests"
Return hashes of the content that would be returned by bzz-raw
request.
* swarm/api/http: handle error in TestBzzGetPath
* swarm/api: remove extra blank line in comment
ethkey is a new tool that serves as a command line interface to
the basic key management functionalities of geth. It currently
supports:
- generating keyfiles
- inspecting keyfiles (print public and private key)
- signing messages
- verifying signed messages
This change inlines the logic of bytesAreProper at its sole
callsite, ABI.Unpack, and applies the multiple-of-32 test only in
the case of unpacking methods. Event data is not required to be a
multiple of 32 bytes long.
Changed the communication protocol for ordinary message,
according to EIP 627. Messages will be send in bundles, i.e.
array of messages will be sent instead of single message.
* crypto: ensure that VerifySignature rejects malleable signatures
It already rejected them when using libsecp256k1, make sure the nocgo
version does the same thing.
* crypto: simplify check
* crypto: fix build
* swarm/api: url scheme bzz-list for getting list of files from manifest
Replace query parameter list=true for listing all files contained
in a swarm manifest with a new URL scheme bzz-list.
* swarm: replaace bzzr and bzzi schemes with bzz-raw and bzz-immutable
New URI Shemes are added and old ones are deprecated, but not removed.
Old Schemes bzzr and bzzi are functional for backward compatibility.
* swarm/api: completely remove bzzr and bzzi schemes
Remove old schemes in favour of bzz-raw and
bzz-immutable.
* swarm/api: revert "completely remove bzzr and bzzi schemes"
Keep bzzr and bzzi schemes for backward compatibility. At least
until 0.3 swarm release.
Specifying ENS API CLI flag, env variable or configuration
field is required for ENS resolving. Backward compatibility is
preserved with --ens-api="" CLI flag value.
Merge with changes that implement config file PR #15548.
Field *EnsApi string* in swarm/api.Config is replaced with
*EnsAPIs []string*.
A new field *EnsDisabled bool* is added to swarm/api.Config for
easy way to disable ENS resolving with config file.
Signature of function swarm.NewSwarm is changed and simplified.
This commit adds a TOML configuration option to swarm. It reuses
the TOML configuration structure used in geth with swarm
customized items.
The commit:
* Adds a "dumpconfig" command to the swarm executable which
allows printing the (default) configuration to stdout, which
then can be redirected to a file in order to customize it.
* Adds a "--config <file>" option to the swarm executable which will
allow to load a configuration file in TOML format from the
specified location in order to initialize the Swarm node The
override priorities are like follows: environment variables
override command line arguments override config file override
default config.
With this change,
key, err := crypto.HexToECDSA("000000...")
returns nil key and an error instead of a non-nil key with nil X
and Y inside. Issue found by @guidovranken.
Now that the AES salt has been moved to the payload, padding must
be adjusted to hide it, lest an attacker guesses that the packet
uses symmetric encryption.
We need those operations for p2p/enr.
Also upgrade github.com/btcsuite/btcd/btcec to the latest version
and improve BenchmarkSha3. The benchmark printed extra output
that confused tools like benchstat and ignored N.
Allow multiple --ens-api flags to be specified with value format
[tld:][contract-addr@]url.
Backward compatibility with only one --ens-api flag and --ens-addr
flag is preserved and conflict cases are handled:
- multiple --ens-api with --ens-addr returns an error
- single --ens-api with contract address and --ens-addr with
different contract address returns an error
Previously implemented --ens-endpoint is removed. Its functionality
is replaced with multiple --ens-api flags.
Generator in the current lib uses -2 as the y point when doing
ScalarBaseMult, this makes it so that points/signatures generated
from libs like py_ecc don't match/validate as pretty much all
other libs (including libsnark) have (1, 2) as the standard
generator.
This does not affect consensus as the generator is never used in
the VM, points are always explicitly defined and there is not
ScalarBaseMult op - it only makes it so that doing "import
github.com/ethereum/go-ethereum/crypto/bn256" doesn't generate
bad points in userland tools.
Updated use of Parallel and added some subtests to help isolate
them. Increased timeout in RequestHeadersByNumber so it
doesn't time out and causes other tests to break.
p2p/simulations: introduce dialBan
- Refactor simulations/network connection getters to support
avoiding simultaneous dials between two peers If two peers dial
simultaneously, the connection will be dropped to help avoid
that, we essentially lock the connection object with a
timestamp which serves as a ban on dialing for a period of time
(dialBanTimeout).
- The connection getter InitConn can be wrapped and passed to the
nodes via adapters.NodeConfig#Reachable field and then used by
the respective services when they initiate connections. This
massively stablise the emerging connectivity when running with
hundreds of nodes bootstrapping a network.
p2p: add Inbound public method to p2p.Peer
p2p/simulations: Add server id to logs to support debugging
in-memory network simulations when multiple peers are logging.
p2p: SetupConn now returns error. The dialer checks the error and
only calls resolve if the actual TCP dial fails.
* core/vm: track 63/64 call gas off stack
Gas calculations in gasCall* relayed the available gas for calls by
replacing it on the stack. This lead to inconsistent traces, which we
papered over by copying the pre-execution stack in trace mode.
This change relays available gas using a temporary variable, off the
stack, and allows removing the weird copy.
* core/vm: remove stackCopy
* core/vm: pop call gas into pool
* core/vm: to -> addr
* trie: make fullnode children hash calculation concurrently
* trie: thread out only on topmost fullnode
* trie: clean up full node children hash calculation
* trie: minor code fixups
* cmd, consensus, eth: split ethash related config to it own
* eth, consensus: minor polish
* eth, consenus, console: compress pow testing config field to single one
* consensus, eth: document pow mode
* eth, internal: Implement using trie diffs
* eth, internal: Changes in response to review
* eth: More fixes to getModifiedAccountsBy*
* eth: minor polishes on error capitalization
* .dockerignore, internal/build: Read git information directly from file
This commit changes the way of retrieving git commit and branch for build
environment from running git command to reading git files directly.
This commit also adds required git files into Docker build context.
fixes: #15346
* .dockerignore: workaround for including some files in .git
github.com/stretchr/testify is a useful library for doing
assertion in tests. It makes assertions in test more less verbose and
more comfortable to read and use.
This changes behaviour: before if the owner/amount didn't match, it
resulted in a successful execution without doing anything; now it
results in a failed execution using the revert opcode (remainder gas
is not consumed).
* core: allow price bump at threshold
* core: test changes to allow price bump at threshold
* core: reinstate tx replacement test underneath threshold
* core: minor test failure message cleanups
This PR implements the new LES protocol version extensions:
* new and more efficient Merkle proofs reply format (when replying to
a multiple Merkle proofs request, we just send a single set of trie
nodes containing all necessary nodes)
* BBT (BloomBitsTrie) works similarly to the existing CHT and contains
the bloombits search data to speed up log searches
* GetTxStatusMsg returns the inclusion position or the
pending/queued/unknown state of a transaction referenced by hash
* an optional signature of new block data (number/hash/td) can be
included in AnnounceMsg to provide an option for "very light
clients" (mobile/embedded devices) to skip expensive Ethash check
and accept multiple signatures of somewhat trusted servers (still a
lot better than trusting a single server completely and retrieving
everything through RPC). The new client mode is not implemented in
this PR, just the protocol extension.
* cmd, consensus, core, miner: instatx clique for --dev
* cmd, consensus, clique: support configurable --dev block times
* cmd, core: allow --dev to use persistent storage too
If the file already existed, the WriteResponse.Size was being set
as the length of the entire file, not just the amount that was
written to the existing file.
Fixes#15216
The accountCache contains a file cache, and remembers from
scan to scan what files were present earlier. Thus, whenever
there's a change, the scan phase only bothers processing new
and removed files.
Building geth requires both a Go (version 1.7 or later) and a C compiler.
Building geth requires both a Go (version 1.9 or later) and a C compiler.
You can install them using your favourite package manager.
Once the dependencies are installed, run
@ -32,13 +34,13 @@ The go-ethereum project comes with several wrappers/executables found in the `cm
| Command | Description |
|:----------:|-------------|
| **`geth`** | Our main Ethereum CLI client. It is the entry point into the Ethereum network (main-, test- or private net), capable of running as a full node (default) archive node (retaining all historical state) or a light node (retrieving data live). It can be used by other processes as a gateway into the Ethereum network via JSON RPC endpoints exposed on top of HTTP, WebSocket and/or IPC transports. `geth --help` and the [CLI Wiki page](https://github.com/ethereum/go-ethereum/wiki/Command-Line-Options) for command line options. |
| **`geth`** | Our main Ethereum CLI client. It is the entry point into the Ethereum network (main-, test- or private net), capable of running as a full node (default), archive node (retaining all historical state) or a light node (retrieving data live). It can be used by other processes as a gateway into the Ethereum network via JSON RPC endpoints exposed on top of HTTP, WebSocket and/or IPC transports. `geth --help` and the [CLI Wiki page](https://github.com/ethereum/go-ethereum/wiki/Command-Line-Options) for command line options. |
| `abigen` | Source code generator to convert Ethereum contract definitions into easy to use, compile-time type-safe Go packages. It operates on plain [Ethereum contract ABIs](https://github.com/ethereum/wiki/wiki/Ethereum-Contract-ABI) with expanded functionality if the contract bytecode is also available. However it also accepts Solidity source files, making development much more streamlined. Please see our [Native DApps](https://github.com/ethereum/go-ethereum/wiki/Native-DApps:-Go-bindings-to-Ethereum-contracts) wiki page for details. |
| `bootnode` | Stripped down version of our Ethereum client implementation that only takes part in the network node discovery protocol, but does not run any of the higher level application protocols. It can be used as a lightweight bootstrap node to aid in finding peers in private networks. |
| `evm` | Developer utility version of the EVM (Ethereum Virtual Machine) that is capable of running bytecode snippets within a configurable environment and execution mode. Its purpose is to allow insolated, fine-grained debugging of EVM opcodes (e.g. `evm --code 60ff60ff --debug`). |
| `evm` | Developer utility version of the EVM (Ethereum Virtual Machine) that is capable of running bytecode snippets within a configurable environment and execution mode. Its purpose is to allow isolated, fine-grained debugging of EVM opcodes (e.g. `evm --code 60ff60ff --debug`). |
| `gethrpctest` | Developer utility tool to support our [ethereum/rpc-test](https://github.com/ethereum/rpc-tests) test suite which validates baseline conformity to the [Ethereum JSON RPC](https://github.com/ethereum/wiki/wiki/JSON-RPC) specs. Please see the [test suite's readme](https://github.com/ethereum/rpc-tests/blob/master/README.md) for details. |
| `rlpdump` | Developer utility tool to convert binary RLP ([Recursive Length Prefix](https://github.com/ethereum/wiki/wiki/RLP)) dumps (data encoding used by the Ethereum protocol both network as well as consensus wise) to user friendlier hierarchical representation (e.g. `rlpdump --hex CE0183FFFFFFC4C304050583616263`). |
| `swarm` | swarm daemon and tools. This is the entrypoint for the swarm network. `swarm --help` for command line options and subcommands. See https://swarm-guide.readthedocs.io for swarm documentation. |
| `swarm` | Swarm daemon and tools. This is the entrypoint for the Swarm network. `swarm --help` for command line options and subcommands. See [Swarm README](https://github.com/ethereum/go-ethereum/tree/master/swarm) for more information. |
| `puppeth` | a CLI wizard that aids in creating a new Ethereum network. |
## Running geth
@ -56,20 +58,18 @@ the user doesn't care about years-old historical data, so we can fast-sync quick
state of the network. To do so:
```
$ geth --fast --cache=512 console
$ geth console
```
This command will:
* Start geth in fast sync mode (`--fast`), causing it to download more data in exchange for avoiding
processing the entire history of the Ethereum network, which is very CPU intensive.
* Bump the memory allowance of the database to 512MB (`--cache=512`), which can help significantly in
sync times especially for HDD users. This flag is optional and you can set it as high or as low as
you'd like, though we'd recommend the 512MB - 2GB range.
* Start geth in fast sync mode (default, can be changed with the `--syncmode` flag), causing it to
download more data in exchange for avoiding processing the entire history of the Ethereum network,
which is very CPU intensive.
* Start up Geth's built-in interactive [JavaScript console](https://github.com/ethereum/go-ethereum/wiki/JavaScript-Console),
(via the trailing `console` subcommand) through which you can invoke all official [`web3` methods](https://github.com/ethereum/wiki/wiki/JavaScript-API)
as well as Geth's own [management APIs](https://github.com/ethereum/go-ethereum/wiki/Management-APIs).
This too is optional and if you leave it out you can always attach to an already running Geth instance
This tool is optional and if you leave it out you can always attach to an already running Geth instance
with `geth attach`.
### Full node on the Ethereum test network
@ -80,12 +80,11 @@ entire system. In other words, instead of attaching to the main network, you wan
network with your node, which is fully equivalent to the main network, but with play-Ether only.
```
$ geth --testnet --fast --cache=512 console
$ geth --testnet console
```
The `--fast`, `--cache` flags and `console` subcommand have the exact same meaning as above and they
are equally useful on the testnet too. Please see above for their explanations if you've skipped to
here.
The `console` subcommand have the exact same meaning as above and they are equally useful on the
testnet too. Please see above for their explanations if you've skipped to here.
Specifying the `--testnet` flag however will reconfigure your Geth instance a bit:
@ -102,6 +101,14 @@ over between the main network and test network, you should make sure to always u
for play-money and real-money. Unless you manually move accounts, Geth will by default correctly
separate the two networks and will not make any accounts available between them.*
### Full node on the Rinkeby test network
The above test network is a cross client one based on the ethash proof-of-work consensus algorithm. As such, it has certain extra overhead and is more susceptible to reorganization attacks due to the network's low difficulty / security. Go Ethereum also supports connecting to a proof-of-authority based test network called [*Rinkeby*](https://www.rinkeby.io) (operated by members of the community). This network is lighter, more secure, but is only supported by go-ethereum.
```
$ geth --rinkeby console
```
### Configuration
As an alternative to passing the numerous flags to the `geth` binary, you can also pass a configuration file via:
@ -125,15 +132,17 @@ One of the quickest ways to get Ethereum up and running on your machine is by us
```
docker run -d --name ethereum-node -v /Users/alice/ethereum:/root \
-p 8545:8545 -p 30303:30303 \
ethereum/client-go --fast --cache=512
ethereum/client-go
```
This will start geth in fastsync mode with a DB memory allowance of 512MB just as the above command does. It will also create a persistent volume in your home directory for saving your blockchain as well as map the default ports. There is also an `alpine` tag available for a slim version of the image.
This will start geth in fast-sync mode with a DB memory allowance of 1GB just as the above command does. It will also create a persistent volume in your home directory for saving your blockchain as well as map the default ports. There is also an `alpine` tag available for a slim version of the image.
Do not forget `--rpcaddr 0.0.0.0`, if you want to access RPC from other containers and/or hosts. By default, `geth` binds to the local interface and RPC endpoints is not accessible from the outside.
### Programatically interfacing Geth nodes
As a developer, sooner rather than later you'll want to start interacting with Geth and the Ethereum
network via your own programs and not manually through the console. To aid this, Geth has builtin
network via your own programs and not manually through the console. To aid this, Geth has built-in
support for a JSON-RPC based APIs ([standard APIs](https://github.com/ethereum/wiki/wiki/JSON-RPC) and
[Geth specific APIs](https://github.com/ethereum/go-ethereum/wiki/Management-APIs)). These can be
exposed via HTTP, WebSockets and IPC (unix sockets on unix based platforms, and named pipes on Windows).
@ -159,7 +168,7 @@ HTTP based JSON-RPC API options:
*`--ipcpath` Filename for IPC socket/pipe within the datadir (explicit paths escape it)
You'll need to use your own programming environments' capabilities (libraries, tools, etc) to connect
via HTTP, WS or IPC to a Geth node configured with the above flags and you'll need to speak [JSON-RPC](http://www.jsonrpc.org/specification)
via HTTP, WS or IPC to a Geth node configured with the above flags and you'll need to speak [JSON-RPC](https://www.jsonrpc.org/specification)
on all transports. You can reuse the same connection for multiple requests!
**Note: Please understand the security implications of opening up an HTTP/WS based transport before
@ -264,7 +273,7 @@ instance for mining, run it with all your usual flags, extended by:
// {{$contract.Type}}{{.Normalized.Name}}Iterator is returned from Filter{{.Normalized.Name}} and is used to iterate over the raw logs and unpacked data for {{.Normalized.Name}} events raised by the {{$contract.Type}} contract.
type {{$contract.Type}}{{.Normalized.Name}}Iterator struct {
Event *{{$contract.Type}}{{.Normalized.Name}} // Event containing the contract specifics and raw log
contract *bind.BoundContract // Generic contract to use for unpacking event data
event string // Event name to use for unpacking event data
logs chan types.Log // Log channel receiving the found contract events
sub ethereum.Subscription // Subscription for errors, completion and termination
done bool // Whether the subscription completed delivering logs
fail error // Occurred error to stop iteration
}
// Next advances the iterator to the subsequent event, returning whether there
// are any more events found. In case of a retrieval or parsing error, false is
// returned and Error() can be queried for the exact failure.
debsrc [ -signer key-id ] [ -upload dest ] -- creates a debian source package
nsis -- creates a Windows NSIS installer
@ -58,6 +59,8 @@ import (
"time"
"github.com/ethereum/go-ethereum/internal/build"
"github.com/ethereum/go-ethereum/params"
sv"github.com/ethereum/go-ethereum/swarm/version"
)
var(
@ -76,51 +79,84 @@ var (
executablePath("geth"),
executablePath("puppeth"),
executablePath("rlpdump"),
executablePath("swarm"),
executablePath("wnode"),
}
// Files that end up in the swarm*.zip archive.
swarmArchiveFiles=[]string{
"COPYING",
executablePath("swarm"),
}
// A debian package is created for all executables listed here.
debExecutables=[]debExecutable{
{
Name:"abigen",
BinaryName:"abigen",
Description:"Source code generator to convert Ethereum contract definitions into easy to use, compile-time type-safe Go packages.",
},
{
Name:"bootnode",
BinaryName:"bootnode",
Description:"Ethereum bootnode.",
},
{
Name:"evm",
BinaryName:"evm",
Description:"Developer utility version of the EVM (Ethereum Virtual Machine) that is capable of running bytecode snippets within a configurable environment and execution mode.",
},
{
Name:"geth",
BinaryName:"geth",
Description:"Ethereum CLI client.",
},
{
Name:"puppeth",
BinaryName:"puppeth",
Description:"Ethereum private network manager.",
},
{
Name:"rlpdump",
BinaryName:"rlpdump",
Description:"Developer utility tool that prints RLP structures.",
},
{
Name:"swarm",
Description:"Ethereum Swarm daemon and tools",
},
{
Name:"wnode",
BinaryName:"wnode",
Description:"Ethereum Whisper diagnostic tool",
},
}
// A debian package is created for all executables listed here.
debSwarmExecutables=[]debExecutable{
{
BinaryName:"swarm",
PackageName:"ethereum-swarm",
Description:"Ethereum Swarm daemon and tools",
},
}
debEthereum=debPackage{
Name:"ethereum",
Version:params.Version,
Executables:debExecutables,
}
debSwarm=debPackage{
Name:"ethereum-swarm",
Version:sv.Version,
Executables:debSwarmExecutables,
}
// Debian meta packages to build and push to Ubuntu PPA
debPackages=[]debPackage{
debSwarm,
debEthereum,
}
// Packages to be cross-compiled by the xgo command
--lightkdf Reduce key-derivation RAM & CPU usage at some expense of KDF strength
--nousb Disables monitoring for and managing USB hardware wallets
--rpcaddr value HTTP-RPC server listening interface (default: "localhost")
--rpcport value HTTP-RPC server listening port (default: 8550)
--signersecret value A file containing the password used to encrypt signer credentials, e.g. keystore credentials and ruleset hash
--4bytedb value File containing 4byte-identifiers (default: "./4byte.json")
--4bytedb-custom value File used for writing new 4byte-identifiers submitted via API (default: "./4byte-custom.json")
--auditlog value File used to emit audit logs. Set to "" to disable (default: "audit.log")
--rules value Enable rule-engine (default: "rules.json")
--stdio-ui Use STDIN/STDOUT as a channel for an external UI. This means that an STDIN/STDOUT is used for RPC-communication with a e.g. a graphical user interface, and can be used when the signer is started by an external process.
--stdio-ui-test Mechanism to test interface between signer and UI. Requires 'stdio-ui'.
--help, -h show help
--version, -v print the version
```
Example:
```
signer -keystore /my/keystore -chainid 4
```
## Security model
The security model of the signer is as follows:
* One critical component (the signer binary / daemon) is responsible for handling cryptographic operations: signing, private keys, encryption/decryption of keystore files.
* The signer binary has a well-defined 'external' API.
* The 'external' API is considered UNTRUSTED.
* The signer binary also communicates with whatever process that invoked the binary, via stdin/stdout.
* This channel is considered 'trusted'. Over this channel, approvals and passwords are communicated.
The general flow for signing a transaction using e.g. geth is as follows:
In this case, `geth` would be started with `--externalsigner=http://localhost:8550` and would relay requests to `eth.sendTransaction`.
## TODOs
Some snags and todos
* [ ] The signer should take a startup param "--no-change", for UIs that do not contain the capability
to perform changes to things, only approve/deny. Such a UI should be able to start the signer in
a more secure mode by telling it that it only wants approve/deny capabilities.
* [x] It would be nice if the signer could collect new 4byte-id:s/method selectors, and have a
secondary database for those (`4byte_custom.json`). Users could then (optionally) submit their collections for
inclusion upstream.
* It should be possible to configure the signer to check if an account is indeed known to it, before
passing on to the UI. The reason it currently does not, is that it would make it possible to enumerate
accounts if it immediately returned "unknown account".
* [x] It should be possible to configure the signer to auto-allow listing (certain) accounts, instead of asking every time.
* [x] Done Upon startup, the signer should spit out some info to the caller (particularly important when executed in `stdio-ui`-mode),
invoking methods with the following info:
* [x] Version info about the signer
* [x] Address of API (http/ipc)
* [ ] List of known accounts
* [ ] Have a default timeout on signing operations, so that if the user has not answered within e.g. 60 seconds, the request is rejected.
* [ ]`account_signRawTransaction`
* [ ]`account_bulkSignTransactions([] transactions)` should
* only exist if enabled via config/flag
* only allow non-data-sending transactions
* all txs must use the same `from`-account
* let the user confirm, showing
* the total amount
* the number of unique recipients
* Geth todos
- The signer should pass the `Origin` header as call-info to the UI. As of right now, the way that info about the request is
put together is a bit of a hack into the http server. This could probably be greatly improved
- Relay: Geth should be started in `geth --external_signer localhost:8550`.
- Currently, the Geth APIs use `common.Address` in the arguments to transaction submission (e.g `to` field). This
type is 20 `bytes`, and is incapable of carrying checksum information. The signer uses `common.MixedcaseAddress`, which
retains the original input.
- The Geth api should switch to use the same type, and relay `to`-account verbatim to the external api.
* [x] Storage
* [x] An encrypted key-value storage should be implemented
* See [rules.md](rules.md) for more info about this.
* Another potential thing to introduce is pairing.
* To prevent spurious requests which users just accept, implement a way to "pair" the caller with the signer (external API).
* Thus geth/mist/cpp would cryptographically handshake and afterwards the caller would be allowed to make signing requests.
* This feature would make the addition of rules less dangerous.
* Wallets / accounts. Add API methods for wallets.
## Communication
### External API
The signer listens to HTTP requests on `rpcaddr`:`rpcport`, with the same JSONRPC standard as Geth. The messages are
expected to be JSON [jsonrpc 2.0 standard](http://www.jsonrpc.org/specification).
Some of these call can require user interaction. Clients must be aware that responses
may be delayed significantly or may never be received if a users decides to ignore the confirmation request.
The External API is **untrusted** : it does not accept credentials over this api, nor does it expect
that requests have any authority.
### UI API
The signer has one native console-based UI, for operation without any standalone tools.
However, there is also an API to communicate with an external UI. To enable that UI,
the signer needs to be executed with the `--stdio-ui` option, which allocates the
`stdin`/`stdout` for the UI-api.
An example (insecure) proof-of-concept of has been implemented in `pythonsigner.py`.
The model is as follows:
* The user starts the UI app (`pythonsigner.py`).
* The UI app starts the `signer` with `--stdio-ui`, and listens to the
process output for confirmation-requests.
* The `signer` opens the external http api.
* When the `signer` receives requests, it sends a `jsonrpc` request via `stdout`.
* The UI app prompts the user accordingly, and responds to the `signer`
* The `signer` signs (or not), and responds to the original request.
## External API
See the [external api changelog](extapi_changelog.md) for information about changes to this API.
### Encoding
- number: positive integers that are hex encoded
- data: hex encoded data
- string: ASCII string
All hex encoded values must be prefixed with `0x`.
## Methods
### account_new
#### Create new password protected account
The signer will generate a new private key, encrypts it according to [web3 keystore spec](https://github.com/ethereum/wiki/wiki/Web3-Secret-Storage-Definition) and stores it in the keystore directory.
The client is responsible for creating a backup of the keystore. If the keystore is lost there is no method of retrieving lost accounts.
#### Arguments
None
#### Result
- address [string]: account address that is derived from the generated key
Signs a transactions and responds with the signed transaction in RLP encoded form.
#### Arguments
2. transaction object:
-`from` [address]: account to send the transaction from
-`to` [address]: receiver account. If omitted or `0x`, will cause contract creation.
-`gas` [number]: maximum amount of gas to burn
-`gasPrice` [number]: gas price
-`value` [number:optional]: amount of Wei to send with the transaction
-`data` [data:optional]: input data
-`nonce` [number]: account nonce
3. method signature [string:optional]
- The method signature, if present, is to aid decoding the calldata. Should consist of `methodname(paramtype,...)`, e.g. `transfer(uint256,address)`. The signer may use this data to parse the supplied calldata, and show the user. The data, however, is considered totally untrusted, and reliability is not expected.
"message":"Transaction data did not match ABI-interface: WARNING: Supplied data is stuffed with extra data. \nWant 0000000000000002000000000000000000000000000000000000000000000012\nHave 0000000000000000000000000000000000000000000000000000000000000012\nfor method safeSend(address)"
}
],
"meta":{
"remote":"127.0.0.1:48492",
"local":"localhost:8550",
"scheme":"HTTP/1.1"
}
}
]
}
```
One which has missing `to`, but with no `data`:
```json
{
"jsonrpc":"2.0",
"id":3,
"method":"ApproveTx",
"params":[
{
"transaction":{
"from":"",
"to":null,
"gas":"0x0",
"gasPrice":"0x0",
"value":"0x0",
"nonce":"0x0",
"data":null,
"input":null
},
"call_info":[
{
"type":"CRITICAL",
"message":"Tx will create contract with empty code!"
}
],
"meta":{
"remote":"signer binary",
"local":"main",
"scheme":"in-proc"
}
}
]
}
```
### ApproveExport
Invoked when a request to export an account has been made.
The UI should show the info to the user. Does not expect response.
#### Sample call
```json
{
"jsonrpc":"2.0",
"id":9,
"method":"ShowInfo",
"params":[
{
"text":"Tests completed"
}
]
}
```
### ShowError
The UI should show the info to the user. Does not expect response.
```json
{
"jsonrpc":"2.0",
"id":2,
"method":"ShowError",
"params":[
{
"text":"Testing 'ShowError'"
}
]
}
```
### OnApproved
`OnApprovedTx` is called when a transaction has been approved and signed. The call contains the return value that will be sent to the external caller. The return value from this method is ignored - the reason for having this callback is to allow the ruleset to keep track of approved transactions.
When implementing rate-limited rules, this callback should be used.
TLDR; Use this method to keep track of signed transactions, instead of using the data in `ApproveTx`.
### OnSignerStartup
This method provide the UI with information about what API version the signer uses (both internal and external) aswell as build-info and external api,
in k/v-form.
Example call:
```json
{
"jsonrpc":"2.0",
"id":1,
"method":"OnSignerStartup",
"params":[
{
"info":{
"extapi_http":"http://localhost:8550",
"extapi_ipc":null,
"extapi_version":"2.0.0",
"intapi_version":"1.2.0"
}
}
]
}
```
### Rules for UI apis
A UI should conform to the following rules.
* A UI MUST NOT load any external resources that were not embedded/part of the UI package.
* For example, not load icons, stylesheets from the internet
* Not load files from the filesystem, unless they reside in the same local directory (e.g. config files)
* A Graphical UI MUST show the blocky-identicon for ethereum addresses.
* A UI MUST warn display approproate warning if the destination-account is formatted with invalid checksum.
* A UI MUST NOT open any ports or services
* The signer opens the public port
* A UI SHOULD verify the permissions on the signer binary, and refuse to execute or warn if permissions allow non-user write.
* A UI SHOULD inform the user about the `SHA256` or `MD5` hash of the binary being executed
* A UI SHOULD NOT maintain a secondary storage of data, e.g. list of accounts
* The signer provides accounts
* A UI SHOULD, to the best extent possible, use static linking / bundling, so that required libraries are bundled
along with the UI.
### UI Implementations
There are a couple of implementation for a UI. We'll try to keep this list up to date.
| Name | Repo | UI type| No external resources| Blocky support| Verifies permissions | Hash information | No secondary storage | Statically linked| Can modify parameters|
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
