This is a simple solution to having the proxy keep up to date with the
state of the cluster. Basically, it uses the cluster configuration
provided at start up (i.e. with `-initial-cluster-state`) to determine
where to reach peer(s) in the cluster, and then it will periodically hit
the `/members` endpoint of those peer(s) (using the same mechanism that
`-cluster-state=existing` does to initialise) to update the set of valid
client URLs to proxy to.
This does not address discovery (#1376), and it would probably be better
to update the set of proxyURLs dynamically whenever we fetch the new
state of the cluster; but it needs a bit more thinking to have this done
in a clean way with the proxy interface.
Example in Procfile works again.
- Update all of the ports to use the new IANA port numbers
- Update the stats section to talk about the id fields
- Remove mention of the modules
- Remove -L from all of the curl commands since it is no longer needed
- Point people at the clustering.md guide for the cluster APIs
It returns error messaage like this now:
'{"errorCode":100,"message":"Key not found","cause":"/1/pants","index":10}'
The commit trims '/1' prefix from cause field if exists.
This is a hack to make it display well. It is correct because all error causes
that contain Path puts Path at the head of the string.
As explained in #1366, the leader will fail to transmit the missed
logs if the leader receives a hearbeat response from a follower
that is not yet matched in the leader. In other words, there are
append responses that do not explicitly reject an append but
implied a gap.
This commit is based on @xiangli-cmu's idea. We should only acknowledge
upto the index of logs in the append message. This way responses to
heartbeats would never interfer with the log synchronization because
their log index is always 0.
Fixes#1366
Continue using hex everywhere. Including here.
TODO: cleanup the printing of the structs which currently have decimal
to/from:
`{Type:MsgAppResp To:9973738105406047488 From:17050684879817348455 T...`
1) Don't panic since we know exactly where this is coming from and don't
need the user to see a full back trace
2) Add docs explaining this situation a bit further
3) Cleanup the error to look like other similiar errors
When adding new member, the etcdserver generates the ID based on the current time
and the given peerurls. We include time to add the uniqueness, since the node with
same peerurls should be able to (add, then remove) several times.
Removes the notion of name being anything more than advisory or
command-line grouping, and adds checks for bootstrapping the command
line. IDs are consistent if the URLs are consistent.
This adds the remaining two stats endpoints: `/v2/stats/self`, for
various statistics on the EtcdServer, and `/v2/stats/leader`, for
statistics on a leader's followers.
By and large most of the stats code is copied across from 0.4.x, updated
where necessary to integrate with the new decoupling of raft from
transport.
This does not satisfactorily resolve the question of name vs ID. In the
old world, names were unique in the cluster and transmitted over the
wire, so they could be used safely in all statistics. In the new world,
a given EtcdServer only knows its own name, and it is instead IDs that
are communicated among the cluster members. Hence in most places here we
simply substitute a string-encoded ID in place of name, and only where
possible do we retain the actual given name of the EtcdServer.
We use nodeID as the default dir previously. It works fine before we do dynamic nodeID
generation (introducing time). After the change the dynamic nodeID will change every
time we restart the etcd process. If the user does not provide the data dir, the default
dir will change every time. It is not the desired behavior.
In this commit, we change the default data dir to node name. If the user changes the node
name and does provide the data dir, etcd still cannot recover from previous state. But it
is much better than using nodeID. And it is actually a doucmentation issue.
Conflicts:
main.go
SSLv3 is no longer considered secure, and is not supported by golang
clients. Set the minimum version of all TLSConfigs that etcd uses to
ensure that only TLS >=1.0 can be used.
Since these are arrays instead of maps, the "keys" here are actually
(useless) goto labels. What really matters is that the ordering is
the same between the constant declarations and the array.
Before this commit, compact always compact log at current appliedindex of raft.
This prevents us from doing non-blocking snapshot since we have to make snapshot
and compact atomically. To prepare for non-blocking snapshot, this commit make
compact supports index and nodes parameters. After completing snapshot, the applier
should call compact with the snapshot index and the nodes at snapshot index to do
a compaction at snapsohot index.
In preperation for adding the ability to join a machine to an existing
cluster force the user to specify whether they expect this to me a new
cluster or an active one.
The error for not specifying the initial-cluster-state is:
```
etcd: initial cluster state unset and no wal found
```
send should not attach current term to msgProp. Send should simply do proxy for msgProp without
changing its term. msgProp has a special term 0, which indicates that it is a local message.
It's slightly unclear why we expose this timeout as being configurable,
and the `-timeout` flag does not exist in 0.4.x, so for now, remove the
flag until we have evidence that it is needed.
This adds a StartIndex field to the Watcher interface, which represents
the Etcd-Index at which the Watcher is created.
Also refactors the HTTP tests to use a table for most handleWatch tests
The usage of removed:
1. tell removed node about its removal explicitly using msgDenied
2. prevent removed node disrupt cluster progress by launching leader election
It is set when apply node removal, or receive msgDenied.
The -addr, -bind-addr, -peer-addr, and peer-bind-addr flags are
superseded by -advertise-client-urls, -listen-client-urls,
-advertise-peer-urls, and -listen-peer-urls, respectively.
If any of the former flags are provided, however, they will still
work. If the new counterparts to the old flags are provided, etcd
will log an error and fail.
This introduces two new concepts: the cluster and the member.
Members are logical etcd instances that have a name, raft ID, and a list
of peer and client addresses.
A cluster is made up of a list of members.
This prevents the bug like this:
1. a node sends join to a cluster and succeeds
2. it starts with empty peers and waits for sync, but it have not
received anything
3. election timeout passes, and it promotes itself to leader
4. it commits some log entry
5. its log conflicts with the cluster's
The work being done in the tests is completely wasted, as we do not
need to test the udnerlying x509 library. Faking out the parser function
allows the tests to run much faster without having to carry massive
fixtures, either.
This adds an EtcdIndex field to store.Event and uses that as the header
instead of the node's modifiedIndex. To facilitate this in a non-racy
way, we set the EtcdIndex while holding the lock.
Configure is used to propose config change. AddNode and RemoveNode is
used to apply cluster change to raft state machine. They are the
basics for dynamic configuration.
golang's concept of "zero" time does not align with the zero value of
a unix timestamp. Initializing time.Time with a unix timestamp of 0
makes time.Time.IsZero fail. Solve this by initializing time.Time only
if we care about the time.
This is to fix possible testing failures caused by sync tests.
Changes:
1. Get rid of time sleep operations, which introduces uncertainty.
2. Use fake Store.
This changes etcdserver.Server to an interface, with the former Server
(now "EtcdServer") becoming the canonical/production implementation.
This will facilitate better testing of the http server et al with mock
implementations of the interface.
It also more clearly defines the boundary for users of the Server.
The ReverseProxy code from the standard library doesn't actually
give us the control that we want. Pull it down and rip out what
we don't need, adding tests in the process.
All available endpoints are attempted when proxying a request. If a
proxied request fails, the upstream will be considered unavailable
for 5s and no more requests will be proxied to it. After the 5s is
up, the endpoint will be put back to rotation.
Make raftIndex section to be expected raftIndex of next entry.
It makes filename more intuitive and straight-forward.
The commit also adds comments for filename format.
golang convention dictates that the individual characters in an
abbreviation should all have the same case. Use ID instead of Id.
The protobuf generator still generates code that does not meet
this convention, but that's a fight for another day.
Add basic input validation of all query parameters supported by
serveKeys. Also restructures etcdhttp a bit to better facilitate
testing.
Test coverage is slightly improved.
Using 0xBEEF is annoying in examples because it makes it makes it look
like the user can use ascii or something. In the Procfile use
0x0,0x1,0x2,etc and use 0xBAD0 in test.
This adds a build script that attempts to be as user friendly as
possible: if they have already set $GOPATH and/or $GOBIN, use those
environment variables. If not, create a gopath for them in this
directory. This should facilitate both `go get` and `git clone` usage.
The `test` script is updated, and the new `cover` script facilitates
easy coverage generation for the repo's constituent packages by setting
the PKG environment variable.
Why do this? `go test ./...` has a ton of annoying output:
```
? github.com/coreos/etcd [no test files]
? github.com/coreos/etcd/crc [no test files]
? github.com/coreos/etcd/elog [no test files]
? github.com/coreos/etcd/error [no test files]
ok github.com/coreos/etcd/etcdserver 0.267s
ok github.com/coreos/etcd/etcdserver/etcdhttp 0.022s
? github.com/coreos/etcd/etcdserver/etcdserverpb [no test files]
ok github.com/coreos/etcd/raft 0.157s
? github.com/coreos/etcd/raft/raftpb [no test files]
ok github.com/coreos/etcd/snap 0.018s
? github.com/coreos/etcd/snap/snappb [no test files]
third_party/code.google.com/p/gogoprotobuf/proto/testdata/test.pb.go🔢
undefined: __emptyarchive__.Extension
ok github.com/coreos/etcd/store 4.247s
ok
github.com/coreos/etcd/third_party/code.google.com/p/go.net/context
2.724s
FAIL
github.com/coreos/etcd/third_party/code.google.com/p/gogoprotobuf/proto
[build failed]
ok
github.com/coreos/etcd/third_party/github.com/stretchr/testify/assert
0.013s
ok github.com/coreos/etcd/wait 0.010s
ok github.com/coreos/etcd/wal 0.024s
? github.com/coreos/etcd/wal/walpb [no test files]
```
And we have no had to manually configure drone.io which I want to avoid:
https://drone.io/github.com/coreos/etcd/admin
The documentation for these options has been incorrect ever since their
addition via commit 084dcb55. This broke upgrading a CoreOS stable
cluster to 410.0.0 because the user was using the example TOML config
which contains:
http_write_timeout = 10
This was silently ignored in the old stable version which predates the
addition of these options. After upgrading etcd began failing with:
Type mismatch for 'config.Config.http_write_timeout': Expected float
but found 'int64’.
Original issue: https://github.com/coreos/update_engine/issues/45
The default is for connections to last forever[1]. This leads to fds
leaking. I set the timeout so high by default so that watches don't have
to keep retrying but perhaps we should set it slower.
Tested on a cluster with lots of clients and it seems to have relieved
the problem.
[1] https://groups.google.com/forum/#!msg/golang-nuts/JFhGwh1q9xU/heh4J8pul3QJ
Original Commit: 084dcb5596
From: philips <brandon@ifup.org>
Many http clients will missbehave unless they get an initial http-
response, even when long-polling. It also saves the user/client from
having to handle headers on the first action of the watch, but rather
handle the response immediately.
Original commit: 2338481bb1
From: Christoffer Vikström
To make configuration change safe without adding configuration protocol:
1. We only allow to add/remove one node at a time.
2. We only allow one uncommitted configuration entry in the log.
These two rules can make sure there is no disjoint quorums in both current cluster and the
future(after applied any number of committed entries or uncommitted entries in log) clusters.
We add a type field in Entry structure for two reasons:
1. Statemachine needs to know if there is a pending configuration change.
2. Configuration entry should be executed by raft package rather application who is using raft.
The default is for connections to last forever[1]. This leads to fds
leaking. I set the timeout so high by default so that watches don't have
to keep retrying but perhaps we should set it slower.
Tested on a cluster with lots of clients and it seems to have relieved
the problem.
[1] https://groups.google.com/forum/#!msg/golang-nuts/JFhGwh1q9xU/heh4J8pul3QJ
Many http clients will missbehave unless they get an initial http-
response, even when long-polling. It also saves the user/client from
having to handle headers on the first action of the watch, but rather
handle the response immediately.
Currently the only way we know that a peer isn't getting a heartbeat is
an edge triggered event from go raft on every missed heartbeat. This
means that we need to do some book keeping in order to do exponential
backoff.
The upside is that instead of screaming thousands of log lines before a
machine hits the default removal of 30 minutes it is only ~100.
The cluster configuration operation is originally to make sure
the instance won't be added back automatically between removal and
check for the number of existing peer-mode instances. But this
could make some node removed before the removal command.
Use longer sync interval instead to avoid this problem.
Changed the LeaderInfo struct "start time" field from "startTime" to "StartTime" so that it is an exported identifier. This required adding the `json:"startTime"` structure field tag so that the encoding/json package correctly performs JSON encoding (i.e. the correct property name --> startTime).
The go-etcd client waits for the response from the paused node. And the test waits for the reponse to continue.
Actually we do not even need that small test, since we will check the machine status afterwards.
A peer might be removed during a network partiton. When it comes back it
will not have received any of the log entries that would have notified
it of its removal and go onto propose a vote. This will disrupt the
cluster and the cluster should give the machine feedback that it is no
longer a member.
The term of a denied vote is MaxUint64. The notification of the removal
is a raft event. These two modification are quick heck.
In reaction to this notification the machine should shutdown. In this
case the shutdown just moves it towards becoming a standby server.
return a valid URL struct from sanitizeURL()
pass the URL struct above to sanitizeBindAddr()
Since url.Parse() will return an error when parsing an already-parsed
ipv6 url string, (e.g. [http://[fe80::6203:8ff:fe9e:ace%25eth0]:7001),
so I just return the valid URL struct from sanitizeURL() and send it to
sanitizeBindAddr(), then there is no need to parse it again in sanitizeBindAddr().
Besides, for IPV6 url, the percent sign should be escaped, see:
http://en.wikipedia.org/wiki/IPv6_address#Link-local_addresses_and_zone_indices
Change log:
1. PeerServer
- estimate initial mode from its log through removedInLog variable
- refactor FindCluster to return the estimation
- refactor Start to call FindCluster explicitly
- move raftServer start and cluster init from FindCluster to Start
- remove stopNotify from PeerServer because it is not used anymore
2. Etcd
- refactor Run logic to fit the specification
3. ClusterConfig
- rename promoteDelay to removeDelay for better naming
- add SyncClusterInterval field to ClusterConfig
- commit command to set default cluster config when cluster is created
- store cluster config info into key space for consistency
- reload cluster config when reboot
4. add StandbyServer
5. Error
- remove unused EcodePromoteError
Peer server will be started and stopped repeatedly in the design.
This step ensures its stop doesn't affect the next start.
The patch includes goroutine stop and timer trigger remove.
- don't close ready channel until PeerServer is listening.
avoids possible panic in Stop() if PeerServer is nil.
- avoid data race in Run() (err variable was shared between 2 goroutines)
- avoid data race in PeerServer Start/Stop (PeerServer.closeChan)
1. We use PUT request to do a V2 join. So we should redirect a PUT request rather than a POST.
2. /admin only accept V2Join request. Send out V2Join instead of V1Join.
DEBIAN_FRONTEND=noninteractive tells apt-get install to use the
noninteractive front end for debconf. This keeps the system from
attempting to interact with the user during package configuration.
Without it, debconf issues warnings like:
```
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (@INC contains:
/etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2
/usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14
/usr/local/lib/site_perl .) at
/usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
```
The use case is to make it easy to embed etcd into another programs.
Currently, the only way of doing this to copy and paste the contents
of the `main()` in etcd.go.
With this commit, contents of the `main()` is moved into `Etcd.Run()`.
Added timeout goroutine to stop waiting on lock after timeout
expiration.
This necessitated reworking the flow of the acquire handler. createNode
now _only_ creates the node; it no longer waits on the lock itself.
getLockIndex (perhaps this is poorly named) extracts out the lock
checking routine so that it can be used by "get" and "watch", both.
get() was added to instantaneously attempt to acquire a lock with no
waiting.
If a lock fails to acquire, for whatever reason, an error is returned,
resulting in a code 500 to the client.
These tests were timing out because the combination of the GET and JSON
decode was taking longer than the timeout. Pull the JSON decode out of
the go routine that is being timed.
Snapshot can be tens of MB. We need to have a minute level timeout for sending snapshot.
We add a separate snapshot http client for snapshot request, which has 120s timeout.
The waitIndex was being pulled from the wrong node in the lock parent which caused the watch to be returned immediately. This caused a continuous set of calls while a client was waiting for a lock.
Heartbeat timeout means the period length that indicates heartbeat is
out of service, which is different from heartbeat interval.
So we should use '-peer-heartbeat-interval' instead of
'-peer-heartbeat-timeout' in etcd.
'-peer-heartbeat-timeout' is deprecated but still could be used.
Before this change if the value of a Node was "" it would get dropped from the json
encoding and the Node.Value field would be missing. Fix this problem by making
NodeExtern.Value a *string so that an empty string will be encoded but a nil value
will drop the field.
The boot order for peers is -discovery, -peers, log data, forming
new cluster itself.
Special rules:
1. If discovery succeeds, it would find peers specified by discover URL
only.
2. Etcd would fail when meeting bad -discovery, no -peers and log data.
Add TestDiscoveryDownNoBackupPeersWithDataDir as the test.
Lets get a bit more consistent in printing the versions:
```
$ ./bin/etcd -version
etcd version 0.3.0+git
$ ./bin/etcdctl -v
etcdctl version 0.3.0+git
```
This should cover most aspects of the v1 API but being new to the etcd
code base I cannot promise that or that the tests are even correct.
They do pass though :)
ModifiedIndex in the lock directory is not incremented and the event
store has only 1000 histories for now.
Acquiring a lock is not queued while (index - ModifiedIndex) > 1000 and
prevIndex exists.
If closeChan is available and stopWatchChan was closed by defer
select selects a single channel randomly. This causes a panic sending to closed
channel.
etcd is open source, Apache 2.0 licensed and accepts contributions via Github pull requests.
This document outlines some of the conventions on commit message formatting, contact points for developers and other resources to make getting your contribution into etcd easier.
etcd is Apache 2.0 licensed and accepts contributions via Github pull requests. This document outlines some of the conventions on commit message formatting, contact points for developers and other resources to make getting your contribution into etcd easier.
# Email and chat
For simplicity etcd discussions happen on coreos-dev and in #coreos-dev.
As the community grows we will move to a dedicated mailing list and IRC channel.
- IRC: #[coreos](irc://irc.freenode.org:6667/#coreos) IRC channel on freenode.org
## Getting Started
## Getting started
- Fork the repository on GitHub
- Read the README.md for build instructions
@ -22,54 +18,45 @@ This is a rough outline of what a contributor's workflow looks like:
- Create a topic branch from where you want to base your work. This is usually master.
- Make commits of logical units.
- Make sure your commit messages are in the proper format, see below
- Make sure your commit messages are in the proper format (see below).
- Push your changes to a topic branch in your fork of the repository.
- Submit a pull request to coreos/etcd
- Submit a pull request to coreos/etcd.
- Your PR must receive a LGTM from two maintainers found in the MAINTAINERS file.
Thanks for you contributions!
Thanks for your contributions!
### Format of the commit message
### Code style
etcd follow a rough convention for commit messages borrowed from Angularjs.
This is an example of a commit:
The coding style suggested by the Golang community is used in etcd. See the [style doc](https://code.google.com/p/go-wiki/wiki/CodeReviewComments) for details.
Please follow this style to make etcd easy to review, maintain and develop.
### Format of the Commit Message
We follow a rough convention for commit messages that is designed to answer two
questions: what changed and why. The subject line should feature the what and
the body of the commit should describe the why.
```
feat(scripts/test-cluster): add a cluster test command
scripts: add the test-cluster command
this uses tmux to setup a test cluster that you can easily kill and
start for debugging.
Fixes #38
```
To make it more formal it looks something like this:
The format can be described more formally as follows:
```
<type>(<scope>): <subject>
<subsystem>: <what changed>
<BLANK LINE>
<body>
<why this change was made>
<BLANK LINE>
<footer>
```
The first line is the subject and should not be longer than 70 characters, the second line is always blank and other lines should be wrapped at 80 characters.
This allows the message to be easier to read on github as well as in various git tools.
### Subject line
The subject line contains succinct description of the change.
### Allowed <type>
- feat (feature)
- fix (bug fix)
- docs (documentation)
- style (formatting, missing semi colons, …)
- refactor
- test (when adding missing tests)
- chore (maintain)
### Allowed <scope>
Scopes could be anything specifying place of the commit change. For example store, api, etc.
### More details on commits
For more details see the [angularjs commit style guide](https://docs.google.com/a/coreos.com/document/d/1QrDFcIiPjSLDn3EL15IJygNPiHORgU1_OOAqWjiDU5Y/edit#).
The first line is the subject and should be no longer than 70 characters, the
second line is always blank, and other lines should be wrapped at 80 characters.
This allows the message to be easier to read on GitHub as well as in various
Return an HTTP 200 OK response code and a representation of all members in the etcd cluster:
```
Example Request: GET
http://localhost:2379/v2/admin/members/
Response formats: JSON
Example Response:
```
```json
{
"members":[
{
"id":"272e204152",
"name":"node1",
"peerURLs":[
"http://10.0.0.10:2379"
],
"clientURLs":[
"http://10.0.0.10:2380"
]
},
{
"id":"2225373f43",
"name":"node2",
"peerURLs":[
"http://127.0.0.11:2379"
],
"clientURLs":[
"http://127.0.0.11:2380"
]
},
]
}
```
### POST /v2/admin/members/
Add a member to the cluster.
Returns an HTTP 201 response code and the representation of added member with a newly generated a memberID when successful. Returns a string describing the failure condition when unsuccessful.
If the POST body is malformed an HTTP 400 will be returned. If the member exists in the cluster or existed in the cluster at some point in the past an HTTP 500(TODO: fix this) will be returned. If the cluster fails to process the request within timeout an HTTP 500 will be returned, though the request may be processed later.
```
Example Request: POST
http://localhost:2379/v2/admin/members/
Body:
[{"PeerURLs":["http://10.0.0.10:2379"]}]
Respose formats: JSON
Example Response:
```
```json
[
{
"id":"3777296169",
"peerURLs":[
"http://10.0.0.10:2379"
],
},
]
```
### DELETE /v2/admin/members/:id
Remove a member from the cluster.
Returns empty when successful. Returns a string describing the failure condition when unsuccessful.
If the member does not exist in the cluster an HTTP 500(TODO: fix this) will be returned. If the cluster fails to process the request within timeout an HTTP 500 will be returned, though the request may be processed later.
|-peer-addr |-initial-advertise-peer-urls |If specified, peer-addr will be used as the only peer URL. Error if both flags specified.|
|-addr |-advertise-client-urls |If specified, addr will be used as the only client URL. Error if both flags specified.|
|-peer-bind-addr |-listen-peer-urls |If specified, peer-bind-addr will be used as the only peer bind URL. Error if both flags specified.|
|-bind-addr |-listen-client-urls |If specified, bind-addr will be used as the only client bind URL. Error if both flags specified.|
|-peers |none |Deprecated. The -initial-cluster flag provides a similar concept with different semantics. Please read this guide on cluster startup.|
|-peers-file |none |Deprecated. The -initial-cluster flag provides a similar concept with different semantics. Please read this guide on cluster startup.|
This document defines the various terms used in etcd documentation, command line and source code.
### Node
Node is an instance of raft state machine.
It has a unique identification, and records other nodes' progress internally when it is the leader.
### Member
Member is an instance of etcd. It hosts a node, and provides service to clients.
### Cluster
Cluster consists of several members.
The node in each member follows raft consensus protocol to replicate logs. Cluster receives proposals from members, commits them and apply to local store.
`activeSize` is the maximum number of peers that can join the cluster and participate in the consensus protocol.
The size of cluster is controlled to be around a certain number. If it is not, standby-mode instances will join or peer-mode instances will be removed to make it happen.
`removeDelay` indicates the minimum time that a machine has been observed to be unresponsive before it is removed from the cluster.
### Get Cluster Config
```sh
curl -L http://127.0.0.1:7001/v2/admin/config
```
```json
{
"activeSize":3,
"removeDelay":1800,
"syncInterval":5
}
```
## Remove Machines
At times you may want to manually remove a machine. Using the machines endpoint
you can find and remove machines.
First, list all the machines in the cluster.
```sh
curl -L http://127.0.0.1:7001/v2/admin/machines
```
```json
[
{
"clientURL":"http://127.0.0.1:4001",
"name":"peer1",
"peerURL":"http://127.0.0.1:7001",
"state":"leader"
},
{
"clientURL":"http://127.0.0.1:4002",
"name":"peer2",
"peerURL":"http://127.0.0.1:7002",
"state":"follower"
},
{
"clientURL":"http://127.0.0.1:4003",
"name":"peer3",
"peerURL":"http://127.0.0.1:7003",
"state":"follower"
}
]
```
Then take a closer look at the machine you want to remove.
Starting an etcd cluster can be painful since each node needs to know of another node in the cluster to get started. If you are trying to bring up a cluster all at once, say using a cloud formation, you also need to coordinate who will be the initial cluster leader. The discovery protocol helps you by providing an automated way to discover other existing peers in a cluster.
Starting an etcd cluster requires that each node knows another in the cluster. If you are trying to bring up a cluster all at once, say using a cloud formation, you also need to coordinate who will be the initial cluster leader. The discovery protocol helps you by providing an automated way to discover other existing peers in a cluster.
For more information on how etcd can locate the cluster, see the [finding the cluster][cluster-finding] documentation.
Please note - at least 3 nodes are required for [cluster availability][optimal-cluster-size].
To use the discovery API, you must first create a token for your etcd cluster. Visit [https://discovery.etcd.io/new](https://discovery.etcd.io/new) to create a new token.
To use the discovery API, you must first create a unique discovery URL for your etcd cluster. Visit [https://discovery.etcd.io/new](https://discovery.etcd.io/new) to create a new discovery URL.
You can inspect the list of peers by viewing `https://discovery.etcd.io/<token>`.
You can inspect the list of peers by viewing `https://discovery.etcd.io/<cluster id>`.
### Start etcd With the Discovery Flag
@ -19,9 +26,10 @@ Specify the `-discovery` flag when you start each etcd instance. The list of exi
The discovery API communicates with a separate etcd cluster to store and retrieve the list of peers. CoreOS provides [https://discovery.etcd.io](https://discovery.etcd.io) as a free service, but you can easily run your own etcd cluster for this purpose. Here's an example using an etcd cluster located at `10.10.10.10:4001`:
If you're interested in how to discovery API works behind the scenes, read about the [Discovery Protocol](https://github.com/coreos/etcd/blob/master/Documentation/discovery-protocol.md).
@ -43,3 +52,9 @@ The Discovery API submits the `-peer-addr` of each etcd instance to the configur
## Stale Peers
The discovery API will automatically clean up the address of a stale peer that is no longer part of the cluster. The TTL for this process is a week, which should be long enough to handle any extremely long outage you may encounter. There is no harm in having stale peers in the list until they are cleaned up, since an etcd instance only needs to connect to one valid peer in the cluster to join.
## Lifetime of a Discovery URL
A discovery URL identifies a single etcd cluster. Do not re-use discovery URLs for new clusters.
When a machine starts with a new discovery URL the discovery URL will be activated and record the machine's metadata. If you destroy the whole cluster and attempt to bring the cluster back up with the same discovery URL it will fail. This is intentional because all of the registered machines are gone including their logs so there is nothing to recover the killed cluster.
If one machine disconnects from the cluster, it could rejoin the cluster automatically when the communication is recovered.
If one machine is killed, it could rejoin the cluster when started with old name. If the peer address is changed, etcd will treat the new peer address as the refreshed one, which benefits instance migration, or virtual machine boot with different IP. The peer-address-changing functionality is only supported when the majority of the cluster is alive, because this behavior needs the consensus of the etcd cluster.
**Note:** For now, it is user responsibility to ensure that the machine doesn't join the cluster that has the member with the same name. Or unexpected error will happen. It would be improved sooner or later.
### Killing Nodes in the Cluster
@ -161,14 +168,8 @@ To do this just change the `-*-file` flags to `-peer-*-file`.
If you are using SSL for server-to-server communication, you must use it on all instances of etcd.
### Bootstrapping a new cluster by name
### What size cluster should I use?
An etcd server is uniquely defined by the peer addresses it listens to. Suppose, however, that you wish to start over, while maintaining the data from the previous cluster -- that is, to pretend that this machine has never joined a cluster before.
Every command the client sends to the master is broadcast to all of the followers.
The command is not committed until the majority of the cluster peers receive that command.
Because of this majority voting property, the ideal cluster should be kept small to keep speed up and be made up of an odd number of peers.
Odd numbers are good because if you have 8 peers the majority will be 5 and if you have 9 peers the majority will still be 5.
The result is that an 8 peer cluster can tolerate 3 peer failures and a 9 peer cluster can tolerate 4 machine failures.
And in the best case when all 9 peers are responding the cluster will perform at the speed of the fastest 5 machines.
You can use `--initial-cluster-name` to generate a new unique ID for each node, as a shared token that every node understands. Nodes also take this into account for bootstrapping the new cluster ID, so it also provides a way for a machine to listen on the same interfaces, disconnect from one cluster, and join a different cluster.
Individual node configuration options can be set in three places:
1. Command line flags
2. Environment variables
@ -10,35 +12,51 @@ Options set on the command line take precedence over all other sources.
Options set in environment variables take precedence over options set in
configuration files.
## Cluster Configuration
Cluster-wide settings are configured via the `/config` admin endpoint and additionally in the configuration file. Values contained in the configuration file will seed the cluster setting with the provided value. After the cluster is running, only the admin endpoint is used.
The full documentation is contained in the [API docs](https://github.com/coreos/etcd/blob/master/Documentation/api.md#cluster-config).
*`activeSize` - the maximum number of peers that can participate in the consensus protocol. Other peers will join as standbys.
*`removeDelay` - the minimum time in seconds that a machine has been observed to be unresponsive before it is removed from the cluster.
*`syncInterval` - the amount of time in seconds between cluster sync when it runs in standby mode.
## Command Line Flags
### Required
*`-name` - The node name. Defaults to the hostname.
*`-name` - The node name. Defaults to a UUID.
### Optional
*`-addr` - The advertised public hostname:port for client communication. Defaults to `127.0.0.1:4001`.
*`-discovery` - A URL to use for discovering the peer list. (i.e `"https://discovery.etcd.io/your-unique-key"`).
*`-bind-addr` - The listening hostname for client communication. Defaults to advertised ip.
*`-http-read-timeout` - The number of seconds before an HTTP read operation is timed out.
*`-http-write-timeout` - The number of seconds before an HTTP write operation is timed out.
*`-bind-addr` - The listening hostname for client communication. Defaults to advertised IP.
*`-peers` - A comma separated list of peers in the cluster (i.e `"203.0.113.101:7001,203.0.113.102:7001"`).
*`-peers-file` - The file path containing a comma separated list of peers in the cluster.
*`-ca-file` - The path of the client CAFile. Enables client cert authentication when present.
*`-cert-file` - The cert file of the client.
*`-key-file` - The key file of the client.
*`-config` - The path of the etcd config file. Defaults to `/etc/etcd/etcd.conf`.
*`-config` - The path of the etcd configuration file. Defaults to `/etc/etcd/etcd.conf`.
*`-cors` - A comma separated white list of origins for cross-origin resource sharing.
*`-cpuprofile` - The path to a file to output cpu profile data. Enables cpu profiling when present.
*`-cpuprofile` - The path to a file to output CPU profile data. Enables CPU profiling when present.
*`-data-dir` - The directory to store log and snapshot. Defaults to the current working directory.
*`-max-result-buffer` - The max size of result buffer. Defaults to `1024`.
*`-max-cluster-size` - The max size of the cluster. Defaults to `9`.
*`-max-retry-attempts` - The max retry attempts when trying to join a cluster. Defaults to `3`.
*`-peer-addr` - The advertised public hostname:port for server communication. Defaults to `127.0.0.1:7001`.
*`-peer-bind-addr` - The listening hostname for server communication. Defaults to advertised ip.
*`-peer-bind-addr` - The listening hostname for server communication. Defaults to advertised IP.
*`-peer-ca-file` - The path of the CAFile. Enables client/peer cert authentication when present.
*`-peer-cert-file` - The cert file of the server.
*`-peer-key-file` - The key file of the server.
*`-peer-election-timeout` - The number of milliseconds to wait before the leader is declared unhealthy.
*`-peer-heartbeat-interval` - The number of milliseconds in between heartbeat requests
*`-snapshot=false` - Disable log snapshots. Defaults to `true`.
*`-cluster-active-size` - The expected number of instances participating in the consensus protocol. Only applied if the etcd instance is the first peer in the cluster.
*`-cluster-remove-delay` - The number of seconds before one node is removed from the cluster since it cannot be connected at all. Only applied if the etcd instance is the first peer in the cluster.
*`-cluster-sync-interval` - The number of seconds between synchronization for standby-mode instance with the cluster. Only applied if the etcd instance is the first peer in the cluster.
*`-v` - Enable verbose logging. Defaults to `false`.
*`-vv` - Enable very verbose logging. Defaults to `false`.
@ -44,10 +44,10 @@ See an [example collectd deploy script](https://github.com/coreos/etcd/contrib/c
## Profiling
etcd exposes profiling information from the Go pprof package over HTTP.
The basic browseable interface is served by etcd at the `/debug/pprof` HTTP endpoint (i.e. `http://127.0.0.1:4001/debug/pprof`).
The basic browsable interface is served by etcd at the `/debug/pprof` HTTP endpoint (i.e. `http://127.0.0.1:4001/debug/pprof`).
For more information on using profiling tools, see http://blog.golang.org/profiling-go-programs.
**NOTE**: In the following examples you need to ensure that the `./bin/etcd` is identical to the `./bin/etcd` that you are targetting (same git hash, arch, platform, etc).
**NOTE**: In the following examples you need to ensure that the `./bin/etcd` is identical to the `./bin/etcd` that you are targeting (same git hash, arch, platform, etc).
Peer discovery uses the following sources in this order: log data in `-data-dir`, `-discovery` and `-peers`.
If log data is provided, etcd will concatenate possible peers from three sources: the log data, the `-discovery` option, and `-peers`. Then it tries to join cluster through them one by one. If all connection attempts fail (which indicates that the majority of the cluster is currently down), it will restart itself based on the log data, which helps the cluster to recover from a full outage.
Without log data, the instance is assumed to be a brand new one. If possible targets are provided by `-discovery` and `-peers`, etcd will make a best effort attempt to join them, and if none is reachable it will exit. Otherwise, if no `-discovery` or `-peers` option is provided, a new cluster will always be started.
This ensures that users can always restart the node safely with the same command (without --force), and etcd will either reconnect to the old cluster if it is still running or recover its cluster from a outage.
Adding peers in an etcd cluster adds network, CPU, and disk overhead to the leader since each one requires replication.
Peers primarily provide resiliency in the event of a leader failure but the benefit of more failover nodes decreases as the cluster size increases.
A lightweight alternative is the standby.
Standbys are a way for an etcd node to forward requests along to the cluster but the standbys are not part of the Raft cluster themselves.
This provides an easier API for local applications while reducing the overhead required by a regular peer node.
Standbys also act as standby nodes in the event that a peer node in the cluster has not recovered after a long duration.
## Configuration Parameters
There are three configuration parameters used by standbys: active size, remove delay and standby sync interval.
The active size specifies a target size for the number of peers in the cluster.
If there are not enough peers to meet the active size, standbys will send join requests until the peer count is equal to the active size.
If there are more peers than the target active size then peers are removed by the leader and will become standbys.
The remove delay specifies how long the cluster should wait before removing a dead peer.
By default this is 30 minutes.
If a peer is inactive for 30 minutes then the peer is removed.
The standby sync interval specifies the synchronization interval of standbys with the cluster.
By default this is 5 seconds.
After each interval, standbys synchronize information with cluster.
## Logical Workflow
### Start a etcd machine
#### Main logic
```
If find existing standby cluster info:
Goto standby loop
Find cluster as required
If determine to start peer server:
Goto peer loop
Else:
Goto standby loop
Peer loop:
Start peer mode
If running:
Wait for stop
Goto standby loop
Standby loop:
Start standby mode
If running:
Wait for stop
Goto peer loop
```
#### [Cluster finding logic][cluster-finding.md]
#### Join request logic:
```
Fetch machine info
If cannot match version:
return false
If active size <= peer count:
return false
If it has existed in the cluster:
return true
If join request fails:
return false
return true
```
**Note**
1. [TODO] The running mode cannot be determined by log, because the log may be outdated. But the log could be used to estimate its state.
2. Even if sync cluster fails, it will restart still for recovery from full outage.
#### Peer mode start logic
```
Start raft server
Start other helper routines
```
#### Peer mode auto stop logic
```
When removed from the cluster:
Stop raft server
Stop other helper routines
```
#### Standby mode run logic
```
Loop:
Sleep for some time
Sync cluster, and write cluster info into disk
Check active size and send join request if needed
If succeed:
Clear cluster info from disk
Return
```
#### Serve Requests as Standby
Return '404 Page Not Found' always on peer address. This is because peer address is used for raft communication and cluster management, which should not be used in standby mode.
Serve requests from client:
```
Redirect all requests to client URL of leader
```
**Note**
1. The leader here implies the one in raft cluster when doing the latest successful synchronization.
2. [IDEA] We could extend HTTP Redirect to multiple possible targets.
### Join Request Handling
```
If machine has existed in the cluster:
Return
If peer count < active size:
Add peer
Increase peer count
```
### Remove Request Handling
```
If machine exists in the cluster:
Remove peer
Decrease peer count
```
## Cluster Monitor Logic
### Active Size Monitor:
This is only run by current cluster leader.
```
Loop:
Sleep for some time
If peer count > active size:
Remove randomly selected peer
```
### Peer Activity Monitor
This is only run by current cluster leader.
```
Loop:
Sleep for some time
For each peer:
If peer last activity time > remove delay:
Remove the peer
Goto Loop
```
## Cluster Cases
### Create Cluster with Thousands of Instances
First few machines run in peer mode.
All the others check the status of the cluster and run in standby mode.
### Recover from full outage
Machines with log data restart with join failure.
Machines in peer mode recover heartbeat between each other.
Machines in standby mode always sync the cluster. If sync fails, it uses the first address from data log as redirect target.
### Kill one peer machine
Leader of the cluster lose the connection with the peer.
When the time exceeds remove delay, it removes the peer from the cluster.
Machine in standby mode finds one available place of the cluster. It sends join request and joins the cluster.
**Note**
1. [TODO] Machine which was divided from majority and was removed from the cluster will distribute running of the cluster if the new node uses the same name.
### Kill one standby machine
No change for the cluster.
## Cons
1. New instance cannot join immediately after one peer is kicked out of the cluster, because the leader doesn't know the info about the standby instances.
2. It may introduce join collision
3. Cluster needs a good interval setting to balance the join delay and join collision.
## Future Attack Plans
1. Based on heartbeat miss and remove delay, standby could adjust its next check time.
2. Preregister the promotion target when heartbeat miss happens.
3. Get the estimated cluster size from the check happened in the sync interval, and adjust sync interval dynamically.
4. Accept join requests based on active size and alive peers.
@ -77,3 +96,8 @@ A detailed recap of client functionalities can be found in the [clients compatib
- [mattn/etcdenv](https://github.com/mattn/etcdenv) - "env" shebang with etcd integration
- [kelseyhightower/confd](https://github.com/kelseyhightower/confd) - Manage local app config files using templates and data from etcd
- [configdb](https://git.autistici.org/ai/configdb/tree/master) - A REST relational abstraction on top of arbitrary database backends, aimed at storing configs and inventories.
- [scrz](https://github.com/scrz/scrz) - Container manager, stores configuration in etcd.
- [fleet](https://github.com/coreos/fleet) - Distributed init system
etcd has a number of modules that are built on top of the core etcd API.
These modules provide things like dashboards, locks and leader election.
These modules provide things like dashboards, locks and leader election (removed).
**Warning**: Modules are deprecated from v0.4 until we have a solid base we can apply them back onto.
For now, we are choosing to focus on raft algorithm and core etcd to make sure that it works correctly and fast.
And it is time consuming to maintain these modules in this period, given that etcd's API changes from time to time.
Moreover, the lock module has some unfixed bugs, which may mislead users.
But we also notice that these modules are popular and useful, and plan to add them back with full functionality as soon as possible.
### Dashboard
@ -15,6 +21,7 @@ Use the `-cors='*'` flag to allow your browser to request information from the c
The Lock module implements a fair lock that can be used when lots of clients want access to a single resource.
A lock can be associated with a value.
The value is unique so if a lock tries to request a value that is already queued for a lock then it will find it and watch until that value obtains the lock.
You may supply a `timeout` which will cancel the lock request if it is not obtained within `timeout` seconds. If `timeout` is not supplied, it is presumed to be infinite. If `timeout` is `0`, the lock request will fail if it is not immediately acquired.
If you lock the same value on a key from two separate curl sessions they'll both return at the same time.
Here's the API:
@ -31,13 +38,19 @@ curl -X POST http://127.0.0.1:4001/mod/v2/lock/customer1?ttl=60
curl -X POST http://127.0.0.1:4001/mod/v2/lock/customer1?ttl=60 -d value=bar
```
**Acquire a lock for "customer1" that is associated with the value "bar" only if it is done within 2 seconds**
```sh
curl -X POST http://127.0.0.1:4001/mod/v2/lock/customer1?ttl=60 -d value=bar -d timeout=2
```
**Renew the TTL on the "customer1" lock for index 2**
```sh
curl -X PUT http://127.0.0.1:4001/mod/v2/lock/customer1?ttl=60 -d index=2
```
**Renew the TTL on the "customer1" lock for value "customer1"**
**Renew the TTL on the "customer1" lock for value "bar"**
```sh
curl -X PUT http://127.0.0.1:4001/mod/v2/lock/customer1?ttl=60 -d value=bar
etcd's Raft consensus algorithm is most efficient in small clusters between 3 and 9 peers. Let's briefly explore how etcd works internally to understand why.
## Writing to etcd
Writes to an etcd peer are always redirected to the leader of the cluster and distributed to all of the peers immediately. A write is only considered successful when a majority of the peers acknowledge the write.
For example, in a 5 node cluster, a write operation is only as fast as the 3rd fastest machine. This is the main reason for keeping your etcd cluster below 9 nodes. In practice, you only need to worry about write performance in high latency environments such as a cluster spanning multiple data centers.
## Leader Election
The leader election process is similar to writing a key — a majority of the cluster must acknowledge the new leader before cluster operations can continue. The longer each node takes to elect a new leader means you have to wait longer before you can write to the cluster again. In low latency environments this process takes milliseconds.
## Odd Cluster Size
The other important cluster optimization is to always have an odd cluster size. Adding an odd node to the cluster doesn't change the size of the majority and therefore doesn't increase the total latency of the majority as described above. But you do gain a higher tolerance for peer failure by adding the extra machine. You can see this in practice when comparing two even and odd sized clusters:
| Cluster Size | Majority | Failure Tolerance |
|--------------|------------|-------------------|
| 8 machines | 5 machines | 3 machines |
| 9 machines | 5 machines | **4 machines** |
As you can see, adding another node to bring the cluster up to an odd size is always worth it. During a network partition, an odd cluster size also guarantees that there will almost always be a majority of the cluster that can continue to operate and be the source of truth when the partition ends.
etcd's Raft consensus algorithm is most efficient in small clusters between 3 and 9 peers. For clusters larger than 9, etcd will select a subset of instances to participate in the algorithm in order to keep it efficient. The end of this document briefly explores how etcd works internally and why these choices have been made.
## Cluster Management
Currently, each CoreOS machine is an etcd peer — if you have 30 CoreOS machines, you have 30 etcd peers and end up with a cluster size that is way too large. If desired, you may manually stop some of these etcd instances to increase cluster performance.
You can manage the active cluster size through the [cluster config API](https://github.com/coreos/etcd/blob/master/Documentation/api.md#cluster-config). `activeSize` represents the etcd peers allowed to actively participate in the consensus algorithm.
Functionality is being developed to expose two different types of followers: active and benched followers. Active followers will influence operations within the cluster. Benched followers will not participate, but will transparently proxy etcd traffic to an active follower. This allows every CoreOS machine to expose etcd on port 4001 for ease of use. Benched followers will have the ability to transition into an active follower if needed.
If the total number of etcd instances exceeds this number, additional peers are started as [standbys](https://github.com/coreos/etcd/blob/master/Documentation/design/standbys.md), which can be promoted to active participation if one of the existing active instances has failed or been removed.
## Internals of etcd
### Writing to etcd
Writes to an etcd peer are always redirected to the leader of the cluster and distributed to all of the peers immediately. A write is only considered successful when a majority of the peers acknowledge the write.
For example, in a cluster with 5 peers, a write operation is only as fast as the 3rd fastest machine. This is the main reason for keeping the number of active peers below 9. In practice, you only need to worry about write performance in high latency environments such as a cluster spanning multiple data centers.
### Leader Election
The leader election process is similar to writing a key — a majority of the active peers must acknowledge the new leader before cluster operations can continue. The longer each peer takes to elect a new leader means you have to wait longer before you can write to the cluster again. In low latency environments this process takes milliseconds.
### Odd Active Cluster Size
The other important cluster optimization is to always have an odd active cluster size (i.e. `activeSize`). Adding an odd node to the number of peers doesn't change the size of the majority and therefore doesn't increase the total latency of the majority as described above. But, you gain a higher tolerance for peer failure by adding the extra machine. You can see this in practice when comparing two even and odd sized clusters:
| Active Peers | Majority | Failure Tolerance |
|--------------|------------|-------------------|
| 1 peers | 1 peers | None |
| 3 peers | 2 peers | 1 peer |
| 4 peers | 3 peers | 1 peer |
| 5 peers | 3 peers | **2 peers** |
| 6 peers | 4 peers | 2 peers |
| 7 peers | 4 peers | **3 peers** |
| 8 peers | 5 peers | 3 peers |
| 9 peers | 5 peers | **4 peers** |
As you can see, adding another peer to bring the number of active peers up to an odd size is always worth it. During a network partition, an odd number of active peers also guarantees that there will almost always be a majority of the cluster that can continue to operate and be the source of truth when the partition ends.
Etcd supports SSL/TLS as well as authentication through client certificates, both for clients to server as well as peer (server to server / cluster) communication.
Etcd supports SSL/TLS and client cert authentication for clients to server, as well as server to server communication.
To get up and running you first need to have a CA certificate and a signed key pair for your node. It is recommended to create and sign a new key pair for every node in a cluster.
For convenience the [etcd-ca](https://github.com/coreos/etcd-ca) tool provides an easy interface to certificate generation, alternatively this site provides a good reference on how to generate self-signed key pairs:
First, you need to have a CA cert `clientCA.crt` and signed key pair `client.crt`, `client.key`.
This site has a good reference for how to generate self-signed key pairs:
http://www.g-loaded.eu/2005/11/10/be-your-own-ca/
For testing you can use the certificates in the `fixtures/ca` directory.
## Basic setup
Let's configure etcd to use this keypair:
Etcd takes several certificate related configuration options, either through command-line flags or environment variables:
**Client-to-server communication:**
`--cert-file=<path>`: Certificate used for SSL/TLS connections **to** etcd. When this option is set, you can reach etcd through HTTPS - for example at `https://127.0.0.1:4001`
`--key-file=<path>`: Key for the certificate. Must be unencrypted.
`--ca-file=<path>`: When this is set etcd will check all incoming HTTPS requests for a client certificate signed by the supplied CA, requests that don't supply a valid client certificate will fail.
The peer options work the same way as the client-to-server options:
`--peer-cert-file=<path>`: Certificate used for SSL/TLS connections between peers. This will be used both for listening on the peer address as well as sending requests to other peers.
`--peer-key-file=<path>`: Key for the certificate. Must be unencrypted.
`--peer-ca-file=<path>`: When set, etcd will check all incoming peer requests from the cluster for valid client certificates signed by the supplied CA.
If either a client-to-server or peer certificate is supplied the key must also be set. All of these configuration options are also available through the environment variables, `ETCD_CA_FILE`, `ETCD_PEER_CA_FILE` and so on.
## Example 1: Client-to-server transport security with HTTPS
For this you need your CA certificate (`ca.crt`) and signed key pair (`server.crt`, `server.key`) ready. If you just want to test the functionality, there are example certificates provided in the [etcd git repository](https://github.com/coreos/etcd/tree/master/fixtures/ca) (namely `server.crt` and `server.key.insecure`).
Assuming you have these files ready, let's configure etcd to use them to provide simple HTTPS transport security.
*`-f` - forces a new machine configuration, even if an existing configuration is found. (WARNING: data loss!)
*`-cert-file` and `-key-file` specify the location of the cert and key files to be used for for transport layer security between the client and server.
You can now test the configuration using HTTPS:
This should start up fine and you can now test the configuration by speaking HTTPS to etcd:
You should be able to see the handshake succeed. Because we use self-signed certificates with our own certificate authorities you need to provide the CA to curl using the `--cacert` option. Another possibility would be to add your CA certificate to the trusted certificates on your system (usually in `/etc/ssl/certs`).
**OSX 10.9+ Users**: curl 7.30.0 on OSX 10.9+ doesn't understand certificates passed in on the command line.
Instead you must import the dummy ca.crt directly into the keychain or add the `-k` flag to curl to ignore errors.
@ -35,42 +52,28 @@ If you want to test without the `-k` flag run `open ./fixtures/ca/ca.crt` and fo
Please remove this certificate after you are done testing!
If you know of a workaround let us know.
```
...
SSLv3, TLS handshake, Finished (20):
...
```
## Example 2: Client-to-server authentication with HTTPS client certificates
And also the response from the etcd server:
For now we've given the etcd client the ability to verify the server identity and provide transport security. We can however also use client certificates to prevent unauthorized access to etcd.
```json
{
"action":"set",
"key":"/foo",
"modifiedIndex":3,
"value":"bar"
}
```
The clients will provide their certificates to the server and the server will check whether the cert is signed by the supplied CA and decide whether to serve the request.
## Authentication with HTTPS Client Certificates
We can also do authentication using CA certs.
The clients will provide their cert to the server and the server will check whether the cert is signed by the CA and decide whether to serve the request.
You need the same files mentioned in the first example for this, as well as a key pair for the client (`client.crt`, `client.key`) signed by the same certificate authority.
@ -107,7 +110,28 @@ And also the response from the server:
}
```
### Why SSLv3 alert handshake failure when using SSL client auth?
## Example 3: Transport security & client certificates in a cluster
Etcd supports the same model as above for **peer communication**, that means the communication between etcd nodes in a cluster.
Assuming we have our `ca.crt` and two nodes with their own keypairs (`node1.crt`&`node1.key`, `node2.crt`&`node2.key`) signed by this CA, we launch etcd as follows:
```sh
DISCOVERY_URL=... # from https://discovery.etcd.io/new
The etcd nodes will form a cluster and all communication between nodes in the cluster will be encrypted and authenticated using the client certificates. You will see in the output of etcd that the addresses it connects to use HTTPS.
## Frequently Asked Questions
### I'm seeing a SSLv3 alert handshake failure when using SSL client authentication?
The `crypto/tls` package of `golang` checks the key usage of the certificate public key before using it.
To use the certificate public key to do client auth, we need to add `clientAuth` to `Extended Key Usage` when creating the certificate public key.
@ -128,3 +152,8 @@ When creating the cert be sure to reference it in the `-extensions` flag:
### With peer certificate authentication I receive "certificate is valid for 127.0.0.1, not $MY_IP"
Make sure that you sign your certificates with a Subject Name your node's public IP address. The `etcd-ca` tool for example provides an `--ip=` option for its `new-cert` command.
If you need your certificate to be signed for your node's FQDN in its Subject Name then you could use Subject Alternative Names (short IP SNAs) to add your IP address. This is not [currently supported](https://github.com/coreos/etcd-ca/issues/29) by `etcd-ca` but can be done [with openssl](http://wiki.cacert.org/FAQ/subjectAltName).
The default settings in etcd should work well for installations on a local network where the average network latency is low.
However, when using etcd across multiple data centers or over networks with high latency you may need to tweak the heartbeat and election timeout settings.
However, when using etcd across multiple data centers or over networks with high latency you may need to tweak the heartbeat interval and election timeout settings.
### Timeouts
The network isn't the only source of latency. Each request and response may be impacted by slow disks on both the leader and follower. Each of these timeouts represents the total time from request to successful response from the other machine.
The underlying distributed consensus protocol relies on two separate timeouts to ensure that nodes can handoff leadership if one stalls or goes offline.
The first timeout is called the *Heartbeat Timeout*.
### Time Parameters
The underlying distributed consensus protocol relies on two separate time parameters to ensure that nodes can handoff leadership if one stalls or goes offline.
The first parameter is called the *Heartbeat Interval*.
This is the frequency with which the leader will notify followers that it is still the leader.
etcd batches commands together for higher throughput so this heartbeat timeout is also a delay for how long it takes for commands to be committed.
By default, etcd uses a `50ms` heartbeat timeout.
etcd batches commands together for higher throughput so this heartbeat interval is also a delay for how long it takes for commands to be committed.
By default, etcd uses a `50ms` heartbeat interval.
The second timeout is the *Election Timeout*.
The second parameter is the *Election Timeout*.
This timeout is how long a follower node will go without hearing a heartbeat before attempting to become leader itself.
By default, etcd uses a `200ms` election timeout.
Adjusting these values is a trade off.
Lowering the heartbeat timeout will cause individual commands to be committed faster but it will lower the overall throughput of etcd.
If your etcd instances have low utilization then lowering the heartbeat timeout can improve your command response time.
Lowering the heartbeat interval will cause individual commands to be committed faster but it will lower the overall throughput of etcd.
If your etcd instances have low utilization then lowering the heartbeat interval can improve your command response time.
The election timeout should be set based on the heartbeat timeout and your network ping time between nodes.
The election timeout should be set based on the heartbeat interval and your network ping time between nodes.
Election timeouts should be at least 10 times your ping time so it can account for variance in your network.
For example, if the ping time between your nodes is 10ms then you should have at least a 100ms election timeout.
You should also set your election timeout to at least 4 to 5 times your heartbeat timeout to account for variance in leader replication.
For a heartbeat timeout of 50ms you should set your election timeout to at least 200ms - 250ms.
You should also set your election timeout to at least 4 to 5 times your heartbeat interval to account for variance in leader replication.
For a heartbeat interval of 50ms you should set your election timeout to at least 200ms - 250ms.
You can override the default values on the command line:
etcd clusters can be upgraded by doing a rolling upgrade or all at once. We make every effort to test this process, but please be sure to backup your data [by etcd-dump](https://github.com/AaronO/etcd-dump), or make a copy of data directory beforehand.
## Upgrade Process
- Stop the old etcd processes
- Upgrade the etcd binary
- Restart the etcd instance using the original --name, --address, --peer-address and --data-dir.
## Rolling Upgrade
During an upgrade, etcd clusters are designed to continue working in a mix of old and new versions. It's recommended to converge on the new version quickly. Using new API features before the entire cluster has been upgraded is only supported as a best effort. Each instance's version can be found with `curl http://127.0.0.1:4001/version`.
## All at Once
If downtime is not an issue, the easiest way to upgrade your cluster is to shutdown all of the etcd instances and restart them with the new binary. The current state of the cluster is saved to disk and will be loaded into the cluster when it restarts.
cli.go is simple, fast, and fun package for building command line apps in Go. The goal is to enable developers to write fast and distributable command line applications in an expressive way.
You can view the API docs here:
http://godoc.org/github.com/codegangsta/cli
## Overview
Command line apps are usually so tiny that there is absolutely no reason why your code should *not* be self-documenting. Things like generating help text and parsing command flags/options should not hinder productivity when writing a command line app.
This is where cli.go comes into play. cli.go makes command line programming fun, organized, and expressive!
## Installation
Make sure you have a working Go environment (go 1.1 is *required*). [See the install instructions](http://golang.org/doc/install.html).
To install cli.go, simply run:
```
$ go get github.com/codegangsta/cli
```
Make sure your PATH includes to the `$GOPATH/bin` directory so your commands can be easily used:
```
export PATH=$PATH:$GOPATH/bin
```
## Getting Started
One of the philosophies behind cli.go is that an API should be playful and full of discovery. So a cli.go app can be as little as one line of code in `main()`.
``` go
package main
import (
"os"
"github.com/codegangsta/cli"
)
func main() {
cli.NewApp().Run(os.Args)
}
```
This app will run and show help text, but is not very useful. Let's give an action to execute and some help documentation:
``` go
package main
import (
"os"
"github.com/codegangsta/cli"
)
func main() {
app := cli.NewApp()
app.Name = "boom"
app.Usage = "make an explosive entrance"
app.Action = func(c *cli.Context) {
println("boom! I say!")
}
app.Run(os.Args)
}
```
Running this already gives you a ton of functionality, plus support for things like subcommands and flags, which are covered below.
## Example
Being a programmer can be a lonely job. Thankfully by the power of automation that is not the case! Let's create a greeter app to fend off our demons of loneliness!
``` go
/* greet.go */
package main
import (
"os"
"github.com/codegangsta/cli"
)
func main() {
app := cli.NewApp()
app.Name = "greet"
app.Usage = "fight the loneliness!"
app.Action = func(c *cli.Context) {
println("Hello friend!")
}
app.Run(os.Args)
}
```
Install our command to the `$GOPATH/bin` directory:
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS
--version Shows version information
```
### Arguments
You can lookup arguments by calling the `Args` function on cli.Context.
``` go
...
app.Action = func(c *cli.Context) {
println("Hello", c.Args()[0])
}
...
```
### Flags
Setting and querying flags is simple.
``` go
...
app.Flags = []cli.Flag {
cli.StringFlag{"lang", "english", "language for the greeting"},
}
app.Action = func(c *cli.Context) {
name := "someone"
if len(c.Args()) > 0 {
name = c.Args()[0]
}
if c.String("lang") == "spanish" {
println("Hola", name)
} else {
println("Hello", name)
}
}
...
```
#### Alternate Names
You can set alternate (or short) names for flags by providing a comma-delimited list for the Name. e.g.
``` go
app.Flags = []cli.Flag {
cli.StringFlag{"lang, l", "english", "language for the greeting"},
}
```
That flag can then be set with `--lang spanish` or `-l spanish`. Note that giving two different forms of the same flag in the same command invocation is an error.
### Subcommands
Subcommands can be defined for a more git-like command line app.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
