*: bump to v2.0.4

etcdtcl: fix etcdctl cluster-health ignores SSL settings

.travis.yml

@@ -1,11 +1,12 @@
 language: go
 sudo: false
 go:
-  - 1.3
+  - 1.4
 
 install:
- - go get code.google.com/p/go.tools/cmd/cover
- - go get code.google.com/p/go.tools/cmd/vet
+ - go get golang.org/x/tools/cmd/cover
+ - go get golang.org/x/tools/cmd/vet
+ - go get github.com/barakmich/go-nyet
 
 script:
  - INTEGRATION=y ./test

Documentation/admin_guide.md

@@ -36,7 +36,7 @@ This can protect you from cluster corruption in case of mis-configuration becaus
 
 #### Optimal Cluster Size
 
-The recommended etcd cluster size is 3, 5 or 7, which is decided by the fault tolerance requirement. A 7-member cluster can provide enough fault tolerance in most cases. While larger cluster provides better fault tolerance, its write performance becomes lower since data needs to be replicated to more machines.
+The recommended etcd cluster size is 3, 5 or 7, which is decided by the fault tolerance requirement. A 7-member cluster can provide enough fault tolerance in most cases. While larger cluster provides better fault tolerance the write performance reduces since data needs to be replicated to more machines.
 
 #### Fault Tolerance Table
 
@@ -55,6 +55,10 @@ It is recommended to have an odd number of members in a cluster. Having an odd c
 
 As you can see, adding another member to bring the size of cluster up to an odd size is always worth it. During a network partition, an odd number of members also guarantees that there will almost always be a majority of the cluster that can continue to operate and be the source of truth when the partition ends.
 
+#### Changing Cluster Size
+
+After your cluster is up and running, adding or removing members is done via [runtime reconfiguration](runtime-configuration.md), which allows the cluster to be modified without downtime. The `etcdctl` tool has a `member list`, `member add` and `member remove` commands to complete this process.
+
 ### Member Migration
 
 When there is a scheduled machine maintenance or retirement, you might want to migrate an etcd member to another machine without losing the data and changing the member ID. 

Documentation/allow_legacy_mode.md

@@ -0,0 +1,120 @@
+## Allow-legacy mode
+
+Allow-legacy is a special mode in etcd that contains logic to enable a running etcd cluster to smoothly transition between major versions of etcd. For example, the internal API versions between etcd 0.4 (internal v1) and etcd 2.0 (internal v2) aren't compatible and the cluster needs to be updated all at once to make the switch. To minimize downtime, allow-legacy coordinates with all of the members of the cluster to shutdown, migration of data and restart onto the new version.
+
+Allow-legacy helps users upgrade v0.4 etcd clusters easily, and allows your etcd cluster to have a minimal amount of downtime -- less than 1 minute for clusters storing less than 50 MB.
+
+It supports upgrading from internal v1 to internal v2 now.
+
+### Setup
+
+This mode is enabled if `ETCD_ALLOW_LEGACY_MODE` is set to true, or etcd is running in CoreOS system.
+
+It treats `ETCD_BINARY_DIR` as the directory for etcd binaries, which is organized in this way:
+
+```
+ETCD_BINARY_DIR
+  |
+  -- 1
+  |
+  -- 2
+```
+
+`1` is etcd with internal v1 protocol. You should use etcd v0.4.7 here. `2` is etcd with internal v2 protocol, which is etcd v2.x.
+
+The default value for `ETCD_BINARY_DIR` is `/usr/libexec/etcd/internal_versions/`.
+
+### Upgrading a Cluster
+
+When starting etcd with a v1 data directory and v1 flags, etcd executes the v0.4.7 binary and runs exactly the same as before. To start the migration, follow the steps below:
+
+![Migration Steps](etcd-migration-steps.png)
+
+#### 1. Check the Cluster Health
+
+Before upgrading, you should check the health of the cluster to double check that everything working perfectly. Check the health by running:
+
+```
+$ etcdctl cluster-health
+cluster is healthy
+member 6e3bd23ae5f1eae0 is healthy
+member 924e2e83e93f2560 is healthy
+member a8266ecf031671f3 is healthy
+```
+
+If the cluster and all members are healthy, you can start the upgrading process. If not, check the unhealthy machines and repair them using [admin guide](./admin_guide.md).
+
+#### 2. Trigger the Upgrade
+
+When you're ready, use the `etcdctl upgrade` command to start the upgrade the etcd cluster to 2.0:
+
+```
+# Defaults work on a CoreOS machine running etcd
+$ etcdctl upgrade
+```
+
+```
+# Advanced example specifying a peer url
+$ etcdctl upgrade --old-version=1 --new-version=2 --peer-url=$PEER_URL
+```
+
+`PEER_URL` can be any accessible peer url of the cluster.
+
+Once triggered, all peer-mode members will print out:
+
+```
+detected next internal version 2, exit after 10 seconds.
+```
+
+#### Parallel Coordinated Upgrade
+
+As part of the upgrade, etcd does internal coordination within the cluster for a brief period and then exits. Clusters storing 50 MB should be unavailable for less than 1 minute.
+
+#### Restart etcd Processes
+
+After the etcd processes exit, they need to be restarted. You can do this manually or configure your unit system to do this automatically. On CoreOS, etcd is already configured to start automatically with systemd.
+
+When restarted, the data directory of each member is upgraded, and afterwards etcd v2.0 will be running and servicing requests. The upgrade is now complete!
+
+Standby-mode members are a special case — they will be upgraded into proxy mode (a new feature in etcd 2.0) upon restarting. When the upgrade is triggered, any standbys will exit with the message:
+
+```
+Detect the cluster has been upgraded to internal API v2. Exit now.
+```
+
+Once restarted, standbys run in v2.0 proxy mode, which proxy user requests to the etcd cluster.
+
+#### 3. Check the Cluster Health
+
+After the upgrade process, you can run the health check again to verify the upgrade. If the cluster is unhealthy or there is an unhealthy member, please refer to start [failure recovery](#failure-recovery).
+
+### Downgrade
+
+If the upgrading fails due to disk/network issues, you still can restart the upgrading process manually. However, once you upgrade etcd to internal v2 protocol, you CANNOT downgrade it back to internal v1 protocol. If you want to downgrade etcd in the future, please backup your v1 data dir beforehand.
+
+### Upgrade Process on CoreOS
+
+When running on a CoreOS system, allow-legacy mode is enabled by default and an automatic update will set up everything needed to execute the upgrade. The `etcd.service` on CoreOS is already configured to restart automatically. All you need to do is run `etcdctl upgrade` when you're ready, as described
+
+### Internal Details
+
+etcd v0.4.7 registers versions of available etcd binaries in its local machine into the key space at bootstrap stage. When the upgrade command is executed, etcdctl checks whether each member has internal-version-v2 etcd binary around. If that is true, each member is asked to record the fact that it needs to be upgraded the next time it reboots, and exits after 10 seconds.
+
+Once restarted, etcd v2.0 sees the upgrade flag recorded. It upgrades the data directory, and executes etcd v2.0.
+
+### Failure Recovery
+
+If `etcdctl cluster-health` says that the cluster is unhealthy, the upgrade process fails, which may happen if the network is broken, or the disk cannot work.
+
+The way to recover it is to manually upgrade the whole cluster to v2.0:
+
+- Log into machines that ran v0.4 peer-mode etcd
+- Stop all etcd services
+- Remove the `member` directory under the etcd data-dir
+- Start etcd service using [2.0 flags](configuration.md). An example for this is:
+```
+$ etcd --data-dir=$DATA_DIR --listen-peer-urls http://$LISTEN_PEER_ADDR \
+    --advertise-client-urls http://$ADVERTISE_CLIENT_ADDR \
+    --listen-client-urls http://$LISTEN_CLIENT_ADDR
+```
+- When this is done, v2.0 etcd cluster should work now.

Documentation/api.md

@@ -305,7 +305,7 @@ We get the index is outdated response, since we miss the 1000 events kept in etc
 {"errorCode":401,"message":"The event in requested index is outdated and cleared","cause":"the requested history has been cleared [1003/7]","index":2002}
 ```
 
-To start watch, frist we need to fetch the current state of key `/foo` and the etcdIndex.
+To start watch, first we need to fetch the current state of key `/foo` and the etcdIndex.
 ```sh
 curl 'http://127.0.0.1:2379/v2/keys/foo' -vv
 ```
@@ -913,19 +913,35 @@ curl http://127.0.0.1:2379/v2/stats/leader
 
 ```json
 {
-    "id": "2c7d3e0b8627375b",
-    "leaderInfo": {
-        "leader": "8a69d5f6b7814500",
-        "startTime": "2014-10-24T13:15:51.184719899-07:00",
-        "uptime": "7m17.859616962s"
+    "followers": {
+        "6e3bd23ae5f1eae0": {
+            "counts": {
+                "fail": 0,
+                "success": 745
+            },
+            "latency": {
+                "average": 0.017039507382550306,
+                "current": 0.000138,
+                "maximum": 1.007649,
+                "minimum": 0,
+                "standardDeviation": 0.05289178277920594
+            }
+        },
+        "a8266ecf031671f3": {
+            "counts": {
+                "fail": 0,
+                "success": 735
+            },
+            "latency": {
+                "average": 0.012124141496598642,
+                "current": 0.000559,
+                "maximum": 0.791547,
+                "minimum": 0,
+                "standardDeviation": 0.04187900156583733
+            }
+        }
     },
-    "name": "infra1",
-    "recvAppendRequestCnt": 3949,
-    "recvBandwidthRate": 561.5729321100841,
-    "recvPkgRate": 9.008227977383449,
-    "sendAppendRequestCnt": 0,
-    "startTime": "2014-10-24T13:15:50.070369454-07:00",
-    "state": "StateFollower"
+    "leader": "924e2e83e93f2560"
 }
 ```
 
@@ -979,19 +995,19 @@ curl http://127.0.0.1:2379/v2/stats/self
 
 ```json
 {
-    "id": "eca0338f4ea31566",
+    "id": "924e2e83e93f2560",
     "leaderInfo": {
-        "leader": "8a69d5f6b7814500",
-        "startTime": "2014-10-24T13:15:51.186620747-07:00",
-        "uptime": "10m47.012122091s"
+        "leader": "924e2e83e93f2560",
+        "startTime": "2015-02-09T11:38:30.177534688-08:00",
+        "uptime": "9m33.891343412s"
     },
-    "name": "node3",
-    "recvAppendRequestCnt": 5835,
-    "recvBandwidthRate": 584.1485698657176,
-    "recvPkgRate": 9.17390765395709,
-    "sendAppendRequestCnt": 0,
-    "startTime": "2014-10-24T13:15:50.072007085-07:00",
-    "state": "StateFollower"
+    "name": "infra3",
+    "recvAppendRequestCnt": 0,
+    "sendAppendRequestCnt": 6535,
+    "sendBandwidthRate": 824.1758351191694,
+    "sendPkgRate": 11.111234716807138,
+    "startTime": "2015-02-09T11:38:28.972034204-08:00",
+    "state": "StateLeader"
 }
 ```
 

Documentation/backward_compatibility.md

@@ -27,6 +27,25 @@ https://github.com/coreos/etcd/blob/master/Documentation/configuration.md.
 
 [migrationtooldoc]: https://github.com/coreos/etcd/blob/master/Documentation/0_4_migration_tool.md
 
+#### Key-Value API
+
+##### Read consistency flag
+
+The consistent flag for read operations is removed in etcd 2.0.0. The normal read operations provides the same consistency guarantees with the 0.4.6 read operations with consistent flag set.
+
+The read consistency guarantees are:
+
+The consistent read guarantees the sequential consistency within one client that talks to one etcd server. Read/Write from one client to one etcd member should be observed in order. If one client write a value to a etcd server successfully, it should be able to get the value out of the server immediately. 
+
+Each etcd member will proxy the request to leader and only return the result to user after the result is applied on the local member. Thus after the write succeed, the user is guaranteed to see the value on the member it sent the request to.
+
+Reads do not provide linearizability. If you want linearizabilable read, you need to set quorum option to true.
+
+**Previous behavior**
+
+We added an option for a consistent read in the old version of etcd since etcd 0.x redirects the write request to the leader. When the user get back the result from the leader, the member it sent the request to originally might not apply the write request yet. With the consistent flag set to true, the client will always send read request to the leader. So one client should be able to see its last write when consistent=true is enabled. There is no order guarantees among different clients.
+
+
 #### Standby
 
 etcd 0.4’s standby mode has been deprecated. [Proxy mode][proxymode] is introduced to solve a subset of problems standby was solving.

Documentation/clustering.md

@@ -4,7 +4,9 @@
 
 Starting an etcd cluster statically requires that each member knows another in the cluster. In a number of cases, you might not know the IPs of your cluster members ahead of time. In these cases, you can bootstrap an etcd cluster with the help of a discovery service.
 
-This guide willcover the following mechanisms for bootstrapping an etcd cluster:
+Once an etcd cluster is up and running, adding or removing members is done via [runtime reconfiguration](runtime-configuration.md).
+
+This guide will cover the following mechanisms for bootstrapping an etcd cluster:
 
 * [Static](#static)
 * [etcd Discovery](#etcd-discovery)
@@ -39,22 +41,22 @@ If you are spinning up multiple clusters (or creating and destroying a single cl
 On each machine you would start etcd with these flags:
 
 ```
-$ etcd -name infra0 -initial-advertise-peer-urls https://10.0.1.10:2380 \
-  -listen-peer-urls https://10.0.1.10:2380 \
+$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2380 \
+  -listen-peer-urls http://10.0.1.10:2380 \
   -initial-cluster-token etcd-cluster-1 \
   -initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380 \
   -initial-cluster-state new
 ```
 ```
-$ etcd -name infra1 -initial-advertise-peer-urls https://10.0.1.11:2380 \
-  -listen-peer-urls https://10.0.1.11:2380 \
+$ etcd -name infra1 -initial-advertise-peer-urls http://10.0.1.11:2380 \
+  -listen-peer-urls http://10.0.1.11:2380 \
   -initial-cluster-token etcd-cluster-1 \
   -initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380 \
   -initial-cluster-state new
 ```
 ```
-$ etcd -name infra2 -initial-advertise-peer-urls https://10.0.1.12:2380 \
-  -listen-peer-urls https://10.0.1.12:2380 \
+$ etcd -name infra2 -initial-advertise-peer-urls http://10.0.1.12:2380 \
+  -listen-peer-urls http://10.0.1.12:2380 \
   -initial-cluster-token etcd-cluster-1 \
   -initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380 \
   -initial-cluster-state new

Documentation/configuration.md

@@ -68,9 +68,11 @@ To start etcd automatically using custom settings at startup in Linux, using a [
 + default: "default=http://localhost:2380,default=http://localhost:7001"
 
 ##### -initial-cluster-state
-+ Initial cluster state ("new" or "existing").
++ Initial cluster state ("new" or "existing"). Set to `new` for all members present during initial static or DNS bootstrapping. If this option is set to `existing`, etcd will attempt to join the existing cluster. If the wrong value is set, etcd will attempt to start but fail safely.
 + default: "new"
 
+[static bootstrap]: clustering.md#static
+
 ##### -initial-cluster-token
 + Initial cluster token for the etcd cluster during bootstrap.
 + default: "etcd-cluster"

Documentation/docker_guide.md

@@ -0,0 +1,88 @@
+# Running etcd under Docker
+
+The following guide will show you how to run etcd under Docker using the [static bootstrap process](clustering.md#static).
+
+## Running etcd in standalone mode
+
+In order to expose the etcd API to clients outside of the Docker host you'll need use the host IP address when configuring etcd.
+
+```
+export HostIP="192.168.12.50"
+```
+
+The following `docker run` command will expose the etcd client API over ports 4001 and 2379, and expose the peer port over 2380.
+
+```
+docker run -d -p 4001:4001 -p 2380:2380 -p 2379:2379 --name etcd quay.io/coreos/etcd:v2.0.3 \
+ -name etcd0 \
+ -advertise-client-urls http://${HostIP}:2379,http://${HostIP}:4001 \
+ -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \
+ -initial-advertise-peer-urls http://${HostIP}:2380 \
+ -listen-peer-urls http://0.0.0.0:2380 \
+ -initial-cluster-token etcd-cluster-1 \
+ -initial-cluster etcd0=http://${HostIP}:2380 \
+ -initial-cluster-state new
+```
+
+Configure etcd clients to use the Docker host IP and one of the listening ports from above.
+
+```
+etcdctl -C http://192.168.12.50:2379 member list
+```
+
+```
+etcdctl -C http://192.168.12.50:4001 member list
+```
+
+## Running a 3 node etcd cluster
+
+Using Docker to setup a multi-node cluster is very similar to the standalone mode configuration.
+The main difference being the value used for the `-initial-cluster` flag, which must contain the peer urls for each etcd member in the cluster.
+
+### etcd0
+
+```
+docker run -d -p 4001:4001 -p 2380:2380 -p 2379:2379 --name etcd quay.io/coreos/etcd:v2.0.3 \
+ -name etcd0 \
+ -advertise-client-urls http://192.168.12.50:2379,http://192.168.12.50:4001 \
+ -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \
+ -initial-advertise-peer-urls http://192.168.12.50:2380 \
+ -listen-peer-urls http://0.0.0.0:2380 \
+ -initial-cluster-token etcd-cluster-1 \
+ -initial-cluster etcd0=http://192.168.12.50:2380,etcd1=http://192.168.12.51:2380,etcd2=http://192.168.12.52:2380 \
+ -initial-cluster-state new
+```
+
+### etcd1
+
+```
+docker run -d -p 4001:4001 -p 2380:2380 -p 2379:2379 --name etcd quay.io/coreos/etcd:v2.0.3 \
+ -name etcd1 \
+ -advertise-client-urls http://192.168.12.51:2379,http://192.168.12.51:4001 \
+ -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \
+ -initial-advertise-peer-urls http://192.168.12.51:2380 \
+ -listen-peer-urls http://0.0.0.0:2380 \
+ -initial-cluster-token etcd-cluster-1 \
+ -initial-cluster etcd0=http://192.168.12.50:2380,etcd1=http://192.168.12.51:2380,etcd2=http://192.168.12.52:2380 \
+ -initial-cluster-state new
+```
+
+### etcd2
+
+```
+docker run -d -p 4001:4001 -p 2380:2380 -p 2379:2379 --name etcd quay.io/coreos/etcd:v2.0.3 \
+ -name etcd2 \
+ -advertise-client-urls http://192.168.12.52:2379,http://192.168.12.52:4001 \
+ -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \
+ -initial-advertise-peer-urls http://192.168.12.52:2380 \
+ -listen-peer-urls http://0.0.0.0:2380 \
+ -initial-cluster-token etcd-cluster-1 \
+ -initial-cluster etcd0=http://192.168.12.50:2380,etcd1=http://192.168.12.51:2380,etcd2=http://192.168.12.52:2380 \
+ -initial-cluster-state new
+```
+
+Once the cluster has been bootstrapped etcd clients can be configured with a list of etcd members:
+
+```
+etcdctl -C http://192.168.12.50:2379,http://192.168.12.51:2379,http://192.168.12.52:2379 member list
+```

Documentation/libraries-and-tools.md

@@ -8,6 +8,7 @@
 - [etcd-fs](https://github.com/xetorthio/etcd-fs) - FUSE filesystem for etcd
 - [etcd-browser](https://github.com/henszey/etcd-browser) - A web-based key/value editor for etcd using AngularJS
 - [etcd-lock](https://github.com/datawisesystems/etcd-lock) - A lock implementation for etcd
+- [etcd-console](https://github.com/matishsiao/etcd-console) - A web-base key/value editor for etcd using PHP
 
 **Go libraries**
 

Documentation/other_apis.md

@@ -99,7 +99,7 @@ curl http://10.0.0.10:2379/v2/members/272e204152 -XDELETE
 
 ## Change the peer urls of a member
 
-Change the peer urls of a given mamber. The member ID must be a hex-encoded uint64. Returns 204 with empty content when successful. Returns a string describing the failure condition when unsuccessful.
+Change the peer urls of a given member. The member ID must be a hex-encoded uint64. Returns 204 with empty content when successful. Returns a string describing the failure condition when unsuccessful.
 
 If the POST body is malformed an HTTP 400 will be returned. If the member does not exist in the cluster an HTTP 404 will be returned. If any of the given peerURLs exists in the cluster an HTTP 409 will be returned. If the cluster fails to process the request within timeout an HTTP 500 will be returned, though the request may be processed later.
 

Documentation/proxy.md

@@ -5,28 +5,28 @@ etcd can now run as a transparent proxy. Running etcd as a proxy allows for easi
 etcd currently supports two proxy modes: `readwrite` and `readonly`. The default mode is `readwrite`, which forwards both read and write requests to the etcd cluster. A `readonly` etcd proxy only forwards read requests to the etcd cluster, and returns `HTTP 501` to all write requests. 
 
 ### Using an etcd proxy
-To start etcd in proxy mode, you need to provide three flags: `proxy`, `listen-client-urls`, and `initial-cluster` (or `discovery-url`). 
+To start etcd in proxy mode, you need to provide three flags: `proxy`, `listen-client-urls`, and `initial-cluster` (or `discovery`). 
 
 To start a readwrite proxy, set `-proxy on`; To start a readonly proxy, set `-proxy readonly`.
 
-The proxy will be listening on `listen-client-urls` and forward requests to the etcd cluster discovered from in `initial-cluster` or `discovery url`. 
+The proxy will be listening on `listen-client-urls` and forward requests to the etcd cluster discovered from in `initial-cluster` or `discovery` url. 
 
 #### Start an etcd proxy with a static configuration
 To start a proxy that will connect to a statically defined etcd cluster, specify the `initial-cluster` flag:
 ```
-etcd -proxy on -client-listen-urls 127.0.0.1:8080 -initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380
+etcd -proxy on -listen-client-urls 127.0.0.1:8080 -initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380
 ```
 
 #### Start an etcd proxy with the discovery service
-If you bootstrap an etcd cluster using the [discovery service][discovery-service], you can also start the proxy with the same `discovery-url`. 
+If you bootstrap an etcd cluster using the [discovery service][discovery-service], you can also start the proxy with the same `discovery`. 
 
-To start a proxy using the discovery service, specify the `discovery-url` flag. The proxy will wait until the etcd cluster defined at the `discovery-url` finishes bootstrapping, and then start to forward the requests. 
+To start a proxy using the discovery service, specify the `discovery` flag. The proxy will wait until the etcd cluster defined at the `discovery` url finishes bootstrapping, and then start to forward the requests. 
 
 ```
-etcd -proxy on -client-listen-urls 127.0.0.1:8080  -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
+etcd -proxy on -listen-client-urls 127.0.0.1:8080  -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
 ```
 
 #### Fallback to proxy mode with discovery service
-If you bootstrap a etcd cluster using [discovery service][discovery-service] with more than the expected number of etcd members, the extra etcd processes will fall back to being `readwrite` proxies by default. They will forward the requests to the cluster as described above. For example, if you create a discovery url with `size=5`, and start ten etcd processes using that same discovery URL, the result will be a cluster with five etcd members and five proxies. Note that this behaviour can be disabled with the `proxy-fallback` flag.
+If you bootstrap a etcd cluster using [discovery service][discovery-service] with more than the expected number of etcd members, the extra etcd processes will fall back to being `readwrite` proxies by default. They will forward the requests to the cluster as described above. For example, if you create a discovery url with `size=5`, and start ten etcd processes using that same discovery url, the result will be a cluster with five etcd members and five proxies. Note that this behaviour can be disabled with the `proxy-fallback` flag.
 
 [discovery-service]: https://github.com/coreos/etcd/blob/master/Documentation/clustering.md#discovery

Documentation/rfc/api_security.md

@@ -0,0 +1,470 @@
+# v2 Auth and Security
+
+## etcd Resources 
+There are three types of resources in etcd
+
+1. user resources: users and roles in the user store
+2. key-value resources: key-value pairs in the key-value store
+3. settings resources: security settings, auth settings, and dynamic etcd cluster settings (election/heartbeat)
+
+### User Resources 
+
+#### Users
+A user is an identity to be authenticated. Each user can have multiple roles. The user has a capability on the resource if one of the roles has that capability.
+
+The special static `root` user has a ROOT role. (Caps for visual aid throughout)
+
+#### Role
+Each role has exact one associated Permission List. An permission list exists for each permission on key-value resources. A role with `manage` permission of a key-value resource can grant/revoke capability of that key-value to other roles.
+
+The special static ROOT role has a full permissions on all key-value resources, the permission to manage user resources and settings resources. Only the ROOT role has the permission to manage user resources and modify settings resources.
+
+#### Permissions
+
+There are two types of permissions, `read` and `write`. All management stems from the ROOT user.
+
+A Permission List is a list of allowed patterns for that particular permission (read or write). Only ALLOW prefixes (incidentally, this is what Amazon S3 does). DENY becomes more complicated and is TBD.
+
+### Key-Value Resources
+A key-value resource is a key-value pairs in the store. Given a list of matching patterns, permission for any given key in a request is granted if any of the patterns in the list match.
+
+The glob match rules are as follows:
+
+* `*` and `\` are special characters, representing "greedy match" and "escape" respectively.
+  * As a corrolary, `\*` and `\\` are the corresponding literal matches. 
+* All other bytes match exactly their bytes, starting always from the *first byte*. (For regex fans, `re.match` in Python) 
+* Examples:
+  * `/foo` matches only the single key/directory of `/foo`
+  * `/foo*` matches the prefix `/foo`, and all subdirectories/keys
+  * `/foo/*/bar` matches the keys bar in any (recursive) subdirectory of `/foo`.
+
+### Settings Resources
+
+Specific settings for the cluster as a whole. This can include adding and removing cluster members, enabling or disabling security, replacing certificates, and any other dynamic configuration by the administrator.
+
+## v2 Auth
+
+### Basic Auth
+We only support [Basic Auth](http://en.wikipedia.org/wiki/Basic_access_authentication) for the first version. Client needs to attach the basic auth to the HTTP Authorization Header. 
+
+### Authorization field for operations
+Added to requests to /v2/keys, /v2/security
+Add code 403 Forbidden to the set of responses from the v2 API
+Authorization: Basic {encoded string}
+
+### Future Work
+Other types of auth can be considered for the future (eg, signed certs, public keys) but the `Authorization:` header allows for other such types
+
+### Things out of Scope for etcd Permissions
+
+* Pluggable AUTH backends like LDAP (other Authorization tokens generated by LDAP et al may be a possiblity)
+* Very fine-grained access controls (eg: users modifying keys outside work hours)
+
+
+
+## API endpoints
+
+An Error JSON corresponds to:
+{
+  "name": "ErrErrorName",
+  "description" : "The longer helpful description of the error."
+}
+
+#### Users
+
+The User JSON object is formed as follows:
+
+```
+{
+  "user": "userName"
+  "password": "password"
+  "roles": [
+    "role1",
+    "role2"
+  ],
+  "grant": [],
+  "revoke": [],
+  "lastModified": "2006-01-02Z04:05:07"
+}
+```
+
+Password is only passed when necessary. Last Modified is set by the server and ignored in all client posts.
+
+**Get a list of users**
+
+GET/HEAD  /v2/security/user
+
+    Sent Headers:
+        Authorization: Basic <BasicAuthString>
+    Possible Status Codes:
+        200 OK
+        403 Forbidden
+    200 Headers:
+        ETag: "<hash of list of users>"
+        Content-type: application/json
+    200 Body:
+        {
+          "users": ["alice", "bob", "eve"]
+        }
+
+**Get User Details**
+
+GET/HEAD  /v2/security/users/alice
+
+    Sent Headers:
+        Authorization: Basic <BasicAuthString>
+    Possible Status Codes:
+        200 OK
+        403 Forbidden
+        404 Not Found
+    200 Headers:
+        ETag: "users/alice:<lastModified>"
+        Content-type: application/json
+    200 Body:
+        {
+          "user" : "alice"
+          "roles" : ["fleet", "etcd"]
+          "lastModified": "2015-02-05Z18:00:00"
+        }
+
+**Create A User**
+
+A user can be created with initial roles, if filled in. However, no roles are required; only the username and password fields
+
+PUT  /v2/security/users/charlie
+
+    Sent Headers:
+        Authorization: Basic <BasicAuthString>
+    Put Body:
+        JSON struct, above, matching the appropriate name and with starting roles.
+    Possible Status Codes:
+        200 OK
+        403 Forbidden
+        409 Conflict (if exists)
+    200 Headers:
+        ETag: "users/charlie:<tzNow>"
+    200 Body: (empty)
+
+**Remove A User**
+
+DELETE  /v2/security/users/charlie
+
+    Sent Headers:
+        Authorization: Basic <BasicAuthString>
+    Possible Status Codes:
+        200 OK
+        403 Forbidden
+        404 Not Found
+    200 Headers:
+    200 Body: (empty)
+
+**Grant a Role(s) to a User**
+
+PUT  /v2/security/users/charlie/grant
+
+    Sent Headers:
+        Authorization: Basic <BasicAuthString>
+    Put Body:
+        { "grantRoles" : ["fleet", "etcd"], (extra JSON data for checking OK) }
+    Possible Status Codes:
+        200 OK
+        403 Forbidden
+        404 Not Found
+        409 Conflict
+    200 Headers:
+        ETag: "users/charlie:<tzNow>"
+    200 Body: 
+        JSON user struct, updated. "roles" now contains the grants, and "grantRoles" is empty. If there is an error in the set of roles to be added, for example, a non-existent role, then 409 is returned, with an error JSON stating why.
+
+**Revoke a Role(s) from a User**
+
+PUT  /v2/security/users/charlie/revoke
+
+    Sent Headers:
+        Authorization: Basic <BasicAuthString>
+    Put Body:
+        { "revokeRoles" : ["fleet"], (extra JSON data for checking OK) }
+    Possible Status Codes:
+        200 OK
+        403 Forbidden
+        404 Not Found
+        409 Conflict
+    200 Headers:
+        ETag: "users/charlie:<tzNow>"
+    200 Body: 
+        JSON user struct, updated. "roles" now doesn't contain the roles, and "revokeRoles" is empty. If there is an error in the set of roles to be removed, for example, a non-existent role, then 409 is returned, with an error JSON stating why.
+
+**Change password**
+
+PUT  /v2/security/users/charlie/password
+
+    Sent Headers:
+        Authorization: Basic <BasicAuthString>
+    Put Body:
+        {"user": "charlie", "password": "newCharliePassword"}
+    Possible Status Codes:
+        200 OK
+        403 Forbidden
+        404 Not Found
+    200 Headers:
+        ETag: "users/charlie:<tzNow>"
+    200 Body:
+        JSON user struct, updated
+
+#### Roles
+
+A full role structure may look like this. A Permission List structure is used for the "permissions", "grant", and "revoke" keys.
+```
+{
+  "role" : "fleet",
+    "permissions" : {
+      "kv" {
+        "read" : [ "/fleet/" ],
+        "write": [ "/fleet/" ],
+      }
+    }
+    "grant" : {"kv": {...}},
+    "revoke": {"kv": {...}},
+    "members" : ["alice", "bob"],
+    "lastModified": "2015-02-05Z18:00:00"
+}
+```
+
+**Get a list of Roles**
+
+GET/HEAD  /v2/security/roles
+
+    Sent Headers:
+        Authorization: Basic <BasicAuthString>
+    Possible Status Codes:
+        200 OK
+        403 Forbidden
+    200 Headers:
+        ETag: "<hash of list of roles>"
+        Content-type: application/json
+    200 Body:
+        {
+          "roles": ["fleet", "etcd", "quay"]
+        }
+
+**Get Role Details**
+
+GET/HEAD  /v2/security/roles/fleet
+
+    Sent Headers:
+        Authorization: Basic <BasicAuthString>
+    Possible Status Codes:
+        200 OK
+        403 Forbidden
+        404 Not Found
+    200 Headers:
+        ETag: "roles/fleet:<lastModified>"
+        Content-type: application/json
+    200 Body:
+        {
+          "role" : "fleet",
+          "read": {
+            "prefixesAllowed": ["/fleet/"],
+          },
+          "write": {
+            "prefixesAllowed": ["/fleet/"],
+          },
+          "members" : ["alice", "bob"] // Reverse map optional?
+          "lastModified": "2015-02-05Z18:00:00"
+        }
+
+**Create A Role**
+
+PUT  /v2/security/roles/rocket
+
+    Sent Headers:
+        Authorization: Basic <BasicAuthString>
+    Put Body:
+        Initial desired JSON state, complete with prefixes and 
+    Possible Status Codes:
+        201 Created
+        403 Forbidden
+        404 Not Found
+        409 Conflict (if exists)
+    200 Headers:
+        ETag: "roles/rocket:<tzNow>"
+    200 Body: 
+        JSON state of the role
+
+**Remove A Role**
+
+DELETE  /v2/security/roles/rocket
+
+    Sent Headers:
+        Authorization: Basic <BasicAuthString>
+    Possible Status Codes:
+        200 OK
+        403 Forbidden
+        404 Not Found
+    200 Headers:
+    200 Body: (empty)
+
+**Update a Role’s Permission List for {read,write}ing**
+
+PUT  /v2/security/roles/rocket/update
+
+    Sent Headers:
+        Authorization: Basic <BasicAuthString>
+    Put Body:
+        {
+          "role" : "rocket",
+          "grant": {
+            "kv": {
+              "read" : [ "/rocket/"]
+            }
+          }, 
+          "revoke": {
+            "kv": {
+              "read" : [ "/fleet/"]
+            }
+          } 
+        }
+    Possible Status Codes:
+        200 OK
+        403 Forbidden
+        404 Not Found
+    200 Headers:
+        ETag: "roles/rocket:<tzNow>"
+    200 Body:
+        JSON state of the role, with change containing empty lists and the deltas applied appropriately.
+
+
+#### TBD Management modification
+
+
+## Example Workflow
+
+Let's walk through an example to show two tenants (applications, in our case) using etcd permissions.
+
+### Enable security
+
+//TODO(barakmich): Maybe this is dynamic? I don't like the idea of rebooting when we don't have to.
+
+#### Default ROOT 
+
+etcd always has a ROOT when started with security enabled. The default username is `root`, and the password is `root`. 
+
+// TODO(barakmich): if the enabling is dynamic, perhaps that'd be a good time to set a password? Thus obviating the next section.
+
+
+### Change root's password
+
+```
+PUT  /v2/security/users/root/password
+    Headers:
+        Authorization: Basic <root:root>
+    Put Body:
+        {"user" : "root", "password": "betterRootPW!"}
+```
+
+//TODO(barakmich): How do you recover the root password? *This* may require a flag and a restart. `--disable-permissions`
+
+### Create Roles for the Applications
+
+Create the rocket role fully specified:
+
+```
+PUT /v2/security/roles/rocket
+    Headers:
+        Authorization: Basic <root:betterRootPW!>
+    Body: 
+        {
+          "role" : "rocket",
+          "permissions" : {
+            "kv": {
+              "read": [
+                "/rocket/"
+              ],
+              "write": [
+                "/rocket/"
+              ]
+            }
+          }
+        }
+```
+
+But let's make fleet just a basic role for now:
+
+```
+PUT /v2/security/roles/fleet
+    Headers:
+      Authorization: Basic <root:betterRootPW!>
+    Body: 
+        {
+            "role" : "fleet",
+        }
+```
+
+### Optional: Add some permissions to the roles
+
+Well, we finally figured out where we want fleet to live. Let's fix it.
+(Note that we avoided this in the rocket case. So this step is optional.)
+
+
+```
+PUT /v2/security/roles/fleet/update
+    Headers:
+        Authorization: Basic <root:betterRootPW!>
+    Put Body:
+        {
+          "role" : "fleet",
+          "grant" : {
+            "kv" : {
+              "read": [
+                "/fleet/"
+              ]
+            }
+          }
+        }
+```
+
+### Create Users
+
+Same as before, let's use rocket all at once and fleet separately
+
+```
+PUT /v2/security/users/rocketuser
+    Headers:
+        Authorization: Basic <root:betterRootPW!>
+    Body:
+        {"user" : "rocketuser", "password" : "rocketpw", "roles" : ["rocket"]}
+```
+
+```
+PUT /v2/security/users/fleetuser
+    Headers:
+        Authorization: Basic <root:betterRootPW!>
+    Body:
+        {"user" : "fleetuser", "password" : "fleetpw"}
+```
+
+### Optional: Grant Roles to Users
+
+Likewise, let's explicitly grant fleetuser access.
+
+```
+PUT /v2/security/users/fleetuser/grant
+    Headers:
+        Authorization: Basic <root:betterRootPW!>
+    Body:
+      {"user": "fleetuser", "grant": ["fleet"]}
+```
+
+#### Start to use fleetuser and rocketuser
+
+
+For example:
+
+```
+PUT /v2/keys/rocket/RocketData
+    Headers:
+        Authorization: Basic <rocketuser:rocketpw>
+```
+
+Reads and writes outside the prefixes granted will fail with a 403 Forbidden.
+

Documentation/runtime-configuration.md

@@ -8,40 +8,40 @@ Reconfiguration requests can only be processed when the the majority of the clus
 
 ## Reconfiguration Use Cases
 
-Let us walk through the four use cases for re-configuring a cluster: replacing a member, increasing or decreasing cluster size, and restarting a cluster from a majority failure.
+Let us walk through some common reasons for reconfiguring a cluster. Most of these just involve combinations of adding or removing a member, which are explained below under [Cluster Reconfiguration Operations](#cluster-reconfiguration-operations).
 
-### Replace a Non-recoverable Member
+### Cycle or Upgrade Multiple Machines
 
-The most common use case of cluster reconfiguration is to replace a member because of a permanent failure of the existing member: for example, hardware failure or data directory corruption.
-It is important to replace failed members as soon as the failure is detected.
-If etcd falls below a simple majority of members it can no longer accept writes: e.g. in a 3 member cluster the loss of two members will cause writes to fail and the cluster to stop operating.
+If you need to move multiple members of your cluster due to planned maintenance (hardware upgrades, network downtime, etc.), it is recommended to modify members one at a time.
 
-If you want to migrate a running member to another machine, please refer [member migration section][member migration].
+It is safe to remove the leader, however there is a brief period of downtime while the election process takes place. If your cluster holds more than 50MB, it is recommended to [migrate the member's data directory][member migration].
 
-[member migration]: https://github.com/coreos/etcd/blob/master/Documentation/admin_guide.md#member-migration
+[member migration]: admin_guide.md#member-migration
 
-### Increase Cluster Size
+### Change the Cluster Size
 
-To make your cluster more resilient to machine failure you can increase the size of the cluster.
-For example, if the cluster consists of three machines, it can tolerate one failure.
-If we increase the cluster size to five, it can tolerate two machine failures.
+Increasing the cluster size can enhance [failure tolerance][fault tolerance table] and provide better read performance. Since clients can read from any member, increasing the number of members increases the overall read throughput.
 
-Increasing the cluster size can also provide better read performance.
-When a client accesses etcd, the normal read gets the data from the local copy of each member (members always shares the same view of the cluster at the same index, which is guaranteed by the sequential consistency of etcd).
-Since clients can read from any member, increasing the number of members thus increases overall read throughput.
+Decreasing the cluster size can improve the write performance of a cluster, with a trade-off of decreased resilience. Writes into the cluster are replicated to a majority of members of the cluster before considered committed. Decreasing the cluster size lowers the majority, and each write is committed more quickly.
 
-### Decrease Cluster Size
+[fault tolerance table]: admin_guide.md#fault-tolerance-table
 
-To improve the write performance of a cluster, you might want to trade off resilience by removing members.
-etcd replicates the data to the majority of members of the cluster before committing the write.
-Decreasing the cluster size means the etcd cluster has to do less work for each write, thus increasing the write performance.
+### Replace A Failed Machine
+
+If a machine fails due to hardware failure, data directory corruption, or some other fatal situation, it should be replaced as soon as possible. Machines that have failed but haven't been removed adversely affect your quorum and reduce the tolerance for an additional failure.
+
+To replace the machine, follow the instructions for [removing the member][remove member] from the cluster, and then [add a new member][add member] in its place. If your cluster holds more than 50MB, it is recommended to [migrate the failed member's data directory][member migration] if you can still access it.
+
+[remove member]: #remove-a-member
+[add member]: #add-a-new-member
 
 ### Restart Cluster from Majority Failure
 
 If the majority of your cluster is lost, then you need to take manual action in order to recover safely.
-The basic steps in the recovery process include creating a new cluster using the old data, forcing a single member to act as the leader, and finally using runtime configuration to add members to this new cluster.
+The basic steps in the recovery process include [creating a new cluster using the old data][disaster recovery], forcing a single member to act as the leader, and finally using runtime configuration to [add new members][add member] to this new cluster one at a time.
 
-TODO: https://github.com/coreos/etcd/issues/1242
+[add member]: #add-a-new-member
+[disaster recovery]: admin_guide.md#disaster-recovery
 
 ## Cluster Reconfiguration Operations
 
@@ -61,7 +61,7 @@ If you want to use the member API directly you can find the documentation [here]
 
 ### Remove a Member
 
-First, we need to find the target member:
+First, we need to find the target member's ID. You can list all members with `etcdctl`:
 
 ```
 $ etcdctl member list
@@ -84,27 +84,27 @@ The target member will stop itself at this point and print out the removal in th
 etcd: this member has been permanently removed from the cluster. Exiting.
 ```
 
-Removal of the leader is safe, but the cluster will be out of progress for a period of election timeout because it needs to elect the new leader.
+It is safe to remove the leader, however the cluster will be inactive while a new leader is elected. This duration is normally the period of election timeout plus the voting process.
 
-### Add a Member
+### Add a New Member
 
 Adding a member is a two step process:
 
  * Add the new member to the cluster via the [members API](https://github.com/coreos/etcd/blob/master/Documentation/other_apis.md#post-v2members) or the `etcdctl member add` command.
- * Start the member with the correct configuration.
+ * Start the new member with the new cluster configuration, including a list of the updated members (existing members + the new member).
 
-Using `etcdctl` let's add the new member to the cluster:
+Using `etcdctl` let's add the new member to the cluster by specifing its [name](configuration.md#-name) and [advertised peer URLs](configuration.md#-initial-advertise-peer-urls):
 
 ```
 $ etcdctl member add infra3 http://10.0.1.13:2380
 added member 9bf1b35fc7761a23 to cluster
+
 ETCD_NAME="infra3"
 ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380,infra3=http://10.0.1.13:2380"
 ETCD_INITIAL_CLUSTER_STATE=existing
 ```
 
-> Notice that infra3 was added to the cluster using its advertised peer URL.
-
+`etcdctl` has informed the cluster about the new member and printed out the environment variables needed to successfully start it.
 Now start the new etcd process with the relevant flags for the new member:
 
 ```
@@ -116,8 +116,8 @@ $ etcd -listen-client-urls http://10.0.1.13:2379 -advertise-client-urls http://1
 
 The new member will run as a part of the cluster and immediately begin catching up with the rest of the cluster.
 
-If you are adding multiple members the best practice is to configure the new member, then start the process, then configure the next, and so on.
-A common case is increasing a cluster from 1 to 3: if you add one member to a 1-node cluster, the cluster cannot make progress before the new member starts because it needs two members as majority to agree on the consensus.
+If you are adding multiple members the best practice is to configure a single member at a time and verify it starts correctly before adding more new members.
+If you add a new member to a 1-node cluster, the cluster cannot make progress before the new member starts because it needs two members as majority to agree on the consensus. You will only see this behavior between the time `etcdctl member add` informs the cluster about the new member and the new member successfully establishing a connection to the existing one.
 
 #### Error Cases
 

Documentation/tuning.md

@@ -11,11 +11,11 @@ The underlying distributed consensus protocol relies on two separate time parame
 The first parameter is called the *Heartbeat Interval*.
 This is the frequency with which the leader will notify followers that it is still the leader.
 etcd batches commands together for higher throughput so this heartbeat interval is also a delay for how long it takes for commands to be committed.
-By default, etcd uses a `50ms` heartbeat interval.
+By default, etcd uses a `100ms` heartbeat interval.
 
 The second parameter is the *Election Timeout*.
 This timeout is how long a follower node will go without hearing a heartbeat before attempting to become leader itself.
-By default, etcd uses a `200ms` election timeout.
+By default, etcd uses a `1000ms` election timeout.
 
 Adjusting these values is a trade off.
 Lowering the heartbeat interval will cause individual commands to be committed faster but it will lower the overall throughput of etcd.

Godeps/Godeps.json

@@ -1,6 +1,6 @@
 {
 	"ImportPath": "github.com/coreos/etcd",
-	"GoVersion": "go1.3.1",
+	"GoVersion": "go1.4.1",
 	"Packages": [
 		"./..."
 	],

Procfile

@@ -2,4 +2,4 @@
 etcd1: bin/etcd -name infra1 -listen-client-urls http://localhost:4001 -advertise-client-urls http://localhost:4001 -listen-peer-urls http://localhost:7001 -initial-advertise-peer-urls http://localhost:7001 -initial-cluster-token etcd-cluster-1 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003' -initial-cluster-state new
 etcd2: bin/etcd -name infra2 -listen-client-urls http://localhost:4002 -advertise-client-urls http://localhost:4002 -listen-peer-urls http://localhost:7002 -initial-advertise-peer-urls http://localhost:7002 -initial-cluster-token etcd-cluster-1 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003' -initial-cluster-state new
 etcd3: bin/etcd -name infra3 -listen-client-urls http://localhost:4003 -advertise-client-urls http://localhost:4003 -listen-peer-urls http://localhost:7003 -initial-advertise-peer-urls http://localhost:7003 -initial-cluster-token etcd-cluster-1 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003' -initial-cluster-state new
-proxy: bin/etcd -proxy=on -bind-addr 127.0.0.1:8080 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003'
+proxy: bin/etcd -name proxy1 -proxy=on -bind-addr 127.0.0.1:8080 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003'

README.md

@@ -5,8 +5,7 @@
 
 ![etcd Logo](logos/etcd-horizontal-color.png)
 
-A highly-available key value store for shared configuration and service discovery.
-etcd is inspired by [Apache ZooKeeper][zookeeper] and [doozer][doozer], with a focus on being:
+etcd is a distributed, consistent key value store for shared configuration and service discovery with a focus on being:
 
 * *Simple*: curl'able user facing API (HTTP+JSON)
 * *Secure*: optional SSL client cert authentication

build

@@ -12,7 +12,7 @@ ln -s ${PWD} $GOPATH/src/${REPO_PATH}
 eval $(go env)
 
 # Static compilation is useful when etcd is run in a container
-CGO_ENABLED=0 go build -a -ldflags '-s' -o bin/etcd ${REPO_PATH}
-CGO_ENABLED=0 go build -a -ldflags '-s' -o bin/etcdctl ${REPO_PATH}/etcdctl
+CGO_ENABLED=0 go build -a -installsuffix cgo -ldflags '-s' -o bin/etcd ${REPO_PATH}
+CGO_ENABLED=0 go build -a -installsuffix cgo -ldflags '-s' -o bin/etcdctl ${REPO_PATH}/etcdctl
 go build -o bin/etcd-migrate ${REPO_PATH}/tools/etcd-migrate
 go build -o bin/etcd-dump-logs ${REPO_PATH}/tools/etcd-dump-logs

client/keys_test.go

@@ -89,7 +89,7 @@ func TestV2KeysURLHelper(t *testing.T) {
 
 func TestGetAction(t *testing.T) {
 	ep := url.URL{Scheme: "http", Host: "example.com/v2/keys"}
-	wantURL := &url.URL{
+	baseWantURL := &url.URL{
 		Scheme: "http",
 		Host:   "example.com",
 		Path:   "/v2/keys/foo/bar",
@@ -117,7 +117,7 @@ func TestGetAction(t *testing.T) {
 		}
 		got := *f.HTTPRequest(ep)
 
-		wantURL := wantURL
+		wantURL := baseWantURL
 		wantURL.RawQuery = tt.wantQuery
 
 		err := assertResponse(got, wantURL, wantHeader, nil)
@@ -129,7 +129,7 @@ func TestGetAction(t *testing.T) {
 
 func TestWaitAction(t *testing.T) {
 	ep := url.URL{Scheme: "http", Host: "example.com/v2/keys"}
-	wantURL := &url.URL{
+	baseWantURL := &url.URL{
 		Scheme: "http",
 		Host:   "example.com",
 		Path:   "/v2/keys/foo/bar",
@@ -166,7 +166,7 @@ func TestWaitAction(t *testing.T) {
 		}
 		got := *f.HTTPRequest(ep)
 
-		wantURL := wantURL
+		wantURL := baseWantURL
 		wantURL.RawQuery = tt.wantQuery
 
 		err := assertResponse(got, wantURL, wantHeader, nil)

discovery/discovery_test.go

@@ -193,14 +193,14 @@ func TestCheckCluster(t *testing.T) {
 			})
 		}
 		c := &clientWithResp{rs: rs}
-		d := discovery{cluster: cluster, id: 1, c: c}
+		dBase := discovery{cluster: cluster, id: 1, c: c}
 
 		cRetry := &clientWithRetry{failTimes: 3}
 		cRetry.rs = rs
 		fc := clockwork.NewFakeClock()
 		dRetry := discovery{cluster: cluster, id: 1, c: cRetry, clock: fc}
 
-		for _, d := range []discovery{d, dRetry} {
+		for _, d := range []discovery{dBase, dRetry} {
 			go func() {
 				for i := uint(1); i <= maxRetryInTest; i++ {
 					fc.BlockUntil(1)
@@ -263,7 +263,7 @@ func TestWaitNodes(t *testing.T) {
 	for i, tt := range tests {
 		// Basic case
 		c := &clientWithResp{nil, &watcherWithResp{tt.rs}}
-		d := &discovery{cluster: "1000", c: c}
+		dBase := &discovery{cluster: "1000", c: c}
 
 		// Retry case
 		retryScanResp := make([]*client.Response, 0)
@@ -291,7 +291,7 @@ func TestWaitNodes(t *testing.T) {
 			clock:   fc,
 		}
 
-		for _, d := range []*discovery{d, dRetry} {
+		for _, d := range []*discovery{dBase, dRetry} {
 			go func() {
 				for i := uint(1); i <= maxRetryInTest; i++ {
 					fc.BlockUntil(1)

etcdctl/command/backup_command.go

@@ -15,6 +15,7 @@
 package command
 
 import (
+	"fmt"
 	"log"
 	"os"
 	"path"
@@ -71,7 +72,12 @@ func handleBackup(c *cli.Context) {
 	}
 	defer w.Close()
 	wmetadata, state, ents, err := w.ReadAll()
-	if err != nil {
+	switch err {
+	case nil:
+	case wal.ErrSnapshotNotFound:
+		fmt.Printf("Failed to find the match snapshot record %+v in wal %v.", walsnap, srcWAL)
+		fmt.Printf("etcdctl will add it back. Start auto fixing...")
+	default:
 		log.Fatal(err)
 	}
 	var metadata etcdserverpb.Metadata
@@ -88,4 +94,7 @@ func handleBackup(c *cli.Context) {
 	if err := neww.Save(state, ents); err != nil {
 		log.Fatal(err)
 	}
+	if err := neww.SaveSnapshot(walsnap); err != nil {
+		log.Fatal(err)
+	}
 }

etcdctl/command/cluster_health.go

@@ -0,0 +1,141 @@
+package command
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"os"
+	"sort"
+	"strings"
+	"time"
+
+	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
+	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/go-etcd/etcd"
+	"github.com/coreos/etcd/etcdserver/stats"
+)
+
+func NewClusterHealthCommand() cli.Command {
+	return cli.Command{
+		Name:   "cluster-health",
+		Usage:  "check the health of the etcd cluster",
+		Flags:  []cli.Flag{},
+		Action: handleClusterHealth,
+	}
+}
+
+func handleClusterHealth(c *cli.Context) {
+	endpoints, err := getEndpoints(c)
+	if err != nil {
+		handleError(ErrorFromEtcd, err)
+	}
+	tr, err := getTransport(c)
+	if err != nil {
+		handleError(ErrorFromEtcd, err)
+	}
+
+	client := etcd.NewClient(endpoints)
+	client.SetTransport(tr)
+
+	if c.GlobalBool("debug") {
+		go dumpCURL(client)
+	}
+
+	if ok := client.SyncCluster(); !ok {
+		handleError(FailedToConnectToHost, errors.New("cannot sync with the cluster using endpoints "+strings.Join(endpoints, ", ")))
+	}
+
+	// do we have a leader?
+	ep, ls0, err := getLeaderStats(tr, client.GetCluster())
+	if err != nil {
+		fmt.Println("cluster is unhealthy")
+		os.Exit(1)
+	}
+
+	// is raft stable and making progress?
+	client = etcd.NewClient([]string{ep})
+	client.SetTransport(tr)
+	resp, err := client.Get("/", false, false)
+	if err != nil {
+		fmt.Println("cluster is unhealthy")
+		os.Exit(1)
+	}
+	rt0, ri0 := resp.RaftTerm, resp.RaftIndex
+	time.Sleep(time.Second)
+
+	resp, err = client.Get("/", false, false)
+	if err != nil {
+		fmt.Println("cluster is unhealthy")
+		os.Exit(1)
+	}
+	rt1, ri1 := resp.RaftTerm, resp.RaftIndex
+
+	if rt0 != rt1 {
+		fmt.Println("cluster is unhealthy")
+		os.Exit(1)
+	}
+
+	if ri1 == ri0 {
+		fmt.Println("cluster is unhealthy")
+		os.Exit(1)
+	}
+
+	// are all the members makeing progress?
+	_, ls1, err := getLeaderStats(tr, []string{ep})
+	if err != nil {
+		fmt.Println("cluster is unhealthy")
+		os.Exit(1)
+	}
+
+	fmt.Println("cluster is healthy")
+	// self is healthy
+	var prints []string
+
+	prints = append(prints, fmt.Sprintf("member %s is healthy\n", ls1.Leader))
+	for name, fs0 := range ls0.Followers {
+		fs1, ok := ls1.Followers[name]
+		if !ok {
+			fmt.Println("Cluster configuration changed during health checking. Please retry.")
+			os.Exit(1)
+		}
+		if fs1.Counts.Success <= fs0.Counts.Success {
+			prints = append(prints, fmt.Sprintf("member %s is unhealthy\n", name))
+		} else {
+			prints = append(prints, fmt.Sprintf("member %s is healthy\n", name))
+		}
+	}
+
+	sort.Strings(prints)
+	for _, p := range prints {
+		fmt.Print(p)
+	}
+	os.Exit(0)
+}
+
+func getLeaderStats(tr *http.Transport, endpoints []string) (string, *stats.LeaderStats, error) {
+	// go-etcd does not support cluster stats, use http client for now
+	// TODO: use new etcd client with new member/stats endpoint
+	httpclient := http.Client{
+		Transport: tr,
+	}
+
+	for _, ep := range endpoints {
+		resp, err := httpclient.Get(ep + "/v2/stats/leader")
+		if err != nil {
+			continue
+		}
+		defer resp.Body.Close()
+		if resp.StatusCode != http.StatusOK {
+			continue
+		}
+
+		ls := &stats.LeaderStats{}
+		d := json.NewDecoder(resp.Body)
+		err = d.Decode(ls)
+		if err != nil {
+			continue
+		}
+		return ep, ls, nil
+	}
+	return "", nil, errors.New("no leader")
+}

etcdctl/command/member_commands.go

@@ -134,10 +134,10 @@ func actionMemberAdd(c *cli.Context) {
 	}
 
 	conf := []string{}
-	for _, m := range members {
-		for _, u := range m.PeerURLs {
-			n := m.Name
-			if m.ID == newID {
+	for _, memb := range members {
+		for _, u := range memb.PeerURLs {
+			n := memb.Name
+			if memb.ID == newID {
 				n = newName
 			}
 			conf = append(conf, fmt.Sprintf("%s=%s", n, u))
@@ -156,16 +156,42 @@ func actionMemberRemove(c *cli.Context) {
 		fmt.Fprintln(os.Stderr, "Provide a single member ID")
 		os.Exit(1)
 	}
+	removalID := args[0]
 
 	mAPI := mustNewMembersAPI(c)
-	mID := args[0]
-	ctx, cancel := context.WithTimeout(context.Background(), client.DefaultRequestTimeout)
-	err := mAPI.Remove(ctx, mID)
-	cancel()
+	// Get the list of members.
+	listctx, listCancel := context.WithTimeout(context.Background(), client.DefaultRequestTimeout)
+	members, err := mAPI.List(listctx)
+	listCancel()
 	if err != nil {
-		fmt.Fprintln(os.Stderr, err.Error())
+		fmt.Fprintln(os.Stderr, "Error while verifying ID against known members:", err.Error())
+		os.Exit(1)
+	}
+	// Sanity check the input.
+	foundID := false
+	for _, m := range members {
+		if m.ID == removalID {
+			foundID = true
+		}
+		if m.Name == removalID {
+			// Note that, so long as it's not ambiguous, we *could* do the right thing by name here.
+			fmt.Fprintf(os.Stderr, "Found a member named %s; if this is correct, please use its ID, eg:\n\tetcdctl member remove %s\n", m.Name, m.ID)
+			fmt.Fprintf(os.Stderr, "For more details, read the documentation at https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md#remove-a-member\n\n")
+		}
+	}
+	if !foundID {
+		fmt.Fprintf(os.Stderr, "Couldn't find a member in the cluster with an ID of %s.\n", removalID)
 		os.Exit(1)
 	}
 
-	fmt.Printf("Removed member %s from cluster\n", mID)
+	// Actually attempt to remove the member.
+	ctx, removeCancel := context.WithTimeout(context.Background(), client.DefaultRequestTimeout)
+	err = mAPI.Remove(ctx, removalID)
+	removeCancel()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Recieved an error trying to remove member %s: %s", removalID, err.Error())
+		os.Exit(1)
+	}
+
+	fmt.Printf("Removed member %s from cluster\n", removalID)
 }

etcdctl/command/upgrade.go

@@ -23,6 +23,7 @@ import (
 	"os"
 
 	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
+	"github.com/coreos/etcd/pkg/transport"
 )
 
 func UpgradeCommand() cli.Command {
@@ -32,7 +33,10 @@ func UpgradeCommand() cli.Command {
 		Flags: []cli.Flag{
 			cli.StringFlag{Name: "old-version", Value: "1", Usage: "Old internal version"},
 			cli.StringFlag{Name: "new-version", Value: "2", Usage: "New internal version"},
-			cli.StringFlag{Name: "peer-url", Value: "", Usage: "An etcd peer url string"},
+			cli.StringFlag{Name: "peer-url", Value: "http://localhost:7001", Usage: "An etcd peer url string"},
+			cli.StringFlag{Name: "peer-cert-file", Value: "", Usage: "identify HTTPS peer using this SSL certificate file"},
+			cli.StringFlag{Name: "peer-key-file", Value: "", Usage: "identify HTTPS peer using this SSL key file"},
+			cli.StringFlag{Name: "peer-ca-file", Value: "", Usage: "verify certificates of HTTPS-enabled peers using this CA bundle"},
 		},
 		Action: handleUpgrade,
 	}
@@ -47,7 +51,12 @@ func handleUpgrade(c *cli.Context) {
 		fmt.Printf("Do not support upgrade to version %s\n", c.String("new-version"))
 		os.Exit(1)
 	}
-	t, err := getTransport(c)
+	tls := transport.TLSInfo{
+		CAFile:   c.String("peer-ca-file"),
+		CertFile: c.String("peer-cert-file"),
+		KeyFile:  c.String("peer-key-file"),
+	}
+	t, err := transport.NewTransport(tls)
 	if err != nil {
 		log.Fatal(err)
 	}

etcdctl/main.go

@@ -39,6 +39,7 @@ func main() {
 	}
 	app.Commands = []cli.Command{
 		command.NewBackupCommand(),
+		command.NewClusterHealthCommand(),
 		command.NewMakeCommand(),
 		command.NewMakeDirCommand(),
 		command.NewRemoveCommand(),

etcdmain/config.go

@@ -231,6 +231,9 @@ func (cfg *config) Parse(arguments []string) error {
 		return ErrConflictBootstrapFlags
 	}
 
+	flags.SetBindAddrFromAddr(cfg.FlagSet, "peer-bind-addr", "peer-addr")
+	flags.SetBindAddrFromAddr(cfg.FlagSet, "bind-addr", "addr")
+
 	cfg.lpurls, err = flags.URLsFromFlags(cfg.FlagSet, "listen-peer-urls", "peer-bind-addr", cfg.peerTLSInfo)
 	if err != nil {
 		return err

etcdmain/config_test.go

@@ -151,6 +151,35 @@ func TestConfigParsingOtherFlags(t *testing.T) {
 	}
 }
 
+func TestConfigParsingV1Flags(t *testing.T) {
+	args := []string{
+		"-peer-addr=127.0.0.1:7001",
+		"-addr=127.0.0.1:4001",
+	}
+	wcfg := NewConfig()
+	wcfg.lpurls = []url.URL{{Scheme: "http", Host: "0.0.0.0:7001"}}
+	wcfg.apurls = []url.URL{{Scheme: "http", Host: "127.0.0.1:7001"}}
+	wcfg.lcurls = []url.URL{{Scheme: "http", Host: "0.0.0.0:4001"}}
+	wcfg.acurls = []url.URL{{Scheme: "http", Host: "127.0.0.1:4001"}}
+
+	cfg := NewConfig()
+	if err := cfg.Parse(args); err != nil {
+		t.Fatal(err)
+	}
+	if !reflect.DeepEqual(cfg.lpurls, wcfg.lpurls) {
+		t.Errorf("listen peer urls = %+v, want %+v", cfg.lpurls, wcfg.lpurls)
+	}
+	if !reflect.DeepEqual(cfg.apurls, wcfg.apurls) {
+		t.Errorf("advertise peer urls = %+v, want %+v", cfg.apurls, wcfg.apurls)
+	}
+	if !reflect.DeepEqual(cfg.lcurls, wcfg.lcurls) {
+		t.Errorf("listen client urls = %+v, want %+v", cfg.lcurls, wcfg.lcurls)
+	}
+	if !reflect.DeepEqual(cfg.acurls, wcfg.acurls) {
+		t.Errorf("advertise client urls = %+v, want %+v", cfg.acurls, wcfg.acurls)
+	}
+}
+
 func TestConfigParsingConflictClusteringFlags(t *testing.T) {
 	conflictArgs := [][]string{
 		[]string{

etcdmain/etcd.go

@@ -15,11 +15,15 @@
 package etcdmain
 
 import (
+	"encoding/json"
 	"fmt"
+	"io/ioutil"
 	"log"
 	"net"
 	"net/http"
 	"os"
+	"path"
+	"reflect"
 	"strings"
 	"time"
 
@@ -27,7 +31,7 @@ import (
 	"github.com/coreos/etcd/etcdserver"
 	"github.com/coreos/etcd/etcdserver/etcdhttp"
 	"github.com/coreos/etcd/pkg/cors"
-	"github.com/coreos/etcd/pkg/fileutil"
+	"github.com/coreos/etcd/pkg/osutil"
 	"github.com/coreos/etcd/pkg/transport"
 	"github.com/coreos/etcd/pkg/types"
 	"github.com/coreos/etcd/proxy"
@@ -70,7 +74,10 @@ func Main() {
 		}
 	}
 
+	osutil.HandleInterrupts()
+
 	<-stopped
+	osutil.Exit(0)
 }
 
 // startEtcd launches the etcd server and HTTP handlers for client/server communication.
@@ -84,12 +91,6 @@ func startEtcd(cfg *config) (<-chan struct{}, error) {
 		cfg.dir = fmt.Sprintf("%v.etcd", cfg.name)
 		log.Printf("no data-dir provided, using default data-dir ./%s", cfg.dir)
 	}
-	if err := os.MkdirAll(cfg.dir, privateDirMode); err != nil {
-		return nil, fmt.Errorf("cannot create data directory: %v", err)
-	}
-	if err := fileutil.IsDirWriteable(cfg.dir); err != nil {
-		return nil, fmt.Errorf("cannot write to data directory: %v", err)
-	}
 
 	pt, err := transport.NewTimeoutTransport(cfg.peerTLSInfo, rafthttp.ConnReadTimeout, rafthttp.ConnWriteTimeout)
 	if err != nil {
@@ -163,6 +164,7 @@ func startEtcd(cfg *config) (<-chan struct{}, error) {
 		return nil, err
 	}
 	s.Start()
+	osutil.RegisterInterruptHandler(s.Stop)
 
 	if cfg.corsInfo.String() != "" {
 		log.Printf("etcd: cors = %s", cfg.corsInfo)
@@ -171,7 +173,7 @@ func startEtcd(cfg *config) (<-chan struct{}, error) {
 		Handler: etcdhttp.NewClientHandler(s),
 		Info:    cfg.corsInfo,
 	}
-	ph := etcdhttp.NewPeerHandler(s.Cluster, s.RaftHandler())
+	ph := etcdhttp.NewPeerHandler(s.Cluster, etcdserver.RaftTimer(s), s.RaftHandler())
 	// Start the peer server in a goroutine
 	for _, l := range plns {
 		go func(l net.Listener) {
@@ -216,15 +218,71 @@ func startProxy(cfg *config) error {
 		return err
 	}
 
-	// TODO(jonboulle): update peerURLs dynamically (i.e. when updating
-	// clientURLs) instead of just using the initial fixed list here
-	peerURLs := cls.PeerURLs()
+	if cfg.dir == "" {
+		cfg.dir = fmt.Sprintf("%v.etcd", cfg.name)
+		log.Printf("no proxy data-dir provided, using default proxy data-dir ./%s", cfg.dir)
+	}
+	cfg.dir = path.Join(cfg.dir, "proxy")
+	err = os.MkdirAll(cfg.dir, 0700)
+	if err != nil {
+		return err
+	}
+
+	var peerURLs []string
+	clusterfile := path.Join(cfg.dir, "cluster")
+
+	b, err := ioutil.ReadFile(clusterfile)
+	switch {
+	case err == nil:
+		urls := struct{ PeerURLs []string }{}
+		err := json.Unmarshal(b, &urls)
+		if err != nil {
+			return err
+		}
+		peerURLs = urls.PeerURLs
+		log.Printf("proxy: using peer urls %v from cluster file ./%s", peerURLs, clusterfile)
+	case os.IsNotExist(err):
+		peerURLs = cls.PeerURLs()
+		log.Printf("proxy: using peer urls %v ", peerURLs)
+	default:
+		return err
+	}
+
 	uf := func() []string {
-		cls, err := etcdserver.GetClusterFromPeers(peerURLs, tr)
+		gcls, err := etcdserver.GetClusterFromRemotePeers(peerURLs, tr)
+		// TODO: remove the 2nd check when we fix GetClusterFromPeers
+		// GetClusterFromPeers should not return nil error with an invaild empty cluster
 		if err != nil {
 			log.Printf("proxy: %v", err)
 			return []string{}
 		}
+		if len(gcls.Members()) == 0 {
+			return cls.ClientURLs()
+		}
+		cls = gcls
+
+		urls := struct{ PeerURLs []string }{cls.PeerURLs()}
+		b, err := json.Marshal(urls)
+		if err != nil {
+			log.Printf("proxy: error on marshal peer urls %s", err)
+			return cls.ClientURLs()
+		}
+
+		err = ioutil.WriteFile(clusterfile+".bak", b, 0600)
+		if err != nil {
+			log.Printf("proxy: error on writing urls %s", err)
+			return cls.ClientURLs()
+		}
+		err = os.Rename(clusterfile+".bak", clusterfile)
+		if err != nil {
+			log.Printf("proxy: error on updating clusterfile %s", err)
+			return cls.ClientURLs()
+		}
+		if !reflect.DeepEqual(cls.PeerURLs(), peerURLs) {
+			log.Printf("proxy: updated peer urls in cluster file from %v to %v", peerURLs, cls.PeerURLs())
+		}
+		peerURLs = cls.PeerURLs()
+
 		return cls.ClientURLs()
 	}
 	ph := proxy.NewHandler(pt, uf)

etcdserver/cluster.go

@@ -56,14 +56,21 @@ type ClusterInfo interface {
 
 // Cluster is a list of Members that belong to the same raft cluster
 type Cluster struct {
-	id      types.ID
-	token   string
-	members map[types.ID]*Member
+	id    types.ID
+	token string
+	store store.Store
+	// index is the raft index that cluster is updated at bootstrap
+	// from remote cluster info.
+	// It may have a higher value than local raft index, because it
+	// displays a further view of the cluster.
+	// TODO: upgrade it as last modified index
+	index uint64
+
+	sync.Mutex // guards members and removed map
+	members    map[types.ID]*Member
 	// removed contains the ids of removed members in the cluster.
 	// removed id cannot be reused.
 	removed map[types.ID]bool
-	store   store.Store
-	sync.Mutex
 }
 
 // NewClusterFromString returns a Cluster instantiated from the given cluster token
@@ -229,6 +236,8 @@ func (c *Cluster) SetID(id types.ID) { c.id = id }
 
 func (c *Cluster) SetStore(st store.Store) { c.store = st }
 
+func (c *Cluster) UpdateIndex(index uint64) { c.index = index }
+
 func (c *Cluster) Recover() {
 	c.members, c.removed = membersFromStore(c.store)
 }
@@ -346,6 +355,20 @@ func (c *Cluster) UpdateRaftAttributes(id types.ID, raftAttr RaftAttributes) {
 	c.members[id].RaftAttributes = raftAttr
 }
 
+// Validate ensures that there is no identical urls in the cluster peer list
+func (c *Cluster) Validate() error {
+	urlMap := make(map[string]bool)
+	for _, m := range c.Members() {
+		for _, url := range m.PeerURLs {
+			if urlMap[url] {
+				return fmt.Errorf("duplicate url %v in cluster config", url)
+			}
+			urlMap[url] = true
+		}
+	}
+	return nil
+}
+
 func membersFromStore(st store.Store) (map[types.ID]*Member, map[types.ID]bool) {
 	members := make(map[types.ID]*Member)
 	removed := make(map[types.ID]bool)

etcdserver/cluster_util.go

@@ -0,0 +1,123 @@
+// Copyright 2015 CoreOS, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package etcdserver
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"log"
+	"net/http"
+	"sort"
+	"strconv"
+	"time"
+
+	"github.com/coreos/etcd/pkg/types"
+)
+
+// isMemberBootstrapped tries to check if the given member has been bootstrapped
+// in the given cluster.
+func isMemberBootstrapped(cl *Cluster, member string, tr *http.Transport) bool {
+	rcl, err := getClusterFromRemotePeers(getRemotePeerURLs(cl, member), false, tr)
+	if err != nil {
+		return false
+	}
+	id := cl.MemberByName(member).ID
+	m := rcl.Member(id)
+	if m == nil {
+		return false
+	}
+	if len(m.ClientURLs) > 0 {
+		return true
+	}
+	return false
+}
+
+// GetClusterFromRemotePeers takes a set of URLs representing etcd peers, and
+// attempts to construct a Cluster by accessing the members endpoint on one of
+// these URLs. The first URL to provide a response is used. If no URLs provide
+// a response, or a Cluster cannot be successfully created from a received
+// response, an error is returned.
+func GetClusterFromRemotePeers(urls []string, tr *http.Transport) (*Cluster, error) {
+	return getClusterFromRemotePeers(urls, true, tr)
+}
+
+// If logerr is true, it prints out more error messages.
+func getClusterFromRemotePeers(urls []string, logerr bool, tr *http.Transport) (*Cluster, error) {
+	cc := &http.Client{
+		Transport: tr,
+		Timeout:   time.Second,
+	}
+	for _, u := range urls {
+		resp, err := cc.Get(u + "/members")
+		if err != nil {
+			if logerr {
+				log.Printf("etcdserver: could not get cluster response from %s: %v", u, err)
+			}
+			continue
+		}
+		b, err := ioutil.ReadAll(resp.Body)
+		if err != nil {
+			if logerr {
+				log.Printf("etcdserver: could not read the body of cluster response: %v", err)
+			}
+			continue
+		}
+		var membs []*Member
+		if err := json.Unmarshal(b, &membs); err != nil {
+			if logerr {
+				log.Printf("etcdserver: could not unmarshal cluster response: %v", err)
+			}
+			continue
+		}
+		id, err := types.IDFromString(resp.Header.Get("X-Etcd-Cluster-ID"))
+		if err != nil {
+			if logerr {
+				log.Printf("etcdserver: could not parse the cluster ID from cluster res: %v", err)
+			}
+			continue
+		}
+		var index uint64
+		// The header at or before v2.0.3 doesn't have this field. For backward
+		// compatibility, it checks whether the field exists.
+		if indexStr := resp.Header.Get("X-Raft-Index"); indexStr != "" {
+			index, err = strconv.ParseUint(indexStr, 10, 64)
+			if err != nil {
+				if logerr {
+					log.Printf("etcdserver: could not parse raft index: %v", err)
+				}
+				continue
+			}
+		}
+		cl := NewClusterFromMembers("", id, membs)
+		cl.UpdateIndex(index)
+		return cl, nil
+	}
+	return nil, fmt.Errorf("etcdserver: could not retrieve cluster information from the given urls")
+}
+
+// getRemotePeerURLs returns peer urls of remote members in the cluster. The
+// returned list is sorted in ascending lexicographical order.
+func getRemotePeerURLs(cl ClusterInfo, local string) []string {
+	us := make([]string, 0)
+	for _, m := range cl.Members() {
+		if m.Name == local {
+			continue
+		}
+		us = append(us, m.PeerURLs...)
+	}
+	sort.Strings(us)
+	return us
+}

etcdserver/config.go

@@ -62,15 +62,8 @@ func (c *ServerConfig) VerifyBootstrapConfig() error {
 		return fmt.Errorf("initial cluster state unset and no wal or discovery URL found")
 	}
 
-	// No identical IPs in the cluster peer list
-	urlMap := make(map[string]bool)
-	for _, m := range c.Cluster.Members() {
-		for _, url := range m.PeerURLs {
-			if urlMap[url] {
-				return fmt.Errorf("duplicate url %v in cluster config", url)
-			}
-			urlMap[url] = true
-		}
+	if err := c.Cluster.Validate(); err != nil {
+		return err
 	}
 
 	// Advertised peer URLs must match those in the cluster peer list
@@ -83,9 +76,11 @@ func (c *ServerConfig) VerifyBootstrapConfig() error {
 	return nil
 }
 
-func (c *ServerConfig) WALDir() string { return path.Join(c.DataDir, "wal") }
+func (c *ServerConfig) MemberDir() string { return path.Join(c.DataDir, "member") }
 
-func (c *ServerConfig) SnapDir() string { return path.Join(c.DataDir, "snap") }
+func (c *ServerConfig) WALDir() string { return path.Join(c.MemberDir(), "wal") }
+
+func (c *ServerConfig) SnapDir() string { return path.Join(c.MemberDir(), "snap") }
 
 func (c *ServerConfig) ShouldDiscover() bool { return c.DiscoveryURL != "" }
 
@@ -99,6 +94,7 @@ func (c *ServerConfig) print(initial bool) {
 		log.Println("etcdserver: force new cluster")
 	}
 	log.Printf("etcdserver: data dir = %s", c.DataDir)
+	log.Printf("etcdserver: member dir = %s", c.MemberDir())
 	log.Printf("etcdserver: heartbeat = %dms", c.TickMs)
 	log.Printf("etcdserver: election = %dms", c.ElectionTicks*int(c.TickMs))
 	log.Printf("etcdserver: snapshot count = %d", c.SnapCount)

etcdserver/config_test.go

@@ -129,8 +129,8 @@ func TestBootstrapConfigVerify(t *testing.T) {
 
 func TestSnapDir(t *testing.T) {
 	tests := map[string]string{
-		"/":            "/snap",
-		"/var/lib/etc": "/var/lib/etc/snap",
+		"/":            "/member/snap",
+		"/var/lib/etc": "/var/lib/etc/member/snap",
 	}
 	for dd, w := range tests {
 		cfg := ServerConfig{
@@ -144,8 +144,8 @@ func TestSnapDir(t *testing.T) {
 
 func TestWALDir(t *testing.T) {
 	tests := map[string]string{
-		"/":            "/wal",
-		"/var/lib/etc": "/var/lib/etc/wal",
+		"/":            "/member/wal",
+		"/var/lib/etc": "/var/lib/etc/member/wal",
 	}
 	for dd, w := range tests {
 		cfg := ServerConfig{

etcdserver/etcdhttp/client_test.go

@@ -1064,13 +1064,13 @@ func TestServeMembersFail(t *testing.T) {
 
 func TestWriteEvent(t *testing.T) {
 	// nil event should not panic
-	rw := httptest.NewRecorder()
-	writeKeyEvent(rw, nil, dummyRaftTimer{})
-	h := rw.Header()
+	rec := httptest.NewRecorder()
+	writeKeyEvent(rec, nil, dummyRaftTimer{})
+	h := rec.Header()
 	if len(h) > 0 {
 		t.Fatalf("unexpected non-empty headers: %#v", h)
 	}
-	b := rw.Body.String()
+	b := rec.Body.String()
 	if len(b) > 0 {
 		t.Fatalf("unexpected non-empty body: %q", b)
 	}

etcdserver/etcdhttp/http_test.go

@@ -76,13 +76,13 @@ func (fs *errServer) UpdateMember(ctx context.Context, m etcdserver.Member) erro
 
 func TestWriteError(t *testing.T) {
 	// nil error should not panic
-	rw := httptest.NewRecorder()
-	writeError(rw, nil)
-	h := rw.Header()
+	rec := httptest.NewRecorder()
+	writeError(rec, nil)
+	h := rec.Header()
 	if len(h) > 0 {
 		t.Fatalf("unexpected non-empty headers: %#v", h)
 	}
-	b := rw.Body.String()
+	b := rec.Body.String()
 	if len(b) > 0 {
 		t.Fatalf("unexpected non-empty body: %q", b)
 	}

etcdserver/etcdhttp/peer.go

@@ -18,6 +18,7 @@ import (
 	"encoding/json"
 	"log"
 	"net/http"
+	"strconv"
 
 	"github.com/coreos/etcd/etcdserver"
 	"github.com/coreos/etcd/rafthttp"
@@ -28,9 +29,10 @@ const (
 )
 
 // NewPeerHandler generates an http.Handler to handle etcd peer (raft) requests.
-func NewPeerHandler(clusterInfo etcdserver.ClusterInfo, raftHandler http.Handler) http.Handler {
+func NewPeerHandler(clusterInfo etcdserver.ClusterInfo, timer etcdserver.RaftTimer, raftHandler http.Handler) http.Handler {
 	mh := &peerMembersHandler{
 		clusterInfo: clusterInfo,
+		timer:       timer,
 	}
 
 	mux := http.NewServeMux()
@@ -43,6 +45,7 @@ func NewPeerHandler(clusterInfo etcdserver.ClusterInfo, raftHandler http.Handler
 
 type peerMembersHandler struct {
 	clusterInfo etcdserver.ClusterInfo
+	timer       etcdserver.RaftTimer
 }
 
 func (h *peerMembersHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
@@ -50,6 +53,7 @@ func (h *peerMembersHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	w.Header().Set("X-Etcd-Cluster-ID", h.clusterInfo.ID().String())
+	w.Header().Set("X-Raft-Index", strconv.FormatUint(h.timer.Index(), 10))
 
 	if r.URL.Path != peerMembersPrefix {
 		http.Error(w, "bad path", http.StatusBadRequest)

etcdserver/etcdhttp/peer_test.go

@@ -33,7 +33,7 @@ func TestNewPeerHandlerOnRaftPrefix(t *testing.T) {
 	h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		w.Write([]byte("test data"))
 	})
-	ph := NewPeerHandler(&fakeCluster{}, h)
+	ph := NewPeerHandler(&fakeCluster{}, &dummyRaftTimer{}, h)
 	srv := httptest.NewServer(ph)
 	defer srv.Close()
 
@@ -91,7 +91,7 @@ func TestServeMembersGet(t *testing.T) {
 		id:      1,
 		members: map[uint64]*etcdserver.Member{1: &memb1, 2: &memb2},
 	}
-	h := &peerMembersHandler{clusterInfo: cluster}
+	h := &peerMembersHandler{clusterInfo: cluster, timer: &dummyRaftTimer{}}
 	msb, err := json.Marshal([]etcdserver.Member{memb1, memb2})
 	if err != nil {
 		t.Fatal(err)

etcdserver/server.go

@@ -18,13 +18,11 @@ import (
 	"encoding/json"
 	"expvar"
 	"fmt"
-	"io/ioutil"
 	"log"
 	"math/rand"
 	"net/http"
 	"path"
 	"regexp"
-	"sort"
 	"sync/atomic"
 	"time"
 
@@ -141,11 +139,12 @@ type EtcdServer struct {
 // NewServer creates a new EtcdServer from the supplied configuration. The
 // configuration is considered static for the lifetime of the EtcdServer.
 func NewServer(cfg *ServerConfig) (*EtcdServer, error) {
-	st := store.New()
+	st := store.New(StoreAdminPrefix, StoreKeysPrefix)
 	var w *wal.WAL
 	var n raft.Node
 	var s *raft.MemoryStorage
 	var id types.ID
+
 	walVersion, err := wal.DetectVersion(cfg.DataDir)
 	if err != nil {
 		return nil, err
@@ -154,18 +153,18 @@ func NewServer(cfg *ServerConfig) (*EtcdServer, error) {
 		return nil, fmt.Errorf("unknown wal version in data dir %s", cfg.DataDir)
 	}
 	haveWAL := walVersion != wal.WALNotExist
-
 	ss := snap.New(cfg.SnapDir())
+
 	switch {
 	case !haveWAL && !cfg.NewCluster:
-		us := getOtherPeerURLs(cfg.Cluster, cfg.Name)
-		existingCluster, err := GetClusterFromPeers(us, cfg.Transport)
+		existingCluster, err := GetClusterFromRemotePeers(getRemotePeerURLs(cfg.Cluster, cfg.Name), cfg.Transport)
 		if err != nil {
 			return nil, fmt.Errorf("cannot fetch cluster info from peer urls: %v", err)
 		}
 		if err := ValidateClusterAndAssignIDs(cfg.Cluster, existingCluster); err != nil {
 			return nil, fmt.Errorf("error validating peerURLs %s: %v", existingCluster, err)
 		}
+		cfg.Cluster.UpdateIndex(existingCluster.index)
 		cfg.Cluster.SetID(existingCluster.id)
 		cfg.Cluster.SetStore(st)
 		cfg.Print()
@@ -175,26 +174,36 @@ func NewServer(cfg *ServerConfig) (*EtcdServer, error) {
 			return nil, err
 		}
 		m := cfg.Cluster.MemberByName(cfg.Name)
-		if isBootstrapped(cfg) {
+		if isMemberBootstrapped(cfg.Cluster, cfg.Name, cfg.Transport) {
 			return nil, fmt.Errorf("member %s has already been bootstrapped", m.ID)
 		}
 		if cfg.ShouldDiscover() {
-			s, err := discovery.JoinCluster(cfg.DiscoveryURL, cfg.DiscoveryProxy, m.ID, cfg.Cluster.String())
+			str, err := discovery.JoinCluster(cfg.DiscoveryURL, cfg.DiscoveryProxy, m.ID, cfg.Cluster.String())
 			if err != nil {
 				return nil, err
 			}
-			if cfg.Cluster, err = NewClusterFromString(cfg.Cluster.token, s); err != nil {
+			if cfg.Cluster, err = NewClusterFromString(cfg.Cluster.token, str); err != nil {
 				return nil, err
 			}
+			if err := cfg.Cluster.Validate(); err != nil {
+				return nil, fmt.Errorf("bad discovery cluster: %v", err)
+			}
 		}
 		cfg.Cluster.SetStore(st)
 		cfg.PrintWithInitial()
 		id, n, s, w = startNode(cfg, cfg.Cluster.MemberIDs())
 	case haveWAL:
-		if walVersion != wal.WALv0_5 {
-			if err := upgradeWAL(cfg, walVersion); err != nil {
-				return nil, err
-			}
+		// Run the migrations.
+		if err := upgradeWAL(cfg.DataDir, cfg.Name, walVersion); err != nil {
+			return nil, err
+		}
+
+		if err := fileutil.IsDirWriteable(cfg.DataDir); err != nil {
+			return nil, fmt.Errorf("cannot write to data directory: %v", err)
+		}
+
+		if err := fileutil.IsDirWriteable(cfg.MemberDir()); err != nil {
+			return nil, fmt.Errorf("cannot write to member directory: %v", err)
 		}
 
 		if cfg.ShouldDiscover() {
@@ -253,7 +262,7 @@ func NewServer(cfg *ServerConfig) (*EtcdServer, error) {
 	tr := rafthttp.NewTransporter(cfg.Transport, id, cfg.Cluster.ID(), srv, srv.errorc, sstats, lstats)
 	// add all the remote members into sendhub
 	for _, m := range cfg.Cluster.Members() {
-		if m.Name != cfg.Name {
+		if m.ID != id {
 			tr.AddPeer(m.ID, m.PeerURLs)
 		}
 	}
@@ -385,8 +394,23 @@ func (s *EtcdServer) run() {
 				if err := s.store.Recovery(rd.Snapshot.Data); err != nil {
 					log.Panicf("recovery store error: %v", err)
 				}
-				s.Cluster.Recover()
+
+				// It avoids snapshot recovery overwriting newer cluster and
+				// transport setting, which may block the communication.
+				if s.Cluster.index < rd.Snapshot.Metadata.Index {
+					s.Cluster.Recover()
+					// recover raft transport
+					s.r.transport.RemoveAllPeers()
+					for _, m := range s.Cluster.Members() {
+						if m.ID == s.ID() {
+							continue
+						}
+						s.r.transport.AddPeer(m.ID, m.PeerURLs)
+					}
+				}
+
 				appliedi = rd.Snapshot.Metadata.Index
+				confState = rd.Snapshot.Metadata.ConfState
 				log.Printf("etcdserver: recovered from incoming snapshot at index %d", snapi)
 			}
 			// TODO(bmizerany): do this in the background, but take
@@ -820,88 +844,3 @@ func (s *EtcdServer) snapshot(snapi uint64, confState *raftpb.ConfState) {
 func (s *EtcdServer) PauseSending() { s.r.pauseSending() }
 
 func (s *EtcdServer) ResumeSending() { s.r.resumeSending() }
-
-// isBootstrapped tries to check if the given member has been bootstrapped
-// in the given cluster.
-func isBootstrapped(cfg *ServerConfig) bool {
-	cl := cfg.Cluster
-	member := cfg.Name
-
-	us := getOtherPeerURLs(cl, member)
-	rcl, err := getClusterFromPeers(us, false, cfg.Transport)
-	if err != nil {
-		return false
-	}
-	id := cl.MemberByName(member).ID
-	m := rcl.Member(id)
-	if m == nil {
-		return false
-	}
-	if len(m.ClientURLs) > 0 {
-		return true
-	}
-	return false
-}
-
-// GetClusterFromPeers takes a set of URLs representing etcd peers, and
-// attempts to construct a Cluster by accessing the members endpoint on one of
-// these URLs. The first URL to provide a response is used. If no URLs provide
-// a response, or a Cluster cannot be successfully created from a received
-// response, an error is returned.
-func GetClusterFromPeers(urls []string, tr *http.Transport) (*Cluster, error) {
-	return getClusterFromPeers(urls, true, tr)
-}
-
-// If logerr is true, it prints out more error messages.
-func getClusterFromPeers(urls []string, logerr bool, tr *http.Transport) (*Cluster, error) {
-	cc := &http.Client{
-		Transport: tr,
-		Timeout:   time.Second,
-	}
-	for _, u := range urls {
-		resp, err := cc.Get(u + "/members")
-		if err != nil {
-			if logerr {
-				log.Printf("etcdserver: could not get cluster response from %s: %v", u, err)
-			}
-			continue
-		}
-		b, err := ioutil.ReadAll(resp.Body)
-		if err != nil {
-			if logerr {
-				log.Printf("etcdserver: could not read the body of cluster response: %v", err)
-			}
-			continue
-		}
-		var membs []*Member
-		if err := json.Unmarshal(b, &membs); err != nil {
-			if logerr {
-				log.Printf("etcdserver: could not unmarshal cluster response: %v", err)
-			}
-			continue
-		}
-		id, err := types.IDFromString(resp.Header.Get("X-Etcd-Cluster-ID"))
-		if err != nil {
-			if logerr {
-				log.Printf("etcdserver: could not parse the cluster ID from cluster res: %v", err)
-			}
-			continue
-		}
-		return NewClusterFromMembers("", id, membs), nil
-	}
-	return nil, fmt.Errorf("etcdserver: could not retrieve cluster information from the given urls")
-}
-
-// getOtherPeerURLs returns peer urls of other members in the cluster. The
-// returned list is sorted in ascending lexicographical order.
-func getOtherPeerURLs(cl ClusterInfo, self string) []string {
-	us := make([]string, 0)
-	for _, m := range cl.Members() {
-		if m.Name == self {
-			continue
-		}
-		us = append(us, m.PeerURLs...)
-	}
-	sort.Strings(us)
-	return us
-}

etcdserver/server_test.go

@@ -1027,8 +1027,8 @@ func TestPublish(t *testing.T) {
 		t.Errorf("method = %s, want PUT", r.Method)
 	}
 	wm := Member{ID: 1, Attributes: Attributes{Name: "node1", ClientURLs: []string{"http://a", "http://b"}}}
-	if w := path.Join(memberStoreKey(wm.ID), attributesSuffix); r.Path != w {
-		t.Errorf("path = %s, want %s", r.Path, w)
+	if wpath := path.Join(memberStoreKey(wm.ID), attributesSuffix); r.Path != wpath {
+		t.Errorf("path = %s, want %s", r.Path, wpath)
 	}
 	var gattr Attributes
 	if err := json.Unmarshal([]byte(r.Val), &gattr); err != nil {
@@ -1072,8 +1072,8 @@ func TestPublishRetry(t *testing.T) {
 
 	action := n.Action()
 	// multiple Proposes
-	if n := len(action); n < 2 {
-		t.Errorf("len(action) = %d, want >= 2", n)
+	if cnt := len(action); cnt < 2 {
+		t.Errorf("len(action) = %d, want >= 2", cnt)
 	}
 }
 
@@ -1135,7 +1135,7 @@ func TestGetOtherPeerURLs(t *testing.T) {
 	}
 	for i, tt := range tests {
 		cl := NewClusterFromMembers("", types.ID(0), tt.membs)
-		urls := getOtherPeerURLs(cl, tt.self)
+		urls := getRemotePeerURLs(cl, tt.self)
 		if !reflect.DeepEqual(urls, tt.wurls) {
 			t.Errorf("#%d: urls = %+v, want %+v", i, urls, tt.wurls)
 		}
@@ -1393,6 +1393,7 @@ func (s *nopTransporter) Handler() http.Handler               { return nil }
 func (s *nopTransporter) Send(m []raftpb.Message)             {}
 func (s *nopTransporter) AddPeer(id types.ID, us []string)    {}
 func (s *nopTransporter) RemovePeer(id types.ID)              {}
+func (s *nopTransporter) RemoveAllPeers()                     {}
 func (s *nopTransporter) UpdatePeer(id types.ID, us []string) {}
 func (s *nopTransporter) Stop()                               {}
 func (s *nopTransporter) Pause()                              {}

etcdserver/storage.go

@@ -16,6 +16,8 @@ package etcdserver
 
 import (
 	"log"
+	"os"
+	"path"
 
 	pb "github.com/coreos/etcd/etcdserver/etcdserverpb"
 	"github.com/coreos/etcd/migrate"
@@ -91,14 +93,47 @@ func readWAL(waldir string, snap walpb.Snapshot) (w *wal.WAL, id, cid types.ID,
 
 // upgradeWAL converts an older version of the etcdServer data to the newest version.
 // It must ensure that, after upgrading, the most recent version is present.
-func upgradeWAL(cfg *ServerConfig, ver wal.WalVersion) error {
-	if ver == wal.WALv0_4 {
+func upgradeWAL(baseDataDir string, name string, ver wal.WalVersion) error {
+	switch ver {
+	case wal.WALv0_4:
 		log.Print("etcdserver: converting v0.4 log to v2.0")
-		err := migrate.Migrate4To2(cfg.DataDir, cfg.Name)
+		err := migrate.Migrate4To2(baseDataDir, name)
 		if err != nil {
 			log.Fatalf("etcdserver: failed migrating data-dir: %v", err)
 			return err
 		}
+		fallthrough
+	case wal.WALv2_0:
+		err := makeMemberDir(baseDataDir)
+		if err != nil {
+			return err
+		}
+		fallthrough
+	case wal.WALv2_0_1:
+		fallthrough
+	default:
+		log.Printf("datadir is valid for the 2.0.1 format")
+	}
+	return nil
+}
+
+func makeMemberDir(dir string) error {
+	membdir := path.Join(dir, "member")
+	_, err := os.Stat(membdir)
+	switch {
+	case err == nil:
+		return nil
+	case !os.IsNotExist(err):
+		return err
+	}
+	if err := os.MkdirAll(membdir, 0700); err != nil {
+		return err
+	}
+	names := []string{"snap", "wal"}
+	for _, name := range names {
+		if err := os.Rename(path.Join(dir, name), path.Join(membdir, name)); err != nil {
+			return err
+		}
 	}
 	return nil
 }

integration/cluster_test.go

@@ -186,13 +186,13 @@ func clusterMustProgress(t *testing.T, membs []*member) {
 
 	for i, m := range membs {
 		u := m.URL()
-		cc := mustNewHTTPClient(t, []string{u})
-		kapi := client.NewKeysAPI(cc)
-		ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
-		if _, err := kapi.Watch(key, resp.Node.ModifiedIndex).Next(ctx); err != nil {
+		mcc := mustNewHTTPClient(t, []string{u})
+		mkapi := client.NewKeysAPI(mcc)
+		mctx, mcancel := context.WithTimeout(context.Background(), requestTimeout)
+		if _, err := mkapi.Watch(key, resp.Node.ModifiedIndex).Next(mctx); err != nil {
 			t.Fatalf("#%d: watch on %s error: %v", i, u, err)
 		}
-		cancel()
+		mcancel()
 	}
 }
 
@@ -526,7 +526,7 @@ func (m *member) Launch() error {
 	m.s.SyncTicker = time.Tick(500 * time.Millisecond)
 	m.s.Start()
 
-	m.raftHandler = &testutil.PauseableHandler{Next: etcdhttp.NewPeerHandler(m.s.Cluster, m.s.RaftHandler())}
+	m.raftHandler = &testutil.PauseableHandler{Next: etcdhttp.NewPeerHandler(m.s.Cluster, m.s, m.s.RaftHandler())}
 
 	for _, ln := range m.PeerListeners {
 		hs := &httptest.Server{
@@ -547,6 +547,24 @@ func (m *member) Launch() error {
 	return nil
 }
 
+func (m *member) WaitOK(t *testing.T) {
+	cc := mustNewHTTPClient(t, []string{m.URL()})
+	kapi := client.NewKeysAPI(cc)
+	for {
+		ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
+		_, err := kapi.Get(ctx, "/")
+		if err != nil {
+			time.Sleep(tickDuration)
+			continue
+		}
+		cancel()
+		break
+	}
+	for m.s.Leader() == 0 {
+		time.Sleep(tickDuration)
+	}
+}
+
 func (m *member) URL() string { return m.ClientURLs[0].String() }
 
 func (m *member) Pause() {

integration/member_test.go

@@ -15,9 +15,14 @@
 package integration
 
 import (
+	"fmt"
 	"io/ioutil"
 	"os"
+	"reflect"
 	"testing"
+
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
+	"github.com/coreos/etcd/client"
 )
 
 func TestPauseMember(t *testing.T) {
@@ -74,3 +79,44 @@ func TestLaunchDuplicateMemberShouldFail(t *testing.T) {
 		t.Errorf("unexpect successful launch")
 	}
 }
+
+func TestSnapshotAndRestartMember(t *testing.T) {
+	defer afterTest(t)
+	m := mustNewMember(t, "snapAndRestartTest")
+	m.SnapCount = 100
+	m.Launch()
+	defer m.Terminate(t)
+	m.WaitOK(t)
+
+	resps := make([]*client.Response, 120)
+	var err error
+	for i := 0; i < 120; i++ {
+		cc := mustNewHTTPClient(t, []string{m.URL()})
+		kapi := client.NewKeysAPI(cc)
+		ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
+		key := fmt.Sprintf("foo%d", i)
+		resps[i], err = kapi.Create(ctx, "/"+key, "bar", -1)
+		if err != nil {
+			t.Fatalf("#%d: create on %s error: %v", i, m.URL(), err)
+		}
+		cancel()
+	}
+	m.Stop(t)
+	m.Restart(t)
+
+	for i := 0; i < 120; i++ {
+		cc := mustNewHTTPClient(t, []string{m.URL()})
+		kapi := client.NewKeysAPI(cc)
+		ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
+		key := fmt.Sprintf("foo%d", i)
+		resp, err := kapi.Get(ctx, "/"+key)
+		if err != nil {
+			t.Fatalf("#%d: get on %s error: %v", i, m.URL(), err)
+		}
+		cancel()
+
+		if !reflect.DeepEqual(resp.Node, resps[i].Node) {
+			t.Errorf("#%d: node = %v, want %v", i, resp.Node, resps[i].Node)
+		}
+	}
+}

integration/v2_http_kv_test.go

@@ -327,21 +327,21 @@ func TestV2Delete(t *testing.T) {
 
 	v := url.Values{}
 	v.Set("value", "XXX")
-	resp, err := tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/foo"), v)
+	r, err := tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/foo"), v)
 	if err != nil {
 		t.Error(err)
 	}
-	resp.Body.Close()
-	resp, err = tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/emptydir?dir=true"), v)
+	r.Body.Close()
+	r, err = tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/emptydir?dir=true"), v)
 	if err != nil {
 		t.Error(err)
 	}
-	resp.Body.Close()
-	resp, err = tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/foodir/bar?dir=true"), v)
+	r.Body.Close()
+	r, err = tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/foodir/bar?dir=true"), v)
 	if err != nil {
 		t.Error(err)
 	}
-	resp.Body.Close()
+	r.Body.Close()
 
 	tests := []struct {
 		relativeURL string
@@ -423,17 +423,17 @@ func TestV2CAD(t *testing.T) {
 
 	v := url.Values{}
 	v.Set("value", "XXX")
-	resp, err := tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/foo"), v)
+	r, err := tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/foo"), v)
 	if err != nil {
 		t.Error(err)
 	}
-	resp.Body.Close()
+	r.Body.Close()
 
-	resp, err = tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/foovalue"), v)
+	r, err = tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/foovalue"), v)
 	if err != nil {
 		t.Error(err)
 	}
-	resp.Body.Close()
+	r.Body.Close()
 
 	tests := []struct {
 		relativeURL string
@@ -582,11 +582,11 @@ func TestV2Get(t *testing.T) {
 
 	v := url.Values{}
 	v.Set("value", "XXX")
-	resp, err := tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/foo/bar/zar"), v)
+	r, err := tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/foo/bar/zar"), v)
 	if err != nil {
 		t.Error(err)
 	}
-	resp.Body.Close()
+	r.Body.Close()
 
 	tests := []struct {
 		relativeURL string
@@ -676,11 +676,11 @@ func TestV2QuorumGet(t *testing.T) {
 
 	v := url.Values{}
 	v.Set("value", "XXX")
-	resp, err := tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/foo/bar/zar?quorum=true"), v)
+	r, err := tc.PutForm(fmt.Sprintf("%s%s", u, "/v2/keys/foo/bar/zar?quorum=true"), v)
 	if err != nil {
 		t.Error(err)
 	}
-	resp.Body.Close()
+	r.Body.Close()
 
 	tests := []struct {
 		relativeURL string

main.go

@@ -24,9 +24,26 @@
 package main
 
 import (
+	"log"
+	"os"
+	"strconv"
+
 	"github.com/coreos/etcd/etcdmain"
+	"github.com/coreos/etcd/migrate/starter"
+	"github.com/coreos/etcd/pkg/coreos"
 )
 
 func main() {
+	if str := os.Getenv("ETCD_ALLOW_LEGACY_MODE"); str != "" {
+		v, err := strconv.ParseBool(str)
+		if err != nil {
+			log.Fatalf("failed to parse ETCD_ALLOW_LEGACY_MODE=%s as bool", str)
+		}
+		if v {
+			starter.StartDesiredVersion(os.Args[1:])
+		}
+	} else if coreos.IsCoreOS() {
+		starter.StartDesiredVersion(os.Args[1:])
+	}
 	etcdmain.Main()
 }

migrate/etcd4.go

@@ -103,7 +103,7 @@ func Migrate4To2(dataDir string, name string) error {
 	st2 := cfg4.HardState2()
 
 	// If we've got the most recent snapshot, we can use it's committed index. Still likely less than the current actual index, but worth it for the replay.
-	if snap2 != nil {
+	if snap2 != nil && st2.Commit < snap2.Metadata.Index {
 		st2.Commit = snap2.Metadata.Index
 	}
 
@@ -175,8 +175,8 @@ func GuessNodeID(nodes map[string]uint64, snap4 *Snapshot4, cfg *Config4, name s
 			delete(snapNodes, p.Name)
 		}
 		if len(snapNodes) == 1 {
-			for name, id := range nodes {
-				log.Printf("Autodetected from snapshot: name %s", name)
+			for nodename, id := range nodes {
+				log.Printf("Autodetected from snapshot: name %s", nodename)
 				return id
 			}
 		}
@@ -186,8 +186,8 @@ func GuessNodeID(nodes map[string]uint64, snap4 *Snapshot4, cfg *Config4, name s
 		delete(nodes, p.Name)
 	}
 	if len(nodes) == 1 {
-		for name, id := range nodes {
-			log.Printf("Autodetected name %s", name)
+		for nodename, id := range nodes {
+			log.Printf("Autodetected name %s", nodename)
 			return id
 		}
 	}

migrate/functional/README.md

@@ -0,0 +1,27 @@
+
+etcd migration functional tests
+=====
+
+This functional test suite deploys a etcd cluster using processes, and asserts etcd is functioning properly.
+
+Dependencies
+------------
+
+The test suite can only be run in CoreOS system. It's recommended to run this in a virtual machine environment on CoreOS (e.g. using coreos-vagrant). The only dependency for the tests not provided on the CoreOS image is go.
+
+Usage
+-----
+
+Set environment variables point to the respective binaries that are used to drive the actual tests:
+
+```
+$ export ETCD_V1_BIN=/path/to/v1_etcd
+$ export ETCD_V2_BIN=/path/to/v2_etcd
+$ export ETCDCTL_BIN=/path/to/etcdctl
+```
+
+Then the tests can be run:
+
+```
+$ go test github.com/coreos/etcd/migrate/functional
+```

migrate/functional/fixtures/ca.crt

@@ -0,0 +1,30 @@
+-----BEGIN CERTIFICATE-----
+MIIFNDCCAx6gAwIBAgIBATALBgkqhkiG9w0BAQUwLTEMMAoGA1UEBhMDVVNBMRAw
+DgYDVQQKEwdldGNkLWNhMQswCQYDVQQLEwJDQTAeFw0xNDAzMTMwMjA5MDlaFw0y
+NDAzMTMwMjA5MDlaMC0xDDAKBgNVBAYTA1VTQTEQMA4GA1UEChMHZXRjZC1jYTEL
+MAkGA1UECxMCQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDdlBlw
+Jiakc4C1UpMUvQ+2fttyBMfMLivQgj51atpKd8qIBvpZwz1wtpzdRG0hSYMF0IUk
+MfBqyg+T5tt2Lfs3Gx3cYKS7G0HTfmABC7GdG8gNvEVNl/efxqvhis7p7hur765e
+J+N2GR4oOOP5Wa8O5flv10cp3ZJLhAguc2CONLzfh/iAYAItFgktGHXJ/AnUhhaj
+KWdKlK9Cv71YsRPOiB1hCV+LKfNSqrXPMvQ4sarz3yECIBhpV/KfskJoDyeNMaJd
+gabX/S7gUCd2FvuOpGWdSIsDwyJf0tnYmQX5XIQwBZJib/IFMmmoVNYc1bFtYvRH
+j0g0Ax4tHeXU/0mglqEcaTuMejnx8jlxZAM8Z94wHLfKbtaP0zFwMXkaM4nmfZqh
+vLZwowDGMv9M0VRFEhLGYIc3xQ8G2u8cFAGw1UqTxKhwAdRmrcFaQ38sk4kziy0u
+AkpGavS7PKcFjjB/fdDFO/kwGQOthX/oTn9nP3BT+IK2h1A6ATMPI4lVnhb5/KBt
+9M/fGgbiU+I9QT0Ilz/LlrcCuzyRXREvIZvoUL77Id+JT3qQxqPn/XMKLN4WEFII
+112MFGqCD85JZzNoC4RkZd8kFlR4YJWsS4WqJlWprESr5cCDuLviK+31cnIRF4fJ
+mz0gPsVgY7GFEan3JJnL8oRUVzdTPKfPt0atsQIDAQABo2MwYTAOBgNVHQ8BAf8E
+BAMCAAQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUnVlVvktY+zlLpG43nTpG
+AWmUkrYwHwYDVR0jBBgwFoAUnVlVvktY+zlLpG43nTpGAWmUkrYwCwYJKoZIhvcN
+AQEFA4ICAQAqIcPFux3V4h1N0aGM4fCS/iT50TzDnRb5hwILKbmyA6LFnH4YF7PZ
+aA0utDNo1XSRDMpR38HWk0weh5Sfx6f2danaKZHAsea8oVEtdrz16ZMOvoh0CPIM
+/hn0CGQOoXDADDNFASuExhhpoyYkDqTVTCQ/zbhZg1mjBljJ+BBzlSgeoE4rUDpn
+nuDcmD9LtjpsVQL+J662rd51xV4Z6a7aZLvN9GfO8tYkfCGCD9+fGh1Cpz0IL7qw
+VRie+p/XpjoHemswnRhYJ4wn10a1UkVSR++wld6Gvjb9ikyr9xVyU5yrRM55pP2J
+VguhzjhTIDE1eDfIMMxv3Qj8+BdVQwtKFD+zQYQcbcjsvjTErlS7oCbM2DVlPnRT
+QaCM0q0yorfzc4hmml5P95ngz2xlohavgNMhsYIlcWyq3NVbm7mIXz2pjqa16Iit
+vL7WX6OVupv/EOMRx5cVcLqqEaYJmAlNd/CCD8ihDQCwoJ6DJhczPRexrVp+iZHK
+SnIUONdXb/g8ungXUGL1jGNQrWuq49clpI5sLWNjMDMFAQo0qu5bLkOIMlK/evCt
+gctOjXDvGXCk5h6Adf14q9zDGFdLoxw0/aciUSn9IekdzYPmkYUTifuzkVRsPKzS
+nmI4dQvz0rHIh4FBUKWWrJhRWhrv9ty/YFuJXVUHeAwr5nz6RFZ4wQ==
+-----END CERTIFICATE-----

migrate/functional/fixtures/ca.key

@@ -0,0 +1,54 @@
+-----BEGIN RSA PRIVATE KEY-----
+Proc-Type: 4,ENCRYPTED
+DEK-Info: DES-EDE3-CBC,d2e70acc12a86116
+
+uAmKZ41MiYTa7CappCFEcLL/kWRX4rE8DJG3sL59lv3j/6bYFkdczy3kgrEWm4Pn
++pveJEssQkszXHkjA3vHx8nlTvfQOwa7ggcc76LNYj1sPHawVRNA0pb6WvjDzN7D
+JMgAnptVuZGP8N6ZIzFvr5Rf58ar5Y2aI7Ti6KxLZvqYojgvz5dzGimC3+SwDlFy
+Q2kwBA/HT4X9w2qSxpQ7WGPw2pkYILZ4Nxfqh9PWHd0Pk1d9KoLhbU5LEtGSy/y9
+9jqKsUqBzp9905t7d2KmFDF9Nd7XvHrDZDPILlKcQYnBxg6c1ChH1NkIqdAW7lQ6
+dAKAFZlMpVb/ArFBjhioljBIO+gLcWxYseHXbteOgbC1cw5xcBTHqH+7RotFH1VO
+ya0DFeW2CyPj4mp7vORD+IOVQaG4H5j1vJXqA9OPBziZR+lHvD0gVJqZIquXIQlW
+MBpX5CfV/3xITb6o0wA2OG2qlNM+VbKzg/cqh/kkusAqcfXIByh16K85k4jwPrBG
+wsYWABgw1vLlrCJ7ug6P2rb6VmzTbMqe4gpqUROgCS36ARjs5eDBDYZsX6NaGSh6
+twAUfzpwoGNuHwUpIYf5BjH1me+tnM0S8tAEtCFf9hy88nCg6v22cWQuAD6+6P6B
+Skl/UYT4sxeeETFv7Vf70wLnBMA3/uymBM75FhPyD5Vvg9fxz7aAJbfB2ovUVZ/v
+l3HCsCo8y7DtEXoiBmPCH28JWVhIZbmP3dYnU8c86SubhNWm0yjJIIwoghyFmCcO
+Wjs0XkVUUa9fGrl6Mc6XQIGsS6UdQkFoIcO+dtIFPg5C5GWnPnF53ro0J4pGcyR0
+zgt9ubCcFKNz5Cbcfw7fKJwswMt6zXtFxE/tVvOq2EPAPrmYYwPrnvbSNbuVL+as
+OT5ukITR9MDsYR/19jFUsdRDjSvUQVwqH7PiKwTnZouuJUhYHfj3Bjhz6cWzadcd
+pNdxqSgEeSzvaz390p1dOpN/0d1ItXlp3za6JZUarVkx8yH9UCFfpEEisPYgTASf
+F2xIrWHgZY+87OjPluU+Gym12ldcs0dbySgsxhKZMyAUd0DB2Knnmug+cqVvN+xo
+rJ2pD7J08zmQSRGyAUsbeUnuGb6fGNxaD5QpEN7nK4x3K1Q5N9QQ3RwL4Ik6jV0N
+eO0LzXF/BZbOAvl/OXAse1f5c7FO21oUw6u6iI0xvTJAcnaH/0eE2N6Y9Lwt507K
+HxhuN5j58/sOeb6kfkX563SoKSdYSrBqIaogDZFCtKpEBevsRM+QRdzAc//Fm67U
+Zs2K/ADM8+IaQN7uhm8IAPtWEnJ5+9rM2PCF0NX+7qa9HtZxTd0cqbeL8Ayx4i/T
+dHvN8k3kPuC+6He7+eZR6EQpN5GPt5SX3QGgKOQbbwBgF8mS/R0zaZpHvaqTY4Bi
+RfsLbRBGoTvR8YjqaQW91tExe5FghH7k02slSGzEzgs/ZhqPMCLNC7uFcSKcx9jA
+Bj+GmrYOMrUOYLQPT1iRtBFjLEUGPlvUGlaJS/JcvBN6DPW375tQHk7kbpVcudPh
+6vVXftuDiYEJk1TIQLt3QdC9s6ieVuAds4KDjYaTZz4s5W2Lkwo5AZzwLeMRank1
+96okoO1qRaDgagHsG8yPIwq+8/b/8dNl7E+wsbAWwLXLhYZGqDmHm/16pv/Ck59W
+LXLoJfrOdKBoxTTZulIsTISZ14Bj87QWPW26kI6So9V5vN60rb2MWrd+HU46Qapi
+JCsfCVsi715GUh4IkqAnec26TuXW2THcOp3p19SyubuJ33XqUR9H7BOZuBsIFeZV
+8sihbgjJ/zb7fZ7AGT3VmAxEtgFi8u2NOBN/WqYb++khtXgnIbOhBx9PuhOBofrO
+4M0R5s6F2SpbX2LEBJFN48wIlRmSMTsKdmZmA7f0IuxjYIcotBdRCGoXRlJJnZeH
+7WriXQJsq0517GlrqgYMDx26xHJy/ao+zcDxsCtftzAQvENuGr1lzsCdIcGXs+FU
+7C8qdmqSXgZgltFQpyR7+PMikXcdYdzkT3BjFh+VKJNiAeGXNnVXQH7L/V49zaij
+BRYWWtHwEDz50vSzZz3fnrFl6Pk8tny4bKoLjB4vBjMlb4yte7LcK+vbfDdreISD
+cDqfpzjAmIpv1GoQFKWGLQjagvwiAfOA8GUivEG9SQSAAImkV9qkr5qYzM7Jn2WU
+icA8D0YfuILpGxTOQc1SgDMOiGboCB+f7cxPsjXHbVahNyxxAbDbTjbc6v7q1oiy
+PESoLaBR0Bi0tdKivvbB63ok2Kq9XneFrQeCIyrhkXIvYDEwdcoCBpL1DEotbU+D
+YjZTLr4UW92xi1M4d94zmG6pyJsfC4sHGflY5paml9dLiEy78rCPfrJkrSSUplf+
+8CjfUoZsbq3haE0N4TbqV0I0W2Fm/a6U113CTRYxj9DeA3m/HFU3TLzk9Vg/vGxP
+/xltsu/wd/GoyoD9OhWhW1Ck9dtQ0G64hQjeXVd/pzsDCMT8hrtKSlX1Q7vK96ml
+OJ9Ju/CdhX2lJA8BrGVh4HS1fsuNFjr5KqZAY6MwFpjAPqvqD7WFE3Yflk5/7VtX
+bsvBZoN2vp9hprXsgm8/KmSNnWxzQY1Nps4XjRJVYeTmND5EyQClGJyCYKg0QVDo
+7L/2GAhnOrSLkAHOcYAlrNhZ85yBiLhjJcvWyT6DDcMpCusgictI2Qv2ZjMmz46v
+62PzHm0/Z3yQMcJnpRO79OdodbY22Eg9xZGGhBp1Xbm/OXYLaEpGW9S7DqPvlD5v
+O+VxENxJNwDELK9H2auGJAQdORwgF0VfvZxN6tGRyb7eI6aJj04YYMBkg5Nds+AR
+sNEdGNzqKm8sWvINSoX+BCOyjElOSRW0glK+ala5Y7/mM3+KOWgMas2LZBcLZfBr
+1/Z0DPIA2CkFtT1VsBKa+fSkEN0s+PRLRV/QWrcMbkSvIaKcswMwoyvI6OcddUEz
+YgjAOZ3TdnRm1DMqZHIsPOj+3xQv6nETqSwhvLJT1wJwnJQVbxjZwoUmJKSsZDEB
+2xL9OWlhFNY2qS7F77vv2ZUJYLYniiTGrC09AAQ4ti8zWnY1gqtaCp+1wynt/Abs
+9gGcbEIaQGWhpVjPtlKjNm86jGP0IXPaAgaOViIuBH+0GeVOLuUMLvb0nL0NWMJa
+-----END RSA PRIVATE KEY-----

migrate/functional/fixtures/server.crt

@@ -0,0 +1,31 @@
+-----BEGIN CERTIFICATE-----
+MIIFWzCCA0WgAwIBAgIBAjALBgkqhkiG9w0BAQUwLTEMMAoGA1UEBhMDVVNBMRAw
+DgYDVQQKEwdldGNkLWNhMQswCQYDVQQLEwJDQTAeFw0xNDAzMTMwMjA5MjJaFw0y
+NDAzMTMwMjA5MjJaMEUxDDAKBgNVBAYTA1VTQTEQMA4GA1UEChMHZXRjZC1jYTEP
+MA0GA1UECxMGc2VydmVyMRIwEAYDVQQDEwkxMjcuMC4wLjEwggIiMA0GCSqGSIb3
+DQEBAQUAA4ICDwAwggIKAoICAQDI3EvqJrLWsnPbjAT8ENiMRyBINhhafubi5Nb+
+glEzkbC2kv2zXkVkpkBubDRwyh3eomSbdwKYk3yz+IopT753teJueRpMPq9Ayr/+
+PZl4Y1tG04KcjfOvOls6zPsDfHzluR8TE705If5wwZu3Bdwxzdtx9T0ROzIEgRt0
+Axuce5qkg93IWNxOrIr+4LCxYfTpvpTXO20lz0IuQNm1Opo9PVoWn7PXdOmuCzSG
+2hW1DcKqSyQP7IkplBJS0EhoovIsXavSkPKJssvQj73ZFIBVgKhXuHmPNdrypaQk
+CtxsqbVdOOlojItqYTTDAiadwRQWkYgDOSQCGJiPqYVJx+rH4MlzxQ6n9x2qIcne
+lfMr+VFDEc1YvHu1XLMg5b1ImD6ChutYW0RhFJ3CQVdQR2i4kJ8T1DSJYLISMODZ
+ux1cZaUoSL/EkrC5/8POWZmP8nJXO6A4wrZDHF30/qWpo+T5PvsA6cABfX1jkcTx
+PBXGK1qOZ8rToTxprJ2zc3zuZNxSgM32nzjcPUgn559Mgdl0HR4c4JeTZGsebWmx
+MWmkz//BV4eUaGHqCpzRQHf3YIxysvDC2Xf4z2Alk8AlLRXp7/ksatdxAtyc+y8+
+MWCc6N0YbI9zjv+ezCBqR+mu1P5Tb0HebPFz3dOdIpiC3kU8QyMEagw8u5xliZs4
+AxwdNwIDAQABo3IwcDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwHQYD
+VR0OBBYEFD6UrVN8uolWz6et79jVeZetjd4XMB8GA1UdIwQYMBaAFJ1ZVb5LWPs5
+S6RuN506RgFplJK2MA8GA1UdEQQIMAaHBH8AAAEwCwYJKoZIhvcNAQEFA4ICAQCo
+sKn1Rjx0tIVWAZAZB4lCWvkQDp/txnb5zzQUlKhIW2o98IklASmOYYyZbE2PXlda
+/n8TwKIzWgIoNh5AcgLWhtASrnZdGFXY88n5jGk6CVZ1+Dl+IX99h+r+YHQzf1jU
+BjGrZHGv3pPjwhFGDS99lM/TEBk/eLI2Kx5laL+nWMTwa8M1OwSIh6ZxYPVlWUqb
+rurk5l/YqW+UkYIXIQhe6LwtB7tBjr6nDIWBfHQ7uN8IdB8VIAF6lejr22VmERTW
+j+zJ5eTzuQN1f0s930mEm8pW7KgGxlEqrUlSJtxlMFCv6ZHZk1Y4yEiOCBKlPNme
+X3B+lhj//PH3gLNm3+ZRr5ena3k+wL9Dd3d3GDCIx0ERQyrGS/rJpqNPI+8ZQlG0
+nrFlm7aP6UznESQnJoSFbydiD0EZ4hXSdmDdXQkTklRpeXfMcrYBGN7JrGZOZ2T2
+WtXBMx2bgPeEH50KRrwUMFe122bchh0Fr+hGvNK2Q9/gRyQPiYHq6vSF4GzorzLb
+aDuWA9JRH8/c0z8tMvJ7KjmmmIxd39WWGZqiBrGQR7utOJjpQl+HCsDIQM6yZ/Bu
+RpwKj2yBz0OQg4tWbtqUuFkRMTkCR6vo3PadgO1VWokM7UFUXlScnYswcM5EwnzJ
+/IsYJ2s1V706QVUzAGIbi3+wYi3enk7JfYoGIqa2oA==
+-----END CERTIFICATE-----

migrate/functional/fixtures/server.key.insecure

@@ -0,0 +1,51 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIJKAIBAAKCAgEAyNxL6iay1rJz24wE/BDYjEcgSDYYWn7m4uTW/oJRM5GwtpL9
+s15FZKZAbmw0cMod3qJkm3cCmJN8s/iKKU++d7XibnkaTD6vQMq//j2ZeGNbRtOC
+nI3zrzpbOsz7A3x85bkfExO9OSH+cMGbtwXcMc3bcfU9ETsyBIEbdAMbnHuapIPd
+yFjcTqyK/uCwsWH06b6U1zttJc9CLkDZtTqaPT1aFp+z13Tprgs0htoVtQ3Cqksk
+D+yJKZQSUtBIaKLyLF2r0pDyibLL0I+92RSAVYCoV7h5jzXa8qWkJArcbKm1XTjp
+aIyLamE0wwImncEUFpGIAzkkAhiYj6mFScfqx+DJc8UOp/cdqiHJ3pXzK/lRQxHN
+WLx7tVyzIOW9SJg+gobrWFtEYRSdwkFXUEdouJCfE9Q0iWCyEjDg2bsdXGWlKEi/
+xJKwuf/DzlmZj/JyVzugOMK2Qxxd9P6lqaPk+T77AOnAAX19Y5HE8TwVxitajmfK
+06E8aayds3N87mTcUoDN9p843D1IJ+efTIHZdB0eHOCXk2RrHm1psTFppM//wVeH
+lGhh6gqc0UB392CMcrLwwtl3+M9gJZPAJS0V6e/5LGrXcQLcnPsvPjFgnOjdGGyP
+c47/nswgakfprtT+U29B3mzxc93TnSKYgt5FPEMjBGoMPLucZYmbOAMcHTcCAwEA
+AQKCAgBS1vCESKOXgo/f61ae8v+skyUQQyc2I4Jr739wBiUhRKQCGIuDr4ylHyAR
+qpTSM7mv+X/O0n2CmcljnEy3Dwl568zQTSf4bB3xde1LGPKzwR6DDnaexLjM+x9n
+F+UqoewM/pV/U7PF3WxH6sGi8UrIS6OG02L1OVm+m9TLuwBnQF8eHLiaiXOLCwRk
+bBzTe5f70zslrX+tiVY9J0fiw6GbQjNmg0UzxicePcbTGxy6yEsR2t2rp51GRahs
++TPz28hPXe6gcGFnQxNmF/JvllH7cY18aDvSQZ7kVkZlCwmv0ypWoUM6eESDgkW1
+a6yrgVccm7bhxW5BYw2AqqSrMkV0oMcCUjh2rYvex7w6dM374Ok3DD/dXjTHLNV5
++0tHMxXUiCKwe7hVEg+iGD4E1jap5n5c4RzpEtAXsGEK5WUBksHi9qOBv+lubjZn
+Kcfbos+BbnmUCU3MmU48EZwyFQIu9djkLXfJV2Cbbg9HmkrIOYgi4tFjoBKeQLE4
+6GCucMWnNfMO7Kq/z7c+7sfWOAA55pu0Ojel8VH6US+Y/1mEuSUhQudrJn8GxAmc
+4t+C2Ie1Q1bK3iJbd0NUqtlwd9xI9wQgCbaxfQceUmBBjuTUu3YFctZ7Jia7h18I
+gZ3wsKfySDhW29XTFvnT3FUpc+AN9Pv4sB7uobm6qOBV8/AdKQKCAQEA1zwIuJki
+bSgXxsD4cfKgQsyIk0eMj8bDOlf/A8AFursXliH3rRASoixXNgzWrMhaEIE2BeeT
+InE13YCUjNCKoz8oZJqKYpjh3o/diZf1vCo6m/YUSR+4amynWE4FEAa58Og2WCJ3
+Nx8/IMpmch2VZ+hSQuNr5uvpH84+eZADQ1GB6ypzqxb5HjIEeryLJecDQGe4ophd
+JCo3loezq/K0XJQI8GTBe2GQPjXSmLMZKksyZoWEXAaC1Q+sdJWZvBpm3GfVQbXu
+q7wyqTMknVIlEOy0sHxstsbayysSFFQ/fcgKjyQb8f4efOkyQg8mH5vQOZghbHJ+
+7I8wVSSBt+bE2wKCAQEA7udRoo2NIoIpJH+2+SPqJJVq1gw/FHMM4oXNZp+AAjR1
+hTWcIzIXleMyDATl5ZFzZIY1U2JMifS5u2R7fDZEu9vfZk4e6BJUJn+5/ahjYFU8
+m8WV4rFWR6XN0SZxPb43Mn6OO7EoMqr8InRufiN4LwIqnPqDm2D9Fdijb9QFJ2UG
+QLKNnIkLTcUfx1RYP4T48CHkeZdxV8Cp49SzSSV8PbhIVBx32bm/yO6nLHoro7Wl
+YqXGW0wItf2BUA5a5eYNO0ezVkOkTp2aj/p9i+0rqbsYa480hzlnOzYI5F72Z8V2
+iPltUAeQn53Vg1azySa1x8/0Xp5nVsgQSh18CH3p1QKCAQBxZv4pVPXgkXlFjTLZ
+xr5Ns7pZ7x7OOiluuiJw9WGPazgYMDlxA8DtlXM11Tneu4lInOu73LGXOhLpa+/Y
+6Z/CN2qu5wX2wRpwy1gsQNaGl7FdryAtDvt5h1n8ms7sDL83gQHxGee6MUpvmnSz
+t4aawrtk5rJZbv7bdS1Rm2E8vNs47psXD/mdwTi++kxOYhNCgeO0N5cLkPrM4x71
+f+ErzguPrWaL/XGkdXNKZULjF8+sWLjOS9fvLlzs6E2h4D9F7addAeCIt5XxtDKc
+eUVyT2U8f7I/8zIgTccu0tzJBvcZSCs5K20g3zVNvPGXQd9KGS+zFfht51vN4HhA
+TuR1AoIBAGuQBKZeexP1bJa9VeF4dRxBldeHrgMEBeIbgi5ZU+YqPltaltEV5Z6b
+q1XUArpIsZ6p+mpvkKxwXgtsI1j6ihnW1g+Wzr2IOxEWYuQ9I3klB2PPIzvswj8B
+/NfVKhk1gl6esmVXzxR4/Yp5x6HNUHhBznPdKtITaf+jCXr5B9UD3DvW6IF5Bnje
+bv9tD0qSEQ71A4xnTiXHXfZxNsOROA4F4bLVGnUR97J9GRGic/GCgFMY9mT2p9lg
+qQ8lV3G5EW4GS01kqR6oQQXgLxSIFSeXUFhlIq5bfwoeuwQvaVuxgTwMqVXmAgyL
+oK1ApTPE1QWAsLLFORvOed8UxVqBbn0CggEBALfr/wheXCKLdzFzm03sO1i9qVz2
+vnpxzexXW3V/TtM6Dff2ojgkDC+CVximtAiLA/Wj60hXnQxw53g5VVT5rESx0J3c
+pq+azbi1eWzFeOrqJvKQhMfYc0nli7YuGnPkKzeepJJtWZHYkAjL4QZAn1jt0RqV
+DQmlGPGiOuGP8uh59c23pbjgh4eSJnvhOT2BFKhKZpBdTBYeiQiZBqIyme8rNTFr
+NmpBxtUr77tccVTrcWWhhViG36UNpetAP7b5QCHScIXZJXrEqyK5HaePqi5UMH8o
+alSz6s2REG/xP7x54574TvRG/3cIamv1AfZAOjin7BwhlSLhPl2eeh4Cgas=
+-----END RSA PRIVATE KEY-----

migrate/functional/member.go

@@ -0,0 +1,278 @@
+// Copyright 2015 CoreOS, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package functional
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+	"log"
+	"net/http"
+	"net/url"
+	"os"
+	"os/exec"
+	"strconv"
+	"strings"
+	"time"
+)
+
+var binDir = ".versions"
+
+type Proc struct {
+	*exec.Cmd
+	Name    string
+	DataDir string
+	URL     string
+	PeerURL string
+
+	stderr io.ReadCloser
+}
+
+func NewProcWithDefaultFlags(path string) *Proc {
+	var args []string
+	dir, err := ioutil.TempDir(os.TempDir(), "etcd")
+	if err != nil {
+		fmt.Printf("unexpected TempDir error: %v", err)
+		os.Exit(1)
+	}
+	args = append(args, "--data-dir="+dir)
+	args = append(args, "--name=default")
+	p := &Proc{
+		Cmd:     exec.Command(path, args...),
+		Name:    "default",
+		DataDir: dir,
+		URL:     "http://127.0.0.1:4001",
+		PeerURL: "http://127.0.0.1:7001",
+	}
+	// always expect to use start_desired_verson mode
+	p.Env = append(p.Env,
+		"ETCD_BINARY_DIR="+binDir,
+	)
+	return p
+}
+
+func NewProcWithV1Flags(path string) *Proc {
+	p := NewProcWithDefaultFlags(path)
+	p.SetV1PeerAddr("127.0.0.1:7001")
+	return p
+}
+
+func NewProcWithV2Flags(path string) *Proc {
+	p := NewProcWithDefaultFlags(path)
+	p.SetV2PeerURL("http://127.0.0.1:7001")
+	return p
+}
+
+func (p *Proc) SetV2PeerURL(url string) {
+	p.Args = append(p.Args,
+		"-listen-peer-urls="+url,
+		"-initial-advertise-peer-urls="+url,
+		"-initial-cluster",
+		p.Name+"="+url,
+	)
+	p.PeerURL = url
+}
+
+func (p *Proc) SetV1PeerAddr(addr string) {
+	p.Args = append(p.Args,
+		"-peer-addr="+addr,
+	)
+	p.PeerURL = "http://" + addr
+}
+
+func (p *Proc) SetV1Addr(addr string) {
+	p.Args = append(p.Args,
+		"-addr="+addr,
+	)
+	p.URL = "http://" + addr
+}
+
+func (p *Proc) SetV1Peers(peers []string) {
+	p.Args = append(p.Args,
+		"-peers="+strings.Join(peers, ","),
+	)
+}
+
+func (p *Proc) SetName(name string) {
+	p.Args = append(p.Args,
+		"-name="+name,
+	)
+	p.Name = name
+}
+
+func (p *Proc) SetDataDir(dataDir string) {
+	p.Args = append(p.Args,
+		"-data-dir="+dataDir,
+	)
+	p.DataDir = dataDir
+}
+
+func (p *Proc) SetSnapCount(cnt int) {
+	p.Args = append(p.Args,
+		"-snapshot-count="+strconv.Itoa(cnt),
+	)
+}
+
+func (p *Proc) SetDiscovery(url string) {
+	p.Args = append(p.Args,
+		"-discovery="+url,
+	)
+}
+
+func (p *Proc) SetPeerTLS(certFile, keyFile, caFile string) {
+	p.Args = append(p.Args,
+		"-peer-cert-file="+certFile,
+		"-peer-key-file="+keyFile,
+		"-peer-ca-file="+caFile,
+	)
+	u, err := url.Parse(p.PeerURL)
+	if err != nil {
+		log.Panicf("unexpected parse error: %v", err)
+	}
+	u.Scheme = "https"
+	p.PeerURL = u.String()
+}
+
+func (p *Proc) CleanUnsuppportedV1Flags() {
+	var args []string
+	for _, arg := range p.Args {
+		if !strings.HasPrefix(arg, "-peers=") {
+			args = append(args, arg)
+		}
+	}
+	p.Args = args
+}
+
+func (p *Proc) Start() error {
+	if err := p.Cmd.Start(); err != nil {
+		return err
+	}
+	for k := 0; k < 50; k++ {
+		_, err := http.Get(p.URL)
+		if err == nil {
+			return nil
+		}
+		time.Sleep(100 * time.Millisecond)
+	}
+	return fmt.Errorf("instance %s failed to be available after a long time", p.Name)
+}
+
+func (p *Proc) Stop() {
+	if err := p.Cmd.Process.Kill(); err != nil {
+		fmt.Printf("Process Kill error: %v", err)
+		return
+	}
+	p.Cmd.Wait()
+}
+
+func (p *Proc) Restart() error {
+	p.Stop()
+	return p.Start()
+}
+
+func (p *Proc) Terminate() {
+	p.Stop()
+	os.RemoveAll(p.DataDir)
+}
+
+type ProcGroup []*Proc
+
+func NewProcInProcGroupWithV1Flags(path string, num int, idx int) *Proc {
+	pg := NewProcGroupWithV1Flags(path, num)
+	return pg[idx]
+}
+
+func NewProcGroupWithV1Flags(path string, num int) ProcGroup {
+	pg := make([]*Proc, num)
+	for i := 0; i < num; i++ {
+		pg[i] = NewProcWithDefaultFlags(path)
+		pg[i].SetName(fmt.Sprintf("etcd%d", i))
+		pg[i].SetV1PeerAddr(fmt.Sprintf("127.0.0.1:%d", 7001+i))
+		pg[i].SetV1Addr(fmt.Sprintf("127.0.0.1:%d", 4001+i))
+		if i > 0 {
+			pg[i].SetV1Peers([]string{"127.0.0.1:7001"})
+		}
+	}
+	return pg
+}
+
+func NewProcGroupViaDiscoveryWithV1Flags(path string, num int, url string) ProcGroup {
+	pg := make([]*Proc, num)
+	for i := range pg {
+		pg[i] = NewProcWithDefaultFlags(path)
+		pg[i].SetName(fmt.Sprintf("etcd%d", i))
+		pg[i].SetDiscovery(url)
+		pg[i].SetV1PeerAddr(fmt.Sprintf("127.0.0.1:%d", 7001+i))
+		pg[i].SetV1Addr(fmt.Sprintf("127.0.0.1:%d", 4001+i))
+	}
+	return pg
+}
+
+func (pg ProcGroup) SetPeerTLS(certFile, keyFile, caFile string) {
+	for i := range pg {
+		pg[i].SetPeerTLS(certFile, keyFile, caFile)
+	}
+}
+
+func (pg ProcGroup) InheritDataDir(opg ProcGroup) {
+	for i := range pg {
+		pg[i].SetDataDir(opg[i].DataDir)
+	}
+}
+
+func (pg ProcGroup) SetSnapCount(count int) {
+	for i := range pg {
+		pg[i].SetSnapCount(count)
+	}
+}
+
+func (pg ProcGroup) CleanUnsuppportedV1Flags() {
+	for _, p := range pg {
+		p.CleanUnsuppportedV1Flags()
+	}
+}
+
+func (pg ProcGroup) Start() error {
+	for _, p := range pg {
+		if err := p.Start(); err != nil {
+			return err
+		}
+	}
+	// leave time for instances to sync and write some entries into disk
+	// TODO: use more reliable method
+	time.Sleep(time.Second)
+	return nil
+}
+
+func (pg ProcGroup) Wait() error {
+	for _, p := range pg {
+		if err := p.Wait(); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (pg ProcGroup) Stop() {
+	for _, p := range pg {
+		p.Stop()
+	}
+}
+
+func (pg ProcGroup) Terminate() {
+	for _, p := range pg {
+		p.Terminate()
+	}
+}

migrate/functional/upgrade_test.go

@@ -0,0 +1,414 @@
+package functional
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"os"
+	"os/exec"
+	"path"
+	"path/filepath"
+	"testing"
+)
+
+var (
+	v1BinPath      = path.Join(binDir, "1")
+	v2BinPath      = path.Join(binDir, "2")
+	etcdctlBinPath string
+)
+
+func init() {
+	os.RemoveAll(binDir)
+	if err := os.Mkdir(binDir, 0700); err != nil {
+		fmt.Printf("unexpected Mkdir error: %v\n", err)
+		os.Exit(1)
+	}
+	if err := os.Symlink(absPathFromEnv("ETCD_V1_BIN"), v1BinPath); err != nil {
+		fmt.Printf("unexpected Symlink error: %v\n", err)
+		os.Exit(1)
+	}
+	if err := os.Symlink(absPathFromEnv("ETCD_V2_BIN"), v2BinPath); err != nil {
+		fmt.Printf("unexpected Symlink error: %v\n", err)
+		os.Exit(1)
+	}
+	etcdctlBinPath = os.Getenv("ETCDCTL_BIN")
+
+	mustExist(v1BinPath)
+	mustExist(v2BinPath)
+	mustExist(etcdctlBinPath)
+}
+
+func TestStartNewMember(t *testing.T) {
+	tests := []*Proc{
+		NewProcWithDefaultFlags(v2BinPath),
+		NewProcWithV1Flags(v2BinPath),
+		NewProcWithV2Flags(v2BinPath),
+	}
+	for i, tt := range tests {
+		if err := tt.Start(); err != nil {
+			t.Fatalf("#%d: Start error: %v", i, err)
+		}
+		defer tt.Terminate()
+
+		ver, err := checkInternalVersion(tt.URL)
+		if err != nil {
+			t.Fatalf("#%d: checkVersion error: %v", i, err)
+		}
+		if ver != "2" {
+			t.Errorf("#%d: internal version = %s, want %s", i, ver, "2")
+		}
+	}
+}
+
+func TestStartV2Member(t *testing.T) {
+	tests := []*Proc{
+		NewProcWithDefaultFlags(v2BinPath),
+		NewProcWithV1Flags(v2BinPath),
+		NewProcWithV2Flags(v2BinPath),
+	}
+	for i, tt := range tests {
+		// get v2 data dir
+		p := NewProcWithDefaultFlags(v2BinPath)
+		if err := p.Start(); err != nil {
+			t.Fatalf("#%d: Start error: %v", i, err)
+		}
+		p.Stop()
+		tt.SetDataDir(p.DataDir)
+		if err := tt.Start(); err != nil {
+			t.Fatalf("#%d: Start error: %v", i, err)
+		}
+		defer tt.Terminate()
+
+		ver, err := checkInternalVersion(tt.URL)
+		if err != nil {
+			t.Fatalf("#%d: checkVersion error: %v", i, err)
+		}
+		if ver != "2" {
+			t.Errorf("#%d: internal version = %s, want %s", i, ver, "2")
+		}
+	}
+}
+
+func TestStartV1Member(t *testing.T) {
+	tests := []*Proc{
+		NewProcWithDefaultFlags(v2BinPath),
+		NewProcWithV1Flags(v2BinPath),
+		NewProcWithV2Flags(v2BinPath),
+	}
+	for i, tt := range tests {
+		// get v1 data dir
+		p := NewProcWithDefaultFlags(v1BinPath)
+		if err := p.Start(); err != nil {
+			t.Fatalf("#%d: Start error: %v", i, err)
+		}
+		p.Stop()
+		tt.SetDataDir(p.DataDir)
+		if err := tt.Start(); err != nil {
+			t.Fatalf("#%d: Start error: %v", i, err)
+		}
+		defer tt.Terminate()
+
+		ver, err := checkInternalVersion(tt.URL)
+		if err != nil {
+			t.Fatalf("#%d: checkVersion error: %v", i, err)
+		}
+		if ver != "1" {
+			t.Errorf("#%d: internal version = %s, want %s", i, ver, "1")
+		}
+	}
+}
+
+func TestUpgradeV1Cluster(t *testing.T) {
+	// get v2-desired v1 data dir
+	pg := NewProcGroupWithV1Flags(v1BinPath, 3)
+	if err := pg.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	cmd := exec.Command(etcdctlBinPath, "upgrade", "--peer-url", pg[1].PeerURL)
+	if err := cmd.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	if err := cmd.Wait(); err != nil {
+		t.Fatalf("Wait error: %v", err)
+	}
+	t.Logf("wait until etcd exits...")
+	if err := pg.Wait(); err != nil {
+		t.Fatalf("Wait error: %v", err)
+	}
+
+	npg := NewProcGroupWithV1Flags(v2BinPath, 3)
+	npg.InheritDataDir(pg)
+	npg.CleanUnsuppportedV1Flags()
+	if err := npg.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	defer npg.Terminate()
+
+	for _, p := range npg {
+		ver, err := checkInternalVersion(p.URL)
+		if err != nil {
+			t.Fatalf("checkVersion error: %v", err)
+		}
+		if ver != "2" {
+			t.Errorf("internal version = %s, want %s", ver, "2")
+		}
+	}
+}
+
+func TestUpgradeV1SnapshotedCluster(t *testing.T) {
+	// get v2-desired v1 data dir
+	pg := NewProcGroupWithV1Flags(v1BinPath, 3)
+	pg.SetSnapCount(10)
+	if err := pg.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	cmd := exec.Command(etcdctlBinPath, "upgrade", "--peer-url", pg[1].PeerURL)
+	if err := cmd.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	if err := cmd.Wait(); err != nil {
+		t.Fatalf("Wait error: %v", err)
+	}
+	t.Logf("wait until etcd exits...")
+	if err := pg.Wait(); err != nil {
+		t.Fatalf("Wait error: %v", err)
+	}
+	for _, p := range pg {
+		// check it has taken snapshot
+		fis, err := ioutil.ReadDir(path.Join(p.DataDir, "snapshot"))
+		if err != nil {
+			t.Fatalf("unexpected ReadDir error: %v", err)
+		}
+		if len(fis) == 0 {
+			t.Fatalf("unexpected no-snapshot data dir")
+		}
+	}
+
+	npg := NewProcGroupWithV1Flags(v2BinPath, 3)
+	npg.InheritDataDir(pg)
+	npg.CleanUnsuppportedV1Flags()
+	if err := npg.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	defer npg.Terminate()
+
+	for _, p := range npg {
+		ver, err := checkInternalVersion(p.URL)
+		if err != nil {
+			t.Fatalf("checkVersion error: %v", err)
+		}
+		if ver != "2" {
+			t.Errorf("internal version = %s, want %s", ver, "2")
+		}
+	}
+}
+
+func TestJoinV1Cluster(t *testing.T) {
+	pg := NewProcGroupWithV1Flags(v1BinPath, 1)
+	if err := pg.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	pg.Stop()
+	npg := NewProcGroupWithV1Flags(v2BinPath, 3)
+	npg[0].SetDataDir(pg[0].DataDir)
+	if err := npg.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	defer npg.Terminate()
+
+	for _, p := range npg {
+		ver, err := checkInternalVersion(p.URL)
+		if err != nil {
+			t.Fatalf("checkVersion error: %v", err)
+		}
+		if ver != "1" {
+			t.Errorf("internal version = %s, want %s", ver, "1")
+		}
+	}
+}
+
+func TestJoinV1ClusterViaDiscovery(t *testing.T) {
+	dp := NewProcWithDefaultFlags(v1BinPath)
+	dp.SetV1Addr("127.0.0.1:5001")
+	dp.SetV1PeerAddr("127.0.0.1:8001")
+	if err := dp.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	defer dp.Terminate()
+
+	durl := "http://127.0.0.1:5001/v2/keys/cluster/"
+	pg := NewProcGroupViaDiscoveryWithV1Flags(v1BinPath, 1, durl)
+	if err := pg.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	pg.Stop()
+	npg := NewProcGroupViaDiscoveryWithV1Flags(v2BinPath, 3, durl)
+	npg[0].SetDataDir(pg[0].DataDir)
+	if err := npg.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	defer npg.Terminate()
+
+	for _, p := range npg {
+		ver, err := checkInternalVersion(p.URL)
+		if err != nil {
+			t.Fatalf("checkVersion error: %v", err)
+		}
+		if ver != "1" {
+			t.Errorf("internal version = %s, want %s", ver, "1")
+		}
+	}
+}
+
+func TestUpgradeV1Standby(t *testing.T) {
+	// get v1 standby data dir
+	pg := NewProcGroupWithV1Flags(v1BinPath, 3)
+	if err := pg.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	req, err := http.NewRequest("PUT", pg[0].PeerURL+"/v2/admin/config", bytes.NewBufferString(`{"activeSize":3,"removeDelay":1800,"syncInterval":5}`))
+	if err != nil {
+		t.Fatalf("NewRequest error: %v", err)
+	}
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		t.Fatalf("http Do error: %v", err)
+	}
+	if resp.StatusCode != http.StatusOK {
+		t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusOK)
+	}
+
+	p := NewProcInProcGroupWithV1Flags(v2BinPath, 4, 3)
+	if err := p.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	fmt.Println("checking new member is in standby mode...")
+	mustExist(path.Join(p.DataDir, "standby_info"))
+	ver, err := checkInternalVersion(p.URL)
+	if err != nil {
+		t.Fatalf("checkVersion error: %v", err)
+	}
+	if ver != "1" {
+		t.Errorf("internal version = %s, want %s", ver, "1")
+	}
+
+	fmt.Println("upgrading the whole cluster...")
+	cmd := exec.Command(etcdctlBinPath, "upgrade", "--peer-url", pg[0].PeerURL)
+	if err := cmd.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	if err := cmd.Wait(); err != nil {
+		t.Fatalf("Wait error: %v", err)
+	}
+	fmt.Println("waiting until peer-mode etcd exits...")
+	if err := pg.Wait(); err != nil {
+		t.Fatalf("Wait error: %v", err)
+	}
+	fmt.Println("restarting the peer-mode etcd...")
+	npg := NewProcGroupWithV1Flags(v2BinPath, 3)
+	npg.InheritDataDir(pg)
+	npg.CleanUnsuppportedV1Flags()
+	if err := npg.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	defer npg.Terminate()
+	fmt.Println("waiting until standby-mode etcd exits...")
+	if err := p.Wait(); err != nil {
+		t.Fatalf("Wait error: %v", err)
+	}
+	fmt.Println("restarting the standby-mode etcd...")
+	np := NewProcInProcGroupWithV1Flags(v2BinPath, 4, 3)
+	np.SetDataDir(p.DataDir)
+	np.CleanUnsuppportedV1Flags()
+	if err := np.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	defer np.Terminate()
+
+	fmt.Println("checking the new member is in v2 proxy mode...")
+	ver, err = checkInternalVersion(np.URL)
+	if err != nil {
+		t.Fatalf("checkVersion error: %v", err)
+	}
+	if ver != "2" {
+		t.Errorf("internal version = %s, want %s", ver, "1")
+	}
+	if _, err := os.Stat(path.Join(np.DataDir, "proxy")); err != nil {
+		t.Errorf("stat proxy dir error = %v, want nil", err)
+	}
+}
+
+func TestUpgradeV1TLSCluster(t *testing.T) {
+	// get v2-desired v1 data dir
+	pg := NewProcGroupWithV1Flags(v1BinPath, 3)
+	pg.SetPeerTLS("./fixtures/server.crt", "./fixtures/server.key.insecure", "./fixtures/ca.crt")
+	if err := pg.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	cmd := exec.Command(etcdctlBinPath,
+		"upgrade", "--peer-url", pg[1].PeerURL,
+		"--peer-cert-file", "./fixtures/server.crt",
+		"--peer-key-file", "./fixtures/server.key.insecure",
+		"--peer-ca-file", "./fixtures/ca.crt",
+	)
+	if err := cmd.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	if err := cmd.Wait(); err != nil {
+		t.Fatalf("Wait error: %v", err)
+	}
+	t.Logf("wait until etcd exits...")
+	if err := pg.Wait(); err != nil {
+		t.Fatalf("Wait error: %v", err)
+	}
+
+	npg := NewProcGroupWithV1Flags(v2BinPath, 3)
+	npg.SetPeerTLS("./fixtures/server.crt", "./fixtures/server.key.insecure", "./fixtures/ca.crt")
+	npg.InheritDataDir(pg)
+	npg.CleanUnsuppportedV1Flags()
+	if err := npg.Start(); err != nil {
+		t.Fatalf("Start error: %v", err)
+	}
+	defer npg.Terminate()
+
+	for _, p := range npg {
+		ver, err := checkInternalVersion(p.URL)
+		if err != nil {
+			t.Fatalf("checkVersion error: %v", err)
+		}
+		if ver != "2" {
+			t.Errorf("internal version = %s, want %s", ver, "2")
+		}
+	}
+}
+
+func absPathFromEnv(name string) string {
+	path, err := filepath.Abs(os.Getenv(name))
+	if err != nil {
+		fmt.Printf("unexpected Abs error: %v\n", err)
+	}
+	return path
+}
+
+func mustExist(path string) {
+	if _, err := os.Stat(path); err != nil {
+		fmt.Printf("%v\n", err)
+		os.Exit(1)
+	}
+}
+
+func checkInternalVersion(url string) (string, error) {
+	resp, err := http.Get(url + "/version")
+	if err != nil {
+		return "", err
+	}
+	b, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return "", err
+	}
+	var m map[string]string
+	err = json.Unmarshal(b, &m)
+	return m["internalVersion"], err
+}

migrate/snapshot.go

@@ -43,7 +43,7 @@ type Snapshot4 struct {
 	} `json:"peers"`
 }
 
-type sstore struct {
+type Store4 struct {
 	Root           *node
 	CurrentIndex   uint64
 	CurrentVersion int
@@ -63,6 +63,24 @@ type node struct {
 	Children   map[string]*node // for directory
 }
 
+func deepCopyNode(n *node, parent *node) *node {
+	out := &node{
+		Path:          n.Path,
+		CreatedIndex:  n.CreatedIndex,
+		ModifiedIndex: n.ModifiedIndex,
+		Parent:        parent,
+		ExpireTime:    n.ExpireTime,
+		ACL:           n.ACL,
+		Value:         n.Value,
+		Children:      make(map[string]*node),
+	}
+	for k, v := range n.Children {
+		out.Children[k] = deepCopyNode(v, out)
+	}
+
+	return out
+}
+
 func replacePathNames(n *node, s1, s2 string) {
 	n.Path = path.Clean(strings.Replace(n.Path, s1, s2, 1))
 	for _, c := range n.Children {
@@ -87,9 +105,23 @@ func pullNodesFromEtcd(n *node) map[string]uint64 {
 	return out
 }
 
-func fixEtcd(n *node) {
-	n.Path = "/0"
-	machines := n.Children["machines"]
+func fixEtcd(etcdref *node) *node {
+	n := &node{
+		Path:          "/0",
+		CreatedIndex:  etcdref.CreatedIndex,
+		ModifiedIndex: etcdref.ModifiedIndex,
+		ExpireTime:    etcdref.ExpireTime,
+		ACL:           etcdref.ACL,
+		Children:      make(map[string]*node),
+	}
+
+	var machines *node
+	if machineOrig, ok := etcdref.Children["machines"]; ok {
+		machines = deepCopyNode(machineOrig, n)
+	}
+	if machines == nil {
+		return n
+	}
 	n.Children["members"] = &node{
 		Path:          "/0/members",
 		CreatedIndex:  machines.CreatedIndex,
@@ -97,6 +129,7 @@ func fixEtcd(n *node) {
 		ExpireTime:    machines.ExpireTime,
 		ACL:           machines.ACL,
 		Children:      make(map[string]*node),
+		Parent:        n,
 	}
 	for name, c := range machines.Children {
 		q, err := url.ParseQuery(c.Value)
@@ -121,29 +154,32 @@ func fixEtcd(n *node) {
 			ModifiedIndex: c.ModifiedIndex,
 			ExpireTime:    c.ExpireTime,
 			ACL:           c.ACL,
-			Children: map[string]*node{
-				"attributes": &node{
-					Path:          path.Join("/0/members", m.ID.String(), "attributes"),
-					CreatedIndex:  c.CreatedIndex,
-					ModifiedIndex: c.ModifiedIndex,
-					ExpireTime:    c.ExpireTime,
-					ACL:           c.ACL,
-					Value:         string(attrBytes),
-				},
-				"raftAttributes": &node{
-					Path:          path.Join("/0/members", m.ID.String(), "raftAttributes"),
-					CreatedIndex:  c.CreatedIndex,
-					ModifiedIndex: c.ModifiedIndex,
-					ExpireTime:    c.ExpireTime,
-					ACL:           c.ACL,
-					Value:         string(raftBytes),
-				},
-			},
+			Children:      make(map[string]*node),
+			Parent:        n.Children["members"],
 		}
+		attrs := &node{
+			Path:          path.Join("/0/members", m.ID.String(), "attributes"),
+			CreatedIndex:  c.CreatedIndex,
+			ModifiedIndex: c.ModifiedIndex,
+			ExpireTime:    c.ExpireTime,
+			ACL:           c.ACL,
+			Value:         string(attrBytes),
+			Parent:        newNode,
+		}
+		newNode.Children["attributes"] = attrs
+		raftAttrs := &node{
+			Path:          path.Join("/0/members", m.ID.String(), "raftAttributes"),
+			CreatedIndex:  c.CreatedIndex,
+			ModifiedIndex: c.ModifiedIndex,
+			ExpireTime:    c.ExpireTime,
+			ACL:           c.ACL,
+			Value:         string(raftBytes),
+			Parent:        newNode,
+		}
+		newNode.Children["raftAttributes"] = raftAttrs
 		n.Children["members"].Children[m.ID.String()] = newNode
 	}
-	delete(n.Children, "machines")
-
+	return n
 }
 
 func mangleRoot(n *node) *node {
@@ -157,15 +193,15 @@ func mangleRoot(n *node) *node {
 	}
 	newRoot.Children["1"] = n
 	etcd := n.Children["_etcd"]
-	delete(n.Children, "_etcd")
 	replacePathNames(n, "/", "/1/")
-	fixEtcd(etcd)
-	newRoot.Children["0"] = etcd
+	newZero := fixEtcd(etcd)
+	newZero.Parent = newRoot
+	newRoot.Children["0"] = newZero
 	return newRoot
 }
 
 func (s *Snapshot4) GetNodesFromStore() map[string]uint64 {
-	st := &sstore{}
+	st := &Store4{}
 	if err := json.Unmarshal(s.State, st); err != nil {
 		log.Fatal("Couldn't unmarshal snapshot")
 	}
@@ -174,7 +210,7 @@ func (s *Snapshot4) GetNodesFromStore() map[string]uint64 {
 }
 
 func (s *Snapshot4) Snapshot2() *raftpb.Snapshot {
-	st := &sstore{}
+	st := &Store4{}
 	if err := json.Unmarshal(s.State, st); err != nil {
 		log.Fatal("Couldn't unmarshal snapshot")
 	}

migrate/standby.go

@@ -0,0 +1,70 @@
+// Copyright 2015 CoreOS, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package migrate
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"os"
+)
+
+type StandbyInfo4 struct {
+	Running      bool
+	Cluster      []*MachineMessage
+	SyncInterval float64
+}
+
+// MachineMessage represents information about a peer or standby in the registry.
+type MachineMessage struct {
+	Name      string `json:"name"`
+	State     string `json:"state"`
+	ClientURL string `json:"clientURL"`
+	PeerURL   string `json:"peerURL"`
+}
+
+func (si *StandbyInfo4) ClientURLs() []string {
+	var urls []string
+	for _, m := range si.Cluster {
+		urls = append(urls, m.ClientURL)
+	}
+	return urls
+}
+
+func (si *StandbyInfo4) InitialCluster() string {
+	b := &bytes.Buffer{}
+	first := true
+	for _, m := range si.Cluster {
+		if !first {
+			fmt.Fprintf(b, ",")
+		}
+		first = false
+		fmt.Fprintf(b, "%s=%s", m.Name, m.PeerURL)
+	}
+	return b.String()
+}
+
+func DecodeStandbyInfo4FromFile(path string) (*StandbyInfo4, error) {
+	var info StandbyInfo4
+	file, err := os.OpenFile(path, os.O_RDONLY, 0600)
+	if err != nil {
+		return nil, err
+	}
+	defer file.Close()
+	if err = json.NewDecoder(file).Decode(&info); err != nil {
+		return nil, err
+	}
+	return &info, nil
+}

migrate/starter/starter.go

@@ -0,0 +1,397 @@
+// Copyright 2015 CoreOS, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package starter
+
+import (
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io/ioutil"
+	"log"
+	"net/http"
+	"net/url"
+	"os"
+	"path"
+	"strings"
+	"syscall"
+
+	"github.com/coreos/etcd/client"
+	"github.com/coreos/etcd/etcdmain"
+	"github.com/coreos/etcd/migrate"
+	"github.com/coreos/etcd/pkg/flags"
+	"github.com/coreos/etcd/pkg/osutil"
+	etcdversion "github.com/coreos/etcd/version"
+	"github.com/coreos/etcd/wal"
+
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
+)
+
+type version string
+
+const (
+	internalV1      version = "1"
+	internalV2      version = "2"
+	internalV2Proxy version = "2.proxy"
+	internalUnknown version = "unknown"
+
+	v0_4      version = "v0.4"
+	v2_0      version = "v2.0"
+	v2_0Proxy version = "v2.0 proxy"
+	empty     version = "empty"
+	unknown   version = "unknown"
+
+	defaultInternalV1etcdBinaryDir = "/usr/libexec/etcd/internal_versions/"
+)
+
+var (
+	v2SpecialFlags = []string{
+		"initial-cluster",
+		"listen-peer-urls",
+		"listen-client-urls",
+		"proxy",
+	}
+)
+
+func StartDesiredVersion(args []string) {
+	fs, err := parseConfig(args)
+	if err != nil {
+		return
+	}
+	if fs.Lookup("version").Value.String() == "true" {
+		fmt.Println("etcd version", etcdversion.Version)
+		os.Exit(0)
+	}
+
+	ver := checkInternalVersion(fs)
+	log.Printf("starter: start etcd version %s", ver)
+	switch ver {
+	case internalV1:
+		startInternalV1()
+	case internalV2:
+	case internalV2Proxy:
+		if _, err := os.Stat(standbyInfo4(fs.Lookup("data-dir").Value.String())); err != nil {
+			log.Printf("starter: Detect standby_info file exists, and add --proxy=on flag to ensure it runs in v2.0 proxy mode.")
+			log.Printf("starter: Before removing v0.4 data, --proxy=on flag MUST be added.")
+		}
+		// append proxy flag to args to trigger proxy mode
+		os.Args = append(os.Args, "-proxy=on")
+	default:
+		log.Panicf("starter: unhandled start version")
+	}
+}
+
+func checkInternalVersion(fs *flag.FlagSet) version {
+	// If it uses 2.0 env var explicitly, start 2.0
+	for _, name := range v2SpecialFlags {
+		if fs.Lookup(name).Value.String() != "" {
+			return internalV2
+		}
+	}
+
+	dataDir := fs.Lookup("data-dir").Value.String()
+	if dataDir == "" {
+		log.Fatalf("starter: please set --data-dir or ETCD_DATA_DIR for etcd")
+	}
+	// check the data directory
+	dataver, err := wal.DetectVersion(dataDir)
+	if err != nil {
+		log.Fatalf("starter: failed to detect etcd version in %v: %v", dataDir, err)
+	}
+	log.Printf("starter: detect etcd version %s in %s", dataver, dataDir)
+	switch dataver {
+	case wal.WALv2_0:
+		return internalV2
+	case wal.WALv2_0Proxy:
+		return internalV2Proxy
+	case wal.WALv0_4:
+		standbyInfo, err := migrate.DecodeStandbyInfo4FromFile(standbyInfo4(dataDir))
+		if err != nil && !os.IsNotExist(err) {
+			log.Fatalf("starter: failed to decode standbyInfo in %v: %v", dataDir, err)
+		}
+		inStandbyMode := standbyInfo != nil && standbyInfo.Running
+		if inStandbyMode {
+			ver, err := checkInternalVersionByClientURLs(standbyInfo.ClientURLs(), clientTLSInfo(fs))
+			if err != nil {
+				log.Printf("starter: failed to check start version through peers: %v", err)
+				return internalV1
+			}
+			if ver == internalV2 {
+				osutil.Unsetenv("ETCD_DISCOVERY")
+				os.Args = append(os.Args, "-initial-cluster", standbyInfo.InitialCluster())
+				return internalV2Proxy
+			}
+			return ver
+		}
+		ver, err := checkInternalVersionByDataDir4(dataDir)
+		if err != nil {
+			log.Fatalf("starter: failed to check start version in %v: %v", dataDir, err)
+		}
+		return ver
+	case wal.WALNotExist:
+		discovery := fs.Lookup("discovery").Value.String()
+		dpeers, err := getPeersFromDiscoveryURL(discovery)
+		if err != nil {
+			log.Printf("starter: failed to get peers from discovery %s: %v", discovery, err)
+		}
+		peerStr := fs.Lookup("peers").Value.String()
+		ppeers := getPeersFromPeersFlag(peerStr, peerTLSInfo(fs))
+
+		urls := getClientURLsByPeerURLs(append(dpeers, ppeers...), peerTLSInfo(fs))
+		ver, err := checkInternalVersionByClientURLs(urls, clientTLSInfo(fs))
+		if err != nil {
+			log.Printf("starter: failed to check start version through peers: %v", err)
+			return internalV2
+		}
+		return ver
+	}
+	// never reach here
+	log.Panicf("starter: unhandled etcd version in %v", dataDir)
+	return internalUnknown
+}
+
+func checkInternalVersionByDataDir4(dataDir string) (version, error) {
+	// check v0.4 snapshot
+	snap4, err := migrate.DecodeLatestSnapshot4FromDir(snapDir4(dataDir))
+	if err != nil {
+		return internalUnknown, err
+	}
+	if snap4 != nil {
+		st := &migrate.Store4{}
+		if err := json.Unmarshal(snap4.State, st); err != nil {
+			return internalUnknown, err
+		}
+		dir := st.Root.Children["_etcd"]
+		n, ok := dir.Children["next-internal-version"]
+		if ok && n.Value == "2" {
+			return internalV2, nil
+		}
+	}
+
+	// check v0.4 log
+	ents4, err := migrate.DecodeLog4FromFile(logFile4(dataDir))
+	if err != nil {
+		return internalUnknown, err
+	}
+	for _, e := range ents4 {
+		cmd, err := migrate.NewCommand4(e.GetCommandName(), e.GetCommand(), nil)
+		if err != nil {
+			return internalUnknown, err
+		}
+		setcmd, ok := cmd.(*migrate.SetCommand)
+		if !ok {
+			continue
+		}
+		if setcmd.Key == "/_etcd/next-internal-version" && setcmd.Value == "2" {
+			return internalV2, nil
+		}
+	}
+	return internalV1, nil
+}
+
+func getClientURLsByPeerURLs(peers []string, tls *TLSInfo) []string {
+	c, err := newDefaultClient(tls)
+	if err != nil {
+		log.Printf("starter: new client error: %v", err)
+		return nil
+	}
+	var urls []string
+	for _, u := range peers {
+		resp, err := c.Get(u + "/etcdURL")
+		if err != nil {
+			log.Printf("starter: failed to get /etcdURL from %s", u)
+			continue
+		}
+		b, err := ioutil.ReadAll(resp.Body)
+		if err != nil {
+			log.Printf("starter: failed to read body from %s", u)
+			continue
+		}
+		urls = append(urls, string(b))
+	}
+	return urls
+}
+
+func checkInternalVersionByClientURLs(urls []string, tls *TLSInfo) (version, error) {
+	c, err := newDefaultClient(tls)
+	if err != nil {
+		return internalUnknown, err
+	}
+	for _, u := range urls {
+		resp, err := c.Get(u + "/version")
+		if err != nil {
+			log.Printf("starter: failed to get /version from %s", u)
+			continue
+		}
+		b, err := ioutil.ReadAll(resp.Body)
+		if err != nil {
+			log.Printf("starter: failed to read body from %s", u)
+			continue
+		}
+
+		var m map[string]string
+		err = json.Unmarshal(b, &m)
+		if err != nil {
+			log.Printf("starter: failed to unmarshal body %s from %s", b, u)
+			continue
+		}
+		switch m["internalVersion"] {
+		case "1":
+			return internalV1, nil
+		case "2":
+			return internalV2, nil
+		default:
+			log.Printf("starter: unrecognized internal version %s from %s", m["internalVersion"], u)
+		}
+	}
+	return internalUnknown, fmt.Errorf("failed to get version from urls %v", urls)
+}
+
+func getPeersFromDiscoveryURL(discoverURL string) ([]string, error) {
+	if discoverURL == "" {
+		return nil, nil
+	}
+
+	u, err := url.Parse(discoverURL)
+	if err != nil {
+		return nil, err
+	}
+	token := u.Path
+	u.Path = ""
+	c, err := client.NewHTTPClient(&http.Transport{}, []string{u.String()})
+	if err != nil {
+		return nil, err
+	}
+	dc := client.NewDiscoveryKeysAPI(c)
+
+	ctx, cancel := context.WithTimeout(context.Background(), client.DefaultRequestTimeout)
+	resp, err := dc.Get(ctx, token)
+	cancel()
+	if err != nil {
+		return nil, err
+	}
+	peers := make([]string, 0)
+	// append non-config keys to peers
+	for _, n := range resp.Node.Nodes {
+		if g := path.Base(n.Key); g == "_config" || g == "_state" {
+			continue
+		}
+		peers = append(peers, n.Value)
+	}
+	return peers, nil
+}
+
+func getPeersFromPeersFlag(str string, tls *TLSInfo) []string {
+	peers := trimSplit(str, ",")
+	for i, p := range peers {
+		peers[i] = tls.Scheme() + "://" + p
+	}
+	return peers
+}
+
+func startInternalV1() {
+	p := os.Getenv("ETCD_BINARY_DIR")
+	if p == "" {
+		p = defaultInternalV1etcdBinaryDir
+	}
+	p = path.Join(p, "1")
+	err := syscall.Exec(p, os.Args, syscall.Environ())
+	if err != nil {
+		log.Fatalf("starter: failed to execute internal v1 etcd: %v", err)
+	}
+}
+
+func newDefaultClient(tls *TLSInfo) (*http.Client, error) {
+	tr := &http.Transport{}
+	if tls.Scheme() == "https" {
+		tlsConfig, err := tls.ClientConfig()
+		if err != nil {
+			return nil, err
+		}
+		tr.TLSClientConfig = tlsConfig
+	}
+	return &http.Client{Transport: tr}, nil
+}
+
+type value struct {
+	isBoolFlag bool
+	s          string
+}
+
+func (v *value) String() string { return v.s }
+
+func (v *value) Set(s string) error {
+	v.s = s
+	return nil
+}
+
+func (v *value) IsBoolFlag() bool { return v.isBoolFlag }
+
+type boolFlag interface {
+	flag.Value
+	IsBoolFlag() bool
+}
+
+// parseConfig parses out the input config from cmdline arguments and
+// environment variables.
+func parseConfig(args []string) (*flag.FlagSet, error) {
+	fs := flag.NewFlagSet("full flagset", flag.ContinueOnError)
+	etcdmain.NewConfig().VisitAll(func(f *flag.Flag) {
+		_, isBoolFlag := f.Value.(boolFlag)
+		fs.Var(&value{isBoolFlag: isBoolFlag}, f.Name, "")
+	})
+	if err := fs.Parse(args); err != nil {
+		return nil, err
+	}
+	if err := flags.SetFlagsFromEnv(fs); err != nil {
+		return nil, err
+	}
+	return fs, nil
+}
+
+func clientTLSInfo(fs *flag.FlagSet) *TLSInfo {
+	return &TLSInfo{
+		CAFile:   fs.Lookup("ca-file").Value.String(),
+		CertFile: fs.Lookup("cert-file").Value.String(),
+		KeyFile:  fs.Lookup("key-file").Value.String(),
+	}
+}
+
+func peerTLSInfo(fs *flag.FlagSet) *TLSInfo {
+	return &TLSInfo{
+		CAFile:   fs.Lookup("peer-ca-file").Value.String(),
+		CertFile: fs.Lookup("peer-cert-file").Value.String(),
+		KeyFile:  fs.Lookup("peer-key-file").Value.String(),
+	}
+}
+
+func snapDir4(dataDir string) string {
+	return path.Join(dataDir, "snapshot")
+}
+
+func logFile4(dataDir string) string {
+	return path.Join(dataDir, "log")
+}
+
+func standbyInfo4(dataDir string) string {
+	return path.Join(dataDir, "standby_info")
+}
+
+func trimSplit(s, sep string) []string {
+	trimmed := strings.Split(s, sep)
+	for i := range trimmed {
+		trimmed[i] = strings.TrimSpace(trimmed[i])
+	}
+	return trimmed
+}

migrate/starter/starter_test.go

@@ -0,0 +1,70 @@
+// Copyright 2015 CoreOS, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package starter
+
+import (
+	"flag"
+	"reflect"
+	"testing"
+)
+
+func TestParseConfig(t *testing.T) {
+	tests := []struct {
+		args  []string
+		wvals map[string]string
+	}{
+		{
+			[]string{"--name", "etcd", "--data-dir", "dir"},
+			map[string]string{
+				"name":     "etcd",
+				"data-dir": "dir",
+			},
+		},
+		{
+			[]string{"--name=etcd", "--data-dir=dir"},
+			map[string]string{
+				"name":     "etcd",
+				"data-dir": "dir",
+			},
+		},
+		{
+			[]string{"--version", "--name", "etcd"},
+			map[string]string{
+				"version": "true",
+				"name":    "etcd",
+			},
+		},
+		{
+			[]string{"--version=true", "--name", "etcd"},
+			map[string]string{
+				"version": "true",
+				"name":    "etcd",
+			},
+		},
+	}
+	for i, tt := range tests {
+		fs, err := parseConfig(tt.args)
+		if err != nil {
+			t.Fatalf("#%d: unexpected parseConfig error: %v", i, err)
+		}
+		vals := make(map[string]string)
+		fs.Visit(func(f *flag.Flag) {
+			vals[f.Name] = f.Value.String()
+		})
+		if !reflect.DeepEqual(vals, tt.wvals) {
+			t.Errorf("#%d: vals = %+v, want %+v", i, vals, tt.wvals)
+		}
+	}
+}

migrate/starter/tls_info.go

@@ -0,0 +1,120 @@
+// Copyright 2015 CoreOS, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package starter
+
+import (
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/pem"
+	"fmt"
+	"io/ioutil"
+)
+
+// TLSInfo holds the SSL certificates paths.
+type TLSInfo struct {
+	CertFile string `json:"CertFile"`
+	KeyFile  string `json:"KeyFile"`
+	CAFile   string `json:"CAFile"`
+}
+
+func (info TLSInfo) Scheme() string {
+	if info.KeyFile != "" && info.CertFile != "" {
+		return "https"
+	} else {
+		return "http"
+	}
+}
+
+// Generates a tls.Config object for a server from the given files.
+func (info TLSInfo) ServerConfig() (*tls.Config, error) {
+	// Both the key and cert must be present.
+	if info.KeyFile == "" || info.CertFile == "" {
+		return nil, fmt.Errorf("KeyFile and CertFile must both be present[key: %v, cert: %v]", info.KeyFile, info.CertFile)
+	}
+
+	var cfg tls.Config
+
+	tlsCert, err := tls.LoadX509KeyPair(info.CertFile, info.KeyFile)
+	if err != nil {
+		return nil, err
+	}
+
+	cfg.Certificates = []tls.Certificate{tlsCert}
+
+	if info.CAFile != "" {
+		cfg.ClientAuth = tls.RequireAndVerifyClientCert
+		cp, err := newCertPool(info.CAFile)
+		if err != nil {
+			return nil, err
+		}
+
+		cfg.RootCAs = cp
+		cfg.ClientCAs = cp
+	} else {
+		cfg.ClientAuth = tls.NoClientCert
+	}
+
+	return &cfg, nil
+}
+
+// Generates a tls.Config object for a client from the given files.
+func (info TLSInfo) ClientConfig() (*tls.Config, error) {
+	var cfg tls.Config
+
+	if info.KeyFile == "" || info.CertFile == "" {
+		return &cfg, nil
+	}
+
+	tlsCert, err := tls.LoadX509KeyPair(info.CertFile, info.KeyFile)
+	if err != nil {
+		return nil, err
+	}
+
+	cfg.Certificates = []tls.Certificate{tlsCert}
+
+	if info.CAFile != "" {
+		cp, err := newCertPool(info.CAFile)
+		if err != nil {
+			return nil, err
+		}
+
+		cfg.RootCAs = cp
+	}
+
+	return &cfg, nil
+}
+
+// newCertPool creates x509 certPool with provided CA file
+func newCertPool(CAFile string) (*x509.CertPool, error) {
+	certPool := x509.NewCertPool()
+	pemByte, err := ioutil.ReadFile(CAFile)
+	if err != nil {
+		return nil, err
+	}
+
+	for {
+		var block *pem.Block
+		block, pemByte = pem.Decode(pemByte)
+		if block == nil {
+			return certPool, nil
+		}
+		cert, err := x509.ParseCertificate(block.Bytes)
+		if err != nil {
+			return nil, err
+		}
+		certPool.AddCert(cert)
+	}
+
+}

pkg/coreos/coreos.go

@@ -0,0 +1,27 @@
+// Copyright 2015 CoreOS, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package coreos
+
+import (
+	"io/ioutil"
+	"strings"
+)
+
+func IsCoreOS() bool {
+	b, err := ioutil.ReadFile("/usr/lib/os-release")
+	if err != nil {
+		return false
+	}
+	return strings.Contains(string(b), "ID=coreos")
+}

pkg/flags/flag.go

@@ -85,6 +85,21 @@ func SetFlagsFromEnv(fs *flag.FlagSet) error {
 	return err
 }
 
+// SetBindAddrFromAddr sets the value of bindAddr flag from the value
+// of addr flag. Both flags' Value must be of type IPAddressPort. If the
+// bindAddr flag is set and the addr flag is unset, it will set bindAddr to
+// 0.0.0.0:port of addr. Otherwise, it keeps the original values.
+func SetBindAddrFromAddr(fs *flag.FlagSet, bindAddrFlagName, addrFlagName string) {
+	if IsSet(fs, bindAddrFlagName) || !IsSet(fs, addrFlagName) {
+		return
+	}
+	addr := *fs.Lookup(addrFlagName).Value.(*IPAddressPort)
+	addr.IP = "0.0.0.0"
+	if err := fs.Set(bindAddrFlagName, addr.String()); err != nil {
+		log.Panicf("etcdmain: unexpected flags set error: %v", err)
+	}
+}
+
 // URLsFromFlags decides what URLs should be using two different flags
 // as datasources. The first flag's Value must be of type URLs, while
 // the second must be of type IPAddressPort. If both of these flags
@@ -119,3 +134,13 @@ func URLsFromFlags(fs *flag.FlagSet, urlsFlagName string, addrFlagName string, t
 
 	return []url.URL(*fs.Lookup(urlsFlagName).Value.(*URLsValue)), nil
 }
+
+func IsSet(fs *flag.FlagSet, name string) bool {
+	set := false
+	fs.Visit(func(f *flag.Flag) {
+		if f.Name == name {
+			set = true
+		}
+	})
+	return set
+}

pkg/flags/flag_test.go

@@ -81,6 +81,49 @@ func TestSetFlagsFromEnvBad(t *testing.T) {
 	}
 }
 
+func TestSetBindAddrFromAddr(t *testing.T) {
+	tests := []struct {
+		args  []string
+		waddr *IPAddressPort
+	}{
+		// no flags set
+		{
+			args:  []string{},
+			waddr: &IPAddressPort{},
+		},
+		// addr flag set
+		{
+			args:  []string{"-addr=192.0.3.17:4001"},
+			waddr: &IPAddressPort{IP: "0.0.0.0", Port: 4001},
+		},
+		// bindAddr flag set
+		{
+			args:  []string{"-bind-addr=127.0.0.1:4001"},
+			waddr: &IPAddressPort{IP: "127.0.0.1", Port: 4001},
+		},
+		// both addr flags set
+		{
+			args:  []string{"-bind-addr=127.0.0.1:4001", "-addr=192.0.3.17:4001"},
+			waddr: &IPAddressPort{IP: "127.0.0.1", Port: 4001},
+		},
+	}
+	for i, tt := range tests {
+		fs := flag.NewFlagSet("test", flag.PanicOnError)
+		fs.Var(&IPAddressPort{}, "addr", "")
+		bindAddr := &IPAddressPort{}
+		fs.Var(bindAddr, "bind-addr", "")
+		if err := fs.Parse(tt.args); err != nil {
+			t.Errorf("#%d: failed to parse flags: %v", i, err)
+			continue
+		}
+		SetBindAddrFromAddr(fs, "bind-addr", "addr")
+
+		if !reflect.DeepEqual(bindAddr, tt.waddr) {
+			t.Errorf("#%d: bindAddr = %+v, want %+v", i, bindAddr, tt.waddr)
+		}
+	}
+}
+
 func TestURLsFromFlags(t *testing.T) {
 	tests := []struct {
 		args     []string

pkg/osutil/interrupt_unix.go

@@ -0,0 +1,81 @@
+// Copyright 2015 CoreOS, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// +build !windows,!plan9
+
+// InterruptHandler is a function that is called on receiving a
+// SIGTERM or SIGINT signal.
+
+package osutil
+
+import (
+	"log"
+	"os"
+	"os/signal"
+	"sync"
+	"syscall"
+)
+
+type InterruptHandler func()
+
+var (
+	interruptRegisterMu, interruptExitMu sync.Mutex
+	// interruptHandlers holds all registered InterruptHandlers in order
+	// they will be executed.
+	interruptHandlers = []InterruptHandler{}
+)
+
+// RegisterInterruptHandler registers a new InterruptHandler. Handlers registered
+// after interrupt handing was initiated will not be executed.
+func RegisterInterruptHandler(h InterruptHandler) {
+	interruptRegisterMu.Lock()
+	defer interruptRegisterMu.Unlock()
+	interruptHandlers = append(interruptHandlers, h)
+}
+
+// HandleInterrupts calls the handler functions on receiving a SIGINT or SIGTERM.
+func HandleInterrupts() {
+	notifier := make(chan os.Signal, 1)
+	signal.Notify(notifier, syscall.SIGINT, syscall.SIGTERM)
+
+	go func() {
+		sig := <-notifier
+
+		interruptRegisterMu.Lock()
+		ihs := make([]InterruptHandler, len(interruptHandlers))
+		copy(ihs, interruptHandlers)
+		interruptRegisterMu.Unlock()
+
+		interruptExitMu.Lock()
+
+		log.Printf("received %v signal, shutting down...", sig)
+
+		for _, h := range ihs {
+			h()
+		}
+		signal.Stop(notifier)
+		pid := syscall.Getpid()
+		// exit directly if it is the "init" process, since the kernel will not help to kill pid 1.
+		if pid == 1 {
+			os.Exit(0)
+		}
+		syscall.Kill(pid, sig.(syscall.Signal))
+	}()
+}
+
+// Exit relays to os.Exit if no interrupt handlers are running, blocks otherwise.
+func Exit(code int) {
+	interruptExitMu.Lock()
+	os.Exit(code)
+}

pkg/osutil/interrupt_windows.go

@@ -0,0 +1,32 @@
+// Copyright 2015 CoreOS, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// +build windows
+
+package osutil
+
+import "os"
+
+type InterruptHandler func()
+
+// RegisterInterruptHandler is a no-op on windows
+func RegisterInterruptHandler(h InterruptHandler) {}
+
+// HandleInterrupts is a no-op on windows
+func HandleInterrupts() {}
+
+// Exit calls os.Exit
+func Exit(code int) {
+	os.Exit(code)
+}

pkg/osutil/osutil.go

@@ -0,0 +1,35 @@
+// Copyright 2015 CoreOS, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package osutil
+
+import (
+	"os"
+	"strings"
+)
+
+func Unsetenv(key string) error {
+	envs := os.Environ()
+	os.Clearenv()
+	for _, e := range envs {
+		strs := strings.SplitN(e, "=", 2)
+		if strs[0] == key {
+			continue
+		}
+		if err := os.Setenv(strs[0], strs[1]); err != nil {
+			return err
+		}
+	}
+	return nil
+}

pkg/osutil/osutil_test.go

@@ -0,0 +1,88 @@
+// Copyright 2015 CoreOS, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package osutil
+
+import (
+	"os"
+	"os/signal"
+	"reflect"
+	"syscall"
+	"testing"
+	"time"
+)
+
+func TestUnsetenv(t *testing.T) {
+	tests := []string{
+		"data",
+		"space data",
+		"equal=data",
+	}
+	for i, tt := range tests {
+		key := "ETCD_UNSETENV_TEST"
+		if os.Getenv(key) != "" {
+			t.Fatalf("#%d: cannot get empty %s", i, key)
+		}
+		env := os.Environ()
+		if err := os.Setenv(key, tt); err != nil {
+			t.Fatalf("#%d: cannot set %s: %v", i, key, err)
+		}
+		if err := Unsetenv(key); err != nil {
+			t.Errorf("#%d: unsetenv %s error: %v", i, key, err)
+		}
+		if g := os.Environ(); !reflect.DeepEqual(g, env) {
+			t.Errorf("#%d: env = %+v, want %+v", i, g, env)
+		}
+	}
+}
+
+func waitSig(t *testing.T, c <-chan os.Signal, sig os.Signal) {
+	select {
+	case s := <-c:
+		if s != sig {
+			t.Fatalf("signal was %v, want %v", s, sig)
+		}
+	case <-time.After(1 * time.Second):
+		t.Fatalf("timeout waiting for %v", sig)
+	}
+}
+
+func TestHandleInterrupts(t *testing.T) {
+	for _, sig := range []syscall.Signal{syscall.SIGINT, syscall.SIGTERM} {
+		n := 1
+		RegisterInterruptHandler(func() { n++ })
+		RegisterInterruptHandler(func() { n *= 2 })
+
+		c := make(chan os.Signal, 2)
+		signal.Notify(c, sig)
+
+		HandleInterrupts()
+		syscall.Kill(syscall.Getpid(), sig)
+
+		// we should receive the signal once from our own kill and
+		// a second time from HandleInterrupts
+		waitSig(t, c, sig)
+		waitSig(t, c, sig)
+
+		if n == 3 {
+			t.Fatalf("interrupt handlers were called in wrong order")
+		}
+		if n != 4 {
+			t.Fatalf("interrupt handlers were not called properly")
+		}
+		// reset interrupt handlers
+		interruptHandlers = interruptHandlers[:0]
+		interruptExitMu.Unlock()
+	}
+}

pkg/transport/keepalive_listener.go

@@ -15,6 +15,7 @@
 package transport
 
 import (
+	"crypto/tls"
 	"net"
 	"time"
 )
@@ -22,18 +23,26 @@ import (
 // NewKeepAliveListener returns a listener that listens on the given address.
 // http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html
 func NewKeepAliveListener(addr string, scheme string, info TLSInfo) (net.Listener, error) {
-	ln, err := NewListener(addr, scheme, info)
+	l, err := net.Listen("tcp", addr)
 	if err != nil {
 		return nil, err
 	}
+
+	if !info.Empty() && scheme == "https" {
+		cfg, err := info.ServerConfig()
+		if err != nil {
+			return nil, err
+		}
+
+		return newTLSKeepaliveListener(l, cfg), nil
+	}
+
 	return &keepaliveListener{
-		Listener: ln,
+		Listener: l,
 	}, nil
 }
 
-type keepaliveListener struct {
-	net.Listener
-}
+type keepaliveListener struct{ net.Listener }
 
 func (kln *keepaliveListener) Accept() (net.Conn, error) {
 	c, err := kln.Listener.Accept()
@@ -48,3 +57,37 @@ func (kln *keepaliveListener) Accept() (net.Conn, error) {
 	tcpc.SetKeepAlivePeriod(30 * time.Second)
 	return tcpc, nil
 }
+
+// A tlsKeepaliveListener implements a network listener (net.Listener) for TLS connections.
+type tlsKeepaliveListener struct {
+	net.Listener
+	config *tls.Config
+}
+
+// Accept waits for and returns the next incoming TLS connection.
+// The returned connection c is a *tls.Conn.
+func (l *tlsKeepaliveListener) Accept() (c net.Conn, err error) {
+	c, err = l.Listener.Accept()
+	if err != nil {
+		return
+	}
+	tcpc := c.(*net.TCPConn)
+	// detection time: tcp_keepalive_time + tcp_keepalive_probes + tcp_keepalive_intvl
+	// default on linux:  30 + 8 * 30
+	// default on osx:    30 + 8 * 75
+	tcpc.SetKeepAlive(true)
+	tcpc.SetKeepAlivePeriod(30 * time.Second)
+	c = tls.Server(c, l.config)
+	return
+}
+
+// NewListener creates a Listener which accepts connections from an inner
+// Listener and wraps each connection with Server.
+// The configuration config must be non-nil and must have
+// at least one certificate.
+func newTLSKeepaliveListener(inner net.Listener, config *tls.Config) net.Listener {
+	l := &tlsKeepaliveListener{}
+	l.Listener = inner
+	l.config = config
+	return l
+}

pkg/transport/keepalive_listener_test.go

@@ -15,7 +15,9 @@
 package transport
 
 import (
+	"crypto/tls"
 	"net/http"
+	"os"
 	"testing"
 )
 
@@ -34,4 +36,29 @@ func TestNewKeepAliveListener(t *testing.T) {
 		t.Fatalf("unexpected Accept error: %v", err)
 	}
 	conn.Close()
+	ln.Close()
+
+	// tls
+	tmp, err := createTempFile([]byte("XXX"))
+	if err != nil {
+		t.Fatalf("unable to create tmpfile: %v", err)
+	}
+	defer os.Remove(tmp)
+	tlsInfo := TLSInfo{CertFile: tmp, KeyFile: tmp}
+	tlsInfo.parseFunc = fakeCertificateParserFunc(tls.Certificate{}, nil)
+	tlsln, err := NewKeepAliveListener("127.0.0.1:0", "https", tlsInfo)
+	if err != nil {
+		t.Fatalf("unexpected NewKeepAliveListener error: %v", err)
+	}
+
+	go http.Get("https://" + tlsln.Addr().String())
+	conn, err = tlsln.Accept()
+	if err != nil {
+		t.Fatalf("unexpected Accept error: %v", err)
+	}
+	if _, ok := conn.(*tls.Conn); !ok {
+		t.Errorf("failed to accept *tls.Conn")
+	}
+	conn.Close()
+	tlsln.Close()
 }

pkg/wait/wait_time.go

@@ -0,0 +1,62 @@
+/*
+   Copyright 2015 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
+package wait
+
+import (
+	"sync"
+	"time"
+)
+
+type WaitTime interface {
+	// Wait returns a chan that waits on the given deadline.
+	// The chan will be triggered when Trigger is called with a
+	// deadline that is later than the one it is waiting for.
+	// The given deadline MUST be unique. The deadline should be
+	// retrived by calling time.Now() in most cases.
+	Wait(deadline time.Time) <-chan struct{}
+	// Trigger triggers all the waiting chans with an earlier deadline.
+	Trigger(deadline time.Time)
+}
+
+type timeList struct {
+	l sync.Mutex
+	m map[int64]chan struct{}
+}
+
+func NewTimeList() *timeList {
+	return &timeList{m: make(map[int64]chan struct{})}
+}
+
+func (tl *timeList) Wait(deadline time.Time) <-chan struct{} {
+	tl.l.Lock()
+	defer tl.l.Unlock()
+	ch := make(chan struct{}, 1)
+	// The given deadline SHOULD be unique.
+	tl.m[deadline.UnixNano()] = ch
+	return ch
+}
+
+func (tl *timeList) Trigger(deadline time.Time) {
+	tl.l.Lock()
+	defer tl.l.Unlock()
+	for t, ch := range tl.m {
+		if t < deadline.UnixNano() {
+			delete(tl.m, t)
+			close(ch)
+		}
+	}
+}

pkg/wait/wait_time_test.go

@@ -0,0 +1,85 @@
+/*
+
+   Copyright 2015 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+package wait
+
+import (
+	"testing"
+	"time"
+)
+
+func TestWaitTime(t *testing.T) {
+	wt := NewTimeList()
+	ch1 := wt.Wait(time.Now())
+	t1 := time.Now()
+	wt.Trigger(t1)
+	select {
+	case <-ch1:
+	case <-time.After(10 * time.Millisecond):
+		t.Fatalf("cannot receive from ch as expected")
+	}
+
+	ch2 := wt.Wait(time.Now())
+	t2 := time.Now()
+	wt.Trigger(t1)
+	select {
+	case <-ch2:
+		t.Fatalf("unexpected to receive from ch")
+	case <-time.After(10 * time.Millisecond):
+	}
+	wt.Trigger(t2)
+	select {
+	case <-ch2:
+	case <-time.After(10 * time.Millisecond):
+		t.Fatalf("cannot receive from ch as expected")
+	}
+}
+
+func TestWaitTestStress(t *testing.T) {
+	chs := make([]<-chan struct{}, 0)
+	wt := NewTimeList()
+	for i := 0; i < 10000; i++ {
+		chs = append(chs, wt.Wait(time.Now()))
+	}
+	wt.Trigger(time.Now())
+
+	for _, ch := range chs {
+		select {
+		case <-ch:
+		case <-time.After(10 * time.Millisecond):
+			t.Fatalf("cannot receive from ch as expected")
+		}
+	}
+}
+
+func BenchmarkWaitTime(b *testing.B) {
+	t := time.Now()
+	wt := NewTimeList()
+	for i := 0; i < b.N; i++ {
+		wt.Wait(t)
+	}
+}
+
+func BenchmarkTriggerAnd10KWaitTime(b *testing.B) {
+	for i := 0; i < b.N; i++ {
+		t := time.Now()
+		wt := NewTimeList()
+		for j := 0; j < 10000; j++ {
+			wt.Wait(t)
+		}
+		wt.Trigger(time.Now())
+	}
+}

raft/log_test.go

@@ -296,15 +296,15 @@ func TestCompactionSideEffects(t *testing.T) {
 		t.Errorf("lastIndex = %d, want %d", raftLog.lastIndex(), lastIndex)
 	}
 
-	for i := offset; i <= raftLog.lastIndex(); i++ {
-		if raftLog.term(i) != i {
-			t.Errorf("term(%d) = %d, want %d", i, raftLog.term(i), i)
+	for j := offset; j <= raftLog.lastIndex(); j++ {
+		if raftLog.term(j) != j {
+			t.Errorf("term(%d) = %d, want %d", j, raftLog.term(j), j)
 		}
 	}
 
-	for i := offset; i <= raftLog.lastIndex(); i++ {
-		if !raftLog.matchTerm(i, i) {
-			t.Errorf("matchTerm(%d) = false, want true", i)
+	for j := offset; j <= raftLog.lastIndex(); j++ {
+		if !raftLog.matchTerm(j, j) {
+			t.Errorf("matchTerm(%d) = false, want true", j)
 		}
 	}
 
@@ -354,9 +354,9 @@ func TestNextEnts(t *testing.T) {
 		raftLog.maybeCommit(5, 1)
 		raftLog.appliedTo(tt.applied)
 
-		ents := raftLog.nextEnts()
-		if !reflect.DeepEqual(ents, tt.wents) {
-			t.Errorf("#%d: ents = %+v, want %+v", i, ents, tt.wents)
+		nents := raftLog.nextEnts()
+		if !reflect.DeepEqual(nents, tt.wents) {
+			t.Errorf("#%d: nents = %+v, want %+v", i, nents, tt.wents)
 		}
 	}
 }
@@ -649,10 +649,10 @@ func TestTerm(t *testing.T) {
 		{offset + num, 0},
 	}
 
-	for i, tt := range tests {
+	for j, tt := range tests {
 		term := l.term(tt.index)
 		if !reflect.DeepEqual(term, tt.w) {
-			t.Errorf("#%d: at = %d, want %d", i, term, tt.w)
+			t.Errorf("#%d: at = %d, want %d", j, term, tt.w)
 		}
 	}
 }
@@ -712,18 +712,18 @@ func TestSlice(t *testing.T) {
 		{offset + num, offset + num + 1, nil, true},
 	}
 
-	for i, tt := range tests {
+	for j, tt := range tests {
 		func() {
 			defer func() {
 				if r := recover(); r != nil {
 					if !tt.wpanic {
-						t.Errorf("%d: panic = %v, want %v: %v", i, true, false, r)
+						t.Errorf("%d: panic = %v, want %v: %v", j, true, false, r)
 					}
 				}
 			}()
 			g := l.slice(tt.from, tt.to)
 			if !reflect.DeepEqual(g, tt.w) {
-				t.Errorf("#%d: from %d to %d = %v, want %v", i, tt.from, tt.to, g, tt.w)
+				t.Errorf("#%d: from %d to %d = %v, want %v", j, tt.from, tt.to, g, tt.w)
 			}
 		}()
 	}

raft/node_test.go

@@ -332,10 +332,10 @@ func TestNodeStart(t *testing.T) {
 	}
 
 	n.Propose(ctx, []byte("foo"))
-	if g := <-n.Ready(); !reflect.DeepEqual(g, wants[1]) {
-		t.Errorf("#%d: g = %+v,\n             w   %+v", 2, g, wants[1])
+	if g2 := <-n.Ready(); !reflect.DeepEqual(g2, wants[1]) {
+		t.Errorf("#%d: g = %+v,\n             w   %+v", 2, g2, wants[1])
 	} else {
-		storage.Append(g.Entries)
+		storage.Append(g2.Entries)
 		n.Advance()
 	}
 

raft/raft_paper_test.go

@@ -774,7 +774,7 @@ func TestVoteRequest(t *testing.T) {
 		{[]pb.Entry{{Term: 1, Index: 1}}, 2},
 		{[]pb.Entry{{Term: 1, Index: 1}, {Term: 2, Index: 2}}, 3},
 	}
-	for i, tt := range tests {
+	for j, tt := range tests {
 		r := newRaft(1, []uint64{1, 2, 3}, 10, 1, NewMemoryStorage(), 0)
 		r.Step(pb.Message{
 			From: 2, To: 1, Type: pb.MsgApp, Term: tt.wterm - 1, LogTerm: 0, Index: 0, Entries: tt.ents,
@@ -788,7 +788,7 @@ func TestVoteRequest(t *testing.T) {
 		msgs := r.readMessages()
 		sort.Sort(messageSlice(msgs))
 		if len(msgs) != 2 {
-			t.Fatalf("#%d: len(msg) = %d, want %d", i, len(msgs), 2)
+			t.Fatalf("#%d: len(msg) = %d, want %d", j, len(msgs), 2)
 		}
 		for i, m := range msgs {
 			if m.Type != pb.MsgVote {

raft/raft_test.go

@@ -510,7 +510,7 @@ func TestOldMessages(t *testing.T) {
 	// commit a new entry
 	tt.send(pb.Message{From: 1, To: 1, Type: pb.MsgProp, Entries: []pb.Entry{{Data: []byte("somedata")}}})
 
-	l := &raftLog{
+	ilog := &raftLog{
 		storage: &MemoryStorage{
 			ents: []pb.Entry{
 				{}, {Data: nil, Term: 1, Index: 1},
@@ -521,7 +521,7 @@ func TestOldMessages(t *testing.T) {
 		unstable:  unstable{offset: 5},
 		committed: 4,
 	}
-	base := ltoa(l)
+	base := ltoa(ilog)
 	for i, p := range tt.peers {
 		if sm, ok := p.(*raft); ok {
 			l := ltoa(sm.raftLog)
@@ -548,7 +548,7 @@ func TestProposal(t *testing.T) {
 		{newNetwork(nil, nopStepper, nopStepper, nil, nil), true},
 	}
 
-	for i, tt := range tests {
+	for j, tt := range tests {
 		send := func(m pb.Message) {
 			defer func() {
 				// only recover is we expect it to panic so
@@ -556,7 +556,7 @@ func TestProposal(t *testing.T) {
 				if !tt.success {
 					e := recover()
 					if e != nil {
-						t.Logf("#%d: err: %s", i, e)
+						t.Logf("#%d: err: %s", j, e)
 					}
 				}
 			}()
@@ -591,7 +591,7 @@ func TestProposal(t *testing.T) {
 		}
 		sm := tt.network.peers[1].(*raft)
 		if g := sm.Term; g != 1 {
-			t.Errorf("#%d: term = %d, want %d", i, g, 1)
+			t.Errorf("#%d: term = %d, want %d", j, g, 1)
 		}
 	}
 }
@@ -603,7 +603,7 @@ func TestProposalByProxy(t *testing.T) {
 		newNetwork(nil, nil, nopStepper),
 	}
 
-	for i, tt := range tests {
+	for j, tt := range tests {
 		// promote 0 the leader
 		tt.send(pb.Message{From: 1, To: 1, Type: pb.MsgHup})
 
@@ -629,7 +629,7 @@ func TestProposalByProxy(t *testing.T) {
 		}
 		sm := tt.peers[1].(*raft)
 		if g := sm.Term; g != 1 {
-			t.Errorf("#%d: term = %d, want %d", i, g, 1)
+			t.Errorf("#%d: term = %d, want %d", j, g, 1)
 		}
 	}
 }
@@ -1601,8 +1601,8 @@ func newNetwork(peers ...Interface) *network {
 	npeers := make(map[uint64]Interface, size)
 	nstorage := make(map[uint64]*MemoryStorage, size)
 
-	for i, p := range peers {
-		id := peerAddrs[i]
+	for j, p := range peers {
+		id := peerAddrs[j]
 		switch v := p.(type) {
 		case nil:
 			nstorage[id] = NewMemoryStorage()

raft/rafttest/network.go

@@ -0,0 +1,157 @@
+package rafttest
+
+import (
+	"math/rand"
+	"sync"
+	"time"
+
+	"github.com/coreos/etcd/raft/raftpb"
+)
+
+// a network interface
+type iface interface {
+	send(m raftpb.Message)
+	recv() chan raftpb.Message
+	disconnect()
+	connect()
+}
+
+// a network
+type network interface {
+	// drop message at given rate (1.0 drops all messages)
+	drop(from, to uint64, rate float64)
+	// delay message for (0, d] randomly at given rate (1.0 delay all messages)
+	// do we need rate here?
+	delay(from, to uint64, d time.Duration, rate float64)
+	disconnect(id uint64)
+	connect(id uint64)
+	// heal heals the network
+	heal()
+}
+
+type raftNetwork struct {
+	mu           sync.Mutex
+	disconnected map[uint64]bool
+	dropmap      map[conn]float64
+	delaymap     map[conn]delay
+	recvQueues   map[uint64]chan raftpb.Message
+}
+
+type conn struct {
+	from, to uint64
+}
+
+type delay struct {
+	d    time.Duration
+	rate float64
+}
+
+func newRaftNetwork(nodes ...uint64) *raftNetwork {
+	pn := &raftNetwork{
+		recvQueues:   make(map[uint64]chan raftpb.Message),
+		dropmap:      make(map[conn]float64),
+		delaymap:     make(map[conn]delay),
+		disconnected: make(map[uint64]bool),
+	}
+
+	for _, n := range nodes {
+		pn.recvQueues[n] = make(chan raftpb.Message, 1024)
+	}
+	return pn
+}
+
+func (rn *raftNetwork) nodeNetwork(id uint64) iface {
+	return &nodeNetwork{id: id, raftNetwork: rn}
+}
+
+func (rn *raftNetwork) send(m raftpb.Message) {
+	rn.mu.Lock()
+	to := rn.recvQueues[m.To]
+	if rn.disconnected[m.To] {
+		to = nil
+	}
+	drop := rn.dropmap[conn{m.From, m.To}]
+	delay := rn.delaymap[conn{m.From, m.To}]
+	rn.mu.Unlock()
+
+	if to == nil {
+		return
+	}
+	if drop != 0 && rand.Float64() < drop {
+		return
+	}
+	// TODO: shall we delay without blocking the send call?
+	if delay.d != 0 && rand.Float64() < delay.rate {
+		rd := rand.Int63n(int64(delay.d))
+		time.Sleep(time.Duration(rd))
+	}
+
+	select {
+	case to <- m:
+	default:
+		// drop messages when the receiver queue is full.
+	}
+}
+
+func (rn *raftNetwork) recvFrom(from uint64) chan raftpb.Message {
+	rn.mu.Lock()
+	fromc := rn.recvQueues[from]
+	if rn.disconnected[from] {
+		fromc = nil
+	}
+	rn.mu.Unlock()
+
+	return fromc
+}
+
+func (rn *raftNetwork) drop(from, to uint64, rate float64) {
+	rn.mu.Lock()
+	defer rn.mu.Unlock()
+	rn.dropmap[conn{from, to}] = rate
+}
+
+func (rn *raftNetwork) delay(from, to uint64, d time.Duration, rate float64) {
+	rn.mu.Lock()
+	defer rn.mu.Unlock()
+	rn.delaymap[conn{from, to}] = delay{d, rate}
+}
+
+func (rn *raftNetwork) heal() {
+	rn.mu.Lock()
+	defer rn.mu.Unlock()
+	rn.dropmap = make(map[conn]float64)
+	rn.delaymap = make(map[conn]delay)
+}
+
+func (rn *raftNetwork) disconnect(id uint64) {
+	rn.mu.Lock()
+	defer rn.mu.Unlock()
+	rn.disconnected[id] = true
+}
+
+func (rn *raftNetwork) connect(id uint64) {
+	rn.mu.Lock()
+	defer rn.mu.Unlock()
+	rn.disconnected[id] = false
+}
+
+type nodeNetwork struct {
+	id uint64
+	*raftNetwork
+}
+
+func (nt *nodeNetwork) connect() {
+	nt.raftNetwork.connect(nt.id)
+}
+
+func (nt *nodeNetwork) disconnect() {
+	nt.raftNetwork.disconnect(nt.id)
+}
+
+func (nt *nodeNetwork) send(m raftpb.Message) {
+	nt.raftNetwork.send(m)
+}
+
+func (nt *nodeNetwork) recv() chan raftpb.Message {
+	return nt.recvFrom(nt.id)
+}

raft/rafttest/network_test.go

@@ -0,0 +1,58 @@
+package rafttest
+
+import (
+	"testing"
+	"time"
+
+	"github.com/coreos/etcd/raft/raftpb"
+)
+
+func TestNetworkDrop(t *testing.T) {
+	// drop around 10% messages
+	sent := 1000
+	droprate := 0.1
+	nt := newRaftNetwork(1, 2)
+	nt.drop(1, 2, droprate)
+	for i := 0; i < sent; i++ {
+		nt.send(raftpb.Message{From: 1, To: 2})
+	}
+
+	c := nt.recvFrom(2)
+
+	received := 0
+	done := false
+	for !done {
+		select {
+		case <-c:
+			received++
+		default:
+			done = true
+		}
+	}
+
+	drop := sent - received
+	if drop > int((droprate+0.1)*float64(sent)) || drop < int((droprate-0.1)*float64(sent)) {
+		t.Errorf("drop = %d, want around %d", drop, droprate*float64(sent))
+	}
+}
+
+func TestNetworkDelay(t *testing.T) {
+	sent := 1000
+	delay := time.Millisecond
+	delayrate := 0.1
+	nt := newRaftNetwork(1, 2)
+
+	nt.delay(1, 2, delay, delayrate)
+	var total time.Duration
+	for i := 0; i < sent; i++ {
+		s := time.Now()
+		nt.send(raftpb.Message{From: 1, To: 2})
+		total += time.Since(s)
+	}
+
+	w := time.Duration(float64(sent)*delayrate/2) * delay
+	// there are pretty overhead in the send call, since it genarete random numbers.
+	if total < w+10*delay {
+		t.Errorf("total = %v, want > %v", total, w)
+	}
+}

raft/rafttest/node.go

@@ -0,0 +1,114 @@
+package rafttest
+
+import (
+	"log"
+	"time"
+
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
+	"github.com/coreos/etcd/raft"
+	"github.com/coreos/etcd/raft/raftpb"
+)
+
+type node struct {
+	raft.Node
+	id     uint64
+	iface  iface
+	stopc  chan struct{}
+	pausec chan bool
+
+	// stable
+	storage *raft.MemoryStorage
+	state   raftpb.HardState
+}
+
+func startNode(id uint64, peers []raft.Peer, iface iface) *node {
+	st := raft.NewMemoryStorage()
+	rn := raft.StartNode(id, peers, 10, 1, st)
+	n := &node{
+		Node:    rn,
+		id:      id,
+		storage: st,
+		iface:   iface,
+		pausec:  make(chan bool),
+	}
+	n.start()
+	return n
+}
+
+func (n *node) start() {
+	n.stopc = make(chan struct{})
+	ticker := time.Tick(5 * time.Millisecond)
+
+	go func() {
+		for {
+			select {
+			case <-ticker:
+				n.Tick()
+			case rd := <-n.Ready():
+				if !raft.IsEmptyHardState(rd.HardState) {
+					n.state = rd.HardState
+					n.storage.SetHardState(n.state)
+				}
+				n.storage.Append(rd.Entries)
+				// TODO: make send async, more like real world...
+				for _, m := range rd.Messages {
+					n.iface.send(m)
+				}
+				n.Advance()
+			case m := <-n.iface.recv():
+				n.Step(context.TODO(), m)
+			case <-n.stopc:
+				n.Stop()
+				log.Printf("raft.%d: stop", n.id)
+				n.Node = nil
+				close(n.stopc)
+				return
+			case p := <-n.pausec:
+				recvms := make([]raftpb.Message, 0)
+				for p {
+					select {
+					case m := <-n.iface.recv():
+						recvms = append(recvms, m)
+					case p = <-n.pausec:
+					}
+				}
+				// step all pending messages
+				for _, m := range recvms {
+					n.Step(context.TODO(), m)
+				}
+			}
+		}
+	}()
+}
+
+// stop stops the node. stop a stopped node might panic.
+// All in memory state of node is discarded.
+// All stable MUST be unchanged.
+func (n *node) stop() {
+	n.iface.disconnect()
+	n.stopc <- struct{}{}
+	// wait for the shutdown
+	<-n.stopc
+}
+
+// restart restarts the node. restart a started node
+// blocks and might affect the future stop operation.
+func (n *node) restart() {
+	// wait for the shutdown
+	<-n.stopc
+	n.Node = raft.RestartNode(n.id, 10, 1, n.storage, 0)
+	n.start()
+	n.iface.connect()
+}
+
+// pause pauses the node.
+// The paused node buffers the received messages and replies
+// all of them when it resumes.
+func (n *node) pause() {
+	n.pausec <- true
+}
+
+// resume resumes the paused node.
+func (n *node) resume() {
+	n.pausec <- false
+}

raft/rafttest/node_test.go

@@ -0,0 +1,112 @@
+package rafttest
+
+import (
+	"testing"
+	"time"
+
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
+	"github.com/coreos/etcd/raft"
+)
+
+func TestBasicProgress(t *testing.T) {
+	peers := []raft.Peer{{1, nil}, {2, nil}, {3, nil}, {4, nil}, {5, nil}}
+	nt := newRaftNetwork(1, 2, 3, 4, 5)
+
+	nodes := make([]*node, 0)
+
+	for i := 1; i <= 5; i++ {
+		n := startNode(uint64(i), peers, nt.nodeNetwork(uint64(i)))
+		nodes = append(nodes, n)
+	}
+
+	time.Sleep(50 * time.Millisecond)
+	for i := 0; i < 1000; i++ {
+		nodes[0].Propose(context.TODO(), []byte("somedata"))
+	}
+
+	time.Sleep(100 * time.Millisecond)
+	for _, n := range nodes {
+		n.stop()
+		if n.state.Commit != 1006 {
+			t.Errorf("commit = %d, want = 1006", n.state.Commit)
+		}
+	}
+}
+
+func TestRestart(t *testing.T) {
+	peers := []raft.Peer{{1, nil}, {2, nil}, {3, nil}, {4, nil}, {5, nil}}
+	nt := newRaftNetwork(1, 2, 3, 4, 5)
+
+	nodes := make([]*node, 0)
+
+	for i := 1; i <= 5; i++ {
+		n := startNode(uint64(i), peers, nt.nodeNetwork(uint64(i)))
+		nodes = append(nodes, n)
+	}
+
+	time.Sleep(50 * time.Millisecond)
+	for i := 0; i < 300; i++ {
+		nodes[0].Propose(context.TODO(), []byte("somedata"))
+	}
+	nodes[1].stop()
+	for i := 0; i < 300; i++ {
+		nodes[0].Propose(context.TODO(), []byte("somedata"))
+	}
+	nodes[2].stop()
+	for i := 0; i < 300; i++ {
+		nodes[0].Propose(context.TODO(), []byte("somedata"))
+	}
+	nodes[2].restart()
+	for i := 0; i < 300; i++ {
+		nodes[0].Propose(context.TODO(), []byte("somedata"))
+	}
+	nodes[1].restart()
+
+	// give some time for nodes to catch up with the raft leader
+	time.Sleep(300 * time.Millisecond)
+	for _, n := range nodes {
+		n.stop()
+		if n.state.Commit != 1206 {
+			t.Errorf("commit = %d, want = 1206", n.state.Commit)
+		}
+	}
+}
+
+func TestPause(t *testing.T) {
+	peers := []raft.Peer{{1, nil}, {2, nil}, {3, nil}, {4, nil}, {5, nil}}
+	nt := newRaftNetwork(1, 2, 3, 4, 5)
+
+	nodes := make([]*node, 0)
+
+	for i := 1; i <= 5; i++ {
+		n := startNode(uint64(i), peers, nt.nodeNetwork(uint64(i)))
+		nodes = append(nodes, n)
+	}
+
+	time.Sleep(50 * time.Millisecond)
+	for i := 0; i < 300; i++ {
+		nodes[0].Propose(context.TODO(), []byte("somedata"))
+	}
+	nodes[1].pause()
+	for i := 0; i < 300; i++ {
+		nodes[0].Propose(context.TODO(), []byte("somedata"))
+	}
+	nodes[2].pause()
+	for i := 0; i < 300; i++ {
+		nodes[0].Propose(context.TODO(), []byte("somedata"))
+	}
+	nodes[2].resume()
+	for i := 0; i < 300; i++ {
+		nodes[0].Propose(context.TODO(), []byte("somedata"))
+	}
+	nodes[1].resume()
+
+	// give some time for nodes to catch up with the raft leader
+	time.Sleep(300 * time.Millisecond)
+	for _, n := range nodes {
+		n.stop()
+		if n.state.Commit != 1206 {
+			t.Errorf("commit = %d, want = 1206", n.state.Commit)
+		}
+	}
+}

raft/util.go

@@ -84,10 +84,10 @@ func DescribeMessage(m pb.Message, f EntryFormatter) string {
 // Entry for debugging.
 func DescribeEntry(e pb.Entry, f EntryFormatter) string {
 	var formatted string
-	if f == nil {
-		formatted = fmt.Sprintf("%q", e.Data)
-	} else {
+	if e.Type == pb.EntryNormal && f != nil {
 		formatted = f(e.Data)
+	} else {
+		formatted = fmt.Sprintf("%q", e.Data)
 	}
 	return fmt.Sprintf("%d/%d %s %s", e.Term, e.Index, e.Type, formatted)
 }

rafthttp/entry_reader.go

@@ -54,7 +54,9 @@ func (er *entryReader) readEntries() ([]raftpb.Entry, error) {
 		}
 		er.ents.Add()
 	}
-	er.lastIndex.Set(int64(ents[l-1].Index))
+	if l > 0 {
+		er.lastIndex.Set(int64(ents[l-1].Index))
+	}
 	return ents, nil
 }
 

rafthttp/peer.go

@@ -196,7 +196,7 @@ func (p *peer) handle() {
 				p.errored = err
 			}
 			if p.active {
-				log.Printf("sender: the connection with %s becomes inactive", p.id)
+				log.Printf("sender: the connection with %s became inactive", p.id)
 				p.active = false
 			}
 			if m.Type == raftpb.MsgApp {
@@ -204,7 +204,7 @@ func (p *peer) handle() {
 			}
 		} else {
 			if !p.active {
-				log.Printf("sender: the connection with %s becomes active", p.id)
+				log.Printf("sender: the connection with %s became active", p.id)
 				p.active = true
 				p.errored = nil
 			}

rafthttp/streamer.go

@@ -194,6 +194,12 @@ func (s *streamWriter) handle(w WriteFlusher) {
 	ew := newEntryWriter(w, s.to)
 	defer ew.stop()
 	for ents := range s.q {
+		// Considering Commit in MsgApp is not recovered when received,
+		// zero-entry appendEntry messages have no use to raft state machine.
+		// Drop it here because it is useless.
+		if len(ents) == 0 {
+			continue
+		}
 		start := time.Now()
 		if err := ew.writeEntries(ents); err != nil {
 			log.Printf("rafthttp: encountered error writing to server log stream: %v", err)
@@ -289,9 +295,6 @@ func (s *streamReader) handle(r io.Reader) {
 			}
 			return
 		}
-		// Considering Commit in MsgApp is not recovered, zero-entry appendEntry
-		// messages have no use to raft state machine. Drop it here because
-		// we don't have easy way to recover its Index easily.
 		if len(ents) == 0 {
 			continue
 		}

rafthttp/transport.go

@@ -37,6 +37,7 @@ type Transporter interface {
 	Send(m []raftpb.Message)
 	AddPeer(id types.ID, urls []string)
 	RemovePeer(id types.ID)
+	RemoveAllPeers()
 	UpdatePeer(id types.ID, urls []string)
 	Stop()
 }
@@ -132,8 +133,26 @@ func (t *transport) AddPeer(id types.ID, urls []string) {
 func (t *transport) RemovePeer(id types.ID) {
 	t.mu.Lock()
 	defer t.mu.Unlock()
-	t.peers[id].Stop()
+	t.removePeer(id)
+}
+
+func (t *transport) RemoveAllPeers() {
+	t.mu.Lock()
+	defer t.mu.Unlock()
+	for id, _ := range t.peers {
+		t.removePeer(id)
+	}
+}
+
+// the caller of this function must have the peers mutex.
+func (t *transport) removePeer(id types.ID) {
+	if peer, ok := t.peers[id]; ok {
+		peer.Stop()
+	} else {
+		log.Panicf("rafthttp: unexpected removal of unknown peer '%d'", id)
+	}
 	delete(t.peers, id)
+	delete(t.leaderStats.Followers, id.String())
 }
 
 func (t *transport) UpdatePeer(id types.ID, urls []string) {

scripts/build-release

@@ -42,7 +42,7 @@ function package {
 	cp etcd/README.md ${target}/README.md
 	cp etcd/etcdctl/README.md ${target}/README-etcdctl.md
 
-	cp -R etcd/Documentation/2.0 ${target}/Documentation
+	cp -R etcd/Documentation ${target}/Documentation
 }
 
 function main {

snap/snapshotter.go

@@ -86,25 +86,25 @@ func (s *Snapshotter) Load() (*raftpb.Snapshot, error) {
 }
 
 func loadSnap(dir, name string) (*raftpb.Snapshot, error) {
-	var err error
-	var b []byte
-
 	fpath := path.Join(dir, name)
-	defer func() {
-		if err != nil {
-			renameBroken(fpath)
-		}
-	}()
-
-	b, err = ioutil.ReadFile(fpath)
+	snap, err := Read(fpath)
 	if err != nil {
-		log.Printf("snap: snapshotter cannot read file %v: %v", name, err)
+		renameBroken(fpath)
+	}
+	return snap, err
+}
+
+// Read reads the snapshot named by snapname and returns the snapshot.
+func Read(snapname string) (*raftpb.Snapshot, error) {
+	b, err := ioutil.ReadFile(snapname)
+	if err != nil {
+		log.Printf("snap: snapshotter cannot read file %v: %v", snapname, err)
 		return nil, err
 	}
 
 	var serializedSnap snappb.Snapshot
 	if err = serializedSnap.Unmarshal(b); err != nil {
-		log.Printf("snap: corrupted snapshot file %v: %v", name, err)
+		log.Printf("snap: corrupted snapshot file %v: %v", snapname, err)
 		return nil, err
 	}
 
@@ -115,13 +115,13 @@ func loadSnap(dir, name string) (*raftpb.Snapshot, error) {
 
 	crc := crc32.Update(0, crcTable, serializedSnap.Data)
 	if crc != serializedSnap.Crc {
-		log.Printf("snap: corrupted snapshot file %v: crc mismatch", name)
+		log.Printf("snap: corrupted snapshot file %v: crc mismatch", snapname)
 		return nil, ErrCRCMismatch
 	}
 
 	var snap raftpb.Snapshot
 	if err = snap.Unmarshal(serializedSnap.Data); err != nil {
-		log.Printf("snap: corrupted snapshot file %v: %v", name, err)
+		log.Printf("snap: corrupted snapshot file %v: %v", snapname, err)
 		return nil, err
 	}
 	return &snap, nil

store/event_test.go

@@ -88,8 +88,8 @@ func TestFullEventQueue(t *testing.T) {
 
 	// Add
 	for i := 0; i < 1000; i++ {
-		e := newEvent(Create, "/foo", uint64(i), uint64(i))
-		eh.addEvent(e)
+		ce := newEvent(Create, "/foo", uint64(i), uint64(i))
+		eh.addEvent(ce)
 		e, err := eh.scan("/foo", true, uint64(i-1))
 		if i > 0 {
 			if e == nil || err != nil {

store/heap_test.go

@@ -51,10 +51,10 @@ func TestHeapUpdate(t *testing.T) {
 
 	// add from older expire time to earlier expire time
 	// the path is equal to ttl from now
-	for i, n := range kvs {
+	for i := range kvs {
 		path := fmt.Sprintf("%v", 10-i)
 		m := time.Duration(10 - i)
-		n = newKV(nil, path, path, 0, nil, "", time.Now().Add(time.Second*m))
+		n := newKV(nil, path, path, 0, nil, "", time.Now().Add(time.Second*m))
 		kvs[i] = n
 		h.push(n)
 	}

store/node.go

@@ -369,10 +369,13 @@ func (n *node) Compare(prevValue string, prevIndex uint64) (ok bool, which int)
 // If the node is a key-value pair, it will clone the pair.
 func (n *node) Clone() *node {
 	if !n.IsDir() {
-		return newKV(n.store, n.Path, n.Value, n.CreatedIndex, n.Parent, n.ACL, n.ExpireTime)
+		newkv := newKV(n.store, n.Path, n.Value, n.CreatedIndex, n.Parent, n.ACL, n.ExpireTime)
+		newkv.ModifiedIndex = n.ModifiedIndex
+		return newkv
 	}
 
 	clone := newDir(n.store, n.Path, n.CreatedIndex, n.Parent, n.ACL, n.ExpireTime)
+	clone.ModifiedIndex = n.ModifiedIndex
 
 	for key, child := range n.Children {
 		clone.Children[key] = child.Clone()

store/store.go

@@ -25,6 +25,7 @@ import (
 
 	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/jonboulle/clockwork"
 	etcdErr "github.com/coreos/etcd/error"
+	"github.com/coreos/etcd/pkg/types"
 )
 
 // The default version to set when the store is first initialized.
@@ -68,21 +69,27 @@ type store struct {
 	ttlKeyHeap     *ttlKeyHeap  // need to recovery manually
 	worldLock      sync.RWMutex // stop the world lock
 	clock          clockwork.Clock
+	readonlySet    types.Set
 }
 
-func New() Store {
-	s := newStore()
+// The given namespaces will be created as initial directories in the returned store.
+func New(namespaces ...string) Store {
+	s := newStore(namespaces...)
 	s.clock = clockwork.NewRealClock()
 	return s
 }
 
-func newStore() *store {
+func newStore(namespaces ...string) *store {
 	s := new(store)
 	s.CurrentVersion = defaultVersion
 	s.Root = newDir(s, "/", s.CurrentIndex, nil, "", Permanent)
+	for _, namespace := range namespaces {
+		s.Root.Add(newDir(s, namespace, s.CurrentIndex, s.Root, "", Permanent))
+	}
 	s.Stats = newStats()
 	s.WatcherHub = newWatchHub(1000)
 	s.ttlKeyHeap = newTtlKeyHeap()
+	s.readonlySet = types.NewUnsafeSet(append(namespaces, "/")...)
 	return s
 }
 
@@ -203,7 +210,7 @@ func (s *store) CompareAndSwap(nodePath string, prevValue string, prevIndex uint
 
 	nodePath = path.Clean(path.Join("/", nodePath))
 	// we do not allow the user to change "/"
-	if nodePath == "/" {
+	if s.readonlySet.Contains(nodePath) {
 		return nil, etcdErr.NewError(etcdErr.EcodeRootROnly, "/", s.CurrentIndex)
 	}
 
@@ -258,7 +265,7 @@ func (s *store) Delete(nodePath string, dir, recursive bool) (*Event, error) {
 
 	nodePath = path.Clean(path.Join("/", nodePath))
 	// we do not allow the user to change "/"
-	if nodePath == "/" {
+	if s.readonlySet.Contains(nodePath) {
 		return nil, etcdErr.NewError(etcdErr.EcodeRootROnly, "/", s.CurrentIndex)
 	}
 
@@ -401,7 +408,7 @@ func (s *store) Update(nodePath string, newValue string, expireTime time.Time) (
 
 	nodePath = path.Clean(path.Join("/", nodePath))
 	// we do not allow the user to change "/"
-	if nodePath == "/" {
+	if s.readonlySet.Contains(nodePath) {
 		return nil, etcdErr.NewError(etcdErr.EcodeRootROnly, "/", s.CurrentIndex)
 	}
 
@@ -461,7 +468,7 @@ func (s *store) internalCreate(nodePath string, dir bool, value string, unique,
 	nodePath = path.Clean(path.Join("/", nodePath))
 
 	// we do not allow the user to change "/"
-	if nodePath == "/" {
+	if s.readonlySet.Contains(nodePath) {
 		return nil, etcdErr.NewError(etcdErr.EcodeRootROnly, "/", currIndex)
 	}
 

store/store_test.go

@@ -23,6 +23,15 @@ import (
 	etcdErr "github.com/coreos/etcd/error"
 )
 
+func TestNewStoreWithNamespaces(t *testing.T) {
+	s := newStore("/0", "/1")
+
+	_, err := s.Get("/0", false, false)
+	assert.Nil(t, err, "")
+	_, err = s.Get("/1", false, false)
+	assert.Nil(t, err, "")
+}
+
 // Ensure that the store can retrieve an existing value.
 func TestStoreGetValue(t *testing.T) {
 	s := newStore()
@@ -433,22 +442,24 @@ func TestStoreDeleteDiretoryFailsIfNonRecursiveAndDir(t *testing.T) {
 }
 
 func TestRootRdOnly(t *testing.T) {
-	s := newStore()
+	s := newStore("/0")
 
-	_, err := s.Set("/", true, "", Permanent)
-	assert.NotNil(t, err, "")
+	for _, tt := range []string{"/", "/0"} {
+		_, err := s.Set(tt, true, "", Permanent)
+		assert.NotNil(t, err, "")
 
-	_, err = s.Delete("/", true, true)
-	assert.NotNil(t, err, "")
+		_, err = s.Delete(tt, true, true)
+		assert.NotNil(t, err, "")
 
-	_, err = s.Create("/", true, "", false, Permanent)
-	assert.NotNil(t, err, "")
+		_, err = s.Create(tt, true, "", false, Permanent)
+		assert.NotNil(t, err, "")
 
-	_, err = s.Update("/", "", Permanent)
-	assert.NotNil(t, err, "")
+		_, err = s.Update(tt, "", Permanent)
+		assert.NotNil(t, err, "")
 
-	_, err = s.CompareAndSwap("/", "", 0, "", Permanent)
-	assert.NotNil(t, err, "")
+		_, err = s.CompareAndSwap(tt, "", 0, "", Permanent)
+		assert.NotNil(t, err, "")
+	}
 }
 
 func TestStoreCompareAndDeletePrevValue(t *testing.T) {
@@ -778,9 +789,10 @@ func TestStoreWatchStream(t *testing.T) {
 // Ensure that the store can recover from a previously saved state.
 func TestStoreRecover(t *testing.T) {
 	s := newStore()
-	var eidx uint64 = 3
+	var eidx uint64 = 4
 	s.Create("/foo", true, "", false, Permanent)
 	s.Create("/foo/x", false, "bar", false, Permanent)
+	s.Update("/foo/x", "barbar", Permanent)
 	s.Create("/foo/y", false, "baz", false, Permanent)
 	b, err := s.Save()
 
@@ -788,9 +800,11 @@ func TestStoreRecover(t *testing.T) {
 	s2.Recovery(b)
 
 	e, err := s.Get("/foo/x", false, false)
+	assert.Equal(t, e.Node.CreatedIndex, uint64(2), "")
+	assert.Equal(t, e.Node.ModifiedIndex, uint64(3), "")
 	assert.Equal(t, e.EtcdIndex, eidx, "")
 	assert.Nil(t, err, "")
-	assert.Equal(t, *e.Node.Value, "bar", "")
+	assert.Equal(t, *e.Node.Value, "barbar", "")
 
 	e, err = s.Get("/foo/y", false, false)
 	assert.Equal(t, e.EtcdIndex, eidx, "")

test

@@ -15,7 +15,7 @@ COVER=${COVER:-"-cover"}
 source ./build
 
 # Hack: gofmt ./ will recursively check the .git directory. So use *.go for gofmt.
-TESTABLE_AND_FORMATTABLE="client discovery error etcdctl/command etcdmain etcdserver etcdserver/etcdhttp etcdserver/etcdhttp/httptypes migrate pkg/fileutil pkg/flags pkg/idutil pkg/ioutil pkg/netutil pkg/pbutil pkg/types pkg/transport pkg/wait proxy raft rafthttp snap store wal"
+TESTABLE_AND_FORMATTABLE="client discovery error etcdctl/command etcdmain etcdserver etcdserver/etcdhttp etcdserver/etcdhttp/httptypes migrate pkg/fileutil pkg/flags pkg/idutil pkg/ioutil pkg/netutil pkg/osutil pkg/pbutil pkg/types pkg/transport pkg/wait proxy raft rafthttp snap store wal"
 FORMATTABLE="$TESTABLE_AND_FORMATTABLE *.go etcdctl/ integration"
 
 # user has not provided PKG override
@@ -60,4 +60,13 @@ if [ -n "${vetRes}" ]; then
 	exit 255
 fi
 
+if command -v go-nyet >/dev/null 2>&1; then
+  echo "Checking go-nyet..."
+  nyetRes=$(go-nyet -exitWith 0 $FMT)
+  if [ -n "${nyetRes}" ]; then
+          echo -e "go-nyet checking failed:\n${nyetRes}"
+          exit 255
+  fi
+fi
+
 echo "Success"

tools/etcd-dump-logs/main.go

@@ -32,24 +32,47 @@ import (
 
 func main() {
 	from := flag.String("data-dir", "", "")
+	snapfile := flag.String("start-snap", "", "The base name of snapshot file to start dumping")
+	index := flag.Uint64("start-index", 0, "The index to start dumping")
 	flag.Parse()
 	if *from == "" {
-		log.Fatal("Must provide -data-dir flag")
+		log.Fatal("Must provide -data-dir flag.")
+	}
+	if *snapfile != "" && *index != 0 {
+		log.Fatal("start-snap and start-index flags cannot be used together.")
 	}
 
-	ss := snap.New(snapDir(*from))
-	snapshot, err := ss.Load()
-	var walsnap walpb.Snapshot
-	switch err {
-	case nil:
-		walsnap.Index, walsnap.Term = snapshot.Metadata.Index, snapshot.Metadata.Term
-		nodes := genIDSlice(snapshot.Metadata.ConfState.Nodes)
-		fmt.Printf("Snapshot:\nterm=%d index=%d nodes=%s\n",
-			walsnap.Term, walsnap.Index, nodes)
-	case snap.ErrNoSnapshot:
-		fmt.Printf("Snapshot:\nempty\n")
-	default:
-		log.Fatalf("Failed loading snapshot: %v", err)
+	var (
+		walsnap  walpb.Snapshot
+		snapshot *raftpb.Snapshot
+		err      error
+	)
+
+	isIndex := *index != 0
+
+	if isIndex {
+		fmt.Printf("Start dumping log entries from index %d.\n", *index)
+		walsnap.Index = *index
+	} else {
+		if *snapfile == "" {
+			ss := snap.New(snapDir(*from))
+			snapshot, err = ss.Load()
+		} else {
+			snapshot, err = snap.Read(path.Join(snapDir(*from), *snapfile))
+		}
+
+		switch err {
+		case nil:
+			walsnap.Index, walsnap.Term = snapshot.Metadata.Index, snapshot.Metadata.Term
+			nodes := genIDSlice(snapshot.Metadata.ConfState.Nodes)
+			fmt.Printf("Snapshot:\nterm=%d index=%d nodes=%s\n",
+				walsnap.Term, walsnap.Index, nodes)
+		case snap.ErrNoSnapshot:
+			fmt.Printf("Snapshot:\nempty\n")
+		default:
+			log.Fatalf("Failed loading snapshot: %v", err)
+		}
+		fmt.Println("Start dupmping log entries from snapshot.")
 	}
 
 	w, err := wal.Open(walDir(*from), walsnap)
@@ -58,7 +81,7 @@ func main() {
 	}
 	wmetadata, state, ents, err := w.ReadAll()
 	w.Close()
-	if err != nil {
+	if err != nil && (!isIndex || err != wal.ErrSnapshotNotFound) {
 		log.Fatalf("Failed reading WAL: %v", err)
 	}
 	id, cid := parseWALMetadata(wmetadata)
@@ -102,9 +125,9 @@ func main() {
 	}
 }
 
-func walDir(dataDir string) string { return path.Join(dataDir, "wal") }
+func walDir(dataDir string) string { return path.Join(dataDir, "member", "wal") }
 
-func snapDir(dataDir string) string { return path.Join(dataDir, "snap") }
+func snapDir(dataDir string) string { return path.Join(dataDir, "member", "snap") }
 
 func parseWALMetadata(b []byte) (id, cid types.ID) {
 	var metadata etcdserverpb.Metadata

version/version.go

@@ -15,6 +15,6 @@
 package version
 
 var (
-	Version         = "2.0.0"
+	Version         = "2.0.4"
 	InternalVersion = "2"
 )

wal/util.go

@@ -28,10 +28,12 @@ import (
 type WalVersion string
 
 const (
-	WALUnknown  WalVersion = "Unknown WAL"
-	WALNotExist WalVersion = "No WAL"
-	WALv0_4     WalVersion = "0.4.x"
-	WALv0_5     WalVersion = "0.5.x"
+	WALUnknown   WalVersion = "Unknown WAL"
+	WALNotExist  WalVersion = "No WAL"
+	WALv0_4      WalVersion = "0.4.x"
+	WALv2_0      WalVersion = "2.0.0"
+	WALv2_0Proxy WalVersion = "2.0 proxy"
+	WALv2_0_1    WalVersion = "2.0.1"
 )
 
 func DetectVersion(dirpath string) (WalVersion, error) {
@@ -48,15 +50,31 @@ func DetectVersion(dirpath string) (WalVersion, error) {
 		return WALNotExist, nil
 	}
 	nameSet := types.NewUnsafeSet(names...)
+	if nameSet.Contains("member") {
+		ver, err := DetectVersion(path.Join(dirpath, "member"))
+		if ver == WALv2_0 {
+			return WALv2_0_1, nil
+		} else if ver == WALv0_4 {
+			// How in the blazes did it get there?
+			return WALUnknown, nil
+		}
+		return ver, err
+	}
 	if nameSet.ContainsAll([]string{"snap", "wal"}) {
 		// .../wal cannot be empty to exist.
 		if Exist(path.Join(dirpath, "wal")) {
-			return WALv0_5, nil
+			return WALv2_0, nil
 		}
 	}
+	if nameSet.ContainsAll([]string{"proxy"}) {
+		return WALv2_0Proxy, nil
+	}
 	if nameSet.ContainsAll([]string{"snapshot", "conf", "log"}) {
 		return WALv0_4, nil
 	}
+	if nameSet.ContainsAll([]string{"standby_info"}) {
+		return WALv0_4, nil
+	}
 
 	return WALUnknown, nil
 }

wal/util_test.go

@@ -28,7 +28,8 @@ func TestDetectVersion(t *testing.T) {
 		wver  WalVersion
 	}{
 		{[]string{}, WALNotExist},
-		{[]string{"snap/", "wal/", "wal/1"}, WALv0_5},
+		{[]string{"member/", "member/wal/", "member/wal/1", "member/snap/"}, WALv2_0_1},
+		{[]string{"snap/", "wal/", "wal/1"}, WALv2_0},
 		{[]string{"snapshot/", "conf", "log"}, WALv0_4},
 		{[]string{"weird"}, WALUnknown},
 		{[]string{"snap/", "wal/"}, WALUnknown},

wal/wal.go

@@ -203,7 +203,7 @@ func openAtIndex(dirpath string, snap walpb.Snapshot, all bool) (*WAL, error) {
 // ReadAll reads out all records of the current WAL.
 // If it cannot read out the expected snap, it will return ErrSnapshotNotFound.
 // If loaded snap doesn't match with the expected one, it will return
-// ErrSnapshotMismatch.
+// all the records and error ErrSnapshotMismatch.
 // TODO: detect not-last-snap error.
 // TODO: maybe loose the checking of match.
 // After ReadAll, the WAL will be ready for appending new records.
@@ -256,9 +256,9 @@ func (w *WAL) ReadAll() (metadata []byte, state raftpb.HardState, ents []raftpb.
 		state.Reset()
 		return nil, state, nil, err
 	}
+	err = nil
 	if !match {
-		state.Reset()
-		return nil, state, nil, ErrSnapshotNotFound
+		err = ErrSnapshotNotFound
 	}
 
 	// close decoder, disable reading
@@ -269,7 +269,7 @@ func (w *WAL) ReadAll() (metadata []byte, state raftpb.HardState, ents []raftpb.
 	// create encoder (chain crc with the decoder), enable appending
 	w.encoder = newEncoder(w.f, w.decoder.lastCRC())
 	w.decoder = nil
-	return metadata, state, ents, nil
+	return metadata, state, ents, err
 }
 
 // Cut closes current file written and creates a new one ready to append.

wal/wal_test.go

@@ -323,23 +323,23 @@ func TestRecoverAfterCut(t *testing.T) {
 	}
 	defer os.RemoveAll(p)
 
-	w, err := Create(p, []byte("metadata"))
+	md, err := Create(p, []byte("metadata"))
 	if err != nil {
 		t.Fatal(err)
 	}
 	for i := 0; i < 10; i++ {
-		if err = w.SaveSnapshot(walpb.Snapshot{Index: uint64(i)}); err != nil {
+		if err = md.SaveSnapshot(walpb.Snapshot{Index: uint64(i)}); err != nil {
 			t.Fatal(err)
 		}
 		es := []raftpb.Entry{{Index: uint64(i)}}
-		if err = w.Save(raftpb.HardState{}, es); err != nil {
+		if err = md.Save(raftpb.HardState{}, es); err != nil {
 			t.Fatal(err)
 		}
-		if err = w.Cut(); err != nil {
+		if err = md.Cut(); err != nil {
 			t.Fatal(err)
 		}
 	}
-	w.Close()
+	md.Close()
 
 	if err := os.Remove(path.Join(p, walName(4, 4))); err != nil {
 		t.Fatal(err)