Lazyfs adds additional overhead causing arm64 machines in CI to not reach robustness test qps requirements due to lack of compute power.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
Creation of event history requires each client to return consistent
events. If clients observed inconsistent view of some revision, merging
will fail and return diff between two clients. This however doesn't
provide hint on what kind of issue happend.
This PR helps cases where there is an error with single watch
stream (like event duplication) by running normal watch validation even
without full event history.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
scheduleCompaction function
To improve traceability of backend database usage, Added below parameter
related to backend database usage metrics inside scheduledCompaction
function.
current-db-size-bytes
current-db-size
current-db-size-in-use-bytes
current-db-size-in-use
Signed-off-by: Rahul More <rahulbapumore@gmail.com>
Unlike SnapshotWithVersion, the client.Snapshot doesn't wait for first
response. The server could open db after we close connection or shutdown
the server. We can read few bytes to ensure server opens boltdb.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Hi,
I think, it's a high time for me to retire. No contributions over last 6m.
There is a hope, I will be back working on etcd, but over last 6m it kept being just 2 months away.
Thank you all,
Piotr
Signed-off-by: Piotr Tabor <piotr.tabor@gmail.com>
This should not change the current behavior, but will cause dependabot
to update the digest of the latest base image weekly.
Signed-off-by: Wes Panther <wpanther@google.com>
We have completed memory profiling for our workflows so can consider removing this additional action now.
Signed-off-by: James Blair <mail@jamesblair.net>
1. ignore old leader's leases revoking request
2. double check current member's leadership before perform lease renew request
3. etcdserve: ensure current member's leadership before performing lease checkpoint request
Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
Ignore Raft messages when the `To` field mismatches the local member ID.
In cases where incorrect Raft messages are dispatched, potentially due
to a malfunctioning switch, this proactive check prevents panics,
such as "tocommit is out of range".
Signed-off-by: Neil Shen <overvenus@gmail.com>
Disable following redirects from peer HTTP communication on the client's side.
Etcd server may run into SSRF (Server-side request forgery) when adding a new
member. If users provide a malicious peer URL, the existing etcd members may be
redirected to another unexpected internal URL when getting the new member's
version.
Signed-off-by: Ivan Valdes <ivan@vald.es>
If one of nodes is unhealthy the test would never finish as watchers
would never reach max revision.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
v2 store is no longer available in v3.6.
We can remove apply logic for it as they will never be used.
Only v2 PUT is neeeded as it applies to v3 storage and etcd v3.5 uses it for setting member
attributes and cluster version.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
It's possible that etcd server may run into SSRF situation when adding a new member. If users provide a malicious peer URL, the existing etcd members may be redirected to other unexpected internal URL when getting the new member's version.
Signed-off-by: James Blair <mail@jamesblair.net>
KubeconNA 2023 Contribfest issue #16893 . Recently in #16801 we introduced on demand github actions runners for the arm64 platform. Having on demand runner infrastructure in place means we should now have enough capacity to begin running arm64 tests for every pull request. Currently we have:
.github/workflows/e2e-arm64-template.yaml - Shared template
.github/workflows/e2e-arm64-nightly.yaml - Runs template against both
release-3.5 and main branches nightly. Moving forward we can just rename
.github/workflows/e2e-arm64-template.yaml to
.github/workflows/e2e-arm64.yaml and delete the other file. We can then
just make the template file a standard workflow that will run on pull
request.
Signed-off-by: Ming Li <mli103hawk@gmail.com>
Use a template to consolidate arm64 and amd64 test workflows. Enable
running armd64 tests on every push and pull request.
Signed-off-by: Ivan Valdes <ivan@vald.es>
To keep etcd projects up to date with the latest patch releases &
incorporate the latest security updates.
Signed-off-by: Arjun Malhotra <am2cj@virginia.edu>
the bump-revision and mark-compated flags were added to `snapshot restore` and required documentation.
Signed off by Stephen Pawulski spawulski@gmail.com
Signed-off-by: spawulski <spawulski@lotlinx.com>
Trial running a small robustness workflow for arm64 on every pull request
KubeconNA 2023 Contribfest issue #16863
Signed-off-by: Etienne Simard <esimard@ripple.com>
This commit will allow scripts/genproto.sh to fall back to gsed if
available, which is required for the script to work on macOS.
Signed-off-by: Michael Shen <mishen@umich.edu>
Adding workflow_dispatch as an "on" trigger enables
manual testing by maintainers, without having to wait for
the nightly cron schedule.
@ahrtr requested this temporary change in order to trigger
the arm64 jobs via CI.
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
This change switches over from 2x self-managed runners where
side effects are possible between builds, to a pool of servers
where each build runs in an isolated VM.
The service is provided by actuated.dev, and sponsored by
both Ampere and the CNCF.
Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
The #15544 has removed the `build_cov` build. And after go1.20, we use
`-cover` buildflag to enable coverage exporter. We don't need to
maintain main_test.go anymore.
```bash
➜ pwd
/home/fuwei/go/src/go.etcd.io/etcd/etcdctl
➜ go build -o /tmp/etcdctl -cover ./
➜ mkdir /tmp/etcdctl-covdata
➜ GOCOVERDIR=/tmp/etcdctl-covdata /tmp/etcdctl get /health
➜ go tool covdata percent -i=/tmp/etcdctl-covdata
go.etcd.io/etcd/etcdctl/v3 coverage: 66.7% of statements
go.etcd.io/etcd/etcdctl/v3/ctlv3 coverage: 83.3% of statements
go.etcd.io/etcd/etcdctl/v3/ctlv3/command coverage: 15.4% of statements
```
REF: https://go.dev/testing/coverage/
Signed-off-by: Wei Fu <fuweid89@gmail.com>
The osutil.Unsetenv function is not used. Today, os.Unsetenv exists
in the standard library, and is already used elsewhere in etcd.
Removing this function could break things that may be importing this
package. According to pkg.go.dev, it seems like the only thing is
etcd forks, so it may be worth getting rid of this code. See:
https://pkg.go.dev/github.com/coreos/etcd/pkg/osutil?tab=importedby
Signed-off-by: Evan Jones <ej@evanjones.ca>
Fix mixin cluster parameter otherwise generated mixins looks like this:
```json
{"type":"prometheus","uid":"${datasource}"},"label":"cluster","name":"job","query":"label_values(etcd_server_has_leader{job=~\".*etcd.*\"}, job)","refresh":2,"type":"query"}]},"time":{"from":"now-15m","to":"now"},"timezone": "`}}{{ .Values.grafana.defaultDashboardsTimezone }}{{`","title":"etcd","uid":"c2f4e12cdf69feb95caa41a5a1b423d9"}`}}
```
where name is job when the variable name used in dashboard queries is cluster.
Signed-off-by: QuentinBisson <quentin@giantswarm.io>
Add two separate probes, one for liveness and one for readiness. The liveness probe would check that the local individual node is up and running, or else restart the node, while the readiness probe would check that the cluster is ready to serve traffic. This would make etcd health-check fully Kubernetes API complient.
Signed-off-by: Siyuan Zhang <sizhang@google.com>
The proc.Stop just sends the SIGTERM to etcd server. The etcd server
needs time to teardown. If we don't wait for it, the etcd server will
hold the port and then next test case, like TestGrpcProxyAutoSync, will
fail to listen on the port 20000.
```bash
file_/__w/etcd/etcd/tests/fixtures/server.crt_--peer-key-file_/__w/etcd/etcd/tests/fixtures/server.key.insecure_--cert-file_/__w/etcd/etcd/tests/fixtures/server2.crt_--key-file_/__w/etcd/etcd/tests/fixtures/server2.key.insecure_--tls-min-version_TLS1.2_--tls-max-version_TLS1.3) (32856): {"level":"info","ts":"2023-10-13T06:12:32.718845Z","caller":"embed/etcd.go:394","msg":"closing etcd server","name":"e1","data-dir":"/tmp/TestEtcdTLSVersion284993522/001","advertise-peer-urls":["https://127.0.0.1:20000"],"advertise-client-urls":["https://0.0.0.0:0"]}
2023-10-13T06:12:32.7399709Z --- PASS: TestEtcdTLSVersion (1.03s)
2023-10-13T06:12:32.7400181Z === RUN TestGrpcProxyAutoSync
2023-10-13T06:12:32.7401203Z logger.go:130: 2023-10-13T06:12:32.719Z INFO starting server... {"name": "TestGrpcProxyAutoSync-test-0"}
2023-10-13T06:12:32.7408306Z logger.go:130: 2023-10-13T06:12:32.719Z INFO spawning process {"args": ["/__w/etcd/etcd/bin/etcd", "--name=TestGrpcProxyAutoSync-test-0", "--listen-client-urls=http://localhost:20000", "--advertise-client-urls=http://localhost:20000", "--listen-peer-urls=http://localhost:20001", "--initial-advertise-peer-urls=http://localhost:20001", "--initial-cluster-token=new", "--data-dir", "/tmp/TestGrpcProxyAutoSync1139610722/001", "--snapshot-count=10000", "--initial-cluster-token=new", "--initial-cluster=TestGrpcProxyAutoSync-test-0=http://localhost:20001", "--initial-cluster-state=new"], "working-dir": "/__w/etcd/etcd/tests/e2e", "name": "TestGrpcProxyAutoSync-test-0", "environment-variables": ["ETCD_VERIFY=all", "EXPECT_DEBUG=true", "PATH=/__t/go/1.21.3/x64/bin:/go/bin:/__t/go/1.21.3/x64/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "ETCD_UNSUPPORTED_ARCH=amd64"]}
...
2023-10-13T06:12:32.7559855Z /__w/etcd/etcd/bin/etcd (TestGrpcProxyAutoSync-test-0) (32863): {"level":"fatal","ts":"2023-10-13T06:12:32.735191Z","caller":"etcdmain/etcd.go:181","msg":"discovery failed","error":"listen tcp 127.0.0.1:20000: bind: address already in use","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:181\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:267"}
```
Signed-off-by: Wei Fu <fuweid89@gmail.com>
E2e tests has been flaky, with failfast and 2 scenarios tests are twice
as fragile. Any early failure of one of the scenarios cases second to be
canceled. Retryign always require running both scenarios.
Intead let's just wait as in other tests allowing us to retry
just the scenario that failed
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
From the algorithm below, 'ss' should be sorted.
Also removes 'fmt.Println', because the idiomatic tests would not print.
Signed-off-by: Jes Cok <xigua67damn@gmail.com>
Copy the tools/.golangci.yaml and run the linters for which we have
already fixed. The temp .golangci.yaml will be removed when we fixes all
the linters' issues.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Changed TraceKey/StartTimeKey/TokenFieldNameGRPCKey to struct{} to
follow the correct usage of context. Similar patch to #8901.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
1. Manually updated go source file to remove the usage of v1 grpc-gateway;
2. Execute ./scripts/fix.sh
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Signed-off-by: Wei Fu <fuweid89@gmail.com>
The HashByRev-goroutines exit since receive `donec` notification. The
Check-computed-hashes goroutine could not have chance to get the hash
result and be stuck forever. We should add case for donec when we wait
for hash result.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
github.com/gyuho/gocovmerge has already been removed or turned into
a private repo. I believe it's also forked from github.com/wadey/gocovmerge.
So replace it with the original repo github.com/alexfalkowski/gocovmerge.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Use data race detection by default for all amd64 and arm64 tests.
Remove redundant parameters in github workflows.
Signed-off-by: James Blair <mail@jamesblair.net>
From the name of func 'UniqueURLsFromFlag', we can tell that UniqueURLs'uss
should not have duplicates. The current implemention of UniqueURLs'Set
has a bug to make it unique.
Fixes: #16307.
Signed-off-by: Jes Cok <xigua67damn@gmail.com>
If the cluster serves requests slowly, the database has few revision
number and then Compact won't trigger BatchCommit. Add a loop to check
the last revision is big enough to trigger panic.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
This is the latest semconv package used in etcd's dependencies.
Switching to that version reduces the overall package dependencies of
the project (and helps downstream projects which track this,
e.g. Kubernetes).
Signed-off-by: Stephen Kitt <skitt@redhat.com>
The PageWriter has cache buffer so that it doesn't call the Writer until
the cache is almost full. Since the data's length is random, the pending
bytes should be always less than cache buffer size, instead of page
size.
Fix: #16255
Signed-off-by: Wei Fu <fuweid89@gmail.com>
ExpectProcess's Stop method uses 'strings.Contains' to check
the returned err, however, this can be avoided. os.ErrProcessDone's
error message is the same as the hardcoded string. So I think
this explicit error is what this method wants to compare.
Signed-off-by: Jes Cok <xigua67damn@gmail.com>
Uses new grafonnet lib to declare dashboard. Generated dashboard has same layout, but now has timeseries panels instead of deprecated graphs
Signed-off-by: Vitaly Zhuravlev <v-zhuravlev@users.noreply.github.com>
Otherwise common metrics like 'process_resident_memory_bytes' can return non etcd metrics (when cluster label is 'cluster' for example)
Signed-off-by: Vitaly Zhuravlev <v-zhuravlev@users.noreply.github.com>
Let's say there is command which outputs one line and exit with error.
There are three goroutines to acquire the lock:
1. ep.read()
2. ep.waitSaveExitErr()
3. ep.Expect()
When ep.read goroutine reads the log but it doesn't acquire the lock in
time, the ep.waitSaveExitErr acquires the lock and updates the
`exitErr`. And then ep.Expect acquires lock but it doesn't see any log
yet and then returns err.
It's hard to reproduce it in local. Add the extra sleep can reproduce it.
```diff
diff --git a/pkg/expect/expect.go b/pkg/expect/expect.go
index a512a3ce4..602bea73f 100644
--- a/pkg/expect/expect.go
+++ b/pkg/expect/expect.go
@@ -128,6 +128,7 @@ func (ep *ExpectProcess) tryReadNextLine(r *bufio.Reader) error {
printDebugLines := os.Getenv("EXPECT_DEBUG") != ""
l, err := r.ReadString('\n')
+ time.Sleep(10 * time.Millisecond)
ep.mu.Lock()
defer ep.mu.Unlock()
```
See it once in Github Action [1]. In order to fix it, the patch introduces
`readCloseCh` to wait for ep.read to get all the data and retry it.
[1]: https://github.com/etcd-io/etcd/pull/16137#issuecomment-1605838518
Signed-off-by: Wei Fu <fuweid89@gmail.com>
The golang-jwt library supports this already, so supporting it is just a
matter of wiring things up.
Signed-off-by: Tom Wieczorek <twieczorek@mirantis.com>
fix the unexpected blocking when using Barrier.Wait(), e.g.
NewBarrier(client, "a").Wait() will block if key "a" is not existed but "a0" is existed, but it should return immediately.
Signed-off-by: zhangwenkang <zwenkang@vmware.com>
1. rename confChangeCh to raftAdvancedC
2. rename waitApply to confChanged
3. add comments and test assertion
Signed-off-by: Chao Chen <chaochn@amazon.com>
For the pkg/expect package, if the process has been stopped but there is
no `Close()` call, the `ExitCode()` won't return exit code correctly.
The `ExitCode()` should check `exitErr` and return exit code if cmd isn't nil.
And introduces `exitCode` to return correct exit code based on the
process is signaled or exited.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Having too many delete requests is bad as they are not unique requests, so
linearization is more prone to timeout on them.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
For now we just validate stale read revision, but not response content.
Reason is that etcd model only stores latest version of keys, and no
history like real etcd.
Validating stale read contents needs to be done outside of model
as storing whole history is just to costly for linearization validation.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
It is just to complicated to support starting from non-empty etcd.
Existing implementation was very naive to assume that we can build
full state from just one request. We might consider implementing
validation of non-empty history in future, but for now settting
this limit should clean up the code and speed up development.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
This contains a slight refactoring to expose enough information
to write meaningful tests for auth applier v3.
Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
check ScheduledCompactKeyName and FinishedCompactKeyName
before writing hash to hashstore. If they do not match, then it means this compaction has once been interrupted and its hash value is invalid. In such cases, we won't write the hash values to the hashstore, and avoids the incorrect corruption alarm.
Signed-off-by: caojiamingalan <alan.c.19971111@gmail.com>
Mitigates #15993 by not checking each key individually for permission
when auth is entirely disabled or admin user is calling the method.
Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
For the sub test TestKVDelete/ClientAutoTLS, the timeout might be caused
by IO or task scheduled. It's really hard to reproduce this case. And
we don't add `VERBOSE=1` for e2e test, the pipeline doesn't show the
`EXPECT_DEBUG` log in time. It's also hard to debug when flaky case
show. So, this patch will enable `VERBOSE=1` to get timestamp in time.
And the one operation `del a c -w json` will be performed after two seconds
delay, we can increase the timeout from 15s to 30s just in case that the
put or del operations take more time in unstable env.
```
Warning: Keys between "a" and "c" will be deleted. Please interrupt the
command within next 2 seconds to cancel.
```
Signed-off-by: Wei Fu <fuweid89@gmail.com>
If start grpc proxy with --resolver-prefix, memberlist will return all alive proxy nodes, when one grpc proxy node is down, it is expected to not return the down node, but it is still return
Signed-off-by: yellowzf <zzhf3311@163.com>
* Use mod revision for optimistic concurrency.
* Introduce range requests as more general then get
* Add kubernetes specific traffic generation, for now using pull, but
expected to evolve to use watch.
* Introduce kubernetes specific test scenario
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
Increase request to 1000 to increase sample size/reduce variability and increase tolerance threshold from 10 to 15%.
Signed-off-by: James Blair <mail@jamesblair.net>
In a [scheduled test][1], the error shows
```
2023-04-19T11:16:15.8166316Z traffic.go:96: rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
```
According to [grpc-keepalive@v1.51.0][2], each frame from server will
fresh the `lastRead` and it won't file `Ping` frame to server. But the
client used by [`tombstone` request][3] might hit the race. Since we use
5ms as timeout, the client might not receive the result of `Ping` from
server in time. The keepalive will mark it timeout and close the
connection.
I didn't reproduce it in my local. If we add the sleep before update
`lastRead`, it can reproduce it sometimes. Still investigating this
part.
```diff
diff --git a/internal/transport/http2_client.go b/internal/transport/http2_client.go
index d518b07e..bee9c00a 100644
--- a/internal/transport/http2_client.go
+++ b/internal/transport/http2_client.go
@@ -1560,6 +1560,7 @@ func (t *http2Client) reader(errCh chan<- error) {
t.controlBuf.throttle()
frame, err := t.framer.fr.ReadFrame()
if t.keepaliveEnabled {
+ time.Sleep(2 * time.Millisecond)
atomic.StoreInt64(&t.lastRead, time.Now().UnixNano())
}
if err != nil {
```
`DialKeepAliveTime` is always >= [10s][4]. I think we should increase
the timeout to avoid flaky caused by unstable env.
And in a [scheduled test][5], the error shows
```
logger.go:130: 2023-04-22T10:45:52.646Z INFO Failed to trigger failpoint {"failpoint": "blackhole", "error": "context deadline exceeded"}
```
Before sending `Status` to member, the client doesn't [pick][6] the
connection in time (100ms) and returns the error.
The `waitTillSnapshot` is used to ensure that it is good enough to
trigger snapshot transfer. And we have 1min timeout for
injectFailpoints, so I think we can remove the 100ms timeout to reduce
unnecessary stop.
```
injectFailpoints(1min timeout)
failpoint.Inject
triggerBlockhole.Trigger
blackhole
waitTillSnapshot
```
> NOTE: I didn't reproduce it either. :(
Reference:
[1]: <https://github.com/etcd-io/etcd/actions/runs/4741737098/jobs/8419176899>
[2]: <eeb9afa1f6/internal/transport/http2_client.go (L1647)>
[3]: <7450cd886d/tests/robustness/traffic.go (L94)>
[4]: <eeb9afa1f6/dialoptions.go (L445)>
[5]: <https://github.com/etcd-io/etcd/actions/runs/4772033408/jobs/8484334015>
[6]: <eeb9afa1f6/clientconn.go (L932)>
REF: #15763
Signed-off-by: Wei Fu <fuweid89@gmail.com>
The schwag was introduced to generate swagger with authorization support
[1][1] in 2017. And in 2018, the grpc-gateway supports to render
security fields by protoc-gen-swagger [2][2]. After several years, I
think it's good to use upstream protoc supports.
NOTE:
The json's key in `rpc.swagger.json` has been reordered so that it seems
that there's a lot of changes. How to verify it:
```bash
$ # use jq -S to sort the key
$ latest_commit="https://raw.githubusercontent.com/etcd-io/etcd/228f493c7697ce3e9d3a1d831bcffad175846c75/Documentation/dev-guide/apispec/swagger/rpc.swagger.json"
$ curl -s "${latest_commit}" | jq -S . > /tmp/old.json
$ cat Documentation/dev-guide/apispec/swagger/rpc.swagger.json | jq -S . > /tmp/new.json
$ diff --color -u /tmp/old.json /tmp/new.json
```
```diff
--- /tmp/old.json 2023-04-26 10:58:07.142311861 +0800
+++ /tmp/new.json 2023-04-26 10:58:12.170299194 +0800
@@ -1523,11 +1523,14 @@
"type": "object"
},
"protobufAny": {
+ "description": "`Any` contains an arbitrary serialized protocol buffer message along with a\nURL that describes the type of the serialized message.\n\nProtobuf library provides support to pack/unpack Any values in the form\nof utility functions or additional generated methods of the Any type.\n\nExample 1: Pack and unpack a message in C++.\n\n Foo foo = ...;\n Any any;\n any.PackFrom(foo);\n ...\n if (any.UnpackTo(&foo)) {\n ...\n }\n\nExample 2: Pack and unpack a message in Java.\n\n Foo foo = ...;\n Any any = Any.pack(foo);\n ...\n if (any.is(Foo.class)) {\n foo = any.unpack(Foo.class);\n }\n\n Example 3: Pack and unpack a message in Python.\n\n foo = Foo(...)\n any = Any()\n any.Pack(foo)\n ...\n if any.Is(Foo.DESCRIPTOR):\n any.Unpack(foo)\n ...\n\n Example 4: Pack and unpack a message in Go\n\n foo := &pb.Foo{...}\n any, err := ptypes.MarshalAny(foo)\n ...\n foo := &pb.Foo{}\n if err := ptypes.UnmarshalAny(any, foo); err != nil {\n ...\n }\n\nThe pack methods provided by protobuf library will by default use\n'type.googleapis.com/full.type.name' as the type URL and the unpack\nmethods only use the fully qualified type name after the last '/'\nin the type URL, for example \"foo.bar.com/x/y.z\" will yield type\nname \"y.z\".\n\n\nJSON\n====\nThe JSON representation of an `Any` value uses the regular\nrepresentation of the deserialized, embedded message, with an\nadditional field `@type` which contains the type URL. Example:\n\n package google.profile;\n message Person {\n string first_name = 1;\n string last_name = 2;\n }\n\n {\n \"@type\": \"type.googleapis.com/google.profile.Person\",\n \"firstName\": <string>,\n \"lastName\": <string>\n }\n\nIf the embedded message type is well-known and has a custom JSON\nrepresentation, that representation will be embedded adding a field\n`value` which holds the custom JSON in addition to the `@type`\nfield. Example (for message [google.protobuf.Duration][]):\n\n {\n \"@type\": \"type.googleapis.com/google.protobuf.Duration\",\n \"value\": \"1.212s\"\n }",
"properties": {
"type_url": {
+ "description": "A URL/resource name that uniquely identifies the type of the serialized\nprotocol buffer message. This string must contain at least\none \"/\" character. The last segment of the URL's path must represent\nthe fully qualified name of the type (as in\n`path/google.protobuf.Duration`). The name should be in a canonical form\n(e.g., leading \".\" is not accepted).\n\nIn practice, teams usually precompile into the binary all types that they\nexpect it to use in the context of Any. However, for URLs which use the\nscheme `http`, `https`, or no scheme, one can optionally set up a type\nserver that maps type URLs to message definitions as follows:\n\n* If no scheme is provided, `https` is assumed.\n* An HTTP GET on the URL must yield a [google.protobuf.Type][]\n value in binary format, or produce an error.\n* Applications are allowed to cache lookup results based on the\n URL, or have them precompiled into a binary to avoid any\n lookup. Therefore, binary compatibility needs to be preserved\n on changes to types. (Use versioned type names to manage\n breaking changes.)\n\nNote: this functionality is not currently available in the official\nprotobuf release, and it is not used for type URLs beginning with\ntype.googleapis.com.\n\nSchemes other than `http`, `https` (or the empty scheme) might be\nused with implementation specific semantics.",
"type": "string"
},
"value": {
+ "description": "Must be a valid serialized protocol buffer of the above specified type.",
"format": "byte",
"type": "string"
}
```
REF:
1: <https://github.com/etcd-io/etcd/pull/7999#issuecomment-307512043>
2: <https://github.com/grpc-ecosystem/grpc-gateway/pull/547>
Signed-off-by: Wei Fu <fuweid89@gmail.com>
It's to deflake TestAuthMemberRemove.
When the client has multiple endpoints, the client might send a request
with valid token to the follower member which hasn't received token
replicated log yet. The member will reject the request.
For instance, the maintenance.Status API will return "auth: invalid auth
token". But the client doesn't identify the error. The client won't retry to
refresh auth token. The maintenance.Status should togRPCError before return
so that the client can reflesh token. It's align with existing API.
Since the maintenance client always creates one connection to target
member, the member will have the token after refresh auth.
Maybe we can introduce a sync to wait for member is ready with token,
instead of refreshing.
Fixes: #15758
Signed-off-by: Wei Fu <fuweid89@gmail.com>
It's followup of #15667.
This patch is to use zaptest/observer as base to provide a similar
function to pkg/expect.Expect.
The test env
```bash
11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
mkdir /sys/fs/cgroup/etcd-followup-15667
echo 0-2 | tee /sys/fs/cgroup/etcd-followup-15667/cpuset.cpus # three cores
```
Before change:
* memory.peak: ~ 681 MiB
* Elapsed (wall clock) time (h:mm:ss or m:ss): 6:14.04
After change:
* memory.peak: ~ 671 MiB
* Elapsed (wall clock) time (h:mm:ss or m:ss): 6:13.07
Based on the test result, I think it's safe to be enabled by default.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
The TestV3WatchRestoreSnapshotUnsync setups three members' cluster.
Before serving any update requests from client, after leader elected,
each member will have index 8 log: 3 x ConfChange +
3 x ClusterMemberAttrSet + 1 x ClusterVersionSet.
Based on the config (SnapshotCount: 10, CatchUpCount: 5), we need to
file update requests to trigger snapshot at least twice.
T1: L(snapshot-index: 11, compacted-index: 6) F_m0(index: 8)
T2: L(snapshot-index: 22, compacted-index: 17) F_m0(index: 8, out of date)
After member0 recovers from network partition, it will reject leader's
request and return hint (index:8, term:x). If it happens after
second snapshot, leader will find out the index:8 is out of date and
force to transfer snapshot.
However, the client only files 15 update requests and leader doesn't
finish the process of snapshot in time. Since the last of
compacted-index is 6, leader can still replicate index:9 to member0
instead of snapshot.
```bash
cd tests/integration
CLUSTER_DEBUG=true go test -v -count=1 -run TestV3WatchRestoreSnapshotUnsync ./
...
INFO m2.raft 3da8ba707f1a21a4 became leader at term 2 {"member": "m2"}
...
INFO m2 triggering snapshot {"member": "m2", "local-member-id": "3da8ba707f1a21a4", "local-member-applied-index": 22, "local-member-snapshot-index": 11, "local-member-snapshot-count": 10, "snapshot-forced": false}
...
cluster.go:1359: network partition between: 99626fe5001fde8b <-> 1c964119da6db036
cluster.go:1359: network partition between: 99626fe5001fde8b <-> 3da8ba707f1a21a4
cluster.go:416: WaitMembersForLeader
INFO m0.raft 99626fe5001fde8b became follower at term 2 {"member": "m0"}
INFO m0.raft raft.node: 99626fe5001fde8b elected leader 3da8ba707f1a21a4 at term 2 {"member": "m0"}
DEBUG m2.raft 3da8ba707f1a21a4 received MsgAppResp(rejected, hint: (index 8, term 2)) from 99626fe5001fde8b for index 23 {"member": "m2"}
DEBUG m2.raft 3da8ba707f1a21a4 decreased progress of 99626fe5001fde8b to [StateReplicate match=8 next=9 inflight=15] {"member": "m2"}
DEBUG m0 Applying entries {"member": "m0", "num-entries": 15}
DEBUG m0 Applying entry {"member": "m0", "index": 9, "term": 2, "type": "EntryNormal"}
....
INFO m2 saved snapshot {"member": "m2", "snapshot-index": 22}
INFO m2 compacted Raft logs {"member": "m2", "compact-index": 17}
```
To fix this issue, the patch uses log monitor to watch "compacted Raft
log" and expect that two members should compact log twice.
Fixes: #15545
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Got a confirmation from Sam that he is retiring his contributions.
Removing Tobias as raft was moved to separate repo.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
Progress notifications requested using ProgressRequest were sent
directly using the ctrlStream, which means that they could race
against watch responses in the watchStream.
This would especially happen when the stream was not synced - e.g. if
you requested a progress notification on a freshly created unsynced
watcher, the notification would typically arrive indicating a revision
for which not all watch responses had been sent.
This changes the behaviour so that v3rpc always goes through the watch
stream, using a new RequestProgressAll function that closely matches
the behaviour of the v3rpc code - i.e.
1. Generate a message with WatchId -1, indicating the revision for
*all* watchers in the stream
2. Guarantee that a response is (eventually) sent
The latter might require us to defer the response until all watchers
are synced, which is likely as it should be. Note that we do *not*
guarantee that the number of progress notifications matches the number
of requests, only that eventually at least one gets sent.
Signed-off-by: Peter Wortmann <peter.wortmann@skao.int>
This issue is somewhat easily reproduced simply by bombarding the
server with requests for progress notifications, which eventually
leads to one being delivered ahead of the payload message. This is
then caught by the watch response validation code previously added by
Marek Siarkowicz.
Signed-off-by: Peter Wortmann <peter.wortmann@skao.int>
This will fail basically every time, as the progress notification
request catches the watcher in an asynchronised state.
Signed-off-by: Peter Wortmann <peter.wortmann@skao.int>
Difference in load configuration for watch delay tests show how huge the
impact is. Even with random write scheduler grpc under http
server can only handle 500 KB with 2 seconds delay. On the other hand,
separate grpc server easily hits 10, 100 or even 1000 MB within 100 miliseconds.
Priority write scheduler that was used in most previous releases
is far worse than random one.
Tests configured to only 5 MB to avoid flakes and taking too long to fill
etcd.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
Add item [server/embed: fix data race when starting both secure & insecure
gRPC servers on the same address] to 3.4/3.5 changelog.
REF: #15509
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Problem: during restore in watchableStore.Restore, synced watchers are moved to unsynced.
minRev will be behind since it's not updated when watcher stays synced.
Solution: update minRev
fixes: https://github.com/etcd-io/etcd/issues/15271
Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
There are two goroutines accessing the `gs` grpc server var. Before
insecure `gs` server start, the `gs` can be changed to secure server and
then the client will fail to connect to etcd with insecure request. It
is data-race. We should use argument for reference in the new goroutine.
fix: #15495
Signed-off-by: Wei Fu <fuweid89@gmail.com>
The original testcase uses `return` statement which skips `restore`
case. It's aimed to enable `restore` testcase.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
The cluster version will be initialized after the member becomes leader.
The update is handled asynchronously. It couldn't be updated if the member
has been closed and the go-runtime picks the `s.stopping` channel first.
```go
// e2a5df534c/server/etcdserver/server.go (L2170)
func (s *EtcdServer) monitorClusterVersions() {
...
for {
select {
case <-s.firstCommitInTerm.Receive():
case <-time.After(monitorVersionInterval):
case <-s.stopping:
return
}
...
}
}
```
Or after the `s.stopping` has been closed, the [UpdateClusterVersion][1] won't
file GoAttach successfully. For the #15409, we can see the warn log
`server has stopped; skipping GoAttach` from GoAttach:
```plain
https://github.com/etcd-io/etcd/actions/runs/4340931587/jobs/7580103902
logger.go:130: 2023-03-06T07:36:44.253Z WARN default stopping grpc server due to error {"error": "accept tcp 127.0.0.1:2379: use of closed network connection"}
logger.go:130: 2023-03-06T07:36:44.253Z WARN default stopped grpc server due to error {"error": "accept tcp 127.0.0.1:2379: use of closed network connection"}
logger.go:130: 2023-03-06T07:36:44.253Z ERROR default setting up serving from embedded etcd failed. {"error": "accept tcp 127.0.0.1:2379: use of closed network connection"}
logger.go:130: 2023-03-06T07:36:44.253Z ERROR default setting up serving from embedded etcd failed. {"error": "http: Server closed"}
logger.go:130: 2023-03-06T07:36:44.253Z INFO default skipped leadership transfer for single voting member cluster {"local-member-id": "8e9e05c52164694d", "current-leader-member-id": "8e9e05c52164694d"}
logger.go:130: 2023-03-06T07:36:44.253Z WARN default server has stopped; skipping GoAttach
...
```
If the cluster version isn't updated, the minimum storage version will
be v3.5 because the [AuthStatus][2] is introduced in [v3.5][3].
The compare will fail.
To fix this issue, we should wait for cluster version to become ready
after server is ready to serve request.
[1]: <e2a5df534c/server/etcdserver/adapters.go (L45)>
[2]: <071e70cdc4>
[3]: <1b4e54c238>
Signed-off-by: Wei Fu <fuweid89@gmail.com>
The TestLeasingDeleteRangeContendTxn is trying to test for RangeDelete when
the target resources are being updated. When the `txnLeasing` wants a
server-side transaction, it needs to ensure all the keys mod revision should
be leass than what it saw. If the compare fails, it will repeat to apply the
server-side transaction until it is sucessful. I believe the test-case is
trying to verify how the `txnLeasing` handles the race issue.
Before the patch #15401, the resource-updating goroutine keeps updating until
the RangeDelete finishes. The testcase is flaky because two goroutines are
sharing one `ctx` and grpc-go client won't wait for the response if `ctx`
has been canceled.
For example,
| DelLease Goroutine | PutLease Goroutine | ETCD Server | Key/0 Status |
| -- | --- | -- | -- |
| deleted | | | version = 0 |
| | send update(key/0=123) req | received update(key/0=123) req | version = 0 |
| cancel | | | version = 0 |
| | exit because of cancel | | version = 0 |
| get key/0 by putkv | | | version = 0 |
| | | applied update(key/0=123) | version = 1 |
| get key/0 by raw-cli | | | version = 1 |
So `raw-cli` gets `[key/0=123]` while the `putkv` gets `[]`. If `putkv`
applies two update reqs to ETCD server and the last one is canceled
before apply, the error will be like:
```
expected [key:"key/0" version:2 value:"123" ], got [key:"key/0" version:1 value:"123" ]
```
The resource-updating goroutine should not share the ctx with RangeDelete here.
And I also revert current main branch because the resource-update goroutine
only updates 8 times and might exit before `RangeDelete`. In this case,
the `txnLeasing` is not handling the race issue.
Fixes: #15352
Signed-off-by: Wei Fu <fuweid89@gmail.com>
The huge (100k+) value was justified when storev2 was being dumped completely with every snapshot.
With storev2 being decomissioned we can checkpoint more frequently for faster recovery.
Signed-off-by: James Blair <mail@jamesblair.net>
Fixes etcd-io#15352.
Depending on the goroutine scheduling, the expected count of 8 might not
have been reached yet. This ensures the routine won't stop earlier than
that.
Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
When the devcontainer starts it will run make build to ensure compiled binaries are immediately available for use.
I've confirmed make test is also behaving as expected within the devcontainer.
Signed-off-by: James Blair <mail@jamesblair.net>
status.Error() returns an error representing error code and msg,
status.Error() is better than status.New().Err().
Signed-off-by: xakdwch <xakdwch5@gmail.com>
Problem: time.Now() uses wall clock reading. See https://github.com/golang/go/blob/master/src/time/time.go#L17
"later time-telling operations use the wall clock reading, but later time-measuring operations, specifically comparisons and subtractions, use the monotonic clock reading."
This can cause 'Return' to be before 'Call' and wrong order of operations from different clients.
Solution: use same base time for all clients and only use 'time-measuring' operations to record timestamps for history.
Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
raftBeforeFollowerSend can only be triggered on Follower and won't work on 1 node cluster.
Had to split RandomFailpoint into RandomOneNodeClusterFailpoint and RandomMultiNodeClusterFailpoint
Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
When the linearizability test cases run with three members, it might take
7-8s to stop three members, especially stopping the leader. The leader
will transfer the leadership and it might take more time to stop peer
listener.
In order to reduce the runtime, this commit is using signal kill to force
stop members instead of graceful shutdown.
REF: #15086
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Historic capnslog timestamps are in microsecond resolution. We need to match that when we migrate to the zap logger.
Signed-off-by: James Blair <mail@jamesblair.net>
Added optional TLS min/max protocol version and command line switches to set
versions for the etcd server.
If max version is not explicitly set by the user, let Go select the max
version which is currently TLSv1.3. Previously max version was set to TLSv1.2.
Signed-off-by: Tero Saarni <tero.saarni@est.tech>
The old name(raftDone) of the channel(notifyc) which indicates the apply has been
completed is left unchanged in the comments, resulting in confusion when reading
the source code.
Signed-off-by: caojiamingalan <alan.c.19971111@gmail.com>
After setting the ComparionSleepInterval, we can use time.Ticker
instead of time.After to optimize the scheduleComparison(),
otherwise it will fail in the 'TestStoreCompact(t)' test.
Signed-off-by: guozhao <guozhao@360.cn>
When promoting a learner, we need to wait until the leader's applied ID
catches up to the commitId. Afterwards, check whether the learner ID
exist or not, and return `membership.ErrIDNotFound` directly in the API
if the member ID not found, to avoid the request being unnecessarily
delivered to raft.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
This CL refactors the tests to reuse a single workflow that has
parameters. This is then reused for PRs/pushes and the nightlies.
Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
We need to return io.ErrUnexpectedEOF in the error chain, so that
etcdserver can repair it automatically.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
The tests are subtle as they skip if the binary is not generated in the local directory.
$ go build . && go test
PASS
ok go.etcd.io/etcd/v3/tools/etcd-dump-logs 0.769s
Signed-off-by: Piotr Tabor <ptab@google.com>
We should call Wait for grpc-proxy process stop before start. Otherwise,
the tcp port won't be released.
Fixes: #14926
Signed-off-by: Wei Fu <fuweid89@gmail.com>
target.Endpoint and some other fields are deprecated, URL field is
suggested to use instead
path is required to be stripped of "/" prefix for naming/resolver to
work porperly
Signed-off-by: Ramil Mirhasanov <ramil600@yahoo.com>
This PR will add `trivy-nightly-scan` for etcd images with versions `3.4.22` and `3.5.6` to scan for vulnerabilities everyday at 2AM UTC.
Signed-off-by: ArkaSaha30 <arkasaha30@gmail.com>
Currently when triggering release, it always pull remote repo and
checkout main branch. Any changes which are merged into the target
release branch (e.g. release-3.5) will be ignored. It isn't
convenient for test, including in github workflow and local environment.
So we need to support triggering release in current branch.
Note: --current-branch should only be called with DRY_RUN=true
Signed-off-by: Benjamin Wang <wachao@vmware.com>
go install golang.org/x/vuln/cmd/govulncheck@latest && govulncheck ./...
shell: /usr/bin/bash -e {0}
env:
GOROOT: /opt/hostedtoolcache/go/1.19.4/x64
go: golang.org/x/vuln/cmd/govulncheck@latest: no matching versions for query "latest"
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Found 1 known vulnerability.
Vulnerability #1: GO-2022-1144
An attacker can cause excessive memory growth in a Go server
accepting HTTP/2 requests. HTTP/2 server connections contain a
cache of HTTP header keys sent by the client. While the total
number of entries in this cache is capped, an attacker sending
very large keys can cause the server to allocate approximately
64 MiB per open connection.
Call stacks in your code:
Error: tools/etcd-dump-metrics/main.go:158:5: go.etcd.io/etcd/v3/tools/etcd-dump-metrics.main calls go.etcd.io/etcd/server/v3/embed.StartEtcd, which eventually calls golang.org/x/net/http2.Server.ServeConn
Found in: golang.org/x/net/http2@v0.2.0
Fixed in: golang.org/x/net/http2@v0.4.0
More info: https://pkg.go.dev/vuln/GO-2022-1144
Error: Process completed with exit code 3.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
$ govulncheck ./...
govulncheck is an experimental tool. Share feedback at https://go.dev/s/govulncheck-feedback.
Scanning for dependencies with known vulnerabilities...
Found 1 known vulnerability.
Vulnerability #1: GO-2022-1144
An attacker can cause excessive memory growth in a Go server
accepting HTTP/2 requests. HTTP/2 server connections contain a
cache of HTTP header keys sent by the client. While the total
number of entries in this cache is capped, an attacker sending
very large keys can cause the server to allocate approximately
64 MiB per open connection.
Call stacks in your code:
tools/etcd-dump-metrics/main.go:159:31: go.etcd.io/etcd/v3/tools/etcd-dump-metrics.main$4 calls go.etcd.io/etcd/server/v3/embed.StartEtcd, which eventually calls golang.org/x/net/http2.ConfigureServer$1
Found in: golang.org/x/net/http2@v0.2.0
Fixed in: golang.org/x/net/http2@v1.19.4
More info: https://pkg.go.dev/vuln/GO-2022-1144
Vulnerability #2: GO-2022-1144
An attacker can cause excessive memory growth in a Go server
accepting HTTP/2 requests. HTTP/2 server connections contain a
cache of HTTP header keys sent by the client. While the total
number of entries in this cache is capped, an attacker sending
very large keys can cause the server to allocate approximately
64 MiB per open connection.
Call stacks in your code:
contrib/lock/storage/storage.go:106:28: go.etcd.io/etcd/v3/contrib/lock/storage.main calls net/http.ListenAndServe
contrib/raftexample/httpapi.go:113:31: go.etcd.io/etcd/v3/contrib/raftexample.serveHTTPKVAPI$1 calls net/http.Server.ListenAndServe
tools/etcd-dump-metrics/main.go:159:31: go.etcd.io/etcd/v3/tools/etcd-dump-metrics.main$4 calls go.etcd.io/etcd/server/v3/embed.StartEtcd, which eventually calls net/http.Serve
tools/etcd-dump-metrics/main.go:159:31: go.etcd.io/etcd/v3/tools/etcd-dump-metrics.main$4 calls go.etcd.io/etcd/server/v3/embed.StartEtcd, which eventually calls net/http.Server.Serve
Found in: net/http@go1.19.3
Fixed in: net/http@go1.19.4
More info: https://pkg.go.dev/vuln/GO-2022-1144
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Many of the existing PSC members are now emeritus maintainers.
Considering the current state of the project, we may want to keep
PSC with the current maintainers. We will list volunteers as we
add them.
Signed-off-by: Sahdev Zala <spzala@us.ibm.com>
Start with a simple job against main that runs for 3h by repeating it
an order of magnitude more often than the PR job.
Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
The pb provides an accessor method to get field and it will not panic if
the owner is nil. And add non-empty RangeRespone into the test case.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
This shortens operation history and avoids having to many failed requests.
Failed requests are problematic as too many of them can cause linearizability
verification complexity to become exponential.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
Previously etcdservers depends on raft/raftpb/raft.proto directly.
After moving raft to a separate repo, we need to add raft to the
tools/mod, and get raft included in the -I protc flags.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Just execute ./script/fix.sh after updating raftexample to use
the new raft module go.etcd.io/raft/v3.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Just replaced all go.etcd.io/etcd/raft/v3 with go.etcd.io/raft/v3
under directory contrib/raftexample.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
TODO:
1. Update Documentation/contributor-guide/modules.svg;
2. Update bill-of-materials.json when raft and raftexample are removed in future;
Signed-off-by: Benjamin Wang <wachao@vmware.com>
No need to generate proto file;
No need to test coverage for raft;
No need to run any test for raft module;
NO need to run any test for raftexample;
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Executed commands below,
1. Removed go.etcd.io/raft/v3 => ../raft;
2. go get go.etcd.io/raft/v3@eaa6808e1f7ab2247c13778250f70520b0527ff1;
3. go mod tidy
Note that after execuing command `go mod tidy`, the dependency
"go.etcd.io/etcd/raft/v3 v3.5.6" was added automatically. When we
remove raft and the raftexample, and it will cleanup automatically
when executing `go mod tidy` again.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Executed commands below,
1. Removed go.etcd.io/raft/v3 => ../raft;
2. go get go.etcd.io/raft/v3@eaa6808e1f7ab2247c13778250f70520b0527ff1;
3. go mod tidy
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Add go.mod and go.sum to fix issue below,
```
$ go build
../../server/etcdserver/api/snap/snapshotter.go:33:2: missing go.sum entry for module providing package go.etcd.io/raft/v3 (imported by go.etcd.io/etcd/server/v3/etcdserver/api/snap); to add:
go get go.etcd.io/etcd/server/v3/etcdserver/api/snap@v3.6.0-alpha.0
../../server/storage/wal/walpb/record.pb.go:14:2: missing go.sum entry for module providing package go.etcd.io/raft/v3/raftpb (imported by go.etcd.io/etcd/v3/tools/etcd-dump-logs); to add:
go get go.etcd.io/etcd/v3/tools/etcd-dump-logs
```
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Executed commands below,
1. Removed go.etcd.io/raft/v3 => ../raft;
2. go get go.etcd.io/raft/v3@eaa6808e1f7ab2247c13778250f70520b0527ff1
3. go mod tidy
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Executed commands below,
1. Removed go.etcd.io/raft/v3 => ../raft;
2. go get go.etcd.io/raft/v3@eaa6808e1f7ab2247c13778250f70520b0527ff1
3. go mod tidy
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Build flags read from env GO_BUILD_FLAGS are prepended to
flags in Makefile and sent to scripts/build.sh for building
executables (etcd, etcdctl & etcdutl).
Signed-off-by: Bhargav Ravuri <bhargav.ravuri@infracloud.io>
When two members in a 5 member cluster are corrupted, and they
have different hashes, etcd will raise alarm for both members,
but the order isn't guaranteed. But if the two corrupted members
have the same hash, then the order is guaranteed. The leader
always raise alarm in the same order as the member list.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
There is data race on `stop` channel. After verify write-timeout successfully,
the case won't wait for `blocker` to receive close signal from `stop` channel.
If the new `blocker`, which is to read-timeout verifier, get dial's result
immediately, the new `blocker` might fetch the message from `stop` channel
before old one and then close the connection, which causes that the
`conn.Read` returns `EOF` when it reads data.
How to reproduce this in linux devbox?
Use `taskset` to limit the test process in one-cpu.
```bash
cd ./client/pkg/transport
go test -c -o /tmp/test --race=true ./
taskset -c 0 /tmp/test -test.run TestWriteReadTimeoutListener -test.v -test.cpu 4 -test.count=10000 -test.failfast
```
To fix this, suggest to use seperate `stop` channel to prevent from data
race.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
If quorum doesn't exist, we don't know which members data are
corrupted. In such situation, we intentionally set the memberID
as 0, it means it affects the whole cluster.
It's align with what we did for 3.4 and 3.5 in
https://github.com/etcd-io/etcd/issues/14849
Signed-off-by: Benjamin Wang <wachao@vmware.com>
The change did in https://github.com/etcd-io/etcd/pull/14824 fixed
the test instead of the product code. It isn't correct. After we
fixed the product code in this PR, we can revert the change in
that PR.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
When the leader detects data inconsistency by comparing hashes,
currently it assumes that the follower is the corrupted member.
It isn't correct, the leader might be the corrupted member as well.
We should depend on quorum to identify the corrupted member.
For example, for 3 member cluster, if 2 members have the same hash,
the the member with different hash is the corrupted one. For 5 member
cluster, if 3 members have the same same, the corrupted member is one
of the left two members; it's also possible that both the left members
are corrupted.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Two chanages:
1. Add field `HashRevision` into `HashKVResponse`;
2. Display the new field when executing `etcdctl endpoint hash`.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Comments fixed as per goword in go _test package files that
shell function go_srcs_in_module lists as per changes on #14827
Helps in #14827
Signed-off-by: Bhargav Ravuri <bhargav.ravuri@infracloud.io>
The shell func go_srcs_in_module will now list
1. go src files
2. go test files belong to same packages
3. go test files that belong to _test packages
Fixes#14827
Signed-off-by: Bhargav Ravuri <bhargav.ravuri@infracloud.io>
Comments fixed as per goword in go test files that shell
function go_srcs_in_module lists as per changes on #14827
Helps in #14827
Signed-off-by: Bhargav Ravuri <bhargav.ravuri@infracloud.io>
The shell function go_srcs_in_module will list go test files
along with go src files in the specified module. This helps
in identifying the copyright header misses in test files.
Fixes#14827
Signed-off-by: Bhargav Ravuri <bhargav.ravuri@infracloud.io>
If the corrupted member has been elected as leader, the memberID in alert
response won't be the corrupted one. It will be a smaller follower ID since
the raftCluster.Members always sorts by ID. We should check the leader
ID and decide to use which memberID.
Fixes: #14823
Signed-off-by: Wei Fu <fuweid89@gmail.com>
This changes the builds to always add -trimpath which removes specific
build time paths from the binary (like current directories etc).
Improves build reproducability to make the final binary independent from
the specific build path.
Lastly, when stripping debug symbols, also add -w to strip DWARF symbols
as well which aren't needed in that case either.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
1. Fixed the test failures which are caused by recent test framework rafactoring;
2. renamed the file to promote_experimental_flag_test.go.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
--warning-unary-request-duration is a duplicate of --experimental-warning-unary-request-duration
experimental-warning-unary-request-duration will be removed in v3.7.
fixes https://github.com/etcd-io/etcd/issues/13783
Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
The correct param forMaxCallRecvMsgSize is '--max-recv-bytes' instead of '--max-request-bytes', so I fixed the documentation and description.
Signed-off-by: cleverhu <shouping.hu@daocloud.io>
When users use the TLS CommonName based authentication, the
authTokenBundle is always nil. But it's possible for the clients
to get `rpctypes.ErrAuthOldRevision` response when the clients
concurrently modify auth data (e.g, addUser, deleteUser etc.).
In this case, there is no need to refresh the token; instead the
clients just need to retry the operations (e.g. Put, Delete etc).
Signed-off-by: Benjamin Wang <wachao@vmware.com>
CodeQL@v1 is going to be deprecated in three weeks. This PR updates the
branches to track only the currently maintained release branches and
moves the actions to its v2 version.
Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
ExpectProcess and ExpectFunc now take the exit code of the process into
account, not just the matching of the tty output.
This also refactors the many tests that were previously succeeding on
matching an output from a failing cmd execution.
Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
The etcdctl and etcdutl built with `-tags cov` mode will show go-test result
after each execution, like
```
...
PASS
coverage: 0.0% of statements in ./...
```
Since the PASS is not real command, the `source completion` command will
fail with command-not-found error. And there is no easy way to disable
the (*testing.M).Run's output. Therefore, this patch uses build tag !cov
to disable cases when enable coverage.
Fixes: #14694
Signed-off-by: Wei Fu <fuweid89@gmail.com>
This commit introduces the max inflight bytes setting at the Config level, and
tests that raft flow control honours it.
Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
This commit plumbs the max total byte size of the Inflights type higher up the
stack to the ProgressTracker.
Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
The Inflights type has limits on the message size and the number of inflight
messages. However, a single large entry that exceeds the size limit can still
be sent. In combination with the max messages count limit, many large messages
can be sent in a row and overflow the receiver. In effect, the "max" values act
as "target" rather than hard limits.
This commit adds an additional soft limit on the total size of inflight
messages, which catches such situations and prevents the receiver overflow.
Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
Tobias Grieger is actively working on etcd/raft, and has already
been added back as a raft maintainer, so no reason to keep him
in the Emeritus Maintainers list.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
This commit removes the ability to apply log entries at the same time as
applying a snapshot. Doing so it possible, but it leads to complex code and
raises questions about what should be applied first. It also raises additional
complexity when we start allowing concurrent, asynchronous log appends and log
application. It's easiest to just disallow this.
Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Also rename hasNextEnts to hasNextCommittedEnts.
Also rename maxNextEntsSize to maxNextCommittedEntsSize.
Pure refactor.
Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
When e2e test cases specify the DataDirPath and there are more than
one member in the cluster, we need to create a subdirectory for each
member. Otherwise all members share the same directory and accordingly
lead to conflict.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Deprecated: use WithTransportCredentials and insecure.NewCredentials() instead. Will be supported throughout 1.x.
Signed-off-by: Ramil Mirhasanov <ramil600@yahoo.com>
`unsafeCommit` is called by both `(*batchTxBuffered) commit` and
`(*backend) defrag`. When users perform the defragmentation
operation, etcd doesn't update the consistent index. If etcd
crashes(e.g. panicking) in the process for whatever reason, then
etcd replays the WAL entries starting from the latest snapshot,
accordingly it may re-apply entries which might have already been
applied, eventually the revision isn't consistent with other members.
Refer to discussion in https://github.com/etcd-io/etcd/pull/14685
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Use array indexing to clean up the code and make it constant time.
Also, add a test for IsResponseMsg.
Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
This commit makes the rarely used `raftpb.Message.Snapshot` field nullable.
In doing so, it reduces the memory size of a `raftpb.Message` message from
264 bytes to 128 bytes — a 52% reduction in size.
While this commit does not change the protobuf encoding, it does change
how that encoding is used. `(gogoproto.nullable) = false` instruct the
generated proto marshaling logic to always encode a value for the field,
even if that value is empty. `(gogoproto.nullable) = true` instructs the
generated proto marshaling logic to omit an encoded value for the field
if the field is nil.
This raises compatibility concerns in both directions. Messages encoded
by new binary versions without a `Snapshot` field will be decoded as an
empty field by old binary versions. In other words, old binary versions
can't tell the difference. However, messages encoded by old binary versions
with an empty Snapshot field will be decoded as a non-nil, empty field by
new binary versions. As a result, new binary versions need to be prepared
to handle such messages.
While Message.Snapshot is not intentionally part of the external interface
of this library, it was possible for users of the library to access it and
manipulate it. As such, this change may be considered a breaking change.
Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
This fix avoids the assumption of knowing the current version of the
binary. We can query the binary with the version flag to get the actual
version of the given binary we upgrade and downgrade to. The
respectively reported versions should match what is returned by the
version endpoint.
Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
This commit removes the verbose comparisons in tests, in favor of using
assert/require helpers from the testify packages.
Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
Make the field name and comment clearer on the fact that it's used both in
StateProbe and StateReplicate. The old name ProbeSent was slightly confusing,
and also triggered thinking that it's used only in StateProbe.
Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
When Inflights to a particular node is full, i.e. MaxInflightMsgs for the
append messages flow is saturated, it is still necessary to continue sending
MsgApp to ensure progress. Currently this is achieved by "forgetting" the first
in-flight message in the window, which frees up quota for one new MsgApp.
This new message is constructed in such a way that it potentially has multiple
entries, or a large entry. The effect of this is that the in-flight limitations
can be exceeded arbitrarily, for as long as the flow to this node continues
being saturated. In particular, if a follower is stuck, the leader will keep
sending entries to it.
This commit makes the MsgApp empty when Inflights is saturated, and prevents
the described leakage of Entries to slow followers.
Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
Previously, Progress update on MsgApp send was scattered across raft.go and
tracker/progress.go. This commit better encapsulates this logic in the Progress
type.
Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
- avoid large indented blocks, leave the main block unindented
- declare pb.Message inlined in the sending call
Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
This commit adds a data-driven test which simulates conditions under which Raft
messages flow to a particular node is throttled while in StateReplicate. The
test demonstrates that MsgApp messages with non-empty Entries may "leak" to a
paused stream every time there is successful heartbeat exchange.
Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
Check the values of myKey and myRev first in Unlock method to prevent calling Unlock without Lock. Because this may cause the value of pfx to be deleted by mistake.
Signed-off-by: chenyahui <cyhone@qq.com>
Afer moving `ClusterVersion` and related constants into e2e packages,
some e2e test cases are broken, so we need to update them to use the
correct definitions.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
ClusterContext is used by "e2e" or "integration" to extend the
ClusterConfig. The common test cases shouldn't care about what
data is encoded or included; instead "e2e" or "integration"
framework should decode or parse it separately.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
The client may connect to a different endpint as the target to be
maintained. When auth is enabled, the target endpoint might haven't
finished applying the authentiation request, so it might reject the
corresponding maintenance request due to permission denied.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
The existing client may connect to different endpoint from the
specific endpoint to be maintained. Maintenance operations do not
go through raft at all, so it might run into issue if the server
hasn't finished applying the authentication request.
Let's work with an example. Assuming the existing client connects to
ep1, while the user wants to maintain ep2. If we getToken again, it
sends an authentication request, which goes through raft. When the
specific endpoint receives the maintenance request, it might haven't
finished previous authentication request, but the new token is already
carried in the context, so it will reject the maintenance request
due to invalid token.
We already have retry logic in `unaryClientInterceptor` and
`streamClientInterceptor`. When the token expires, it can automatically
refresh the token, so it should be safe to remove the `getToken`
logic in `maintenance.dial`
Signed-off-by: Benjamin Wang <wachao@vmware.com>
All maintenance APIs require admin privilege when auth is enabled,
otherwise, the request will be rejected. If auth isn't enabled,
then no such requirement any more.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
In the TestDowngradeUpgradeCluster case, the brand-new cluster is using
simple-config-changer, which means that entries has been committed
before leader election and these entries will be applied when etcdserver
starts to receive apply-requests. The simple-config-changer will mark
the `confState` dirty and the storage backend precommit hook will update
the `confState`.
For the new cluster, the storage version is nil at the beginning. And
it will be v3.5 if the `confState` record has been committed. And it
will be >v3.5 if the `storageVersion` record has been committed.
When the new cluster is ready, the leader will set init cluster version
with v3.6.x. And then it will trigger the `monitorStorageVersion` to
update the `storageVersion` to v3.6.x. If the `confState` record has
been updated before cluster version update, we will get storageVersion
record.
If the storage backend doesn't commit in time, the
`monitorStorageVersion` won't update the version because of `cannot
detect storage schema version: missing confstate information`.
And then we file the downgrade request before next round of
`monitorStorageVersion`(per 4 second), the cluster version will be
v3.5.0 which is equal to the `UnsafeDetectSchemaVersion`'s result.
And we won't see that `The server is ready to downgrade`.
It is easy to reproduce the issue if you use cpuset or taskset to limit
in two cpus.
So, we should wait for the new cluster's storage ready before downgrade
request.
Fixes: #14540
Signed-off-by: Wei Fu <fuweid89@gmail.com>
We defines two common `WithAuth` functions for e2e and integration
test respectively. They are calling `integration.WithAuth` and
`e2e.WithAuth` respectively.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
It doesn't make sense to always pass a AuthConfig parameter for
test cases which do not enable auth at all. So refactoring the
Client interface method so that it accepts a `ClientOption`
variadic parameter.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Check the client count before creating the ephemeral key, do not
create the key if there are already too many clients. Check the
count after creating the key again, if the total kvs is bigger
than the expected count, then check the rev of the current key,
and take action accordingly based on its rev. If its rev is in
the first ${count}, then it's valid client, otherwise, it should
fail.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Rebased this PR. There is no response from the original author,
so Benjamin (ahrtr@) continue to work on this PR.
Signed-off-by: Vitalii Levitskii <vitalii@uber.com>
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Two fixes:
1. netutil: add url comparison without resolver to URLStringsEqual
2. client/v3: Add backoff before retry when watch stream returns unavailable
Signed-off-by: Benjamin Wang <wachao@vmware.com>
The client retries connection without backoff when the server is gone
after the watch stream is established. This results in high CPU usage
in the client process. This change introduces backoff when the stream is
failed and unavailable.
Signed-off-by: Hisanobu Tomari <posco.grubb@gmail.com>
These are nonsensical and a network implementation is not required
to handle them correctly, so panic instead of sending them out.
Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Since http2 spec defines the receive windows's size and max size of
frame in the stream, the underlayer - gRPC client can pre-read data
from server even if the application layer hasn't read it yet.
And the initialized cluster has 20KiB snapshot, which can be pre-read
by underlayer. We should increase the snapshot's size, just in case
that the io.Copy won't return the canceled or timeout error.
Fixes: #14477
Signed-off-by: Wei Fu <fuweid89@gmail.com>
etcdserver starts to using generic in https://github.com/etcd-io/etcd/pull/14515,
so any applications which depend on package `go.etcd.io/etcd/server/v3' must be
built with go version >= 1.18.
Most likely we may depends on go 1.18 new features in other packages, i.e. client, pkg,
so we just simplify the changelog message to require applications depending on any
etcd 3.6 pacakges must be built with go version >= 1.18, although only etcdserver uses
go generic for now.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
This commit simplifies the logic added in 37c7e4d to auto-leave joint
configurations. It does so by making the following adjustments to the
code:
- remove the `oldApplied <= r.pendingConfIndex` condition. This does
not seem necessary. When a node first attempts to auto-leave a joint
config, it will bump `r.pendingConfIndex` when proposing. In cases
where `oldApplied >= r.pendingConfIndex`, the proposal must have
already been applied. Reviewers should double check this.
- use raft.Step instead of custom proposal code. This code was already
present in stepLeader, so there was no reason to duplicate it. This
would have avoided bugs like the one we fixed in #14538.
- use `confChangeToMsg` to generate message, to centralize the creation
of all `MsgProp{EntryConfChange}` messages.
Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
This commit ensures that the raft leader eagerly broadcasts a MsgApp to
each follower when initiating an automatic transition out of a joint
configuration. This had been missed previously, which could lead to
delayed completion of an auto-transition.
Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
This commit updates the `RawNode`'s `prevHardSt` to the new HardState in
`acceptReady` instead of on `Advance`. This aligns the handling of
`prevHardSt` with that of `prevSoftSt` (and other fields like `msgs`)
and simplifies the logic in `Advance`.
Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
When a key-value store corruption check happens immediately after a
compaction, the revision at which the key-value store hash is computed,
is the compacted revision itself.
In that case, the hash computation logic was incorrect because it
returned an ErrCompacted error; this error should instead be returned when
the revision at which the key-value store is hashed, is strictly lower
than the compacted revision.
Fixes#14325
Signed-off-by: Jeremy Leach <44558776+jbml@users.noreply.github.com>
Some comments in the file are formatted automatically into ugly style,
because the hierarchical structure is missing. After removing the
leading numbers in the comments, `go fmt` will not format the comments
anymore.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
This reverts commit d73a986e4edb15ef9dbfc994f1cbf5e96694d877, which
was added only for benchmarking purposes.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
This makes the alternative option of implementing the leader's self-ack
of entry append the default.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
This is tested directly at the level of `RawNode` in
`TestRawNodeReadIndex`. `*node` is a thin wrapper around `RawNode` so
this is sufficient.
The reason to remove the test is that it now incurs data races
since it's not possible to adjust the `readStates` and `step`
fields while the node is running, and there is no primitive
to synchronize with its goroutine. This could all be fixed
but it's not worth it.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
This now needs an additional Ready cycle to apply the previous conf change,
so the finalizing conf change does too.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
Leader only acks to itself on `(*raft).advance` so we have to
make this test a bit more like the real thing.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
This was expecting the progress of the leader to be updated as a
result of MsgProp but it is now happening in `(*raft).advance`.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
This fixes essentially all tests using this, since now they don't have
to do anything special about the extra cycle introduced for single
nodes.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
Switched this to baking the conf changes into the initial state
to have fewer cycles to walk through in the test.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
This needed to apply entries from CommittedEntries, not Entries.
Previously the test got away with it because the two slices were
equal. Now it was hanging because when it proposed the second
conf change the first one hadn't applied yet, and so it got dropped,
and the test would hang.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
`StartNode` runs a naked goroutine, so it's impossible to test against
it in a way that will reliably produce contained test failures when
assertions are hit on the `(*node).run` goroutine.
This commit introduces a harness that we can use in tests to wrap
this goroutine and allow it to defer errors to `*testing.T`.
Note that tests of `Node` still need to be architected carefully
since it's easy to produce a deadlock in them should things not
go exactly as planned.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
See https://github.com/etcd-io/etcd/issues/14370.
When run in a single-voter configuration, prior to this PR
raft would emit `HardState`s that would emit a proposed `Entry`
simultaneously in `CommittedEntries` and `Entries`.
To be correct, this requires users of the raft library to enforce an
ordering between appending to the log and notifying the client about
`CommittedEntries` also present in `Entries`. This was easy to miss.
Walk back this behavior to arrive at a simpler contract: what's
emitted in `CommittedEntries` is truly committed, i.e. present
in stable storage on a quorum of voters.
This in turn pessimizes the single-voter case: rather than fully
handling an `Entry` in just one `Ready`, now two are required,
and in particular one has to do extra work to save on allocations.
We accept this as a good tradeoff, since raft primarily serves
multi-voter configurations.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
Show-cases the current behavior and changes made in future commits for [^1].
The test demonstrates that a single-voter raft instance will emit an
entry as committed while it still needs to be appended to the log.
[^1]: https://github.com/etcd-io/etcd/issues/14370
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
This is a speed-of-light benchmark that uses an in-memory single-voter
RawNode to sequentially propose and process entries.
As a bonus, it also measures the number of calls to the underlying
Storage. Calls to the Storage are cheap since the benchmark is in-
memory, but in a real-world implementation, especially one that doesn't
cache results, additional calls to the Storage interface can translate
to a heavy hit as they might involve additional I/O.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
RecentActive is now initialized to true in `becomeLeader`. Both
configuration changes and CheckQuorum make sure not to break this,
so we now now that the leader is always RecentActive.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
submitConcurrentWatch use sleep 3s to wait for all the watch connections
ready. When the number of connections increases, like 1000, the 3s is not
enough and the test case becomes flaky.
In this commit, spawn curl process and check the ouput line with
`created":true}}` to make sure that the connection has been initialized
and ready to receive the events. It is reliable to test the following
range request.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
github.com/golang-jwt/jwt adds go mod support startig from 4.0.0,
and it's backwards-compatible with existing v3.x.y tags.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
It isn't valid to start with quote character ' for flag -ldflags.
Go cmd older than 1.19 just ignores the error. Starting from go 1.19,
Go cmd will fail with error message something like below,
```
stderr: invalid value "'-X=go.etcd.io/etcd/api/v3/version.GitSHA=01250c9'" for flag -ldflags: parameter may not start with quote character '
```
Actually we don't have such quote character ' when building
etcd/etcdctl/etcdutl.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
The default Go version used by golinter is coming from the go.mod
file, fallback on the env var `GOVERSION`. So no need to configure
the go version in .golangci.yaml.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
The rules should be:
1. Each patch release should only include change against previous
patch, such as 3.5.4 vs 3.5.5.
2. For the first release (e.g. 3.6.0) for each minor version, we
should only describe new features or changes which are not
included in any releases in previous minor version.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Problem: both SIGQUIT_ETCD_AND_REMOVE_DATA_AND_STOP_AGENT and test.sh
will attempt to stop agents and remove directories.
Solution: since test.sh creates directories and starts test, it should be
responsible for cleanup.
See https://github.com/etcd-io/etcd/issues/14384
Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
There are duplicated items for https://github.com/etcd-io/etcd/pull/14441.
Since it's a etcdctl side change, so we should keep the item under etcdctl,
and remove the duplicated item under "etcd server"
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Add an overview and initial development guidelines. Restructured
the doc for a better readabiltiy and easier review, and per the
previous review feedback. The TODOs will be addressed iteratively.
Related #13775
Signed-off-by: Sahdev Zala <spzala@us.ibm.com>
Add feature request and test flake template to stop users from using wrong or empty
template
Redirect questions to discuss page to remove load from maintainers.
Remove distributors appliations as they are not used at all.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
1. authentication data not loaded on member startup
2. etcdctl move-leader may fail for multiple endpoints
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Due to a duplicate call of clientConfigFromCmd, the move-leader command
would fail with "conflicting environment variable is shadowed by corresponding command-line flag".
Also in scenarios where no command-line flag was supplied.
Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
A WAL object was closed by defer, however the WAL was rewritten afterwards,
so defer closed already closed WAL but not the new one. It caused a data
race between writing file and cleaning up a temporary test directory,
which led to a non-deterministic bug.
Fixes#14332
Signed-off-by: Vladimir Sokolov <vsvastey@gmail.com>
The alarm list is the only exception that doesn't move consistent_index
forward. The reproduction steps are as simple as,
```
etcd --snapshot-count=5 &
for i in {1..6}; do etcdctl alarm list; done
kill -9 <etcd_pid>
etcd
```
Signed-off-by: Benjamin Wang <wachao@vmware.com>
For a cluster with only one member, the raft always send identical
unstable entries and committed entries to etcdserver, and etcd
responds to the client once it finishes (actually partially) the
applying workflow.
When the client receives the response, it doesn't mean etcd has already
successfully saved the data, including BoltDB and WAL, because:
1. etcd commits the boltDB transaction periodically instead of on each request;
2. etcd saves WAL entries in parallel with applying the committed entries.
Accordingly, it may run into a situation of data loss when the etcd crashes
immediately after responding to the client and before the boltDB and WAL
successfully save the data to disk.
Note that this issue can only happen for clusters with only one member.
For clusters with multiple members, it isn't an issue, because etcd will
not commit & apply the data before it being replicated to majority members.
When the client receives the response, it means the data must have been applied.
It further means the data must have been committed.
Note: for clusters with multiple members, the raft will never send identical
unstable entries and committed entries to etcdserver.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
When we can't reach quorum, we were waiting forever and never sending
the systemd notify message. As a result, systemd would eventually time out
and restart the etcd process which likely would make the unhealthy cluster
in an even worse state
Improves #13785
Signed-off-by: Nicolai Moore <niconorsk@gmail.com>
Notes:
1. compactPhysical in ctlCtx and withQuota aren't used at all, they are dead code.
2. quotaBackendBytes in ctlCtx isn't used either. Instead, users (test cases) set the QuotaBackendBytes directly.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
etcdctl doesn't depend on the following packages at all,
1. go.etcd.io/etcd/client/v2
2. go.etcd.io/etcd/etcdutl/v3
3. go.etcd.io/etcd/raft/v3
4. go.etcd.io/etcd/server/v3
Signed-off-by: Benjamin Wang <wachao@vmware.com>
There are two cases, when interrupted by users, then forcibly kill
all processes. Otherwise, gracefully terminate all processes.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Streams are now closed after being used in the lessor `keepAliveOnce` method.
This prevents the "failed to receive lease keepalive request from gRPC stream"
message from being logged by the server after the context is cancelled by the
client.
Signed-off-by: Justin Kolberg <amd.prophet@gmail.com>
Only `net.TCPConn` supports `SetKeepAlive` and `SetKeepAlivePeriod`
by default, so if you want to warp multiple layers of net.Listener,
the `keepaliveListener` should be the one which is closest to the
original `net.Listener` implementation, namely `TCPListener`.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
The proxy must be waiting for the etcd to be running, but the current
implementation hard codes the wating time as 5 seconds. The improvement
is to dynamically check whether the etcd is running, and start the
proxy when etcd port is reachable.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
As protobuf doesn't have required field, user may send an empty
WatchRequest by mistake. Currently, etcd will ignore the invalid request
and keep the stream opening. If we don't reject the invalid request by
closing the stream, it would be better to leave a log there.
This commit also fixes a typo in the comment.
Signed-off-by: spacewander <spacewanderlzx@gmail.com>
To verify distributed tracing feature is correctly setup, this PR adds
an integration test for this feature.
In the process of writing the test, I discovered a goroutine leak due to
the TraceProvider not being closed. This PR fixs this issue as well.
Signed-off-by: Yingrong Zhao <yingrong.zhao@gmail.com>
--max-txn-ops flag allows users to define the maximum number of operations permitted in
a transaction during syncing updates.if unlimited syncing may fail when number of txn ops
exceeds the maximum number of server side.
Signed-off-by: xakdwch <xakdwch5@gmail.com>
Problem: We pass grpc context down to applier in readonly serializable txn.
This context can be cancelled for example due to timeout.
This will trigger panic inside applyTxn
Solution: Only panic for transactions with write operations
fixes https://github.com/etcd-io/etcd/issues/14110
Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
The `ErrFileNotFound` was used for for three cases:
1. There is no any WAL files (probably due to no read permission);
2. There is no WAL files matching the snapshot index;
3. The WAL file seqs do not increase continuously.
It's not good for debug when users see the `ErrFileNotFound` error,
so in this PR, a different error is returned for each case above.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
This should aid in debugging test flakes, especially in tests where the process is restarted very often and thus changes its pid.
Now it's a lot easier to grep for different members, also when different tests fail at the same time.
The test TestDowngradeUpgradeClusterOf3 as mentioned in #13167 is a good example for that.
Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
Here we add file `.golangci.yaml`, to have a common configuration for
static analysis. If you use the following IDEs/editors, they will read
the file:
- `vim` with vim-go.
- VSCode.
The initial configuration file is based on Kubernetes' HEAD. We removed
a custom kubernetes-linter for custom kubernetes-logging. We also
changed to Golang version 1.17 (etcd's current target Golang version)
from 1.18.
Also, we added a new target to `Makefile`: `lint`. NOTE that auto-fixing
should be part of a later commit, once we are all happy with how the
configuration file is working for us.
As expected, this change fixes two issues found by this `.golangci.yaml`
in file `contrib/lock/client/client.go`:
- Dead code, removed.
- Innefective assignment, removed.
Finally, we are updating `CONTRIBUTING.md` to mention `golangci-lint`.
We will add a GitHub-action to run `golangci-lint` in a future change.
Local testing done:
- `make build`.
- `make test`.
Both are happy.
This is the initial step to fix
https://github.com/etcd-io/etcd/issues/14164.
Signed-off-by: Ramsés Morales <ramses@gmail.com>
My responsibilities have long since been handed off to ptab@google.com and siarkowicz@google.com. I'd like to reflect that here in the MAINTAINERS file.
When asking CNCF for support due to reduced community activity we stumbled upon a problem,
governing board was surprised that project is unmaintained as there is so many maintainers listed for the project.
Feedback was to improve accuracy of maintainers list so it reflects only active maintainers.
Self nominating process it not enough and we need to start enforcing
moving maintainers to emeritus status. Period of 12 months to take maternity leave into account.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
In the test `testNonleaderElectionTimeoutRandomized`, the possible values of election timeout randomized out should be 10-19(inclusive), but the current test only tests the possible values as 11-19(inclusive), which is incorrect here. We need to expand the scope of the tests to ensure robustness, and also to make the logic clearer.
Signed-off-by: Zike Yang <zike@apache.org>
Updating the locking example to avoid relying on GC to break the lease.
Updating the documentation to reflect the build process better, improve
error messages and logging to not confuse first time users.
Fixes#14224
Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
The ROADMAP.md file was deleted as a part of commit
a722827c96 and the readme
file was not updated. I could not find a suitable alternative
resource so I am removing the link.
Signed-off-by: Austin Benoit <22805659+AustinBenoit@users.noreply.github.com>
ClusterID, memberID and leaseID are just identifies, so it doesn't
make sense to print them in decimal. Instead, we should always print
them in hexdecimal.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
`go install` is the recommended way to build and install packages in
module mode. `go get` should be used with the -d flag to adjust the
current module's dependencies without building packages, and use of
`go get` to build and install packages is deprecated. In a future
release, the -d flag will always be enabled.
Refer to https://go.dev/doc/go1.16.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
`Poll` means that the elements in the heap will be removed.
And it is more appropriate to use `Peek` to get the top of the heap.
Signed-off-by: SimFG <1142838399@qq.com>
The default max stream is 250 in http2. When there are more then
250 streams, the client side may be blocked until some previous
streams are released. So we need to support configuring a larger
`MaxConcurrentStreams`.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
The golang buildin package `flag` doesn't support `uint32` data
type, so we need to support it via the `flag.Var`.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
There is no update on the original PR (see below) for more then 2
weeks. So Benjamin(@ahrtr) continues to work on the PR. The first
step is to rebase the PR, because there are lots of conflicts with
the main branch.
The change to go.mod and go.sum reverted, because they are not needed.
The e2e test cases are also reverted, because they are not correct.
```
https://github.com/etcd-io/etcd/pull/14081
```
Signed-off-by: nic-chen <chenjunxu6@gmail.com>
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Make sure that WithPrefix correctly set the flag, and add test.
Also, add test for WithFromKey.
fixes#14056
Signed-off-by: Sahdev Zala <spzala@us.ibm.com>
Changes:
1. Upgrade go version to 1.16 in both go.mod and pieplines;
2. Upgrade boltdb from 1.3.3 to 1.3.6;
Signed-off-by: Benjamin Wang <wachao@vmware.com>
etcdctl/ctlv3: migrate cheggaaa/pb.v1 to cheggaaa/pb/v3
This commit also changes the format of the progress bar, from using a
custom progress bar to the default provided by the library.
Old behaviour:
./benchmarkv1 put
0 / 10000 B ! 0.00%
3987 / 10000 Boooooooooooooom ! 39.87%
10000 / 10000 Boooooooooooooooooooooooooooooooooooooooooooo! 100.00% 1s
New behaviour:
./benchmark put
6536 / 10000 [----------------------->________________] 65.36% 7053 p/s
10000 / 10000 [---------------------------------------] 100.00% 7581 p/s
Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info>
This PR describes the fact that different workloads in the `check perf` command
are different, and the results might vary.
Signed-off-by: Eduardo Patrocinio <epatro@gmail.com>
Currently the max size of each WAL entry is hard coded as 10MB. If users
set a value > 10MB for the flag --max-request-bytes, then etcd may run
into a situation that it successfully processes a big request, but fails
to decode it when replaying the WAL file on startup.
On the other hand, we can't just remove the limitation, because if a
WAL entry is somehow corrupted, and its recByte is a huge value, then
etcd may run out of memory. So the solution is to restrict the max size
of each WAL entry as a dynamic value, which is the remaining size of
the WAL file.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
The FileReader interface is the wrapper of io.Reader. It provides
the fs.FileInfo as well. The FileBufReader struct is the wrapper of
bufio.Reader, it also provides fs.FileInfo.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Before this change:
```
gwmux, err = sctx.registerGateway([]grpc.DialOption{grpc.WithInsecure()})
if err != nil { return err; }
```
caused the Etcd.Serve() method finish without sctx.serversC channel being closed.
The code closing Etcd got stuck waiting for all the servers to be running.
Signed-off-by: Piotr Tabor <ptab@google.com>
The TestFilePipelineFailLockFile case maybe test the line 77 in the alloc function. But the LockFile will not throw an error when the file lock is hold. And TestFilePipelineFailPreallocate and TestFilePipelineFailLockFile are exactly the same except for the comments.
Signed-off-by: SimFG <1142838399@qq.com>
We have already defined all the constant etcd versions in the
centralized place api/version/version.go. So we should replace all
the versions with the centralized definitions.
The versions such as V3_5, V3_6, etc. are defined everywhere in
the repo. In this commit, we get all of the constant versions defined
in a centralized place, so that they can be reused by all cases.
The current Makefile doesn't allow the compilation of the tools directory.
This commit creates a build library file, updates the Makefile
and a top level script fod building tools.
To build the tools, you can run make build_tools.
As before, you can run make build to build etcd binaries.
The current Makefile doesn't allow the compilation of the tools directory.
This commit updates the Makefile and the build file to add an option
to build the tools.
To build the tools, you can run make build_tools.
This PR:
- moves wrapping of appliers (due to Alarms) out of server.go into uber_applier.go
- clearly devides the application logic into: chain of:
a) 'WrapApply' (generic logic across all the methods)
b) dispatcher (translation of Apply into specific method like 'Put')
c) chain of 'wrappers' of the specific methods (like Put).
- when we do recovery (restore from snapshot) we create new instance of appliers.
The purpose is to make sure we control all the depencies of the apply process, i.e.
we can supply e.g. special instance of 'backend' to the application logic.
The PR removes calls to applierV3base logic from server.go that is NOT part of 'application'.
The original idea was that read-only transaction and Range call shared logic with Apply,
so they can call appliers directly (but bypassing all 'corrupt', 'quota' and 'auth' wrappers).
This PR moves all the logic to a separate file (that later can become package on its own).
In lack of supermajority, we sometimes required to hold on to
important decisions for long time. In order to speed up, after
giving enough time for supermajority, use lazy consensus.
This PR removes additional clone when building artifacts.
When releasing v3.5.4 this clone was main cause of issues and
confusion about what release script is doing.
release.sh script already clones repo in /tmp/ directory, so clonning
before build is not needed. As precautions for bug in script leaving
/tmp/ clone in bad state I moved "Verify the latest commit has the
version tag" and added "Verify the clean working tree" to be always run
before build.
Goroutine for new directors would live past director scope. Tests
could occassionally fail if this goroutine had log events after
test execution should have ended.
Benjamin is one of the current reviewers. Benjamin has proven high level of etcd expertise
and impact on the project, including delivering fixes to data-corruption bugs in etcd,
care about test quality and overall project health.
Per discussion with existing maintainers, I nominate him for the maintainer role.
Benjamin, thank you for your outstanding contributions to the project.
Usually the consistent_index should be greater than the index of the
latest snapshot with suffix .snap. But for the snapshot coming from the
leader, the consistent_index should be equal to the snapshot index.
Add missing install commands. Update benchmark to go install as go
get usage is deprecated. Also, improve overall instructions and
experience.
related #13991
Problem: TestLeaseGrantAndList is flaky because lists won't match at the end.
Test uses revision to verify that all members got leases. But checking for revision isn't enough.
Solution: use size of the list to stop polling.
When clients have no permission to perform whatever operation, then
the applying may fail. We should also move consistent_index forward
in this case, otherwise the consitent_index may smaller than the
snapshot index.
Downstream users of etcd experience build issues when using dependencies
which require more recent (incompatible) versions of opentelemetry. This
commit upgrades the dependencies so that downstream users stop
experiencing these issues.
The first issue is to support linearizable lease renewal request.
The second issue is to prevent the consistent_index coming from
a snapshot is overwritten by the local old value.
Update raftexample to save the snapshot file and WAL snapshot entry
before hardstate to ensure the snapshot exists during recovery.
Otherwise if there is a failure after storing the hard state there may
be reference to a non-existent snapshot.
This PR introduces the fix from #10219 to the raftexample.
When a user sets up a Mirror with a restricted user that doesn't have
access to the `foo` path, we will fail to get the most recent revision
due to permissions issues.
With this change, when a prefix is provided we will get the initial
revision from the prefix rather than /foo. This allows restricted users
to setup sync.
When etcdserver receives a LeaseRenew request, it may be still in
progress of processing the LeaseGrantRequest on exact the same
leaseID. Accordingly it may return a TTL=0 to client due to the
leaseID not found error. So the leader should wait for the appliedID
to be available before processing client requests.
humanize.Time() drops precision resulting in some events reporting they took
"now" time to complete. Using time.Duration.String() results in accurate
duration being reported.
Fixes#13905
Reason to store CI and term in backend was to make db fully independent
snapshot, it was never meant to interfere with apply logic. Skip of CI
was introduced for v2->v3 migration where we wanted to prevent it from
decreasing when replaying wal in
https://github.com/etcd-io/etcd/pull/5391. By mistake it was added to
apply flow during refactor in
https://github.com/etcd-io/etcd/pull/12855#commitcomment-70713670.
Consistency index and term should only be negotiated and used by raft to make
decisions. Their values should only driven by raft state machine and
backend should only be responsible for storing them.
Currently there are a handful of tests within etcd that silently fail
because LeakDetection will skip the test before it manages to hit this
check.
Here we move the check to the beginning of the process to highlight
these cases earlier, and to avoid them accidentally presenting as leaks.
Removed the fields consistentIdx and consistentTerm from struct EtcdServer,
and added applyingIndex and applyingTerm into struct consistentIndex in
package cindex. We may remove the two fields completely if we decide to
remove the OnPreCommitUnsafe, and it will depend on the performance test
result.
Previously the SetConsistentIndex() is called during the apply workflow,
but it's outside the db transaction. If a commit happens between SetConsistentIndex
and the following apply workflow, and etcd crashes for whatever reason right
after the commit, then etcd commits an incomplete transaction to db.
Eventually etcd runs into the data inconsistency issue.
In this commit, we move the SetConsistentIndex into a txPostLockHook, so
it will be executed inside the transaction lock.
We don't consistently reach the same etcd server during the lifetime of
a test and in some cases, this means that this test will flake if an
etcd server was slow to update its state and the test hits the outdated
server.
Here we switch to using an `Eventually` case which will wait upto a
second for the expected result before failing - with a 10ms gap between
invocations.
```
[tests(dani/leasefix)] $ gotestsum -- ./common -tags integration -count 100 -timeout 15m -run TestLeaseGrantAndList
✓ common (2m26.71s)
DONE 1600 tests in 147.258s
```
I think strong (not-equal) relationship was too restrictive when expressed with 1s granularity.
```
logger.go:130: 2022-04-03T22:15:15.242+0200 WARN m1 leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"member": "m1", "to": "cb785755eb80ac1", "heartbeat-interval": "10ms", "expected-duration": "20ms", "exceeded-duration": "24.666613ms"}
logger.go:130: 2022-04-03T22:15:15.262+0200 INFO m-1 published local member to cluster through raft {"member": "m-1", "local-member-id": "e2dd9f523aa7be87", "local-member-attributes": "{Name:m-1 ClientURLs:[unix://127.0.0.1:2196386040]}", "cluster-id": "b4b8e7e41c23c8b5", "publish-timeout": "5.2s"}
v3_lease_test.go:415: Expected lease ttl (4m58s) to be greather than (4m58s)
```
The code now ensures that each of the test is running in its own directory as opposed to shared os.tempdir.
```
$ (cd tests && env go test -timeout=15m --race go.etcd.io/etcd/tests/v3/integration/clientv3/examples -run ExampleAuth)
2022/04/03 10:24:59 Running tests (examples): ...
2022/04/03 10:24:59 the function can be called only in the test context. Was integration.BeforeTest() called ?
2022/04/03 10:24:59 2022-04-03T10:24:59.462+0200 INFO m0 LISTEN GRPC {"member": "m0", "grpcAddr": "localhost:m0", "m.Name": "m0"}
```
Nearly none of the tests was checking the value... just assuming WaitLeader success.
```
maintenance_test.go:277: Waiting for leader...
logger.go:130: 2022-04-03T08:01:09.914+0200 INFO m0 cluster version differs from storage version. {"member": "m0", "cluster-version": "3.6.0", "storage-version": "3.5.0"}
logger.go:130: 2022-04-03T08:01:09.915+0200 WARN m0 leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"member": "m0", "to": "2acc3d3b521981", "heartbeat-interval": "10ms", "expected-duration": "20ms", "exceeded-duration": "103.756219ms"}
logger.go:130: 2022-04-03T08:01:09.916+0200 INFO m0 updated storage version {"member": "m0", "new-storage-version": "3.6.0"}
...
logger.go:130: 2022-04-03T08:01:09.926+0200 INFO grpc [[roundrobin] roundrobinPicker: Build called with info: {map[0xc002630ac0:{{unix:localhost:m0 localhost <nil> 0 <nil>}} 0xc002630af0:{{unix:localhost:m1 localhost <nil> 0 <nil>}} 0xc002630b20:{{unix:localhost:m2 localhost <nil> 0 <nil>}}]}]
logger.go:130: 2022-04-03T08:01:09.926+0200 WARN m0 apply request took too long {"member": "m0", "took": "114.661766ms", "expected-duration": "100ms", "prefix": "", "request": "header:<ID:12658633312866157316 > cluster_version_set:<ver:\"3.6.0\" > ", "response": ""}
logger.go:130: 2022-04-03T08:01:09.927+0200 INFO m0 cluster version is updated {"member": "m0", "cluster-version": "3.6"}
logger.go:130: 2022-04-03T08:01:09.955+0200 INFO m2.raft 9f96af25a04e2ec3 [logterm: 2, index: 8, vote: 9903a56eaf96afac] ignored MsgVote from 2acc3d3b521981 [logterm: 2, index: 8] at term 2: lease is not expired (remaining ticks: 10) {"member": "m2"}
logger.go:130: 2022-04-03T08:01:09.955+0200 INFO m0.raft 9903a56eaf96afac [logterm: 2, index: 8, vote: 9903a56eaf96afac] ignored MsgVote from 2acc3d3b521981 [logterm: 2, index: 8] at term 2: lease is not expired (remaining ticks: 5) {"member": "m0"}
logger.go:130: 2022-04-03T08:01:09.955+0200 INFO m0.raft 9903a56eaf96afac [term: 2] received a MsgAppResp message with higher term from 2acc3d3b521981 [term: 3] {"member": "m0"}
logger.go:130: 2022-04-03T08:01:09.955+0200 INFO m0.raft 9903a56eaf96afac became follower at term 3 {"member": "m0"}
logger.go:130: 2022-04-03T08:01:09.955+0200 INFO m0.raft raft.node: 9903a56eaf96afac lost leader 9903a56eaf96afac at term 3 {"member": "m0"}
maintenance_test.go:279: Leader established.
```
Tmp
```
% (cd client/v3 && env go test -short -timeout=3m --race ./...)
--- FAIL: TestAuthTokenBundleNoOverwrite (0.00s)
client_test.go:210: listen unix /var/folders/t1/3m8z9xz93t9c3vpt7zyzjm6w00374n/T/TestAuthTokenBundleNoOverwrite3197524989/001/etcd-auth-test:0: bind: invalid argument
FAIL
FAIL go.etcd.io/etcd/client/v3 4.270s
```
The reason was that the path exceeded 108 chars (that is too much for socket).
In the mitigation we first change chroot (working directory) to the tempDir... such the path is 'local'.
We've had these since in one form or another since 23a302364c
(Makefile: initial commit, 2017-09-29), but in at least some cases the
underlying shell does not pipefail, a test failure gets swallowed, and
the make call exits zero despite failing the tests [1]:
$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_etcd/109/pull-ci-openshift-etcd-openshift-4.11-unit/1509260812278042624/artifacts/test/build-log.txt
TEST_OPTS: PASSES='unit'
log-file: test-MTY0ODY3MTA1MQo.log
PASSES='unit' ./test.sh 2>&1 | tee test-MTY0ODY3MTA1MQo.log
% env GO111MODULE=off go get github.com/myitcv/gobin
Running with --race
Starting at: Wed Mar 30 20:10:52 UTC 2022
'unit' started at Wed Mar 30 20:10:52 UTC 2022
% (cd api && env go test -short -timeout=3m --race ./...)
stderr: authpb/auth.pb.go:12:2: open /go/pkg/mod/github.com/gogo/protobuf@v1.3.2/gogoproto: permission denied
stderr: authpb/auth.pb.go:13:2: open /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto: permission denied
stderr: etcdserverpb/rpc.pb.go:17:2: open /go/pkg/mod/google.golang.org/genproto@v0.0.0-20210602131652-f16073e35f0c/googleapis/api/annotations: permission denied
stderr: etcdserverpb/rpc.pb.go:18:2: open /go/pkg/mod/google.golang.org/grpc@v1.38.0: permission denied
stderr: etcdserverpb/rpc.pb.go:19:2: open /go/pkg/mod/google.golang.org/grpc@v1.38.0/codes: permission denied
stderr: etcdserverpb/rpc.pb.go:20:2: open /go/pkg/mod/google.golang.org/grpc@v1.38.0/status: permission denied
stderr: etcdserverpb/gw/rpc.pb.gw.go:17:2: open /go/pkg/mod/github.com/golang/protobuf@v1.5.2/descriptor: permission denied
stderr: etcdserverpb/gw/rpc.pb.gw.go:19:2: open /go/pkg/mod/github.com/grpc-ecosystem/grpc-gateway@v1.16.0/runtime: permission denied
stderr: etcdserverpb/gw/rpc.pb.gw.go:20:2: open /go/pkg/mod/github.com/grpc-ecosystem/grpc-gateway@v1.16.0/utilities: permission denied
FAIL: (code:1):
% (cd api && env go test -short -timeout=3m --race ./...)
stderr: etcdserverpb/gw/rpc.pb.gw.go:23:2: open /go/pkg/mod/google.golang.org/grpc@v1.38.0/grpclog: permission denied
stderr: version/version.go:23:2: open /go/pkg/mod/github.com/coreos/go-semver@v0.3.0/semver: permission denied
FAIL: 'unit' failed at Wed Mar 30 20:10:52 UTC 2022
! egrep "(--- FAIL:|DATA RACE|panic: test timed out|appears to have leaked)" -B50 -A10 test-MTY0ODY3MTA1MQo.log
We can't drop the log aggregation, because the log files are used for
the panic/race grepping. But I'm dropping the tee (so no more
synchronous updates, but we no longer have to worry about pipefail
handling). And then if the test script fails, I'm dumping the log
file to stdout and exiting 1, so the overall run fails.
[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_etcd/109/pull-ci-openshift-etcd-openshift-4.11-unit/1509260812278042624
This change is to ensure that all members returned during the client's
AutoSync are started and are not learners, which are not valid
etcd members to make requests to.
When performing the downgrade operation, users can confirm whether each member
is ready to be downgraded using the field 'storageVersion'. If it's equal to the
'target version' in the downgrade command, then it's ready to be downgraded;
otherwise, the etcd member is still in progress of processing the db file.
Not all systems include binaries in the same location. On my (NixOS, so
albeit a little weird) system these binaries exist in very different
locations.
This test switches to looking up the paths from the users PATH or skips
the test if they do not exist to improve the `make test` experience on
such systems.
TestMutexTryLock(SingleNode|MultiNode) were being skipped as they had duplicate calls to integration2.BeginTest. These duplicate calls registered a second handler for leak detection which failed and skipped the test as the wrappers had already started a cluster.
part of #13698
This test was being skipped as it has a duplicate call to
integration2.BeginTest. This duplicate call registered a second handler
for leak detection which failed and skipped the test as the wrapper
(runWatchTest) has already started a cluster.
The ClientConfig is a fully declarive configuration, so it makes more
sense to rename it to ConfigSpec. It can also mitigate the confusion
between Config and ClientConfig.
When there is a `raftpb.EntryConfChange` after creating the snapshot,
then the confState included in the snapshot is out of date. so We need
to update the confState before sending a snapshot to a follower.
Currently the discovery url is just one endpoint. But actually it
should be the same as the etcdctl, which means that it should be
a list of endpoints. When one endpoint is down, the clientv3 can
fail over to the next endpoint automatically.
Problem: Defrag was implemented before custom bolt options were added.
Currently defrag doesn't restore backend options.
For example BackendFreelistType will be unset after defrag.
Solution: save bolt db options and use them in defrag.
The repository github.com/myitcv/gobin has already been archived,
and the `go install` command accepts arguments with version suffixs
starting from 1.16 (for example, go install example.com/cmd@v1.0.0).
So there is no reason to continue to use gobin.
Marek is one of the current reviewers. He has consistently
performed all the tasks to earn a top-level maintainer role as
described in the
https://github.com/etcd-io/etcd/blob/main/GOVERNANCE.md#maintainers
Per discussion with existing maintainers, I nominate him for the
maintainer role.
Marek, thank you for your outstanding contributions to the project.
When getting warnings from the retry_interceptors that RPCs failed, they
provide very little context as to what exactly failed.
I used this patch to try debugging
https://groups.google.com/g/etcd-dev/c/3hxxr9ktdpM.
This attempts to fix a special case of the problem described in #12385,
where trying to do `clientv3.Watch` with an expired token would result
in `ErrGRPCPermissionDenied`, due to the failing authorization check in
`isWatchPermitted`. Furthermore, the client can't auto recover, since
`shouldRefreshToken` rightly returns false for the permission denied
error.
In this case, we would like to have a runbook to dynamically disable
auth, without causing any disruption. Doing so would immediately expire
all existing tokens, which would then cause the behavior described
above. This means existing watchers would still work for a period of
time after disabling auth, until they have to reconnect, e.g. due to a
rolling restart of server nodes.
This commit adds a client-side fix and a server-side fix, either of
which is sufficient to get the added test case to pass. Note that it is
an e2e test case instead of an integration one, as the reconnect only
happens if the server node is stopped via SIGINT or SIGTERM.
A generic fix for the problem described in #12385 would be better, as
that shall also fix this special case. However, the fix would likely be
a lot more involved, as some untangling of authn/authz is required.
In the sample code demonstrating how to specify a client request
timeout, the `cancel()` is called immediately after the Put, but it
should be deferred instead, giving the Put enough time to complete.
In the canonical Golang context
[docs](https://pkg.go.dev/context#WithTimeout), the sample code sets a
`defer cancel()` immediately after context creation, and with this
commit we adhere to that convention.
fixes:
```json
{
"level": "warn",
"ts": "2021-12-29T09:56:42.439-0800",
"logger": "etcd-client",
"caller": "v3@v3.5.1/retry_interceptor.go:62",
"msg": "retrying of unary invoker failed",
"target": "etcd-endpoints://0xc000213340/localhost:2379",
"attempt": 0,
"error": "rpc error: code = Canceled desc = context canceled"
}
```
raft: fix goroutine leaks in TestCommitPagination
The goroutine created with n.run() will leak if we forget to call n.Stop().
We can replay the goroutine leaks by using [goleak](https://github.com/uber-go/goleak):
```
$ cd raft && env go test -short -v -timeout=3m --race -run=TestCommitPagination.
... ...
raft2021/12/27 20:47:15 INFO: raft.node: 1 elected leader 1 at term 1
leaks.go:78: found unexpected goroutines:
[Goroutine 20 in state select, with go.etcd.io/etcd/raft/v3.(*node).run on top of the stack:
goroutine 20 [select]:
go.etcd.io/etcd/raft/v3.(*node).run(0xc00036f260)
/home/yuanting/work/dev/goprojects/etcd/raft/node.go:344 +0xc1d
created by go.etcd.io/etcd/raft/v3.TestCommitPagination
/home/yuanting/work/dev/goprojects/etcd/raft/node_test.go:920 +0x554
]
--- FAIL: TestCommitPagination (0.45s)
FAIL
FAIL go.etcd.io/etcd/raft/v3 0.508s
FAIL
```
This has been additionally verified by running the tests locally as a
basic smoke test. GitHub Actions doesn't provide MacOS M1 (arm64) yet,
so there's no good way to automate testing.
Ran `TMPDIR=/tmp make test` locally. The `TMPDIR` bit is needed so
there's no really long path used that breaks Unix socket setup in one of
the tests.
To avoid inconsistant behavior during cluster upgrade we are feature
gating persistance behind cluster version. This should ensure that
all cluster members are upgraded to v3.6 before changing behavior.
To allow backporting this fix to v3.5 we are also introducing flag
--experimental-enable-lease-checkpoint-persist that will allow for
smooth upgrade in v3.5 clusters with this feature enabled.
Current checkpointing mechanism is buggy. New checkpoints for any lease
are scheduled only until the first leader change. Added fix for that
and a test that will check it.
Upgrading from v1.0.1.
Upgrading related dependencies
------------------------------
The following dependencies also had to be upgraded:
- go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.26.1
From v0.25.0. This gets rid of a transitive dependency on go.opentelemetry.io/otel@v1.0.1.
- google.golang.org/genproto@v0.0.0-20211118181313-81c1377c94b1
This is because etcd v3.5 will panic when it encounters
ClusterVersionSet entry with version >3.5.0. For downgrades to v3.5 to
work we need to make sure this entry is snapshotted.
Problem with old code was that during downgrade only members with
downgrade target version were allowed to join. This is unrealistic as
it doesn't handle any members to disconnect/rejoin.
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
To make it possible to alert on misconfigured etcd clusters that have
missing/superfluous peers, expose the list of peers as a metric.
This metric can, for example, be compared to the control-plane nodes of
a kubernetes cluster.
Resolves#12450
This commits adds support to unix/unixs socket URLs, which currently
fail with the message "URL address does not have the form "host:port".
It also replaces the work started in #11747.
Json encoding is the default zap encoding value and can not be changeable.
This PR enables configuring zap encoding to console via new flag `log-format`.
prefixArgs uses os.Setenv in e2e tests instead envMap.
This creates overwrites in some test cases and have an impact
on test quality and isolation between tests.
This PR uses ctlcontext envMap in each tests with high priority
and merges os environment variables with low priority.
Storage version should follow cluster version. During upgrades this
should be immidiate as storage version can be always upgraded as storage
is backward compatible. During downgrades it will be delayed and will
require time for incompatible changes to be snapshotted.
As storage version change can happen long after cluster is running, we
need to add a step during bootstrap to validate if loaded data can be
understood by migrator.
To fix a bug in the retry logic caused when the auth token is cleared after receiving `ErrInvalidAuthToken` from the server and the subsequent call to `getToken` also fails due to some reason (eg. context deadline exceeded).
This leaves the client without a token and the retry will continue to fail with `ErrUserEmpty` unless the token is refreshed.
A client-side optimization was made in #6100 to filter ascending key sorts to avoid an unnecessary re-sort since this is the order already returned by the back-end logic.
It seems to me that this really belongs on the server side since it's tied to the server implementation and should apply for any caller of the kv api (for example non-go clients).
Related, the client/v3 syncer depends on this default sorting which isn't explicit in the kv api contract. So I'm proposing the required sort parameters be included explicitly; it will take the fast path either way.
add 'FAIL:' to egrep
```shell
[root@LF-136-9 etcd]# egrep "(--- FAIL:|DATA RACE|panic: test timed out|appears to have leaked)" -B50 -A10 test-MTYyNjIyOTc0MQo.log
[root@LF-136-9 etcd]# egrep "(--- FAIL:|FAIL:|DATA RACE|panic: test timed out|appears to have leaked)" -B50 -A10 test-MTYyNjIyOTc0MQo.log
ok go.etcd.io/etcd/server/v3/auth 3.247s
ok go.etcd.io/etcd/server/v3/config 0.047s
ok go.etcd.io/etcd/server/v3/datadir 0.035s
ok go.etcd.io/etcd/server/v3/embed 1.944s
ok go.etcd.io/etcd/server/v3/etcdmain 0.326s
FAIL go.etcd.io/etcd/server/v3/etcdserver [build failed]
? go.etcd.io/etcd/server/v3/etcdserver/api [no test files]
ok go.etcd.io/etcd/server/v3/etcdserver/api/etcdhttp 0.110s
ok go.etcd.io/etcd/server/v3/etcdserver/api/membership 0.479s
ok go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp 0.251s
ok go.etcd.io/etcd/server/v3/etcdserver/api/snap 0.045s
? go.etcd.io/etcd/server/v3/etcdserver/api/snap/snappb [no test files]
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2auth 1.470s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2discovery 0.088s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2error 0.034s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2http 0.128s
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2http/httptypes 0.033s
? go.etcd.io/etcd/server/v3/etcdserver/api/v2stats [no test files]
ok go.etcd.io/etcd/server/v3/etcdserver/api/v2store 0.068s
? go.etcd.io/etcd/server/v3/etcdserver/api/v2v3 [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3alarm [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3client [no test files]
ok go.etcd.io/etcd/server/v3/etcdserver/api/v3compactor 1.793s
? go.etcd.io/etcd/server/v3/etcdserver/api/v3election [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3election/v3electionpb [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3election/v3electionpb/gw [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3lock [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3lock/v3lockpb [no test files]
? go.etcd.io/etcd/server/v3/etcdserver/api/v3lock/v3lockpb/gw [no test files]
ok go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc 0.089s
ok go.etcd.io/etcd/server/v3/etcdserver/cindex 0.045s
ok go.etcd.io/etcd/server/v3/lease 3.324s
ok go.etcd.io/etcd/server/v3/lease/leasehttp 2.096s
? go.etcd.io/etcd/server/v3/lease/leasepb [no test files]
? go.etcd.io/etcd/server/v3/mock/mockstorage [no test files]
? go.etcd.io/etcd/server/v3/mock/mockstore [no test files]
? go.etcd.io/etcd/server/v3/mock/mockwait [no test files]
ok go.etcd.io/etcd/server/v3/mvcc 8.805s
ok go.etcd.io/etcd/server/v3/mvcc/backend 1.983s
? go.etcd.io/etcd/server/v3/mvcc/backend/testing [no test files]
? go.etcd.io/etcd/server/v3/mvcc/buckets [no test files]
? go.etcd.io/etcd/server/v3/proxy/grpcproxy [no test files]
? go.etcd.io/etcd/server/v3/proxy/grpcproxy/adapter [no test files]
? go.etcd.io/etcd/server/v3/proxy/grpcproxy/cache [no test files]
ok go.etcd.io/etcd/server/v3/proxy/httpproxy 0.046s
ok go.etcd.io/etcd/server/v3/proxy/tcpproxy 0.035s
? go.etcd.io/etcd/server/v3/verify [no test files]
ok go.etcd.io/etcd/server/v3/wal 0.513s
ok go.etcd.io/etcd/server/v3/wal/walpb 0.045s
FAIL
FAIL: (code:2):
% (cd server && env go test -short -timeout=3m --race --cpu=16 ./...)
FAIL: 'unit' failed at Wed Jul 14 10:29:37 CST 2021
```
When a unary request takes more than predefined duration, this request
is defined as "expensive" and a warning is printed. The expensive request
duration is hard-coded to 300 ms. It can be not enough for example
for transactions with a lot of operations. The warnings just blow up
the log files and reduce throughput.
This fix allows user to configure the "expensive" request duration.
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
The file `zap_raft.go` adds the raft.Logger proxy logger on top of `*zap.Logger`.
Adding a proxy requires adding the option `zap.AddCallerSkip(1)`,
so that the logging message specifies the correct caller,
two of the three constructors in the `zap_raft.go` adds this option.
This commit fixes the third constructor so that it also adds `zap.AddCallerSkip`.
Before fix:
`{"level":"info","ts":"2021-07-22T17:46:01.435Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"bd07d29169ff0c5a [logterm: 2, index: 8, vote: 38447ba545569bbe] ignored MsgPreVote from c7baeaad79d6d5ed [logterm: 2, index: 8] at term 2: lease is not expired (remaining ticks: 10)"}`
After fix:
`{"level":"info","ts":"2021-07-22T17:46:51.227Z","logger":"raft","caller":"raft/raft.go:859","msg":"bd07d29169ff0c5a [logterm: 2, index: 8, vote: c7baeaad79d6d5ed] ignored MsgPreVote from 38447ba545569bbe [logterm: 2, index: 8] at term 2: lease is not expired (remaining ticks: 9)"}`
During the refactoring process, duplicate logging
of the send buffer overflow event was added.
Each of these log lines logs exactly the same information, the logging
context is sufficient to distinguish the cause.
Additionally, the unnecessary context (in parentheses) in the log
message was removed, which was necessary without the zap context (with
the old logger), but now only confuses.
If one of the nodes in the cluster has lost a dns record,
restarting the second node will break it.
This PR makes an attempt to add a comparison without using a resolver,
which allows to protect cluster from dns errors and does not break
the current logic of comparing urls in the URLStringsEqual function.
You can read more in the issue #7798Fixes#7798
we found a lease leak issue:
if a new member(by member add) is recovered by snapshot, and then
become leader, the lease will never expire afterwards. leader will
log the revoke failure caused by "invalid auth token", since the
token provider is not functional, and drops all generated token
from upper layer, which in this case, is the lease revoking
routine.
When running 100 times in row those tests flaked around 10-20%. Based on
some experimentation 10 keys was enough to ensure that wal snapshot is
created and prevented any flakes.
Use the FileLockEx wrapper and the corresponding LOCKFILE_* and error
constants from the golang.org/x/sys/windows package rather than
implementing these in the fileutil package.
`Client.dial` can be called multiple times. For example, from regular
`NewClient` and [from the `Maintenance`
wrapper](6d451ab61d/client/v3/maintenance.go (L81-L84)).
This ends up creating two (if `Client.dial` is called twice) independent
`credentials.Bundle` instances:
- the first one is wired into `PerRPCCredentials` callback in gRPC
client.
- the second one is in `Client.authTokenBundle` and receives the latest
auth token updates.
When using username/password authentication and the server is configured
to issue JWTs, token rotation logic breaks because of the above
`credentials.Bundle` confusion.
There are many developers on the web, confused by how to install etcd v3 for the development of go project.
A number of similar errors were encountered, as follows:
https://github.com/etcd-io/etcd/issues/13108
```
➜ ~ go version
go version go1.16.5 darwin/amd64
➜ ~ go get go.etcd.io/etcd/clientv3
go: downloading golang.org/x/net v0.0.0-20201021035429-f5854403a974
# github.com/coreos/etcd/clientv3/balancer/picker
go/pkg/mod/github.com/coreos/etcd@v3.3.25+incompatible/clientv3/balancer/picker/err.go:37:44: undefined: balancer.PickOptions
go/pkg/mod/github.com/coreos/etcd@v3.3.25+incompatible/clientv3/balancer/picker/roundrobin_balanced.go:55:54: undefined: balancer.PickOptions
# github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
go/pkg/mod/github.com/coreos/etcd@v3.3.25+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:114:78: undefined: resolver.BuildOption
go/pkg/mod/github.com/coreos/etcd@v3.3.25+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:182:31: undefined: resolver.ResolveNowOption
```
There are many examples of confusion here:
- https://github.com/etcd-io/etcd/issues/12577
- https://github.com/etcd-io/etcd/issues/12181
- https://github.com/etcd-io/etcd/issues/12068
- https://github.com/etcd-io/etcd/issues/11931
......
So, As suggested by @lilic , I submitted Readme update.
There are many developers on the web, confused by how to install etcd v3 for the development of go project.
A number of similar errors were encountered, as follows:
https://github.com/etcd-io/etcd/issues/13108
```
➜ ~ go version
go version go1.16.5 darwin/amd64
➜ ~ go get go.etcd.io/etcd/clientv3
go: downloading golang.org/x/net v0.0.0-20201021035429-f5854403a974
# github.com/coreos/etcd/clientv3/balancer/picker
go/pkg/mod/github.com/coreos/etcd@v3.3.25+incompatible/clientv3/balancer/picker/err.go:37:44: undefined: balancer.PickOptions
go/pkg/mod/github.com/coreos/etcd@v3.3.25+incompatible/clientv3/balancer/picker/roundrobin_balanced.go:55:54: undefined: balancer.PickOptions
# github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
go/pkg/mod/github.com/coreos/etcd@v3.3.25+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:114:78: undefined: resolver.BuildOption
go/pkg/mod/github.com/coreos/etcd@v3.3.25+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:182:31: undefined: resolver.ResolveNowOption
```
There are many examples of confusion here:
- https://github.com/etcd-io/etcd/issues/12577
- https://github.com/etcd-io/etcd/issues/12181
- https://github.com/etcd-io/etcd/issues/12068
- https://github.com/etcd-io/etcd/issues/11931
......
So, As suggested by @lilic , I submitted Readme update.
Narrowly prevent etcd from crashing when given a bad ACTIVATE payload, e.g.:
$ curl -d "{\"action\":\"ACTIVATE\"}" ${ETCD}/v3/maintenance/alarm
curl: (52) Empty reply from server
Ensure the client which access etcd via grpc-gateway won't
be limited by the MaxCallRecvMsgSize. Here we choose the same
default value of etcdcli as grpc-gateway's MaxCallRecvMsgSize.
Fix https://github.com/etcd-io/etcd/issues/12576
- Use go testing infrastructure, such that logs are printed only in
case of failure.
- Split the test into multiple sub-round and sub-round-testcases such
that tree of execution reflects the logic.
- Use 'development' loggers that are formatting the output to be human
readable.
---
(If this PR fixes a github issue, please add `Fixes #<xyz>`)
Fixes#13023
(or if this PR is one task of a github issue, please add `Master Issue: #<xyz>`
to link to the master issue)
Master Issue: #13023
*Motivation*
Currently, StubServer can stop itself (by Stop()) before it start its
grpc server (created by Start()). This race condition may lead to a
goroutine leak mentioned by #13023.
*Modifications*
This PR add a channel to force the Stop() started after Start()
*Verify this change*
Please pick either of following options.
- This change is already covered by existing tests, such as *(please describe tests)*.
Test/TestEtcdGrpcResolver
Used github.com/stretchr/testify/assert to replace the AssertEqual function definition. Required the use of copyToInterface to copy a string slice to the empty interface slice
Due to flakes in CI and locally which resulted due to shared servers
there is suspcion that fixed port numbers are to blame. This reuses port
and address.
- ErrGRPCNotCapable("etcdserver: not capable") -> codes.FailedPrecondition (it will not autofix, it requires new version of server)
- ErrGPRCNotSupportedForLearner("etcdserver: rpc not supported for learner") -> codes.FailedPrecondition (as long as its learner, the call will not work)
- ErrGRPCClusterVersionUnavailable("etcdserver: cluster version not found during downgrade") -> codes.FailedPrecondition (backend does not contain the version (old etcd?) so retry will not help)
https://github.com/etcd-io/etcd/runs/2599598633?check_suite_focus=true
```
{"level":"warn","ts":"2021-05-17T09:55:30.246Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000539880/#initially=[unix://localhost:m30]","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: rpc not supported for learner"}
{"level":"warn","ts":"2021-05-17T09:55:30.270Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying
of unary invoker
failed","target":"etcd-endpoints://0xc000539880/#initially=[unix://localhost:m30]","attempt":1,"error":"rpc
error: code = Unavailable desc = etcdserver: rpc not supported for
learner"}`
```
During review of: https://github.com/etcd-io/etcd/pull/12988 spotted
that PUT is actially writing to v3-backend.
If we are replaying WAL log, it might happened that backend's
applied_index is > than the WAL's log entry. In such situation we should
skip applying on backend V3.
I think both the methods (setVersion, setMembersAttributes) are in
practice idempotent so its not that 'serious' problem, but for
formal correctness adding the proper checks.
Thanks to this change:
- all the maps bucket -> buffer are indexed by int's instead of
string. No need to do: byte[] -> string -> hash conversion on each
access.
- buckets are strongly typed in backend/mvcc API.
description:Report a bug encountered while operating etcd
labels:
- type/bug
body:
- type:checkboxes
id:confirmations
attributes:
label:Bug report criteria
description:Please confirm this bug report meets the following criteria.
options:
- label:This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
- label:This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- label:You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- label:Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.
- type:markdown
attributes:
value:|
Please fill the form below and provide as much information as possible.
Not doing so may result in your bug not being addressed in a timely manner.
- type:textarea
id:problem
attributes:
label:What happened?
validations:
required:true
- type:textarea
id:expected
attributes:
label:What did you expect to happen?
validations:
required:true
- type:textarea
id:repro
attributes:
label:How can we reproduce it (as minimally and precisely as possible)?
validations:
required:true
- type:textarea
id:additional
attributes:
label:Anything else we need to know?
- type:textarea
id:etcdVersion
attributes:
label:Etcd version (please run commands below)
value:|
<details>
```console
$ etcd --version
# paste output here
$ etcdctl version
# paste output here
```
</details>
validations:
required:true
- type:textarea
id:config
attributes:
label:Etcd configuration (command line flags or environment variables)
value:|
<details>
# paste your configuration here
</details>
- type:textarea
id:etcdDebugInformation
attributes:
label:Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
value:|
<details>
```console
$ etcdctl member list -w table
# paste output here
$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here
```
</details>
- type:textarea
id:logs
attributes:
label:Relevant log output
description:Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
- label:I have reviewed the [community membership guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/community-membership.md)
required:true
- label:The members are actively contributing to 1 or more etcd subprojects
required:true
- label:The members are being sponsored by two current reviewers or a current maintainer.
required:true
- type:textarea
id:rationale
attributes:
label:How do the new members meet the regular active contribution requirements?
# Configuration for probot-stale - https://github.com/probot/stale
# Number of days of inactivity before an Issue or Pull Request becomes stale
@ -12,8 +13,7 @@ onlyLabels: []
# Issues or Pull Requests with these labels will never be considered stale. Set to `[]` to disable
exemptLabels:
- "area/security"
- "Investigating"
- "stage/tracked"
# Set to true to ignore issues in a project (defaults to false)
exemptProjects:false
@ -28,11 +28,7 @@ exemptAssignees: false
staleLabel:stale
# Comment to post when marking as stale. Set to `false` to disable
markComment:>
This issue has been automatically marked as stale because it has not had
recent activity. It will be closed after 21 days if no further activity
occurs. Thank you for your contributions.
markComment:This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.
SSH into the instance, and install the github action self-hosted runner (see [install scripts](https://github.com/etcd-io/etcd/settings/actions/runners/new?arch=arm64&os=linux)).
### Step 3. Configure github action on the host
SSH into the instance, and configure the github action self-hosted runner.
First, we need disable ICU install (see [actions/runner issue on ARM64](https://github.com/actions/runner/issues/629)):
Previous change logs can be found at [CHANGELOG-3.2](https://github.com/etcd-io/etcd/blob/main/CHANGELOG-3.2.md).
The minimum recommended etcd versions to run in **production** are 3.2.28+, 3.3.18+, and 3.4.2+.
Previous change logs can be found at [CHANGELOG-3.2](https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.2.md).
<hr>
## v3.3.27 (2021-10-15)
## v3.3.25 (2020 TBD)
See [code changes](https://github.com/etcd-io/etcd/compare/v3.3.26...v3.3.27) and [v3.3 upgrade guide](https://etcd.io/docs/latest/upgrades/upgrade_3_3/) for any breaking changes.
### Other
- Updated [base image](https://github.com/etcd-io/etcd/pull/13386) from `debian:buster-v1.4.0` to `debian:bullseye-20210927` to fix the following critical CVEs:
- [CVE-2021-3711](https://nvd.nist.gov/vuln/detail/CVE-2021-3711): miscalculation of a buffer size in openssl's SM2 decryption
- [CVE-2021-35942](https://nvd.nist.gov/vuln/detail/CVE-2021-35942): integer overflow flaw in glibc
- [CVE-2019-9893](https://nvd.nist.gov/vuln/detail/CVE-2019-9893): incorrect syscall argument generation in libseccomp
- [CVE-2021-36159](https://nvd.nist.gov/vuln/detail/CVE-2021-36159): libfetch in apk-tools mishandles numeric strings in FTP and HTTP protocols to allow out of bound reads.
<hr>
## v3.3.26 (2021-10-03)
See [code changes](https://github.com/etcd-io/etcd/compare/v3.3.25...v3.3.26) and [v3.3 upgrade guide](https://etcd.io/docs/latest/upgrades/upgrade_3_3/) for any breaking changes.
### Package `clientv3`
- Fix [auth token invalid after watch reconnects](https://github.com/etcd-io/etcd/pull/12264). Get AuthToken automatically when clientConn is ready.
### Package `fileutil`
- Fix [constant](https://github.com/etcd-io/etcd/pull/12440) for linux locking.
### Go
- Compile with [*Go 1.12.17*](https://golang.org/doc/devel/release.html#go1.12).
<hr>
## v3.3.25 (2020-08-24)
See [code changes](https://github.com/etcd-io/etcd/compare/v3.3.23...v3.3.25) and [v3.3 upgrade guide](https://etcd.io/docs/latest/upgrades/upgrade_3_3/) for any breaking changes.
@ -730,7 +757,7 @@ See [code changes](https://github.com/etcd-io/etcd/compare/v3.3.2...v3.3.3) and
- For every compaction period or 1-hour, compactor uses the last revision that was fetched before compaction period, to discard historical data.
- The retention window of compaction period moves for every given compaction period or hour.
- For instance, when hourly writes are 100 and `etcd --auto-compaction-mode=periodic --auto-compaction-retention=24h`, `v3.2.x`, `v3.3.0`, `v3.3.1`, and `v3.3.2` compact revision 2400, 2640, and 2880 for every 2.4-hour, while `v3.3.3`*or later* compacts revision 2400, 2500, 2600 for every 1-hour.
- Futhermore, when `etcd --auto-compaction-mode=periodic --auto-compaction-retention=30m` and writes per minute are about 1000, `v3.3.0`, `v3.3.1`, and `v3.3.2` compact revision 30000, 33000, and 36000, for every 3-minute, while `v3.3.3`*or later* compacts revision 30000, 60000, and 90000, for every 30-minute.
- Furthermore, when `etcd --auto-compaction-mode=periodic --auto-compaction-retention=30m` and writes per minute are about 1000, `v3.3.0`, `v3.3.1`, and `v3.3.2` compact revision 30000, 33000, and 36000, for every 3-minute, while `v3.3.3`*or later* compacts revision 30000, 60000, and 90000, for every 30-minute.
### Metrics, Monitoring
@ -933,7 +960,7 @@ See [security doc](https://etcd.io/docs/latest/op-guide/security/) for more deta
- Periodic compactor continues to record latest revisions for every 1/10 of given compaction period (e.g. 1-hour when `etcd --auto-compaction-mode=periodic --auto-compaction-retention=10h`).
- For every 1/10 of given compaction period, compactor uses the last revision that was fetched before compaction period, to discard historical data.
- The retention window of compaction period moves for every 1/10 of given compaction period.
- For instance, when hourly writes are 100 and `--auto-compaction-retention=10`, v3.1 compacts revision 1000, 2000, and 3000 for every 10-hour, while v3.2.x, v3.3.0, v3.3.1, and v3.3.2 compact revision 1000, 1100, and 1200 for every 1-hour. Futhermore, when writes per minute are 1000, v3.3.0, v3.3.1, and v3.3.2 with `--auto-compaction-mode=periodic --auto-compaction-retention=30m` compact revision 30000, 33000, and 36000, for every 3-minute with more finer granularity.
- For instance, when hourly writes are 100 and `--auto-compaction-retention=10`, v3.1 compacts revision 1000, 2000, and 3000 for every 10-hour, while v3.2.x, v3.3.0, v3.3.1, and v3.3.2 compact revision 1000, 1100, and 1200 for every 1-hour. Furthermore, when writes per minute are 1000, v3.3.0, v3.3.1, and v3.3.2 with `--auto-compaction-mode=periodic --auto-compaction-retention=30m` compact revision 30000, 33000, and 36000, for every 3-minute with more finer granularity.
- Whether compaction succeeds or not, this process repeats for every 1/10 of given compaction period. If compaction succeeds, it just removes compacted revision from historical revision records.
- Serve [`/health` endpoint as unhealthy](https://github.com/etcd-io/etcd/pull/8272) when [alarm (e.g. `NOSPACE`) is raised or there's no leader](https://github.com/etcd-io/etcd/issues/8207).
- [Replace balancer with upstream grpc solution](https://github.com/etcd-io/etcd/pull/16844).
- Fix [race condition when accessing cfg.Endpoints in dial()](https://github.com/etcd-io/etcd/pull/16857).
- Fix [invalid authority header issue in single endpoint scenario](https://github.com/etcd-io/etcd/pull/16988).
### Dependencies
- Compile binaries using [go 1.20.11](https://github.com/etcd-io/etcd/pull/16916).
- Upgrade [bbolt to 1.3.8](https://github.com/etcd-io/etcd/pull/16834).
- Upgrade gRPC to 1.58.3 in https://github.com/etcd-io/etcd/pull/16997 and https://github.com/etcd-io/etcd/pull/16999. Note that gRPC server will reject requests with connection header (refer to https://github.com/grpc/grpc-go/pull/4803).
<hr>
## v3.4.27 (2023-07-11)
### etcd server
- Fix [corruption check may get a `ErrCompacted` error when server has just been compacted](https://github.com/etcd-io/etcd/pull/16047)
- Improve [Lease put performance for the case that auth is disabled or the user is admin](https://github.com/etcd-io/etcd/pull/16020)
- Fix [embed: nil pointer dereference when stopServer](https://github.com/etcd-io/etcd/pull/16195)
### etcdctl v3
- Add [optional --bump-revision and --mark-compacted flag to etcdctl snapshot restore operation](https://github.com/etcd-io/etcd/pull/16193).
### Dependencies
- Compile binaries using [go 1.19.10](https://github.com/etcd-io/etcd/pull/16038).
<hr>
## v3.4.26 (2023-05-12)
### etcd server
- Fix [LeaseTimeToLive API may return keys to clients which have no read permission on the keys](https://github.com/etcd-io/etcd/pull/15814).
The minimum recommended etcd versions to run in **production** are 3.2.28+, 3.3.18+, and 3.4.2+.
### Dependencies
- Compile binaries using [go 1.19.9](https://github.com/etcd-io/etcd/pull/15823)
<hr>
## v3.4.25 (2023-04-14)
### etcd server
- Add [`etcd --tls-min-version --tls-max-version`](https://github.com/etcd-io/etcd/pull/15486) to enable support for TLS 1.3.
- Add [`etcd --listen-client-http-urls`](https://github.com/etcd-io/etcd/pull/15620) flag to support separating http server from grpc one, thus giving full immunity to [watch stream starvation under high read load](https://github.com/etcd-io/etcd/issues/15402).
- Change [http2 frame scheduler to random algorithm](https://github.com/etcd-io/etcd/pull/15478)
- Fix [server/embed: fix data race when starting both secure & insecure gRPC servers on the same address](https://github.com/etcd-io/etcd/pull/15518)
- Fix [wsproxy did not print log in JSON format](https://github.com/etcd-io/etcd/pull/15662).
- Fix [CVE-2021-28235](https://nvd.nist.gov/vuln/detail/CVE-2021-28235) by [clearing password after authenticating the user](https://github.com/etcd-io/etcd/pull/15655).
- Fix [etcdserver may panic when parsing a JWT token without username or revision](https://github.com/etcd-io/etcd/pull/15677).
- Fix [Watch response traveling back in time when reconnecting member downloads snapshot from the leader](https://github.com/etcd-io/etcd/pull/15520).
- Fix [Requested watcher progress notifications are not synchronised with stream](https://github.com/etcd-io/etcd/pull/15697).
### Package `clientv3`
- Reverted the fix to [auth invalid token and old revision errors in watch](https://github.com/etcd-io/etcd/pull/15542).
- Compile binaries using [Go 1.19.8](https://github.com/etcd-io/etcd/pull/15652).
- Upgrade [golang.org/x/net to v0.7.0](https://github.com/etcd-io/etcd/pull/15333).
### Docker image
- Fix [etcd docker images all tagged with amd64 architecture](https://github.com/etcd-io/etcd/pull/15681)
<hr>
## v3.4.24 (2023-02-16)
### etcd server
- Fix [etcdserver might promote a non-started learner](https://github.com/etcd-io/etcd/pull/15097).
- Improve [mvcc: reduce count-only range overhead](https://github.com/etcd-io/etcd/pull/15099)
- Improve [mvcc: push down RangeOptions.limit argv into index tree to reduce memory overhead](https://github.com/etcd-io/etcd/pull/15137)
- Improve [server: set multiple concurrentReadTx instances share one txReadBuffer](https://github.com/etcd-io/etcd/pull/15195)
- Fix [aligning zap log timestamp resolution to microseconds](https://github.com/etcd-io/etcd/pull/15241). Etcd now uses zap timestamp format: `2006-01-02T15:04:05.999999Z0700` (microsecond instead of milliseconds precision).
- Fix [consistently format IPv6 addresses for comparison](https://github.com/etcd-io/etcd/pull/15188)
### Package `clientv3`
- Fix [etcd might send duplicated events to watch clients](https://github.com/etcd-io/etcd/pull/15275).
### Dependencies
- Upgrade [bbolt to v1.3.7](https://github.com/etcd-io/etcd/pull/15223).
- Upgrade [github.com/grpc-ecosystem/grpc-gateway](https://github.com/grpc-ecosystem/grpc-gateway/releases) from [v1.9.5](https://github.com/grpc-ecosystem/grpc-gateway/releases/tag/v1.9.5) to [v1.11.0](https://github.com/grpc-ecosystem/grpc-gateway/releases/tag/v1.11.0).
### Docker image
- Updated [base image from base-debian11 to static-debian11 and removed dependency on busybox](https://github.com/etcd-io/etcd/pull/15038).
<hr>
## v3.4.23 (2022-12-21)
### etcd server
- Fix [Remove memberID from data corrupt alarm](https://github.com/etcd-io/etcd/pull/14853).
- Fix [nil pointer panic for readonly txn due to nil response](https://github.com/etcd-io/etcd/pull/14900).
- Bumped [some dependencies](https://github.com/etcd-io/etcd/pull/15019) to address some HIGH Vulnerabilities.
### Package `clientv3`
- Fix [Refreshing token on CommonName based authentication causes segmentation violation in client](https://github.com/etcd-io/etcd/pull/14792).
- Compile binaries using [Go 1.17.13](https://github.com/etcd-io/etcd/pull/15019).
### Docker image
- Use [distroless base image](https://github.com/etcd-io/etcd/pull/15017) to address critical Vulnerabilities.
<hr>
## v3.4.22 (2022-11-02)
### etcd server
- Fix [memberID equals zero in corruption alarm](https://github.com/etcd-io/etcd/pull/14530)
- Fix [auth invalid token and old revision errors in watch](https://github.com/etcd-io/etcd/pull/14548)
- Fix [avoid closing a watch with ID 0 incorrectly](https://github.com/etcd-io/etcd/pull/14562)
- Fix [auth: fix data consistency issue caused by recovery from snapshot](https://github.com/etcd-io/etcd/pull/14649)
### Package `netutil`
- Fix [netutil: add url comparison without resolver to URLStringsEqual](https://github.com/etcd-io/etcd/pull/14577)
### Package `clientv3`
- Fix [Add backoff before retry when watch stream returns unavailable](https://github.com/etcd-io/etcd/pull/14581).
### etcd grpc-proxy
- Add [`etcd grpc-proxy start --listen-cipher-suites`](https://github.com/etcd-io/etcd/pull/14601) flag to support adding configurable cipher list.
<hr>
## v3.4.21 (2022-09-15)
### etcd server
- Fix [Durability API guarantee broken in single node cluster](https://github.com/etcd-io/etcd/pull/14423)
- Fix [Panic due to nil log object](https://github.com/etcd-io/etcd/pull/14420)
- Fix [authentication data not loaded on member startup](https://github.com/etcd-io/etcd/pull/14410)
### etcdctl v3
- Fix [etcdctl move-leader may fail for multiple endpoints](https://github.com/etcd-io/etcd/pull/14441)
<hr>
## v3.4.20 (2022-08-06)
### Package `clientv3`
- Fix [filter learners members during autosync](https://github.com/etcd-io/etcd/pull/14236).
### etcd server
- Add [`etcd --max-concurrent-streams`](https://github.com/etcd-io/etcd/pull/14251) flag to configure the max concurrent streams each client can open at a time, and defaults to math.MaxUint32.
- Add [`etcd --experimental-enable-lease-checkpoint-persist`](https://github.com/etcd-io/etcd/pull/14253) flag to enable checkpoint persisting.
- Fix [Lease checkpoints don't prevent to reset ttl on leader change](https://github.com/etcd-io/etcd/pull/14253), requires enabling checkpoint persisting.
- Fix [Protect rangePermCache with a RW lock correctly](https://github.com/etcd-io/etcd/pull/14230)
- Fix [raft: postpone MsgReadIndex until first commit in the term](https://github.com/etcd-io/etcd/pull/14258)
- Fix [etcdserver: resend ReadIndex request on empty apply request](https://github.com/etcd-io/etcd/pull/14269)
- Fix [remove temp files in snap dir when etcdserver starting](https://github.com/etcd-io/etcd/pull/14246)
- Fix [Etcdserver is still in progress of processing LeaseGrantRequest when it receives a LeaseKeepAliveRequest on the same leaseID](https://github.com/etcd-io/etcd/pull/14177)
- Fix [Grant lease with negative ID can possibly cause db out of sync](https://github.com/etcd-io/etcd/pull/14239)
- Fix [Allow non mutating requests pass through quotaKVServer when NOSPACE](https://github.com/etcd-io/etcd/pull/14254)
<hr>
## v3.4.19 (2022-07-12)
See [code changes](https://github.com/etcd-io/etcd/compare/v3.4.18...v3.4.19) and [v3.4 upgrade guide](https://etcd.io/docs/latest/upgrades/upgrade_3_4/) for any breaking changes.
### etcd server
- Fix [exclude the same alarm type activated by multiple peers](https://github.com/etcd-io/etcd/pull/13475).
- Fix [lease leak issue due to tokenProvider isn't enabled when restoring auth store from a snapshot](https://github.com/etcd-io/etcd/pull/13206).
- Fix [the race condition between goroutine and channel on the same leases to be revoked](https://github.com/etcd-io/etcd/pull/14150).
- Fix [lessor may continue to schedule checkpoint after stepping down leader role](https://github.com/etcd-io/etcd/pull/14150).
### Package `clientv3`
- Fix [a bug of not refreshing expired tokens](https://github.com/etcd-io/etcd/pull/13999).
### Dependency
- Upgrade [go.etcd.io/bbolt](https://github.com/etcd-io/bbolt/releases) from [v1.3.3](https://github.com/etcd-io/bbolt/releases/tag/v1.3.3) to [v1.3.6](https://github.com/etcd-io/bbolt/releases/tag/v1.3.6).
### Security
- Upgrade [golang.org/x/crypto](https://github.com/etcd-io/etcd/pull/14179) to v0.0.0-20220411220226-7b82a4e95df4 to address [CVE-2022-27191 ](https://github.com/advisories/GHSA-8c26-wmh5-6g9v).
- Upgrade [gopkg.in/yaml.v2](https://github.com/etcd-io/etcd/pull/14192) to v2.4.0 to address [CVE-2019-11254](https://github.com/advisories/GHSA-wxc4-f4m6-wwqv).
- Compile with [Go 1.16+](https://go.dev/doc/devel/release#go1.16).
- etcd uses [go modules](https://github.com/etcd-io/etcd/pull/14136) (instead of vendor dir) to track dependencies.
<hr>
## v3.4.18 (2021-10-15)
See [code changes](https://github.com/etcd-io/etcd/compare/v3.4.17...v3.4.18) and [v3.4 upgrade guide](https://etcd.io/docs/latest/upgrades/upgrade_3_4/) for any breaking changes.
### Metrics, Monitoring
See [List of metrics](https://etcd.io/docs/latest/metrics/) for all metrics per release.
- Updated [base image](https://github.com/etcd-io/etcd/pull/13386) from `debian:buster-v1.4.0` to `debian:bullseye-20210927` to fix the following critical CVEs:
- [CVE-2021-3711](https://nvd.nist.gov/vuln/detail/CVE-2021-3711): miscalculation of a buffer size in openssl's SM2 decryption
- [CVE-2021-35942](https://nvd.nist.gov/vuln/detail/CVE-2021-35942): integer overflow flaw in glibc
- [CVE-2019-9893](https://nvd.nist.gov/vuln/detail/CVE-2019-9893): incorrect syscall argument generation in libseccomp
- [CVE-2021-36159](https://nvd.nist.gov/vuln/detail/CVE-2021-36159): libfetch in apk-tools mishandles numeric strings in FTP and HTTP protocols to allow out of bound reads.
<hr>
## v3.4.17 (2021-10-03)
See [code changes](https://github.com/etcd-io/etcd/compare/v3.4.16...v3.4.17) and [v3.4 upgrade guide](https://etcd.io/docs/latest/upgrades/upgrade_3_4/) for any breaking changes.
### `etcdctl`
- Fix [etcdctl check datascale command](https://github.com/etcd-io/etcd/pull/11896) to work with https endpoints.
### gRPC gateway
- Add [`MaxCallRecvMsgSize`](https://github.com/etcd-io/etcd/pull/13077) support for http client.
### Dependency
- Replace [`github.com/dgrijalva/jwt-go with github.com/golang-jwt/jwt'](https://github.com/etcd-io/etcd/pull/13378).
### Go
- Compile with [*Go 1.12.17*](https://golang.org/doc/devel/release.html#go1.12).
<hr>
@ -141,6 +400,7 @@ See [code changes](https://github.com/etcd-io/etcd/compare/v3.4.10...v3.4.11) an
### Metrics, Monitoring
- Add [`os_fd_used` and `os_fd_limit` to monitor current OS file descriptors](https://github.com/etcd-io/etcd/pull/12214).
@ -492,7 +752,7 @@ See [code changes](https://github.com/etcd-io/etcd/compare/v3.3.0...v3.4.0) and
- For every compaction period or 1-hour, compactor uses the last revision that was fetched before compaction period, to discard historical data.
- The retention window of compaction period moves for every given compaction period or hour.
- For instance, when hourly writes are 100 and `etcd --auto-compaction-mode=periodic --auto-compaction-retention=24h`, `v3.2.x`, `v3.3.0`, `v3.3.1`, and `v3.3.2` compact revision 2400, 2640, and 2880 for every 2.4-hour, while `v3.3.3`*or later* compacts revision 2400, 2500, 2600 for every 1-hour.
- Futhermore, when `etcd --auto-compaction-mode=periodic --auto-compaction-retention=30m` and writes per minute are about 1000, `v3.3.0`, `v3.3.1`, and `v3.3.2` compact revision 30000, 33000, and 36000, for every 3-minute, while `v3.3.3`*or later* compacts revision 30000, 60000, and 90000, for every 30-minute.
- Furthermore, when `etcd --auto-compaction-mode=periodic --auto-compaction-retention=30m` and writes per minute are about 1000, `v3.3.0`, `v3.3.1`, and `v3.3.2` compact revision 30000, 33000, and 36000, for every 3-minute, while `v3.3.3`*or later* compacts revision 30000, 60000, and 90000, for every 30-minute.
- Make [Lease `Lookup` non-blocking with concurrent `Grant`/`Revoke`](https://github.com/etcd-io/etcd/pull/9229).
- Make etcd server return `raft.ErrProposalDropped` on internal Raft proposal drop in [v3 applier](https://github.com/etcd-io/etcd/pull/9549) and [v2 applier](https://github.com/etcd-io/etcd/pull/9558).
@ -838,7 +1098,7 @@ See [security doc](https://etcd.io/docs/latest/op-guide/security/) for more deta
- If watch response events exceed this server-side request limit and watch request is created with `fragment` field `true`, the server will split watch events into a set of chunks, each of which is a subset of watch events below server-side request limit.
- Useful when client-side has limited bandwidths.
- For example, watch response contains 10 events, where each event is 1 MiB. And server `etcd --max-request-bytes` flag value is 1 MiB. Then, server will send 10 separate fragmented events to the client.
- For example, watch response contains 5 events, where each event is 2 MiB. And server `etcd --max-request-bytes` flag value is 1 MiB and `clientv3.Config.MaxCallRecvMsgSize` is 1 MiB. Then, server will try to send 5 separate fragmented events to the client, and the client will error with `"code = ResourceExhausted desc = grpc: received message larger than max (...)"`.
- For example, watch response contains 5 events, where each event is 2 MiB. And server `etcd --max-recv-bytes` flag value is 1 MiB and `clientv3.Config.MaxCallRecvMsgSize` is 1 MiB. Then, server will try to send 5 separate fragmented events to the client, and the client will error with `"code = ResourceExhausted desc = grpc: received message larger than max (...)"`.
- Client must implement fragmented watch event merge (which `clientv3` does in etcd v3.4).
- Add [`raftAppliedIndex` field to `etcdserverpb.StatusResponse`](https://github.com/etcd-io/etcd/pull/9176) for current Raft applied index.
- Add [`errors` field to `etcdserverpb.StatusResponse`](https://github.com/etcd-io/etcd/pull/9206) for server-side error.
@ -972,7 +1232,7 @@ Note: **v3.5 will deprecate `etcd --log-package-levels` flag for `capnslog`**; `
- Now [`(r *raft) Step` returns `raft.ErrProposalDropped`](https://github.com/etcd-io/etcd/pull/9137) if a proposal has been ignored.
- e.g. a node is removed from cluster, or [`raftpb.MsgProp` arrives at current leader while there is an ongoing leadership transfer](https://github.com/etcd-io/etcd/issues/8975).
- Improve [Raft `becomeLeader` and `stepLeader`](https://github.com/etcd-io/etcd/pull/9073) by keeping track of latest `pb.EntryConfChange` index.
- Previously record `pendingConf` boolean field scanning the entire tail of the log, which can delay hearbeat send.
- Previously record `pendingConf` boolean field scanning the entire tail of the log, which can delay heartbeat send.
- Fix [missing learner nodes on `(n *node) ApplyConfChange`](https://github.com/etcd-io/etcd/pull/9116).
- Add [`raft.Config.MaxUncommittedEntriesSize`](https://github.com/etcd-io/etcd/pull/10167) to limit the total size of the uncommitted entries in bytes.
- Once exceeded, raft returns `raft.ErrProposalDropped` error.
- Compile binaries using [go 1.20.13](https://github.com/etcd-io/etcd/pull/17275)
## v3.5.11 (2023-12-07)
### etcd server
- Fix distributed tracing by ensuring `--experimental-distributed-tracing-sampling-rate` configuration option is available to [set tracing sample rate](https://github.com/etcd-io/etcd/pull/16951).
- Fix [url redirects while checking peer urls during new member addition](https://github.com/etcd-io/etcd/pull/16986)
### Dependencies
- Compile binaries using [go 1.20.12](https://github.com/etcd-io/etcd/pull/17077)
- Fix [CVE-2023-47108](https://github.com/advisories/GHSA-8pgv-569h-w5rw) by [bumping go.opentelemetry.io/otel to 1.20.0 and go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc to 0.46.0](https://github.com/etcd-io/etcd/pull/16946).
<hr>
## v3.5.10 (2023-10-27)
### etcd server
- Fix [`--socket-reuse-port` and `--socket-reuse-address` not able to be set in configuration file](https://github.com/etcd-io/etcd/pull/16435).
- Fix [corruption check may get a `ErrCompacted` error when server has just been compacted](https://github.com/etcd-io/etcd/pull/16048)
- Improve [Lease put performance for the case that auth is disabled or the user is admin](https://github.com/etcd-io/etcd/pull/16019)
- Improve [Skip getting authInfo from incoming context when auth is disabled](https://github.com/etcd-io/etcd/pull/16241)
- Fix [Hash and HashKV have duplicated RESTful API](https://github.com/etcd-io/etcd/pull/16490)
### etcdutl v3
- Add [optional --bump-revision and --mark-compacted flag to etcdutl snapshot restore operation](https://github.com/etcd-io/etcd/pull/16165).
### etcdctl v3
- Add [optional --bump-revision and --mark-compacted flag to etcdctl snapshot restore operation](https://github.com/etcd-io/etcd/pull/16165).
### etcd grpc-proxy
- Fix [Memberlist results not updated when proxy node down](https://github.com/etcd-io/etcd/pull/15907).
### Package `clientv3`
- Fix [Multiple endpoints with same prefix got mixed up](https://github.com/etcd-io/etcd/pull/15939)
- Fix [Unexpected blocking when barrier waits on a nonexistent key](https://github.com/etcd-io/etcd/pull/16188)
- Fix [Reset auth token when failing to authenticate due to auth being disabled](https://github.com/etcd-io/etcd/pull/16241)
- Fix [panic in etcd validate secure endpoints](https://github.com/etcd-io/etcd/pull/16565)
### Dependencies
- Compile binaries using [go 1.20.10](https://github.com/etcd-io/etcd/pull/16745).
- Upgrade gRPC to 1.58.3 in https://github.com/etcd-io/etcd/pull/16625, https://github.com/etcd-io/etcd/pull/16781 and https://github.com/etcd-io/etcd/pull/16790. Note that gRPC server will reject requests with connection header (refer to https://github.com/grpc/grpc-go/pull/4803).
- Upgrade [bbolt to v1.3.8](https://github.com/etcd-io/etcd/pull/16833)
<hr>
## v3.5.9 (2023-05-11)
### etcd server
- Fix [LeaseTimeToLive API may return keys to clients which have no read permission on the keys](https://github.com/etcd-io/etcd/pull/15815).
### Dependencies
- Compile binaries using [go 1.19.9](https://github.com/etcd-io/etcd/pull/15822).
<hr>
## v3.5.8 (2023-04-13)
### etcd server
- Add [`etcd --tls-min-version --tls-max-version`](https://github.com/etcd-io/etcd/pull/15483) to enable support for TLS 1.3.
- Add [`etcd --listen-client-http-urls`](https://github.com/etcd-io/etcd/pull/15589) flag to support separating http server from grpc one, thus giving full immunity to [watch stream starvation under high read load](https://github.com/etcd-io/etcd/issues/15402).
- Change [http2 frame scheduler to random algorithm](https://github.com/etcd-io/etcd/pull/15452)
- Fix [Watch response traveling back in time when reconnecting member downloads snapshot from the leader](https://github.com/etcd-io/etcd/pull/15515)
- Fix [race when starting both secure & insecure gRPC servers on the same address](https://github.com/etcd-io/etcd/pull/15517)
- Fix [aligning zap log timestamp resolution to microseconds](https://github.com/etcd-io/etcd/pull/15240). Etcd now uses zap timestamp format: `2006-01-02T15:04:05.999999Z0700` (microsecond instead of milliseconds precision).
- Fix [wsproxy did not print log in JSON format](https://github.com/etcd-io/etcd/pull/15661).
- Fix [CVE-2021-28235](https://nvd.nist.gov/vuln/detail/CVE-2021-28235) by [clearing password after authenticating the user](https://github.com/etcd-io/etcd/pull/15653).
- Fix [etcdserver may panic when parsing a JWT token without username or revision](https://github.com/etcd-io/etcd/pull/15676).
- Fix [Requested watcher progress notifications are not synchronised with stream](https://github.com/etcd-io/etcd/pull/15695).
### Package `netutil`
- Fix [consistently format IPv6 addresses for comparison](https://github.com/etcd-io/etcd/pull/15187).
### Package `clientv3`
- Fix [etcd might send duplicated events to watch clients](https://github.com/etcd-io/etcd/pull/15274).
- Compile binaries using [Go 1.17.13](https://github.com/etcd-io/etcd/pull/15019)
- Bumped [some dependencies](https://github.com/etcd-io/etcd/pull/15018) to address some HIGH Vulnerabilities.
### Docker image
- Use [distroless base image](https://github.com/etcd-io/etcd/pull/15016) to address critical Vulnerabilities.
- Updated [base image from base-debian11 to static-debian11 and removed dependency on busybox](https://github.com/etcd-io/etcd/pull/15037).
<hr>
## v3.5.6 (2022-11-21)
### etcd server
- Fix [auth invalid token and old revision errors in watch](https://github.com/etcd-io/etcd/pull/14547)
- Fix [avoid closing a watch with ID 0 incorrectly](https://github.com/etcd-io/etcd/pull/14563)
- Fix [auth: fix data consistency issue caused by recovery from snapshot](https://github.com/etcd-io/etcd/pull/14648)
- Fix [revision might be inconsistency between members when etcd crashes during processing defragmentation operation](https://github.com/etcd-io/etcd/pull/14733)
- Fix [timestamp in inconsistent format](https://github.com/etcd-io/etcd/pull/14799)
- Fix [Failed resolving host due to lost DNS record](https://github.com/etcd-io/etcd/pull/14573)
### Package `clientv3`
- Fix [Add backoff before retry when watch stream returns unavailable](https://github.com/etcd-io/etcd/pull/14582).
- Fix [stack overflow error in double barrier](https://github.com/etcd-io/etcd/pull/14658)
- Fix [Refreshing token on CommonName based authentication causes segmentation violation in client](https://github.com/etcd-io/etcd/pull/14790).
### etcd grpc-proxy
- Add [`etcd grpc-proxy start --listen-cipher-suites`](https://github.com/etcd-io/etcd/pull/14500) flag to support adding configurable cipher list.
<hr>
## v3.5.5 (2022-09-15)
### Deprecations
- Deprecated [SetKeepAlive and SetKeepAlivePeriod in limitListenerConn](https://github.com/etcd-io/etcd/pull/14366).
### Package `clientv3`
- Fix [do not overwrite authTokenBundle on dial](https://github.com/etcd-io/etcd/pull/14132).
- Fix [IsOptsWithPrefix returns false even if WithPrefix() is included](https://github.com/etcd-io/etcd/pull/14187).
### etcd server
- [Build official darwin/arm64 artifacts](https://github.com/etcd-io/etcd/pull/14436).
- Add [`etcd --max-concurrent-streams`](https://github.com/etcd-io/etcd/pull/14219) flag to configure the max concurrent streams each client can open at a time, and defaults to math.MaxUint32.
- Add [`etcd --experimental-compact-hash-check-enabled --experimental-compact-hash-check-time`](https://github.com/etcd-io/etcd/issues/14039) flags to support enabling reliable corruption detection on compacted revisions.
- Fix [unexpected error during txn](https://github.com/etcd-io/etcd/issues/14110).
- Fix [lease leak issue due to tokenProvider isn't enabled when restoring auth store from a snapshot](https://github.com/etcd-io/etcd/pull/13205).
- Fix [the race condition between goroutine and channel on the same leases to be revoked](https://github.com/etcd-io/etcd/pull/14087).
- Fix [lessor may continue to schedule checkpoint after stepping down leader role](https://github.com/etcd-io/etcd/pull/14087).
- Fix [Restrict the max size of each WAL entry to the remaining size of the WAL file](https://github.com/etcd-io/etcd/pull/14127).
- Fix [Protect rangePermCache with a RW lock correctly](https://github.com/etcd-io/etcd/pull/14227)
- Fix [memberID equals zero in corruption alarm](https://github.com/etcd-io/etcd/pull/14272)
- Fix [Durability API guarantee broken in single node cluster](https://github.com/etcd-io/etcd/pull/14424)
- Fix [etcd fails to start after performing alarm list operation and then power off/on](https://github.com/etcd-io/etcd/pull/14429)
- Fix [authentication data not loaded on member startup](https://github.com/etcd-io/etcd/pull/14409)
### etcdctl v3
- Fix [etcdctl move-leader may fail for multiple endpoints](https://github.com/etcd-io/etcd/pull/14434)
The minimum recommended etcd versions to run in **production** are 3.2.28+, 3.3.18+, and 3.4.2+.
### Other
- [Bump golang.org/x/crypto to latest version](https://github.com/etcd-io/etcd/pull/13996) to address [CVE-2022-27191](https://github.com/advisories/GHSA-8c26-wmh5-6g9v).
- [Bump OpenTelemetry to 1.0.1 and gRPC to 1.41.0](https://github.com/etcd-io/etcd/pull/14312).
<hr>
## v3.5.4 (2022-04-24)
### etcd server
- Fix [etcd panic on startup (auth enabled)](https://github.com/etcd-io/etcd/pull/13946)
### package `client/pkg/v3`
- [Revert the change of trimming the trailing dot from SRV.Target](https://github.com/etcd-io/etcd/pull/13950) returned by DNS lookup
<hr>
## v3.5.3 (2022-04-13)
### etcd server
- Fix [Provide a better liveness probe for when etcd runs as a Kubernetes pod](https://github.com/etcd-io/etcd/pull/13706)
- Fix [Inconsistent revision and data occurs](https://github.com/etcd-io/etcd/pull/13908)
- Fix [Etcdserver is still in progress of processing LeaseGrantRequest when it receives a LeaseKeepAliveRequest on the same leaseID](https://github.com/etcd-io/etcd/pull/13932)
- Fix [consistent_index coming from snapshot is overwritten by the old local value](https://github.com/etcd-io/etcd/pull/13933)
- [Update container base image snapshot](https://github.com/etcd-io/etcd/pull/13862)
See [code changes](https://github.com/etcd-io/etcd/compare/v3.5.1...v3.5.2) and [v3.5 upgrade guide](https://etcd.io/docs/latest/upgrades/upgrade_3_5/) for any breaking changes.
### etcd server
- Fix [exclude the same alarm type activated by multiple peers](https://github.com/etcd-io/etcd/pull/13476).
- Add [`etcd --experimental-enable-lease-checkpoint-persist`](https://github.com/etcd-io/etcd/pull/13508) flag to enable checkpoint persisting.
- Fix [Lease checkpoints don't prevent to reset ttl on leader change](https://github.com/etcd-io/etcd/pull/13508), requires enabling checkpoint persisting.
- Fix [assertion failed due to tx closed when recovering v3 backend from a snapshot db](https://github.com/etcd-io/etcd/pull/13501)
- Fix [segmentation violation(SIGSEGV) error due to premature unlocking of watchableStore](https://github.com/etcd-io/etcd/pull/13541)
See [code changes](https://github.com/etcd-io/etcd/compare/v3.5.0...v3.5.1) and [v3.5 upgrade guide](https://etcd.io/docs/latest/upgrades/upgrade_3_5/) for any breaking changes.
### etcd server
- Fix [self-signed-cert-validity parameter cannot be specified in the config file](https://github.com/etcd-io/etcd/pull/13237).
- Fix [ensure that cluster members stored in v2store and backend are in sync](https://github.com/etcd-io/etcd/pull/13348)
- Endpoints self identify now as `etcd-endpoints://{id}/{authority}` where authority is based on first endpoint passed, for example `etcd-endpoints://0xc0009d8540/localhost:2079`
### Other
- Updated [base image](https://github.com/etcd-io/etcd/pull/13386) from `debian:buster-v1.4.0` to `debian:bullseye-20210927` to fix the following critical CVEs:
- [CVE-2021-3711](https://nvd.nist.gov/vuln/detail/CVE-2021-3711): miscalculation of a buffer size in openssl's SM2 decryption
- [CVE-2021-35942](https://nvd.nist.gov/vuln/detail/CVE-2021-35942): integer overflow flaw in glibc
- [CVE-2019-9893](https://nvd.nist.gov/vuln/detail/CVE-2019-9893): incorrect syscall argument generation in libseccomp
- [CVE-2021-36159](https://nvd.nist.gov/vuln/detail/CVE-2021-36159): libfetch in apk-tools mishandles numeric strings in FTP and HTTP protocols to allow out of bound reads.
<hr>
## v3.5.0 (2021-06)
See [code changes](https://github.com/etcd-io/etcd/compare/v3.4.0...v3.5.0) and [v3.5 upgrade guide](https://etcd.io/docs/latest/upgrades/upgrade_3_5/) for any breaking changes.
- [v3.5.0](https://github.com/etcd-io/etcd/releases/tag/v3.5.0) (2020 TBD), see [code changes](https://github.com/etcd-io/etcd/compare/v3.5.0-rc.1...v3.5.0).
- [v3.5.0-rc.1](https://github.com/etcd-io/etcd/releases/tag/v3.5.0-rc.1) (2020 TBD), see [code changes](https://github.com/etcd-io/etcd/compare/v3.5.0-rc.0...v3.5.0-rc.1).
- [v3.5.0-rc.0](https://github.com/etcd-io/etcd/releases/tag/v3.5.0-rc.0) (2020 TBD), see [code changes](https://github.com/etcd-io/etcd/compare/v3.4.0...v3.5.0-rc.0).
- [v3.5.0](https://github.com/etcd-io/etcd/releases/tag/v3.5.0) (2021 TBD), see [code changes](https://github.com/etcd-io/etcd/compare/v3.5.0-rc.1...v3.5.0).
- [v3.5.0-rc.1](https://github.com/etcd-io/etcd/releases/tag/v3.5.0-rc.1) (2021-06-10), see [code changes](https://github.com/etcd-io/etcd/compare/v3.5.0-rc.0...v3.5.0-rc.1).
- [v3.5.0-rc.0](https://github.com/etcd-io/etcd/releases/tag/v3.5.0-rc.0) (2021-06-04), see [code changes](https://github.com/etcd-io/etcd/compare/v3.5.0-beta.4...v3.5.0-rc.0).
- [v3.5.0-beta.4](https://github.com/etcd-io/etcd/releases/tag/v3.5.0-beta.4) (2021-05-26), see [code changes](https://github.com/etcd-io/etcd/compare/v3.5.0-beta.3...v3.5.0-beta.4).
- [v3.5.0-beta.3](https://github.com/etcd-io/etcd/releases/tag/v3.5.0-beta.3) (2021-05-18), see [code changes](https://github.com/etcd-io/etcd/compare/v3.5.0-beta.2...v3.5.0-beta.3).
- [v3.5.0-beta.2](https://github.com/etcd-io/etcd/releases/tag/v3.5.0-beta.2) (2021-05-18), see [code changes](https://github.com/etcd-io/etcd/compare/v3.5.0-beta.1...v3.5.0-beta.2).
- [v3.5.0-beta.1](https://github.com/etcd-io/etcd/releases/tag/v3.5.0-beta.1) (2021-05-18), see [code changes](https://github.com/etcd-io/etcd/compare/v3.4.0...v3.5.0-beta.1).
**Again, before running upgrades from any previous release, please make sure to read change logs below and [v3.5 upgrade guide](https://etcd.io/docs/latest/upgrades/upgrade_3_5/).**
@ -25,15 +269,15 @@ See [code changes](https://github.com/etcd-io/etcd/compare/v3.4.0...v3.5.0) and
- `go.etcd.io/clientv3/snapshot` SnapshotManager class have moved to `go.etcd.io/clientv3/etcdctl`.
The method `snapshot.Save` to download a snapshot from the remote server was preserved in 'go.etcd.io/clientv3/snapshot`.
- `go.etcd.io/client' package got migrated to 'go.etcd.io/client/v2'.
- Changed behavior of clienv3 API [MemberList](https://github.com/etcd-io/etcd/pull/11639).
- Changed behavior of clientv3 API [MemberList](https://github.com/etcd-io/etcd/pull/11639).
- Previously, it is directly served with server's local data, which could be stale.
- Now, it is served with linearizable guarantee. If the server is disconnected from quorum, `MemberList` call will fail.
- [gRPC gateway](https://github.com/grpc-ecosystem/grpc-gateway) only supports [`/v3`](TODO) endpoint.
- `curl -L http://localhost:2379/v3beta/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'` does work in v3.5. Use `curl -L http://localhost:2379/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'` instead.
- `curl -L http://localhost:2379/v3beta/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'` doesn't work in v3.5. Use `curl -L http://localhost:2379/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'` instead.
- **`etcd --experimental-enable-v2v3` flag remains experimental and to be deprecated.**
- v2 storage emulation feature will be deprecated in the next release.
- etcd 3.5 is the last version that supports V2 API. Flags `--enable-v2` and `--experimental-enable-v2v3` [are now deprecated](https://github.com/etcd-io/etcd/pull/) and will be removed in etcd v3.6 release.
- etcd 3.5 is the last version that supports V2 API. Flags `--enable-v2` and `--experimental-enable-v2v3` [are now deprecated](https://github.com/etcd-io/etcd/pull/12940) and will be removed in etcd v3.6 release.
- **`etcd --experimental-backend-bbolt-freelist-type` flag has been deprecated.** Use **`etcd --backend-bbolt-freelist-type`** instead. The default type is hashmap and it is stable now.
- **`etcd --debug` flag has been deprecated.** Use **`etcd --log-level=debug`** instead.
@ -87,6 +331,7 @@ See [code changes](https://github.com/etcd-io/etcd/compare/v3.4.0...v3.5.0) and
- Changed [the format of WAL entries related to auth for not keeping password as a plain text](https://github.com/etcd-io/etcd/pull/11943).
- Add third party [Security Audit Report](https://github.com/etcd-io/etcd/pull/12201).
- A [log warning](https://github.com/etcd-io/etcd/pull/12242) is added when etcd uses any existing directory that has a permission different than 700 on Linux and 777 on Windows.
- Add optional [`ClientCertFile` and `ClientKeyFile`](https://github.com/etcd-io/etcd/pull/12705) options for peer and client tls configuration when split certificates are used.
### Metrics, Monitoring
@ -107,9 +352,12 @@ Note that any `etcd_debugging_*` metrics are experimental and subject to change.
- Add [`TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256` and `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256` to `etcd --cipher-suites`](https://github.com/etcd-io/etcd/pull/11864).
- Automatically [create parent directory if it does not exist](https://github.com/etcd-io/etcd/pull/9626) (fix [issue#9609](https://github.com/etcd-io/etcd/issues/9609)).
- v4.0 will configure `etcd --enable-v2=true --enable-v2v3=/aaa` to enable v2 API server that is backed by **v3 storage**.
@ -117,6 +365,7 @@ Note that any `etcd_debugging_*` metrics are experimental and subject to change.
- `etcd --experimental-backend-bbolt-freelist-type` has been deprecated.
- Support [downgrade API](https://github.com/etcd-io/etcd/pull/11715).
- Deprecate v2 apply on cluster version. [Use v3 request to set cluster version and recover cluster version from v3 backend](https://github.com/etcd-io/etcd/pull/11427).
- [Use v2 api to update cluster version to support mixed version cluster during upgrade](https://github.com/etcd-io/etcd/pull/12988).
- [Fix corruption bug in defrag](https://github.com/etcd-io/etcd/pull/11613).
- Fix [quorum protection logic when promoting a learner](https://github.com/etcd-io/etcd/pull/11640).
- Improve [peer corruption checker](https://github.com/etcd-io/etcd/pull/11621) to work when peer mTLS is enabled.
@ -159,13 +408,14 @@ Note that any `etcd_debugging_*` metrics are experimental and subject to change.
- Add [`etcd --socket-reuse-address`](https://github.com/etcd-io/etcd/pull/12702) flag
- Setting this flag enables `SO_REUSEADDR` which allows binding to an address in `TIME_WAIT` state, improving etcd restart time.
- Reduce [around 30% memory allocation by logging range response size without marshal](https://github.com/etcd-io/etcd/pull/12871).
- `ETCD_VERIFY="all"` environment triggers [additional verification of consistency](https://github.com/etcd-io/etcd/pull/) of etcd data-dir files.
- `ETCD_VERIFY="all"` environment triggers [additional verification of consistency](https://github.com/etcd-io/etcd/pull/12901) of etcd data-dir files.
- Add [`etcd --enable-log-rotation`](https://github.com/etcd-io/etcd/pull/12774) boolean flag which enables log rotation if true.
- Add [`etcd --log-rotation-config-json`](https://github.com/etcd-io/etcd/pull/12774) flag which allows passthrough of JSON config to configure log rotation for a file output target.
- Add experimental distributed tracing boolean flag [`--experimental-enable-distributed-tracing`](https://github.com/etcd-io/etcd/pull/12919) which enables tracing.
- Add [`etcd --experimental-distributed-tracing-address`](https://github.com/etcd-io/etcd/pull/12919) string flag which allows configuring the OpenTelemetry collector address.
- Add [`etcd --experimental-distributed-tracing-service-name`](https://github.com/etcd-io/etcd/pull/12919) string flag which allows changing the default "etcd" service name.
- Add [`etcd --experimental-distributed-tracing-instance-id`](https://github.com/etcd-io/etcd/pull/12919) string flag which configures an instance ID, which must be unique per etcd instance.
- Add [`--experimental-bootstrap-defrag-threshold-megabytes`](https://github.com/etcd-io/etcd/pull/12941) which configures a threshold for the unused db size and etcdserver will automatically perform defragmentation on bootstrap when it exceeds this value. The functionality is disabled if the value is 0.
### Package `runtime`
@ -226,13 +476,14 @@ Note that any `etcd_debugging_*` metrics are experimental and subject to change.
- Add [`etcdctl member list -w=json --hex`](https://github.com/etcd-io/etcd/pull/11812) flag to print memberListResponse in hex format json.
- Changed [`etcdctl lock <lockname> exec-command`](https://github.com/etcd-io/etcd/pull/12829) to return exit code of exec-command.
- `curl -L http://localhost:2379/v3beta/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'` does work in v3.5. Use `curl -L http://localhost:2379/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'` instead.
- Set [`enable-grpc-gateway`](https://github.com/etcd-io/etcd/pull/12297) flag to true when using a config file to keep the defaults the same as the command line configuration.
### gRPC Proxy
@ -249,7 +500,8 @@ Note that any `etcd_debugging_*` metrics are experimental and subject to change.
- [Fix a data corruption bug by saving consistent index](https://github.com/etcd-io/etcd/pull/11652).
Previous change logs can be found at [CHANGELOG-3.5](https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md).
<hr>
## v3.6.0 (TBD)
See [code changes](https://github.com/etcd-io/etcd/compare/v3.5.0...v3.6.0).
### Breaking Changes
-`etcd` will no longer start on data dir created by newer versions (for example etcd v3.6 will not run on v3.7+ data dir). To downgrade data dir please check out `etcdutl migrate` command.
-`etcd` doesn't support serving client requests on the peer listen endpoints (--listen-peer-urls). See [pull/13565](https://github.com/etcd-io/etcd/pull/13565).
-`etcdctl` will sleep(2s) in case of range delete without `--range` flag. See [pull/13747](https://github.com/etcd-io/etcd/pull/13747)
- Applications which depend on etcd v3.6 packages must be built with go version >= v1.18.
- Package `mvcc/backend` was moved to `storage/backend`
- Package `mvcc/buckets` was moved to `storage/schema`
- Package `wal` was moved to `storage/wal`
- Package `datadir` was moved to `storage/datadir`
### Package `raft`
- [Decouple raft from etcd](https://github.com/etcd-io/etcd/issues/14713). Migrated raft to a separate [repository](https://github.com/etcd-io/raft), and renamed raft module to `go.etcd.io/raft/v3`.
### etcd server
- Add [`etcd --log-format`](https://github.com/etcd-io/etcd/pull/13339) flag to support log format.
- Add [`etcd --experimental-max-learners`](https://github.com/etcd-io/etcd/pull/13377) flag to allow configuration of learner max membership.
- Add [`etcd --experimental-enable-lease-checkpoint-persist`](https://github.com/etcd-io/etcd/pull/13508) flag to handle upgrade from v3.5.2 clusters with this feature enabled.
- Add [`etcdctl make-mirror --rev`](https://github.com/etcd-io/etcd/pull/13519) flag to support incremental mirror.
- Add [v3 discovery](https://github.com/etcd-io/etcd/pull/13635) to bootstrap a new etcd cluster.
- Add [field `storage`](https://github.com/etcd-io/etcd/pull/13772) into the response body of endpoint `/version`.
- Add [`etcd --max-concurrent-streams`](https://github.com/etcd-io/etcd/pull/14169) flag to configure the max concurrent streams each client can open at a time, and defaults to math.MaxUint32.
- Add [`etcd grpc-proxy --experimental-enable-grpc-logging`](https://github.com/etcd-io/etcd/pull/14266) flag to logging all grpc requests and responses.
- Add [`etcd --experimental-compact-hash-check-enabled --experimental-compact-hash-check-time`](https://github.com/etcd-io/etcd/issues/14039) flags to support enabling reliable corruption detection on compacted revisions.
- Add [Protection on maintenance request when auth is enabled](https://github.com/etcd-io/etcd/pull/14663).
- Graduated [`--experimental-warning-unary-request-duration` to `--warning-unary-request-duration`](https://github.com/etcd-io/etcd/pull/14414). Note the experimental flag is deprecated and will be decommissioned in v3.7.
- Add [field `hash_revision` into `HashKVResponse`](https://github.com/etcd-io/etcd/pull/14537).
- Add [`etcd --experimental-snapshot-catch-up-entries`](https://github.com/etcd-io/etcd/pull/15033) flag to configure number of entries for a slow follower to catch up after compacting the raft storage entries and defaults to 5k.
- Decreased [`--snapshot-count` default value from 100,000 to 10,000](https://github.com/etcd-io/etcd/pull/15408)
- Add [`etcd --tls-min-version --tls-max-version`](https://github.com/etcd-io/etcd/pull/15156) to enable support for TLS 1.3.
- Add [`etcd --experimental-stop-grpc-service-on-defrag`](https://github.com/etcd-io/etcd/pull/16278) to enable client failover on defrag.
### etcd grpc-proxy
- Add [`etcd grpc-proxy start --endpoints-auto-sync-interval`](https://github.com/etcd-io/etcd/pull/14354) flag to enable and configure interval of auto sync of endpoints with server.
- Add [`etcd grpc-proxy start --listen-cipher-suites`](https://github.com/etcd-io/etcd/pull/14308) flag to support adding configurable cipher list.
- Compile with [Go 1.21+](https://go.dev/doc/devel/release#go1.21.minor). Please refer to [gc-guide](https://go.dev/doc/gc-guide) to configure `GOGC` and `GOMEMLIMIT` properly.
### Other
- Use Distroless as base image to make the image less vulnerable and reduce image size.
- [Upgrade grpc-gateway from v1 to v2](https://github.com/etcd-io/etcd/pull/16595).
Previous change logs can be found at [CHANGELOG-3.x](https://github.com/etcd-io/etcd/blob/main/CHANGELOG-3.x.md).
The minimum recommended etcd versions to run in **production** are 3.2.28+, 3.3.18+, and 3.4.2+.
Previous change logs can be found at [CHANGELOG-3.x](https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.x.md).
<hr>
## v4.0.0 (TBD)
See [code changes](https://github.com/etcd-io/etcd/compare/v3.5.0...v4.0.0) and [v4.0 upgrade guide](https://etcd.io/docs/latest/upgrades/upgrade_4_0/) for any breaking changes.
@ -18,10 +13,6 @@ See [code changes](https://github.com/etcd-io/etcd/compare/v3.5.0...v4.0.0) and
### Breaking Changes
- [Secure etcd by default](https://github.com/etcd-io/etcd/issues/9475)?
- Change `/health` endpoint output.
- Previously, `{"health":"true"}`.
- Now, `{"health":true}`.
- Breaks [Kubernetes `kubectl get componentstatuses` command](https://github.com/kubernetes/kubernetes/issues/58240).
- Deprecate [`etcd --proxy*`](TODO) flags; **no more v2 proxy**.
- Deprecate [v2 storage backend](https://github.com/etcd-io/etcd/issues/9232); **no more v2 store**.
- v2 API is still supported via [v2 emulation](TODO).
The minimum recommended etcd versions to run in **production** are v3.4.22+ and v3.5.6+. Refer to the [versioning policy](https://etcd.io/docs/v3.5/op-guide/versioning/) for more details.
### v3.5 data corruption issue
Running etcd v3.5.2, v3.5.1 and v3.5.0 under high load can cause a data corruption issue.
If etcd process is killed, occasionally some committed transactions are not reflected on all the members.
Recommendation is to upgrade to v3.5.4+.
If you have encountered data corruption, please follow instructions on https://etcd.io/docs/v3.5/op-guide/data_corruption/.
## Change log rules
1. Each patch release only includes changes against previous patch release.
For example, the change log of v3.5.5 should only include items which are new to v3.5.4.
2. For the first release (e.g. 3.4.0, 3.5.0, 3.6.0, 4.0.0 etc.) for each minor or major
version, it only includes changes which are new to the first release of previous minor
or major version. For example, v3.5.0 should only include items which are new to v3.4.0,
and v3.6.0 should only include items which are new to v3.5.0.
etcd is Apache 2.0 licensed and accepts contributions via GitHub pull requests. This document outlines some of the conventions on commit message formatting, contact points for developers, and other resources to help get contributions into etcd.
Reporting bugs is one of the best ways to contribute. However, a good bug report has some very specific qualities, so please read over our short document on [reporting bugs](https://etcd.io/docs/latest/reporting_bugs) before submitting a bug report. This document might contain links to known issues, another good reason to take a look there before reporting a bug.
## Contribution flow
etcd is Apache 2.0 licensed and accepts contributions via GitHub pull requests.
This document outlines basics of contributing to etcd.
This is a rough outline of what a contributor's workflow looks like:
* [Find something to work on](#Find-something-to-work-on)
* [Setup development environment](#Setup-development-environment)
* [Implement your change](#Implement-your-change)
* [Commit your change](#Commit-your-change)
* [Create a pull request](#Create-a-pull-request)
* [Get your pull request reviewed](#Get-your-pull-request-reviewed)
- Create a topic branch from where to base the contribution. This is usually main.
- Make commits of logical units.
- Make sure commit messages are in the proper format (see below).
- Push changes in a topic branch to a personal fork of the repository.
- Submit a pull request to etcd-io/etcd.
- The PR must receive a LGTM from two maintainers found in the MAINTAINERS file.
If you have any questions about, please reach out using one of the methods listed in [contact].
Thanks for contributing!
[contact]: ./README.md#Contact
### Code style
## Learn more about etcd
The coding style suggested by the Golang community is used in etcd. See the [style doc](https://github.com/golang/go/wiki/CodeReviewComments) for details.
Before making a change please look through resources below to learn more about etcd and tools used for development.
Please follow this style to make etcd easy to review, maintain and develop.
*Please learn about [Git](https://github.com/git-guides) version control system used in etcd.
* Read the [etcd learning resources](https://etcd.io/docs/v3.5/learning/)
* Read the [etcd community membership](/Documentation/contributor-guide/community-membership.md)
* Watch [etcd deep dive](https://www.youtube.com/watch?v=D2pm6ufIt98&t=927s)
* Watch [etcd code walk through](https://www.youtube.com/watch?v=H3XaSF6wF7w)
### Format of the commit message
## Find something to work on
We follow a rough convention for commit messages that is designed to answer two
questions: what changed and why. The subject line should feature the what and
the body of the commit should describe the why.
All the work in etcd project is tracked in [github issue tracker].
Issues should be properly labeled making it easy to find something for you.
Depending on your interest and experience you should check different labels:
* If you are just starting, check issues labeled with [good first issue].
* When you feel more conformable in your contributions, checkout [help wanted].
* Advanced contributors can try to help with issues labeled [priority/important] covering most relevant work at the time.
If any of aforementioned labels don't have unassigned issues, please [contact] one of the [maintainers] asking to triage more issues.
For both options the only supported architecture is `linux-amd64`. Bug reports for other environments will generally be ignored. Supporting new environments requires introduction of proper tests and maintainer support that is currently lacking in the etcd project.
If you would like etcd to support your preferred environment you can [file an issue].
### Option 1 - Manually setup local environment
This is the original etcd development environment, is most supported and is backwards compatible for development of older etcd versions.
Follow the steps below to setup the environment:
- [Clone the repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository)
- Install Go by following [installation](https://go.dev/doc/install). Please check minimal go version in [go.mod file](./go.mod#L3).
- Install build tools:
-`make`: For debian based distributions you can run `sudo apt-get install build-essential`
-`protoc`: You can download for your os. Use version [`v3.20.3`](https://github.com/protocolbuffers/protobuf/releases/tag/v3.20.3).
-`yamllint`: For debian based distribution you can run `sudo apt-get install yamllint`
- Verify that everything is installed by running `make build`
Note: `make build` runs with `-v`. Other build flags can be added through env `GO_BUILD_FLAGS`, **if required**. Eg.,
```console
GO_BUILD_FLAGS="-buildmode=pie" make build
```
### Option 2 - Automatically setup devcontainer
This is a more recently added environmnent that aims to make it faster for new contributors to get started with etcd. This option is supported for etcd versions 3.6 onwards.
This option can be [used locally](https://code.visualstudio.com/docs/devcontainers/tutorial) on a system running Visual Studio Code and Docker, or in a remote cloud based [Codespaces](https://github.com/features/codespaces) environment.
To get started, create a codespace for this repository by clicking this 👇
[](https://github.com/codespaces/new?hide_repo_select=true&ref=main&repo=11225014)
A codespace will open in a web-based version of Visual Studio Code. The [dev container](.devcontainer/devcontainer.json) is fully configured with software needed for this project.
**Note**: Dev containers is an open spec which is supported by [GitHub Codespaces](https://github.com/codespaces) and [other tools](https://containers.dev/supporting).
[file an issue]: https://github.com/etcd-io/etcd/issues/new/choose
## Implement your change
etcd code should follow coding style suggested by the Golang community.
See the [style doc](https://github.com/golang/go/wiki/CodeReviewComments) for details.
Please ensure that your change passes static analysis (requires [golangci-lint](https://golangci-lint.run/usage/install/)):
-`make verify` to verify if all checks pass.
-`make verify-*` to verify a single check, for example `make verify-bom` to verify if bill-of-materials.json file is up-to-date.
-`make fix` to fix all checks.
-`make fix-*` to fix a single checks, for example `make fix-bom` to update bill-of-materials.json.
Please ensure that your change passes tests.
-`make test-unit` to run unit tests.
-`make test-integration` to run integration tests.
-`make test-e2e` to run e2e tests.
All changes are expected to come with unit test.
All new features are expected to have either e2e or integration tests.
## Commit your change
etcd follows a rough convention for commit messages:
* First line:
* Should start name of package (for example `etcdserver`, `etcdctl`) followed by `:` character.
* Describe the `what` behind the change
* Optionally author might provide the `why` behind the change in the main commit message body.
* Last line should be `Signed-off-by: firstname lastname <email@example.com>` (can be automatically generate by providing `--signoff` to git commit command).
Example of commit message:
```
etcdserver: add grpc interceptor to log info on incoming requests
@ -51,44 +127,29 @@ remote client info, request content (with value field redacted), request
handling latency, response size, etc. Uses zap logger if available,
etcd is a distributed key-value store designed to reliably and quickly preserve and provide access to critical data. It enables reliable distributed coordination through distributed locking, leader elections, and write barriers. An etcd cluster is intended for high availability and permanent data storage and retrieval.
These documents have moved to the [etcd-io/website repo](https://github.com/etcd-io/website/), and can be viewed live at [https://etcd.io/docs/latest/](https://etcd.io/docs/latest/).
This directory includes etcd project internal documentation for new and existing contributors.
For user and developer documentation please go to [etcd.io](https://etcd.io/),
which is developed in [website](https://github.com/etcd-io/website/) repo.
* New development occurs on the [main branch][main].
* Main branch should always have a green build!
* Backwards-compatible bug fixes should target the main branch and subsequently be ported to stable branches.
* Once the main branch is ready for release, it will be tagged and become the new stable branch.
The etcd team has adopted a *rolling release model* and supports two stable versions of etcd.
### Main branch
The `main` branch is our development branch. All new features land here first.
To try new and experimental features, pull `main` and play with it. Note that `main` may not be stable because new features may introduce bugs.
Before the release of the next stable version, feature PRs will be frozen. A [release manager](./release.md#release-management) will be assigned to major/minor version and will lead the etcd community in test, bug-fix and documentation of the release for one to two weeks.
### Stable branches
All branches with prefix `release-` are considered _stable_ branches.
After every minor release ([semver.org](https://semver.org/)), we will have a new stable branch for that release, managed by a [patch release manager](./release.md#release-management). We will keep fixing the backwards-compatible bugs for the latest two stable releases. A _patch_ release to each supported release branch, incorporating any bug fixes, will be once every two weeks, given any patches.
| Member | Active contributor in the community | Sponsored by 2 reviewers and multiple contributions | etcd GitHub org member |
| Reviewer | Review contributions from other members | History of review and authorship | [OWNERS] file reviewer entry |
| Maintainer | Set direction and priorities for the project | Demonstrated responsibility and excellent technical judgement | [OWNERS] file approver entry |
## New contributors
New contributors should be welcomed to the community by existing members,
helped with PR workflow, and directed to relevant documentation and
communication channels.
## Established community members
Established community members are expected to demonstrate their adherence to the
principles in this document, familiarity with project organization, roles,
policies, procedures, conventions, etc., and technical and/or writing ability.
Role-specific expectations, responsibilities, and requirements are enumerated
below.
## Member
Members are continuously active contributors in the community. They can have
issues and PRs assigned to them. Members are expected to remain active
contributors to the community.
**Defined by:** Member of the etcd GitHub organization.
### Requirements
- Enabled [two-factor authentication] on their GitHub account
- Have made multiple contributions to the project or community. Contribution may include, but is not limited to:
- Authoring or reviewing PRs on GitHub. At least one PR must be **merged**.
- Filing or commenting on issues on GitHub
- Contributing to community discussions (e.g. meetings, Slack, email discussion
forums, Stack Overflow)
- Subscribed to etcd-dev@googlegroups.com
- Have read the [contributor guide]
- Sponsored by one active maintainer or two reviewers.
- Sponsors must be from multiple member companies to demonstrate integration across community.
- With no objections from other maintainers
- Open a [membership nomination] issue against the etcd-io/etcd repo
- Ensure your sponsors are @mentioned on the issue
- Make sure that the list of contributions included is representative of your work on the project.
- Members can be removed by a supermajority of the maintainers or can resign by notifying
the maintainers.
### Responsibilities and privileges
- Responsive to issues and PRs assigned to them
- Granted "triage access" to etcd project
- Active owner of code they have contributed (unless ownership is explicitly transferred)
- Code is well tested
- Tests consistently pass
- Addresses bugs or issues discovered after code is accepted
**Note:** members who frequently contribute code are expected to proactively
perform code reviews and work towards becoming a *reviewer*.
## Reviewers
Reviewers are contributors who have demonstrated greater skill in
reviewing the code from other contributors. They are knowledgeable about both
the codebase and software engineering principles. Their LGTM counts towards
merging a code change into the project. A reviewer is generally on the ladder towards
maintainership.
**Defined by:***reviewers* entry in the [OWNERS] file.
### Requirements
- member for at least 3 months.
- Primary reviewer for at least 5 PRs to the codebase.
- Reviewed or contributed at least 20 substantial PRs to the codebase.
- Knowledgeable about the codebase.
- Sponsored by two active maintainers.
- Sponsors must be from multiple member companies to demonstrate integration across community.
- With no objections from other maintainers
- Reviewers can be removed by a supermajority of the maintainers or can resign by notifying
the maintainers.
### Responsibilities and privileges
- Code reviewer status may be a precondition to accepting large code contributions
- Responsible for project quality control via code reviews
- Focus on code quality and correctness, including testing and factoring
- May also review for more holistic issues, but not a requirement
- Expected to be responsive to review requests
- Assigned PRs to review related to area of expertise
- Assigned test bugs related to area of expertise
- Granted "triage access" to etcd project
## Maintainers
Maintainers are first and foremost contributors that have shown they
are committed to the long term success of a project. Maintainership is about building
trust with the current maintainers and being a person that they can
depend on to make decisions in the best interest of the project in a consistent manner.
**Defined by:***approvers* entry in the [OWNERS] file.
### Requirements
- Deep understanding of the technical goals and direction of the project
- Deep understanding of the technical domain of the project
- Sustained contributions to design and direction by doing all of:
- Authoring and reviewing proposals
- Initiating, contributing and resolving discussions (emails, GitHub issues, meetings)
- Identifying subtle or complex issues in designs and implementation PRs
- Directly contributed to the project through implementation and / or review
- Sponsored by two active maintainers and elected by supermajority
- Sponsors must be from multiple member companies to demonstrate integration across community.
- To become a maintainer send an email with your candidacy to etcd-maintainers-private@googlegroups.com
- Ensure your sponsors are @mentioned on the email
- Include a list of contributions representative of your work on the project.
- Existing maintainers vote will privately and respond to the email with either acceptance or with feedback for suggested improvement.
- With your membership approved you are expected to:
- Open a PR and add an entry to the [OWNERS] file
- Subscribe to etcd-maintainers@googlegroups.com and etcd-maintainers-private@googlegroups.com
- Request to join to [etcd-maintainer teams of etcd organization of GitHub](https://github.com/orgs/etcd-io/teams/maintainers-etcd)
- Request to join to the private slack channel for etcd maintainers on [kubernetes slack](http://slack.kubernetes.io/)
- Request access to etcd-development GCP project where we publish releases
- Request access to passwords shared between maintainers
### Responsibilities and privileges
- Make and approve technical design decisions
- Set technical direction and priorities
- Define milestones and releases
- Mentor and guide reviewers, and contributors to the project.
- Participate when called upon in the [security disclosure and release process]
- Ensure continued health of the project
- Adequate test coverage to confidently release
- Tests are passing reliably (i.e. not flaky) and are fixed when they fail
- Ensure a healthy process for discussion and decision making is in place.
- Work with other maintainers to maintain the project's overall health and success holistically
### Retiring
Life priorities, interests, and passions can change. Maintainers can retire and
move to [emeritus maintainers]. If a maintainer needs to step down, they should
inform other maintainers, if possible, help find someone to pick up the related
work. At the very least, ensure the related work can be continued. Afterward
they can remove themselves from list of existing maintainers.
If a maintainer has not been performing their duties for period of 12 months,
they can be removed by other maintainers. In that case inactive maintainer will
be first notified via an email. If situation doesn't improve, they will be
removed. If an emeritus maintainer wants to regain an active role, they can do
so by renewing their contributions. Active maintainers should welcome such a move.
Retiring of other maintainers or regaining the status should require approval
of at least two active maintainers.
## Acknowledgements
Contributor roles and responsibilities were written based on [Kubernetes community membership]
The dependabot is enabled & [configured](https://github.com/etcd-io/etcd/blob/main/.github/dependabot.yml) to
manage dependencies for etcd `main` branch. But dependabot doesn't work well for multi-module repository like `etcd`,
see [dependabot-core/issues/6678](https://github.com/dependabot/dependabot-core/issues/6678).
Usually human intervention is required each time when dependabot automatically opens some PRs to bump dependencies.
Please see guidance below.
### Dependencies used in workflows
The PRs which automatically bump dependencies (see examples below) used in workflows are fine, and can be approved & merged directly as long as all checks are successful.
- [build(deps): bump github/codeql-action from 2.2.11 to 2.2.12](https://github.com/etcd-io/etcd/pull/15736)
- [build(deps): bump actions/checkout from 3.5.0 to 3.5.2](https://github.com/etcd-io/etcd/pull/15735)
- [build(deps): bump ossf/scorecard-action from 2.1.2 to 2.1.3](https://github.com/etcd-io/etcd/pull/15607)
### Bumping order
When multiple etcd modules depend on the same package, please bump the package version for all the modules in the correct order. The rule is simple:
if module A depends on module B, then bump the dependency for module B before module A. If the two modules do not depend on each other, then
it doesn't matter to bump which module first. For example, multiple modules depend on `github.com/spf13/cobra`, we need to bump the dependency
in the following order,
- go.etcd.io/etcd/pkg/v3
- go.etcd.io/etcd/server/v3
- go.etcd.io/etcd/etcdctl/v3
- go.etcd.io/etcd/etcdutl/v3
- go.etcd.io/etcd/tests/v3
- go.etcd.io/etcd/v3
- go.etcd.io/etcd/tools/v3
For more details about etcd Golang modules, please check <https://etcd.io/docs/next/dev-internal/modules>
Note the module `go.etcd.io/etcd/tools/v3` doesn't depend on any other modules, nor by any other modules, so it doesn't matter when to bump dependencies for it.
### Steps to bump a dependency
Use the `github.com/spf13/cobra` as an example, follow steps below to bump it from 1.6.1 to 1.7.0 for module `go.etcd.io/etcd/etcdctl/v3`,
```bash
cd${ETCD_ROOT_DIR}/etcdctl
go get github.com/spf13/cobra@v1.7.0
go mod tidy
cd ..
./scripts/fix.sh
```
Execute the same steps for all other modules. When you finish bumping the dependency for all modules, then commit the change,
```bash
git add .
git commit --signoff -m "dependency: bump github.com/spf13/cobra from 1.6.1 to 1.7.0"
```
Please close the related PRs which were automatically opened by dependabot.
When you bump multiple dependencies in one PR, it's recommended to create a separate commit for each dependency. But it isn't a must; for example,
you can get all dependencies bumping for the module `go.etcd.io/etcd/tools/v3` included in one commit.
### Indirect dependencies
Usually we don't bump a dependency if all modules just indirectly depend on it, such as `github.com/go-logr/logr`.
If an indirect dependency (e.g. `D1`) causes any CVE or bugs which affect etcd, usually the module (e.g. `M1`, not part of etcd, but used by etcd)
which depends on it should bump the dependency (`D1`), and then etcd just needs to bump `M1`. However, if the module (`M1`) somehow doesn't
bump the problematic dependency, then etcd can still bump it (`D1`) directly following the same steps above. But as a long-term solution, etcd should
try to remove the dependency on such module (`M1`) that lack maintenance.
For mixed cases, in which some modules directly while others indirectly depend on a dependency, we have multiple options,
- Bump the dependency for all modules, no matter it's direct or indirect dependency.
- Bump the dependency only for modules which directly depend on it.
We should try to follow the first way, and temporarily fall back to the second one if we run into any issue on the first way. Eventually we
should fix the issue and ensure all modules depend on the same version of the dependency.
### Known incompatible dependency updates
#### arduino/setup-protoc
Please refer to [build(deps): bump arduino/setup-protoc from 1.3.0 to 2.0.0](https://github.com/etcd-io/etcd/pull/16016)
### Rotation worksheet
The dependabot scheduling interval is weekly; it means dependabot will automatically raise a bunch of PRs per week.
Usually human intervention is required each time. We have a [rotation worksheet](https://docs.google.com/spreadsheets/d/1jodHIO7Dk2VWTs1IRnfMFaRktS9IH8XRyifOnPdSY8I/edit#gid=1394774387),
and everyone is welcome to participate; you just need to register your name in the worksheet.
## Stable branches
Usually we don't proactively bump dependencies for stable releases unless there are any CVEs or bugs that affect etcd.
If we have to do it, then follow the same guidance above. Note that there is no `./scripts/fix.sh` in release-3.4, so no need to
execute it for 3.4.
## Golang versions
The etcd project aims to maintain a development branch that is on the latest [Go version](https://go.dev/dl), ideally this will align with the Go version in use for Kubernetes project development. For an example on how to update etcd to a new minor release of Go refer to issue <https://github.com/etcd-io/etcd/issues/16393> and the linked pull requests.
Suggested steps for performing a minor version upgrade for the etcd development branch:
1. Carefully review new Go version release notes and potentially related blog for any deprecations, performance impacts or other considerations.
2. Create a github issue to signal intent to upgrade and invite discussion, example <https://github.com/etcd-io/etcd/issues/16393>.
3. Complete the upgrade locally in your development environment.
4. Run performance benchmarks locally to compare before and after.
5. Raise a pull request for the changes, example <https://github.com/etcd-io/etcd/pull/16394>.
Stable etcd release branches will be maintained to stay on the latest patch release of a supported Go version, however upgrading minor versions will be avoided unless the minor version in use is now out of support. Refer to the [Go release policy](https://go.dev/doc/devel/release).
For an example of how to update etcd to a new patch release of Go refer to issue <https://github.com/etcd-io/etcd/issues/16343> and the linked pull requests.
This document provides an overview of etcd features and general development guidelines for adding and deprecating them. The project maintainers can override these guidelines per the need of the project following the project governance.
## Overview
The etcd features fall into three stages, experimental, stable, and unsafe.
### Experimental
Any new feature is usually added as an experimental feature. An experimental feature is characterized as below:
- Might be buggy due to a lack of user testing. Enabling the feature may not work as expected.
- Disabled by default when added initially.
- Support for such a feature may be dropped at any time without notice
- Feature related issues may be given lower priorities.
- It can be removed in the next minor or major release without following the feature deprecation policy unless it graduates to the stable future.
### Stable
A stable feature is characterized as below:
- Supported as part of the supported releases of etcd.
- May be enabled by default.
- Discontinuation of support must follow the feature deprecation policy.
### Unsafe
Unsafe features are rare and listed under the `Unsafe feature:` section in the etcd usage documentation. By default, they are disabled. They should be used with caution following documentation. An unsafe feature can be removed in the next minor or major release without following feature deprecation policy.
## Development Guidelines
### Adding a new feature
Any new enhancements to the etcd are typically added as an experimental feature. The general development requirements are listed below. They can be somewhat flexible depending on the scope of the feature and review discussions, and will evolve over time.
- Open an issue
- It must provide a clear need for the proposed feature.
- It should list development work items as checkboxes. There must be one work item towards future graduation to the stable future.
- Label the issue with `type/feature` and `experimental`.
- Keep the issue open for tracking purpose until a decision is made on graduation.
- Open a Pull Request (PR)
- Provide unit tests. Integreation tests are also recommended as possible.
- Provide robust e2e test coverage. If the feature being added is complicated or quickly needed, maintainers can decide to go with e2e tests for basic coverage initially and have robust coverage added at the later time before feature graduation to the stable feature.
- Provide logs for proper debugging.
- Provide metrics and benchmarks as needed.
- The Feature should be disabled by default.
- Any configuration flags related to the implementation of the feature must be prefixed with `experimental` e.g. `--experimental-feature-name`.
- Add a CHANGELOG entry.
- At least two maintainers must approve feature requirements and related code changes.
### Graduating an Experimental feature to Stable
It is important that experimental features don't get stuck in that stage. They should be revisited and moved to the stable stage following the graduation steps as described here.
#### Locate graduation candidate
Decide if an experimental feature is ready for graduation to the stable stage.
- Find the issue that was used to enable the experimental feature initially. One way to find such issues is to search for issues with `type/feature` and `experimental` labels.
- Fix any known open issues against the feature.
- Make sure the feature was enabled for at least one previous release. Check the PR(s) reference from the issue to see when the feature related code changes were merged.
#### Provide implementation
If an experimental feature is found ready for graduation to the stable stage, open a Pull Request (PR) with the following changes.
- Add robust e2e tests if not already provided.
- Add a new stable feature flag identical to the experimental feature flag but without the `--experimental` prefix.
- Deprecate the experimental feature following the [feature deprecation policy](#Deprecating-a-feature).
- Implementation must ensure that both the graduated and deprecated experimental feature flags work as expected. Note that both these flags will co-exist for the timeframe described in the feature deprecation policy.
- Enable the graduated feature by default if needed.
At least two maintainers must approve the work. Patch releases should not be considered for graduation.
### Deprecating a feature
#### Experimental
An experimental feature deprecates when it graduates to the stable stage.
- Add a deprecation message in the documentation of the experimental feature with a recommendation to use related stable feature. e.g. `DEPRECATED. Use <feature-name> instead.`
- Add a `deprecated` label in the issue that was initially used to enable the experimental feature.
#### Stable
As the project evolves, a stable feature may sometimes need to be deprecated and removed. Such a situation should be handled using the steps below:
- Create an issue for tracking purpose.
- Add a deprecation message in the feature usage documentation before a planned release for feature deprecation. e.g. `To be deprecated in <release>.`. If a new feature replaces the `To be deprecated` feature, then also provide a message saying so. e.g. `Use <feature-name> instead.`.
- Deprecate the feature in the planned release with a message as part of the feature usage documentation. e.g. `DEPRECATED`. If a new feature replaces the deprecated feature, then also provide a message saying so. e.g. `DEPRECATED. Use <feature-name> instead.`.
- Add a `deprecated` label in the related issue.
Remove the deprecated feature in the following release. Close any related issue(s). At least two maintainers must approve the work. Patch releases should not be considered for deprecation.
For testing and development deployments, the quickest and easiest way is to configure a local cluster. For a production deployment, refer to the [clustering][clustering] section.
## Local standalone cluster
### Starting a cluster
Run the following to deploy an etcd cluster as a standalone cluster:
```
$ ./etcd
...
```
If the `etcd` binary is not present in the current working directory, it might be located either at `$GOPATH/bin/etcd` or at `/usr/local/bin/etcd`. Run the command appropriately.
The running etcd member listens on `localhost:2379` for client requests.
### Interacting with the cluster
Use `etcdctl` to interact with the running cluster:
1. Store an example key-value pair in the cluster:
```
$ ./etcdctl put foo bar
OK
```
If OK is printed, storing key-value pair is successful.
2. Retrieve the value of `foo`:
```
$ ./etcdctl get foo
bar
```
If `bar` is returned, interaction with the etcd cluster is working as expected.
## Local multi-member cluster
### Starting a cluster
A `Procfile` at the base of the etcd git repository is provided to easily configure a local multi-member cluster. To start a multi-member cluster, navigate to the root of the etcd source tree and perform the following:
1. Install `goreman` to control Procfile-based applications:
```
$ go install github.com/mattn/goreman@latest
```
The installation will place executables in the $GOPATH/bin. If $GOPATH environment variable is not set, the tool will be installed into the $HOME/go/bin. Make sure that $PATH is set accordingly in your environment.
2. Start a cluster with `goreman` using etcd's stock Procfile:
```
$ goreman -f Procfile start
```
The members start running. They listen on `localhost:2379`, `localhost:22379`, and `localhost:32379` respectively for client requests.
### Interacting with the cluster
Use `etcdctl` to interact with the running cluster:
1. Print the list of members:
```
$ etcdctl --write-out=table --endpoints=localhost:2379 member list
```
The list of etcd members are displayed as follows:
Restarting the member re-establish the connection. `etcdctl` will now be able to retrieve the key successfully. To learn more about interacting with etcd, read [interacting with etcd section][interacting].
etcd uses the [zap][zap] library for logging application output categorized into *levels*. A log message's level is determined according to these conventions:
* Debug: Everything is still fine, but even common operations may be logged, and less helpful but more quantity of notices. Usually not used in production.
* Examples:
* Send a normal message to a remote peer
* Write a log entry to disk
* Info: Normal, working log information, everything is fine, but helpful notices for auditing or common operations. Should rather not be logged more frequently than once per a few seconds in normal server's operation.
* Examples:
* Startup configuration
* Start to do snapshot
* Warning: (Hopefully) Temporary conditions that may cause errors, but may work fine. A replica disappearing (that may reconnect) is a warning.
* Examples:
* Failure to send raft message to a remote peer
* Failure to receive heartbeat message within the configured election timeout
* Error: Data has been lost, a request has failed for a bad reason, or a required resource has been lost.
* Examples:
* Failure to allocate disk space for WAL
* Panic: Unrecoverable or unexpected error situation that requires stopping execution.
* Examples:
* Failure to create the database
* Fatal: Unrecoverable or unexpected error situation that requires immediate exit. Mostly used in the test.
The guide talks about how to release a new version of etcd.
The procedure includes some manual steps for sanity checking, but it can probably be further scripted. Please keep this document up-to-date if making changes to the release process.
## Release management
etcd community members are assigned to manage the release each etcd major/minor version as well as manage patches
and to each stable release branch. The managers are responsible for communicating the timelines and status of each
release and for ensuring the stability of the release branch.
| 3.4 patch (post 3.4.0) | Benjamin Wang [@ahrtr](https://github.com/ahrtr) |
| 3.5 patch (post 3.5.0) | Marek Siarkowicz [@serathius](https://github.com/serathius) |
All releases version numbers follow the format of [semantic versioning 2.0.0](http://semver.org/).
### Major, minor version release, or its pre-release
- Ensure the relevant milestone on GitHub is complete. All referenced issues should be closed, or moved elsewhere.
- Ensure the latest upgrade documentation is available.
- Bump [hardcoded MinClusterVerion in the repository](https://github.com/etcd-io/etcd/blob/v3.4.15/version/version.go#L29), if necessary.
- Add feature capability maps for the new version, if necessary.
### Patch version release
- To request a backport, devlopers submit cherrypick PRs targeting the release branch. The commits should not include merge commits. The commits should be restricted to bug fixes and security patches.
- The cherrypick PRs should target the appropriate release branch (`base:release-<major>-<minor>`). `hack/patch/cherrypick.sh` may be used to automatically generate cherrypick PRs.
- The release patch manager reviews the cherrypick PRs. Please discuss carefully what is backported to the patch release. Each patch release should be strictly better than it's predecessor.
- The release patch manager will cherry-pick these commits starting from the oldest one into stable branch.
## Write release note
- Write introduction for the new release. For example, what major bug we fix, what new features we introduce or what performance improvement we make.
- Put `[GH XXXX]` at the head of change line to reference Pull Request that introduces the change. Moreover, add a link on it to jump to the Pull Request.
- Find PRs with `release-note` label and explain them in `NEWS` file, as a straightforward summary of changes for end-users.
## Build and push the release artifacts
- Ensure `docker` is available.
Run release script in root directory:
```
DRY_RUN=false ./scripts/release.sh ${VERSION}
```
It generates all release binaries and images under directory ./release.
Binaries are pushed to gcr.io and images are pushed to quay.io and gcr.io.
## Publish release page in GitHub
- Set release title as the version name.
- Follow the format of previous release pages.
- Attach the generated binaries and signatures.
- Select whether it is a pre-release.
- Publish the release!
## Announce to the etcd-dev Googlegroup
- Follow the format of [previous release emails](https://groups.google.com/g/etcd-dev).
- Make sure to include a list of authors that contributed since the previous release - something like the following might be handy:
- Create new stable branch through `git push origin ${VERSION_MAJOR}.${VERSION_MINOR}` if this is a major stable release. This assumes `origin` corresponds to "https://github.com/etcd-io/etcd".
- Bump [hardcoded Version in the repository](https://github.com/etcd-io/etcd/blob/v3.4.15/version/version.go#L30) to the version `${VERSION}+git`.
If any part of the etcd project has bugs or documentation mistakes, please let us know by [opening an issue][etcd-issue]. We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check that an issue reporting the same problem does not already exist.
To make the bug report accurate and easy to understand, please try to create bug reports that are:
- Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the etcd server, please attach the etcd log (the starting log with etcd configuration is especially important).
- Reproducible. Include the steps to reproduce the problem. We understand some issues might be hard to reproduce, please includes the steps that might lead to the problem. If possible, please attach the affected etcd data dir and stack strace to the bug report.
- Isolated. Please try to isolate and reproduce the bug with minimum dependencies. It would significantly slow down the speed to fix a bug if too many dependencies are involved in a bug report. Debugging external systems that rely on etcd is out of scope, but we are happy to provide guidance in the right direction or help with using etcd itself.
- Unique. Do not duplicate existing bug report.
- Scoped. One bug per report. Do not follow up with another bug inside one report.
It may be worthwhile to read [Elika Etemad’s article on filing good bug reports][filing-good-bugs] before creating a bug report.
We might ask for further information to locate a bug. A duplicated bug report will be closed.
## Frequently asked questions
### How to get a stack trace
``` bash
$ kill -QUIT $PID
```
### How to get etcd version
``` bash
$ etcd --version
```
### How to get etcd configuration and log when it runs as systemd service ‘etcd2.service’
``` bash
$ sudo systemctl cat etcd2
$ sudo journalctl -u etcd2
```
Due to an upstream systemd bug, journald may miss the last few log lines when its processes exit. If journalctl says etcd stopped without fatal or panic message, try `sudo journalctl -f -t etcd2` to get full log.
etcd uses GitHub milestones to track all tasks in each major or minor release. The `roadmap.md` file only records the
most important tasks for each release. The list is based on current maintainers capacity that may shift over time.
Proposed milestones is what we think we can deliver with people we have. If we have more support on the important
stuff, we could pick up more items from backlog. Note that etcd will continue to mainly focus on technical debt over
the next few major or minor releases.
Each item has an assigned priority. Refer to [priority definitions](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/triage_issues.md#step-5---prioritise-the-issue).
## v3.6.0
For a full list of tasks in `v3.6.0`, please see [milestone etcd-v3.6](https://github.com/etcd-io/etcd/milestone/38).
| [Support downgrade](https://github.com/etcd-io/etcd/issues/11716) | priority/important-soon | In progress | etcd will support downgrade starting from 3.6.0. But it will also support offline downgrade from 3.5 to 3.4. |
| [StoreV2 deprecation](https://github.com/etcd-io/etcd/issues/12913) | priority/important-soon | In progress | This task will be covered in both 3.6 and 3.7. |
| [Release raft 3.6.0](https://github.com/etcd-io/raft/issues/89) | priority/important-soon | Not started | etcd 3.6.0 will depends on raft 3.6.0 |
| [Release bbolt 1.4.0](https://github.com/etcd-io/bbolt/issues/553) | priority/important-soon | Not started | etcd 3.6.0 will depends on bbolt 1.4.0 |
| [Support /livez and /readyz endpoints](https://github.com/etcd-io/etcd/issues/16007) | priority/important-longterm | In progress | It provides clearer APIs, and can also workaround the stalled writes issue |
| [Bump gRPC](https://github.com/etcd-io/etcd/issues/16290) | priority/important-longterm | Completed | It isn't guaranteed to be resolved in 3.6, and might be postponed to 3.7 depending on the effort and risk. |
| [Deprecate grpc-gateway or bump it](https://github.com/etcd-io/etcd/issues/14499) | priority/important-longterm | Completed | It isn't guaranteed to be resolved in 3.6, and might be postponed to 3.7 depending on the effort and risk. |
| [bbolt: Add logger into bbolt](https://github.com/etcd-io/bbolt/issues/509) | priority/important-longterm | Completed | It's important to diagnose bbolt issues |
| [bbolt: Add surgery commands](https://github.com/etcd-io/bbolt/issues/370) | priority/important-longterm | Completed | Surgery commands are important for fixing corrupted db files |
| [Evaluate and (Gradulate or deprecate/remove) experimental features](https://github.com/etcd-io/etcd/issues/16292) | priority/backlog | Not started | This task will be covered in both 3.6 and 3.7. |
## v3.7.0
For a full list of tasks in `v3.7.0`, please see [milestone etcd-v3.7](https://github.com/etcd-io/etcd/milestone/39).
| [Refactor lease: Lease might be revoked by mistake by old leader](https://github.com/etcd-io/etcd/issues/15247) | P1 | to be investigated & discussed |
| [Integrate raft's new feature (async write) into etcd](https://github.com/etcd-io/etcd/issues/16291) | P1 | It should can improve the performance |
| [bbolt: Support customizing the bbolt rebalance threshold](https://github.com/etcd-io/bbolt/issues/422) | P2 | It may get rid of etcd's defragmentation. Both bbolt and etcd need to be changed. |
| [Evaluate and (graduate or deprecate/remove) experimental features](https://github.com/etcd-io/etcd/issues/16292) | P2 | Finish the remaining tasks 3.7. |
The `etcd` issues are listed at <https://github.com/etcd-io/etcd/issues> and are identified with labels. For example, an issue that is identified as a bug will be set to label `type/bug`.
The etcd project uses labels to indicate common attributes such as `area`, `type` and `priority` of incoming issues.
New issues will often start out without any labels, but typically `etcd` maintainers, reviewers and members will add labels by following these triage guidelines. The detailed list of labels can be found at <https://github.com/etcd-io/etcd/labels>.
## Scope
This document serves as the primary guidelines for triaging incoming issues in `etcd`.
All contributors are encouraged and welcome to help manage issues which will help reduce burden on project maintainers, though the work and responsibilities discussed in this document are created with `etcd` project reviewers and members in mind as these individuals will have triage access to the etcd project which is a requirement for actions like applying labels or closing issues.
Refer to [etcd community membership](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/community-membership.md) for guidance on becoming and etcd project member or reviewer.
## Step 1 - Find an issue to triage
To get started you can use the following recommended issue searches to identify issues that are in need of triage:
* [Issues that have no labels](https://github.com/etcd-io/etcd/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated+no%3Alabel)
* [Issues created recently](https://github.com/etcd-io/etcd/issues?q=is%3Aissue+is%3Aopen+)
* [Issues not assigned but linked pr](https://github.com/etcd-io/etcd/issues?q=is%3Aopen+is%3Aissue+no%3Aassignee+linked%3Apr)
* [Issues with no comments](https://github.com/etcd-io/etcd/issues?q=is%3Aopen+is%3Aissue+comments%3A0+)
* [Issues with help wanted](https://github.com/etcd-io/etcd/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22+)
## Step 2 - Check the issue is valid
Before we start adding labels or trying to work out a priority, our first triage step needs to be working out if the issue actually belongs to the etcd project and is not a duplicate.
### Issues that don't belong to etcd
Sometime issues are reported that actually belongs to other projects that `etcd` use. For example, `grpc` or `golang` issues. Such issues should be addressed by asking reporter to open issues in appropriate other project.
These issues can generally be closed unless a maintainer and issue reporter see a need to keep it open for tracking purpose. If you have triage permissions please close it, alternatively mention the @etcd-io/members group to request a member with triage access close the issue.
### Duplicate issues
If an issue is a duplicate, add a comment stating so along with a reference for the original issue and if you have triage permissions please close it, alternatively mention the @etcd-io/members group to request a member with triage access close the issue.
## Step 3 - Apply the appropriate type label
Adding a `type` label to an issue helps create visibility on the health of the project and helps contributors identify potential priorities, i.e. addressing existing bugs or test flakes before implementing new features.
### Support requests
As a general rule the focus for etcd support is to address common themes in a broad way that helps all users, i.e. through channels like known issues, frequently asked questions and high quality documentation. To make the best use of project members time we should avoid providing 1:1 support if a broad approach is available.
Some people mistakenly use our GitHub bug report or feature request templates to file support requests. Usually they are asking for help operating or configuring some aspect of etcd. Support requests for etcd should instead be raised as [discussions](https://github.com/etcd-io/etcd/discussions).
Common types of support requests are:
1. Questions about configuring or operating existing well documented etcd features, for example <https://github.com/etcd-io/etcd/issues/15945>. Note - If an existing feature is not well documented please apply the `area/documentation` label and propose documentation improvements that would prevent future users from stumbling on the problem again.
2. Bug reports or questions about unspported versions of etcd, for example <https://github.com/etcd-io/etcd/issues/15796>. When responding to these issues please refer to our [supported versions documentation](https://etcd.io/docs/latest/op-guide/versioning) and encourage the reporter to upgrade to a recent patch release of a supported version as soon as possible. We should limit the effort supporting users that do not make the effort to run a supported version of etcd or ensure their version is patched.
3. Bug reports that do not provide a complete list of steps to reproduce issue and/or contributors are not able to reproduce the issue, for example <https://github.com/etcd-io/etcd/issues/15740>. We should limit the effort we put into reproducing issues ourselves and motivate users to provide necessary information to accept the bug report.
4. General questions that are filed using feature request or bug report issue templates, for example <https://github.com/etcd-io/etcd/issues/15914>. Note - These types of requests may surface good additions to our [frequently asked questions](https://etcd.io/docs/v3.5/faq).
If you identify that an issue is a support request please:
1. Add the `type/support` or `type/question` label.
2. Add the following comment to inform the issue creator that discussions should be used instead and that this issue will be converted to a discussion.
> Thank you for your question, this support issue will be moved to our [Discussion Forums](https://github.com/etcd-io/etcd/discussions).
>
> We are trying to consolidate the channels to which questions for help/support are posted so that we can improve our efficiency in responding to your requests, and to make it easier for you to find answers to frequently asked questions and how to address common use cases.
>
> We regularly see messages posted in multiple forums, with the full response thread only in one place or, worse, spread across multiple forums. Also, the large volume of support issues on GitHub is making it difficult for us to use issues to identify real bugs.
>
> Members of the etcd community use Discussion Forums to field support requests. Before posting a new question, please search these for answers to similar questions, and also familiarize yourself with:
> Again, thanks for using etcd and raising this question.
>
> The etcd team
3. Finally, click `Convert to discussion` on the right hand panel, selecting the appropriate discussion category.
### Bug reports
If an issue has been raised as a bug it should already have the `type/bug` label, however if this is missing for an issue you determine to be a bug please add the label manually.
The next step is to validate if the issue is indeed a bug. If not, add a comment with findings and close trivial issue. For non-trivial issue, wait to hear back from issue reporter and see if there is any objection. If issue reporter does not reply in 30 days, close the issue.
If the problem can not be reproduced or requires more information, leave a comment for the issue reporter as soon as possible while the issue will be fresh for the issue reporter.
### Feature requests
New feature requests should be created via the etcd feature request template and in theory already have the `type/feature` label, however if this is missing for an issue you determine to be a feature please add the label manually.
### Test flakes
Test flakes are a specific type of bug that the etcd project tracks separately as these are a priority to address. These should be created via the test flake template and in theory already have the `type/flake` label, however if this is missing for an issue you determine to be related to a flaking test please add the label manually.
## Step 4 - Define the areas impacted
Adding an `area` label to an issue helps create visibility on which areas of the etcd project require attention and helps contributors find issues to work on relating to their particular skills or knowledge of the etcd codebase.
If an issue crosses multiple domains please add additional `area` labels to reflect that.
Below is a brief summary of the area labels in active use by the etcd project along with any notes on their use:
| Label | Notes |
| --- | --- |
| area/external | Tracking label for issues raised that are external to etcd. |
| area/community | |
| area/raft | |
| area/clientv3 | |
| area/performance | |
| area/security | |
| area/tls | |
| area/auth | |
| area/etcdctl | |
| area/etcdutl | |
| area/contrib | Not to be confused with `area/community` this label is specifically used for issues relating to community maintained scripts or files in the `contrib/` directory which aren't part of the core etcd project. |
| area/documentation | |
| area/tooling | Generally used in relation to the third party / external utilities or tools that are used in various stages of the etcd build, test or release process, for example tooling to create sboms. |
| area/testing | |
| area/robustness-testing | |
## Step 5 - Prioritise the issue
If an issue lacks a priority label it has not been formally prioritized yet.
Adding a `priority` label helps the etcd project understand what is important and should be worked on now, and conversely what is not as important and is on the project backlog.
|Priority label|What it means|Examples|
|---|---|---|
| `priority/critical-urgent` | Maintainers are responsible for making sure that these issues (in their area) are being actively worked on—i.e., drop what you're doing. Stuff is burning. These should be fixed before the next release. | user-visible critical bugs in core features <br> broken builds on tier1 supported platforms <br> tests and critical security issues |
| `priority/important-soon` | Must be staffed and worked on either currently or very soon—ideally in time for the next release. | |
| `priority/important-longterm` | Important over the long term, but may not be currently staffed and/or may require multiple releases to complete. | |
| `priority/backlog` | General agreement that this is a nice-to-have, but no one's available to work on it anytime soon. Community contributions would be most welcome in the meantime, though it might take a while to get them reviewed if reviewers are fully occupied with higher-priority issues—for example, immediately before a release.| |
| `priority/awaiting-more-evidence` | Possibly useful, but not yet enough support to actually get it done. | Mostly placeholders for potentially good ideas, so that they don't get completely forgotten, and can be referenced or deduped every time they come up |
## Step 6 - Support new contributors
As part of the `etcd` triage process once the `kind` and `area` have been determined, please consider if the issue would be suitable for a less experienced contributor. The `good first issue` label is a subset of the `help wanted` label, indicating that members have committed to providing extra assistance for new contributors. All `good first issue` items also have the `help wanted` label.
### Help wanted
Items marked with the `help wanted` label need to ensure that they meet these criteria:
* **Low Barrier to Entry** - It should be easy for new contributors.
* **Clear** - The task is agreed upon and does not require further discussions in the community.
* **Goldilocks priority** - The priority should not be so high that a core contributor should do it, but not too low that it isn’t useful enough for a core contributor to spend time reviewing it, answering questions, helping get it into a release, etc.
### Good first issue
Items marked with `good first issue` are intended for first-time contributors. It indicates that members will keep an eye out for these pull requests and shepherd it through our processes.
New contributors should not be left to find an approver, ping for reviews, decipher test commands, or identify that their build failed due to a flake. It is important to make new contributors feel welcome and valued. We should assure them that they will have an extra level of help with their first contribution.
After a contributor has successfully completed one or two `good first issue` items, they should be ready to move on to `help wanted` items.
* **No Barrier to Entry** - The task is something that a new contributor can tackle without advanced setup or domain knowledge.
* **Solution Explained** - The recommended solution is clearly described in the issue.
* **Gives Examples** - Link to examples of similar implementations so new contributors have a reference guide for their changes.
* **Identifies Relevant Code** - The relevant code and tests to be changed should be linked in the issue.
* **Ready to Test** - There should be existing tests that can be modified, or existing test cases fit to be copied. If the area of code doesn’t have tests, before labeling the issue, add a test fixture. This prep often makes a great help wanted task!
## Step 7 - Follow up
Once initial triage has been completed, issues need to be re-evaluated over time to ensure they don't become stale incorrectly.
### Track important issues
If an issue is at risk of being closed by stale bot in future, but is an important issuefor the etcd project, then please apply the `stage/tracked` label and remove any `stale` labels that exist. This will ensure the project does not lose sight of the issue.
### Close incomplete issues
Issues that lack enough information from the issue reporter should be closed if issue reporter do not provide information in 30 days. Issues can always be re-opened at a later date if new information is provided.
### Check for incomplete work
If an issue owned by a developer has no pull request created in 30 days, contact the issue owner and kindly ask about the status of their work, or to release ownership on the issue if needed.
The `etcd` PRs are listed at https://github.com/etcd-io/etcd/pulls
A PR can have various labels, milestone, reviewer etc. The detailed list of labels can be found at
https://github.com/kubernetes/kubernetes/labels
Following are few example searches on PR for convenience:
* [Open PRS for milestone etcd-v3.6](https://github.com/etcd-io/etcd/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+milestone%3Aetcd-v3.6)
* [PRs under investigation](https://github.com/etcd-io/etcd/labels/Investigating)
## Scope
These guidelines serves as a primary document for managing PRs in `etcd`. Everyone is welcome to help manage PRs but the work and responsibilities discussed in this document is created with `etcd` maintainers and active contributors in mind.
## Handle inactive PRs
Poke PR owner if review comments are not addressed in 15 days. If PR owner does not reply in 90 days, update the PR with a new commit if possible. If not, inactive PR should be closed after 180 days.
## Poke reviewer if needed
Reviewers are responsive in a timely fashion, but considering everyone is busy, give them some time after requesting review if quick response is not provided. If response is not provided in 10 days, feel free to contact them via adding a comment in the PR or sending an email or message on the Slack.
## Verify important labels are in place
Make sure that appropriate reviewers are added to the PR. Also, make sure that a milestone is identified. If any of these or other important labels are missing, add them. If a correct label cannot be decided, leave a comment for the maintainers to do so as needed.
"title":"Stream result of v3electionpbLeaderResponse"
}
},
"default":{
"description":"An unexpected error response",
"description":"An unexpected error response.",
"schema":{
"$ref":"#/definitions/runtimeError"
"$ref":"#/definitions/rpcStatus"
}
}
},
@ -131,9 +136,9 @@
}
},
"default":{
"description":"An unexpected error response",
"description":"An unexpected error response.",
"schema":{
"$ref":"#/definitions/runtimeError"
"$ref":"#/definitions/rpcStatus"
}
}
},
@ -164,9 +169,9 @@
}
},
"default":{
"description":"An unexpected error response",
"description":"An unexpected error response.",
"schema":{
"$ref":"#/definitions/runtimeError"
"$ref":"#/definitions/rpcStatus"
}
}
},
@ -203,7 +208,7 @@
"revision":{
"type":"string",
"format":"int64",
"description":"revision is the key-value store revision when the request was applied.\nFor watch progress responses, the header.revision indicates progress. All future events\nrecieved in this stream are guaranteed to have a higher revision number than the\nheader.revision number."
"description":"revision is the key-value store revision when the request was applied, and it's\nunset (so 0) in case of calls not interacting with key-value store.\nFor watch progress responses, the header.revision indicates progress. All future events\nreceived in this stream are guaranteed to have a higher revision number than the\nheader.revision number."
"description":"revision is the key-value store revision when the request was applied.\nFor watch progress responses, the header.revision indicates progress. All future events\nrecieved in this stream are guaranteed to have a higher revision number than the\nheader.revision number."
"description":"revision is the key-value store revision when the request was applied, and it's\nunset (so 0) in case of calls not interacting with key-value store.\nFor watch progress responses, the header.revision indicates progress. All future events\nreceived in this stream are guaranteed to have a higher revision number than the\nheader.revision number."
The infrastructure to build for arm64 is provided by [Equinix Metal](https://www.equinix.com/) via the [CNCF Community Infrastructure Lab](https://github.com/cncf/cluster/issues).
Previously, several maintainers were responsible for managing two bare-metal machines with a self-hosted runner installed. This was a manual process, and side effects could be left over from previous builds.
As part of a joint program between Ampere and the CNCF, [actuated.dev](https://actuated.dev) is providing managed Arm64 builds.
To use the new infrastructure, add the following to your workflow:
```yaml
runs-on:actuated-arm64-8cpu-8gb
```
The vCPUs and RAM are customizable, i.e. `actuated-arm64-8cpu-16gb` or `actuated-arm64-8cpu-32gb`.
For urgent support, contact @alexellis or the [actuated team](https://actuated.dev).
| Summary | Code refactor in v3.5.0 resulted in consistent index not being saved atomically. Independent crash could lead to committed transactions are not reflected on all the members. |
| Impact | No user reported problems in production as triggering the issue required frequent crashes, however issue was critical enough to motivate a public statement. Main impact comes from loosing user trust into etcd reliability. |
## Background
etcd v3 state is preserved on disk in two forms write ahead log (WAL) and database state (DB).
etcd v3.5 also still maintains v2 state, however it's deprecated and not relevant to the issue in this postmortem.
WAL stores history of changes for etcd state and database represents state at one point.
To know which point of history database is representing, it stores consistent index (CI).
It's a special metadata field that points to last entry in WAL that it has seen.
When etcd is updating database state, it replays entries from WAL and updates the consistent index to point to new entry.
This operation is required to be [atomic](https://en.wikipedia.org/wiki/Atomic_commit).
A partial fail would mean that database and WAL would no longer match, so some entries would be either skipped (if only CI is updated) or executed twice (if only changes are applied).
This is especially important for distributed system like etcd, where there are multiple cluster members, each applying the WAL entries to their database.
Correctness of the system depends on assumption that every member of the cluster, while replying WAL entries, will reach the same state.
## Root cause
To simplify managing consistency index, etcd has introduced backend hooks in https://github.com/etcd-io/etcd/pull/12855.
Goal was to ensure that consistency index is always updated, by automatically triggering update during commit.
Implementation was as follows, before applying the WAL entries, etcd updated in memory value of consistent index.
As part of transaction commit process, a database hook would read the value of consistent index and store it to database.
Problem is that in memory value of consistent index is shared, and there might be other in flight transactions apart from serial WAL apply flow.
So if we imagine scenario:
1. etcd server starts an apply workflow, and it just sets a new consistent index value.
2. The periodic commit is triggered, and it executes the backend hook and saves consistent index from apply workflow.
3. etcd server finished an apply workflow, saves new changes and saves same value of consistent index again.
Between second and third point there is a very small window where consistent index is increased without applying entry from WAL.
## Trigger
If etcd crashed after consistency index is saved, but before to apply workflow finished it would lead to data inconsistency.
When recovering the data etcd would skip executing changes from failed apply workflow, assuming they have been already executed.
This follows the issue reports and code used to reproduce the issue where trigger was etcd crashing under high request load.
Etcd v3.5.0 was released with bug (https://github.com/etcd-io/etcd/pull/13505) that could cause etcd to crash that was fixed in v3.5.1.
Apart from that all reports described etcd running under high memory pressure, causing it to go out of memory from time to time.
Reproduction run etcd under high stress and randomly killed one of the members using SIGKILL signal (not recoverable immediate process death).
## Detection
For single member cluster it is totally undetectable.
There is no mechanism or tool for verifying that state database matches WAL.
In cluster with multiple members it would mean that one of the members that crashed, will missing changes from failed apply workflow.
This means that it will have different state of database and will return different hash via `HashKV` grpc call.
There is an automatic mechanism to detect data inconsistency.
It can be executed during etcd start via `--experimental-initial-corrupt-check` and periodically via `--experimental-corrupt-check-time`.
Both checks however have a flaw, they depend on `HashKV` grpc method, which might fail causing the check to pass.
In multi member etcd cluster, each member can run with different performance and be at different stage of applying the WAL log.
Comparing database hashes between multiple etcd members requires all hashes to be calculated at the same change.
This is done by requesting hash for the same `revision` (version of key value store).
However, it will not work if the provided revision is not available on the members.
This can happen on very slow members, or in cases where corruption has lead revision numbers to diverge.
This means that for this issue, the corrupt check is only reliable during etcd start just after etcd crashes.
## Impact
We are not aware any cases of users reporting a data corruption in production environment.
However, issue was critical enough to motivate a public statement.
Main impact comes from loosing user trust into etcd reliability.
## Lessons learned
### What went well
* Multiple maintainers were able to work effectively on reproducing and fixing the issue. As they are in different timezones, there was always someone working on the issue.
* When fixing the main data inconsistency we have found multiple other edge cases that could lead to data corruption (https://github.com/etcd-io/etcd/issues/13514, https://github.com/etcd-io/etcd/issues/13922, https://github.com/etcd-io/etcd/issues/13937).
### What went wrong
* No users enable data corruption detection as it is still an experimental feature introduced in v3.3. All reported cases where detected manually making it almost impossible to reproduce.
* etcd has functional tests designed to detect such problems, however they are unmaintained, flaky and are missing crucial scenarios.
* etcd v3.5 release was not qualified as comprehensive as previous ones. Older maintainers run manual qualification process that is no longer known or executed.
* etcd apply code is so complicated that fixing the data inconsistency took almost 2 weeks and multiple tries. Fix needed to be so complicated that we needed to develop automatic validation for it (https://github.com/etcd-io/etcd/pull/13885).
* etcd v3.5 was recommended for production without enough insight on the production adoption. Production ready recommendations based on after some internal feedback... to get diverse usage, but the user's hold on till someone else will discover issues.
### Where we got lucky
* We reproduced the issue using etcd functional only because weird partition setup on workstation. Functional tests store etcd data under `/tmp` usually mounted to in memory filesystem. Problem was reproduced only because one of the maintainers has `/tmp` mounted to standard disk.
## Action items
Action items should directly address items listed in lessons learned.
We should double down on things that went well, fix things that went wrong, and stop depending on luck.
Action fall under three types, and we should have at least one item per type. Types:
* Prevent - Prevent similar issues from occurring. In this case, what testing we should introduce to find data inconsistency issues before release, preventing publishing broken release.
* Detect - Be more effective in detecting when similar issues occur. In this case, improve mechanism to detect data inconsistency issue so users will be automatically informed.
* Mitigate - Reduce time to recovery for users. In this case, how we ensure that users are able to quickly fix data inconsistency.
Actions should not be restricted to fixing the immediate issues and also propose long term strategic improvements.
To reflect this action items should have assigned priority:
* P0 - Critical for reliability of the v3.5 release. Should be prioritized this over all other work and backported to v3.5.
* P1 - Important for long term success of the project. Blocks v3.6 release.
* P2 - Stretch goals that would be nice to have for v3.6, however should not be blocking.
<ahref="https://actuated.dev/"><imgalt="Arm CI sponsored by Actuated"src="https://docs.actuated.dev/images/actuated-badge.png"width="120px"></img></a>
**Note**: The `main` branch may be in an *unstable or even broken state* during development. For stable versions, see [releases][github-release].
@ -24,10 +24,14 @@ etcd is a distributed reliable key-value store for the most critical data of a d
etcd is written in Go and uses the [Raft][] consensus algorithm to manage a highly-available replicated log.
etcd is used [in production by many companies](./ADOPTERS.md), and the development team stands behind it in critical deployment scenarios, where etcd is frequently teamed with applications such as [Kubernetes][k8s], [locksmith][], [vulcand][], [Doorman][], and many others. Reliability is further ensured by [**rigorous testing**](https://github.com/etcd-io/etcd/tree/main/tests/functional).
etcd is used [in production by many companies](./ADOPTERS.md), and the development team stands behind it in critical deployment scenarios, where etcd is frequently teamed with applications such as [Kubernetes][k8s], [locksmith][], [vulcand][], [Doorman][], and many others. Reliability is further ensured by rigorous [**robustness testing**](https://github.com/etcd-io/etcd/tree/main/tests/robustness).
See [etcdctl][etcdctl] for a simple command line client.
<sub>Original image credited to xkcd.com/2347, alterations by Josh Berkus.</sub>
[raft]: https://raft.github.io/
[k8s]: http://kubernetes.io/
[doorman]: https://github.com/youtube/doorman
@ -35,46 +39,9 @@ See [etcdctl][etcdctl] for a simple command line client.
etcd contributors and maintainers have monthly (every four weeks) meetings at 11:00 AM (USA Pacific) on Thursday.
An initial agenda will be posted to the [shared Google docs][shared-meeting-notes] a day before each meeting, and everyone is welcome to suggest additional topics or other agendas.
[Maintainers](OWNERS) strive to shape an inclusive open source project culture where users are heard and contributors feel respected and empowered. Maintainers aim to build productive relationships across different companies and disciplines. Read more about [Maintainers role and responsibilities](Documentation/contributor-guide/community-membership.md#maintainers).
## Getting started
@ -84,11 +51,7 @@ The easiest way to get etcd is to use one of the pre-built release binaries whic
For more installation guides, please check out [play.etcd.io](http://play.etcd.io) and [operating etcd](https://etcd.io/docs/latest/op-guide).
For those wanting to try the very latest version, [build the latest version of etcd][dl-build] from the `main` branch. This first needs [*Go*](https://golang.org/) installed ([version 1.16+](/go.mod#L3) is required). All development occurs on `main`, including new features and bug fixes. Bug fixes are first targeted at `main` and subsequently ported to release branches, as described in the [branch management][branch-management] guide.
*Slack: [#sig-etcd](https://kubernetes.slack.com/archives/C3HD8ARJ5) channel on Kubernetes ([get an invite](http://slack.kubernetes.io/))
*[Community meetings](#community-meetings)
### Community meetings
etcd contributors and maintainers meet every week at 11:00 AM (USA Pacific) on Thursday and meetings alternate between community meetings and issue triage meetings. An initial agenda will be posted to the [shared Google docs][shared-meeting-notes] a day before each meeting, and everyone is welcome to suggest additional topics or other agendas.
Issue triage meetings are aimed at getting through our backlog of PRs and Issues. Triage meetings are open to any contributor; you don't have to be a reviewer or approver to help out! They can also be a good way to get started contributing.
Meeting recordings are uploaded to official etcd [YouTube channel].
Get calendar invitation by joining [etcd-dev](https://groups.google.com/g/etcd-dev) mailing group.
See [CONTRIBUTING](CONTRIBUTING.md) for details on submitting patches and the contribution workflow.
See [CONTRIBUTING](CONTRIBUTING.md) for details on setting up your development environment, submitting patches and the contribution workflow.
Please refer to [community-membership.md](Documentation/contributor-guide/community-membership.md#member) for information on becoming an etcd project member. We welcome and look forward to your contributions to the project!
Please also refer to [roadmap](Documentation/contributor-guide/roadmap.md) to get more details on the priorities for the next few major or minor releases.
## Reporting bugs
See [reporting bugs](https://etcd.io/docs/latest/reporting-bugs) for details about reporting any issues.
See [reporting bugs](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md) for details about reporting any issues. Before opening an issue please check it is not covered in our [frequently asked questions].
@ -194,16 +174,24 @@ See [security disclosure and release process](security/README.md) for details on
## Issue and PR management
See [issue triage guidelines](https://etcd.io/docs/current/triage/issues/) for details on how issues are managed.
See [issue triage guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/triage_issues.md) for details on how issues are managed.
See [PR management](https://etcd.io/docs/current/triage/prs/) for guidelines on how pull requests are managed.
See [PR management](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/triage_prs.md) for guidelines on how pull requests are managed.
## etcd Emeritus Maintainers
These emeritus maintainers dedicated a part of their career to etcd and reviewed code, triaged bugs, and pushed the project forward over a substantial period of time. Their contribution is greatly appreciated.
These emeritus maintainers dedicated a part of their career to etcd and reviewed code, triaged bugs and pushed the project forward over a substantial period of time. Their contribution is greatly appreciated.
This document defines a high level roadmap for etcd development.
The dates below should not be considered authoritative, but rather indicative of the projected timeline of the project. The [milestones defined in GitHub](https://github.com/etcd-io/etcd/milestones) represent the most up-to-date and issue-for-issue plans.
etcd 3.3 is our current stable branch. The roadmap below outlines new features that will be added to etcd, and while subject to change, define what future stable will look like.
### etcd 3.4 (2019)
- Stabilization of 3.3 experimental features
- Support/document downgrade
- Snapshot restore as Go library
- Improved client balancer with new gRPC balancer interface
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
