Commit Graph

377 Commits

Author SHA1 Message Date
a40a6e9ad8 Merge pull request #12805 from ptabor/20210326-minor-test-fixes
tests: logging & temp-dir fixes
2021-03-28 19:14:13 -04:00
f290ab2e60 Update dependecies:
github.com/grpc-ecosystem/grpc-gateway v1.14.6 -> grpc-gateway v1.16.0
  golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e->v0.0.0-20210220033141-f8bda1e9f3ba
2021-03-27 20:48:33 +01:00
c49807f59e Update cmux to 1.5.0.
Executed using:
```
./scripts/update_dep.sh github.com/soheilhy/cmux v0.1.5
```
2021-03-27 11:18:13 +01:00
03f55eeb2c Make NewTmpBackend use testing tmp location (so cleanup). 2021-03-26 13:54:55 +01:00
d0962f10ae Adopt grpc_testing to grpc-1.36. 2021-03-24 23:31:03 +01:00
a60676686b Update dep: grpc: 1.32.0 -> 1.36.0. 2021-03-24 22:27:55 +01:00
45fb7b41d6 Update dep: github.com/golang/protobuf v1.3.5 ->  v1.5.1
Thanks to https://go-review.googlesource.com/c/protobuf/+/300869/ its
feasible now.
2021-03-24 22:27:12 +01:00
8469108548 Merge pull request #12782 from ptabor/20210316-fixes
Integration: Test flakiness fixes
2021-03-19 20:10:11 +01:00
18321a0c85 Merge pull request #12786 from ptabor/20210318-debug-env
client: Bring back ETCD_CLIENT_DEBUG variable interpretation.
2021-03-18 21:55:51 +01:00
9312d1b077 clientv3: Bring back ETCD_CLIENT_DEBUG variable interpretation.
env ETCD_CLIENT_DEBUG supports log levels (debug, info, warn, error, dpanic, panic, fatal).
Only when set, overrides application-wide grpc logging settings.
2021-03-18 18:43:07 +01:00
dac6e37ea1 *: over 20 staticcheck fixes 2021-03-18 15:06:17 +01:00
725a8c5e02 Enable configuring delegated zap-logging for embed server. 2021-03-17 08:17:36 +01:00
6657d5907c Make sure all integration tests have BeforeTest.
The CL disallows to create NewCluster in tests without BeforeTest.
2021-03-16 23:50:01 +01:00
809e7629ed Add integration.BeforeTest to all missing tests. 2021-03-16 23:50:01 +01:00
a84bd093b0 Integration with grpc-settable logger. 2021-03-16 22:50:41 +01:00
a57e967d84 Integration test flakes fixes. 2021-03-16 16:08:18 +01:00
a47c18d30a Fix 2 remaining 'defer AfterTest' calls. 2021-03-13 23:41:29 +01:00
b406647dd7 Fix/remove broken: TestMetricDbSizeDefragDebugging 2021-03-12 23:05:53 +01:00
4eba403ccc Merge pull request #12765 from ptabor/20210312-move-config
Move config (ServerConfig) out of etcdserver package.
2021-03-11 15:05:29 -08:00
fd7fed1511 Move config (ServerConfig) out of etcdserver package.
Motivation:
  - ServerConfig is part of 'embed' public API, while etcdserver is more 'internal'
  - EtcdServer is already too big and config is pretty wide-spread leaf
if we were to split etcdserver (e.g. into pre & post-apply part).
2021-03-11 20:56:22 +01:00
b9226d03f4 Merge pull request #12763 from hexfusion/bump-proto
vendor: bump gogo/proto to v1.3.2
2021-03-11 17:59:07 +01:00
d3aa3fb486 vendor: bump gogo/proto to v1.3.2
Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
2021-03-11 11:27:25 -05:00
c8243a9927 Tests: Functional - in case of failure, log the exception. 2021-03-09 18:19:52 +01:00
b6c2e87a74 Testing: Integration tests does not check whether t==nil 2021-03-09 18:19:52 +01:00
5ddabfdb24 tests: Make tests operate in /tmp director instead of src.
Thanks to this, unix sockets should be not longer
created by integration tests in the the source code directory,
so potentially trigger IDE reloads and unnecessery load (and mess).
2021-03-09 18:19:52 +01:00
bfe02c0526 tests: Cluster creation that failed shouldn't leak goroutines. 2021-03-09 18:19:52 +01:00
41f6cc7234 Tests: Better isolation between store_v2v3 integration tests. 2021-03-09 18:19:51 +01:00
fb1d48e98e Integration tests: Use BeforeTest(t) instead of defer AfterTest().
Thanks to this change, a single method BeforeTest(t) can handle
before-test logic as well as registration of cleanup code
(t.Cleanup(func)).
2021-03-09 18:19:51 +01:00
87258efd90 Integration tests: Use zaptest.Logger based testing.TB
Thanks to this the logs:
  - are automatically printed if the test fails.
  - are in pretty consistent format.
  - are annotated by 'member' information of the cluster emitting them.

Side changes:
  - Set propert default got DefaultWarningApplyDuration (used to be '0')
  - Name the members based on their 'place' on the list (as opposed to
'random')
2021-03-09 18:19:51 +01:00
a46a358577 --experimental-memory-mlock support
The flag protects etcd memory from being swapped out to disk.
This can happen in memory constrained systems where mmaped bbolt
area is natural condidate for swapping out.

This flag should provide better tail latency on the cost of higher RSS
ram usage. If the experiment is successful, the logic should get moved
into bbolt layer, where we can protect specific bbolt instances
(e.g. avoid protecting both during defragmentation).
2021-03-07 12:32:57 +01:00
f7a2389992 Update version of certifi/gocertifi to get rid of WTF Public license
Seems old versions of https://github.com/certifi/gocertifi where
categorized as "Do What The F*ck You Want To Public License".

Update to newer version that is explicit `Mozilla Public License` 2.0 (MPL 2.0).
2021-03-04 09:48:34 +01:00
102c198444 Merge pull request #12705 from astromechza/bm_etcd_peer_server_cert
etcdmain: added peer-client-{client,key}-file parameters for supporting separate client and server certs when communicating between peers
2021-03-02 09:03:35 +01:00
5e10d12996 tests: Fixes a few recently spotted test-flakes
```
Unexpected goroutines running after all test(s).
1 instances of:
syscall.Syscall(...)
	/usr/local/go/src/syscall/asm_linux_386.s:19 +0x5
syscall.Close(...)
	/usr/local/go/src/syscall/zsyscall_linux_386.go:285 +0x3d
internal/poll.(*FD).destroy(...)
	/usr/local/go/src/internal/poll/fd_unix.go:77 +0x30
internal/poll.(*FD).decref(...)
	/usr/local/go/src/internal/poll/fd_mutex.go:213 +0x38
internal/poll.(*FD).Close(...)
	/usr/local/go/src/internal/poll/fd_unix.go:99 +0x43
net.(*netFD).Close(...)
	/usr/local/go/src/net/fd_posix.go:37 +0x49
FAIL	go.etcd.io/etcd/tests/v3/integration/client	0.039s
```

```
--- FAIL: TestServer_TCP_Secure_DelayTx (0.20s)
    server_test.go:110: took 128.026085ms with no latency
    server_test.go:125: took 62.980988ms with latency 50ms��5ms
    server_test.go:133: expected took1 128.026085ms < took2 62.980988ms (with latency)
```

https://github.com/etcd-io/etcd/issues/12372
2021-03-01 18:07:38 +01:00
3d44f5bf80 *: added client-{client,key}-file parameters for supporting separate client and server certs when communicating between peers
In some environments, the CA is not able to sign certificates with both
'client auth' and 'server auth' extended usage parameters and so an operator
needs to be able to set a seperate client certificate to use when making
requests which is different to the certificate used for accepting requests.
This applies to both proxy and etcd member mode and is available as both a CLI
 flag and config file field for peer TLS.

Signed-off-by: Ben Meier <ben.meier@oracle.com>
2021-02-28 14:37:56 +00:00
a7f340216d Reformat code according to 'gotip' rules.
In practices adds annotations in the new syntax:
```
+//go:build !linux
 // +build !linux
```

Fixes failing gotip PASSES='fmt' check:
https://travis-ci.com/github/etcd-io/etcd/jobs/486453806
2021-02-26 10:14:46 +01:00
45b1e6b470 ClientV3: Ordering: Fix the ordering test such it does not fail.
The test depended on very subtle timing semantic and on properties of
'copied' clients.

https://travis-ci.com/github/etcd-io/etcd/jobs/486191449

Examplar failure:
```
{"level":"warn","ts":"2021-02-25T12:34:47.894Z","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000d6fc0/#initially=[unix://localhost:86269902489114839060]","attempt":1,"error":"rpc error: code = Unavailable desc = etcdserver: rpc not supported for learner"}
{"level":"warn","ts":"2021-02-25T12:34:48.163Z","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00035a000/#initially=[unix://localhost:78285857058450835940]","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: not leader"}
{"level":"info","ts":"2021-02-25T12:34:48.255Z","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"warn","ts":"2021-02-25T12:34:48.255Z","caller":"v3/maintenance.go:221","msg":"failed to receive from snapshot stream; closing","error":"rpc error: code = Canceled desc = context canceled"}
{"level":"info","ts":"2021-02-25T12:34:48.255Z","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2021-02-25T12:34:50.255Z","caller":"v3/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2021-02-25T12:34:51.717Z","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"warn","ts":"2021-02-25T12:34:52.017Z","caller":"v3/maintenance.go:221","msg":"failed to receive from snapshot stream; closing","error":"rpc error: code = Canceled desc = context canceled"}
{"level":"info","ts":"2021-02-25T12:34:52.018Z","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"warn","ts":"2021-02-25T12:34:53.018Z","caller":"v3/maintenance.go:221","msg":"failed to receive from snapshot stream; closing","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
--- FAIL: TestEndpointSwitchResolvesViolation (10.12s)
    ordering_util_test.go:81: failed to resolve order violation etcdclient: no cluster members have a revision higher than the previously received revision
```
2021-02-25 22:15:13 +01:00
60d5159091 version: bump up to 3.5.0-alpha.0 2021-02-24 19:55:45 +00:00
1a9c81abda Update grpc dependency to 1.32.
Simplify grpc testing infrastructure to align with upstream changes.
2021-02-23 11:31:50 +01:00
4a1c24556c clientv3: PS: Replace balancer with upstream grpc solution
Addresses comments from: https://github.com/etcd-io/etcd/pull/12671#pullrequestreview-593942302
2021-02-23 10:03:15 +01:00
3d7aac948b Merge pull request #12196 from ironcladlou/metrics-watch-error-fix
etcdserver: fix incorrect metrics generated when clients cancel watches
2021-02-19 12:46:49 -08:00
b67ed4e4aa Merge pull request #12671 from ptabor/20210207-grpc-del-balancer
clientv3: Replace balancer with upstream grpc solution
2021-02-17 22:45:13 +01:00
a836a8045b Get rid of legacy client/v3/naming API.
Update grpcproxy to use the new abstractions.
2021-02-09 11:56:28 +01:00
0b75fede64 Replace client/v3/balancer with standard components: resolver + round_robin LB
This commit significantly reduces volume of custom code
in etcd client v3, while preserving full existing functionality.
2021-02-08 18:50:31 +01:00
8feb55f65c *: implement Endpoint Watch and new Resolver 2021-02-08 20:05:45 +08:00
571ed502d4 endpoints: implement Update method for EndpointManager.
- Add integration test for endpoints and resolver.
2021-02-04 23:30:22 +08:00
d6d03beaea Merge pull request #12538 from lzhfromustc/12_9_GoroutineLeak
test: change channel operations to avoid potential goroutine leaks
2021-02-01 21:16:43 +01:00
0d7a671c75 Tests: Fix vet warnings about Fatal in sub-goroutines.
% (cd tests && go vet ./...)
stderr: # go.etcd.io/etcd/tests/v3/integration/clientv3/concurrency_test
stderr: integration/clientv3/concurrency/election_test.go:74:6: call to (*T).Fatal from a non-test goroutine
stderr: integration/clientv3/concurrency/mutex_test.go:57:4: call to (*T).Fatal from a non-test goroutine
2021-01-31 00:00:26 +01:00
f0ecad00e3 Use temp-directory that is covered by framework level cleanup
Prior to this PR, the e2e tests where creating dirs like:
```
/tmp/testname1.etcd030299846
/tmp/testname0.etcd039445123
/tmp/testname0.etcd206372065
```
and not cleaning them, that led to disk-space-exceeded flakes.

After the PR, the testing.TB tempdir mechanism is used and the names are
being cleaned and are more miningful:

```
../../bin/etcd --name test-TestCtlV3EndpointHashKV-2 --listen-client-urls http://localhost:20010 --advertise-client-urls http://localhost:20010 --listen-peer-urls https://localhost:20011 --initial-advertise-peer-urls https://localhost:20011 --initial-cluster-token new --data-dir /tmp/TestCtlV3EndpointHashKV429176179/003 --snapshot-count 100000 --experimental-initial-corrupt-check --peer-auto-tls --initial-cluster test-TestCtlV3EndpointHashKV-0=https://localhost:20001,test-TestCtlV3EndpointHashKV-1=https://localhost:20006,test-TestCtlV3EndpointHashKV-2=https://localhost:20011
```
2021-01-30 13:25:55 +01:00
5d7c1db3a9 Introduce grpc-1.30+ compatible client/v3/naming API.
This is not yet implementation, just API and tests to be filled
with implementation in next CLs,
tracked by: https://github.com/etcd-io/etcd/issues/12652

We propose here 3 packages:
 - clientv3/naming/endpoints ->
    That is abstraction layer over etcd that allows to write, read &
    watch Endpoints information. It's independent from GRPC API. It hides
    the storage details.

 - clientv3/naming/endpoints/internal ->
    That contains the grpc's compatible Update class to preserve the
    internal JSON mashalling format.

 - clientv3/naming/resolver ->
   That implements the GRPC resolver API, such that etcd can be
   used for connection.Dial in grpc.

Please see the grpc_naming.md document changes & grpcproxy/cluster.go
new integration, to see how the new abstractions work.
2021-01-30 12:32:19 +01:00
9600bbd5fe Update root image for docker-test-image to ubuntu:20.10 2021-01-29 22:16:51 +00:00