fb2e96e651
vendor: Get rid of ./vendor cleanup
...
Updated scripts and documentation to not recommend vendoring.
Implemented best practices for tools installation.
Performed multiple tests to confirm its not breaking any workflows and
has no negative performance impact. Rather see 3x speedup.
1. PASSES="fmt unit integration e2e functional" ./test
2. ./scripts/updatebom.sh
3. ./scripts/updatedep.sh
4. ./scripts/genproto.sh - works - ca be simplified - in follow up PR
5. Installation without explicit GOPATH:
```
% unset GOPATH
% [sudo] rm -rf ~/go
% git clone https://github.com/etcd-io/etcd.git
% time ./build
go: downloading google.golang.org/grpc v1.26.0
go: downloading github.com/jonboulle/clockwork v0.1.0
go: downloading github.com/prometheus/client_golang v1.0.0
go: downloading github.com/soheilhy/cmux v0.1.4
go: downloading github.com/gogo/protobuf v1.2.1
go: downloading sigs.k8s.io/yaml v1.1.0
go: downloading golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2
go: downloading github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903
go: downloading go.etcd.io/bbolt v1.3.5
go: downloading go.uber.org/zap v1.15.0
go: downloading golang.org/x/crypto v0.0.0-20191002192127-34f69633bfdc
go: downloading github.com/golang/protobuf v1.3.2
go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8
go: downloading github.com/beorn7/perks v1.0.0
go: downloading github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4
go: downloading github.com/coreos/go-systemd/v22 v22.0.0
go: downloading gopkg.in/yaml.v2 v2.2.2
go: downloading github.com/coreos/go-semver v0.2.0
go: downloading github.com/sirupsen/logrus v1.4.2
go: downloading golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7
go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: downloading github.com/google/uuid v1.0.0
go: downloading github.com/modern-go/reflect2 v1.0.1
go: downloading github.com/prometheus/common v0.4.1
go: downloading github.com/spf13/cobra v0.0.3
go: downloading github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
go: downloading github.com/gorilla/websocket v0.0.0-20170926233335-4201258b820c
go: downloading github.com/spf13/pflag v1.0.1
go: downloading github.com/json-iterator/go v1.1.7
go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: downloading github.com/google/btree v1.0.0
go: downloading go.uber.org/atomic v1.6.0
go: downloading github.com/prometheus/procfs v0.0.2
go: downloading go.uber.org/multierr v1.5.0
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5
go: downloading github.com/grpc-ecosystem/grpc-gateway v1.9.5
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4
go: downloading github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4
go: downloading google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading golang.org/x/text v0.3.3
go: downloading github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5
go: downloading github.com/bgentry/speakeasy v0.1.0
go: downloading gopkg.in/cheggaaa/pb.v1 v1.0.25
go: downloading github.com/urfave/cli v1.20.0
go: downloading github.com/mattn/go-runewidth v0.0.2
./build 8.22s user 2.31s system 117% cpu 8.961 total
```
Before:
```
% git clone https://github.com/etcd-io/etcd.git && cd etcd && time ./build
Cloning into 'etcd'...
remote: Enumerating objects: 97872, done.
remote: Total 97872 (delta 0), reused 0 (delta 0), pack-reused 97872
Receiving objects: 100% (97872/97872), 58.97 MiB | 19.85 MiB/s, done.
Resolving deltas: 100% (63091/63091), done.
./build 34.97s user 4.15s system 236% cpu 16.555 total
```
6. Rebuild without changes:
```
% time ./build
./build 1.43s user 0.83s system 168% cpu 1.336 total
```
7. Instantation of vendor directory (assuming ./build loaded them to
$GOPATH/pkg):
```
time go mod vendor
go: downloading github.com/inconshreveable/mousetrap v1.0.0
go: downloading github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa
go: downloading github.com/creack/pty v1.1.11
go: downloading github.com/etcd-io/gofail v0.0.0-20190801230047-ad7f989257ca
go: downloading github.com/konsorten/go-windows-terminal-sequences v1.0.1
go mod vendor 0.51s user 0.44s system 110% cpu 0.861 total
```
8. Fresh instantation of vendor:
```
% rm -rf vendor
% [sudo] rm -rf ~/go
% time go mod vendor
go: downloading github.com/coreos/go-systemd/v22 v22.0.0
go: downloading github.com/spf13/cobra v0.0.3
go: downloading github.com/prometheus/client_golang v1.0.0
go: downloading golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7
go: downloading github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4
go: downloading github.com/gogo/protobuf v1.2.1
go: downloading sigs.k8s.io/yaml v1.1.0
go: downloading google.golang.org/grpc v1.26.0
go: downloading github.com/urfave/cli v1.20.0
go: downloading go.uber.org/zap v1.15.0
go: downloading github.com/spf13/pflag v1.0.1
go: downloading github.com/soheilhy/cmux v0.1.4
go: downloading github.com/json-iterator/go v1.1.7
go: downloading github.com/coreos/go-semver v0.2.0
go: downloading github.com/prometheus/common v0.4.1
go: downloading github.com/prometheus/procfs v0.0.2
go: downloading go.uber.org/atomic v1.6.0
go: downloading github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5
go: downloading github.com/golang/protobuf v1.3.2
go: downloading github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4
go: downloading github.com/modern-go/reflect2 v1.0.1
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading go.uber.org/multierr v1.5.0
go: downloading github.com/creack/pty v1.1.11
go: downloading github.com/mattn/go-runewidth v0.0.2
go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: downloading golang.org/x/crypto v0.0.0-20191002192127-34f69633bfdc
go: downloading golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5
go: downloading github.com/jonboulle/clockwork v0.1.0
go: downloading gopkg.in/yaml.v2 v2.2.2
go: downloading github.com/etcd-io/gofail v0.0.0-20190801230047-ad7f989257ca
go: downloading github.com/grpc-ecosystem/grpc-gateway v1.9.5
go: downloading github.com/google/btree v1.0.0
go: downloading google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55
go: downloading github.com/beorn7/perks v1.0.0
go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: downloading github.com/google/uuid v1.0.0
go: downloading golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2
go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8
go: downloading go.etcd.io/bbolt v1.3.5
go: downloading golang.org/x/text v0.3.3
go: downloading gopkg.in/cheggaaa/pb.v1 v1.0.25
go: downloading github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
go: downloading github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4
go: downloading github.com/inconshreveable/mousetrap v1.0.0
go: downloading github.com/gorilla/websocket v0.0.0-20170926233335-4201258b820c
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading github.com/bgentry/speakeasy v0.1.0
go: downloading github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903
go: downloading github.com/sirupsen/logrus v1.4.2
go: downloading github.com/konsorten/go-windows-terminal-sequences v1.0.1
go mod vendor 3.62s user 1.30s system 127% cpu 3.854 total
```
9. Size of the repository - before: 39M, after: 18M
Before:
```
% time git clone https://github.com/etcd-io/etcd.git
Cloning into 'etcd'...
remote: Enumerating objects: 97872, done.
remote: Total 97872 (delta 0), reused 0 (delta 0), pack-reused 97872
Receiving objects: 100% (97872/97872), 58.97 MiB | 20.53 MiB/s, done.
Resolving deltas: 100% (63091/63091), done.
git clone https://github.com/etcd-io/etcd.git 4.66s user 1.02s system 93% cpu 6.068 total
% du -h --exclude .git -d 1
944K ./clientv3
108K ./etcdmain
5.4M ./Documentation
384K ./security
384K ./mvcc
28K ./.github
8.0K ./version
144K ./contrib
240K ./proxy
2.5M ./etcdserver
112K ./embed
536K ./integration
332K ./tools
116K ./lease
108K ./logos
896K ./tests
960K ./raft
216K ./client
52K ./scripts
100K ./hack
464K ./etcdctl
3.0M ./pkg
620K ./functional
136K ./wal
152K ./auth
21M ./vendor
39M
```
After:
```
% time git clone https://github.com/ptabor/etcd.git -b 20200908-no-vendor
Cloning into 'etcd'...
remote: Enumerating objects: 38, done.
remote: Counting objects: 100% (38/38), done.
remote: Compressing objects: 100% (37/37), done.
remote: Total 98489 (delta 10), reused 8 (delta 1), pack-reused 98451
Receiving objects: 100% (98489/98489), 59.23 MiB | 21.26 MiB/s, done.
Resolving deltas: 100% (63572/63572), done.
git clone https://github.com/ptabor/etcd.git -b 20200908-no-vendor 5.56s user 1.05s system 105% cpu 6.260 total
% du -h --exclude .git -d 1
944K ./clientv3
108K ./etcdmain
5.4M ./Documentation
384K ./security
384K ./mvcc
28K ./.github
8.0K ./version
144K ./contrib
240K ./proxy
2.5M ./etcdserver
112K ./embed
536K ./integration
332K ./tools
116K ./lease
108K ./logos
896K ./tests
960K ./raft
216K ./client
56K ./scripts
100K ./hack
464K ./etcdctl
3.0M ./pkg
620K ./functional
136K ./wal
152K ./auth
19M .
```
2020-09-15 14:38:54 +02:00
2d76f9d17d
vendor: Remove /vendor and add to .gitignore
2020-09-15 14:38:47 +02:00
76e769ce95
Merge pull request #12273 from ptabor/2020-09-07-fix-grpc-proxy-tests
...
testing/e2e,grpcproxy: Fix: go test --tags "cluster_proxy" -v ./tests/e2e/...
2020-09-09 12:03:09 -07:00
e81cae77a3
Merge pull request #12274 from ptabor/20200907-fix-cov-e2e-tests
...
tests/e2e,etcdctl,etcdmain: Fix go test --tags cov -v ./tests/e2e
2020-09-09 11:14:18 -07:00
9d5a840942
etcdmain/grpc_proxy: Remove superflous logging line.
2020-09-09 20:04:25 +02:00
7c880e5263
etcdctl: Rename Start / StartWithErrors to MustStart
2020-09-09 19:32:50 +02:00
10fa9614e1
Merge pull request #12271 from jingyih/add_watch_notify_interval_flag_in_testing
...
integration: add WatchProgressNotifyInterval in integration test
2020-09-09 08:56:48 -07:00
c32180d772
tests/e2e,etcdctl,etcdmain: Fix go test --tags cov -v ./tests/e2e
...
This CL fixes:
COVERDIR=./coverage PASSES="build_cov" && go test --tags cov -v ./tests/e2e
and is part of the effort to make:
COVERDIR=coverage PASSES="build_cov cov" ./test
fully pass.
The args passed to ./bin/etcd_test and ./bin/etcdctl_test binaries were
mismatched. The protocol of passing the arguments using
environment variables has been replaces with proper passing of flags.
How the measurement of coverage by e2e tests works:
1. COVERDIR=./coverage PASSES="build_cov" are generating
./bin/etcd_test and ./bin/etcdctl_test binaries.
2. These binaries are tests (as coverage can be computed only for
tests) [see ./main_test.go ./etcdctl/main_test.go], but this tests are
running the main logic of the server and uppon termination (or SIGTERM
signal) are writting proper .coverprofile files in the $COVERDIR folder.
The binaries used to take arguments using env variables, but its not
needed any longer. The binaries can consume any command line arguments
that either test (so --test.fooo) or the original binary can consume.
3. The tests/e2e (when compiled with the --tags cov) are starting the
_test binaries instead of the original binaries, such that the coverage
is being collected.
2020-09-09 12:56:15 +02:00
73817b53fd
integration: add flag WatchProgressNotifyInterval in integration test
2020-09-07 08:32:54 -07:00
093282f5ea
tests/e2e: cluster_proxy tests use CN-less cert for etcd-server auth.
...
Change tests/e2e to use proper (client-nocn.crt) certificate when
running in tags="cluster_proxy" mode.
Thanks to this (and previous in this PR) changes, the following test run
finally succeeds:
./build && go test --tags "cluster_proxy" -v ./tests/e2e/...
2020-09-07 12:55:01 +02:00
2d0ce9de3d
etcdmain: grpc-proxy should only require CN-less certificates for --cert flags.
...
We have following communication schema:
client --- 1 ---> grpc-proxy --- 2 --- > etcd-server
There are 2 sets of flags/certs in grpc proxy [ https://github.com/etcd-io/etcd/blob/master/etcdmain/grpc_proxy.go#L140 ]:
A. (cert-file, key-file, trusted-ca-file, auto-tls) this are controlling [1] so client to proxy connection and in particular they are describing proxy public identity.
B. (cert,key, cacert ) - these are controlling [2] so what's the identity that proxy uses to make connections to the etcd-server.
If 2 (B.) contains certificate with CN and etcd-server is running with --client-cert-auth=true, the CN can be used as identity of 'client' from service perspective. This is permission escalation, that we should forbid.
If 1 (A.) contains certificate with CN - it should be considered perfectly valid. The server can (should) have full identity.
So only --cert flag (and not --cert-file flag) should be validated for empty CN.
2020-09-07 11:59:28 +02:00
2c93127c7b
integration: Regenerate certificates, add client-nocn.crt
...
Executed:
(cd ./integration/fixtures && ./gencerts.sh)
This in particular cereated a new client-nocn.crt (and key) that can be
used for testing grpc-proxy -> etcd-server connections.
2020-09-07 11:48:38 +02:00
966e8cecf0
integration: gencerts.sh cleanup and supports no-CN certs
...
integration/fixtures/gencerts.sh:
- refactored common logic to a helper function
- added definition for client-nocn certificate
(used for grpc-proxy -> etcd-server) communication.
2020-09-07 11:47:24 +02:00
c20cc05fc5
mvcc: Export a "Last DB compaction" timestamp metric ( #12176 )
...
This is to aid with debugging the effectiveness of systems that
manually take care of cluster compaction, and have greater visibity
into recent compactions.
It can be handy to alert on the exactly how long it was since a
compaction (and also to put on dashboards) had happened.
---
Tested using a test cluster, the final result looks like this:
```
root@etcd-1:~# ETCDCTL_API=3 /tmp/test-etcd/etcdctl --endpoints=192.168.232.10:2379 compact 1012
compacted revision 1012
root@etcd-1:~# curl -s 192.168.232.10:2379/metrics | grep last
# HELP etcd_debugging_mvcc_db_compaction_last The unix time since the last db compaction. Resets to 0 on start.
# TYPE etcd_debugging_mvcc_db_compaction_last gauge
etcd_debugging_mvcc_db_compaction_last 1.595873939e+09
```
2020-08-26 16:27:10 -07:00
facd0c9460
Merge pull request #12252 from spzala/changelogfileperm34and33
...
CHANGELOG: file perm updates in 3.4 and 3.3
2020-08-24 12:30:21 -07:00
1c0d73d248
CHANGELOG: file perm updates in 3.4 and 3.3
...
Updates for https://github.com/etcd-io/etcd/pull/12250 and
https://github.com/etcd-io/etcd/pull/12251
2020-08-24 11:58:19 -04:00
ae66916226
pkg: file stat warning ( #12242 )
...
Provide warning and doc instead of enforcing file permission.
2020-08-23 17:20:16 -07:00
c199d3d8c3
server: use buffered channel to avoid goroutine leak ( #11941 )
...
Signed-off-by: Gaurav Singh <gaurav1086@gmail.com >
2020-08-21 18:46:28 -07:00
261aa31dc5
Merge pull request #12243 from hexfusion/bump-x/text
...
vendor: bump golang.org/x/text
2020-08-21 10:03:01 -04:00
100c443a10
vendor: bump golang.org/x/text
...
Signed-off-by: Sam Batschelet <sbatsche@redhat.com >
2020-08-21 09:14:40 -04:00
76993f1fc6
Merge pull request #12240 from avorima/fdusage_readdirnames
...
etcdserver: Use Readdirnames to count fds for FDUsage
2020-08-20 21:58:18 -07:00
ab9c14f477
Merge pull request #12241 from philips/add-asset-transparency-github-action
...
github: workflows: add asset-transparency release action
2020-08-20 14:03:19 -07:00
142358c13d
github: workflows: add asset-transparency release action
...
From etcd-dev discussion:
https://groups.google.com/u/2/g/etcd-dev/c/oMGSBqs_7sc
I have been working on this system called Asset Transparency[1] which
helps users verify they have received the correct contents from a URL.
If you are familiar with the "download a file, download a SHA256SUM
file, run `sha256sum -c`, etc" process? This tool helps to automate
that for users into something like this[2]:
$ tl get https://github.com/etcd-io/etcd/releases/download/v3.4.12/etcd-v3.4.12-darwin-amd64.zip
And a best practice for this Asset Transparency system is that URLs
are registered with the log as soon as possible. Why? Well, the sooner
a URL is entered the longer it can protect people consuming a URL from
unexpected content modification from say a GitHub credential
compromise.
To that end I have written a GitHub Action[3] that will automatically
do that on every release. It is easy to activate and should be hands
free after installation. So, before I enable it I want to see if there
are any concerns from maintainers. The only change to our repo will be
a new file in .github/workflows.
[1] https://www.transparencylog.com
[2] https://github.com/transparencylog/tl
[3] https://github.com/transparencylog/publish-releases-asset-transparency-action
2020-08-20 11:32:36 -07:00
be70400fb5
etcdserver: Use Readdirnames to count fds for FDUsage
...
Readdir already calls Readdirnames, but continues to allocate
os.FileInfo with Lstat for each result.
2020-08-20 16:51:29 +02:00
4b6a0eea49
CHANGELOG: update with server panic fix
...
Signed-off-by: Gyuho Lee <leegyuho@amazon.com >
2020-08-19 20:02:52 -07:00
44dea5df03
Merge pull request #12238 from liggitt/slow-v2-panic
...
etcdserver: Avoid panics logging slow v2 requests in integration tests
2020-08-19 09:20:47 -07:00
ad57fea4c5
etcdserver: Avoid panics logging slow v2 requests in integration tests
2020-08-19 11:30:15 -04:00
cfdc296a3c
CHANGELOG: update release-3.2 dates
...
Signed-off-by: Gyuho Lee <leegyuho@amazon.com >
2020-08-18 09:43:45 -07:00
edaac6e2a9
CHANGELOG: add v3.3.24 release dates
...
Signed-off-by: Gyuho Lee <leegyuho@amazon.com >
2020-08-18 09:30:27 -07:00
e37b28bd28
CHANGELOG: add v3.4.11
...
Signed-off-by: Gyuho Lee <leegyuho@amazon.com >
2020-08-18 09:23:24 -07:00
0526f461e1
Doc: Update curl command to fix 400 Bad Request ( #11911 )
2020-08-16 16:12:39 -07:00
0e4ba37b6c
Merge pull request #12193 from mitake/integration_test
...
test: avoid non existing package for integration test
2020-08-15 23:26:34 -07:00
d35933c351
Merge pull request #12221 from wenjiaswe/changelog_12215
...
CHANGELOG: update from 12215
2020-08-14 13:29:40 -07:00
32982ef469
CHANGELOG: update from 12215
...
Change-Id: I17e076554a56c95fabd95af111eccd8d7409966b
2020-08-14 13:18:02 -07:00
92f9e6eba2
Merge pull request #12216 from jingyih/experimental_flag_for_watch_notify_interval
...
*: add experimental flag for watch notify interval
2020-08-14 12:47:43 -07:00
799b16c2d1
CHANGELOG: update for PR12216
2020-08-14 12:06:38 -07:00
9a698476bf
*: add experimental flag for watch notify interval
2020-08-14 12:01:00 -07:00
06f89cc4f8
Merge pull request #12212 from gyuho/logger
...
*: upgrade zap logger to 1.15, replace global logger
2020-08-13 09:46:44 -07:00
93cf449205
Merge pull request #12214 from gyuho/fd
...
*: optimize runtime.FDUsage + add OS level FD metrics
2020-08-12 18:37:05 -07:00
5678779665
CHANGELOG: update
...
Signed-off-by: Gyuho Lee <leegyuho@amazon.com >
2020-08-12 10:32:27 -07:00
421df2ecbb
etcdserver: add OS level FD metrics
...
Similar counts are exposed via Prometheus.
This adds the one that are perceived by etcd server.
e.g.
os_fd_limit 120000
os_fd_used 14
process_cpu_seconds_total 0.31
process_max_fds 120000
process_open_fds 17
Signed-off-by: Gyuho Lee <leegyuho@amazon.com >
2020-08-12 10:32:27 -07:00
53fdcdc5a2
pkg/runtime: optimize FDUsage by removing sort
...
No need sort when we just want the counts.
Signed-off-by: Gyuho Lee <leegyuho@amazon.com >
2020-08-12 10:32:24 -07:00
d8ed233791
CHANGELOG: update
...
Signed-off-by: Gyuho Lee <leegyuho@amazon.com >
2020-08-12 09:50:00 -07:00
7eac6bd497
*: upgrade zap logger to 1.15, replace global logger
...
Signed-off-by: Gyuho Lee <leegyuho@amazon.com >
2020-08-12 09:50:00 -07:00
ed27d9d2de
Merge pull request #12198 from ptabor/20200803-int-to-string-test-fix
...
etcdserver, wal: Fix tests unintended CASTing of int->String.
2020-08-11 21:35:20 -07:00
8c44d25f2a
Merge pull request #12211 from tangcong/ignore-errcompacted
...
etcdserver: ignore ErrCompacted error
2020-08-11 21:34:38 -07:00
fe36be2251
Merge pull request #12195 from tangcong/optimize-healthcheck
...
*: check health by using v3 range request and its corresponding timeout
2020-08-11 21:32:44 -07:00
8a4c7751d8
CHANGELOG: update for 12195
2020-08-12 08:10:13 +08:00
18adf55c92
Merge pull request #12199 from ptabor/20200803-expect-replace-fix
...
tests/e2e: Update github.com/creack/pty v1.1.7 -> v1.1.11
2020-08-11 11:59:12 -07:00
844091dda3
Merge pull request #12206 from ptabor/20200807-setLoggingDataRace
...
integration: Fix flakes due to .setupLogging race.
2020-08-11 11:58:47 -07:00
