Currently when triggering release, it always pull remote repo and
checkout main branch. Any changes which are merged into the target
release branch (e.g. release-3.5) will be ignored. It isn't
convenient for test, including in github workflow and local environment.
So we need to support triggering release in current branch.
Note: --current-branch should only be called with DRY_RUN=true
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Previously etcdservers depends on raft/raftpb/raft.proto directly.
After moving raft to a separate repo, we need to add raft to the
tools/mod, and get raft included in the -I protc flags.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
No need to generate proto file;
No need to test coverage for raft;
No need to run any test for raft module;
NO need to run any test for raftexample;
Signed-off-by: Benjamin Wang <wachao@vmware.com>
The shell func go_srcs_in_module will now list
1. go src files
2. go test files belong to same packages
3. go test files that belong to _test packages
Fixes#14827
Signed-off-by: Bhargav Ravuri <bhargav.ravuri@infracloud.io>
The shell function go_srcs_in_module will list go test files
along with go src files in the specified module. This helps
in identifying the copyright header misses in test files.
Fixes#14827
Signed-off-by: Bhargav Ravuri <bhargav.ravuri@infracloud.io>
This changes the builds to always add -trimpath which removes specific
build time paths from the binary (like current directories etc).
Improves build reproducability to make the final binary independent from
the specific build path.
Lastly, when stripping debug symbols, also add -w to strip DWARF symbols
as well which aren't needed in that case either.
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
It isn't valid to start with quote character ' for flag -ldflags.
Go cmd older than 1.19 just ignores the error. Starting from go 1.19,
Go cmd will fail with error message something like below,
```
stderr: invalid value "'-X=go.etcd.io/etcd/api/v3/version.GitSHA=01250c9'" for flag -ldflags: parameter may not start with quote character '
```
Actually we don't have such quote character ' when building
etcd/etcdctl/etcdutl.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Problem: both SIGQUIT_ETCD_AND_REMOVE_DATA_AND_STOP_AGENT and test.sh
will attempt to stop agents and remove directories.
Solution: since test.sh creates directories and starts test, it should be
responsible for cleanup.
See https://github.com/etcd-io/etcd/issues/14384
Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
There are two cases, when interrupted by users, then forcibly kill
all processes. Otherwise, gracefully terminate all processes.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
The current Makefile doesn't allow the compilation of the tools directory.
This commit creates a build library file, updates the Makefile
and a top level script fod building tools.
To build the tools, you can run make build_tools.
As before, you can run make build to build etcd binaries.
The current Makefile doesn't allow the compilation of the tools directory.
This commit updates the Makefile and the build file to add an option
to build the tools.
To build the tools, you can run make build_tools.
This PR removes additional clone when building artifacts.
When releasing v3.5.4 this clone was main cause of issues and
confusion about what release script is doing.
release.sh script already clones repo in /tmp/ directory, so clonning
before build is not needed. As precautions for bug in script leaving
/tmp/ clone in bad state I moved "Verify the latest commit has the
version tag" and added "Verify the clean working tree" to be always run
before build.
I think strong (not-equal) relationship was too restrictive when expressed with 1s granularity.
```
logger.go:130: 2022-04-03T22:15:15.242+0200 WARN m1 leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"member": "m1", "to": "cb785755eb80ac1", "heartbeat-interval": "10ms", "expected-duration": "20ms", "exceeded-duration": "24.666613ms"}
logger.go:130: 2022-04-03T22:15:15.262+0200 INFO m-1 published local member to cluster through raft {"member": "m-1", "local-member-id": "e2dd9f523aa7be87", "local-member-attributes": "{Name:m-1 ClientURLs:[unix://127.0.0.1:2196386040]}", "cluster-id": "b4b8e7e41c23c8b5", "publish-timeout": "5.2s"}
v3_lease_test.go:415: Expected lease ttl (4m58s) to be greather than (4m58s)
```
When performing the downgrade operation, users can confirm whether each member
is ready to be downgraded using the field 'storageVersion'. If it's equal to the
'target version' in the downgrade command, then it's ready to be downgraded;
otherwise, the etcd member is still in progress of processing the db file.
The repository github.com/myitcv/gobin has already been archived,
and the `go install` command accepts arguments with version suffixs
starting from 1.16 (for example, go install example.com/cmd@v1.0.0).
So there is no reason to continue to use gobin.
This has been additionally verified by running the tests locally as a
basic smoke test. GitHub Actions doesn't provide MacOS M1 (arm64) yet,
so there's no good way to automate testing.
Ran `TMPDIR=/tmp make test` locally. The `TMPDIR` bit is needed so
there's no really long path used that breaks Unix socket setup in one of
the tests.
Motivation is as follows:
- etcdctl we only depend on clientv3 APIs, no dependencies of bolt, backend, mvcc, file-layout
- etcdctl can be officially supported across wide range of versions, while etcdutl is pretty specific to file format at particular version.
it's step towards desired modules layout, documented in: https://etcd.io/docs/next/dev-internal/modules/
Seems that additinal refresh of go.sum files is needed during the
release process after editing the versions of modules.
Discovered the problem while I was testing whether etcdutl is
properly included in the target binaries using:
```
DRY_RUN=true REPOSITORY=\`pwd\` BRANCH='branch-with-etcdutl' ./scripts/release 3.5.0-foobar.2
```
While it appears that etcd is not vulnerable to CVE-2021-3121,
it is a good idea to update to the new generator so that new
vulnerable code isn't generated in any future APIs. Also, this
lays the issue to rest of whether there is any issue with
etcd and CVE-2021-3121.
Seems old versions of https://github.com/certifi/gocertifi where
categorized as "Do What The F*ck You Want To Public License".
Update to newer version that is explicit `Mozilla Public License` 2.0 (MPL 2.0).
- Fix script that creates manifest-list based multi-arch-images.
The images need to be pushed first.
- Use docker instead of gcloud docker helper
- Make sure docker pushes are properly 'dry run'
- Added preparation instruction to the release script.
- making sure the DRY_RUN mode can finish e2e, so e.g. commits to
local copy of repository are OK in dry-run (while git pushes are NOT).
- better interaction with ./test_lib.sh script.
- more consistent logging
- bringing back s390x architecture that on go 1.14.3 seems to work as
expected.
Change amd64 docker image to use a `-amd64` suffix, consistent with
other architectures. Introduce a multiarch docker manifest list with
the original (no-arch) docker tag.
Fixes#10380
Changes:
- signing tags.
- allows to override BRANCH and REPOSITORY using env variables.
Tested by a release in my private fork:
BRANCH="20201126-ptabor-release" REPOSITORY="git@github.com:ptabor/etcd.git" ./scripts/release 3.5.0-alpha.20
- build & test scripts deprecated. Call *.sh variants.
This will avoid delete the symlinks and get rid of
subtle dependency on 'sed --follow-symlinks' on OsX/BSD sed.
- Fix parameters to mktemp
Examplar invocations:
Edit go.mod files such that all etcd modules are pointing on given version:
```
% DRY_RUN=false TARGET_VERSION="v3.5.13" ./scripts/release_mod.sh update_versions
```
Tag latest commit with current version number for all the modules and push upstream:
```
% DRY_RUN=true REMOTE_REPO="origin" ./scripts/release_mod.sh push_mod_tags
```
* Turn off checking of format of commit message.
* scripts/fix.sh: Fix fixing whitespaces in *.sh scripts
Aparently there is a difference between:
find ./ -print0 -name *.sh and
find ./ -name *.sh -print0
* etcdserver unit tests: Do not call .Fatalf(...) from not test's goroutine.
Fixes following test failures:
https://travis-ci.com/github/etcd-io/etcd/jobs/425920416
```
% (cd server && go vet ./...)
stderr: # go.etcd.io/etcd/server/v3/etcdserver
stderr: etcdserver/server_test.go:1002:4: call to (*T).Fatalf from a non-test goroutine
stderr: etcdserver/server_test.go:1166:4: call to (*T).Fatalf from a non-test goroutine
FAIL: (code:2):
% (cd server && go vet ./...)
FAIL: 'run go vet ./...' checking failed (!=0 return code)
FAIL: 'govet' failed at Tue Nov 3 04:07:47 UTC 2020
```
We make v2 client code a module go.etcd.io/etcd/client/v2.
Pretty mechanical change that can be summarized as:
mkdir client/v2
cd client/v2 && git mod init go.etcd.io/etcd/client/v2
git mv client/*.go client/v2/
find -name '*.go' | xargs sed -i --follow-symlinks 's|/v3/client["]|/client/v2\"|g'
+ fixing changelog, bom, go.mod etc.
This brings consistency between proto-generation code and actual versions of libraries being used in runtime:
github.com/gogo/protobuf v1.2.1,v1.0.0 -> v1.3.1
github.com/golang/protobuf v1.3.2 -> v1.3.5
github.com/grpc-ecosystem/grpc-gateway v1.9.5,v1.4.1,v1.15.2 -> v1.14.6
google.golang.org/grpc v1.26.0 -> v1.29.1
Moved as far as possible, without bumping on grpc 1.30.0 "naming" decomissioning.
Please also notice that gogo/protobuf is likely to reach EOL: https://github.com/gogo/protobuf/issues/691
Replace ./scripts/install_tool.sh with `gobin`, such that we have
consistent handling for all tools needed for build and consistent
versioning within ./tools/mod/go.mod.
Side changes:
- Expose /scripts/fix.sh that fixes formatting and bom across modules
- Expose *.sh variants of scripts like build and ./test (first step
towards replacement).
- Make stderr output of commands explicit and make commands use
different color than callouts.
During move of code to 'api' (0aab02e7b5),
I overlooked that the ./build script is setting the version on the 'symbol'.
I added a code to ./build script that checks whether the symbol's are in sync.
Refactoring script/genproto.sh around state-of-the-art techniques of
managing tooling in go:
- https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module
- uses https://github.com/myitcv/gobin instead of customly created gopath.proto dir
- caches tools between executions
- guaratees hermetics runs (it was not guaranteed for protoc_grpc_gateway that used latest)
The change is no-op for the generated code.
The commit reveals a few 'worring things':
1 We depend on : github.com/grpc-ecosystem/grpc-gateway/@v/v1.4.1/protoc-gen-grpc-gateway
2. And also : github.com/grpc-ecosystem/grpc-gateway/@v/v1.15.0/protoc-gen-swagger/protoc-gen-swagger
3. And on extremely old: github.com/gogo/protobuf@v1.0.0 protoc-gen-gofast that is out of sync with the library linked to binaries: github.com/gogo/protobuf@v1.2.1
The module is supposed to contain minimal set of files that establish
public etcd server API. In particular client libraries for etcd built in
different languages might want to depend on this file.
1. setting environment variable cannot be in quote
2. "--race" testing for unit tests is supposed to be part of linux-amd64-unit-4-cpu-race config.
3. 'run' function in test script should log_error in case of failed
command (wrong operator for ints comparison in bash).
This refactoring offers following benefits:
- Unified way how go test commands are being called (in terms of flags intepretation)
- Uses standard go mechanisms (like go lists) to find files/packages that are subject for test. The mechanism are module aware.
- Added instruction how to install tools needed for the tests/checkers.
- Added colors to the output to make it easier to spot any failure.
Confirmed to work using:
- COVERDIR="./coverage" CPU="4" RACE=false COVER=false PASSES="build build_cov cov" ./test
- CPU="4" RACE=false COVER=false PASSES="e2e functional integration" ./test
- COVERDIR="./coverage" COVER="false" CPU="4" RACE="false" PASSES="fmt build unit build_cov integration e2e integration_e2e grpcproxy cov" ./test
- PASSES=unit PKG=./wal TIMEOUT=1m ./test
- PASSES=integration PKG=./clientv3 TIMEOUT=1m ./test
- PASSES=unit PKG=./wal TESTCASE=TestNew TIMEOUT=1m ./test
- PASSES=unit PKG=./wal TESTCASE="\bTestNew\b" TIMEOUT=1m ./test
- PASSES=integration PKG=./client/integration TESTCASE="\bTestV2NoRetryEOF\b" TIMEOUT=1m ./test
- COVERDIR=coverage PASSES="build_cov cov" ./test
As we get rid of ./vendor there is no need to update the dependencies.
If anyone needs up-to-date vendor directory locally, getting it is as
simple as:
```go mod vendor```
