mirrors/etcd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
22,323 Commits 124 Branches 418 Tags
33d65fc90bcb1e9a284ee6c2eaa0179d0b9cdbe2
Commit Graph

276 Commits

Author SHA1 Message Date
Benjamin Wang
152de1fa7e Still return continuous WAL entries when running into ErrSliceOutOfRange 
Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
 2024-12-21 15:33:14 +00:00
Benjamin Wang
9db12c6d02 Merge pull request #19081 from ahrtr/log_migrage_20241218 
Add more info in the error message in downgrade detection
 2024-12-19 09:41:23 +00:00
Benjamin Wang
2830237e33 Add more info in the error message in downgrade detection 
Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
 2024-12-18 15:23:12 +00:00
Srujan Bharadwaj
5398519dc9 backend: remove ugly nil checks 
To remove the ugly nil checks by instantiating the logger with a no-op logger if it is nil.

Signed-off-by: Srujan Bharadwaj <srujanbharadwaj44@gmail.com>
 2024-12-18 03:04:12 +00:00
Benjamin Wang
9143bef424 Add more debug info when running into ErrSliceOutOfRange when reading WAL 
Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
 2024-12-16 10:36:30 +00:00
Benjamin Wang
50e7f9f697 Add a TODO comment for UnsafeDetectSchemaVersion on how to simplify the implementation 
Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
 2024-12-12 14:49:14 +00:00
Matthieu MOREL
647f1621d6 fix: enable gosimple linter 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-12-03 07:32:22 +01:00
Marek Siarkowicz
348c0cb2be Reuse events between sync loops 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2024-12-02 18:07:25 +01:00
Marek Siarkowicz
1f4439c2eb Extract rangeEvents function 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2024-12-02 18:06:59 +01:00
Marek Siarkowicz
3741000161 Use number of revisions instead of batches to configure TestWatchBatchUnsynced 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2024-12-01 21:02:48 +01:00
Marek Siarkowicz
2a415cc62b Add scenario for 1 events per revison TestWatchBatchUnsynced 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2024-12-01 21:02:48 +01:00
Marek Siarkowicz
e88802ac2a Use table tests for TestWatchBatchUnsynced 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2024-12-01 21:02:48 +01:00
Marek Siarkowicz
1e655bf039 Make batching in TestWatchBatchUnsynced explicit 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2024-12-01 21:02:48 +01:00
Marek Siarkowicz
8c5e42d41b Use assert in TestWatchBatchUnsynced 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2024-12-01 21:02:48 +01:00
Marek Siarkowicz
06a5cd6eb3 Simplify TestSyncWatchers to make recepie for more tests 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2024-12-01 21:02:48 +01:00
Marek Siarkowicz
52ea2f83c5 Use constructor in tests 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2024-12-01 21:02:48 +01:00
Matthieu MOREL
69efe31ea0 fix: enable gofumpt instead of gofmt linter in server 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-11-27 07:22:36 +01:00
Marek Siarkowicz
f77fa903cb Merge pull request #17757 from fuweid/should-fatal-if-panic 
storage/backend: fatal if there is panic during defrag
 2024-11-22 23:38:45 +01:00
Thomas Gosteli
97d19b6e26 build: fix govet shadow err 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-11-08 12:35:42 +01:00
Thomas Gosteli
a19d4087cc fix(defrag): close temp file in case of error 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-11-07 15:18:39 +01:00
Wei Fu
c438fcbafd storage/backend: fatal if there is panic during defrag 
We should exit as soon as possible if there is panic during defrag.
Because that tx might be closed. The inflight request might use invalid
tx and then panic as well. However, the real panic might be shadowed by
new panic. It's related to goroutine schedule. So, we should fatal here.

How to reproduce bbolt#715:

```diff
diff --git a/server/etcdserver/api/v3rpc/maintenance.go b/server/etcdserver/api/v3rpc/maintenance.go
index 4f55c7c74..0a91b4225 100644
--- a/server/etcdserver/api/v3rpc/maintenance.go
+++ b/server/etcdserver/api/v3rpc/maintenance.go
@@ -89,7 +89,13 @@ func NewMaintenanceServer(s *etcdserver.EtcdServer, healthNotifier notifier) pb.
 func (ms *maintenanceServer) Defragment(ctx context.Context, sr *pb.DefragmentRequest) (*pb.DefragmentResponse, error) {
        ms.lg.Info("starting defragment")
        ms.healthNotifier.defragStarted()
-       defer ms.healthNotifier.defragFinished()
+       defer func() {
+               ms.healthNotifier.defragFinished()
+               if rerr := recover(); rerr != nil {
+                       time.Sleep(30 * time.Second)
+                       panic(rerr)
+               }
+       }()
        err := ms.bg.Backend().Defrag()
        if err != nil {
                ms.lg.Warn("failed to defragment", zap.Error(err))
```

```bash
$ make gofail-enable
$ make
$ GOFAIL_HTTP="127.0.0.1:1234" bin/etcd

$ New Terminal
$ curl http://127.0.0.1:1234/defragBeforeRename -XPUT -d'panic()'
$ bin/etcdctl defrag

$ Old Terminal
{"level":"info","ts":"2024-04-09T23:28:54.084434+0800","caller":"v3rpc/maintenance.go:90","msg":"starting defragment"}
{"level":"info","ts":"2024-04-09T23:28:54.087363+0800","caller":"backend/backend.go:509","msg":"defragmenting","path":"default.etcd/member/snap/db","current-db-size-bytes":20480,"current-db-size":"20 kB","current-db-size-in-use-bytes":16384,"current-db-size-in-use":"16 kB"}
{"level":"info","ts":"2024-04-09T23:28:54.091229+0800","caller":"v3rpc/health.go:62","msg":"grpc service status changed","service":"","status":"SERVING"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xbb2553]

goroutine 156 [running]:
go.etcd.io/bbolt.(*Tx).Bucket(...)
        go.etcd.io/bbolt@v1.4.0-alpha.0/tx.go:112
go.etcd.io/etcd/server/v3/storage/backend.(*baseReadTx).UnsafeRange(0xc00061ac80, {0x12b7780, 0x1a73420}, {0x1a113e8, 0xe, 0xe}, {0x0, 0x0, 0x0}, 0x1)
        go.etcd.io/etcd/server/v3/storage/backend/read_tx.go:103 +0x233
go.etcd.io/etcd/server/v3/storage/schema.UnsafeReadStorageVersion({0x7f6280613618?, 0xc00061ac80?})
        go.etcd.io/etcd/server/v3/storage/schema/version.go:35 +0x5d
go.etcd.io/etcd/server/v3/storage/schema.UnsafeDetectSchemaVersion(0xc000334b80, {0x7f6280613618, 0xc00061ac80})
        go.etcd.io/etcd/server/v3/storage/schema/schema.go:94 +0x47
go.etcd.io/etcd/server/v3/storage/schema.DetectSchemaVersion(0xc000334b80, {0x12b77f0, 0xc00061ac80})
        go.etcd.io/etcd/server/v3/storage/schema/schema.go:89 +0xf2
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).StorageVersion(0xc000393c08)
        go.etcd.io/etcd/server/v3/etcdserver/server.go:2216 +0x105
go.etcd.io/etcd/server/v3/etcdserver.(*serverVersionAdapter).GetStorageVersion(0x0?)
        go.etcd.io/etcd/server/v3/etcdserver/adapters.go:77 +0x16
go.etcd.io/etcd/server/v3/etcdserver/version.(*Monitor).UpdateStorageVersionIfNeeded(0xc00002df70)
        go.etcd.io/etcd/server/v3/etcdserver/version/monitor.go:112 +0x5d
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).monitorStorageVersion(0xc000393c08)
        go.etcd.io/etcd/server/v3/etcdserver/server.go:2259 +0xa8
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
        go.etcd.io/etcd/server/v3/etcdserver/server.go:2440 +0x59
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach in goroutine 1
        go.etcd.io/etcd/server/v3/etcdserver/server.go:2438 +0xf9
```

REF: https://github.com/etcd-io/bbolt/issues/715

Signed-off-by: Wei Fu <fuweid89@gmail.com>
 2024-11-06 23:56:54 +00:00
Thomas Gosteli
04c042ceac fix(defrag): handle defragdb failure 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-11-06 12:43:51 +01:00
Thomas Gosteli
35cab80e1f fix(defrag): handle no space left error 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-11-06 11:01:17 +01:00
Benjamin Wang
6883308899 Merge pull request #18800 from mmorel-35/testifylint/require-error 
fix: enable require-error rule from testifylint in client, pkg and server packages
 2024-10-30 23:03:01 +00:00
Matthieu MOREL
3a3115122e fix: enable require-error rule from testifylint in client, pkg and server packages 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-10-28 21:02:58 +01:00
Matthieu MOREL
3abdf612b3 fix: enable errorlint in server directory 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-10-26 08:16:47 +02:00
Thomas Gosteli
ff4a8df3aa fix: enable formatter rule from testifylint 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-10-17 14:10:35 +02:00
Matthieu MOREL
33d7f2d53e fix: enable gofmt and whitespace linters 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-10-11 07:03:18 +02:00
Matthieu MOREL
e06fb81713 fix: enable empty and len rules from testifylint 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-10-10 11:07:21 +00:00
Matthieu MOREL
b23947b604 fix: enable bool-compare rule from testifylint 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-10-06 09:47:02 +02:00
redwrasse
0739142ee3 server: errors.Is conversions 
Signed-off-by: redwrasse <mail@redwrasse.io>
 2024-09-29 05:23:47 -07:00
Lucas Rodriguez
680eadf0d6 Add function to create WAL files 
Signed-off-by: Lucas Rodriguez <lucas.rodriguez9616@gmail.com>
 2024-09-24 12:55:24 -05:00
Lucas Rodriguez
d12bb7e0fb Ensure consistent file permissions on broken WAL 
Signed-off-by: Lucas Rodriguez <lucas.rodriguez9616@gmail.com>
 2024-09-16 14:07:16 -05:00
Jalin Wang
2c53be7c5d etcdserver: rename defaultCompactionSleepInterval var (#18495) 
* etcdserver: rename `minimumBatchInterval`  to `defaultCompactionSleepInterval` and `defaultCompactBatchLimit` to `defaultCompactionBatchLimit`

Signed-off-by: Jalin Wang <JalinWang@outlook.com>
 2024-08-27 14:36:06 +01:00
Wei Fu
bbdc94181a *: keep tombstone if revision == compactAtRev 
Before this patch, the tombstone can be deleted if its revision is equal
compacted revision. It causes that the watch subscriber won't get this
DELETE event. Based on Compact API[1], we should keep tombstone revision
if it's not less than the compaction revision.

> CompactionRequest compacts the key-value store up to a given revision.
> All superseded keys with a revision less than the compaction revision
> will be removed.

[1]: https://etcd.io/docs/latest/dev-guide/api_reference_v3/

Signed-off-by: Wei Fu <fuweid89@gmail.com>
 2024-08-07 21:56:20 +08:00
Wei Fu
5e178e2288 mvcc/*_test.go: should not use duplicate revision.Main for one key 
In commit [[1]], the newTestKeyIndex function creates one key with two
Revision{Main: 14} revisions. However, starting from version [[2]], etcd server
does not allow duplicate keys in a single transaction. This update to
newTestKeyIndex is to avoid confusion and ensure consistency with the
latest etcd server behavior.

REF:

[1]: be80d11948
[2]: https://github.com/etcd-io/etcd/pull/4376

Signed-off-by: Wei Fu <fuweid89@gmail.com>
 2024-07-14 17:30:47 +08:00
Chun-Hung Tseng
5be397a6a8 Fix a linter issue 
golangci-lint reports the following issue:
storage/mvcc/kvstore.go:312:27: (*store).restore - result 0 (error) is always nil (unparam)

It's due to the fact that both Attach() and compactLockfree() within the
function restore() are able to return an error, but we only log them in
the current implementation. Thus, the return value restore() is always
nil, hence the linter warning.

We have agreed to suppress the linter warning for now [1].

Reference:
[1] https://github.com/etcd-io/etcd/pull/18228#issuecomment-2187309957

Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
 2024-06-25 11:45:53 +02:00
MR_G
c0076a788e mvcc: fix typo 
Signed-off-by: MR_G <2044783677@qq.com>
 2024-06-22 00:17:13 +08:00
Cenk Alti
ea46253b42 Improve snapshot status 
Signed-off-by: Cenk Alti <cenkalti@gmail.com>
 2024-06-15 10:11:34 -04:00
Benjamin Wang
45f14539e0 Merge pull request #18164 from andyxning/bugfix_register_of_walWriteSec 
bugfix: register of walWriteSec
 2024-06-14 18:32:15 +01:00
lubronzhan
0b6529c462 mvcc: fix typo 
Signed-off-by: lubronzhan <lubronzhan@gmail.com>
 2024-06-13 00:15:37 -07:00
Andy Xie
4a555fead3 bugfix: register of walWriteSec 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2024-06-13 15:12:58 +08:00
Cenk Alti
b107d2437f cli: Add etcdutl snapshot hashkv command 
Signed-off-by: Cenk Alti <cenkalti@gmail.com>

Apply suggestions from code review

Co-authored-by: Benjamin Wang <benjamin.wang@broadcom.com>
 2024-06-02 00:07:50 -04:00
Benjamin Wang
09f7d7ab92 Merge pull request #17875 from ivanvc/fix-deprecation-comments-added-with-var-naming-lint-rule 
Fix deprecation comments from addressing var-naming work
 2024-04-30 19:51:37 +01:00
Jes Cok
dafadd13c1 all: don't convert byte slice to string when using verb %s 
This is unnecessary, as the documentation for 'go doc fmt' says:
%s the uninterpreted bytes of the string or slice

Signed-off-by: Jes Cok <xigua67damn@gmail.com>
 2024-04-25 23:34:52 +08:00
Ivan Valdes
242f5d60a8 Fix deprecation comments from addressing var-naming work 
The deprecation tag/comment must to be followed by an empty line.

Signed-off-by: Ivan Valdes <ivan@vald.es>
 2024-04-24 14:40:59 -07:00
Wei Fu
94c83a962b server/storage/mvcc: should update currentRev in revMu 
Signed-off-by: Wei Fu <fuweid89@gmail.com>
 2024-04-24 11:08:33 +08:00
Ivan Valdes
a2bf8d7e80 server/config: address golangci var-naming issues 
Addresses issues in V2 Deprecation constant names.

Signed-off-by: Ivan Valdes <ivan@vald.es>
 2024-04-22 17:12:15 -07:00
Benjamin Wang
dd4e35a585 Merge pull request #17815 from fuweid/repro-17780 
[RFC] fix revision loss issue caused by compaction - 17780
 2024-04-22 18:15:44 +01:00
Benjamin Wang
29529c505b Merge pull request #17791 from ahrtr/verify_revision_bootstrap_20240415 
Add verification that the revision shouldn't decrease on bootstrap
 2024-04-22 10:43:01 +01:00