mirrors/etcd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
22,323 Commits 124 Branches 418 Tags
33d65fc90bcb1e9a284ee6c2eaa0179d0b9cdbe2
Commit Graph

62 Commits

Author SHA1 Message Date
Srujan Bharadwaj
5398519dc9 backend: remove ugly nil checks 
To remove the ugly nil checks by instantiating the logger with a no-op logger if it is nil.

Signed-off-by: Srujan Bharadwaj <srujanbharadwaj44@gmail.com>
 2024-12-18 03:04:12 +00:00
Matthieu MOREL
69efe31ea0 fix: enable gofumpt instead of gofmt linter in server 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-11-27 07:22:36 +01:00
Marek Siarkowicz
f77fa903cb Merge pull request #17757 from fuweid/should-fatal-if-panic 
storage/backend: fatal if there is panic during defrag
 2024-11-22 23:38:45 +01:00
Thomas Gosteli
97d19b6e26 build: fix govet shadow err 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-11-08 12:35:42 +01:00
Thomas Gosteli
a19d4087cc fix(defrag): close temp file in case of error 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-11-07 15:18:39 +01:00
Wei Fu
c438fcbafd storage/backend: fatal if there is panic during defrag 
We should exit as soon as possible if there is panic during defrag.
Because that tx might be closed. The inflight request might use invalid
tx and then panic as well. However, the real panic might be shadowed by
new panic. It's related to goroutine schedule. So, we should fatal here.

How to reproduce bbolt#715:

```diff
diff --git a/server/etcdserver/api/v3rpc/maintenance.go b/server/etcdserver/api/v3rpc/maintenance.go
index 4f55c7c74..0a91b4225 100644
--- a/server/etcdserver/api/v3rpc/maintenance.go
+++ b/server/etcdserver/api/v3rpc/maintenance.go
@@ -89,7 +89,13 @@ func NewMaintenanceServer(s *etcdserver.EtcdServer, healthNotifier notifier) pb.
 func (ms *maintenanceServer) Defragment(ctx context.Context, sr *pb.DefragmentRequest) (*pb.DefragmentResponse, error) {
        ms.lg.Info("starting defragment")
        ms.healthNotifier.defragStarted()
-       defer ms.healthNotifier.defragFinished()
+       defer func() {
+               ms.healthNotifier.defragFinished()
+               if rerr := recover(); rerr != nil {
+                       time.Sleep(30 * time.Second)
+                       panic(rerr)
+               }
+       }()
        err := ms.bg.Backend().Defrag()
        if err != nil {
                ms.lg.Warn("failed to defragment", zap.Error(err))
```

```bash
$ make gofail-enable
$ make
$ GOFAIL_HTTP="127.0.0.1:1234" bin/etcd

$ New Terminal
$ curl http://127.0.0.1:1234/defragBeforeRename -XPUT -d'panic()'
$ bin/etcdctl defrag

$ Old Terminal
{"level":"info","ts":"2024-04-09T23:28:54.084434+0800","caller":"v3rpc/maintenance.go:90","msg":"starting defragment"}
{"level":"info","ts":"2024-04-09T23:28:54.087363+0800","caller":"backend/backend.go:509","msg":"defragmenting","path":"default.etcd/member/snap/db","current-db-size-bytes":20480,"current-db-size":"20 kB","current-db-size-in-use-bytes":16384,"current-db-size-in-use":"16 kB"}
{"level":"info","ts":"2024-04-09T23:28:54.091229+0800","caller":"v3rpc/health.go:62","msg":"grpc service status changed","service":"","status":"SERVING"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xbb2553]

goroutine 156 [running]:
go.etcd.io/bbolt.(*Tx).Bucket(...)
        go.etcd.io/bbolt@v1.4.0-alpha.0/tx.go:112
go.etcd.io/etcd/server/v3/storage/backend.(*baseReadTx).UnsafeRange(0xc00061ac80, {0x12b7780, 0x1a73420}, {0x1a113e8, 0xe, 0xe}, {0x0, 0x0, 0x0}, 0x1)
        go.etcd.io/etcd/server/v3/storage/backend/read_tx.go:103 +0x233
go.etcd.io/etcd/server/v3/storage/schema.UnsafeReadStorageVersion({0x7f6280613618?, 0xc00061ac80?})
        go.etcd.io/etcd/server/v3/storage/schema/version.go:35 +0x5d
go.etcd.io/etcd/server/v3/storage/schema.UnsafeDetectSchemaVersion(0xc000334b80, {0x7f6280613618, 0xc00061ac80})
        go.etcd.io/etcd/server/v3/storage/schema/schema.go:94 +0x47
go.etcd.io/etcd/server/v3/storage/schema.DetectSchemaVersion(0xc000334b80, {0x12b77f0, 0xc00061ac80})
        go.etcd.io/etcd/server/v3/storage/schema/schema.go:89 +0xf2
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).StorageVersion(0xc000393c08)
        go.etcd.io/etcd/server/v3/etcdserver/server.go:2216 +0x105
go.etcd.io/etcd/server/v3/etcdserver.(*serverVersionAdapter).GetStorageVersion(0x0?)
        go.etcd.io/etcd/server/v3/etcdserver/adapters.go:77 +0x16
go.etcd.io/etcd/server/v3/etcdserver/version.(*Monitor).UpdateStorageVersionIfNeeded(0xc00002df70)
        go.etcd.io/etcd/server/v3/etcdserver/version/monitor.go:112 +0x5d
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).monitorStorageVersion(0xc000393c08)
        go.etcd.io/etcd/server/v3/etcdserver/server.go:2259 +0xa8
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
        go.etcd.io/etcd/server/v3/etcdserver/server.go:2440 +0x59
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach in goroutine 1
        go.etcd.io/etcd/server/v3/etcdserver/server.go:2438 +0xf9
```

REF: https://github.com/etcd-io/bbolt/issues/715

Signed-off-by: Wei Fu <fuweid89@gmail.com>
 2024-11-06 23:56:54 +00:00
Thomas Gosteli
04c042ceac fix(defrag): handle defragdb failure 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-11-06 12:43:51 +01:00
Thomas Gosteli
35cab80e1f fix(defrag): handle no space left error 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-11-06 11:01:17 +01:00
Matthieu MOREL
3a3115122e fix: enable require-error rule from testifylint in client, pkg and server packages 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-10-28 21:02:58 +01:00
Thomas Gosteli
ff4a8df3aa fix: enable formatter rule from testifylint 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-10-17 14:10:35 +02:00
Matthieu MOREL
33d7f2d53e fix: enable gofmt and whitespace linters 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-10-11 07:03:18 +02:00
Matthieu MOREL
e06fb81713 fix: enable empty and len rules from testifylint 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-10-10 11:07:21 +00:00
redwrasse
0739142ee3 server: errors.Is conversions 
Signed-off-by: redwrasse <mail@redwrasse.io>
 2024-09-29 05:23:47 -07:00
Jes Cok
dafadd13c1 all: don't convert byte slice to string when using verb %s 
This is unnecessary, as the documentation for 'go doc fmt' says:
%s the uninterpreted bytes of the string or slice

Signed-off-by: Jes Cok <xigua67damn@gmail.com>
 2024-04-25 23:34:52 +08:00
idnandre
12fa1bb940 server/storage: change deprecated bolt error to new error package 
Signed-off-by: idnandre <andre@idntimes.com>
 2024-04-03 19:21:22 +07:00
Siyuan Zhang
0a54362cca add key dedupe when a write buffer writeback to an empty read buffer bucket. 
Signed-off-by: Siyuan Zhang <sizhang@google.com>
 2024-03-27 17:03:31 +00:00
Siyuan Zhang
7be3606841 Add tx buffer test case of unordered key writes. 
Signed-off-by: Siyuan Zhang <sizhang@google.com>
 2024-03-27 17:03:31 +00:00
Siyuan Zhang
4346a43cee Add benchmark tests for buffer writeback function. 
Signed-off-by: Siyuan Zhang <sizhang@google.com>
 2024-03-27 17:03:26 +00:00
thirdkeyword
fbda591866 fix some typos 
Signed-off-by: thirdkeyword <fliterdashen@gmail.com>
 2024-03-25 10:34:44 +08:00
Ivan Valdes
66f56d71e4 server: address golangci var-naming issues 
Signed-off-by: Ivan Valdes <ivan@vald.es>
 2024-03-20 21:12:12 -07:00
Siyuan Zhang
3565a822de Add VerifyTxConsistency to backend. 
Signed-off-by: Siyuan Zhang <sizhang@google.com>

Update server/storage/backend/verify.go

Co-authored-by: Benjamin Wang <benjamin.wang@broadcom.com>

Update server/storage/backend/verify.go

Co-authored-by: Benjamin Wang <benjamin.wang@broadcom.com>
 2024-02-22 11:31:16 -08:00
Ivan Valdes
be28833212 etcdutl: Fix snapshot restore memory alloc issue 
When running the snapshot command, allow receiving an initial memory map
allocation for the database, avoiding future memory allocation issues.

Co-authored-by: Benjamin Wang <benjamin.wang@broadcom.com>
Co-authored-by: Fatih USTA <fatihusta86@gmail.com>
Signed-off-by: Ivan Valdes <ivan@vald.es>
 2024-02-13 13:36:13 -08:00
Ivan Valdes
d69adf45f9 server: Implement WithMmapSize option for backend config 
Accept a third argument for NewDefaultBackend for overrides to the
BackendConfig.
Add a new function, WithMmapSize, which modifies the backend config to
provide a custom InitiamMmapSize.

Signed-off-by: Ivan Valdes <ivan@vald.es>
 2024-02-13 13:24:44 -08:00
Benjamin Wang
c12e15d91a Add verification on keys: should be always mononically increasing 
Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
 2024-01-25 18:00:26 +00:00
Siyuan Zhang
30b242f800 Revert "add test cases of duplicate keys in one tx." 
This reverts commit 8d29b245a3.

Signed-off-by: Siyuan Zhang <sizhang@google.com>
 2024-01-19 08:58:48 -08:00
Siyuan Zhang
93afd8e430 Revert "add key dedupe when a write buffer writeback to an empty read buffer bucket." 
This reverts commit 971a4f57cc.

Signed-off-by: Siyuan Zhang <sizhang@google.com>
 2024-01-19 08:58:27 -08:00
Benjamin Wang
690916f018 replace CreateBucket with CreateBucketIfNotExists 
Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
 2024-01-16 09:40:20 +00:00
Benjamin Wang
9ded10ec76 Merge pull request #17237 from ahrtr/bbolt_logger_20240112 
Specify logger for bbolt
 2024-01-13 11:03:14 +00:00
Benjamin Wang
cfbf672b3c specify logger for bbolt 
Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
 2024-01-12 16:38:18 +00:00
Siyuan Zhang
971a4f57cc add key dedupe when a write buffer writeback to an empty read buffer bucket. 
Signed-off-by: Siyuan Zhang <sizhang@google.com>
 2024-01-11 11:31:09 -08:00
Siyuan Zhang
8d29b245a3 add test cases of duplicate keys in one tx. 
Signed-off-by: Siyuan Zhang <sizhang@google.com>
 2024-01-11 11:31:03 -08:00
Siyuan Zhang
db61c9695d commit bbolt transaction if there is any pending deleting operations 
Signed-off-by: Siyuan Zhang <sizhang@google.com>
 2024-01-10 09:33:21 -08:00
Siyuan Zhang
b3a5307640 add tests to test tx delete consistency. 
Signed-off-by: Siyuan Zhang <sizhang@google.com>
 2024-01-10 09:33:13 -08:00
Wei Fu
aa97484166 *: enable goimports in verify-lint 
Signed-off-by: Wei Fu <fuweid89@gmail.com>
 2023-09-21 21:14:09 +08:00
chenyahui
c0aa3b613b Use any instead of interface{} 
Signed-off-by: chenyahui <cyhone@qq.com>
 2023-09-17 17:41:58 +08:00
new-dream
b05d75ab0b server: optimizing memory overhead of copy operation in ConcurrentReadTxn 
Signed-off-by: new-dream <111836360+new-dream@users.noreply.github.com>
 2023-08-30 23:03:15 +08:00
Marek Siarkowicz
53cbd81009 Separate Writer interface from BatchTx interfaces 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2023-07-31 10:18:01 +02:00
Marek Siarkowicz
29769984e6 Remove RLock/RUnlock from BatchTx 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2023-07-28 11:39:50 +02:00
Marek Siarkowicz
b4f8a7be51 server: Remove Lock/Unlock from ReadTx 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2023-07-27 13:34:43 +02:00
caojiamingalan
bc97a94564 Follow up https://github.com/etcd-io/etcd/pull/16068#discussion_r1263664700 
Replace unnecessary Lock()/Unlock()s with RLock()/RUnlock()s

Signed-off-by: caojiamingalan <alan.c.19971111@gmail.com>
 2023-07-14 20:08:25 -05:00
Piotr Tabor
6f899a7b40 Merge pull request #15052 from ptabor/20221228-goimports-fix 
./scripts/fix.sh: Takes care of goimports across the whole project.
 2022-12-29 11:31:22 +01:00
Piotr Tabor
9e1abbab6e Fix goimports in all existing files. Execution of ./scripts/fix.sh 
Signed-off-by: Piotr Tabor <ptab@google.com>
 2022-12-29 09:41:31 +01:00
Wei Fu
4d0b91947e chore: delete // +build buildtag by go fix 
Signed-off-by: Wei Fu <fuweid89@gmail.com>
 2022-12-29 14:17:05 +08:00
Andrew Sims
f656fa0f49 add missing copyright headers 
Signed-off-by: Andrew Sims <andrew.cameron.sims@gmail.com>
 2022-11-23 19:13:43 +11:00
Benjamin Wang
4f824336ad etcdserver: add two failpoints for backend 
1. before and after create boltDB transaction;
2. before and after writebuf back to read buffer;

Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-11-15 08:09:05 +08:00
Benjamin Wang
3f18816e7d etcdserver: add gofail points before and after OnPreCommitUnsafe 
Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-11-14 11:02:18 +08:00
Benjamin Wang
5a3ef953eb etcdserver: call the OnPreCommitUnsafe in unsafeCommit 
`unsafeCommit` is called by both `(*batchTxBuffered) commit` and
`(*backend) defrag`. When users perform the defragmentation
operation, etcd doesn't update the consistent index. If etcd
crashes(e.g. panicking) in the process for whatever reason, then
etcd replays the WAL entries starting from the latest snapshot,
accordingly it may re-apply entries which might have already been
applied, eventually the revision isn't consistent with other members.

Refer to discussion in https://github.com/etcd-io/etcd/pull/14685

Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-11-11 10:57:15 +08:00
Piotr Tabor
d69e07dd3a Verification framework and check whether cindex is not decreasing. 2022-04-22 12:32:05 +02:00
ahrtr
4033f5c2b9 move the consistentIdx and consistentTerm from Etcdserver to cindex package 
Removed the fields consistentIdx and consistentTerm from struct EtcdServer,
and added applyingIndex and applyingTerm into struct consistentIndex in
package cindex. We may remove the two fields completely if we decide to
remove the OnPreCommitUnsafe, and it will depend on the performance test
result.
 2022-04-07 15:16:49 +08:00
ahrtr
e155e50886 rename LockWithoutHook to LockOutsideApply and add LockInsideApply 2022-04-07 05:35:13 +08:00