mirrors/etcd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
22,759 Commits 124 Branches 418 Tags 106 MiB
main
Commit Graph

23 Commits

Author SHA1 Message Date
Srujan Bharadwaj
5398519dc9 backend: remove ugly nil checks 
To remove the ugly nil checks by instantiating the logger with a no-op logger if it is nil.

Signed-off-by: Srujan Bharadwaj <srujanbharadwaj44@gmail.com>
 2024-12-18 03:04:12 +00:00
Matthieu MOREL
69efe31ea0 fix: enable gofumpt instead of gofmt linter in server 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-11-27 07:22:36 +01:00
Marek Siarkowicz
f77fa903cb
Merge pull request #17757 from fuweid/should-fatal-if-panic 
storage/backend: fatal if there is panic during defrag
 2024-11-22 23:38:45 +01:00
Thomas Gosteli
97d19b6e26 build: fix govet shadow err 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-11-08 12:35:42 +01:00
Thomas Gosteli
a19d4087cc fix(defrag): close temp file in case of error 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-11-07 15:18:39 +01:00
Wei Fu
c438fcbafd storage/backend: fatal if there is panic during defrag 
We should exit as soon as possible if there is panic during defrag.
Because that tx might be closed. The inflight request might use invalid
tx and then panic as well. However, the real panic might be shadowed by
new panic. It's related to goroutine schedule. So, we should fatal here.

How to reproduce bbolt#715:

```diff
diff --git a/server/etcdserver/api/v3rpc/maintenance.go b/server/etcdserver/api/v3rpc/maintenance.go
index 4f55c7c74..0a91b4225 100644
--- a/server/etcdserver/api/v3rpc/maintenance.go
+++ b/server/etcdserver/api/v3rpc/maintenance.go
@@ -89,7 +89,13 @@ func NewMaintenanceServer(s *etcdserver.EtcdServer, healthNotifier notifier) pb.
 func (ms *maintenanceServer) Defragment(ctx context.Context, sr *pb.DefragmentRequest) (*pb.DefragmentResponse, error) {
        ms.lg.Info("starting defragment")
        ms.healthNotifier.defragStarted()
-       defer ms.healthNotifier.defragFinished()
+       defer func() {
+               ms.healthNotifier.defragFinished()
+               if rerr := recover(); rerr != nil {
+                       time.Sleep(30 * time.Second)
+                       panic(rerr)
+               }
+       }()
        err := ms.bg.Backend().Defrag()
        if err != nil {
                ms.lg.Warn("failed to defragment", zap.Error(err))
```

```bash
$ make gofail-enable
$ make
$ GOFAIL_HTTP="127.0.0.1:1234" bin/etcd

$ New Terminal
$ curl http://127.0.0.1:1234/defragBeforeRename -XPUT -d'panic()'
$ bin/etcdctl defrag

$ Old Terminal
{"level":"info","ts":"2024-04-09T23:28:54.084434+0800","caller":"v3rpc/maintenance.go:90","msg":"starting defragment"}
{"level":"info","ts":"2024-04-09T23:28:54.087363+0800","caller":"backend/backend.go:509","msg":"defragmenting","path":"default.etcd/member/snap/db","current-db-size-bytes":20480,"current-db-size":"20 kB","current-db-size-in-use-bytes":16384,"current-db-size-in-use":"16 kB"}
{"level":"info","ts":"2024-04-09T23:28:54.091229+0800","caller":"v3rpc/health.go:62","msg":"grpc service status changed","service":"","status":"SERVING"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xbb2553]

goroutine 156 [running]:
go.etcd.io/bbolt.(*Tx).Bucket(...)
        go.etcd.io/bbolt@v1.4.0-alpha.0/tx.go:112
go.etcd.io/etcd/server/v3/storage/backend.(*baseReadTx).UnsafeRange(0xc00061ac80, {0x12b7780, 0x1a73420}, {0x1a113e8, 0xe, 0xe}, {0x0, 0x0, 0x0}, 0x1)
        go.etcd.io/etcd/server/v3/storage/backend/read_tx.go:103 +0x233
go.etcd.io/etcd/server/v3/storage/schema.UnsafeReadStorageVersion({0x7f6280613618?, 0xc00061ac80?})
        go.etcd.io/etcd/server/v3/storage/schema/version.go:35 +0x5d
go.etcd.io/etcd/server/v3/storage/schema.UnsafeDetectSchemaVersion(0xc000334b80, {0x7f6280613618, 0xc00061ac80})
        go.etcd.io/etcd/server/v3/storage/schema/schema.go:94 +0x47
go.etcd.io/etcd/server/v3/storage/schema.DetectSchemaVersion(0xc000334b80, {0x12b77f0, 0xc00061ac80})
        go.etcd.io/etcd/server/v3/storage/schema/schema.go:89 +0xf2
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).StorageVersion(0xc000393c08)
        go.etcd.io/etcd/server/v3/etcdserver/server.go:2216 +0x105
go.etcd.io/etcd/server/v3/etcdserver.(*serverVersionAdapter).GetStorageVersion(0x0?)
        go.etcd.io/etcd/server/v3/etcdserver/adapters.go:77 +0x16
go.etcd.io/etcd/server/v3/etcdserver/version.(*Monitor).UpdateStorageVersionIfNeeded(0xc00002df70)
        go.etcd.io/etcd/server/v3/etcdserver/version/monitor.go:112 +0x5d
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).monitorStorageVersion(0xc000393c08)
        go.etcd.io/etcd/server/v3/etcdserver/server.go:2259 +0xa8
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
        go.etcd.io/etcd/server/v3/etcdserver/server.go:2440 +0x59
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach in goroutine 1
        go.etcd.io/etcd/server/v3/etcdserver/server.go:2438 +0xf9
```

REF: https://github.com/etcd-io/bbolt/issues/715

Signed-off-by: Wei Fu <fuweid89@gmail.com>
 2024-11-06 23:56:54 +00:00
Thomas Gosteli
04c042ceac fix(defrag): handle defragdb failure 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-11-06 12:43:51 +01:00
Thomas Gosteli
35cab80e1f fix(defrag): handle no space left error 
Signed-off-by: Thomas Gosteli <thomas.gosteli@protonmail.ch>
 2024-11-06 11:01:17 +01:00
Jes Cok
dafadd13c1 all: don't convert byte slice to string when using verb %s 
This is unnecessary, as the documentation for 'go doc fmt' says:
%s the uninterpreted bytes of the string or slice

Signed-off-by: Jes Cok <xigua67damn@gmail.com>
 2024-04-25 23:34:52 +08:00
Ivan Valdes
be28833212
etcdutl: Fix snapshot restore memory alloc issue 
When running the snapshot command, allow receiving an initial memory map
allocation for the database, avoiding future memory allocation issues.

Co-authored-by: Benjamin Wang <benjamin.wang@broadcom.com>
Co-authored-by: Fatih USTA <fatihusta86@gmail.com>
Signed-off-by: Ivan Valdes <ivan@vald.es>
 2024-02-13 13:36:13 -08:00
Ivan Valdes
d69adf45f9
server: Implement WithMmapSize option for backend config 
Accept a third argument for NewDefaultBackend for overrides to the
BackendConfig.
Add a new function, WithMmapSize, which modifies the backend config to
provide a custom InitiamMmapSize.

Signed-off-by: Ivan Valdes <ivan@vald.es>
 2024-02-13 13:24:44 -08:00
Benjamin Wang
cfbf672b3c specify logger for bbolt 
Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
 2024-01-12 16:38:18 +00:00
Piotr Tabor
9e1abbab6e Fix goimports in all existing files. Execution of ./scripts/fix.sh 
Signed-off-by: Piotr Tabor <ptab@google.com>
 2022-12-29 09:41:31 +01:00
Benjamin Wang
4f824336ad etcdserver: add two failpoints for backend 
1. before and after create boltDB transaction;
2. before and after writebuf back to read buffer;

Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-11-15 08:09:05 +08:00
ahrtr
4033f5c2b9 move the consistentIdx and consistentTerm from Etcdserver to cindex package 
Removed the fields consistentIdx and consistentTerm from struct EtcdServer,
and added applyingIndex and applyingTerm into struct consistentIndex in
package cindex. We may remove the two fields completely if we decide to
remove the OnPreCommitUnsafe, and it will depend on the performance test
result.
 2022-04-07 15:16:49 +08:00
ahrtr
e155e50886 rename LockWithoutHook to LockOutsideApply and add LockInsideApply 2022-04-07 05:35:13 +08:00
ahrtr
bfd5170f66 add a txPostLockHook into the backend 
Previously the SetConsistentIndex() is called during the apply workflow,
but it's outside the db transaction. If a commit happens between SetConsistentIndex
and the following apply workflow, and etcd crashes for whatever reason right
after the commit, then etcd commits an incomplete transaction to db.
Eventually etcd runs into the data inconsistency issue.

In this commit, we move the SetConsistentIndex into a txPostLockHook, so
it will be executed inside the transaction lock.
 2022-04-07 05:35:13 +08:00
Marek Siarkowicz
73fc864247 tests: Pass logger to backend 2022-04-05 15:53:38 +02:00
ahrtr
836bd6bc3a fix WARNING: DATA RACE issue when multiple goroutines access the backend concurrently 2022-04-03 06:13:09 +08:00
Bogdan Kanivets
01347a8f53 server/storage/backend: restore original bolt db options after defrag 
Problem: Defrag was implemented before custom bolt options were added.
Currently defrag doesn't restore backend options.
For example BackendFreelistType will be unset after defrag.

Solution: save bolt db options and use them in defrag.
 2022-02-11 11:01:27 -08:00
Eng Zer Jun
2a151c8982
*: move from io/ioutil to io and os packages 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-28 00:05:28 +08:00
Geeta Gharpure
817d2f40d1 storage/backend: Add a gauge to indicate if defrag is active 2021-09-27 17:02:13 -07:00
Marek Siarkowicz
5e40a8b00c server: Create storage package and move mvcc files to it 2021-07-12 15:37:21 +02:00