2c12954158
Merge pull request #14049 from serathius/compact-hash
...
Calculate hash during compaction
2022-06-14 11:10:19 +02:00
9612fc1194
tests: Unify TestCompactionHash and extend it to also Delete keys and Defrag
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:20:19 +02:00
0e739da9a4
server: Cache compaction hash for HashByRev API
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:20:18 +02:00
2b090e86a6
server: Extract hasher to separate interface
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:20:18 +02:00
80828b593a
server: Remove duplicated compaction revision
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:20:18 +02:00
34a02ba621
server: Return revision range that hash was calcualted for
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:20:17 +02:00
76d3c527ae
server: Store real rv range in hasher
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
8f7b053480
server: Move adjusting revision to hasher
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
19941654fe
server: Pass revision as int
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
887e53e61a
server: Calculate hash during compaction
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
7381c81a7e
server: Fix range in mock not returning same number of keys and values
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
08a43178fa
server: Move reading KV index inside scheduleCompaction function
...
Makes it easier to test hash match between scheduleCompaction and
HashByRev.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
638ca1006a
server: Return error from scheduleCompaction
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
0f90359c4b
server: Refactor hasher
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:25 +02:00
e62c358793
server: Extract kvHash struct
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:25 +02:00
fcaf76dbca
server: Move unsafeHashByRev to new hash.go file
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:25 +02:00
0984878ae7
server: Extract unsafeHashByRev function
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:25 +02:00
39c6935c65
server: Test HashByRev values to make sure they don't change
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:25 +02:00
7c35dadc25
server: Extract corruption detection to dedicated struct
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:24 +02:00
5f8cd5bd39
Replace all etcd versions with the centralized definitions
...
We have already defined all the constant etcd versions in the
centralized place api/version/version.go. So we should replace all
the versions with the centralized definitions.
2022-06-13 14:08:39 +08:00
b295cebc05
mvcc: improve the use of locks in index.go
...
To make the meaning of the RangeSince function in treeIndex clearer.
2022-06-09 20:35:02 +08:00
b073129d03
Applier does not depend on EtcdServer any longer.
...
All the depencies are explicily passed to the UberApplier factory method.
2022-05-20 14:32:04 +02:00
d69e07dd3a
Verification framework and check whether cindex is not decreasing.
2022-04-22 12:32:05 +02:00
a3650db574
use readTx in (*store).restore
2022-04-08 15:45:05 +08:00
1ea53d527e
server: Save consistency index and term to backend even when they decrease
...
Reason to store CI and term in backend was to make db fully independent
snapshot, it was never meant to interfere with apply logic. Skip of CI
was introduced for v2->v3 migration where we wanted to prevent it from
decreasing when replaying wal in
https://github.com/etcd-io/etcd/pull/5391 . By mistake it was added to
apply flow during refactor in
https://github.com/etcd-io/etcd/pull/12855#commitcomment-70713670 .
Consistency index and term should only be negotiated and used by raft to make
decisions. Their values should only driven by raft state machine and
backend should only be responsible for storing them.
2022-04-07 19:00:03 +02:00
4033f5c2b9
move the consistentIdx and consistentTerm from Etcdserver to cindex package
...
Removed the fields consistentIdx and consistentTerm from struct EtcdServer,
and added applyingIndex and applyingTerm into struct consistentIndex in
package cindex. We may remove the two fields completely if we decide to
remove the OnPreCommitUnsafe, and it will depend on the performance test
result.
2022-04-07 15:16:49 +08:00
e155e50886
rename LockWithoutHook to LockOutsideApply and add LockInsideApply
2022-04-07 05:35:13 +08:00
7ac995cdde
enhanced authBackend to support authReadTx
2022-04-07 05:35:13 +08:00
a4c5da844d
added detailed comment to explain the difference between Lock and LockWithoutHook
2022-04-07 05:35:13 +08:00
bfd5170f66
add a txPostLockHook into the backend
...
Previously the SetConsistentIndex() is called during the apply workflow,
but it's outside the db transaction. If a commit happens between SetConsistentIndex
and the following apply workflow, and etcd crashes for whatever reason right
after the commit, then etcd commits an incomplete transaction to db.
Eventually etcd runs into the data inconsistency issue.
In this commit, we move the SetConsistentIndex into a txPostLockHook, so
it will be executed inside the transaction lock.
2022-04-07 05:35:13 +08:00
ad03f2076a
Merge pull request #13886 from serathius/backend-logger
...
tests: Pass logger to backend
2022-04-05 16:35:07 +02:00
73fc864247
tests: Pass logger to backend
2022-04-05 15:53:38 +02:00
1d3517020b
server: Add verification of whether lock was called within out outside of apply
2022-04-05 15:34:45 +02:00
6c974a3e31
Merge pull request #13867 from serathius/logs-test
...
tests: Use zaptest.NewLogger in tests
2022-04-04 14:47:04 +02:00
804fddf921
tests: Use zaptest.NewLogger in tests
2022-04-04 13:03:15 +02:00
836bd6bc3a
fix WARNING: DATA RACE issue when multiple goroutines access the backend concurrently
2022-04-03 06:13:09 +08:00
9b3b383366
the file server/storage/mvcc/util.go isn't used at all, so removing it
2022-03-31 10:14:46 +08:00
3ecd8d2364
update incorrect migration log message
...
It can be upgrade or downgrade, so we should use word "updated" here.
2022-03-10 04:39:56 +08:00
d578a86a59
fix some log format typos
2022-03-10 03:26:34 +08:00
1ae5aa52de
fix some typos related to downgrade
2022-03-09 16:07:18 +08:00
124bc1ebbc
mvcc: clean up duplicate metrics name
2022-03-01 18:00:07 +08:00
0ecd4de923
delete duplicate metrics rangeCounterDebug
2022-02-28 16:22:40 +08:00
f80f477073
Merge pull request #13644 from Juneezee/refactor/t.TempDir
...
*: use `T.TempDir` to create temporary test directory
2022-02-21 19:52:37 +01:00
aa75fd0850
Merge pull request #13689 from AdamKorcz/fuzz7
...
server/storage/wal: fix oss-fuzz issue 44433
2022-02-14 15:48:55 -05:00
09e35a44ec
Merge pull request #13676 from chaochn47/fix_12535
...
Load all leases from backend
2022-02-14 21:32:33 +01:00
a0347261e4
Merge pull request #13679 from lavacat/defrag-bopts-fix
...
server/storage/backend: restore original bolt db options after defrag
2022-02-14 20:24:02 +01:00
0fcb2a628c
server/storage/wal: fix oss-fuzz issue 44433
2022-02-14 11:50:27 +00:00
fd79af9ee7
load all leases from backend
2022-02-13 20:30:20 -08:00
01347a8f53
server/storage/backend: restore original bolt db options after defrag
...
Problem: Defrag was implemented before custom bolt options were added.
Currently defrag doesn't restore backend options.
For example BackendFreelistType will be unset after defrag.
Solution: save bolt db options and use them in defrag.
2022-02-11 11:01:27 -08:00
029c354316
server/storage/mvcc: fix oss-fuzz issue 44449
2022-02-11 10:57:02 +00:00
