6d1041dc62
Merge pull request #14128 from ahrtr/remove_ioutil
...
replace ioutil with os package
2022-06-17 15:03:07 +08:00
cbf54a0421
Use proper grpc schema (unix:file) for unix sockets.
...
Signed-off-by: Piotr Tabor <ptab@google.com >
2022-06-17 08:57:23 +02:00
17c1dcd614
Update grpc to 1.47 and go-cmp to 0.5.8.
...
Signed-off-by: Piotr Tabor <ptab@google.com >
2022-06-17 08:57:22 +02:00
fa68664a26
update 3.5 and 3.6 changelog to cover the fix of restricting the max size of each WAL entry to remaining size of the WAL file
...
Signed-off-by: Benjamin Wang <wachao@vmware.com >
2022-06-17 10:17:02 +08:00
8038e876d3
replace ioutil with os package
...
Signed-off-by: Benjamin Wang <wachao@vmware.com >
2022-06-17 10:11:30 +08:00
4c95878f00
Merge pull request #14122 from ahrtr/threshold_20220615
...
Restrict the max size of each WAL entry to the remaining size of the WAL file
2022-06-17 08:19:42 +08:00
ccf477d12b
restrict the max size of each WAL entry to the remaining size of the file
...
Currently the max size of each WAL entry is hard coded as 10MB. If users
set a value > 10MB for the flag --max-request-bytes, then etcd may run
into a situation that it successfully processes a big request, but fails
to decode it when replaying the WAL file on startup.
On the other hand, we can't just remove the limitation, because if a
WAL entry is somehow corrupted, and its recByte is a huge value, then
etcd may run out of memory. So the solution is to restrict the max size
of each WAL entry as a dynamic value, which is the remaining size of
the WAL file.
Signed-off-by: Benjamin Wang <wachao@vmware.com >
2022-06-17 05:41:49 +08:00
427372b6ee
Merge pull request #14123 from tsonglew/patch-1
...
fix: typo gouroutine
2022-06-16 13:28:45 +02:00
e5a80f5049
fix: typo gouroutine
...
fix: typo gouroutine
2022-06-16 16:35:06 +08:00
d23917c576
Add FileReader and FileBufReader utilities
...
The FileReader interface is the wrapper of io.Reader. It provides
the fs.FileInfo as well. The FileBufReader struct is the wrapper of
bufio.Reader, it also provides fs.FileInfo.
Signed-off-by: Benjamin Wang <wachao@vmware.com >
2022-06-16 15:32:16 +08:00
a1fe0b8ea3
Merge pull request #14116 from ptabor/20220613-embed-errors
...
Embed server should log errors (and not get stuck)
2022-06-15 22:37:40 +02:00
e61d431792
Merge pull request #14109 from SimFG/fifo_panic
...
schedule: Provide logs when the fifo job panic happens
2022-06-15 22:24:42 +02:00
509a01861c
Merge pull request #14117 from ptabor/20220614-gitignore
...
Extend .gitignore to ignore intelij config/generated files.
2022-06-15 22:10:38 +02:00
c4e7224946
Merge pull request #14121 from L2ncE/main
...
Fix a syntax error in a code comment
2022-06-15 15:41:30 -04:00
637afd359b
Fix a syntax error in a code comment
...
Signed-off-by: L2ncE <llance_24@foxmail.com >
2022-06-15 23:19:12 +08:00
d83925e357
schedule: Provide logs when the fifo job panic happens
...
To make the fifo scheduler better debuggability.
Signed-off-by: SimFG <1142838399@qq.com >
2022-06-15 20:58:17 +08:00
1c401581e3
Close channel of started servers even if embed server fails to start.
...
Before this change:
```
gwmux, err = sctx.registerGateway([]grpc.DialOption{grpc.WithInsecure()})
if err != nil { return err; }
```
caused the Etcd.Serve() method finish without sctx.serversC channel being closed.
The code closing Etcd got stuck waiting for all the servers to be running.
Signed-off-by: Piotr Tabor <ptab@google.com >
2022-06-15 13:50:17 +02:00
fcc8fce4d2
Expand logging in case of embed server not being able to successfully start.
...
So far the errors were directed to Etcd.Errc (channel) that is not being consumed in practice.
Signed-off-by: Piotr Tabor <ptab@google.com >
2022-06-15 13:50:17 +02:00
f9e9f50a13
Extend .gitignore to ignore intelij config/generated files.
...
Signed-off-by: Piotr Tabor <ptab@google.com >
2022-06-15 13:14:08 +02:00
71bba3c761
Merge pull request #14106 from SimFG/remove_case
...
wal: remove the repeated test case
2022-06-14 12:13:00 +02:00
402188a98c
wal: remove the repeated test case
...
The TestFilePipelineFailLockFile case maybe test the line 77 in the alloc function. But the LockFile will not throw an error when the file lock is hold. And TestFilePipelineFailPreallocate and TestFilePipelineFailLockFile are exactly the same except for the comments.
Signed-off-by: SimFG <1142838399@qq.com >
2022-06-14 17:26:20 +08:00
2c12954158
Merge pull request #14049 from serathius/compact-hash
...
Calculate hash during compaction
2022-06-14 11:10:19 +02:00
9612fc1194
tests: Unify TestCompactionHash and extend it to also Delete keys and Defrag
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:20:19 +02:00
f5eadf5452
tests: Add tests for HashByRev HTTP API
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:20:19 +02:00
1581eef888
tests: Add integration tests for compact hash
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:20:19 +02:00
0e739da9a4
server: Cache compaction hash for HashByRev API
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:20:18 +02:00
2b090e86a6
server: Extract hasher to separate interface
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:20:18 +02:00
80828b593a
server: Remove duplicated compaction revision
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:20:18 +02:00
34a02ba621
server: Return revision range that hash was calcualted for
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:20:17 +02:00
76d3c527ae
server: Store real rv range in hasher
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
8f7b053480
server: Move adjusting revision to hasher
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
19941654fe
server: Pass revision as int
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
887e53e61a
server: Calculate hash during compaction
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
7381c81a7e
server: Fix range in mock not returning same number of keys and values
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
08a43178fa
server: Move reading KV index inside scheduleCompaction function
...
Makes it easier to test hash match between scheduleCompaction and
HashByRev.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
638ca1006a
server: Return error from scheduleCompaction
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:26 +02:00
0f90359c4b
server: Refactor hasher
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:25 +02:00
e62c358793
server: Extract kvHash struct
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:25 +02:00
fcaf76dbca
server: Move unsafeHashByRev to new hash.go file
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:25 +02:00
0984878ae7
server: Extract unsafeHashByRev function
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:25 +02:00
39c6935c65
server: Test HashByRev values to make sure they don't change
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:25 +02:00
01e85be159
server: Cover corruptionMonitor with tests
...
Get 100% coverage on InitialCheck and PeriodicCheck functions to avoid
any mistakes.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:25 +02:00
7c35dadc25
server: Extract corruption detection to dedicated struct
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:19:24 +02:00
70fbc0b3e7
server: Extract triggerCorruptAlarm to function
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 18:04:24 +02:00
fc690531e2
Merge pull request #14108 from serathius/tracked
...
Use single "stage/tracked" label to prevent issue from getting stale
2022-06-13 13:03:41 +02:00
bce22b587c
Use single "stage/tracked" label to prevent issue from getting stale
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com >
2022-06-13 11:48:34 +02:00
4ce7a85188
Merge pull request #14097 from patrocinio/buildtools
...
Update build file to run with an argument
2022-06-13 10:55:54 +02:00
4592af25ba
Merge pull request #14107 from ahrtr/version_refactor_20220613
...
Get all constant etcd versions defined in a centralized place
2022-06-13 10:31:46 +02:00
5f8cd5bd39
Replace all etcd versions with the centralized definitions
...
We have already defined all the constant etcd versions in the
centralized place api/version/version.go. So we should replace all
the versions with the centralized definitions.
2022-06-13 14:08:39 +08:00
4f202a6429
Define all versions in a centralized place
...
The versions such as V3_5, V3_6, etc. are defined everywhere in
the repo. In this commit, we get all of the constant versions defined
in a centralized place, so that they can be reused by all cases.
2022-06-13 13:37:41 +08:00
