Commit Graph

159 Commits

Author SHA1 Message Date
7b8fb3cf0a mvcc: add and implement Keep api to index
Keep finds all revisions to be kept for a Compaction at the given rev.
2017-07-31 14:04:03 -07:00
451b062184 mvcc/backend: add TestBackendWritebackForEach to backend_test.go 2017-07-28 09:39:48 -07:00
785deebd62 mvcc/backend: enforce ordering for UnsafeForEach in read_tx.go
This pr changes  UnsafeForEach to traverse on boltdb before on the buffer.
This ordering guarantees that UnsafeForEach traverses in the same order
before or after the commit of buffer.
2017-07-28 09:30:23 -07:00
2a348fb8e9 Merge pull request #8263 from fanminshi/hash_by_rev
api: hash by rev
2017-07-26 11:22:33 -07:00
8609521ce2 mvcc: add TestHashKVWhenCompacting to kvstore_test 2017-07-26 09:48:29 -07:00
deca9879c2 mvcc: add HashByRev to kv.go
HashByRev computes the hash of all MVCC keys up to a given revision.
2017-07-25 17:00:46 -07:00
c06953ae08 mvcc: Add metric for count of db key revisions compacted.
When digging into etcd/boltdb "storage space exceeded" issues, this metric may help answer questions about if/when compactions occured and how much data was freed.
2017-07-20 10:07:56 -07:00
e9d096ae6b mvcc: don't allocate end revision while computing range
Use 'nil' since it's only reading a single key. Also preallocates
the result slice based on limit / number of revisions fetched.

Fixes #8208
2017-07-06 15:59:27 -07:00
870302afa6 mvcc/backend: enable 'NoFreelistSync' by default (linux)
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-07-05 16:10:04 -07:00
318e9c766f *: replace 'boltdb' import paths with 'coreos/bbolt'
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-07-05 14:32:13 -07:00
522e75cb4f mvcc: use GaugeFunc metric to load db size when requested
Relying on mvcc to set the db size metric can cause it to
miss size changes when a txn commits after the last write
completes before a quiescent period. Instead, load the
db size on demand.

Fixes #8146
2017-06-21 23:58:37 -07:00
51a568aa81 mvcc: restore into tree index with one key index
Clobbering the mvcc kvindex with new keyIndexes for each restore
chunk would cause index corruption by dropping historical information.
2017-06-19 12:04:01 -07:00
02164874d9 mvcc: test restore and deletes with small chunk sizes 2017-06-19 12:04:01 -07:00
7f149d8fb6 mvcc: set db size metric on restore
Fixes #8080
2017-06-16 11:27:34 -07:00
da48f1feaf mvcc: create TxnWrites from TxnRead with NewReadOnlyTxnWrite
Already used internally by mvcc, but needed by etcdserver txns.
2017-06-09 09:20:38 -07:00
300feea177 Merge pull request #8052 from heyitsanthony/watch-victim-test
mvcc: test watch victim/delay path
2017-06-08 11:10:33 -07:00
83b2ea2f60 mvcc: test watch victim/delay path
Current tests don't normally trigger the watch victim path because the
constants are too large; set the constants to small values and hammer
the store to cause watch delivery delays.
2017-06-07 17:02:00 -07:00
0352ce79b8 mvcc: count range/put/del operations for txns
Txns were previously only bumping the txn counter; now bumps all operation
counters.
2017-06-07 16:53:50 -07:00
fd71da47d1 mvcc: remove unused store.Equals function 2017-06-07 09:25:42 -07:00
ef63abdf7f mvcc: don't use pointer for storeTxnRead in storeTxnWrite
Saves an allocation when creating a storeTxnWrite.
2017-06-06 09:51:57 -07:00
6846e49edf Merge pull request #7859 from heyitsanthony/cache-consistent-get
mvcc: cache consistent index
2017-05-26 10:52:53 -07:00
ac4855e911 mvcc: benchmark ConsistentIndex 2017-05-26 09:49:40 -07:00
73dee0bec4 mvcc: cache consistentIndex
Called on every entry apply and boltdb requests aren't free.
2017-05-26 09:49:40 -07:00
0506f49f9e backend: don't hold boltdb read txn lock on cursor scanning
Large fetches hold the lock when they do not need to do so.
2017-05-26 09:28:08 -07:00
343a018361 Merge pull request #7900 from heyitsanthony/chunk-restore
mvcc: chunk reads for restoring
2017-05-26 09:21:59 -07:00
8516d8ccc5 backend: force initial mmap size to 0 for windows
boltdb on windows allocates a file with the full mmap size even if the
db is empty. Force the initial mmap size to 0 so there's no huge initial
db file on windows.

Fixes #7910
2017-05-12 14:34:07 -07:00
8468b38631 backend: dynamically set snapshotWarningTimeout based on db size 2017-05-11 15:25:35 -07:00
1aca63e9e0 mvcc: time restore in restore benchmark
This never worked.
2017-05-09 20:14:58 -07:00
163fd2d76b mvcc: chunk reads for restoring
Loading all keys at once would cause etcd to use twice as much
memory than it would need to serve the keys, causing RSS to spike on
boot. Instead, load the keys into the mvcc by chunk. Uses pipelining
for some concurrency.

Fixes #7822
2017-05-09 20:14:58 -07:00
230106dd3c backend: add prometheus metric for large snapshot duration.
FIXES #7878
2017-05-05 17:27:33 -07:00
f7f30f2361 backend: print snapshotting duration warning every 30s
FIXES #7870
2017-05-04 16:41:03 -07:00
14d6ed9e5f *: clear redundant return statement warnings (S1027) 2017-04-21 14:01:00 -07:00
5000d29b4a mvcc: remove stopc select case in Hash
Revert change in 33acbb694b.

Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-04-17 14:19:48 -07:00
8ffd58fb3b mvcc/backend: remove t.tx.DB()==nil checks with GracefulStop
Revert https://github.com/coreos/etcd/pull/6662.

Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-04-17 14:17:00 -07:00
cd470f9ccd Revert "mvcc: test inflight Hash to trigger Size on nil db"
This reverts commit 994e8e4f40.

Since now etcdserver gracefully shuts down the gRPC server
2017-04-17 14:15:43 -07:00
78a5eb79b5 *: add swagger and grpc-gateway assets for v3lock and v3election 2017-04-10 15:21:07 -07:00
f67bdc2eed *: support checking that an interval tree's keys cover an entire interval 2017-04-03 15:38:07 -07:00
161c7f6bdf Merge pull request #7579 from gyuho/fix-defrage
*: fix panic during defrag operation
2017-03-23 10:08:33 -07:00
7ef75e373a Merge pull request #7525 from heyitsanthony/big-backend
etcdserver, backend: configure mmap size based on quota
2017-03-23 10:06:00 -07:00
26abd25cd3 mvcc/backend: hold 'readTx.Lock' until completing bolt.Tx reset
Fix https://github.com/coreos/etcd/issues/7526.

When resetting `bolt.Tx` in `defrag` and `batchTxBuffered.commit`
operation, we do not hold `readTx` lock, so the inflight range
requests can trigger panic in `mvcc.Range` paths. This fixes by
moving mutexes out and hold it while resetting the `readTx`.

Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-03-23 09:47:43 -07:00
7698a2a546 Merge pull request #7553 from xiang90/fix_defrag
backend: add FillPercent option
2017-03-21 11:16:17 -07:00
95870a21eb backend: add FillPercent option 2017-03-21 08:06:03 -07:00
8a3fee15a3 etcdserver, backend: only warn if exceeding max quota 2017-03-17 15:38:57 -07:00
5e4b008106 *: base initial mmap size on quota size 2017-03-17 15:38:49 -07:00
2f1542c06d *: use filepath.Join for files 2017-03-16 07:46:06 -07:00
33acbb694b mvcc: txns and r/w views
Clean-up of the mvcc interfaces to use txn interfaces instead of an id.

Adds support for concurrent read-only mvcc transactions.

Fixes #7083
2017-03-08 20:52:59 -08:00
8d438c2939 backend: readtx
ReadTxs are designed for read-only accesses to the backend using a
read-only boltDB transaction. Since BatchTx's are long-running
transactions, all writes to BatchTx will writeback to ReadTx, overlaying
the base read-only transaction.
2017-03-08 20:52:59 -08:00
3d75395875 *: remove never-unused vars, minor lint fix
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-03-06 14:59:12 -08:00
0914b8b707 test: Fix gosimple errors
Getting gosimple suggestion while running test script, so this PR is for fixing gosimple S1019 check.
raft/node_test.go:456:40: should use make([]raftpb.Entry, 1) instead (S1019)
raft/node_test.go:457:49: should use make([]raftpb.Entry, 1) instead (S1019)
raft/node_test.go:458:43: should use make([]raftpb.Message, 1) instead (S1019)

Refer https://github.com/dominikh/go-tools/blob/master/cmd/gosimple/README.md#checks for more information.
2017-02-09 08:01:28 -05:00
43078d3ced mvcc: remove unused restore method 2016-11-18 23:04:39 +05:30