mirrors/git
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

reftable/stack: handle locked tables during auto-compaction

Browse Source 
When compacting tables, it may happen that we want to compact a set of
tables which are already locked by a concurrent process that compacts
them. In the case where we wanted to perform a full compaction of all
tables it is sensible to bail out in this case, as we cannot fulfill the
requested action.

But when performing auto-compaction it isn't necessarily in our best
interest of us to abort the whole operation. For example, due to the
geometric compacting schema that we use, it may be that process A takes
a lot of time to compact the bulk of all tables whereas process B
appends a bunch of new tables to the stack. B would in this case also
notice that it has to compact the tables that process A is compacting
already and thus also try to compact the same range, probably including
the new tables it has appended. But because those tables are locked
already, it will fail and thus abort the complete auto-compaction. The
consequence is that the stack will grow longer and longer while A isn't
yet done with compaction, which will lead to a growing performance
impact.

Instead of aborting auto-compaction altogether, let's gracefully handle
this situation by instead compacting tables which aren't locked. To do
so, instead of locking from the beginning of the slice-to-be-compacted,
we start locking tables from the end of the slice. Once we hit the first
table that is locked already, we abort. If we succeeded to lock two or
more tables, then we simply reduce the slice of tables that we're about
to compact to those which we managed to lock.

This ensures that we can at least make some progress for compaction in
said scenario. It also helps in other scenarios, like for example when a
process died and left a stale lockfile behind. In such a case we can at
least ensure some compaction on a best-effort basis.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This commit is contained in:
Patrick Steinhardt
2024-08-08 16:06:58 +02:00
committed by Junio C Hamano
parent ed1ad6b44d
commit f234df07f6
3 changed files with 70 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
reftable/stack_test.c
View File

@ -917,13 +917,15 @@ static void test_reftable_stack_auto_compaction_with_locked_tables(void)
write_file_buf(buf.buf, "", 0);
/*
* Ideally, we'd handle the situation where any of the tables is locked
* gracefully. We don't (yet) do this though and thus fail.
* When parts of the stack are locked, then auto-compaction does a best
* effort compaction of those tables which aren't locked. So while this
* would in theory compact all tables, due to the preexisting lock we
* only compact the newest two tables.
*/
err = reftable_stack_auto_compact(st);
EXPECT(err == REFTABLE_LOCK_ERROR);
EXPECT(st->stats.failures == 1);
EXPECT(st->merged->stack_len == 5);
EXPECT_ERR(err);
EXPECT(st->stats.failures == 0);
EXPECT(st->merged->stack_len == 4);
reftable_stack_destroy(st);
strbuf_release(&buf);