From 6eb1a7d7b017299f9e4cfeb9ec40a7da0c2748df Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:36:41 -0400 Subject: [PATCH 01/19] Documentation: describe incremental MIDX format Prepare to implement incremental multi-pack indexes (MIDXs) over the next several commits by first describing the relevant prerequisites (like a new chunk in the MIDX format, the directory structure for incremental MIDXs, etc.) The format is described in detail in the patch contents below, but the high-level description is as follows. Incremental MIDXs live in $GIT_DIR/objects/pack/multi-pack-index.d, and each `*.midx` within that directory has a single "parent" MIDX, which is the MIDX layer immediately before it in the MIDX chain. The chain order resides in a file 'multi-pack-index-chain' in the same directory. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- Documentation/technical/multi-pack-index.txt | 103 +++++++++++++++++++ 1 file changed, 103 insertions(+) diff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt index f2221d2b44..cc063b30be 100644 --- a/Documentation/technical/multi-pack-index.txt +++ b/Documentation/technical/multi-pack-index.txt @@ -61,6 +61,109 @@ Design Details - The MIDX file format uses a chunk-based approach (similar to the commit-graph file) that allows optional data to be added. +Incremental multi-pack indexes +------------------------------ + +As repositories grow in size, it becomes more expensive to write a +multi-pack index (MIDX) that includes all packfiles. To accommodate +this, the "incremental multi-pack indexes" feature allows for combining +a "chain" of multi-pack indexes. + +Each individual component of the chain need only contain a small number +of packfiles. Appending to the chain does not invalidate earlier parts +of the chain, so repositories can control how much time is spent +updating the MIDX chain by determining the number of packs in each layer +of the MIDX chain. + +=== Design state + +At present, the incremental multi-pack indexes feature is missing two +important components: + + - The ability to rewrite earlier portions of the MIDX chain (i.e., to + "compact" some collection of adjacent MIDX layers into a single + MIDX). At present the only supported way of shrinking a MIDX chain + is to rewrite the entire chain from scratch without the `--split` + flag. ++ +There are no fundamental limitations that stand in the way of being able +to implement this feature. It is omitted from the initial implementation +in order to reduce the complexity, but will be added later. + + - Support for reachability bitmaps. The classic single MIDX + implementation does support reachability bitmaps (see the section + titled "multi-pack-index reverse indexes" in + linkgit:gitformat-pack[5] for more details). ++ +As above, there are no fundamental limitations that stand in the way of +extending the incremental MIDX format to support reachability bitmaps. +The design below specifically takes this into account, and support for +reachability bitmaps will be added in a future patch series. It is +omitted from the current implementation for the same reason as above. ++ +In brief, to support reachability bitmaps with the incremental MIDX +feature, the concept of the pseudo-pack order is extended across each +layer of the incremental MIDX chain to form a concatenated pseudo-pack +order. This concatenation takes place in the same order as the chain +itself (in other words, the concatenated pseudo-pack order for a chain +`{$H1, $H2, $H3}` would be the pseudo-pack order for `$H1`, followed by +the pseudo-pack order for `$H2`, followed by the pseudo-pack order for +`$H3`). ++ +The layout will then be extended so that each layer of the incremental +MIDX chain can write a `*.bitmap`. The objects in each layer's bitmap +are offset by the number of objects in the previous layers of the chain. + +=== File layout + +Instead of storing a single `multi-pack-index` file (with an optional +`.rev` and `.bitmap` extension) in `$GIT_DIR/objects/pack`, incremental +MIDXs are stored in the following layout: + +---- +$GIT_DIR/objects/pack/multi-pack-index.d/ +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-chain +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H1.midx +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H2.midx +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H3.midx +---- + +The `multi-pack-index-chain` file contains a list of the incremental +MIDX files in the chain, in order. The above example shows a chain whose +`multi-pack-index-chain` file would contain the following lines: + +---- +$H1 +$H2 +$H3 +---- + +The `multi-pack-index-$H1.midx` file contains the first layer of the +multi-pack-index chain. The `multi-pack-index-$H2.midx` file contains +the second layer of the chain, and so on. + +When both an incremental- and non-incremental MIDX are present, the +non-incremental MIDX is always read first. + +=== Object positions for incremental MIDXs + +In the original multi-pack-index design, we refer to objects via their +lexicographic position (by object IDs) within the repository's singular +multi-pack-index. In the incremental multi-pack-index design, we refer +to objects via their index into a concatenated lexicographic ordering +among each component in the MIDX chain. + +If `objects_nr()` is a function that returns the number of objects in a +given MIDX layer, then the index of an object at lexicographic position +`i` within, say, $H3 is defined as: + +---- +objects_nr($H2) + objects_nr($H1) + i +---- + +(in the C implementation, this is often computed as `i + +m->num_objects_in_base`). + Future Work ----------- From 2678a730093cb8c3e728cc6bc7e1da414fac7953 Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:36:44 -0400 Subject: [PATCH 02/19] midx: add new fields for incremental MIDX chains The incremental MIDX chain feature is designed around the idea of indexing into a concatenated lexicographic ordering of object IDs present in the MIDX. When given an object position, the MIDX machinery needs to be able to locate both (a) which MIDX layer contains the given object, and (b) at what position *within that MIDX layer* that object appears. To do this, three new fields are added to the `struct multi_pack_index`: - struct multi_pack_index *base_midx; - uint32_t num_objects_in_base; - uint32_t num_packs_in_base; These three fields store the pieces of information suggested by their respective field names. In turn, the `num_objects_in_base` and `num_packs_in_base` fields are used to crawl backwards along the `base_midx` pointer to locate the appropriate position for a given object within the MIDX that contains it. The following commits will update various parts of the MIDX machinery (as well as their callers from outside of midx.c and midx-write.c) to be aware and make use of these fields when performing object lookups. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/midx.h b/midx.h index 8554f2d616..020e49f77c 100644 --- a/midx.h +++ b/midx.h @@ -63,6 +63,10 @@ struct multi_pack_index { const unsigned char *chunk_revindex; size_t chunk_revindex_len; + struct multi_pack_index *base_midx; + uint32_t num_objects_in_base; + uint32_t num_packs_in_base; + const char **pack_names; struct packed_git **packs; char object_dir[FLEX_ARRAY]; From 19419821bac508c5f40e1df45ca0b132b37d006b Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:18 -0400 Subject: [PATCH 03/19] midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs The function `nth_midxed_pack_int_id()` takes in a object position in MIDX lexicographic order and returns an identifier of the pack from which that object was selected in the MIDX. Currently, the given object position is an index into the lexicographic order of objects in a single MIDX. Change this position to instead refer into the concatenated lexicographic order of all MIDXs in a MIDX chain. This has two visible effects within the implementation of `prepare_midx_pack()`: - First, the given position is now an index into the concatenated lexicographic order of all MIDXs in the order in which they appear in the MIDX chain. - Second the pack ID returned from this function is now also in the concatenated order of packs among all layers of the MIDX chain in the same order that they appear in the MIDX chain. To do this, introduce the first of two general purpose helpers, this one being `midx_for_object()`. `midx_for_object()` takes a double pointer to a `struct multi_pack_index` as well as an object `pos` in terms of the entire MIDX chain[^1]. The function chases down the '->base_midx' field until it finds the MIDX layer within the chain that contains the given object. It then: - modifies the double pointer to point to the containing MIDX, instead of the tip of the chain, and - returns the MIDX-local position[^2] at which the given object can be found. Use this function within `nth_midxed_pack_int_id()` so that the `pos` it expects is now relative to the entire MIDX chain, and that it returns the appropriate pack position for that object. [^1]: As a reminder, this means that the object is identified among the objects contained in all layers of the incremental MIDX chain, not any particular layer. For example, consider MIDX chain with two individual MIDXs, one with 4 objects and another with 3 objects. If the MIDX with 4 objects appears earlier in the chain, then asking for object 6 would return the second object in the MIDX with 3 objects. [^2]: Building on the previous example, asking for object 6 in a MIDX chain with (4, 3) objects, respectively, this would set the double pointer to point at the MIDX containing three objects, and would return an index to the second object within that MIDX. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/midx.c b/midx.c index 3992b05465..39d358da20 100644 --- a/midx.c +++ b/midx.c @@ -242,6 +242,23 @@ void close_midx(struct multi_pack_index *m) free(m); } +static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos) +{ + struct multi_pack_index *m = *_m; + while (m && pos < m->num_objects_in_base) + m = m->base_midx; + + if (!m) + BUG("NULL multi-pack-index for object position: %"PRIu32, pos); + + if (pos >= m->num_objects + m->num_objects_in_base) + die(_("invalid MIDX object position, MIDX is likely corrupt")); + + *_m = m; + + return pos - m->num_objects_in_base; +} + int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id) { struct strbuf pack_name = STRBUF_INIT; @@ -334,8 +351,10 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos) uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos) { - return get_be32(m->chunk_object_offsets + - (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH); + pos = midx_for_object(&m, pos); + + return m->num_packs_in_base + get_be32(m->chunk_object_offsets + + (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH); } int fill_midx_entry(struct repository *r, From 1820bd878c623a71ffc719ff09071fc6bbc6e34e Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:21 -0400 Subject: [PATCH 04/19] midx: teach `prepare_midx_pack()` about incremental MIDXs The function `prepare_midx_pack()` is part of the midx.h API and loads the pack identified by the MIDX-local 'pack_int_id'. This patch prepares that function to be aware of an incremental MIDX world. To do this, introduce the second of the two general purpose helpers mentioned in the previous commit. This commit introduces `midx_for_pack()`, which is the pack-specific analog of `midx_for_object()`, and works in the same fashion. Like `midx_for_object()`, this function chases down the '->base_midx' field until it finds the MIDX layer within the chain that contains the given pack. Use this function within `prepare_midx_pack()` so that the `pack_int_id` it expects is now relative to the entire MIDX chain, and that it prepares the given pack in the appropriate MIDX. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.c | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/midx.c b/midx.c index 39d358da20..07b3981a7a 100644 --- a/midx.c +++ b/midx.c @@ -259,14 +259,32 @@ static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos) return pos - m->num_objects_in_base; } -int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id) +static uint32_t midx_for_pack(struct multi_pack_index **_m, + uint32_t pack_int_id) +{ + struct multi_pack_index *m = *_m; + while (m && pack_int_id < m->num_packs_in_base) + m = m->base_midx; + + if (!m) + BUG("NULL multi-pack-index for pack ID: %"PRIu32, pack_int_id); + + if (pack_int_id >= m->num_packs + m->num_packs_in_base) + die(_("bad pack-int-id: %u (%u total packs)"), + pack_int_id, m->num_packs + m->num_packs_in_base); + + *_m = m; + + return pack_int_id - m->num_packs_in_base; +} + +int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, + uint32_t pack_int_id) { struct strbuf pack_name = STRBUF_INIT; struct packed_git *p; - if (pack_int_id >= m->num_packs) - die(_("bad pack-int-id: %u (%u total packs)"), - pack_int_id, m->num_packs); + pack_int_id = midx_for_pack(&m, pack_int_id); if (m->packs[pack_int_id]) return 0; From 26afb5afa16c9ea7825fdeae05e31d619849ad4b Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:24 -0400 Subject: [PATCH 05/19] midx: teach `nth_midxed_object_oid()` about incremental MIDXs The function `nth_midxed_object_oid()` returns the object ID for a given object position in the MIDX lexicographic order. Teach this function to instead operate over the concatenated lexicographic order defined in an earlier step so that it is able to be used with incremental MIDXs. To do this, we need to both (a) adjust the bounds check for the given 'n', as well as record the MIDX-local position after chasing the `->base_midx` pointer to find the MIDX which contains that object. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index 07b3981a7a..64a051cca1 100644 --- a/midx.c +++ b/midx.c @@ -338,9 +338,11 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid, struct multi_pack_index *m, uint32_t n) { - if (n >= m->num_objects) + if (n >= m->num_objects + m->num_objects_in_base) return NULL; + n = midx_for_object(&m, n); + oidread(oid, m->chunk_oid_lookup + st_mult(m->hash_len, n), the_repository->hash_algo); return oid; From 60750e1eb967c8622e7140037ec018a87577132b Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:27 -0400 Subject: [PATCH 06/19] midx: teach `nth_bitmapped_pack()` about incremental MIDXs In a similar fashion as in previous commits, teach the function `nth_bitmapped_pack()` about incremental MIDXs by translating the given `pack_int_id` from the concatenated lexical order to a MIDX-local lexical position. When accessing the containing MIDX's array of packs, use the local pack ID. Likewise, when reading the 'BTMP' chunk, use the MIDX-local offset when accessing the data within that chunk. (Note that the both the call to prepare_midx_pack() and the assignment of bp->pack_int_id both care about the global pack_int_id, so avoid shadowing the given 'pack_int_id' parameter). Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/midx.c b/midx.c index 64a051cca1..25350152f1 100644 --- a/midx.c +++ b/midx.c @@ -311,17 +311,19 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, struct bitmapped_pack *bp, uint32_t pack_int_id) { + uint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id); + if (!m->chunk_bitmapped_packs) return error(_("MIDX does not contain the BTMP chunk")); if (prepare_midx_pack(r, m, pack_int_id)) return error(_("could not load bitmapped pack %"PRIu32), pack_int_id); - bp->p = m->packs[pack_int_id]; + bp->p = m->packs[local_pack_int_id]; bp->bitmap_pos = get_be32((char *)m->chunk_bitmapped_packs + - MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id); + MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id); bp->bitmap_nr = get_be32((char *)m->chunk_bitmapped_packs + - MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id + + MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id + sizeof(uint32_t)); bp->pack_int_id = pack_int_id; From 3f5f1cff92dfe64bbbfa9f2fb4ed810125810b1b Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:30 -0400 Subject: [PATCH 07/19] midx: introduce `bsearch_one_midx()` The `bsearch_midx()` function will be extended in a following commit to search for the location of a given object ID across all MIDXs in a chain (or the single non-chain MIDX if no chain is available). While most callers will naturally want to use the updated `bsearch_midx()` function, there are a handful of special cases that will want finer control and will only want to search through a single MIDX. For instance, the object abbreviation code, which cares about object IDs near to where we'd expect to find a match in a MIDX. In that case, we want to look at the nearby matches in each layer of the MIDX chain, not just a single one). Split the more fine-grained control out into a separate function called `bsearch_one_midx()` which searches only a single MIDX. At present both `bsearch_midx()` and `bsearch_one_midx()` have identical behavior, but the following commit will rewrite the former to be aware of incremental MIDXs for the remaining non-special case callers. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.c | 17 +++++++-- midx.h | 5 ++- object-name.c | 99 +++++++++++++++++++++++++++------------------------ 3 files changed, 71 insertions(+), 50 deletions(-) diff --git a/midx.c b/midx.c index 25350152f1..bd6e3f26c9 100644 --- a/midx.c +++ b/midx.c @@ -330,10 +330,21 @@ int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, return 0; } -int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result) +int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, + uint32_t *result) { - return bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup, - the_hash_algo->rawsz, result); + int ret = bsearch_hash(oid->hash, m->chunk_oid_fanout, + m->chunk_oid_lookup, the_hash_algo->rawsz, + result); + if (result) + *result += m->num_objects_in_base; + return ret; +} + +int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, + uint32_t *result) +{ + return bsearch_one_midx(oid, m, result); } struct object_id *nth_midxed_object_oid(struct object_id *oid, diff --git a/midx.h b/midx.h index 020e49f77c..46c53d69ff 100644 --- a/midx.h +++ b/midx.h @@ -90,7 +90,10 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id); int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, struct bitmapped_pack *bp, uint32_t pack_int_id); -int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result); +int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, + uint32_t *result); +int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, + uint32_t *result); off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos); uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos); struct object_id *nth_midxed_object_oid(struct object_id *oid, diff --git a/object-name.c b/object-name.c index 527b853ac4..739d46f9cf 100644 --- a/object-name.c +++ b/object-name.c @@ -134,28 +134,32 @@ static int match_hash(unsigned len, const unsigned char *a, const unsigned char static void unique_in_midx(struct multi_pack_index *m, struct disambiguate_state *ds) { - uint32_t num, i, first = 0; - const struct object_id *current = NULL; - int len = ds->len > ds->repo->hash_algo->hexsz ? - ds->repo->hash_algo->hexsz : ds->len; - num = m->num_objects; + for (; m; m = m->base_midx) { + uint32_t num, i, first = 0; + const struct object_id *current = NULL; + int len = ds->len > ds->repo->hash_algo->hexsz ? + ds->repo->hash_algo->hexsz : ds->len; - if (!num) - return; + if (!m->num_objects) + continue; - bsearch_midx(&ds->bin_pfx, m, &first); + num = m->num_objects + m->num_objects_in_base; - /* - * At this point, "first" is the location of the lowest object - * with an object name that could match "bin_pfx". See if we have - * 0, 1 or more objects that actually match(es). - */ - for (i = first; i < num && !ds->ambiguous; i++) { - struct object_id oid; - current = nth_midxed_object_oid(&oid, m, i); - if (!match_hash(len, ds->bin_pfx.hash, current->hash)) - break; - update_candidates(ds, current); + bsearch_one_midx(&ds->bin_pfx, m, &first); + + /* + * At this point, "first" is the location of the lowest + * object with an object name that could match + * "bin_pfx". See if we have 0, 1 or more objects that + * actually match(es). + */ + for (i = first; i < num && !ds->ambiguous; i++) { + struct object_id oid; + current = nth_midxed_object_oid(&oid, m, i); + if (!match_hash(len, ds->bin_pfx.hash, current->hash)) + break; + update_candidates(ds, current); + } } } @@ -708,37 +712,40 @@ static int repo_extend_abbrev_len(struct repository *r UNUSED, static void find_abbrev_len_for_midx(struct multi_pack_index *m, struct min_abbrev_data *mad) { - int match = 0; - uint32_t num, first = 0; - struct object_id oid; - const struct object_id *mad_oid; + for (; m; m = m->base_midx) { + int match = 0; + uint32_t num, first = 0; + struct object_id oid; + const struct object_id *mad_oid; - if (!m->num_objects) - return; + if (!m->num_objects) + continue; - num = m->num_objects; - mad_oid = mad->oid; - match = bsearch_midx(mad_oid, m, &first); + num = m->num_objects + m->num_objects_in_base; + mad_oid = mad->oid; + match = bsearch_one_midx(mad_oid, m, &first); - /* - * first is now the position in the packfile where we would insert - * mad->hash if it does not exist (or the position of mad->hash if - * it does exist). Hence, we consider a maximum of two objects - * nearby for the abbreviation length. - */ - mad->init_len = 0; - if (!match) { - if (nth_midxed_object_oid(&oid, m, first)) - extend_abbrev_len(&oid, mad); - } else if (first < num - 1) { - if (nth_midxed_object_oid(&oid, m, first + 1)) - extend_abbrev_len(&oid, mad); + /* + * first is now the position in the packfile where we + * would insert mad->hash if it does not exist (or the + * position of mad->hash if it does exist). Hence, we + * consider a maximum of two objects nearby for the + * abbreviation length. + */ + mad->init_len = 0; + if (!match) { + if (nth_midxed_object_oid(&oid, m, first)) + extend_abbrev_len(&oid, mad); + } else if (first < num - 1) { + if (nth_midxed_object_oid(&oid, m, first + 1)) + extend_abbrev_len(&oid, mad); + } + if (first > 0) { + if (nth_midxed_object_oid(&oid, m, first - 1)) + extend_abbrev_len(&oid, mad); + } + mad->init_len = mad->cur_len; } - if (first > 0) { - if (nth_midxed_object_oid(&oid, m, first - 1)) - extend_abbrev_len(&oid, mad); - } - mad->init_len = mad->cur_len; } static void find_abbrev_len_for_pack(struct packed_git *p, From 88f309e095d084d730f908fe1ec01a92c63612c2 Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:33 -0400 Subject: [PATCH 08/19] midx: teach `bsearch_midx()` about incremental MIDXs Now that the special cases callers of `bsearch_midx()` have been dealt with, teach `bsearch_midx()` to handle incremental MIDX chains. The incremental MIDX-aware version of `bsearch_midx()` works by repeatedly searching for a given OID in each layer along the `->base_midx` pointer, stopping either when an exact match is found, or the end of the chain is reached. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index bd6e3f26c9..83857cbd1e 100644 --- a/midx.c +++ b/midx.c @@ -344,7 +344,10 @@ int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result) { - return bsearch_one_midx(oid, m, result); + for (; m; m = m->base_midx) + if (bsearch_one_midx(oid, m, result)) + return 1; + return 0; } struct object_id *nth_midxed_object_oid(struct object_id *oid, From df7ede83be829f983cd54539eb6fe0e6bddc8c9e Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:36 -0400 Subject: [PATCH 09/19] midx: teach `nth_midxed_offset()` about incremental MIDXs In a similar fashion as in previous commits, teach the function `nth_midxed_offset()` about incremental MIDXs. The given object `pos` is used to find the containing MIDX, and translated back into a MIDX-local position by assigning the return value of `midx_for_object()` to it. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/midx.c b/midx.c index 83857cbd1e..346e58dec7 100644 --- a/midx.c +++ b/midx.c @@ -369,6 +369,8 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos) const unsigned char *offset_data; uint32_t offset32; + pos = midx_for_object(&m, pos); + offset_data = m->chunk_object_offsets + (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH; offset32 = get_be32(offset_data + sizeof(uint32_t)); From 3b00e351081ad676e1d1262f079e63bbde5a4314 Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:40 -0400 Subject: [PATCH 10/19] midx: teach `fill_midx_entry()` about incremental MIDXs In a similar fashion as previous commits, teach the `fill_midx_entry()` function to work in a incremental MIDX-aware fashion. This function, unlike others which accept an index into either the lexical order of objects or packs, takes in an object_id, and attempts to fill a caller-provided 'struct pack_entry' with the remaining pieces of information about that object from the MIDX. The function uses `bsearch_midx()` which fills out the frame-local 'pos' variable, recording the given object_id's lexical position within the MIDX chain, if found (if no matching object ID was found, we'll return immediately without filling out the `pack_entry` structure). Once given that position, we jump back through the `->base_midx` pointer to ensure that our `m` points at the MIDX layer which contains the given object_id (and not an ancestor or descendant of it in the chain). Note that we can drop the bounds check "if (pos >= m->num_objects)" because `midx_for_object()` performs this check for us. After that point, we only need to make two special considerations within this function: - First, the pack_int_id returned to us by `nth_midxed_pack_int_id()` is a position in the concatenated lexical order of packs, so we must ensure that we subtract `m->num_packs_in_base` before accessing the MIDX-local `packs` array. - Second, we must avoid translating the `pos` back to a MIDX-local index, since we use it as an argument to `nth_midxed_offset()` which expects a position relative to the concatenated lexical order of objects. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/midx.c b/midx.c index 346e58dec7..5e4e6f9b65 100644 --- a/midx.c +++ b/midx.c @@ -407,14 +407,12 @@ int fill_midx_entry(struct repository *r, if (!bsearch_midx(oid, m, &pos)) return 0; - if (pos >= m->num_objects) - return 0; - + midx_for_object(&m, pos); pack_int_id = nth_midxed_pack_int_id(m, pos); if (prepare_midx_pack(r, m, pack_int_id)) return 0; - p = m->packs[pack_int_id]; + p = m->packs[pack_int_id - m->num_packs_in_base]; /* * We are about to tell the caller where they can locate the From 5d0ee3f6758cda141e9bc999b9aba0e53867538f Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:42 -0400 Subject: [PATCH 11/19] midx: remove unused `midx_locate_pack()` Commit 307d75bbe6 (midx: implement `midx_locate_pack()`, 2023-12-14) introduced `midx_locate_pack()`, which was described at the time as a complement to the function `midx_contains_pack()` which allowed callers to determine where in the MIDX lexical order a pack appeared, as opposed to whether or not it was simply contained. 307d75bbe6 suggests that future patches would be added which would introduce callers for this new function, but none ever were, meaning the function has gone unused since its introduction. Clean this up by in effect reverting 307d75bbe6, which removes the unused functions and inlines its definition back into `midx_contains_pack()`. (Looking back through the list archives when 307d75bbe6 was written, this was in preparation for this[1] patch from back when we had the concept of "disjoint" packs while developing multi-pack verbatim reuse. That concept was abandoned before the series was merged, but I never dropped what would become 307d75bbe6 from the series, leading to the state prior to this commit). [1]: https://lore.kernel.org/git/3019738b52ba8cd78ea696a3b800fa91e722eb66.1701198172.git.me@ttaylorr.com/ Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.c | 13 ++----------- midx.h | 2 -- 2 files changed, 2 insertions(+), 13 deletions(-) diff --git a/midx.c b/midx.c index 5e4e6f9b65..50f131e59a 100644 --- a/midx.c +++ b/midx.c @@ -466,8 +466,7 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name, return strcmp(idx_or_pack_name, idx_name); } -int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name, - uint32_t *pos) +int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) { uint32_t first = 0, last = m->num_packs; @@ -478,11 +477,8 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name, current = m->pack_names[mid]; cmp = cmp_idx_or_pack_name(idx_or_pack_name, current); - if (!cmp) { - if (pos) - *pos = mid; + if (!cmp) return 1; - } if (cmp > 0) { first = mid + 1; continue; @@ -493,11 +489,6 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name, return 0; } -int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) -{ - return midx_locate_pack(m, idx_or_pack_name, NULL); -} - int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id) { if (m->preferred_pack_idx == -1) { diff --git a/midx.h b/midx.h index 46c53d69ff..86af7dfc5e 100644 --- a/midx.h +++ b/midx.h @@ -102,8 +102,6 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid, int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m); int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name); -int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name, - uint32_t *pos); int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id); int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local); From 853165c50a8c6df8424ee7a09d16272485ad1712 Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:46 -0400 Subject: [PATCH 12/19] midx: teach `midx_contains_pack()` about incremental MIDXs Now that the `midx_contains_pack()` versus `midx_locate_pack()` debacle has been cleaned up, teach the former about how to operate in an incremental MIDX-aware world in a similar fashion as in previous commits. Instead of using either of the two `midx_for_object()` or `midx_for_pack()` helpers, this function is split into two: one that determines whether a pack is contained in a single MIDX, and another which calls the former in a loop over all MIDXs. This approach does not require that we change any of the implementation in what is now `midx_contains_pack_1()` as it still operates over a single MIDX. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index 50f131e59a..454c27b673 100644 --- a/midx.c +++ b/midx.c @@ -466,7 +466,8 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name, return strcmp(idx_or_pack_name, idx_name); } -int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) +static int midx_contains_pack_1(struct multi_pack_index *m, + const char *idx_or_pack_name) { uint32_t first = 0, last = m->num_packs; @@ -489,6 +490,14 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) return 0; } +int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) +{ + for (; m; m = m->base_midx) + if (midx_contains_pack_1(m, idx_or_pack_name)) + return 1; + return 0; +} + int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id) { if (m->preferred_pack_idx == -1) { From b31f2aac561d9a819dc4f4ed54fafdeb3041bccd Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:49 -0400 Subject: [PATCH 13/19] midx: teach `midx_preferred_pack()` about incremental MIDXs The function `midx_preferred_pack()` is used to determine the identity of the preferred pack, which is the identity of a unique pack within the MIDX which is used as a tie-breaker when selecting from which pack to represent an object that appears in multiple packs within the MIDX. Historically we have said that the MIDX's preferred pack has the unique property that all objects from that pack are represented in the MIDX. But that isn't quite true: a more precise statement would be that all objects from that pack *which appear in the MIDX* are selected from that pack. This helps us extend the concept of preferred packs across a MIDX chain, where some object(s) in the preferred pack may appear in other packs in an earlier MIDX layer, in which case those object(s) will not appear in a subsequent MIDX layer from either the preferred pack or any other pack. Extend the concept of preferred packs by using the pack which represents the object at the first position in MIDX pseudo-pack order belonging to the current MIDX layer (i.e., at position 'm->num_objects_in_base'). Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/midx.c b/midx.c index 454c27b673..918349576f 100644 --- a/midx.c +++ b/midx.c @@ -501,13 +501,16 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id) { if (m->preferred_pack_idx == -1) { + uint32_t midx_pos; if (load_midx_revindex(m) < 0) { m->preferred_pack_idx = -2; return -1; } - m->preferred_pack_idx = - nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0)); + midx_pos = pack_pos_to_midx(m, m->num_objects_in_base); + + m->preferred_pack_idx = nth_midxed_pack_int_id(m, midx_pos); + } else if (m->preferred_pack_idx == -2) return -1; /* no revindex */ From 97fd770ea1d2ccf9581c4962a1238d853540ec11 Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:52 -0400 Subject: [PATCH 14/19] midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs The function `midx_fanout_add_midx_fanout()` is used to help construct the fanout table when generating a MIDX by reusing data from an existing MIDX. Prepare this function to work with incremental MIDXs by making a few changes: - The bounds checks need to be adjusted to start object lookups taking into account the number of objects in the previous MIDX layer (i.e., by starting the lookups at position `m->num_objects_in_base` instead of position 0). - Likewise, the bounds checks need to end at `m->num_objects_in_base` objects after `m->num_objects`. - Finally, `midx_fanout_add_midx_fanout()` needs to recur on earlier MIDX layers when dealing with an incremental MIDX chain by calling itself when given a MIDX with a non-NULL `base_midx`. Note that after 0c5a62f14b (midx-write.c: do not read existing MIDX with `packs_to_include`, 2024-06-11), we do not use this function with an existing MIDX (incremental or not) when generating a MIDX with --stdin-packs, and likewise for incremental MIDXs. But it is still used when adding the fanout table from an incremental MIDX when generating a non-incremental MIDX (without --stdin-packs, of course). Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx-write.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/midx-write.c b/midx-write.c index 478b42e720..d5275d719b 100644 --- a/midx-write.c +++ b/midx-write.c @@ -196,7 +196,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m, struct pack_midx_entry *e, uint32_t pos) { - if (pos >= m->num_objects) + if (pos >= m->num_objects + m->num_objects_in_base) return 1; nth_midxed_object_oid(&e->oid, m, pos); @@ -247,12 +247,16 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout, uint32_t cur_fanout, int preferred_pack) { - uint32_t start = 0, end; + uint32_t start = m->num_objects_in_base, end; uint32_t cur_object; + if (m->base_midx) + midx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout, + preferred_pack); + if (cur_fanout) - start = ntohl(m->chunk_oid_fanout[cur_fanout - 1]); - end = ntohl(m->chunk_oid_fanout[cur_fanout]); + start += ntohl(m->chunk_oid_fanout[cur_fanout - 1]); + end = m->num_objects_in_base + ntohl(m->chunk_oid_fanout[cur_fanout]); for (cur_object = start; cur_object < end; cur_object++) { if ((preferred_pack > -1) && From b80236d0e3386e5fdfaff0a7480212a3f39cbee3 Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:55 -0400 Subject: [PATCH 15/19] midx: support reading incremental MIDX chains Now that the MIDX machinery's internals have been taught to understand incremental MIDXs over the previous handful of commits, the MIDX machinery itself can begin reading incremental MIDXs. (Note that while the on-disk format for incremental MIDXs has been defined, the writing end has not been implemented. This will take place in the commit after next.) The core of this change involves following the order specified in the MIDX chain in reverse and opening up MIDXs in the chain one-by-one, adding them to the previous layer's `->base_midx` pointer at each step. In order to implement this, the `load_multi_pack_index()` function is taught to call a new `load_multi_pack_index_chain()` function if loading a non-incremental MIDX failed via `load_multi_pack_index_one()`. When loading a MIDX chain, `load_midx_chain_fd_st()` reads each line in the file one-by-one and dispatches calls to `load_multi_pack_index_one()` to read each layer of the MIDX chain. When a layer was successfully read, it is added to the MIDX chain by calling `add_midx_to_chain()` which validates the contents of the `BASE` chunk, performs some bounds checks on the number of combined packs and objects, and attaches the new MIDX by assigning its `base_midx` pointer to the existing part of the chain. As a supplement to this, introduce a new mode in the test-read-midx test-tool which allows us to read the information for a specific MIDX in the chain by specifying its trailing checksum via the command-line arguments like so: $ test-tool read-midx .git/objects [checksum] Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.c | 184 +++++++++++++++++++++++++++++++++++--- midx.h | 7 ++ packfile.c | 5 +- t/helper/test-read-midx.c | 24 +++-- 4 files changed, 201 insertions(+), 19 deletions(-) diff --git a/midx.c b/midx.c index 918349576f..54c06cbb86 100644 --- a/midx.c +++ b/midx.c @@ -91,7 +91,9 @@ static int midx_read_object_offsets(const unsigned char *chunk_start, #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz) -struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local) +static struct multi_pack_index *load_multi_pack_index_one(const char *object_dir, + const char *midx_name, + int local) { struct multi_pack_index *m = NULL; int fd; @@ -99,31 +101,26 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local size_t midx_size; void *midx_map = NULL; uint32_t hash_version; - struct strbuf midx_name = STRBUF_INIT; uint32_t i; const char *cur_pack_name; struct chunkfile *cf = NULL; - get_midx_filename(&midx_name, object_dir); - - fd = git_open(midx_name.buf); + fd = git_open(midx_name); if (fd < 0) goto cleanup_fail; if (fstat(fd, &st)) { - error_errno(_("failed to read %s"), midx_name.buf); + error_errno(_("failed to read %s"), midx_name); goto cleanup_fail; } midx_size = xsize_t(st.st_size); if (midx_size < MIDX_MIN_SIZE) { - error(_("multi-pack-index file %s is too small"), midx_name.buf); + error(_("multi-pack-index file %s is too small"), midx_name); goto cleanup_fail; } - strbuf_release(&midx_name); - midx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0); close(fd); @@ -213,7 +210,6 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local cleanup_fail: free(m); - strbuf_release(&midx_name); free_chunkfile(cf); if (midx_map) munmap(midx_map, midx_size); @@ -222,6 +218,173 @@ cleanup_fail: return NULL; } +void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir) +{ + strbuf_addf(buf, "%s/pack/multi-pack-index.d", object_dir); +} + +void get_midx_chain_filename(struct strbuf *buf, const char *object_dir) +{ + get_midx_chain_dirname(buf, object_dir); + strbuf_addstr(buf, "/multi-pack-index-chain"); +} + +void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir, + const unsigned char *hash, const char *ext) +{ + get_midx_chain_dirname(buf, object_dir); + strbuf_addf(buf, "/multi-pack-index-%s.%s", hash_to_hex(hash), ext); +} + +static int open_multi_pack_index_chain(const char *chain_file, + int *fd, struct stat *st) +{ + *fd = git_open(chain_file); + if (*fd < 0) + return 0; + if (fstat(*fd, st)) { + close(*fd); + return 0; + } + if (st->st_size < the_hash_algo->hexsz) { + close(*fd); + if (!st->st_size) { + /* treat empty files the same as missing */ + errno = ENOENT; + } else { + warning(_("multi-pack-index chain file too small")); + errno = EINVAL; + } + return 0; + } + return 1; +} + +static int add_midx_to_chain(struct multi_pack_index *midx, + struct multi_pack_index *midx_chain, + struct object_id *oids, + int n) +{ + if (midx_chain) { + if (unsigned_add_overflows(midx_chain->num_packs, + midx_chain->num_packs_in_base)) { + warning(_("pack count in base MIDX too high: %"PRIuMAX), + (uintmax_t)midx_chain->num_packs_in_base); + return 0; + } + if (unsigned_add_overflows(midx_chain->num_objects, + midx_chain->num_objects_in_base)) { + warning(_("object count in base MIDX too high: %"PRIuMAX), + (uintmax_t)midx_chain->num_objects_in_base); + return 0; + } + midx->num_packs_in_base = midx_chain->num_packs + + midx_chain->num_packs_in_base; + midx->num_objects_in_base = midx_chain->num_objects + + midx_chain->num_objects_in_base; + } + + midx->base_midx = midx_chain; + midx->has_chain = 1; + + return 1; +} + +static struct multi_pack_index *load_midx_chain_fd_st(const char *object_dir, + int local, + int fd, struct stat *st, + int *incomplete_chain) +{ + struct multi_pack_index *midx_chain = NULL; + struct strbuf buf = STRBUF_INIT; + struct object_id *layers = NULL; + int valid = 1; + uint32_t i, count; + FILE *fp = xfdopen(fd, "r"); + + count = st->st_size / (the_hash_algo->hexsz + 1); + CALLOC_ARRAY(layers, count); + + for (i = 0; i < count; i++) { + struct multi_pack_index *m; + + if (strbuf_getline_lf(&buf, fp) == EOF) + break; + + if (get_oid_hex(buf.buf, &layers[i])) { + warning(_("invalid multi-pack-index chain: line '%s' " + "not a hash"), + buf.buf); + valid = 0; + break; + } + + valid = 0; + + strbuf_reset(&buf); + get_split_midx_filename_ext(&buf, object_dir, layers[i].hash, + MIDX_EXT_MIDX); + m = load_multi_pack_index_one(object_dir, buf.buf, local); + + if (m) { + if (add_midx_to_chain(m, midx_chain, layers, i)) { + midx_chain = m; + valid = 1; + } else { + close_midx(m); + } + } + if (!valid) { + warning(_("unable to find all multi-pack index files")); + break; + } + } + + free(layers); + fclose(fp); + strbuf_release(&buf); + + *incomplete_chain = !valid; + return midx_chain; +} + +static struct multi_pack_index *load_multi_pack_index_chain(const char *object_dir, + int local) +{ + struct strbuf chain_file = STRBUF_INIT; + struct stat st; + int fd; + struct multi_pack_index *m = NULL; + + get_midx_chain_filename(&chain_file, object_dir); + if (open_multi_pack_index_chain(chain_file.buf, &fd, &st)) { + int incomplete; + /* ownership of fd is taken over by load function */ + m = load_midx_chain_fd_st(object_dir, local, fd, &st, + &incomplete); + } + + strbuf_release(&chain_file); + return m; +} + +struct multi_pack_index *load_multi_pack_index(const char *object_dir, + int local) +{ + struct strbuf midx_name = STRBUF_INIT; + struct multi_pack_index *m; + + get_midx_filename(&midx_name, object_dir); + + m = load_multi_pack_index_one(object_dir, midx_name.buf, local); + if (!m) + m = load_multi_pack_index_chain(object_dir, local); + + strbuf_release(&midx_name); + + return m; +} + void close_midx(struct multi_pack_index *m) { uint32_t i; @@ -230,6 +393,7 @@ void close_midx(struct multi_pack_index *m) return; close_midx(m->next); + close_midx(m->base_midx); munmap((unsigned char *)m->data, m->data_len); diff --git a/midx.h b/midx.h index 86af7dfc5e..94de16a8c4 100644 --- a/midx.h +++ b/midx.h @@ -24,6 +24,7 @@ struct bitmapped_pack; #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* "OOFF" */ #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* "LOFF" */ #define MIDX_CHUNKID_REVINDEX 0x52494458 /* "RIDX" */ +#define MIDX_CHUNKID_BASE 0x42415345 /* "BASE" */ #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t)) #define MIDX_LARGE_OFFSET_NEEDED 0x80000000 @@ -50,6 +51,7 @@ struct multi_pack_index { int preferred_pack_idx; int local; + int has_chain; const unsigned char *chunk_pack_names; size_t chunk_pack_names_len; @@ -80,11 +82,16 @@ struct multi_pack_index { #define MIDX_EXT_REV "rev" #define MIDX_EXT_BITMAP "bitmap" +#define MIDX_EXT_MIDX "midx" const unsigned char *get_midx_checksum(struct multi_pack_index *m); void get_midx_filename(struct strbuf *out, const char *object_dir); void get_midx_filename_ext(struct strbuf *out, const char *object_dir, const unsigned char *hash, const char *ext); +void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir); +void get_midx_chain_filename(struct strbuf *buf, const char *object_dir); +void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir, + const unsigned char *hash, const char *ext); struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local); int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id); diff --git a/packfile.c b/packfile.c index 813584646f..1eb18e3041 100644 --- a/packfile.c +++ b/packfile.c @@ -880,7 +880,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len, if (!report_garbage) return; - if (!strcmp(file_name, "multi-pack-index")) + if (!strcmp(file_name, "multi-pack-index") || + !strcmp(file_name, "multi-pack-index.d")) return; if (starts_with(file_name, "multi-pack-index") && (ends_with(file_name, ".bitmap") || ends_with(file_name, ".rev"))) @@ -1064,7 +1065,7 @@ struct packed_git *get_all_packs(struct repository *r) prepare_packed_git(r); for (m = r->objects->multi_pack_index; m; m = m->next) { uint32_t i; - for (i = 0; i < m->num_packs; i++) + for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) prepare_midx_pack(r, m, i); } diff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c index 83effc2b5f..69757e94fc 100644 --- a/t/helper/test-read-midx.c +++ b/t/helper/test-read-midx.c @@ -9,8 +9,10 @@ #include "packfile.h" #include "setup.h" #include "gettext.h" +#include "pack-revindex.h" -static int read_midx_file(const char *object_dir, int show_objects) +static int read_midx_file(const char *object_dir, const char *checksum, + int show_objects) { uint32_t i; struct multi_pack_index *m; @@ -21,6 +23,13 @@ static int read_midx_file(const char *object_dir, int show_objects) if (!m) return 1; + if (checksum) { + while (m && strcmp(hash_to_hex(get_midx_checksum(m)), checksum)) + m = m->base_midx; + if (!m) + return 1; + } + printf("header: %08x %d %d %d %d\n", m->signature, m->version, @@ -54,7 +63,8 @@ static int read_midx_file(const char *object_dir, int show_objects) struct pack_entry e; for (i = 0; i < m->num_objects; i++) { - nth_midxed_object_oid(&oid, m, i); + nth_midxed_object_oid(&oid, m, + i + m->num_objects_in_base); fill_midx_entry(the_repository, &oid, &e, m); printf("%s %"PRIu64"\t%s\n", @@ -111,7 +121,7 @@ static int read_midx_bitmapped_packs(const char *object_dir) if (!midx) return 1; - for (i = 0; i < midx->num_packs; i++) { + for (i = 0; i < midx->num_packs + midx->num_packs_in_base; i++) { if (nth_bitmapped_pack(the_repository, midx, &pack, i) < 0) return 1; @@ -127,16 +137,16 @@ static int read_midx_bitmapped_packs(const char *object_dir) int cmd__read_midx(int argc, const char **argv) { - if (!(argc == 2 || argc == 3)) - usage("read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] "); + if (!(argc == 2 || argc == 3 || argc == 4)) + usage("read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] "); if (!strcmp(argv[1], "--show-objects")) - return read_midx_file(argv[2], 1); + return read_midx_file(argv[2], argv[3], 1); else if (!strcmp(argv[1], "--checksum")) return read_midx_checksum(argv[2]); else if (!strcmp(argv[1], "--preferred-pack")) return read_midx_preferred_pack(argv[2]); else if (!strcmp(argv[1], "--bitmap")) return read_midx_bitmapped_packs(argv[2]); - return read_midx_file(argv[1], 0); + return read_midx_file(argv[1], argv[2], 0); } From 3592796d0a176f51bb257b31dd9afe12a5af7932 Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:37:58 -0400 Subject: [PATCH 16/19] midx: implement verification support for incremental MIDXs Teach the verification implementation used by `git multi-pack-index verify` to perform verification for incremental MIDX chains by independently validating each layer within the chain. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- midx.c | 47 ++++++++++++++++++++++++++++++----------------- midx.h | 2 ++ 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/midx.c b/midx.c index 54c06cbb86..a53d65702d 100644 --- a/midx.c +++ b/midx.c @@ -470,6 +470,13 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, return 0; } +struct packed_git *nth_midxed_pack(struct multi_pack_index *m, + uint32_t pack_int_id) +{ + uint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id); + return m->packs[local_pack_int_id]; +} + #define MIDX_CHUNK_BITMAPPED_PACKS_WIDTH (2 * sizeof(uint32_t)) int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, @@ -818,6 +825,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag uint32_t i; struct progress *progress = NULL; struct multi_pack_index *m = load_multi_pack_index(object_dir, 1); + struct multi_pack_index *curr; verify_midx_error = 0; if (!m) { @@ -840,8 +848,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag if (flags & MIDX_PROGRESS) progress = start_delayed_progress(_("Looking for referenced packfiles"), - m->num_packs); - for (i = 0; i < m->num_packs; i++) { + m->num_packs + m->num_packs_in_base); + for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) { if (prepare_midx_pack(r, m, i)) midx_report("failed to load pack in position %d", i); @@ -861,17 +869,20 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag if (flags & MIDX_PROGRESS) progress = start_sparse_progress(_("Verifying OID order in multi-pack-index"), m->num_objects - 1); - for (i = 0; i < m->num_objects - 1; i++) { - struct object_id oid1, oid2; - nth_midxed_object_oid(&oid1, m, i); - nth_midxed_object_oid(&oid2, m, i + 1); + for (curr = m; curr; curr = curr->base_midx) { + for (i = 0; i < m->num_objects - 1; i++) { + struct object_id oid1, oid2; - if (oidcmp(&oid1, &oid2) >= 0) - midx_report(_("oid lookup out of order: oid[%d] = %s >= %s = oid[%d]"), - i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1); + nth_midxed_object_oid(&oid1, m, m->num_objects_in_base + i); + nth_midxed_object_oid(&oid2, m, m->num_objects_in_base + i + 1); - midx_display_sparse_progress(progress, i + 1); + if (oidcmp(&oid1, &oid2) >= 0) + midx_report(_("oid lookup out of order: oid[%d] = %s >= %s = oid[%d]"), + i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1); + + midx_display_sparse_progress(progress, i + 1); + } } stop_progress(&progress); @@ -881,8 +892,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag * each of the objects and only require 1 packfile to be open at a * time. */ - ALLOC_ARRAY(pairs, m->num_objects); - for (i = 0; i < m->num_objects; i++) { + ALLOC_ARRAY(pairs, m->num_objects + m->num_objects_in_base); + for (i = 0; i < m->num_objects + m->num_objects_in_base; i++) { pairs[i].pos = i; pairs[i].pack_int_id = nth_midxed_pack_int_id(m, i); } @@ -896,16 +907,18 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag if (flags & MIDX_PROGRESS) progress = start_sparse_progress(_("Verifying object offsets"), m->num_objects); - for (i = 0; i < m->num_objects; i++) { + for (i = 0; i < m->num_objects + m->num_objects_in_base; i++) { struct object_id oid; struct pack_entry e; off_t m_offset, p_offset; if (i > 0 && pairs[i-1].pack_int_id != pairs[i].pack_int_id && - m->packs[pairs[i-1].pack_int_id]) - { - close_pack_fd(m->packs[pairs[i-1].pack_int_id]); - close_pack_index(m->packs[pairs[i-1].pack_int_id]); + nth_midxed_pack(m, pairs[i-1].pack_int_id)) { + uint32_t pack_int_id = pairs[i-1].pack_int_id; + struct packed_git *p = nth_midxed_pack(m, pack_int_id); + + close_pack_fd(p); + close_pack_index(p); } nth_midxed_object_oid(&oid, m, pairs[i].pos); diff --git a/midx.h b/midx.h index 94de16a8c4..9d30935589 100644 --- a/midx.h +++ b/midx.h @@ -95,6 +95,8 @@ void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir, struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local); int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id); +struct packed_git *nth_midxed_pack(struct multi_pack_index *m, + uint32_t pack_int_id); int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, struct bitmapped_pack *bp, uint32_t pack_int_id); int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, From 9552c3595a77027a3d29b7a943bc2ecdba1c6813 Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:38:01 -0400 Subject: [PATCH 17/19] t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' Two years ago, commit ff1e653c8e2 (midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP', 2021-08-31) introduced a new environment variable which caused the test suite to write MIDX bitmaps after any 'git repack' invocation. At the time, this was done to help flush out any bugs with MIDX bitmaps that weren't explicitly covered in the t5326-multi-pack-bitmap.sh script. Two years later, that flag has served us well and is no longer providing meaningful coverage, as the script in t5326 has matured substantially and covers many more interesting cases than it did back when ff1e653c8e2 was originally written. Remove the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment variable as it is no longer serving a useful purpose. More importantly, removing this variable clears the way for us to introduce a new one to help similarly flush out bugs related to incremental MIDX chains. Because these incremental MIDX chains are (for now) incompatible with MIDX bitmaps, we cannot have both. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- builtin/repack.c | 12 ++---------- ci/run-build-and-tests.sh | 1 - midx.h | 2 -- t/README | 4 ---- t/t0410-partial-clone.sh | 2 -- t/t5310-pack-bitmaps.sh | 4 ---- t/t5319-multi-pack-index.sh | 3 +-- t/t5326-multi-pack-bitmaps.sh | 3 +-- t/t5327-multi-pack-bitmaps-rev.sh | 5 ++--- t/t7700-repack.sh | 21 +++++++-------------- 10 files changed, 13 insertions(+), 44 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index f0317fa94a..8499bf0e12 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -1217,10 +1217,6 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (!write_midx && (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository())) write_bitmaps = 0; - } else if (write_bitmaps && - git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) && - git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) { - write_bitmaps = 0; } if (pack_kept_objects < 0) pack_kept_objects = write_bitmaps > 0 && !write_midx; @@ -1518,12 +1514,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (run_update_server_info) update_server_info(0); - if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) { - unsigned flags = 0; - if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) - flags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX; - write_midx_file(get_object_directory(), NULL, NULL, flags); - } + if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) + write_midx_file(get_object_directory(), NULL, NULL, 0); cleanup: string_list_clear(&names, 1); diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index 98dda42045..e6fd68630c 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -25,7 +25,6 @@ linux-TEST-vars) export GIT_TEST_COMMIT_GRAPH=1 export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1 export GIT_TEST_MULTI_PACK_INDEX=1 - export GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master export GIT_TEST_NO_WRITE_REV_INDEX=1 export GIT_TEST_CHECKOUT_WORKERS=2 diff --git a/midx.h b/midx.h index 9d30935589..3714cad2cc 100644 --- a/midx.h +++ b/midx.h @@ -29,8 +29,6 @@ struct bitmapped_pack; #define MIDX_LARGE_OFFSET_NEEDED 0x80000000 #define GIT_TEST_MULTI_PACK_INDEX "GIT_TEST_MULTI_PACK_INDEX" -#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \ - "GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP" struct multi_pack_index { struct multi_pack_index *next; diff --git a/t/README b/t/README index d9e0e07506..e8a11926e4 100644 --- a/t/README +++ b/t/README @@ -469,10 +469,6 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack- index to be written after every 'git repack' command, and overrides the 'core.multiPackIndex' setting to true. -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=, when true, sets the -'--bitmap' option on all invocations of 'git multi-pack-index write', -and ignores pack-objects' '--write-bitmap-index'. - GIT_TEST_SIDEBAND_ALL=, when true, overrides the 'uploadpack.allowSidebandAll' setting to true, and when false, forces fetch-pack to not request sideband-all (even if the server advertises diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 2c30c86e7b..34bdb3ab1f 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -5,8 +5,6 @@ test_description='partial clone' . ./test-lib.sh . "$TEST_DIRECTORY"/lib-terminal.sh -# missing promisor objects cause repacks which write bitmaps to fail -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 # When enabled, some commands will write commit-graphs. This causes fsck # to fail when delete_object() is called because fsck will attempt to # verify the out-of-sync commit graph. diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh index d7fd71360e..a6de7c5764 100755 --- a/t/t5310-pack-bitmaps.sh +++ b/t/t5310-pack-bitmaps.sh @@ -5,10 +5,6 @@ test_description='exercise basic bitmap functionality' . ./test-lib.sh . "$TEST_DIRECTORY"/lib-bitmap.sh -# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in -# their place. -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 - # Likewise, allow individual tests to control whether or not they use # the boundary-based traversal. sane_unset GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 10d2a6bf92..6e9ee23398 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -600,8 +600,7 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' ' compare_results_with_midx "after repack" test_expect_success 'multi-pack-index and pack-bitmap' ' - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -c repack.writeBitmaps=true repack -ad && + git -c repack.writeBitmaps=true repack -ad && git multi-pack-index write && git rev-list --test-bitmap HEAD ' diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh index 916da389b6..1cb3e3ff08 100755 --- a/t/t5326-multi-pack-bitmaps.sh +++ b/t/t5326-multi-pack-bitmaps.sh @@ -4,10 +4,9 @@ test_description='exercise basic multi-pack bitmap functionality' . ./test-lib.sh . "${TEST_DIRECTORY}/lib-bitmap.sh" -# We'll be writing our own midx and bitmaps, so avoid getting confused by the +# We'll be writing our own MIDX, so avoid getting confused by the # automatic ones. GIT_TEST_MULTI_PACK_INDEX=0 -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 # This test exercise multi-pack bitmap functionality where the object order is # stored and read from a special chunk within the MIDX, so use the default diff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh index e65e311cd7..23db949c20 100755 --- a/t/t5327-multi-pack-bitmaps-rev.sh +++ b/t/t5327-multi-pack-bitmaps-rev.sh @@ -5,10 +5,9 @@ test_description='exercise basic multi-pack bitmap functionality (.rev files)' . ./test-lib.sh . "${TEST_DIRECTORY}/lib-bitmap.sh" -# We'll be writing our own midx and bitmaps, so avoid getting confused by the -# automatic ones. +# We'll be writing our own MIDX, so avoid getting confused by the automatic +# ones. GIT_TEST_MULTI_PACK_INDEX=0 -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 # Unlike t5326, this test exercise multi-pack bitmap functionality where the # object order is stored in a separate .rev file. diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh index 127efe99f8..8f34f05087 100755 --- a/t/t7700-repack.sh +++ b/t/t7700-repack.sh @@ -70,14 +70,13 @@ test_expect_success 'objects in packs marked .keep are not repacked' ' test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' ' # build on $oid, $packid, and .keep state from previous - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl && + git repack -Adbl && test_has_duplicate_object true ' test_expect_success 'writing bitmaps via config can duplicate .keep objects' ' # build on $oid, $packid, and .keep state from previous - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -c repack.writebitmaps=true repack -Adl && + git -c repack.writebitmaps=true repack -Adl && test_has_duplicate_object true ' @@ -284,8 +283,7 @@ test_expect_success 'repacking fails when missing .pack actually means missing o test_expect_success 'bitmaps are created by default in bare repos' ' git clone --bare .git bare.git && rm -f bare.git/objects/pack/*.bitmap && - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -C bare.git repack -ad && + git -C bare.git repack -ad && bitmap=$(ls bare.git/objects/pack/*.bitmap) && test_path_is_file "$bitmap" ' @@ -296,8 +294,7 @@ test_expect_success 'incremental repack does not complain' ' ' test_expect_success 'bitmaps can be disabled on bare repos' ' - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -c repack.writeBitmaps=false -C bare.git repack -ad && + git -c repack.writeBitmaps=false -C bare.git repack -ad && bitmap=$(ls bare.git/objects/pack/*.bitmap || :) && test -z "$bitmap" ' @@ -308,8 +305,7 @@ test_expect_success 'no bitmaps created if .keep files present' ' keep=${pack%.pack}.keep && test_when_finished "rm -f \"\$keep\"" && >"$keep" && - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -C bare.git repack -ad 2>stderr && + git -C bare.git repack -ad 2>stderr && test_must_be_empty stderr && find bare.git/objects/pack/ -type f -name "*.bitmap" >actual && test_must_be_empty actual @@ -320,8 +316,7 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' ' blob=$(test-tool genrandom big $((1024*1024)) | git -C bare.git hash-object -w --stdin) && git -C bare.git update-ref refs/tags/big $blob && - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -C bare.git repack -ad 2>stderr && + git -C bare.git repack -ad 2>stderr && test_must_be_empty stderr && find bare.git/objects/pack -type f -name "*.bitmap" >actual && test_must_be_empty actual @@ -342,9 +337,7 @@ test_expect_success 'repacking with a filter works' ' ' test_expect_success '--filter fails with --write-bitmap-index' ' - test_must_fail \ - env GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none + test_must_fail git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none ' test_expect_success 'repacking with two filters works' ' From 147c3f6740773d0cf9095ad9fd7c68f1c1348c9a Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:38:04 -0400 Subject: [PATCH 18/19] t/t5313-pack-bounds-checks.sh: prepare for sub-directories Prepare for sub-directories to appear in $GIT_DIR/objects/pack by adjusting the copy, remove, and chmod invocations to perform their behavior recursively. This prepares us for the new $GIT_DIR/objects/pack/multi-pack-index.d directory which will be added in a following commit. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- t/t5313-pack-bounds-checks.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/t/t5313-pack-bounds-checks.sh b/t/t5313-pack-bounds-checks.sh index ceaa6700a2..86fc73f9fb 100755 --- a/t/t5313-pack-bounds-checks.sh +++ b/t/t5313-pack-bounds-checks.sh @@ -7,11 +7,11 @@ TEST_PASSES_SANITIZE_LEAK=true clear_base () { test_when_finished 'restore_base' && - rm -f $base + rm -r -f $base } restore_base () { - cp base-backup/* .git/objects/pack/ + cp -r base-backup/* .git/objects/pack/ } do_pack () { @@ -64,9 +64,9 @@ test_expect_success 'set up base packfile and variables' ' git commit -m base && git repack -ad && base=$(echo .git/objects/pack/*) && - chmod +w $base && + chmod -R +w $base && mkdir base-backup && - cp $base base-backup/ && + cp -r $base base-backup/ && object=$(git rev-parse HEAD:file) ' From fcb2205b77470c60f996a3206b2d4aebf6e951e3 Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Tue, 6 Aug 2024 11:38:07 -0400 Subject: [PATCH 19/19] midx: implement support for writing incremental MIDX chains Now that the rest of the MIDX subsystem and relevant callers have been updated to learn about how to read and process incremental MIDX chains, let's finally update the implementation in `write_midx_internal()` to be able to write incremental MIDX chains. This new feature is available behind the `--incremental` option for the `multi-pack-index` builtin, like so: $ git multi-pack-index write --incremental The implementation for doing so is relatively straightforward, and boils down to a handful of different kinds of changes implemented in this patch: - The `compute_sorted_entries()` function is taught to reject objects which appear in any existing MIDX layer. - Functions like `write_midx_revindex()` are adjusted to write pack_order values which are offset by the number of objects in the base MIDX layer. - The end of `write_midx_internal()` is adjusted to move non-incremental MIDX files when necessary (i.e. when creating an incremental chain with an existing non-incremental MIDX in the repository). There are a handful of other changes that are introduced, like new functions to clear incremental MIDX files that are unrelated to the current chain (using the same "keep_hash" mechanism as in the non-incremental case). The tests explicitly exercising the new incremental MIDX feature are relatively limited for two reasons: 1. Most of the "interesting" behavior is already thoroughly covered in t5319-multi-pack-index.sh, which handles the core logic of reading objects through a MIDX. The new tests in t5334-incremental-multi-pack-index.sh are mostly focused on creating and destroying incremental MIDXs, as well as stitching their results together across layers. 2. A new GIT_TEST environment variable is added called "GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL", which modifies the entire test suite to write incremental MIDXs after repacking when combined with the "GIT_TEST_MULTI_PACK_INDEX" variable. This exercises the long tail of other interesting behavior that is defined implicitly throughout the rest of the CI suite. It is likewise added to the linux-TEST-vars job. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- Documentation/git-multi-pack-index.txt | 11 +- builtin/multi-pack-index.c | 2 + builtin/repack.c | 8 +- ci/run-build-and-tests.sh | 1 + midx-write.c | 312 ++++++++++++++++++++---- midx.c | 62 ++++- midx.h | 4 + packfile.c | 16 +- packfile.h | 4 + t/README | 4 + t/lib-bitmap.sh | 6 +- t/lib-midx.sh | 28 +++ t/t5319-multi-pack-index.sh | 27 +- t/t5326-multi-pack-bitmaps.sh | 1 + t/t5327-multi-pack-bitmaps-rev.sh | 1 + t/t5332-multi-pack-reuse.sh | 2 + t/t5334-incremental-multi-pack-index.sh | 46 ++++ t/t7700-repack.sh | 27 +- 18 files changed, 459 insertions(+), 103 deletions(-) create mode 100755 t/t5334-incremental-multi-pack-index.sh diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt index 3696506eb3..631d5c7d15 100644 --- a/Documentation/git-multi-pack-index.txt +++ b/Documentation/git-multi-pack-index.txt @@ -64,6 +64,12 @@ The file given at `` is expected to be readable, and can contain duplicates. (If a given OID is given more than once, it is marked as preferred if at least one instance of it begins with the special `+` marker). + + --incremental:: + Write an incremental MIDX file containing only objects + and packs not present in an existing MIDX layer. + Migrates non-incremental MIDXs to incremental ones when + necessary. Incompatible with `--bitmap`. -- verify:: @@ -74,6 +80,8 @@ expire:: have no objects referenced by the MIDX (with the exception of `.keep` packs and cruft packs). Rewrite the MIDX file afterward to remove all references to these pack-files. ++ +NOTE: this mode is incompatible with incremental MIDX files. repack:: Create a new pack-file containing objects in small pack-files @@ -95,7 +103,8 @@ repack:: + If `repack.packKeptObjects` is `false`, then any pack-files with an associated `.keep` file will not be selected for the batch to repack. - ++ +NOTE: this mode is incompatible with incremental MIDX files. EXAMPLES -------- diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index 9cf1a32d65..8805cbbeb3 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -129,6 +129,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv, MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX), OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS), + OPT_BIT(0, "incremental", &opts.flags, + N_("write a new incremental MIDX"), MIDX_WRITE_INCREMENTAL), OPT_BOOL(0, "stdin-packs", &opts.stdin_packs, N_("write multi-pack index containing only given indexes")), OPT_FILENAME(0, "refs-snapshot", &opts.refs_snapshot, diff --git a/builtin/repack.c b/builtin/repack.c index 8499bf0e12..7608430a37 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -1514,8 +1514,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (run_update_server_info) update_server_info(0); - if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) - write_midx_file(get_object_directory(), NULL, NULL, 0); + if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) { + unsigned flags = 0; + if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0)) + flags |= MIDX_WRITE_INCREMENTAL; + write_midx_file(get_object_directory(), NULL, NULL, flags); + } cleanup: string_list_clear(&names, 1); diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index e6fd68630c..2e28d02b20 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -25,6 +25,7 @@ linux-TEST-vars) export GIT_TEST_COMMIT_GRAPH=1 export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1 export GIT_TEST_MULTI_PACK_INDEX=1 + export GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master export GIT_TEST_NO_WRITE_REV_INDEX=1 export GIT_TEST_CHECKOUT_WORKERS=2 diff --git a/midx-write.c b/midx-write.c index d5275d719b..a94cb28bfd 100644 --- a/midx-write.c +++ b/midx-write.c @@ -17,6 +17,8 @@ #include "refs.h" #include "revision.h" #include "list-objects.h" +#include "path.h" +#include "pack-revindex.h" #define PACK_EXPIRED UINT_MAX #define BITMAP_POS_UNKNOWN (~((uint32_t)0)) @@ -25,7 +27,11 @@ extern int midx_checksum_valid(struct multi_pack_index *m); extern void clear_midx_files_ext(const char *object_dir, const char *ext, - unsigned char *keep_hash); + const char *keep_hash); +extern void clear_incremental_midx_files_ext(const char *object_dir, + const char *ext, + const char **keep_hashes, + uint32_t hashes_nr); extern int cmp_idx_or_pack_name(const char *idx_or_pack_name, const char *idx_name); @@ -86,6 +92,7 @@ struct write_midx_context { size_t nr; size_t alloc; struct multi_pack_index *m; + struct multi_pack_index *base_midx; struct progress *progress; unsigned pack_paths_checked; @@ -99,6 +106,9 @@ struct write_midx_context { int preferred_pack_idx; + int incremental; + uint32_t num_multi_pack_indexes_before; + struct string_list *to_include; }; @@ -122,6 +132,9 @@ static int should_include_pack(const struct write_midx_context *ctx, */ if (ctx->m && midx_contains_pack(ctx->m, file_name)) return 0; + else if (ctx->base_midx && midx_contains_pack(ctx->base_midx, + file_name)) + return 0; else if (ctx->to_include && !string_list_has_string(ctx->to_include, file_name)) return 0; @@ -338,7 +351,7 @@ static void compute_sorted_entries(struct write_midx_context *ctx, for (cur_fanout = 0; cur_fanout < 256; cur_fanout++) { fanout.nr = 0; - if (ctx->m) + if (ctx->m && !ctx->incremental) midx_fanout_add_midx_fanout(&fanout, ctx->m, cur_fanout, ctx->preferred_pack_idx); @@ -364,6 +377,10 @@ static void compute_sorted_entries(struct write_midx_context *ctx, if (cur_object && oideq(&fanout.entries[cur_object - 1].oid, &fanout.entries[cur_object].oid)) continue; + if (ctx->incremental && ctx->base_midx && + midx_has_oid(ctx->base_midx, + &fanout.entries[cur_object].oid)) + continue; ALLOC_GROW(ctx->entries, st_add(ctx->entries_nr, 1), alloc_objects); @@ -547,10 +564,16 @@ static int write_midx_revindex(struct hashfile *f, void *data) { struct write_midx_context *ctx = data; - uint32_t i; + uint32_t i, nr_base; + + if (ctx->incremental && ctx->base_midx) + nr_base = ctx->base_midx->num_objects + + ctx->base_midx->num_objects_in_base; + else + nr_base = 0; for (i = 0; i < ctx->entries_nr; i++) - hashwrite_be32(f, ctx->pack_order[i]); + hashwrite_be32(f, ctx->pack_order[i] + nr_base); return 0; } @@ -579,12 +602,18 @@ static int midx_pack_order_cmp(const void *va, const void *vb) static uint32_t *midx_pack_order(struct write_midx_context *ctx) { struct midx_pack_order_data *data; - uint32_t *pack_order; + uint32_t *pack_order, base_objects = 0; uint32_t i; trace2_region_enter("midx", "midx_pack_order", the_repository); + if (ctx->incremental && ctx->base_midx) + base_objects = ctx->base_midx->num_objects + + ctx->base_midx->num_objects_in_base; + + ALLOC_ARRAY(pack_order, ctx->entries_nr); ALLOC_ARRAY(data, ctx->entries_nr); + for (i = 0; i < ctx->entries_nr; i++) { struct pack_midx_entry *e = &ctx->entries[i]; data[i].nr = i; @@ -596,12 +625,11 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx) QSORT(data, ctx->entries_nr, midx_pack_order_cmp); - ALLOC_ARRAY(pack_order, ctx->entries_nr); for (i = 0; i < ctx->entries_nr; i++) { struct pack_midx_entry *e = &ctx->entries[data[i].nr]; struct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]]; if (pack->bitmap_pos == BITMAP_POS_UNKNOWN) - pack->bitmap_pos = i; + pack->bitmap_pos = i + base_objects; pack->bitmap_nr++; pack_order[i] = data[i].nr; } @@ -649,7 +677,8 @@ static void prepare_midx_packing_data(struct packing_data *pdata, prepare_packing_data(the_repository, pdata); for (i = 0; i < ctx->entries_nr; i++) { - struct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]]; + uint32_t pos = ctx->pack_order[i]; + struct pack_midx_entry *from = &ctx->entries[pos]; struct object_entry *to = packlist_alloc(pdata, &from->oid); oe_set_in_pack(pdata, to, @@ -897,35 +926,128 @@ cleanup: static int fill_packs_from_midx(struct write_midx_context *ctx, const char *preferred_pack_name, uint32_t flags) { - uint32_t i; + struct multi_pack_index *m; - for (i = 0; i < ctx->m->num_packs; i++) { - ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); + for (m = ctx->m; m; m = m->base_midx) { + uint32_t i; + + for (i = 0; i < m->num_packs; i++) { + ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); - if (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) { /* * If generating a reverse index, need to have * packed_git's loaded to compare their * mtimes and object count. * - * * If a preferred pack is specified, need to * have packed_git's loaded to ensure the chosen * preferred pack has a non-zero object count. */ - if (prepare_midx_pack(the_repository, ctx->m, i)) - return error(_("could not load pack")); + if (flags & MIDX_WRITE_REV_INDEX || + preferred_pack_name) { + if (prepare_midx_pack(the_repository, m, + m->num_packs_in_base + i)) { + error(_("could not load pack")); + return 1; + } - if (open_pack_index(ctx->m->packs[i])) - die(_("could not open index for %s"), - ctx->m->packs[i]->pack_name); + if (open_pack_index(m->packs[i])) + die(_("could not open index for %s"), + m->packs[i]->pack_name); + } + + fill_pack_info(&ctx->info[ctx->nr++], m->packs[i], + m->pack_names[i], + m->num_packs_in_base + i); } + } + return 0; +} - fill_pack_info(&ctx->info[ctx->nr++], ctx->m->packs[i], - ctx->m->pack_names[i], i); +static struct { + const char *non_split; + const char *split; +} midx_exts[] = { + {NULL, MIDX_EXT_MIDX}, + {MIDX_EXT_BITMAP, MIDX_EXT_BITMAP}, + {MIDX_EXT_REV, MIDX_EXT_REV}, +}; + +static int link_midx_to_chain(struct multi_pack_index *m) +{ + struct strbuf from = STRBUF_INIT; + struct strbuf to = STRBUF_INIT; + int ret = 0; + size_t i; + + if (!m || m->has_chain) { + /* + * Either no MIDX previously existed, or it was already + * part of a MIDX chain. In both cases, we have nothing + * to link, so return early. + */ + goto done; } - return 0; + for (i = 0; i < ARRAY_SIZE(midx_exts); i++) { + const unsigned char *hash = get_midx_checksum(m); + + get_midx_filename_ext(&from, m->object_dir, hash, + midx_exts[i].non_split); + get_split_midx_filename_ext(&to, m->object_dir, hash, + midx_exts[i].split); + + if (link(from.buf, to.buf) < 0 && errno != ENOENT) { + ret = error_errno(_("unable to link '%s' to '%s'"), + from.buf, to.buf); + goto done; + } + + strbuf_reset(&from); + strbuf_reset(&to); + } + +done: + strbuf_release(&from); + strbuf_release(&to); + return ret; +} + +static void clear_midx_files(const char *object_dir, + const char **hashes, + uint32_t hashes_nr, + unsigned incremental) +{ + /* + * if incremental: + * - remove all non-incremental MIDX files + * - remove any incremental MIDX files not in the current one + * + * if non-incremental: + * - remove all incremental MIDX files + * - remove any non-incremental MIDX files not matching the current + * hash + */ + struct strbuf buf = STRBUF_INIT; + const char *exts[] = { MIDX_EXT_BITMAP, MIDX_EXT_REV, MIDX_EXT_MIDX }; + uint32_t i, j; + + for (i = 0; i < ARRAY_SIZE(exts); i++) { + clear_incremental_midx_files_ext(object_dir, exts[i], + hashes, hashes_nr); + for (j = 0; j < hashes_nr; j++) + clear_midx_files_ext(object_dir, exts[i], hashes[j]); + } + + if (incremental) + get_midx_filename(&buf, object_dir); + else + get_midx_chain_filename(&buf, object_dir); + + if (unlink(buf.buf) && errno != ENOENT) + die_errno(_("failed to clear multi-pack-index at %s"), buf.buf); + + strbuf_release(&buf); } static int write_midx_internal(const char *object_dir, @@ -940,42 +1062,66 @@ static int write_midx_internal(const char *object_dir, uint32_t i, start_pack; struct hashfile *f = NULL; struct lock_file lk; + struct tempfile *incr; struct write_midx_context ctx = { 0 }; int bitmapped_packs_concat_len = 0; int pack_name_concat_len = 0; int dropped_packs = 0; int result = 0; + const char **keep_hashes = NULL; struct chunkfile *cf; trace2_region_enter("midx", "write_midx_internal", the_repository); - get_midx_filename(&midx_name, object_dir); + ctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL); + if (ctx.incremental && (flags & MIDX_WRITE_BITMAP)) + die(_("cannot write incremental MIDX with bitmap")); + + if (ctx.incremental) + strbuf_addf(&midx_name, + "%s/pack/multi-pack-index.d/tmp_midx_XXXXXX", + object_dir); + else + get_midx_filename(&midx_name, object_dir); if (safe_create_leading_directories(midx_name.buf)) die_errno(_("unable to create leading directories of %s"), midx_name.buf); - if (!packs_to_include) { - /* - * Only reference an existing MIDX when not filtering which - * packs to include, since all packs and objects are copied - * blindly from an existing MIDX if one is present. - */ - ctx.m = lookup_multi_pack_index(the_repository, object_dir); - } + if (!packs_to_include || ctx.incremental) { + struct multi_pack_index *m = lookup_multi_pack_index(the_repository, + object_dir); + if (m && !midx_checksum_valid(m)) { + warning(_("ignoring existing multi-pack-index; checksum mismatch")); + m = NULL; + } - if (ctx.m && !midx_checksum_valid(ctx.m)) { - warning(_("ignoring existing multi-pack-index; checksum mismatch")); - ctx.m = NULL; + if (m) { + /* + * Only reference an existing MIDX when not filtering + * which packs to include, since all packs and objects + * are copied blindly from an existing MIDX if one is + * present. + */ + if (ctx.incremental) + ctx.base_midx = m; + else if (!packs_to_include) + ctx.m = m; + } } ctx.nr = 0; - ctx.alloc = ctx.m ? ctx.m->num_packs : 16; + ctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16; ctx.info = NULL; ALLOC_ARRAY(ctx.info, ctx.alloc); - if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name, - flags) < 0) { - result = 1; + if (ctx.incremental) { + struct multi_pack_index *m = ctx.base_midx; + while (m) { + ctx.num_multi_pack_indexes_before++; + m = m->base_midx; + } + } else if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name, + flags) < 0) { goto cleanup; } @@ -992,7 +1138,8 @@ static int write_midx_internal(const char *object_dir, for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx); stop_progress(&ctx.progress); - if ((ctx.m && ctx.nr == ctx.m->num_packs) && + if ((ctx.m && ctx.nr == ctx.m->num_packs + ctx.m->num_packs_in_base) && + !ctx.incremental && !(packs_to_include || packs_to_drop)) { struct bitmap_index *bitmap_git; int bitmap_exists; @@ -1008,12 +1155,14 @@ static int write_midx_internal(const char *object_dir, * corresponding bitmap (or one wasn't requested). */ if (!want_bitmap) - clear_midx_files_ext(object_dir, ".bitmap", - NULL); + clear_midx_files_ext(object_dir, "bitmap", NULL); goto cleanup; } } + if (ctx.incremental && !ctx.nr) + goto cleanup; /* nothing to do */ + if (preferred_pack_name) { ctx.preferred_pack_idx = -1; @@ -1159,8 +1308,30 @@ static int write_midx_internal(const char *object_dir, pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT); - hold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR); - f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk)); + if (ctx.incremental) { + struct strbuf lock_name = STRBUF_INIT; + + get_midx_chain_filename(&lock_name, object_dir); + hold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR); + strbuf_release(&lock_name); + + incr = mks_tempfile_m(midx_name.buf, 0444); + if (!incr) { + error(_("unable to create temporary MIDX layer")); + return -1; + } + + if (adjust_shared_perm(get_tempfile_path(incr))) { + error(_("unable to adjust shared permissions for '%s'"), + get_tempfile_path(incr)); + return -1; + } + + f = hashfd(get_tempfile_fd(incr), get_tempfile_path(incr)); + } else { + hold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR); + f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk)); + } if (ctx.nr - dropped_packs == 0) { error(_("no pack files to index.")); @@ -1253,14 +1424,55 @@ static int write_midx_internal(const char *object_dir, * have been freed in the previous if block. */ - if (ctx.m) + CALLOC_ARRAY(keep_hashes, ctx.num_multi_pack_indexes_before + 1); + + if (ctx.incremental) { + FILE *chainf = fdopen_lock_file(&lk, "w"); + struct strbuf final_midx_name = STRBUF_INIT; + struct multi_pack_index *m = ctx.base_midx; + + if (!chainf) { + error_errno(_("unable to open multi-pack-index chain file")); + return -1; + } + + if (link_midx_to_chain(ctx.base_midx) < 0) + return -1; + + get_split_midx_filename_ext(&final_midx_name, object_dir, + midx_hash, MIDX_EXT_MIDX); + + if (rename_tempfile(&incr, final_midx_name.buf) < 0) { + error_errno(_("unable to rename new multi-pack-index layer")); + return -1; + } + + keep_hashes[ctx.num_multi_pack_indexes_before] = + xstrdup(hash_to_hex(midx_hash)); + + for (i = 0; i < ctx.num_multi_pack_indexes_before; i++) { + uint32_t j = ctx.num_multi_pack_indexes_before - i - 1; + + keep_hashes[j] = xstrdup(hash_to_hex(get_midx_checksum(m))); + m = m->base_midx; + } + + for (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++) + fprintf(get_lock_file_fp(&lk), "%s\n", keep_hashes[i]); + } else { + keep_hashes[ctx.num_multi_pack_indexes_before] = + xstrdup(hash_to_hex(midx_hash)); + } + + if (ctx.m || ctx.base_midx) close_object_store(the_repository->objects); if (commit_lock_file(&lk) < 0) die_errno(_("could not write multi-pack-index")); - clear_midx_files_ext(object_dir, ".bitmap", midx_hash); - clear_midx_files_ext(object_dir, ".rev", midx_hash); + clear_midx_files(object_dir, keep_hashes, + ctx.num_multi_pack_indexes_before + 1, + ctx.incremental); cleanup: for (i = 0; i < ctx.nr; i++) { @@ -1275,6 +1487,11 @@ cleanup: free(ctx.entries); free(ctx.pack_perm); free(ctx.pack_order); + if (keep_hashes) { + for (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++) + free((char *)keep_hashes[i]); + free(keep_hashes); + } strbuf_release(&midx_name); trace2_region_leave("midx", "write_midx_internal", the_repository); @@ -1311,6 +1528,9 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla if (!m) return 0; + if (m->base_midx) + die(_("cannot expire packs from an incremental multi-pack-index")); + CALLOC_ARRAY(count, m->num_packs); if (flags & MIDX_PROGRESS) @@ -1485,6 +1705,8 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, if (!m) return 0; + if (m->base_midx) + die(_("cannot repack an incremental multi-pack-index")); CALLOC_ARRAY(include_pack, m->num_packs); diff --git a/midx.c b/midx.c index a53d65702d..c867b2b6c2 100644 --- a/midx.c +++ b/midx.c @@ -16,7 +16,10 @@ int midx_checksum_valid(struct multi_pack_index *m); void clear_midx_files_ext(const char *object_dir, const char *ext, - unsigned char *keep_hash); + const char *keep_hash); +void clear_incremental_midx_files_ext(const char *object_dir, const char *ext, + char **keep_hashes, + uint32_t hashes_nr); int cmp_idx_or_pack_name(const char *idx_or_pack_name, const char *idx_name); @@ -521,6 +524,11 @@ int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, return 0; } +int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid) +{ + return bsearch_midx(oid, m, NULL); +} + struct object_id *nth_midxed_object_oid(struct object_id *oid, struct multi_pack_index *m, uint32_t n) @@ -723,7 +731,8 @@ int midx_checksum_valid(struct multi_pack_index *m) } struct clear_midx_data { - char *keep; + char **keep; + uint32_t keep_nr; const char *ext; }; @@ -731,32 +740,63 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS const char *file_name, void *_data) { struct clear_midx_data *data = _data; + uint32_t i; if (!(starts_with(file_name, "multi-pack-index-") && ends_with(file_name, data->ext))) return; - if (data->keep && !strcmp(data->keep, file_name)) - return; - + for (i = 0; i < data->keep_nr; i++) { + if (!strcmp(data->keep[i], file_name)) + return; + } if (unlink(full_path)) die_errno(_("failed to remove %s"), full_path); } void clear_midx_files_ext(const char *object_dir, const char *ext, - unsigned char *keep_hash) + const char *keep_hash) { struct clear_midx_data data; memset(&data, 0, sizeof(struct clear_midx_data)); - if (keep_hash) - data.keep = xstrfmt("multi-pack-index-%s%s", - hash_to_hex(keep_hash), ext); + if (keep_hash) { + ALLOC_ARRAY(data.keep, 1); + + data.keep[0] = xstrfmt("multi-pack-index-%s.%s", keep_hash, ext); + data.keep_nr = 1; + } data.ext = ext; for_each_file_in_pack_dir(object_dir, clear_midx_file_ext, &data); + if (keep_hash) + free(data.keep[0]); + free(data.keep); +} + +void clear_incremental_midx_files_ext(const char *object_dir, const char *ext, + char **keep_hashes, + uint32_t hashes_nr) +{ + struct clear_midx_data data; + uint32_t i; + + memset(&data, 0, sizeof(struct clear_midx_data)); + + ALLOC_ARRAY(data.keep, hashes_nr); + for (i = 0; i < hashes_nr; i++) + data.keep[i] = xstrfmt("multi-pack-index-%s.%s", keep_hashes[i], + ext); + data.keep_nr = hashes_nr; + data.ext = ext; + + for_each_file_in_pack_subdir(object_dir, "multi-pack-index.d", + clear_midx_file_ext, &data); + + for (i = 0; i < hashes_nr; i++) + free(data.keep[i]); free(data.keep); } @@ -774,8 +814,8 @@ void clear_midx_file(struct repository *r) if (remove_path(midx.buf)) die(_("failed to clear multi-pack-index at %s"), midx.buf); - clear_midx_files_ext(r->objects->odb->path, ".bitmap", NULL); - clear_midx_files_ext(r->objects->odb->path, ".rev", NULL); + clear_midx_files_ext(r->objects->odb->path, MIDX_EXT_BITMAP, NULL); + clear_midx_files_ext(r->objects->odb->path, MIDX_EXT_REV, NULL); strbuf_release(&midx); } diff --git a/midx.h b/midx.h index 3714cad2cc..42d4f8d149 100644 --- a/midx.h +++ b/midx.h @@ -29,6 +29,8 @@ struct bitmapped_pack; #define MIDX_LARGE_OFFSET_NEEDED 0x80000000 #define GIT_TEST_MULTI_PACK_INDEX "GIT_TEST_MULTI_PACK_INDEX" +#define GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL \ + "GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL" struct multi_pack_index { struct multi_pack_index *next; @@ -77,6 +79,7 @@ struct multi_pack_index { #define MIDX_WRITE_BITMAP (1 << 2) #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3) #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4) +#define MIDX_WRITE_INCREMENTAL (1 << 5) #define MIDX_EXT_REV "rev" #define MIDX_EXT_BITMAP "bitmap" @@ -101,6 +104,7 @@ int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result); int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result); +int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid); off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos); uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos); struct object_id *nth_midxed_object_oid(struct object_id *oid, diff --git a/packfile.c b/packfile.c index 1eb18e3041..cf12a539ea 100644 --- a/packfile.c +++ b/packfile.c @@ -815,9 +815,10 @@ static void report_pack_garbage(struct string_list *list) report_helper(list, seen_bits, first, list->nr); } -void for_each_file_in_pack_dir(const char *objdir, - each_file_in_pack_dir_fn fn, - void *data) +void for_each_file_in_pack_subdir(const char *objdir, + const char *subdir, + each_file_in_pack_dir_fn fn, + void *data) { struct strbuf path = STRBUF_INIT; size_t dirnamelen; @@ -826,6 +827,8 @@ void for_each_file_in_pack_dir(const char *objdir, strbuf_addstr(&path, objdir); strbuf_addstr(&path, "/pack"); + if (subdir) + strbuf_addf(&path, "/%s", subdir); dir = opendir(path.buf); if (!dir) { if (errno != ENOENT) @@ -847,6 +850,13 @@ void for_each_file_in_pack_dir(const char *objdir, strbuf_release(&path); } +void for_each_file_in_pack_dir(const char *objdir, + each_file_in_pack_dir_fn fn, + void *data) +{ + for_each_file_in_pack_subdir(objdir, NULL, fn, data); +} + struct prepare_pack_data { struct repository *r; struct string_list *garbage; diff --git a/packfile.h b/packfile.h index eb18ec15db..0f78658229 100644 --- a/packfile.h +++ b/packfile.h @@ -55,6 +55,10 @@ struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path); typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len, const char *file_name, void *data); +void for_each_file_in_pack_subdir(const char *objdir, + const char *subdir, + each_file_in_pack_dir_fn fn, + void *data); void for_each_file_in_pack_dir(const char *objdir, each_file_in_pack_dir_fn fn, void *data); diff --git a/t/README b/t/README index e8a11926e4..e93a29de1b 100644 --- a/t/README +++ b/t/README @@ -469,6 +469,10 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack- index to be written after every 'git repack' command, and overrides the 'core.multiPackIndex' setting to true. +GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=, when true, sets +the '--incremental' option on all invocations of 'git multi-pack-index +write'. + GIT_TEST_SIDEBAND_ALL=, when true, overrides the 'uploadpack.allowSidebandAll' setting to true, and when false, forces fetch-pack to not request sideband-all (even if the server advertises diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh index f595937094..62aa6744a6 100644 --- a/t/lib-bitmap.sh +++ b/t/lib-bitmap.sh @@ -1,6 +1,8 @@ # Helpers for scripts testing bitmap functionality; see t5310 for # example usage. +. "$TEST_DIRECTORY"/lib-midx.sh + objdir=.git/objects midx=$objdir/pack/multi-pack-index @@ -264,10 +266,6 @@ have_delta () { test_cmp expect actual } -midx_checksum () { - test-tool read-midx --checksum "$1" -} - # midx_pack_source midx_pack_source () { test-tool read-midx --show-objects .git/objects | grep "^$1 " | cut -f2 diff --git a/t/lib-midx.sh b/t/lib-midx.sh index 1261994744..e38c609604 100644 --- a/t/lib-midx.sh +++ b/t/lib-midx.sh @@ -6,3 +6,31 @@ test_midx_consistent () { test_cmp expect actual && git multi-pack-index --object-dir=$1 verify } + +midx_checksum () { + test-tool read-midx --checksum "$1" +} + +midx_git_two_modes () { + git -c core.multiPackIndex=false $1 >expect && + git -c core.multiPackIndex=true $1 >actual && + if [ "$2" = "sorted" ] + then + sort expect.sorted && + mv expect.sorted expect && + sort actual.sorted && + mv actual.sorted actual + fi && + test_cmp expect actual +} + +compare_results_with_midx () { + MSG=$1 + test_expect_success "check normal git operations: $MSG" ' + midx_git_two_modes "rev-list --objects --all" && + midx_git_two_modes "log --raw" && + midx_git_two_modes "count-objects --verbose" && + midx_git_two_modes "cat-file --batch-all-objects --batch-check" && + midx_git_two_modes "cat-file --batch-all-objects --batch-check --unordered" sorted + ' +} diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 6e9ee23398..4b0b5a5c9f 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -3,8 +3,11 @@ test_description='multi-pack-indexes' . ./test-lib.sh . "$TEST_DIRECTORY"/lib-chunk.sh +. "$TEST_DIRECTORY"/lib-midx.sh GIT_TEST_MULTI_PACK_INDEX=0 +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 +GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0 objdir=.git/objects HASH_LEN=$(test_oid rawsz) @@ -107,30 +110,6 @@ test_expect_success 'write midx with one v1 pack' ' midx_read_expect 1 18 4 $objdir ' -midx_git_two_modes () { - git -c core.multiPackIndex=false $1 >expect && - git -c core.multiPackIndex=true $1 >actual && - if [ "$2" = "sorted" ] - then - sort expect.sorted && - mv expect.sorted expect && - sort actual.sorted && - mv actual.sorted actual - fi && - test_cmp expect actual -} - -compare_results_with_midx () { - MSG=$1 - test_expect_success "check normal git operations: $MSG" ' - midx_git_two_modes "rev-list --objects --all" && - midx_git_two_modes "log --raw" && - midx_git_two_modes "count-objects --verbose" && - midx_git_two_modes "cat-file --batch-all-objects --batch-check" && - midx_git_two_modes "cat-file --batch-all-objects --batch-check --unordered" sorted - ' -} - test_expect_success 'write midx with one v2 pack' ' git pack-objects --index-version=2,0x40 $objdir/pack/test &2 && incrpackid=$(git pack-objects --all --unpacked --incremental .git/objects/pack/pack err && + git repack -Adl --write-bitmap-index 2>err && cat >expect <<-EOF && warning: disabling bitmap writing, as some objects are not being packed EOF @@ -533,11 +536,11 @@ test_expect_success 'setup for --write-midx tests' ' test_expect_success '--write-midx unchanged' ' ( cd midx && - GIT_TEST_MULTI_PACK_INDEX=0 git repack && + git repack && test_path_is_missing $midx && test_path_is_missing $midx-*.bitmap && - GIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx && + git repack --write-midx && test_path_is_file $midx && test_path_is_missing $midx-*.bitmap && @@ -550,7 +553,7 @@ test_expect_success '--write-midx with a new pack' ' cd midx && test_commit loose && - GIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx && + git repack --write-midx && test_path_is_file $midx && test_path_is_missing $midx-*.bitmap && @@ -561,7 +564,7 @@ test_expect_success '--write-midx with a new pack' ' test_expect_success '--write-midx with -b' ' ( cd midx && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -mb && + git repack -mb && test_path_is_file $midx && test_path_is_file $midx-*.bitmap && @@ -574,7 +577,7 @@ test_expect_success '--write-midx with -d' ' cd midx && test_commit repack && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad --write-midx && + git repack -Ad --write-midx && test_path_is_file $midx && test_path_is_missing $midx-*.bitmap && @@ -587,21 +590,21 @@ test_expect_success 'cleans up MIDX when appropriate' ' cd midx && test_commit repack-2 && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx && + git repack -Adb --write-midx && checksum=$(midx_checksum $objdir) && test_path_is_file $midx && test_path_is_file $midx-$checksum.bitmap && test_commit repack-3 && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx && + git repack -Adb --write-midx && test_path_is_file $midx && test_path_is_missing $midx-$checksum.bitmap && test_path_is_file $midx-$(midx_checksum $objdir).bitmap && test_commit repack-4 && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb && + git repack -Adb && find $objdir/pack -type f -name "multi-pack-index*" >files && test_must_be_empty files @@ -622,7 +625,6 @@ test_expect_success '--write-midx with preferred bitmap tips' ' git log --format="create refs/tags/%s/%s %H" HEAD >refs && git update-ref --stdin