mirrors/git
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
72441af7c4
git/diff.h
Kirill Smelkov 72441af7c4 tree-diff: rework diff_tree() to generate diffs for multiparent cases as well 
Previously diff_tree(), which is now named ll_diff_tree_sha1(), was
generating diff_filepair(s) for two trees t1 and t2, and that was
usually used for a commit as t1=HEAD~, and t2=HEAD - i.e. to see changes
a commit introduces.

In Git, however, we have fundamentally built flexibility in that a
commit can have many parents - 1 for a plain commit, 2 for a simple merge,
but also more than 2 for merging several heads at once.

For merges there is a so called combine-diff, which shows diff, a merge
introduces by itself, omitting changes done by any parent. That works
through first finding paths, that are different to all parents, and then
showing generalized diff, with separate columns for +/- for each parent.
The code lives in combine-diff.c .

There is an impedance mismatch, however, in that a commit could
generally have any number of parents, and that while diffing trees, we
divide cases for 2-tree diffs and more-than-2-tree diffs. I mean there
is no special casing for multiple parents commits in e.g.
revision-walker .

That impedance mismatch *hurts* *performance* *badly* for generating
combined diffs - in "combine-diff: optimize combine_diff_path
sets intersection" I've already removed some slowness from it, but from
the timings provided there, it could be seen, that combined diffs still
cost more than an order of magnitude more cpu time, compared to diff for
usual commits, and that would only be an optimistic estimate, if we take
into account that for e.g. linux.git there is only one merge for several
dozens of plain commits.

That slowness comes from the fact that currently, while generating
combined diff, a lot of time is spent computing diff(commit,commit^2)
just to only then intersect that huge diff to almost small set of files
from diff(commit,commit^1).

That's because at present, to compute combine-diff, for first finding
paths, that "every parent touches", we use the following combine-diff
property/definition:

D(A,P1...Pn) = D(A,P1) ^ ... ^ D(A,Pn)      (w.r.t. paths)

where

D(A,P1...Pn) is combined diff between commit A, and parents Pi

and

D(A,Pi) is usual two-tree diff Pi..A

So if any of that D(A,Pi) is huge, tracting 1 n-parent combine-diff as n
1-parent diffs and intersecting results will be slow.

And usually, for linux.git and other topic-based workflows, that
D(A,P2) is huge, because, if merge-base of A and P2, is several dozens
of merges (from A, via first parent) below, that D(A,P2) will be diffing
sum of merges from several subsystems to 1 subsystem.

The solution is to avoid computing n 1-parent diffs, and to find
changed-to-all-parents paths via scanning A's and all Pi's trees
simultaneously, at each step comparing their entries, and based on that
comparison, populate paths result, and deduce we could *skip*
*recursing* into subdirectories, if at least for 1 parent, sha1 of that
dir tree is the same as in A. That would save us from doing significant
amount of needless work.

Such approach is very similar to what diff_tree() does, only there we
deal with scanning only 2 trees simultaneously, and for n+1 tree, the
logic is a bit more complex:

D(T,P1...Pn) calculation scheme
-------------------------------

D(T,P1...Pn) = D(T,P1) ^ ... ^ D(T,Pn)	(regarding resulting paths set)

    D(T,Pj)		- diff between T..Pj
    D(T,P1...Pn)	- combined diff from T to parents P1,...,Pn

We start from all trees, which are sorted, and compare their entries in
lock-step:

     T     P1       Pn
     -     -        -
    |t|   |p1|     |pn|
    |-|   |--| ... |--|      imin = argmin(p1...pn)
    | |   |  |     |  |
    |-|   |--|     |--|
    |.|   |. |     |. |
     .     .        .
     .     .        .

at any time there could be 3 cases:

    1)  t < p[imin];
    2)  t > p[imin];
    3)  t = p[imin].

Schematic deduction of what every case means, and what to do, follows:

1)  t < p[imin]  ->  ∀j t ∉ Pj  ->  "+t" ∈ D(T,Pj)  ->  D += "+t";  t↓

2)  t > p[imin]

    2.1) ∃j: pj > p[imin]  ->  "-p[imin]" ∉ D(T,Pj)  ->  D += ø;  ∀ pi=p[imin]  pi↓
    2.2) ∀i  pi = p[imin]  ->  pi ∉ T  ->  "-pi" ∈ D(T,Pi)  ->  D += "-p[imin]";  ∀i pi↓

3)  t = p[imin]

    3.1) ∃j: pj > p[imin]  ->  "+t" ∈ D(T,Pj)  ->  only pi=p[imin] remains to investigate
    3.2) pi = p[imin]  ->  investigate δ(t,pi)
     |
     |
     v

    3.1+3.2) looking at δ(t,pi) ∀i: pi=p[imin] - if all != ø  ->

                      ⎧δ(t,pi)  - if pi=p[imin]
             ->  D += ⎨
                      ⎩"+t"     - if pi>p[imin]

    in any case t↓  ∀ pi=p[imin]  pi↓

~

For comparison, here is how diff_tree() works:

D(A,B) calculation scheme
-------------------------

    A     B
    -     -
   |a|   |b|    a < b   ->  a ∉ B   ->   D(A,B) +=  +a    a↓
   |-|   |-|    a > b   ->  b ∉ A   ->   D(A,B) +=  -b    b↓
   | |   | |    a = b   ->  investigate δ(a,b)            a↓ b↓
   |-|   |-|
   |.|   |.|
    .     .
    .     .

~~~~~~~~

This patch generalizes diff tree-walker to work with arbitrary number of
parents as described above - i.e. now there is a resulting tree t, and
some parents trees tp[i] i=[0..nparent). The generalization builds on
the fact that usual diff

D(A,B)

is by definition the same as combined diff

D(A,[B]),

so if we could rework the code for common case and make it be not slower
for nparent=1 case, usual diff(t1,t2) generation will not be slower, and
multiparent diff tree-walker would greatly benefit generating
combine-diff.

What we do is as follows:

1) diff tree-walker ll_diff_tree_sha1() is internally reworked to be
   a paths generator (new name diff_tree_paths()), with each generated path
   being `struct combine_diff_path` with info for path, new sha1,mode and for
   every parent which sha1,mode it was in it.

2) From that info, we can still generate usual diff queue with
   struct diff_filepairs, via "exporting" generated
   combine_diff_path, if we know we run for nparent=1 case.
   (see emit_diff() which is now named emit_diff_first_parent_only())

3) In order for diff_can_quit_early(), which checks

       DIFF_OPT_TST(opt, HAS_CHANGES))

   to work, that exporting have to be happening not in bulk, but
   incrementally, one diff path at a time.

   For such consumers, there is a new callback in diff_options
   introduced:

       ->pathchange(opt, struct combine_diff_path *)

   which, if set to !NULL, is called for every generated path.

   (see new compat ll_diff_tree_sha1() wrapper around new paths
    generator for setup)

4) The paths generation itself, is reworked from previous
   ll_diff_tree_sha1() code according to "D(A,P1...Pn) calculation
   scheme" provided above:

   On the start we allocate [nparent] arrays in place what was
   earlier just for one parent tree.

   then we just generalize loops, and comparison according to the
   algorithm.

Some notes(*):

1) alloca(), for small arrays, is used for "runs not slower for
   nparent=1 case than before" goal - if we change it to xmalloc()/free()
   the timings get ~1% worse. For alloca() we use just-introduced
   xalloca/xalloca_free compatibility wrappers, so it should not be a
   portability problem.

2) For every parent tree, we need to keep a tag, whether entry from that
   parent equals to entry from minimal parent. For performance reasons I'm
   keeping that tag in entry's mode field in unused bit - see S_IFXMIN_NEQ.
   Not doing so, we'd need to alloca another [nparent] array, which hurts
   performance.

3) For emitted paths, memory could be reused, if we know the path was
   processed via callback and will not be needed later. We use efficient
   hand-made realloc-style path_appendnew(), that saves us from ~1-1.5%
   of potential additional slowdown.

4) goto(s) are used in several places, as the code executes a little bit
   faster with lowered register pressure.

Also

- we should now check for FIND_COPIES_HARDER not only when two entries
  names are the same, and their hashes are equal, but also for a case,
  when a path was removed from some of all parents having it.

  The reason is, if we don't, that path won't be emitted at all (see
  "a > xi" case), and we'll just skip it, and FIND_COPIES_HARDER wants
  all paths - with diff or without - to be emitted, to be later analyzed
  for being copies sources.

  The new check is only necessary for nparent >1, as for nparent=1 case
  xmin_eqtotal always =1 =nparent, and a path is always added to diff as
  removal.

~~~~~~~~

Timings for

    # without -c, i.e. testing only nparent=1 case
    `git log --raw --no-abbrev --no-renames`

before and after the patch are as follows:

                navy.git        linux.git v3.10..v3.11

    before      0.611s          1.889s
    after       0.619s          1.907s
    slowdown    1.3%            0.9%

This timings show we did no harm to usual diff(tree1,tree2) generation.
From the table we can see that we actually did ~1% slowdown, but I think
I've "earned" that 1% in the previous patch ("tree-diff: reuse base
str(buf) memory on sub-tree recursion", HEAD~~) so for nparent=1 case,
net timings stays approximately the same.

The output also stayed the same.

(*) If we revert 1)-4) to more usual techniques, for nparent=1 case,
    we'll get ~2-2.5% of additional slowdown, which I've tried to avoid, as
   "do no harm for nparent=1 case" rule.

For linux.git, combined diff will run an order of magnitude faster and
appropriate timings will be provided in the next commit, as we'll be
taking advantage of the new diff tree-walker for combined-diff
generation there.

P.S. and combined diff is not some exotic/for-play-only stuff - for
example for a program I write to represent Git archives as readonly
filesystem, there is initial scan with

    `git log --reverse --raw --no-abbrev --no-renames -c`

to extract log of what was created/changed when, as a result building a
map

    {}  sha1    ->  in which commit (and date) a content was added

that `-c` means also show combined diff for merges, and without them, if
a merge is non-trivial (merges changes from two parents with both having
separate changes to a file), or an evil one, the map will not be full,
i.e. some valid sha1 would be absent from it.

That case was my initial motivation for combined diffs speedup.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-04-07 14:40:46 -07:00

362 lines
12 KiB
C
Raw Blame History

/*
* Copyright (C) 2005 Junio C Hamano
*/
#ifndef DIFF_H
#define DIFF_H
#include "tree-walk.h"
#include "pathspec.h"
struct rev_info;
struct diff_options;
struct diff_queue_struct;
struct strbuf;
struct diff_filespec;
struct userdiff_driver;
struct sha1_array;
struct commit;
struct combine_diff_path;
typedef int (*pathchange_fn_t)(struct diff_options *options,
struct combine_diff_path *path);
typedef void (*change_fn_t)(struct diff_options *options,
unsigned old_mode, unsigned new_mode,
const unsigned char *old_sha1,
const unsigned char *new_sha1,
int old_sha1_valid, int new_sha1_valid,
const char *fullpath,
unsigned old_dirty_submodule, unsigned new_dirty_submodule);
typedef void (*add_remove_fn_t)(struct diff_options *options,
int addremove, unsigned mode,
const unsigned char *sha1,
int sha1_valid,
const char *fullpath, unsigned dirty_submodule);
typedef void (*diff_format_fn_t)(struct diff_queue_struct *q,
struct diff_options *options, void *data);
typedef struct strbuf *(*diff_prefix_fn_t)(struct diff_options *opt, void *data);
#define DIFF_FORMAT_RAW 0x0001
#define DIFF_FORMAT_DIFFSTAT 0x0002
#define DIFF_FORMAT_NUMSTAT 0x0004
#define DIFF_FORMAT_SUMMARY 0x0008
#define DIFF_FORMAT_PATCH 0x0010
#define DIFF_FORMAT_SHORTSTAT 0x0020
#define DIFF_FORMAT_DIRSTAT 0x0040
/* These override all above */
#define DIFF_FORMAT_NAME 0x0100
#define DIFF_FORMAT_NAME_STATUS 0x0200
#define DIFF_FORMAT_CHECKDIFF 0x0400
/* Same as output_format = 0 but we know that -s flag was given
* and we should not give default value to output_format.
*/
#define DIFF_FORMAT_NO_OUTPUT 0x0800
#define DIFF_FORMAT_CALLBACK 0x1000
#define DIFF_OPT_RECURSIVE (1 << 0)
#define DIFF_OPT_TREE_IN_RECURSIVE (1 << 1)
#define DIFF_OPT_BINARY (1 << 2)
#define DIFF_OPT_TEXT (1 << 3)
#define DIFF_OPT_FULL_INDEX (1 << 4)
#define DIFF_OPT_SILENT_ON_REMOVE (1 << 5)
#define DIFF_OPT_FIND_COPIES_HARDER (1 << 6)
#define DIFF_OPT_FOLLOW_RENAMES (1 << 7)
#define DIFF_OPT_RENAME_EMPTY (1 << 8)
/* (1 << 9) unused */
#define DIFF_OPT_HAS_CHANGES (1 << 10)
#define DIFF_OPT_QUICK (1 << 11)
#define DIFF_OPT_NO_INDEX (1 << 12)
#define DIFF_OPT_ALLOW_EXTERNAL (1 << 13)
#define DIFF_OPT_EXIT_WITH_STATUS (1 << 14)
#define DIFF_OPT_REVERSE_DIFF (1 << 15)
#define DIFF_OPT_CHECK_FAILED (1 << 16)
#define DIFF_OPT_RELATIVE_NAME (1 << 17)
#define DIFF_OPT_IGNORE_SUBMODULES (1 << 18)
#define DIFF_OPT_DIRSTAT_CUMULATIVE (1 << 19)
#define DIFF_OPT_DIRSTAT_BY_FILE (1 << 20)
#define DIFF_OPT_ALLOW_TEXTCONV (1 << 21)
#define DIFF_OPT_DIFF_FROM_CONTENTS (1 << 22)
#define DIFF_OPT_SUBMODULE_LOG (1 << 23)
#define DIFF_OPT_DIRTY_SUBMODULES (1 << 24)
#define DIFF_OPT_IGNORE_UNTRACKED_IN_SUBMODULES (1 << 25)
#define DIFF_OPT_IGNORE_DIRTY_SUBMODULES (1 << 26)
#define DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG (1 << 27)
#define DIFF_OPT_DIRSTAT_BY_LINE (1 << 28)
#define DIFF_OPT_FUNCCONTEXT (1 << 29)
#define DIFF_OPT_PICKAXE_IGNORE_CASE (1 << 30)
#define DIFF_OPT_TST(opts, flag) ((opts)->flags & DIFF_OPT_##flag)
#define DIFF_OPT_TOUCHED(opts, flag) ((opts)->touched_flags & DIFF_OPT_##flag)
#define DIFF_OPT_SET(opts, flag) (((opts)->flags |= DIFF_OPT_##flag),((opts)->touched_flags |= DIFF_OPT_##flag))
#define DIFF_OPT_CLR(opts, flag) (((opts)->flags &= ~DIFF_OPT_##flag),((opts)->touched_flags |= DIFF_OPT_##flag))
#define DIFF_XDL_TST(opts, flag) ((opts)->xdl_opts & XDF_##flag)
#define DIFF_XDL_SET(opts, flag) ((opts)->xdl_opts |= XDF_##flag)
#define DIFF_XDL_CLR(opts, flag) ((opts)->xdl_opts &= ~XDF_##flag)
#define DIFF_WITH_ALG(opts, flag) (((opts)->xdl_opts & ~XDF_DIFF_ALGORITHM_MASK) | XDF_##flag)
enum diff_words_type {
DIFF_WORDS_NONE = 0,
DIFF_WORDS_PORCELAIN,
DIFF_WORDS_PLAIN,
DIFF_WORDS_COLOR
};
struct diff_options {
const char *orderfile;
const char *pickaxe;
const char *single_follow;
const char *a_prefix, *b_prefix;
unsigned flags;
unsigned touched_flags;
/* diff-filter bits */
unsigned int filter;
int use_color;
int context;
int interhunkcontext;
int break_opt;
int detect_rename;
int irreversible_delete;
int skip_stat_unmatch;
int line_termination;
int output_format;
int pickaxe_opts;
int rename_score;
int rename_limit;
int needed_rename_limit;
int degraded_cc_to_c;
int show_rename_progress;
int dirstat_permille;
int setup;
int abbrev;
const char *prefix;
int prefix_length;
const char *stat_sep;
long xdl_opts;
int stat_width;
int stat_name_width;
int stat_graph_width;
int stat_count;
const char *word_regex;
enum diff_words_type word_diff;
/* this is set by diffcore for DIFF_FORMAT_PATCH */
int found_changes;
/* to support internal diff recursion by --follow hack*/
int found_follow;
void (*set_default)(struct diff_options *);
FILE *file;
int close_file;
struct pathspec pathspec;
pathchange_fn_t pathchange;
change_fn_t change;
add_remove_fn_t add_remove;
diff_format_fn_t format_callback;
void *format_callback_data;
diff_prefix_fn_t output_prefix;
int output_prefix_length;
void *output_prefix_data;
int diff_path_counter;
};
enum color_diff {
DIFF_RESET = 0,
DIFF_PLAIN = 1,
DIFF_METAINFO = 2,
DIFF_FRAGINFO = 3,
DIFF_FILE_OLD = 4,
DIFF_FILE_NEW = 5,
DIFF_COMMIT = 6,
DIFF_WHITESPACE = 7,
DIFF_FUNCINFO = 8
};
const char *diff_get_color(int diff_use_color, enum color_diff ix);
#define diff_get_color_opt(o, ix) \
diff_get_color((o)->use_color, ix)
const char *diff_line_prefix(struct diff_options *);
extern const char mime_boundary_leader[];
extern struct combine_diff_path *diff_tree_paths(
struct combine_diff_path *p, const unsigned char *sha1,
const unsigned char **parent_sha1, int nparent,
struct strbuf *base, struct diff_options *opt);
extern int diff_tree_sha1(const unsigned char *old, const unsigned char *new,
const char *base, struct diff_options *opt);
extern int diff_root_tree_sha1(const unsigned char *new, const char *base,
struct diff_options *opt);
struct combine_diff_path {
struct combine_diff_path *next;
char *path;
unsigned int mode;
unsigned char sha1[20];
struct combine_diff_parent {
char status;
unsigned int mode;
unsigned char sha1[20];
} parent[FLEX_ARRAY];
};
#define combine_diff_path_size(n, l) \
(sizeof(struct combine_diff_path) + \
sizeof(struct combine_diff_parent) * (n) + (l) + 1)
extern void show_combined_diff(struct combine_diff_path *elem, int num_parent,
int dense, struct rev_info *);
extern void diff_tree_combined(const unsigned char *sha1, const struct sha1_array *parents, int dense, struct rev_info *rev);
extern void diff_tree_combined_merge(const struct commit *commit, int dense, struct rev_info *rev);
void diff_set_mnemonic_prefix(struct diff_options *options, const char *a, const char *b);
extern int diff_can_quit_early(struct diff_options *);
extern void diff_addremove(struct diff_options *,
int addremove,
unsigned mode,
const unsigned char *sha1,
int sha1_valid,
const char *fullpath, unsigned dirty_submodule);
extern void diff_change(struct diff_options *,
unsigned mode1, unsigned mode2,
const unsigned char *sha1,
const unsigned char *sha2,
int sha1_valid,
int sha2_valid,
const char *fullpath,
unsigned dirty_submodule1, unsigned dirty_submodule2);
extern struct diff_filepair *diff_unmerge(struct diff_options *, const char *path);
#define DIFF_SETUP_REVERSE 1
#define DIFF_SETUP_USE_CACHE 2
#define DIFF_SETUP_USE_SIZE_CACHE 4
/*
* Poor man's alternative to parse-option, to allow both stuck form
* (--option=value) and separate form (--option value).
*/
extern int parse_long_opt(const char *opt, const char **argv,
const char **optarg);
extern int git_diff_basic_config(const char *var, const char *value, void *cb);
extern int git_diff_ui_config(const char *var, const char *value, void *cb);
extern void diff_setup(struct diff_options *);
extern int diff_opt_parse(struct diff_options *, const char **, int);
extern void diff_setup_done(struct diff_options *);
#define DIFF_DETECT_RENAME 1
#define DIFF_DETECT_COPY 2
#define DIFF_PICKAXE_ALL 1
#define DIFF_PICKAXE_REGEX 2
#define DIFF_PICKAXE_KIND_S 4 /* traditional plumbing counter */
#define DIFF_PICKAXE_KIND_G 8 /* grep in the patch */
extern void diffcore_std(struct diff_options *);
extern void diffcore_fix_diff_index(struct diff_options *);
#define COMMON_DIFF_OPTIONS_HELP \
"\ncommon diff options:\n" \
" -z output diff-raw with lines terminated with NUL.\n" \
" -p output patch format.\n" \
" -u synonym for -p.\n" \
" --patch-with-raw\n" \
" output both a patch and the diff-raw format.\n" \
" --stat show diffstat instead of patch.\n" \
" --numstat show numeric diffstat instead of patch.\n" \
" --patch-with-stat\n" \
" output a patch and prepend its diffstat.\n" \
" --name-only show only names of changed files.\n" \
" --name-status show names and status of changed files.\n" \
" --full-index show full object name on index lines.\n" \
" --abbrev=<n> abbreviate object names in diff-tree header and diff-raw.\n" \
" -R swap input file pairs.\n" \
" -B detect complete rewrites.\n" \
" -M detect renames.\n" \
" -C detect copies.\n" \
" --find-copies-harder\n" \
" try unchanged files as candidate for copy detection.\n" \
" -l<n> limit rename attempts up to <n> paths.\n" \
" -O<file> reorder diffs according to the <file>.\n" \
" -S<string> find filepair whose only one side contains the string.\n" \
" --pickaxe-all\n" \
" show all files diff when -S is used and hit is found.\n" \
" -a --text treat all files as text.\n"
extern int diff_queue_is_empty(void);
extern void diff_flush(struct diff_options*);
extern void diff_warn_rename_limit(const char *varname, int needed, int degraded_cc);
/* diff-raw status letters */
#define DIFF_STATUS_ADDED 'A'
#define DIFF_STATUS_COPIED 'C'
#define DIFF_STATUS_DELETED 'D'
#define DIFF_STATUS_MODIFIED 'M'
#define DIFF_STATUS_RENAMED 'R'
#define DIFF_STATUS_TYPE_CHANGED 'T'
#define DIFF_STATUS_UNKNOWN 'X'
#define DIFF_STATUS_UNMERGED 'U'
/* these are not diff-raw status letters proper, but used by
* diffcore-filter insn to specify additional restrictions.
*/
#define DIFF_STATUS_FILTER_AON '*'
#define DIFF_STATUS_FILTER_BROKEN 'B'
extern const char *diff_unique_abbrev(const unsigned char *, int);
/* do not report anything on removed paths */
#define DIFF_SILENT_ON_REMOVED 01
/* report racily-clean paths as modified */
#define DIFF_RACY_IS_MODIFIED 02
extern int run_diff_files(struct rev_info *revs, unsigned int option);
extern int run_diff_index(struct rev_info *revs, int cached);
extern int do_diff_cache(const unsigned char *, struct diff_options *);
extern int diff_flush_patch_id(struct diff_options *, unsigned char *);
extern int diff_result_code(struct diff_options *, int);
extern void diff_no_index(struct rev_info *, int, const char **, const char *);
extern int index_differs_from(const char *def, int diff_flags);
extern size_t fill_textconv(struct userdiff_driver *driver,
struct diff_filespec *df,
char **outbuf);
extern struct userdiff_driver *get_textconv(struct diff_filespec *one);
extern int parse_rename_score(const char **cp_p);
extern long parse_algorithm_value(const char *value);
extern void handle_deprecated_show_diff_q(struct diff_options *);
extern int print_stat_summary(FILE *fp, int files,
int insertions, int deletions);
extern void setup_diff_pager(struct diff_options *);
#endif /* DIFF_H */
Reference in New Issue View Git Blame Copy Permalink