build(rust): vendor googletest crate

third_party/crates/regex-automata-0.4.18/src/util/empty.rs

@@ -0,0 +1,265 @@
+/*!
+This module provides helper routines for dealing with zero-width matches.
+
+The main problem being solved here is this:
+
+1. The caller wants to search something that they know is valid UTF-8, such
+as a Rust `&str`.
+2. The regex used by the caller can match the empty string. For example, `a*`.
+3. The caller should never get match offsets returned that occur within the
+encoding of a UTF-8 codepoint. It is logically incorrect, and also means that,
+e.g., slicing the `&str` at those offsets will lead to a panic.
+
+So the question here is, how do we prevent the caller from getting match
+offsets that split a codepoint? For example, strictly speaking, the regex `a*`
+matches `☃` at the positions `[0, 0]`, `[1, 1]`, `[2, 2]` and `[3, 3]` since
+the UTF-8 encoding of `☃` is `\xE2\x98\x83`. In particular, the `NFA` that
+underlies all of the matching engines in this crate doesn't have anything in
+its state graph that prevents matching between UTF-8 code units. Indeed, any
+engine derived from the `NFA` will match at those positions by virtue of the
+fact that the `NFA` is byte oriented. That is, its transitions are defined over
+bytes and the matching engines work by proceeding one byte at a time.
+
+(An alternative architecture would be to define the transitions in an `NFA`
+over codepoints, or `char`. And then make the matching engines proceed by
+decoding one codepoint at a time. This is a viable strategy, but it doesn't
+work for DFA matching engines because designing a fast and memory efficient
+transition table for an alphabet as large as Unicode is quite difficult. More
+to the point, the top-level `regex` crate supports matching on arbitrary bytes
+when Unicode mode is disabled and one is searching a `&[u8]`. So in that case,
+you can't just limit yourself to decoding codepoints and matching those. You
+really do need to be able to follow byte oriented transitions on the `NFA`.)
+
+In an older version of the regex crate, we handled this case not in the regex
+engine, but in the iterators over matches. Namely, since this case only arises
+when the match is empty, we "just" incremented the next starting position
+of the search by `N`, where `N` is the length of the codepoint encoded at
+the current position. The alternative or more "natural" solution of just
+incrementing by `1` would result in executing a search of `a*` on `☃` like
+this:
+
+* Start search at `0`.
+* Found match at `[0, 0]`.
+* Next start position is `0`.
+* To avoid an infinite loop, since it's an empty match, increment by `1`.
+* Start search at `1`.
+* Found match at `[1, 1]`. Oops.
+
+But if we instead incremented by `3` (the length in bytes of `☃`), then we get
+the following:
+
+* Start search at `0`.
+* Found match at `[0, 0]`.
+* Next start position is `0`.
+* To avoid an infinite loop, since it's an empty match, increment by `3`.
+* Start search at `3`.
+* Found match at `[3, 3]`.
+
+And we get the correct result. But does this technique work in all cases?
+Crucially, it requires that a zero-width match that splits a codepoint never
+occurs beyond the starting position of the search. Because if it did, merely
+incrementing the start position by the number of bytes in the codepoint at
+the current position wouldn't be enough. A zero-width match could just occur
+anywhere. It turns out that it is _almost_ true. We can convince ourselves by
+looking at all possible patterns that can match the empty string:
+
+* Patterns like `a*`, `a{0}`, `(?:)`, `a|` and `|a` all unconditionally match
+the empty string. That is, assuming there isn't an `a` at the current position,
+they will all match the empty string at the start of a search. There is no way
+to move past it because any other match would not be "leftmost."
+* `^` only matches at the beginning of the haystack, where the start position
+is `0`. Since we know we're searching valid UTF-8 (if it isn't valid UTF-8,
+then this entire problem goes away because it implies your string type supports
+invalid UTF-8 and thus must deal with offsets that not only split a codepoint
+but occur in entirely invalid UTF-8 somehow), it follows that `^` never matches
+between the code units of a codepoint because the start of a valid UTF-8 string
+is never within the encoding of a codepoint.
+* `$` basically the same logic as `^`, but for the end of a string. A valid
+UTF-8 string can't have an incomplete codepoint at the end of it.
+* `(?m:^)` follows similarly to `^`, but it can match immediately following
+a `\n`. However, since a `\n` is always a codepoint itself and can never
+appear within a codepoint, it follows that the position immediately following
+a `\n` in a string that is valid UTF-8 is guaranteed to not be between the
+code units of another codepoint. (One caveat here is that the line terminator
+for multi-line anchors can now be changed to any arbitrary byte, including
+things like `\x98` which might occur within a codepoint. However, this wasn't
+supported by the old regex crate. If it was, it pose the same problems as
+`(?-u:\B)`, as we'll discuss below.)
+* `(?m:$)` a similar argument as for `(?m:^)`. The only difference is that a
+`(?m:$)` matches just before a `\n`. But the same argument applies.
+* `(?Rm:^)` and `(?Rm:$)` weren't supported by the old regex crate, but the
+CRLF aware line anchors follow a similar argument as for `(?m:^)` and `(?m:$)`.
+Namely, since they only ever match at a boundary where one side is either a
+`\r` or a `\n`, neither of which can occur within a codepoint.
+* `\b` only matches at positions where both sides are valid codepoints, so
+this cannot split a codepoint.
+* `\B`, like `\b`, also only matches at positions where both sides are valid
+codepoints. So this cannot split a codepoint either.
+* `(?-u:\b)` matches only at positions where at least one side of it is an ASCII
+word byte. Since ASCII bytes cannot appear as code units in non-ASCII codepoints
+(one of the many amazing qualities of UTF-8), it follows that this too cannot
+split a codepoint.
+* `(?-u:\B)` finally represents a problem. It can matches between *any* two
+bytes that are either both word bytes or non-word bytes. Since code units like
+`\xE2` and `\x98` (from the UTF-8 encoding of `☃`) are both non-word bytes,
+`(?-u:\B)` will match at the position between them.
+
+Thus, our approach of incrementing one codepoint at a time after seeing an
+empty match is flawed because `(?-u:\B)` can result in an empty match that
+splits a codepoint at a position past the starting point of a search. For
+example, searching `(?-u:\B)` on `a☃` would produce the following matches: `[2,
+2]`, `[3, 3]` and `[4, 4]`. The positions at `0` and `1` don't match because
+they correspond to word boundaries since `a` is an ASCII word byte.
+
+So what did the old regex crate do to avoid this? It banned `(?-u:\B)` from
+regexes that could match `&str`. That might sound extreme, but a lot of other
+things were banned too. For example, all of `(?-u:.)`, `(?-u:[^a])` and
+`(?-u:\W)` can match invalid UTF-8 too, including individual code units with a
+codepoint. The key difference is that those expressions could never produce an
+empty match. That ban happens when translating an `Ast` to an `Hir`, because
+that process that reason about whether an `Hir` can produce *non-empty* matches
+at invalid UTF-8 boundaries. Bottom line though is that we side-stepped the
+`(?-u:\B)` issue by banning it.
+
+If banning `(?-u:\B)` were the only issue with the old regex crate's approach,
+then I probably would have kept it. `\B` is rarely used, so it's not such a big
+deal to have to work-around it. However, the problem with the above approach
+is that it doesn't compose. The logic for avoiding splitting a codepoint only
+lived in the iterator, which means if anyone wants to implement their own
+iterator over regex matches, they have to deal with this extremely subtle edge
+case to get full correctness.
+
+Instead, in this crate, we take the approach of pushing this complexity down
+to the lowest layers of each regex engine. The approach is pretty simple:
+
+* If this corner case doesn't apply, don't do anything. (For example, if UTF-8
+mode isn't enabled or if the regex cannot match the empty string.)
+* If an empty match is reported, explicitly check if it splits a codepoint.
+* If it doesn't, we're done, return the match.
+* If it does, then ignore the match and re-run the search.
+* Repeat the above process until the end of the haystack is reached or a match
+is found that doesn't split a codepoint or isn't zero width.
+
+And that's pretty much what this module provides. Every regex engine uses these
+methods in their lowest level public APIs, but just above the layer where
+their internal engine is used. That way, all regex engines can be arbitrarily
+composed without worrying about handling this case, and iterators don't need to
+handle it explicitly.
+
+(It turns out that a new feature I added, support for changing the line
+terminator in a regex to any arbitrary byte, also provokes the above problem.
+Namely, the byte could be invalid UTF-8 or a UTF-8 continuation byte. So that
+support would need to be limited or banned when UTF-8 mode is enabled, just
+like we did for `(?-u:\B)`. But thankfully our more robust approach in this
+crate handles that case just fine too.)
+*/
+
+use crate::util::search::{Input, MatchError};
+
+#[cold]
+#[inline(never)]
+pub(crate) fn skip_splits_fwd<T, F>(
+    input: &Input<'_>,
+    init_value: T,
+    match_offset: usize,
+    find: F,
+) -> Result<Option<T>, MatchError>
+where
+    F: FnMut(&Input<'_>) -> Result<Option<(T, usize)>, MatchError>,
+{
+    skip_splits(true, input, init_value, match_offset, find)
+}
+
+#[cold]
+#[inline(never)]
+pub(crate) fn skip_splits_rev<T, F>(
+    input: &Input<'_>,
+    init_value: T,
+    match_offset: usize,
+    find: F,
+) -> Result<Option<T>, MatchError>
+where
+    F: FnMut(&Input<'_>) -> Result<Option<(T, usize)>, MatchError>,
+{
+    skip_splits(false, input, init_value, match_offset, find)
+}
+
+fn skip_splits<T, F>(
+    forward: bool,
+    input: &Input<'_>,
+    init_value: T,
+    mut match_offset: usize,
+    mut find: F,
+) -> Result<Option<T>, MatchError>
+where
+    F: FnMut(&Input<'_>) -> Result<Option<(T, usize)>, MatchError>,
+{
+    // If our config says to do an anchored search, then we're definitely
+    // done. We just need to determine whether we have a valid match or
+    // not. If we don't, then we're not allowed to continue, so we report
+    // no match.
+    //
+    // This is actually quite a subtle correctness thing. The key here is
+    // that if we got an empty match that splits a codepoint after doing an
+    // anchored search in UTF-8 mode, then that implies that we must have
+    // *started* the search at a location that splits a codepoint. This
+    // follows from the fact that if a match is reported from an anchored
+    // search, then the start offset of the match *must* match the start
+    // offset of the search.
+    //
+    // It also follows that no other non-empty match is possible. For
+    // example, you might write a regex like '(?:)|SOMETHING' and start its
+    // search in the middle of a codepoint. The first branch is an empty
+    // regex that will bubble up a match at the first position, and then
+    // get rejected here and report no match. But what if 'SOMETHING' could
+    // have matched? We reason that such a thing is impossible, because
+    // if it does, it must report a match that starts in the middle of a
+    // codepoint. This in turn implies that a match is reported whose span
+    // does not correspond to valid UTF-8, and this breaks the promise
+    // made when UTF-8 mode is enabled. (That promise *can* be broken, for
+    // example, by enabling UTF-8 mode but building an by hand NFA that
+    // produces non-empty matches that span invalid UTF-8. This is an unchecked
+    // but documented precondition violation of UTF-8 mode, and is documented
+    // to have unspecified behavior.)
+    //
+    // I believe this actually means that if an anchored search is run, and
+    // UTF-8 mode is enabled and the start position splits a codepoint,
+    // then it is correct to immediately report no match without even
+    // executing the regex engine. But it doesn't really seem worth writing
+    // out that case in every regex engine to save a tiny bit of work in an
+    // extremely pathological case, so we just handle it here.
+    if input.get_anchored().is_anchored() {
+        return Ok(if input.is_char_boundary(match_offset) {
+            Some(init_value)
+        } else {
+            None
+        });
+    }
+    // Otherwise, we have an unanchored search, so just keep looking for
+    // matches until we have one that does not split a codepoint or we hit
+    // EOI.
+    let mut value = init_value;
+    let mut input = input.clone();
+    while !input.is_char_boundary(match_offset) {
+        if forward {
+            // The unwrap is OK here because overflowing usize while
+            // iterating over a slice is impossible, at it would require
+            // a slice of length greater than isize::MAX, which is itself
+            // impossible.
+            input.set_start(input.start().checked_add(1).unwrap());
+        } else {
+            input.set_end(match input.end().checked_sub(1) {
+                None => return Ok(None),
+                Some(end) => end,
+            });
+        }
+        match find(&input)? {
+            None => return Ok(None),
+            Some((new_value, new_match_end)) => {
+                value = new_value;
+                match_offset = new_match_end;
+            }
+        }
+    }
+    Ok(Some(value))
+}